Contribute Morphuntion from Apple as open source software #35

grhoten · 2024-10-30T22:16:13Z

See the LICENSE.txt for copyright and permission details.

This contribution should resolve the following issues: #5, #6, #7, #11, #12, #13, #15, #17, #18, #19
This contribution is also related to the following issues without fully resolving the issues: 3, 4, 8, 10, 21, 23, 24, 25
This contribution also has an implementation that addresses these CLDR issues: 13025, 13563

CLAassistant · 2024-10-30T22:16:19Z

All committers have signed the CLA.

…CENSE.txt for copyright and permission details. This contribution should resolve the following issues: #5, #6, #7, #11, #12, #13, #15, #17, #18, #19 This contribution is also related to the following issues without fully resolving the issues: 3, 4, 8, 10, 21, 23, 24, 25 This contribution also has an implementation that addresses these CLDR issues: 13025, 13563

nciric

First round. I'll go into actual code in the next.

nciric · 2024-11-01T16:36:53Z

morphuntion/CMakeLists.txt

@@ -0,0 +1,118 @@
+#
+# Copyright 2018-2024 Apple Inc. All rights reserved.


Do we keep the Apple copyright message? Mostly a question for Anne and George.

nciric · 2024-11-01T16:40:21Z

morphuntion/CMakeLists.txt

+    add_link_options(-g -fprofile-instr-generate -fcoverage-mapping)
+endif()
+
+#Set these warning properties on a project level


Nit: Some comments start with capital letters, some don't. Some have space between # and text and some don't.

I have changed this file to make the comments more consistent in style.

morphuntion/README.md

nciric · 2024-11-01T16:48:00Z

morphuntion/build.gradle

+
+plugins {
+    id 'net.ltgt.errorprone' version '1.1.1' apply false
+    id 'maven-publish'


Dumb question - what do we use Java for in this project?

It's dictionary-parser. Denny will likely want to look at that code. It can be reimplemented in another programming language when using wikidata as a data source, but I wanted to provide the tool source code so that the logic behind generating the lexical dictionaries is a lot clearer. The tool will likely be deleted once wikidata can be ingested.

This also had a Java wrapper through JNI. That implementation was excluded from this contribution at this time to expedite making this implementation open source.

nciric · 2024-11-01T17:02:12Z

morphuntion/resources/org/unicode/morphuntion/inflection/pronoun.properties

+#
+# Copyright 2022-2024 Apple Inc. All rights reserved.
+#
+yue_HK=yue_Hant


Is this just mapping one form of LangID to another? Why (comment)?

It's an aliasing scheme. In this case, we're making the script explicit for a given region.

nciric · 2024-11-01T17:04:39Z

morphuntion/resources/org/unicode/morphuntion/tokenizer/config_ar.properties

+tokenizer.normalize=\u062A
+tokenizer.replace=\u0629
+
+#tokenizer.decompound=(\u0648|\u0641)?(\u0628|\u0643|\u0644)?(\u0627\u0644)?(.+?)(\u064A|\u0646\u064A|\u0643|\u0646\u0627|\u0643\u0645|\u0643\u0646|\u0647|\u0647\u0627|\u0643\u0646|\u0647\u0645|\u0647\u0646)?


This has been removed. It was informative about the previous implementation, but it's not helpful anymore.

nciric · 2024-11-01T17:07:34Z

morphuntion/src/icu4cxx/BreakIterator.hpp

+/*
+ * Copyright 2016-2024 Apple Inc. All rights reserved.
+ */
+// This is a generated file. Do not edit this file.


Do we care how it's generated? If yes, some epointers to the generator would be good.

This is generated with the generateICUWrapper tool. It takes an existing ICU installation, and generates C++ wrappers around the C API. This provides a natural adaptor for C++ API, but with the stability of the C ABI.

The file can be regenerated when ICU adds new C API.

nciric · 2024-11-01T17:08:39Z

morphuntion/src/icu4cxx/BreakIterator.hpp

+BreakIterator(UBreakIteratorType type, const char * locale, std::u16string_view text) {
+    UErrorCode ec = U_ZERO_ERROR;
+    wrappee_ = ubrk_open(type, locale, (const UChar *)text.data(), (int32_t)text.size(), &ec);
+    ::morphuntion::exception::ICUException::throwOnFailure(ec);


Some companies don't allow exceptions in code. Do these bubble up to top or are they always caught and then API returns an error?

They're not caught on the C++ API. The C API wrappers around the C++ code converts the exceptions to CFErrorRef. If you don't want C++ exceptions, use the C API. If you want C++ exceptions, use the C++ API. C++ library code also generates exceptions. So you can't completely omit them from C++.

nciric · 2024-11-01T17:11:24Z

morphuntion/src/morphuntion/Object.hpp

+/*
+ * Copyright 2016-2024 Apple Inc. All rights reserved.
+ */
+#pragma once


Hopefully this works everywhere we care about (we use guards instead). https://en.wikipedia.org/wiki/Pragma_once

That page doesn't show any compilers that don't support it. I guess the only issue would be if you want to use an old compiler. From a maintenance point of view, this is much more reliable than using macros.

nciric

GitHub makes reviewing 800+ files PR a nightmare (everything is super laggy and slow). Let's land this and then go over important details in follow up PRs.
I guess current legal discussion is preventing the merge.

nciric · 2024-11-01T19:41:45Z

morphuntion/tools/generateICUWrapper/generateICUWrapper.cpp

@@ -0,0 +1,551 @@
+//
+//  main.cpp
+//  icu4cxx_generate


I wonder if, long term, we could convert this into Python or some kind of a template language e.g. https://github.com/kainjow/Mustache

Yes, rewriting this in another programming language is fine. It’s rarely run anyway.

grhoten force-pushed the morphuntion branch from 96540c7 to 7c4b9d5 Compare October 30, 2024 22:29

nciric reviewed Nov 1, 2024

View reviewed changes

grhoten added 2 commits November 1, 2024 14:24

Updates based on review comments.

06935bd

Improve documentation, and refine what is exposed as public API.

8b244d4

nciric approved these changes Nov 8, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Contribute Morphuntion from Apple as open source software #35

Contribute Morphuntion from Apple as open source software #35

grhoten commented Oct 30, 2024

CLAassistant commented Oct 30, 2024 •

edited

Loading

nciric left a comment

nciric Nov 1, 2024

nciric Nov 1, 2024

grhoten Nov 1, 2024

nciric Nov 1, 2024

grhoten Nov 1, 2024

nciric Nov 1, 2024

grhoten Nov 1, 2024

nciric Nov 1, 2024

grhoten Nov 1, 2024

nciric Nov 1, 2024

grhoten Nov 1, 2024

grhoten Nov 1, 2024

nciric Nov 1, 2024

grhoten Nov 1, 2024

nciric Nov 1, 2024

grhoten Nov 1, 2024

nciric left a comment

nciric Nov 1, 2024

grhoten Nov 8, 2024

		@@ -0,0 +1,118 @@
		#
		# Copyright 2018-2024 Apple Inc. All rights reserved.

Contribute Morphuntion from Apple as open source software #35

Are you sure you want to change the base?

Contribute Morphuntion from Apple as open source software #35

Conversation

grhoten commented Oct 30, 2024

CLAassistant commented Oct 30, 2024 • edited Loading

nciric left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nciric left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CLAassistant commented Oct 30, 2024 •

edited

Loading