UnicodeMath support

[suleman-uzair]

This PR supports UnicodeMath and the test case.

close plurimath/plurimath.org/issues/38
close plurimath/plurimath#316

[suleman-uzair]

I have tried some fixes, but I can't get the tests to pass.
I have faced some issues including but not limited to missing libraries that I have added but I can't get following error over the line.

can't find processor for rel_path: "./ox.so", abs_path: "/var/lib/gems/3.1.0/gems/ox-2.14.18/lib/ox.so",

@hmdne Can you please give it a look?

cc: @ronaldtse

[hmdne]

@suleman-uzair
Certain libraries, like nokogiri or ox won't work with Opal, due to them using C extensions a lot and Opal can't compile C to JS, only Ruby to JS. This is why we ported Plurimath to be able to work with oga, which we have successfully patched. So, if something requires one of those libraries, it's considered a bug and such a dependency must be dropped. In particular, there's a new Gem added: twitter-cldr-rb which is a huge library and it adds a lot of dependencies and uses a lot of assumptions (like - an ability to connect to an FTP server, or load a file from the filesystem, which web browsers won't be able to do).

So, what are the possibilities, I see two obvious ones:

1. Try to conditionally use either twitter-cldr-rb or twitter-cldr-js:

# backtick_javascript: true
# This particular thing is a macro that shields code from execution inside Opal. In this case - requirement, which will cause the entire suite to be compiled by Opal.
require "twitter-cldr" unless RUBY_ENGINE == 'opal'

if RUBY_ENGINE == 'opal'
  # Backticks/x-strings allow you to execute JavaScript code, just ensure you pass an array, string, number or boolean (and expect a return of the same form) - but NOT nil, null, undefined, Ruby or JavaScript objects.
  return_value = `Javascript.Code(#{variable_from_ruby})`
else
  return_value = Ruby::Code(variable_from_ruby)
end

Then adjust the build system of plurimath-js to add twitter-cldr-js library as a dependency.

2. Try to extract the parts needed from twitter-cldr-rb and ensure they don't eg. try to load files from filesystem. Below is some hint from me on how to approach such an issue:

if RUBY_ENGINE == 'opal'
  require 'available_locales_opal'
else
  AVAILABLE_LOCALES = YAML.load_file("locales.json")
end

def locale_supported?(locale)
  AVAILABLE_LOCALES.include? locale
end

# available_locales_opal.rb.erb
# This ERB file is executed by Opal at compile time (using MRI), producing an RB file.
# This RB file does not load external resources.
AVAILABLE_LOCALES = <%= YAML.load_file("locales.json").inspect %>

I have tried to build on your branch a little, first there's a patch needed for plurimath due to a tricky regexp:

diff --git a/lib/plurimath/latex/parser.rb b/lib/plurimath/latex/parser.rb
index b0ced07..7eb4aa4 100644
--- a/lib/plurimath/latex/parser.rb
+++ b/lib/plurimath/latex/parser.rb
@@ -8,7 +8,7 @@ module Plurimath
     class Parser
       attr_accessor :text
 
-      TEXT_REGEX = %r(\\(?:mbox|text){[^}]+})
+      TEXT_REGEX = %r(\\(?:mbox|text)\{[^}]+\})
 
       def initialize(text)
         @text = pre_processing(text)

Second, I dropped your fix commits and added stubs. Stubs cause a file to just become compiled as an empty file. That's not a proper fix, but there's a high likelihood that those dependencies are not used outside of tests or are not used by our use of the library.

diff --git a/build.sh b/build.sh
index 4713f5e..1c1f526 100755
--- a/build.sh
+++ b/build.sh
@@ -18,6 +18,25 @@ bundle exec opal --esm -sjruby \
                  -Ivendor/ruby-ll/ext/pureruby/ \
                  -Ivendor/plurimath/lib \
                  -pplurimath \
+                 -snokogiri \
+                 -sox \
+                 -sox/ox \
+                 -sox.so \
+                 -szip \
+                 -sconcurrent \
+                 -sparallel \
+                 -scldr-plurals \
+                 -scldr/export \
+                 -sjava \
+                 -setc \
+                 -sregexp_parser \
+                 -stsort \
+                 -srexml/document \
+                 -stzinfo/data \
+                 -stempfile \
+                 -si18n \
+                 -snet/ftp \
+                 -stwitter_cldr/resources/requirements \
                  -c -e'#' > tmp/plurimath-opal.js
 
 echo "** Minifying Plurimath-Opal with Terser"

Unfortunately, while this successfully compiles our library, the library is unusable due to aforementioned issues.

[ronaldtse]

I think the best way forward is not to use twitter-cldr and go with a plain Ruby solution. The number formatting rules aren't so complex and numerous that we can extract from CLDR (or post-process twitter-cldr) directly.

[suleman-uzair]

- TEXT_REGEX = %r(\(?:mbox|text){[^}]+})
+ TEXT_REGEX = %r(\(?:mbox|text)\{[^}]+\})

@hmdne I will add this patch in my next commit.

I think the best way forward is not to use twitter-cldr and go with a plain Ruby solution.

@ronaldtse I totally agree with you. I will open an issue for this and start working on it right away.

[ronaldtse]

@suleman-uzair so we no longer use twitter-cldr. Is this task ready to go?

[suleman-uzair]

@suleman-uzair so we no longer use twitter-cldr. Is this task ready to go?

@ronaldtse I just updated the Plurimath submodule in the repo, but it's still failing for some reason, I'll look into it right after the current task.

[suleman-uzair]

@ronaldtse, This PR is now finalized using Plurimath's latest state.
Unfortunately, there are two drawbacks to this update for now.

1. XMLL(XML languages, MathML and OMML) examples might not work properly.

Unfortunately, LutaML-Model's Oga support is not quite smooth yet (the issue exists at lutaml/lutaml-model#100), which might cause a crash for some XMLL examples.
All supported languages(except XMLL) to XMLL conversion will be fine since it uses Plurimath's internal Oga classes.

2. Using number formatter in plurimath-js

NumberFormatter depends on BigDecimal for numeric calculations and conversions, and Opal is raising an error similar to undefined method 'BigDecimal' for <NumberFormatter:instance>, So NumberFormatting support will have to wait until an alternative of BigDecimal is implemented in Plurimath.

@hmdne, can you help/guide for BigDecimal support? I see BigDecimal class in Opal.

[hmdne]

@suleman-uzair

It is in stdlib, so you need to require "bigdecimal". We haven't touched that code for a long time, though it should work.

[suleman-uzair]

@ronaldtse, I will merge/finalize this PR after the following PRs are merged:

1. MathML parsing using Oga #322
2. power and fix methods #1
3. NumberFormatter methods refactoring #320
4. Replace YAML and XML processing using lutaml-model unitsml/unitsml-ruby#21

[suleman-uzair]

UPDATE

@ronaldtse, This PR is in progress for now.
Working on Unitsml-Ruby unitsml/unitsml-ruby#21 for changes related to this PR/release.