Merge to master: Release 2.6.0 - Multi-backend stores integrations, I…

…ndexing ontologies data and metadata, URI data fetching (#135) * Feature: Optimize tests run time by 50% (#107) * update bubastis.jar v1.4.0 and fix missing import exception * optimize mappings tests * optimize provisional relation tests * optimize notes tests * fix mappings tests * optimize instances tests * add generate_missing_labels and extract_metadata process options * don't index and extract by default in submission process in tests * optimize ontology submission tests run time * Feature: Add Virtuso, Allegrograph and Graphdb integration to OLD (#106) * setup multi-store unit-tests environment * fix unit tests * add vo parsing optimization * update RDF version replaced RDF::SKOS with RDF::Vocab::SKOS (#131) * Fix: an issue after update RDF gem to 3.0 that frozen request params (#133) * fix an issue after update RDF gem to 3.0 that frozen request params * handlee the case when the sparql endpoint default value is empty * Feature: Migrate SOLR configuration files to use SOLR Schema API (#126) * use standard SOLR in docker compose with no ontoportal old confgis * migrate ontology properties SOLR configuration to use Schema API * migrate ontology classes SOLR configuration to use Schema API * migrate provisional classes indexation to use Schema API and model hooks * update tests to handle the new indexation API * simplify the ontology properties index schema * update class and properties schema to use the existent dynamic names * Feature: Index Ontologies metadata and content & Agents (#130) * use standard SOLR in docker compose with no ontoportal old confgis * migrate ontology properties SOLR configuration to use Schema API * migrate ontology classes SOLR configuration to use Schema API * migrate provisional classes indexation to use Schema API and model hooks * update tests to handle the new indexation API * simplify the ontology properties index schema * update class and properties schema to use the existent dynamic names * index submission and ontologies metadata on save * index agents metadata * add ontology and agent metadata indexation tests * make agent, name , acronym, email and identifiers searchable * unindex ontology submission when archived * make ontology acronym and name searchable * update embedded ontology to all the fields and update submission in save * fix embed docs search tests * rename ontology unindex to unindex_all_data to prevent conflicts * implement index all ontology content * fix unescaping indexed properties naming * fix an issue after update RDF gem to 3.0 that frozen request params * add parallel processing the index_all_data step * clear indexed data after ontology delete * optimize index all data in Virtuoso and GraphDb by pre-fetching all ids - Before optimization - fs ⇒ 15.224490000051446s - ag ⇒ 19.238805999979377s - vo ⇒ 42.95274499990046s - gb ⇒ 33.52821200003382s - After optimization - fs ⇒ 15.369778999942355s - ag ⇒ 17.367580000078306s - vo ⇒ 16.564614000031725s - gb ⇒ 15.431716999970376s * Feature: Add URI fetching related triples and serialization in different formats (#125) * Add raptor library to parse ntriples data * Add resource model to fetch id related triples and serialize it * Add and inhance xml, ntriples, turtle and json serializers * Updating rdf version in goo project * updating resource model * Adding tests for resource model and serializers * update the resource test to have a more complete data to test (array, bnodes, typed values) * re-implement xml serializer using RDF/XML parser instead of Raptor * implement array handelling of resource to_object * Enhance and refactor serializers ntriples, turtle and xml * Enhance and refactor serializers ntriples, turtle and xml * Handle blank nodes and reverse triples - handle blank nodes - fetch reverse triples - generate random name for models in to_object, because when two model created the same time one overrides the other - call the new serializer JSONLD and RDF_XML * Impliment new serializers jsonld and rdf_xml - impliment jsonld serializer that uses json-ld library - revert changes in xml.rb file to the original implimentation, and put the new implimentation in rdf_xml.rb file - Add the media types :jsonld and :rdf_xml * Add json-ld gem * Enhance the test resource - Add some cases to the data tests - refactor the test of the serializers formats * Fix test for fetch-related triples and json * clean and refactor the resource serializer code * Removed unused methods * Extracted duplicated code in methods * Removed skip from the tests --------- Co-authored-by: Syphax bouazzouni <[email protected]> * Feature: Add submission metrics to the indexed data * Feature: isolate ontology submission process steps (#132) * add an abstraction for submission process steps * extract submission generate_rdf step to a file * extract submission generate missing labels steps into a file * extract the submission archiving step into a file * add abstraction to diff tool & extract the submission step to a file * extract the submission metrics generation step to a file * extract the submission properties indexation step into a file * extract the submission terms indexation step into a file * move the extract metadata concern to submission process step file * extract the submission generate obsolete classes step from generate rdf * add the global submission process that call the sub-steps * Feature: Add index all data step (#136) * move the submission_all_data concern to a submission process service * add index_all step to submission parsing steps * add index all data submission status * send note creation notification to also the admins (#137) * change sparql client branch to use development * fix indexing all data been removed after the index terms step --------- Co-authored-by: Imad Bourouche <[email protected]>
lifewatch-eric · May 22, 2024 · 2bc3c98 · 2bc3c98
1 parent 95e7798
commit 2bc3c98
Show file tree

Hide file tree

Showing 92 changed files with 3,635 additions and 7,658 deletions.
diff --git a/.github/workflows/ruby-unit-tests.yml b/.github/workflows/ruby-unit-tests.yml
@@ -7,26 +7,37 @@ on:
 jobs:
   test:
     strategy:
+      fail-fast: false
       matrix:
-        backend: ['ruby'] # ruby runs tests with 4store backend and ruby-agraph runs with AllegroGraph backend
+        goo-slice: [ '20', '100', '500' ]
+        ruby-version: [ '2.7' ]
+        triplestore: [ 'fs', 'ag', 'vo', 'gb' ]
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v3
-    - name: Set up solr configsets
-      run: ./test/solr/generate_ncbo_configsets.sh
-    - name: create config.rb file
-      run: cp config/config.test.rb config/config.rb
-    - name: Build docker-compose
-      run: docker-compose --profile 4store build #profile flag is set in order to build all containers in this step
-    - name: Run unit tests
-      # unit tests are run inside a container
-      # http://docs.codecov.io/docs/testing-with-docker
-      run: |
-        ci_env=`bash <(curl -s https://codecov.io/env)`
-        docker-compose run $ci_env -e CI --rm ${{ matrix.backend }} wait-for-it solr-ut:8983 -- bundle exec rake test TESTOPTS='-v'
-    - name: Upload coverage reports to Codecov
-      uses: codecov/codecov-action@v3
-      with:
-        flags: unittests
-        verbose: true
-        fail_ci_if_error: false # optional (default = false)
+      - uses: actions/checkout@v3
+      - name: create config.rb file
+        run: cp config/config.rb.sample config/config.rb
+      - name: Install Dependencies
+        run: sudo apt-get update && sudo apt-get -y install raptor2-utils
+      - name: Set up JDK 11
+        uses: actions/setup-java@v2
+        with:
+          java-version: '11'
+          distribution: 'adopt'
+      - name: Set up Ruby
+        uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: ${{ matrix.ruby-version }}
+          bundler-cache: true # runs 'bundle install' and caches installed gems automatically
+      - name: Run unit tests
+        # unit tests are run inside a container
+        # http://docs.codecov.io/docs/testing-with-docker
+        run: |
+          ci_env=`bash <(curl -s https://codecov.io/env)`
+          GOO_SLICES=${{ matrix.goo-slice }} bundle exec rake test:docker:${{ matrix.triplestore }} TESTOPTS="-v"
+      - name: Upload coverage reports to Codecov
+        uses: codecov/codecov-action@v3
+        with:
+          flags: unittests
+          verbose: true
+          fail_ci_if_error: false # optional (default = false)
diff --git a/Dockerfile b/Dockerfile
@@ -7,6 +7,7 @@ RUN apt-get update -yqq && apt-get install -yqq --no-install-recommends \
   openjdk-11-jre-headless \
   raptor2-utils \
   wait-for-it \
+  libraptor2-dev \
   && rm -rf /var/lib/apt/lists/*
 
 RUN mkdir -p /srv/ontoportal/ontologies_linked_data

diff --git a/Gemfile b/Gemfile
@@ -21,6 +21,9 @@ gem 'rubyzip', '~> 1.0'
 gem 'thin'
 gem 'request_store'
 gem 'jwt'
+gem 'json-ld', '~> 3.0.2'
+gem "parallel", "~> 1.24"
+
 
 # Testing
 group :test do
@@ -37,5 +40,6 @@ group :development do
   gem 'rubocop', require: false
 end
 # NCBO gems (can be from a local dev path or from rubygems/git)
-gem 'goo', github: 'ontoportal-lirmm/goo', branch: 'master'
-gem 'sparql-client', github: 'ontoportal-lirmm/sparql-client', branch: 'master'
+gem 'goo', github: 'ontoportal-lirmm/goo', branch: 'development'
+gem 'sparql-client', github: 'ontoportal-lirmm/sparql-client', branch: 'development'
+
diff --git a/Gemfile.lock b/Gemfile.lock
@@ -1,12 +1,15 @@
 GIT
   remote: https://github.com/ontoportal-lirmm/goo.git
-  revision: 8ddd2d719617ad082c6964a9efdac153cdd2b48e
-  branch: master
+  revision: 8f0a9a5bddba03d9d660a363c4e6618da479db9f
+  branch: development
   specs:
     goo (0.0.2)
       addressable (~> 2.8)
       pry
-      rdf (= 1.0.8)
+      rdf (= 3.2.11)
+      rdf-raptor
+      rdf-rdfxml
+      rdf-vocab
       redis
       rest-client
       rsolr
@@ -15,13 +18,12 @@ GIT
 
 GIT
   remote: https://github.com/ontoportal-lirmm/sparql-client.git
-  revision: aed51baf4106fd0f3d0e3f9238f0aad9406aa3f0
-  branch: master
+  revision: c96da3ad479724a31ccd6217ab9939dddfaca40e
+  branch: development
   specs:
-    sparql-client (1.0.1)
-      json_pure (>= 1.4)
-      net-http-persistent (= 2.9.4)
-      rdf (>= 1.0)
+    sparql-client (3.2.2)
+      net-http-persistent (~> 4.0, >= 4.0.2)
+      rdf (~> 3.2, >= 3.2.11)
 
 GEM
   remote: https://rubygems.org/
@@ -36,13 +38,14 @@ GEM
       public_suffix (>= 2.0.2, < 6.0)
     ansi (1.5.0)
     ast (2.4.2)
+    base64 (0.2.0)
     bcrypt (3.1.20)
-    bigdecimal (3.1.6)
+    bigdecimal (3.1.7)
     builder (3.2.4)
     coderay (1.1.3)
     concurrent-ruby (1.2.3)
     connection_pool (2.4.1)
-    crack (0.4.6)
+    crack (1.0.0)
       bigdecimal
       rexml
     cube-ruby (0.0.3)
@@ -87,13 +90,17 @@ GEM
       domain_name (~> 0.5)
     i18n (0.9.5)
       concurrent-ruby (~> 1.0)
-    json (2.7.1)
-    json_pure (2.7.1)
-    jwt (2.7.1)
+    json (2.7.2)
+    json-ld (3.0.2)
+      multi_json (~> 1.12)
+      rdf (>= 2.2.8, < 4.0)
+    jwt (2.8.1)
+      base64
     language_server-protocol (3.17.0.3)
     launchy (2.5.2)
       addressable (~> 2.8)
     libxml-ruby (2.9.0)
+    link_header (0.0.8)
     logger (1.6.0)
     macaddr (1.7.2)
       systemu (~> 2.6.5)
@@ -102,10 +109,10 @@ GEM
       net-imap
       net-pop
       net-smtp
-    method_source (1.0.0)
+    method_source (1.1.0)
     mime-types (3.5.2)
       mime-types-data (~> 3.2015)
-    mime-types-data (3.2024.0206)
+    mime-types-data (3.2024.0305)
     mini_mime (1.1.5)
     minitest (4.7.5)
     minitest-reporters (0.14.24)
@@ -114,23 +121,24 @@ GEM
       minitest (>= 2.12, < 5.0)
       powerbar
     multi_json (1.15.0)
-    multipart-post (2.3.0)
-    net-http-persistent (2.9.4)
+    multipart-post (2.4.0)
+    net-http-persistent (4.0.2)
+      connection_pool (~> 2.2)
     net-imap (0.4.10)
       date
       net-protocol
     net-pop (0.1.2)
       net-protocol
     net-protocol (0.2.2)
       timeout
-    net-smtp (0.4.0.1)
+    net-smtp (0.5.0)
       net-protocol
     netrc (0.11.0)
     oj (2.18.5)
     omni_logger (0.1.4)
       logger
     parallel (1.24.0)
-    parser (3.3.0.5)
+    parser (3.3.1.0)
       ast (~> 2.4.1)
       racc
     pony (1.13.1)
@@ -140,21 +148,34 @@ GEM
     pry (0.14.2)
       coderay (~> 1.1)
       method_source (~> 1.0)
-    public_suffix (5.0.4)
+    public_suffix (5.0.5)
     racc (1.7.3)
     rack (1.6.13)
     rack-test (0.8.3)
       rack (>= 1.0, < 3)
     rainbow (3.1.1)
     rake (10.5.0)
-    rdf (1.0.8)
-      addressable (>= 2.2)
-    redis (5.0.8)
-      redis-client (>= 0.17.0)
-    redis-client (0.19.1)
+    rdf (3.2.11)
+      link_header (~> 0.0, >= 0.0.8)
+    rdf-raptor (3.2.0)
+      ffi (~> 1.15)
+      rdf (~> 3.2)
+    rdf-rdfxml (3.2.2)
+      builder (~> 3.2)
+      htmlentities (~> 4.3)
+      rdf (~> 3.2)
+      rdf-xsd (~> 3.2)
+    rdf-vocab (3.2.7)
+      rdf (~> 3.2, >= 3.2.4)
+    rdf-xsd (3.2.1)
+      rdf (~> 3.2)
+      rexml (~> 3.2)
+    redis (5.2.0)
+      redis-client (>= 0.22.0)
+    redis-client (0.22.1)
       connection_pool
     regexp_parser (2.9.0)
-    request_store (1.5.1)
+    request_store (1.6.0)
       rack (>= 1.4)
     rest-client (2.1.0)
       http-accept (>= 1.7.0, < 2.0)
@@ -164,19 +185,19 @@ GEM
     rexml (3.2.6)
     rsolr (1.1.2)
       builder (>= 2.1.2)
-    rubocop (1.60.2)
+    rubocop (1.63.4)
       json (~> 2.3)
       language_server-protocol (>= 3.17.0)
       parallel (~> 1.10)
       parser (>= 3.3.0.2)
       rainbow (>= 2.2.2, < 4.0)
       regexp_parser (>= 1.8, < 3.0)
       rexml (>= 3.2.5, < 4.0)
-      rubocop-ast (>= 1.30.0, < 2.0)
+      rubocop-ast (>= 1.31.1, < 2.0)
       ruby-progressbar (~> 1.7)
       unicode-display_width (>= 2.4.0, < 3.0)
-    rubocop-ast (1.30.0)
-      parser (>= 3.2.1.0)
+    rubocop-ast (1.31.3)
+      parser (>= 3.3.1.0)
     ruby-progressbar (1.13.0)
     ruby2_keywords (0.0.5)
     rubyzip (1.3.0)
@@ -202,13 +223,12 @@ GEM
     unicode-display_width (2.5.0)
     uuid (2.3.9)
       macaddr (~> 1.0)
-    webmock (3.20.0)
+    webmock (3.23.0)
       addressable (>= 2.8.0)
       crack (>= 0.3.2)
       hashdiff (>= 0.4.0, < 2.0.0)
 
 PLATFORMS
-  x86_64-darwin-21
   x86_64-darwin-23
   x86_64-linux
 
@@ -221,13 +241,15 @@ DEPENDENCIES
   faraday (~> 1.9)
   ffi
   goo!
+  json-ld (~> 3.0.2)
   jwt
   libxml-ruby (~> 2.0)
   minitest
   minitest-reporters (>= 0.5.0)
   multi_json (~> 1.0)
   oj (~> 2.0)
   omni_logger
+  parallel (~> 1.24)
   pony
   pry
   rack (~> 1.0)
@@ -246,4 +268,4 @@ DEPENDENCIES
   webmock
 
 BUNDLED WITH
-   2.4.21
+   2.4.22
diff --git a/bin/bubastis-1.4.0.jar b/bin/bubastis-1.4.0.jar
diff --git a/bin/bubastis.jar b/bin/bubastis.jar
@@ -1 +1 @@
-bubastis_1_3.jar
+./bubastis-1.4.0.jar