Sync: bring OntoPortal up-to-date with BioPortal releases 5.26.0 and …

…onward (#2)

* add a script to eradicate (delete data+ files) submissions of an ontology

* Auto stash before merge of "development" and "master"

* omit logs link file

* update the eradicator to support the eradication of not archived submissions if wanted

* fix the delete submission files to not let behind empty directories

* not remove the submission directory beaucse it's already done by the submission.delete

* Update Gemfile.lock

* Reset branch specifier to develop

* extract do_ontology_pull function

* some simple code refactor in the ontology_pull

* simple code refactor of test_ontology_pull

* add a script to do a ontology pull on an ontology on demand

* set the name of the new script in $0

* extract new_file_exists? method from do_ontology_pull

* save the submission in the RemoteFileException

* some automatic code refactor/lint

* use the new do_ontology_pull in the old  do_remote_ontology_pull

* fixed an API call mentioned by @syphax-bouazzouni in ncbo/bioportal-project#254

* fixed an API call mentioned by @syphax-bouazzouni in ncbo/bioportal-project#254

* Gemfile.lock update

* bump up version of actions/checkout from v2->v3

* Gemfile.lock update

* Merge branch 'develop'

* remove forgot variables

* GH Actions unit test workflow refactor

- add ruby versioning via docker-compose.yml file
- bump up ruby v2.6 -> v2.7
- add AllegroGraph backend
- add code coverage

* Remove extra space

* fix for #61

- create contact instance if it doesn't exist
- changed --from-api to --from-apikey
- minor linting

* Restore branch specifier to develop

* Optimization - remove repeated query

* Gemfile.lock update

* Gemfile.lock update

* Gemfile.lock update

* Gemfile had references to develop branch

* implemented #64 - ability to generate labels independently of RDF processing (and vise versa)

* Gemfile.lock update

* fixed a bug in #64

* Relocate docker-compose file and update default configs

* Add GH workflow for publishing docker images

* use ruby native method for listing files instead of a git function

Resolves warning messages when we exclude .git directory from docker image

* remove comment

* capitalize argument in order to be consistent with other scripts

* add arm/64 platform

* additional error handling for SPAM deletion script, #60

* additional error handling for SPAM deletion script, #60

* implemented #67 - improved corrupt data and error handling

* Gemfile.lock update

* exclude test/data/dictionary.txt from git commits

* update version of solr-ut

* Gemfile.lock update

* Restore branch specifier to master

* fixed configuration for the analytics module

* Gemfile.lock update

* implemented #69 - scheduled annotator dictionary file generation should be a configurable option instead of the default

* Gemfile.lock update

* gem update

* create new rake taks for updating purls for all ontologies

moved from ontologies_api/fix_purls.rb

* initial implementation of #70 - Google Analytics v4 Update Compatibility Issue

* added the /data folder to ignore

* update gems

* Gemfile.lock update

* Gemfile.lock update

* Gemfile.lock update

* use patched version of agraph v7.3.1

* unpin faraday gem

* A chnage to reference Analytics Redis from LinkedData block

* Gemfile.lock update

* Gemfile.lock update

* Gemfile.lock update

* Gemfile.lock update

* use assert_operator instead of assert

minitest style guide adherence.
encountered an intermittent unit test failure so assert_operator will provide
better failure feedback than assert

* fixed ncbo_ontology_archive_old_submissions error output

* Gemfile.lock update

* Gemfile.lock update

* Gemfile update

* Gemfile update

* fixes to the analytics script and a new script to generate UA analytics for documentation

* Gemfile.lock update

* Gemfile.lock update

* implemented the first pass at bmir-radx/radx-project#37

* implemented the first pass at bmir-radx/radx-project#37

* set bundler version to be comptatible with ruby 2.7

+ AG v8

* Gemfile.lock update

* Gemfile.lock update

---------

Co-authored-by: Jennifer Vendetti <[email protected]>
Co-authored-by: mdorf <[email protected]>
Co-authored-by: Alex Skrenchuk <[email protected]>

.dockerignore

@@ -1,5 +1,6 @@
 # Git
-#.git
+.git
+.github
 .gitignore
 # Logs
 log/*
@@ -8,3 +9,9 @@ tmp/*
 # Editor temp files
 *.swp
 *.swo
+coverage
+create_permissions.log
+# Ignore generated test data
+test/data/dictionary.txt
+test/data/ontology_files/repo/**/*
+test/data/tmp/*

.github/workflows/docker-image.yml

@@ -0,0 +1,42 @@
+name: Docker Image CI
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  push_to_registry:
+    name: Push Docker image to Docker Hub
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out the repo
+        uses: actions/checkout@v3
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v2
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2
+
+      - name: Log in to Docker Hub
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Extract metadata (tags, labels) for Docker
+        id: meta
+        uses: docker/metadata-action@v4
+        with:
+          images: bioportal/ncbo_cron
+
+      - name: Build and push Docker image
+        uses: docker/build-push-action@v4
+        with:
+          context: .
+          platforms: linux/amd64,linux/arm64
+          build-args: |
+            RUBY_VERSION=2.7
+          push: true
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}

.github/workflows/ruby-unit-tests.yml

@@ -6,15 +6,25 @@ on:
 
 jobs:
   test:
+    strategy:
+      fail-fast: false
+      matrix:
+        backend: ['ncbo_cron', 'ncbo_cron-agraph'] # ruby runs tests with 4store backend and ruby-agraph runs with AllegroGraph backend
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v3
     - name: copy config.rb file from template
       run:  cp config/config.test.rb config/config.rb
     - name: Build docker-compose
-      working-directory: ./test
       run: docker-compose build
     - name: Run unit tests
-      working-directory: ./test
-      run: docker-compose run unit-test wait-for-it solr-ut:8983 -- rake test TESTOPTS='-v'
+      run: |
+        ci_env=`bash <(curl -s https://codecov.io/env)`
+        docker-compose run $ci_env -e CI --rm ${{ matrix.backend }} bundle exec rake test TESTOPTS='-v'
+    - name: Upload coverage reports to Codecov
+      uses: codecov/codecov-action@v3
+      with:
+        flags: unittests
+        verbose: true
+        fail_ci_if_error: false # optional (default = false)
 

.gitignore

@@ -3,6 +3,8 @@
 config/config.rb
 config/config_*.rb
 config/*.p12
+config/*.json
+data/
 projectFilesBackup/
 .ruby-version
 repo*
@@ -11,6 +13,9 @@ repo*
 .DS_Store
 tmp
 
+# Code coverage reports
+coverage*
+
 # Ignore eclipse .project
 .project
 .pmd

Dockerfile

@@ -1,13 +1,29 @@
-FROM ruby:2.6
+ARG RUBY_VERSION
+ARG DISTRO_NAME=bullseye
 
-RUN apt-get update -yqq && apt-get install -yqq --no-install-recommends openjdk-11-jre-headless raptor2-utils wait-for-it
+FROM ruby:$RUBY_VERSION-$DISTRO_NAME
+
+RUN apt-get update -yqq && apt-get install -yqq --no-install-recommends \
+  openjdk-11-jre-headless \
+  raptor2-utils \
+  && rm -rf /var/lib/apt/lists/*
 
-# The Gemfile Caching Trick
-# we install gems before copying the code in its own layer so that gems would not have to get
-# installed every single time code is updated
 RUN mkdir -p /srv/ontoportal/ncbo_cron
+RUN mkdir -p /srv/ontoportal/bundle
 COPY Gemfile* *.gemspec /srv/ontoportal/ncbo_cron/
+
 WORKDIR /srv/ontoportal/ncbo_cron
-RUN gem install bundler -v "$(grep -A 1 "BUNDLED WITH" Gemfile.lock | tail -n 1)"
+
+# set rubygem and bundler to the last version supported by ruby 2.7
+# remove version after ruby v3 upgrade
+RUN gem update --system '3.4.22'
+RUN gem install bundler -v '2.4.22'
+RUN gem update --system
+RUN gem install bundler
+ENV BUNDLE_PATH=/srv/ontoportal/bundle
 RUN bundle install
+
 COPY . /srv/ontoportal/ncbo_cron
+RUN cp /srv/ontoportal/ncbo_cron/config/config.rb.sample /srv/ontoportal/ncbo_cron/config/config.rb
+
+CMD ["/bin/bash"]

Gemfile

@@ -2,13 +2,17 @@ source 'https://rubygems.org'
 
 gemspec
 
-gem 'faraday', '~> 1.9'
 gem 'ffi'
+
+# This is needed temporarily to pull the Google Universal Analytics (UA)
+# data and store it in a file. See (bin/generate_ua_analytics_file.rb)
+# The ability to pull this data from Google will cease on July 1, 2024
 gem "google-apis-analytics_v3"
+
+gem 'google-analytics-data'
 gem 'mail', '2.6.6'
-gem 'minitest', '< 5.0'
 gem 'multi_json'
-gem 'oj', '~> 2.0'
+gem 'oj', '~> 3.0'
 gem 'parseconfig'
 gem 'pony'
 gem 'pry'
@@ -28,6 +32,8 @@ gem 'sparql-client', github: 'ncbo/sparql-client', branch: 'master'
 
 group :test do
   gem 'email_spec'
+  gem 'minitest', '< 5.0'
+  gem 'simplecov'
+  gem 'simplecov-cobertura' # for codecov.io
   gem 'test-unit-minitest'
 end
-