Merge to master: AgroPortal Release 2.8.0 #7
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…es-metadata-to-scheme-file Feature: Update ontology submission model 2023 - Part 1 - Update attributes validators and metadata
Feature: Support multi lingual - add show_language argument to the attributes getters
Now when we ask a language we ether the values with the matching language, or nil or empty if not. Were before we had fallback to show the values in the others languages to not have empty fields
SKOS XL labels are tagged with a language so literal value is empty if it does not match the requested language but the object is still fetched and showed
Feature: Add request all languages unit test
…ields for values in other languages
…ngual-search Feature: Support multilangual search
…organization-models Feature: Update ontology submission model 2023 - Part 2 - Add person & organization models
…tion-models' into feature/add-persons-and-organization-models
…idators Feature: Disable the new validators & fix submission metadata categories
…tes-metadata-to-scheme-file Fix: embed for submission model all the attributes of the object metrics
* Template added at top of file * Consolidation of the "General" properties * Consolidation of "Licensing", "Description" and "Dates" * Consolidation of the "Person and orga" properties * Update ontology_submission.yml * Consolidation of the "Usage" and "Methodology" properties * Consolidation of the "object description properties" * Consolidated the Metrics properties * Consolidation of Links and Relations properties * Consolidation of Content and Media properties * Fix syntax error --------- Co-authored-by: Syphax bouazzouni <[email protected]>
…110) * Fix multiple errors and syntax bugs. * Update ontology_submission.yml
* add usage method to the agent models * optimize the agents usages query, to do batch loading when fetching all agents
* update the lexvo language validator to no check the iso version
* added mail options test to catch auth type case * fixed auth type mail options not being saved
* added mail options test to catch auth type case * fixed auth type mail options not being saved
…metadata (#114) * fix includedInCatalog enforced values to be urls * update accrualMethod to no more be limited to URIs
…n (#96) * fix json serializer nil exception (#79) if hash was missing @context key the json serializer throwed a nil exception * Merge pull request #87 from ontoportal-lirmm/fix/achived-submission-space-optimization Fix: Archived submission space optimization * Merge pull request #71 from ontoportal-lirmm/feature/support-multilingual-read-one-language-from-request-parameter Feature: Support multilingual - Add read one language from request parameter tests * Merge pull request #73 from ontoportal-lirmm/feature/add-request-language-to-context Feature/add request language to context * fix json serializer nil exception (#79) if hash was missing @context key the json serializer throwed a nil exception * Merge pull request #83 from ontoportal-lirmm/feature/add-request-language-to-context Feature: add request language to context follow up * Merge pull request #93 from ontoportal-lirmm/support-multi-lang-part-02 Feature: Add request all languages unit test * Merge pull request #94 from ontoportal-lirmm/feature/support-multi-langual-search Feature: Support multilangual search --------- Co-authored-by: Raimi Solorzano Niederhausen <[email protected]>
* fix json serializer nil exception (#79) if hash was missing @context key the json serializer throwed a nil exception * Merge pull request #72 from ontoportal-lirmm/feature/migrate-attrbiutes-metadata-to-scheme-file Feature: Update ontology submission model 2023 - Part 1 - Update attributes validators and metadata * Merge pull request #82 from ontoportal-lirmm/feature/embed-meterics-for-submissions Feature: embed metrics for submission model * Merge pull request #86 from ontoportal-lirmm/fix/read_restricted_based_on-nil Fix: call to read_restricted? only if instance_to_base_on not nil * Merge pull request #90 from ontoportal-lirmm/feature/add-persons-and-organization-models Feature: Update ontology submission model 2023 - Part 2 - Add person & organization models * in tests no more use the reload argument in the exist? method --------- Co-authored-by: Raimi Solorzano Niederhausen <[email protected]>
* add request language to @context * add subscribed attribute to the user model (#68) Co-authored-by: Syphax bouazzouni <[email protected]> * change name ( language -> lang ) * add @language to @context * in tests no more use the reload argument in the exist? method * rename the INRAETHES test file name * fix mappings test test_external_mapping_with_no_source_ids validity * handle users with no username to_string * add requested language tests * use for Goo the language filter feature branch * add request_store gem * update requested language tests to use RequestStore gem * migrate submission attributes metadata to a yaml scheme file * migrate submission types enforce to the type section * update ontology submission types and validators * add metadata information to some attributes of ontology submission Updated metadata information for version, URI, hasOntologyLanguage, status, hasFormalityLevel, hasOntologySyntax, deprecated, versionIRI attributes * fix a typo in ontology submission curatedOn attribute validator * reading language value from request query parameter called 'language' * update the metadata extractor to handle the new goo validators * fix tests to handle the new ontology submission validators * force averageChildCount metrics to be a float * remove the ontology_submission creationDate attribute default value * disable extractedMeta the keyClasses and exampleIdentifier attributes * remove unique validator fro the URI attribute * make exampleIdentifier default the first class of the ontology * add enforce_symmetric_ontologies proc validator test * implment enforce_symmetric_ontologies proc validator * use enforce_symmetric_ontologies proc validator for som submission attrs * move ontology_submission_validators tests to a seperate file * add status_align_callbacks tests * implement submission status align callbacks * add the new two status onUpdate callback to the submission attribute * remove the old enforce_symmetric_ontologies from the submission tests * implement submission retired validators and callback * use submission retired validators and callbacks * add submission retired status validators and callbacks tests * add submission modification_date_previous_align validator test * implement modification_date_previous_align validator * refactor validators module to extract the callback module * add submission test_has_prior_version_callback * implement and use include_previous_submission onUpdate callback * add ontology update_submissions_has_part test * implement and use ontology update_submissions_has_part callback * implement and use submission include_ontology_views validator * Update ontology_submission.yml Update ontology_submission.yml Update ontology_submission.yml Update ontology_submission.yml Update ontology_submission.yml Update ontology_submission.yml Update ontology_submission.yml Update ontology_submission.yml * Update Gemfile.luck to get the new version of Goo * fix some syntax issue the ontology_submission.yml file * add submission inverse ontology properties callbacks tests * implement and use ontology_inverse_of callback for useImports and usedBy * Update ontology_submission.yml labels : uri,homepage, natural language,langauge, publication Update ontology_submission.yml Labels => Documentation, version, Description, Status, Creation date, contact Update ontology_submission.yml adding labels : Number of classes, Number of individuals, Number of properties, Modification date, Number of individuals, Number of axioms or triples, Keywords, Key classes, Known usage, Notes, Knowledge representation paradigm Update ontology_submission.yml adding labels : Designed for task Update ontology_submission.yml Adding labels : Was generated by, curator, was invalidated by, Endorsed by, Funded or sponsored by, Translator, Subject, Formality level, License, Engineering methodology Update ontology_submission.yml Adding labels : Created With, Prior version, Imports, Created With, Incompatible, Backward compatible, Deprecated, Generally related to, Has equivalences with, Hidden or past name, Coverage, Publisher, Other identifier, Source Update ontology_submission.yml adding labels : Abstract, Has part (has views), Has format, Audience, SPARQL endpoint, Accrual policy, Accrual periodicity, Accrual method, Download URL, Free-text search endpoint, Competency question, logo, depiction, Disparate modelling with, To do list, Preferred namespace URI, Preferred namespace prefix,More permissions, Use guidelines, Curation date, Repository, Bug database, Mailing list, Example of resource, Rights holder, Associated media, Indexed or Included in catalog or repository * refactor and reorder submission attributes and group them by category * disable metadata extraction for hasOntologyLanguage attribute * update Gemfile.lock to get the latest version of Goo * fix syntax issues in the ontology_submission.yml file * enable submission creationDate automatic extraction * implement and use lexvo_language validator * add submission lexvo_language_validator test * remove no more needed concatenate, textarea and isOntology attributes enforce * remove submission metrics attributes and move them to the metrics model * set the default values callbacks to submissions attributes * don't show request language if @context is not enabled * Accrual periodicity values * IncludedInDataCatalog : enforcedValues * apply for some submission attributes ontologies_inverse_of callback * Add metadataMappings to properties Synchronize and consolidate the file * Add metadataMappings values Synchronize and consolidate the file * Add missing properties (Object description properties) & pullLocation * Add obsoleteParent property * Add Missing Metrics properties * fix json serializer nil exception (#79) if hash was missing @context key the json serializer throwed a nil exception * add some metrics attributes as embed values for Submissions * prevent a bug when an embed object is null (e.g submission with no metrics) * Feature: return requested languages in @language - part 2 - multiple langugaes (#80) * remove platform languages * set all to default in language params * upcase the requested language * support select multilanguage * return only existing language * extract get_languages * fix json serializer nil exception * access submission safely * refactor --------- Co-authored-by: Syphax bouazzouni <[email protected]> * add get_languages and get_submission_languages helpers * use get_languages helper to get the current context language * extract get_language helper to get the requested language * get serialized object submission only if include 'submission' attribute (#85) * call to read_restricted? only if instance_to_base_on not nil * remove duplicated gzip? function * add zip_file function to the file utilities module * add archiving submission zipping old uploadedFile file test * add removing archived submission unzipped folder test * implement and extract the new archive action in the submission process * prevent interconnected tests of zipped_submission process tests * add archiving file size threshold for zipping uploaded file * fix ontology submission yaml file syntax issues * add AgentIdentifier model * add FOAF Agent model * update submission person and organization attributes to use Agent model * update Agent model to enforce name to be unique and an auto generated ID * disable extracting metadata for attributes of type * rename Agent 'type' to 'agentType' to not conflicts with JSON-LD @type * remove property: :pageHome form Agent model because it does not exist * make identifier model embded true * add read write access rules for agent and identifier models * remove firstName and lastName attribute from Agent model * enforce Agent affiliations to be all of type Organization * enforce Identifier notation attribute to not be a URL * add agent test for creation validity * enforce to show all attributes of the embed affiliations of Agents * add show_with_language param * update get_submission_languages * change identifier id to make be schemaAgency + notation * make agent identifiers list unique * make agent email attribute unique * make agent name no more unique * change agent affiliations attribute namespace to org:memberOf * fix identifier id to be nil if notation or agentType empty * make email agent attribute optional * add is_organizaion validator to agent and submission affiliation attributes * fix the class portal language test for the new multilanagual support Now when we ask a language we ether the values with the matching language, or nil or empty if not. Were before we had fallback to show the values in the others languages to not have empty fields * optimize request and portal lang tests by parsing the test ontology once * add test request all languages unit test * use the new getters argument to show languages * fix skos xl tests after the new multilangual support SKOS XL labels are tagged with a language so literal value is empty if it does not match the requested language but the object is still fetched and showed * add test for multilingual terms indexation and search * update SOLR terms schema to add dynamicFields for values in all langs * make the indexation use map_attributes with include_languages enabled * override the Ressource to_hash method in Class model to * update the indexed document of a class model to include the dynamic fields for values in other languages * rename the getters to show_all_languages argument to include_languages * add is_person validator to hasCreator and hasContributor attributes * embed all agents attributes for the submission model * fix submission deprecate_previous_submissions callback when no valid * refactor submission agent validators to extract common logic * update the include_ontology_views validator to test prefixed url id * add Agent type to submission endorsedBy attribute * remove URI type of submission usedOntologyEngineeringTool attribute * remove class type for exampleIdentifier and keyClasses attributes * check if object.class respond to attributes before calling it * check if the attribute method is a getter and not serialized method * check if the attribute method is a getter and not serialized method * use the same populate method as the attribute to show languages in properties also * use the same populate method as the attribute to show languages in properties also * Fix bug in authorization.rb * Improved support for OAuth 2.0 and SSO * bundle add oauth2 * remove duplicate find_apikey in the authorization call * add find_access_token method oauth2 authorization * add the unauthorized message the alternative text if oath2 enabled * add user oauth2 tests * enforce user email attribute to be unique * on user save (creation) send a notification if enabled in the arguments * add oauth2 providers configs * implement oauth2 providers token authentification * remove access token authorization to use only apikeys methods * fix tests after enforcing user email uniqueness * override security when sync user with an external provider * fix update_submissions_has_part callback to check if valid before saving * add uri type enforcement to submission URI attribute * disable attributes onUpdate callbacks feature * fix URI extraction process to enforce RDF::URI instead of string * fix test after URI enforced to be URI * disable complex attriubtes validators and callbacks * Update Object description properties Changed enforced values and definitons. * Remove enforced values for pullLocation * fix ontology submission attributes categories * disable complex validators tests * embed for submission model all the attributes of the object metrics * Fix: ontology submission metadata file rework (#108) * Template added at top of file * Consolidation of the "General" properties * Consolidation of "Licensing", "Description" and "Dates" * Consolidation of the "Person and orga" properties * Update ontology_submission.yml * Consolidation of the "Usage" and "Methodology" properties * Consolidation of the "object description properties" * Consolidated the Metrics properties * Consolidation of Links and Relations properties * Consolidation of Content and Media properties * Fix syntax error --------- Co-authored-by: Syphax bouazzouni <[email protected]> * add usage method to the agent models (#109) * Fix multiple errors and syntax bugs in ontology_submission.ym file (#110) * Fix multiple errors and syntax bugs. * Update ontology_submission.yml * Fix: optimize fetch all agents usages query (#111) * add usage method to the agent models * optimize the agents usages query, to do batch loading when fetching all agents * Fix: the lexvo language validator to no check the iso version (#112) * update the lexvo language validator to no check the iso version * Fix: login auth type mail options (#113) * added mail options test to catch auth type case * fixed auth type mail options not being saved * Fix: ontology submisison "includedInDataCatalog" and "accrualMethod" metadata (#114) * fix includedInCatalog enforced values to be urls * update accrualMethod to no more be limited to URIs --------- Co-authored-by: HADDAD Zineddine <[email protected]> Co-authored-by: DJEBABRIA Hiba <[email protected]> Co-authored-by: Raimi Solorzano Niederhausen <[email protected]> Co-authored-by: Clement Jonquet <[email protected]> Co-authored-by: Guillaume Alviset <[email protected]>
…Pattern (#116) * Fix error btw released and creationDate * open the uriRegexPattern property to be a string * fix archived submission notification email to show failure in subject --------- Co-authored-by: Clement Jonquet <[email protected]>
* add ontologies and users analytics tests * extract analytics concern to a file that load and parse data from redis * add users visits per month and pages visited in the previous month stats
…ing no lang first (#128)
…ndexing ontologies data and metadata, URI data fetching (#135)
* Feature: Optimize tests run time by 50% (#107)
* update bubastis.jar v1.4.0 and fix missing import exception
* optimize mappings tests
* optimize provisional relation tests
* optimize notes tests
* fix mappings tests
* optimize instances tests
* add generate_missing_labels and extract_metadata process options
* don't index and extract by default in submission process in tests
* optimize ontology submission tests run time
* Feature: Add Virtuso, Allegrograph and Graphdb integration to OLD (#106)
* setup multi-store unit-tests environment
* fix unit tests
* add vo parsing optimization
* update RDF version replaced RDF::SKOS with RDF::Vocab::SKOS (#131)
* Fix: an issue after update RDF gem to 3.0 that frozen request params (#133)
* fix an issue after update RDF gem to 3.0 that frozen request params
* handlee the case when the sparql endpoint default value is empty
* Feature: Migrate SOLR configuration files to use SOLR Schema API (#126)
* use standard SOLR in docker compose with no ontoportal old confgis
* migrate ontology properties SOLR configuration to use Schema API
* migrate ontology classes SOLR configuration to use Schema API
* migrate provisional classes indexation to use Schema API and model hooks
* update tests to handle the new indexation API
* simplify the ontology properties index schema
* update class and properties schema to use the existent dynamic names
* Feature: Index Ontologies metadata and content & Agents (#130)
* use standard SOLR in docker compose with no ontoportal old confgis
* migrate ontology properties SOLR configuration to use Schema API
* migrate ontology classes SOLR configuration to use Schema API
* migrate provisional classes indexation to use Schema API and model hooks
* update tests to handle the new indexation API
* simplify the ontology properties index schema
* update class and properties schema to use the existent dynamic names
* index submission and ontologies metadata on save
* index agents metadata
* add ontology and agent metadata indexation tests
* make agent, name , acronym, email and identifiers searchable
* unindex ontology submission when archived
* make ontology acronym and name searchable
* update embedded ontology to all the fields and update submission in save
* fix embed docs search tests
* rename ontology unindex to unindex_all_data to prevent conflicts
* implement index all ontology content
* fix unescaping indexed properties naming
* fix an issue after update RDF gem to 3.0 that frozen request params
* add parallel processing the index_all_data step
* clear indexed data after ontology delete
* optimize index all data in Virtuoso and GraphDb by pre-fetching all ids
- Before optimization
- fs ⇒ 15.224490000051446s
- ag ⇒ 19.238805999979377s
- vo ⇒ 42.95274499990046s
- gb ⇒ 33.52821200003382s
- After optimization
- fs ⇒ 15.369778999942355s
- ag ⇒ 17.367580000078306s
- vo ⇒ 16.564614000031725s
- gb ⇒ 15.431716999970376s
* Feature: Add URI fetching related triples and serialization in different formats (#125)
* Add raptor library to parse ntriples data
* Add resource model to fetch id related triples and serialize it
* Add and inhance xml, ntriples, turtle and json serializers
* Updating rdf version in goo project
* updating resource model
* Adding tests for resource model and serializers
* update the resource test to have a more complete data to test (array, bnodes, typed values)
* re-implement xml serializer using RDF/XML parser instead of Raptor
* implement array handelling of resource to_object
* Enhance and refactor serializers ntriples, turtle and xml
* Enhance and refactor serializers ntriples, turtle and xml
* Handle blank nodes and reverse triples
- handle blank nodes
- fetch reverse triples
- generate random name for models in to_object, because when two model created the same time one overrides the other
- call the new serializer JSONLD and RDF_XML
* Impliment new serializers jsonld and rdf_xml
- impliment jsonld serializer that uses json-ld library
- revert changes in xml.rb file to the original implimentation, and put the new implimentation in rdf_xml.rb file
- Add the media types :jsonld and :rdf_xml
* Add json-ld gem
* Enhance the test resource
- Add some cases to the data tests
- refactor the test of the serializers formats
* Fix test for fetch-related triples and json
* clean and refactor the resource serializer code
* Removed unused methods
* Extracted duplicated code in methods
* Removed skip from the tests
---------
Co-authored-by: Syphax bouazzouni <[email protected]>
* Feature: Add submission metrics to the indexed data
* Feature: isolate ontology submission process steps (#132)
* add an abstraction for submission process steps
* extract submission generate_rdf step to a file
* extract submission generate missing labels steps into a file
* extract the submission archiving step into a file
* add abstraction to diff tool & extract the submission step to a file
* extract the submission metrics generation step to a file
* extract the submission properties indexation step into a file
* extract the submission terms indexation step into a file
* move the extract metadata concern to submission process step file
* extract the submission generate obsolete classes step from generate rdf
* add the global submission process that call the sub-steps
* Feature: Add index all data step (#136)
* move the submission_all_data concern to a submission process service
* add index_all step to submission parsing steps
* add index all data submission status
* send note creation notification to also the admins (#137)
* change sparql client branch to use development
* fix indexing all data been removed after the index terms step
---------
Co-authored-by: Imad Bourouche <[email protected]>
syphax-bouazzouni
force-pushed
the
ecoportal-ontoportal-reset
branch
3 times, most recently
from
8cbe2f9 to
ad207e6
Compare
syphax-bouazzouni
force-pushed
the
ecoportal-ontoportal-reset
branch
from
ad207e6 to
4efea88
Compare
syphax-bouazzouni
added a commit
that referenced
this pull request
May 28, 2024
Merge to master: AgroPortal Release 2.8.0
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR syncs Ecoportal with the latest version Agroportal v2.8.0 it includes the following: