Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor: Centralize mapping creation #157

Open
wants to merge 34 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
0cb252f
replace the readers by attr_read
syphax-bouazzouni Jul 27, 2022
5e01d03
remove useless returns
syphax-bouazzouni Jul 27, 2022
782355a
auto lint changes
syphax-bouazzouni Jul 27, 2022
fb2fb07
add the subject_source_id and object_source_id attributes
syphax-bouazzouni Jul 27, 2022
67b90b2
update owlapi-wrapper
syphax-bouazzouni Aug 3, 2022
7eb3424
add concerns glob loader
syphax-bouazzouni Aug 3, 2022
32e6639
add MetadataExtractor concern with extract_version and ontology_iri
syphax-bouazzouni Aug 3, 2022
4996384
use the new extract_metadata and remove the old extract_version
syphax-bouazzouni Aug 3, 2022
c1e485b
update owlapi-wrapper to 1.4.1 upstream
syphax-bouazzouni Aug 3, 2022
6eddae6
Merge branch 'pr/feature/add-extract-ontology-uri-to-submissions' int…
syphax-bouazzouni Aug 5, 2022
764f45e
add the concerns requires
syphax-bouazzouni Aug 8, 2022
0078bfa
add the mapping bulk load tests
syphax-bouazzouni Aug 8, 2022
e99a5cc
change the mapping model to enforce :list for relation
syphax-bouazzouni Aug 8, 2022
763679c
auto-lint of the mappings.rb file
syphax-bouazzouni Aug 8, 2022
c8b68d5
add the Mappings::BulkLoad concern
syphax-bouazzouni Aug 8, 2022
73b15a8
extend the Mappings module with the Mappings::BulkLoad concern
syphax-bouazzouni Aug 8, 2022
d479dc2
revert enforce relation to be :list
syphax-bouazzouni Aug 8, 2022
5b2a233
add check_mapping_exist before the creation
syphax-bouazzouni Aug 19, 2022
089f75b
make the bulk_loader return errors for each unsaved mapping
syphax-bouazzouni Aug 19, 2022
737add9
fix unspecified mapping defaults
syphax-bouazzouni Aug 19, 2022
c7114cc
remove 'process' from the given mappings
syphax-bouazzouni Aug 19, 2022
a40e5e1
rescue mapping process creation errors
syphax-bouazzouni Aug 19, 2022
b3a4d92
fix mappings tests by deleting them after test
syphax-bouazzouni Aug 19, 2022
03fc368
rescue the loading errors in the tests
syphax-bouazzouni Aug 19, 2022
0d524c8
remove 'process' from the given mapping to load in the tests
syphax-bouazzouni Aug 19, 2022
b47236b
save backup_rest_mapping id when creating
syphax-bouazzouni Sep 2, 2022
d0b58ce
fix test_mapping_ontologies_not_found
syphax-bouazzouni Sep 14, 2022
a80ae8c
centralize the mapping creation in the Mappings::Creator module
syphax-bouazzouni Sep 14, 2022
fcaf6d4
extend Mappings with the Creator module
syphax-bouazzouni Sep 14, 2022
bb1a8de
auto lint mappings module
syphax-bouazzouni Sep 14, 2022
5014a11
auto lint mappings models and add validate_size enforce to class_urns
syphax-bouazzouni Sep 14, 2022
2b656eb
re-use the create_mapping method in the BulkLoader
syphax-bouazzouni Sep 14, 2022
7b67f9d
bring ontology acronym if class found
syphax-bouazzouni Sep 14, 2022
7868dd6
Merge branch 'master' into pr/refactor/centralize-mapping-creation
syphax-bouazzouni Sep 14, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added bin/owlapi-wrapper-1.4.1.jar
Binary file not shown.
8 changes: 7 additions & 1 deletion lib/ontologies_linked_data.rb
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,13 @@
# Setup Goo (repo connection and namespaces)
require "ontologies_linked_data/config/config"

project_root = File.dirname(File.absolute_path(__FILE__))

models = Dir.glob("#{project_root}/ontologies_linked_data/concerns/**/*.rb").sort
models.each do |m|
require m
end

# Include other dependent code
require "ontologies_linked_data/security/authorization"
require "ontologies_linked_data/security/access_control"
Expand All @@ -33,7 +40,6 @@
require "ontologies_linked_data/models/base"

# Require all models
project_root = File.dirname(File.absolute_path(__FILE__))

# We need to require deterministic - that is why we have the sort.
models = Dir.glob(project_root + '/ontologies_linked_data/models/**/*.rb').sort
Expand Down
32 changes: 32 additions & 0 deletions lib/ontologies_linked_data/concerns/mappings/mapping_bulk_load.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
module LinkedData
module Concerns
module Mappings
module BulkLoad
# A method to easily add a new mapping without using ontologies_api
# Where the mapping hash contain classes, relation, creator and comment)

def bulk_load_mappings(mappings_hash, user_creator, check_exist: true)
errors = {}
loaded = []
mappings_hash&.each_with_index do |mapping, index|
loaded << load_mapping(mapping, user_creator, check_exist: check_exist)
rescue ArgumentError => e
errors[index] = e.message
end
[loaded, errors]
end

def load_mapping(mapping_hash, user_creator, check_exist: true)
LinkedData::Mappings.create_mapping(mapping_hash: mapping_hash, user_creator: user_creator,
check_exist: check_exist)
end

end
end
end
end





171 changes: 171 additions & 0 deletions lib/ontologies_linked_data/concerns/mappings/mapping_creator.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,171 @@
module LinkedData
module Concerns
module Mappings
module Creator

def create_mapping(mapping_hash:, user_creator:, check_exist: false)
object_class, object_submission,
subject_class, subject_submission = get_mapping_classes(subject_id: mapping_hash[:subject_source_id],
object_id: mapping_hash[:object_source_id],
classes: mapping_hash[:classes])

process = create_mapping_process(mapping_hash, subject_submission&.uri, object_submission&.uri, user_creator)
classes = [subject_class, object_class]

if check_exist && LinkedData::Mappings.check_mapping_exist(classes, process.relation)
raise ArgumentError, 'Mapping already exists'
end

save_process(process)
save_rest_mapping(classes, process)
end

def create_rest_mapping(classes, process)
begin
backup_mapping = LinkedData::Models::RestBackupMapping.new
backup_mapping.uuid = UUID.new.generate
backup_mapping.process = process
class_urns = generate_class_urns(classes)
backup_mapping.class_urns = class_urns
# Insert backup into 4store

raise StandardError, backup_mapping.errors unless backup_mapping.valid?

backup_mapping.save

rescue StandardError => e
raise IOError, "Saving backup mapping has failed. Message: #{e.message.to_s}"
end

#second add the mapping id to current submission graphs
rest_predicate = mapping_predicates()['REST'][0]
begin
classes.each do |c|
sub = c.submission
unless sub.id.to_s['latest'].nil?
#the submission in the class might point to latest
sub = LinkedData::Models::Ontology.find(c.submission.ontology.id).first.latest_submission
end
c_id = c.id
graph_id = sub.id
graph_insert = RDF::Graph.new
graph_insert << [c_id, RDF::URI.new(rest_predicate), backup_mapping.id]
Goo.sparql_update_client.insert_data(graph_insert, graph: graph_id)
end
rescue StandardError => e
# Remove the created backup if the following steps of the mapping fail
backup_mapping.delete
raise StandardError, "Inserting the mapping ID in the submission graphs has failed. Message: #{e.message.to_s}"
end

LinkedData::Models::Mapping.new(classes, 'REST', process, backup_mapping.id)
end

def create_mapping_process(mapping_process_hash, source_uri, object_uri, user)
process = LinkedData::Models::MappingProcess.new
relations_array = Array(mapping_process_hash[:relation]).map { |r| RDF::URI.new(r) }
process.relation = relations_array.first
process.creator = user
process.subject_source_id = RDF::URI.new(source_uri || mapping_process_hash[:subject_source_id])
process.object_source_id = RDF::URI.new(object_uri || mapping_process_hash[:object_source_id])
process.date = mapping_process_hash[:date] ? DateTime.parse(mapping_process_hash[:date]) : DateTime.now
process_fields = %i[source source_name comment name source_contact_info]
process_fields.each do |att|
process.send("#{att}=", mapping_process_hash[att]) if mapping_process_hash[att]
end
process
end

private

def save_rest_mapping(classes, process)
LinkedData::Mappings.create_rest_mapping(classes, process)
rescue StandardError => e
# Remove the created process if the following steps of the mapping fail
process.delete
raise ArgumentError, "Loading mapping has failed. Message: #{e.message.to_s}"
end

def save_process(process)
process.save
rescue StandardError => e
raise ArgumentError, "Loading mapping has failed. Message: #{e.message.to_s} : #{process.errors}"
end

def get_mapping_classes(classes:, subject_id:, object_id:)
subject_submission = find_submission_by_ontology_id(subject_id)
subject_class, subject_submission = find_class(classes.first, subject_submission)

object_submission = find_submission_by_ontology_id(object_id)
object_class, object_submission = find_class(classes.last, object_submission)

[object_class, object_submission, subject_class, subject_submission]
end

# Generate URNs for class mapping (urn:ONT_ACRO:CLASS_URI)
def generate_class_urns(classes)
class_urns = []
classes.each do |c|
next if c.nil?

if c.instance_of? LinkedData::Models::Class
acronym = c.submission.id.to_s.split('/')[-3]
class_urns << RDF::URI.new(LinkedData::Models::Class.urn_id(acronym, c.id.to_s))
else
class_urns << RDF::URI.new(c.urn_id())
end
end
class_urns
end

def find_submission_by_ontology_id(ontology_id)
return nil if ontology_id.nil?

o = LinkedData::Models::Ontology.where(submissions: { uri: ontology_id })
.include(submissions: %i[submissionId submissionStatus uri])
.first
o.nil? ? nil : o.latest_submission
end

def find_ontology_by_class(class_instance)
class_instance.submission.bring :ontology
class_instance.submission.ontology
end

def find_submission_by_class_id(class_id)
params = {
require_exact_match: true,
defType: 'edismax',
qf: 'resource_id'
}
query = class_id
search_response = LinkedData::Models::Class.search(query, params)
search_response = search_response['response']['docs']
search_response.each do |resp|
submission_id = resp['ontologyId']
class_instance = LinkedData::Models::OntologySubmission.find(RDF::URI.new(submission_id)).include(:uri).first
return class_instance unless class_instance.nil?
end
nil
end

def find_class(class_id, submission)
submission = find_submission_by_class_id(class_id) if submission.nil?
c = nil
unless submission.nil?
c = LinkedData::Models::Class.find(RDF::URI.new(class_id))
.in(submission)
.first
if c
c.submission.bring :ontology if c.submission.bring?(:ontology)
c.submission.ontology.bring :acronym if c.submission.ontology.bring?(:acronym)
end

end
[c, submission]
end
end
end
end
end

Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
module LinkedData
module Concerns
module OntologySubmission
module MetadataExtractor

def extract_metadata
version_info = extract_version
ontology_iri = extract_ontology_iri

self.version = version_info if version_info
self.uri = ontology_iri if ontology_iri

end

def extract_version

query = Goo.sparql_query_client.select(:versionInfo).distinct
.from(self.id)
.where([RDF::URI.new('http://bioportal.bioontology.org/ontologies/versionSubject'),
RDF::URI.new('http://www.w3.org/2002/07/owl#versionInfo'),
:versionInfo])

sol = query.each_solution.first || {}
sol[:versionInfo]&.to_s
end

def extract_ontology_iri
query = Goo.sparql_query_client.select(:uri).distinct
.from(self.id)
.where([:uri,
RDF::URI.new('http://www.w3.org/1999/02/22-rdf-syntax-ns#type'),
RDF::URI.new('http://www.w3.org/2002/07/owl#Ontology')])
sol = query.each_solution.first || {}
sol[:uri]&.to_s
end
end
end
end
end
Loading