-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Upgrade from Bulkrax 2.3.0 to 8.0.0, no configuration just yet * Fixes uploads-with-files issue by pointing to bulkrax branch * Work in Progress - tasks to ingest ProQuest ETD zips * WIP - next need to create CSV from array of metadata hashes * WIP - fixed problem creating header row * Fixed embargo logic; fixed CSV structure * Eliminated folder names from metadata csv FileSet entries; copy files to bulkrax zip staging directory, but will need to segregate files from each ETD into separate directoroes * Adds 'bulkrax_identifier' metadata; fixes imports of works w/files, using prerelease of next bulkrax release. * implemented parent work/child FileSet bulkrax_identifier, repaired embargo attributes * refactor file paths for extracted zip; parse creator/contributors * Repair attachment filenames with spaces (or else bulkrax will); fix author parsing * Add degree, advisors, committee members * Add gw_affiliation, date_created * Simplify embargo date; add rights statement; clean up * Fix truncated file; clarify configs, set default rights * Update bulkrax hash, now contains db migration fix * Code cleanup for PR * Add scholarspace-ingest directory and volume mapping * Add mapping for scholarspace-ingest directory * Add CI directive to create ingest folder * Upgrade Bulkrax to 8.1.0 * Allow admin user to visit /importers and /exporters even when there isn't yet an admin set for admin to deposit to * Add fixture zips for bulkrax rspec testing * Add sidekiq inline testing setting * Set testing queue for inline sidekiq * Modify ingest_bulkrax_prep when in test mode * Add bulkrax importer tests * Simplify bulkrax tests * Populates degree and resource_type. License is still WIP, pending input from ScholComm * Added resource_type field ---------
- Loading branch information
Showing
21 changed files
with
637 additions
and
42 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,10 @@ | ||
| # Generated by hyrax:models:install | ||
| class FileSet < ActiveFedora::Base | ||
| # include ::Hyrax::FileSetBehavior | ||
|
|
||
| property :bulkrax_identifier, predicate: ::RDF::URI("https://iro.bl.uk/resource#bulkraxIdentifier"), multiple: false do |index| | ||
| index.as :stored_searchable, :facetable | ||
| end | ||
|
|
||
| include ::Hyrax::FileSetBehavior | ||
| end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,146 @@ | ||
| #!/usr/bin/env ruby | ||
| # frozen_string_literal: true | ||
|
|
||
| require_relative '../config/environment' | ||
|
|
||
| require 'slop' | ||
|
|
||
| def main(opts = {}) | ||
| check_required_params | ||
|
|
||
| update = opts[:importer_id].present? | ||
| port = opts[:port].presence | ||
| url = build_url(opts.delete(:importer_id), opts.delete(:url), port) | ||
|
|
||
| headers = { 'Content-Type' => 'application/json' } | ||
| headers['Authorization'] = "Token: #{opts.delete(:auth_token)}" | ||
| params = build_params(opts) | ||
|
|
||
| logger.info("POST to #{url} - PARAMS #{params}") | ||
|
|
||
| conn = Faraday.new( | ||
| url: url, | ||
| headers: headers | ||
| ) | ||
|
|
||
| response = if update | ||
| conn.put do |request| | ||
| request.body = params.to_json | ||
| end | ||
| else | ||
| conn.post do |request| | ||
| request.body = params.to_json | ||
| end | ||
| end | ||
|
|
||
| puts "#{response.status} - #{response.body.truncate(200)}" | ||
| end | ||
|
|
||
| def check_required_params | ||
| if opts[:importer_id].blank? && invalid?(opts) | ||
| puts 'Missing required parameters' | ||
| help | ||
| end | ||
|
|
||
| if opts[:auth_token].blank? # rubocop:disable Style/GuardClause | ||
| puts 'Missing Authentication Token --auth_token' | ||
| exit | ||
| end | ||
| end | ||
|
|
||
| def invalid?(opts) | ||
| required_params.each do |p| | ||
| return true if opts[p.to_sym].blank? | ||
| end | ||
| return false | ||
| end | ||
|
|
||
| def required_params | ||
| Bulkrax.api_definition['bulkrax']['importer'].map { |key, value| key if value['required'] == true }.compact | ||
| end | ||
|
|
||
| def build_params(opts = {}) | ||
| params = {} | ||
| params[:commit] = opts.delete(:commit) | ||
| parser_fields = { | ||
| metadata_file_name: opts.delete(:metadata_file_name), | ||
| metadata_format: opts.delete(:metadata_format), | ||
| rights_statement: opts.delete(:rights_statement), | ||
| override_rights_statement: opts.delete(:override_rights_statement), | ||
| import_file_path: opts.delete(:import_file_path), | ||
| metadata_prefix: opts.delete(:metadata_prefix), | ||
| set: opts.delete(:set), | ||
| collection_name: opts.delete(:collection_name) | ||
| }.compact | ||
| params[:importer] = opts.compact | ||
| params[:importer][:user_id] = opts.delete(:user_id) | ||
| params[:importer][:admin_set_id] = opts.delete(:admin_set_id) | ||
| params[:importer][:parser_fields] = parser_fields || {} | ||
| return params.compact | ||
| end | ||
|
|
||
| def build_url(importer_id, url, port = nil) | ||
| if url.nil? | ||
| protocol = Rails.application.config.force_ssl ? 'https://' : 'http://' | ||
| host = Rails.application.config.action_mailer.default_url_options[:host] | ||
| url = "#{protocol}#{host}" | ||
| url = "#{url}:#{port}" if port | ||
| end | ||
| path = Bulkrax::Engine.routes.url_helpers.polymorphic_path(Bulkrax::Importer) | ||
| url = File.join(url, path) | ||
| url = File.join(url, importer_id) if importer_id | ||
| return url | ||
| end | ||
|
|
||
| def logger | ||
| Rails.logger | ||
| end | ||
|
|
||
| def version | ||
| puts "Bulkrax #{Bulkrax::VERSION}" | ||
| puts "Slop #{Slop::VERSION}" | ||
| end | ||
|
|
||
| # Format the help for the CLI | ||
| def help | ||
| puts 'CREATE:' | ||
| puts ' bin/importer --name "My Import" --parser_klass Bulkrax::CsvParser --commit "Create and Import" --import_file_path /data/tmp/import.csv --auth_token 12345' | ||
| puts 'UPDATE:' | ||
| puts ' bin/importer --importer_id 1 --commit "Update and Re-Import (update metadata only)" --import_file_path /data/tmp/import.csv --auth_token 12345' | ||
| puts 'PARAMETERS:' | ||
| Bulkrax.api_definition['bulkrax']['importer'].each_pair do |key, value| | ||
| next if key == 'parser_fields' | ||
| puts " --#{key}" | ||
| value.each_pair do |k, v| | ||
| next if k == 'contained_in' | ||
| puts " #{k}: #{v}" | ||
| end | ||
| end | ||
| puts ' --url' | ||
| puts " Repository URL" | ||
| exit | ||
| end | ||
|
|
||
| # Setup the options | ||
| options = Slop.parse do |o| | ||
| o.on '--version', 'Print the version' do | ||
| version | ||
| exit | ||
| end | ||
|
|
||
| o.on '--help', 'Print help' do | ||
| help | ||
| exit | ||
| end | ||
|
|
||
| Bulkrax.api_definition['bulkrax']['importer'].each_pair do |key, value| | ||
| if value['required'].blank? | ||
| o.string "--#{key}", value['definition'], default: nil | ||
| else | ||
| o.string "--#{key}", value['definition'] | ||
| end | ||
| end | ||
| o.string '--url', 'Repository URL' | ||
| end | ||
|
|
||
| main(options.to_hash) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.