Skip to content

Developing backend

Jump to bottom Edit New page
milanmajchrak edited this page Dec 6, 2023 · 26 revisions

Also checkout

Database

($INSTANCE is one number. On dev-5 it can be 5 or 6, see https://github.com/dataquest-dev/DSpace/wiki/Developing-dev-5/ When deployed at customer, it is usally 0)

In order to connect to database, follow these steps:

  1. If run in docker, first exec into it like so: sudo docker exec -it dspacedb$INSTANCE /bin/bash
  2. Then login to postgres with user dspace like so: psql -U dspace -p 543$INSTANCE (You might need to locate psql. It can be placed in postgres installation folder or in its subfolder /bin )
    • As shell suggests, type help for help.
  3. Misc useful commands:
    • \dt to see all tables
    • \d tablename to describe table, e.g. \d eperson
    • select * from eperson; to see all entries in eperson. Note the ; here, which is missing in \d commands.

Alter table

Update Handle table - add an url column there.

  1. Add script to the dspace-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/postgres
  2. Rebuild [dspace-source] to generate fresh dspace folder
  3. Run dspace database migrate command in the dspace/bin

Don't forget to change the database for tests: Testing

Database dumps and migrations

Creating dump

As described here https://www.postgresql.org/docs/current/backup-dump.html it is possible to create dump files and or load them. Creating dump is quite simple, just use pg_dump -U dspace dspace > path/to/file.sql . (NOTE: all programs/scripts are located in [postgres-installation]/bin, best to execute them from there)

In command above, -U dspace means user dspace and second dspace is name of database. I found it easiest to use owner of database.

Importing dump

To import database, it's important to drop all the tables, or alternatively whole database and recreate it. In order to drop and create again, log in as root psql -U postgres. You should then be prompted for password (for user of name postgres, which should be admin). After login check connection info by typing \conninfo. You must not be connected to database you want to drop. Also end any other programs that might be connected, in this scenario namely dspace backend. Drop database by issuing command drop database dspace;.

To create new database, log out by typing \q. Then execute createdb --username=postgres --owner=dspace --encoding=UNICODE dspace.

Log in again as postgres (psql -U postgres), change database to dspace \connect dspace and execute following command: create extension pgcrypto;

Log out again (\q) and execute psql -U postgres dspace < path/to/db_file.sql. Then go to [dspace-installation]/bin and import database with this command dspace database migrate force as well as discover indexes with dspace index-discovery.

Done.

Database installation instructions, in case of first install.

Code

Define autocomplete input field

In dspace/config/submission-forms.xml set input type to <input-type>autocomplete</input-type> and the values will be loaded from the appropriate metadata field.

Harvesting

How to harvest in the Docker

Related issue: https://github.com/dataquest-dev/dspace-angular/issues/48

  1. go to [dspace-angular] - in our case /opt/actions-runner-dq-1/_work/dspace-angular/dspace-angular Dataquest
  2. remove entrypoint in the docker/cli.yml
  3. connect to the docker-cli: docker-compose --env-file build-scripts/run/envs/.default -p dq-d7 -f docker/cli.yml run -v $(pwd)/build-scripts/import/assets:/assets --rm dspace-cli /bin/bash
  4. create collection: ./dspace structure-builder -f /assets/test_community_collection.xml -o /assets/test_import_output.xml -e [email protected]
  5. prepare collection for harvesting: ./dspace harvest -s -c 123456789/2 -a http://lindat.mff.cuni.cz/repository/oai/request -i hdl_11234_3430 -m dc -t 1 -e [email protected]
  6. harvest to the collection: ./dspace harvest -r -c 123456789/2 -a http://lindat.mff.cuni.cz/repository/oai/request -i hdl_11234_3430 -m dc -t 1 -e [email protected]

Unresolved problems:

  • The script harvest.sh harvest the items to the collection 123456789/2 but it works only in the first run because database remember all historical collection ids.

Metadata

How to create a custom local metadata

By configuration

  1. Add the custom metadata to the local-types.xml
  2. Run mvn package in the [dspace-source]/dspace
  3. Run ant fresh_install in the [dspace-source]/dspace/targer/dspace-installer
  4. Run dspace database migrate force in the [dspace-source]/bin

By Admin UI

  1. Login as an Administrator
  2. In the side menu that appears, open the "Registries" menu. Click on "Metadata". This is where all your Metadata fields are defined
  3. Now, you'll see a list of all the metadata schema that DSpace is initialized with. click on the "local" schema to see its fields
  4. The "local" schema is empty by default, so no fields should appear. But, you can add a new field by minimally adding an Element & clicking Save. For example, to create "local.myfield", place the text "myfield" in the Element and click Save. To add a field named "local.myfield.other", add the text "myfield" in the Element and "other" in Qualifier and click Save.

By REST API

Updating a testEnvironment.zip

Some changes are not pushed to the testEnvironment.zip after command mvn package because:

  • '/target/' folder needs to be updated - run mvn clean package not just mvn package
  • After running the test, the 'testEnvironment.zip' is loaded from the .m2 and not created - remove it from the .m2.

License

To add license to files which miss it, call mvn license:format.

Integration Test errors

Expected object but was LinkedHashMap

Use Mathers.is instead of Matchers.hasItem

Create endpoint for new table in DB

Necessary class for create new endpoint for new table in DB:

  1. Object class (Handle.java, Metadatavalue.java...) -> add it to hibernate.cfg
  2. DAO/Impl -> add it to core-dao-services.xml
  3. Service/Impl -> add it to core-services.xml Note: If the service wants to catch the protected constructor of object, it mast be in package org.dspace.content!
  4. RestRepository/IT
  5. Rest
  6. Converter
  7. Resource
  8. Builder
  9. ServiceFacotry/Impl -> add it to core-factory-services.xml

Configure special step e.g. Notice in the submission process for the specific collection

  1. Create a collection - you need to have a handle of that collection
  2. Configure a new submission process in the item-submission.xsl file e.g., https://github.com/dataquest-dev/DSpace/pull/231/files#diff-cbecd668e9534b5ef05a6d8838b62d5f2c3f2ea46fdb1e2f980a2cffeee51d4dR368
  3. Rebuild the DSpace project.

Configure big file limit

  1. Update spring.servlet.multipart.max-file-size and spring.servlet.multipart.max-request-size props to your max upload file limit size in the clarin-dspace.cfg
  2. Update submission-forms.xml - the Big file input field must be set up
<row>
    <field>
        <dc-schema>local</dc-schema>
        <dc-element>bitstream</dc-element>
        <dc-qualifier>redirectToURL</dc-qualifier>
        <repeatable>false</repeatable>
        <label>Big file URL</label>
        <input-type>onebox</input-type>
        <hint>
            The actual maximum upload size of the file is 4GB. To upload the file bigger than maximum
            upload size type the URL of that big file. Admin must know URL to that bitstream file.
            Then click on the 'Save' button and the file will start to upload. The file will be loaded
            from the '/temp' folder of the server. Example: /tomcat/temp/bitstream.png
        </hint>
        <required></required>
        <acl>
            policy=deny,action=read,grantee-type=user,grantee-id=*
        </acl>
    </field>
</row>
  1. Do not forget about nginx and tomcat restrictions
  2. Try to upload big file following this steps: https://github.com/dataquest-dev/DSpace/wiki/For-users#uploading--files-which-are-bigger-than-maximum-upload-size
Add a custom footer
Add a custom sidebar
Clone this wiki locally