API importddiwithjson

[lubitchv]

What this PR does / why we need it:
importddi API does not work at present since during dataset creation it tries to validate required fields but DDI (xml) does not have fields that Dataverse deems to be required, for example "Subject". DDI does not have such field. DDI has keywords but they correspond to dataverse keywords.

There is also sometimes a situation when XML files do not have all the fields that user wants to import to Dataverse, in this case editing of XML is needed before import or updating metadata after import.

This API tries to resolve both situations. API submits xml file and json file. Json file consists of missing fields, for example it can be Subject, Language etc. Json file is similar to a json file for edit dataset metadata API https://guides.dataverse.org/en/latest/api/native-api.html#edit-dataset-metadata

Which issue(s) this PR closes:
Resolves #8210

Suggestions on how to test this:
Integration test DataversesIT#testImportDDI is introduced.

The API can be checked with curl:
curl -H X-Dataverse-key:$API_TOKEN -X POST $SERVER_URL/api/dataverses/$DATAVERSE_ID/datasets/importddiwithjson?replace=true -F 'xml=@src/test/resources/xml/example.xml' -F 'jsonData=@src/test/resources/json/missing-fields.json'

API has following optional parameters:

replace =true/false - if true then replace fields of xml by json fields if the field is in both and creates dataset. If false then add missing files from json to xml and creates dataset. If field exists in both then error is shown.

release=true/false if true then after creation, dataset is published.

pid - import dataset with existing persistentId

Does this PR introduce a user interface change? If mockups are available, please link/include them here:
No

[coveralls]

Coverage decreased (-0.02%) to 18.837% when pulling eafe793 on lubitchv:8210-ddi-import into a629d18 on IQSS:develop.

[pdurbin]

Not sure why the Jenkins job failed. https://jenkins.dataverse.org/job/IQSS-Dataverse-Develop-PR/job/PR-8447/1/consoleFull ended with this:

[JaCoCo plugin] Loading packages..
[JaCoCo plugin] Done.
[JaCoCo plugin] Overall coverage: class: 37.92017, method: 21.559065, line: 18.84934, branch: 15.647721, instruction: 19.508333, complexity: 15.2561245
[Pipeline] script
[Pipeline] {
[Pipeline] fileExists
[Pipeline] error
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: Ansible run terminated abnormally, failing build.

GitHub has been notified of this commit’s build result

Finished: FAILURE

Also, I added this to Review so someone can take a look. Thanks for the pull request, @lubitchv !

---

Update: I merged the latest from develop in eafe793 and now all the test are passing.

[pdurbin]

@lubitchv @scolapasta @qqmyers @landreev thanks for all the discussion around this pull request today.

Please see #8210 (comment) for the direction we're planning. In short, we're closing this pull request that adds a DDI+JSON API endpoint and focusing on fixing the broken importddi endpoint. Existing JSON endpoints can be used afterward to add Subject or other metadata fields. Thanks @lubitchv for taking this on!