10288 Add keywordTermURI metadata in keyword block #10371

stevenferey · 2024-03-13T10:11:56Z

What this PR does / why we need it:

To continue on the subject of OntoPortal integration

Which issue(s) this PR closes:

Special notes for your reviewer:

In addition to the addition of the keywordTermURI field, we added a script allowing administrators to optionally migrate data from keywordValue to keywordTermURI when the data starts with "http".

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

The keyword metadata block is modified:

Before data migration, here is an example with a keywordValue containing data starting with "http" :
After optional data migration :

Is there a release notes update needed for this change?:

Yes, a proposal is visible in the file doc/release-notes/10288-add-term_uri-metadata-in-keyword-block.md

…h its name Controlled Vocabulary URL

…rd-block

jggautier · 2024-04-03T13:59:33Z

Heya @stevenferey. Have you had a chance to check out the guidelines for metadata text at https://docs.google.com/document/d/1tY5t3gjrIgAGoRxVMWQSCh46fnbSmnFDLQ7aLkNLhJ8?

qqmyers · 2024-04-03T14:27:50Z

src/main/java/propertyFiles/citation.properties

 datasetfieldtype.keywordValue.title=Term
 datasetfieldtype.keywordVocabulary.title=Controlled Vocabulary Name
-datasetfieldtype.keywordVocabularyURI.title=Controlled Vocabulary URL
+datasetfieldtype.keywordVocabularyURL.title=Controlled Vocabulary URL


Why does this need to change to URL from URI? (While you're adding Term URI?) Minimally I think we should be consistent and, unless we're sure that there are no URNs in use, sticking with URI might be the better option. (I do see that we focus on URLs in the descriptions and the idea of web locations for things instead of talking about URIs as identifiers that might also be resolvable on the web. I'm not sure what the best way to resolve that is, or that it needs to be handled in this PR.) @jggautier - any thoughts?

I'm not sure and might not understand this enough. But maybe I could help see if there are URNs in use?

In case this info is helpful, I think we used "keywordVocabularyURI" for the database name because DDI Codebook uses "vocabURI". And maybe we thought that most people would be more familiar with the term "URLs", so that's what we used for the label people see on the page and in the tooltips. Of course this was all before I joined Dataverse.

After discussion with Recherche-Data-Gouv team, we decided to rollback modifications from keywordVocabularyURI to keywordVocabularyURL. Update is coming.

Update is done @qqmyers :)

…adata-in-Keyword-block_v2 10288 - rollback keywordVocabularyURL to keywordVocabularyURI

qqmyers · 2024-04-24T13:11:50Z

src/main/resources/db/migration/V6.2.0.1__10288-update_keywordVocabularyURI_metadata.sql

@@ -0,0 +1,5 @@
+-- update of the "keywordVocabularyURI" metadata to make it consistent with its name "Controlled Vocabulary URL"


This is obsolete now?

The PR description also still mentions changing from URI to URL (not in the code, but might confuse in review or QA)

Yes absolutely, I delete the script and update the PR description

qqmyers · 2024-04-25T11:09:07Z

@jggautier - any reason this can't get on the board now to move forward?

jggautier · 2024-04-25T15:27:46Z

Hmm maybe. Could this get on the board in the next sprint's "This Sprint" column while we wait to hear back from @stevenferey about the tooltip text and how it does or doesn't follow the guidelines for metadata text? Or should that be resolved first?

I'd like to make sure that those guidelines are easy to find and are helpful when folks are working on the text associated with metadata, like the field labels and tooltips shown in the UI.

@stevenferey, I asked about this a few weeks ago because the current tooltip text for this Term URI field makes me think that you haven't seen those guidelines because they're hard to find or that the guidelines could be clearer.

stevenferey · 2024-04-26T15:16:00Z

Sorry for the late response,
I was able to read the document after your message.

Is this description better?
Keyword URI points to the web presence of the term. Enter an absolute URI, e.g. starting by https://

We listen to your suggestions for improvement.

jggautier · 2024-04-26T19:50:48Z

Hey @stevenferey. Thanks for taking a look!

The more important thing is to avoid adding instructions in the tooltip for how depositors should add metadata, since people looking for data also see those tooltips and that information isn't relevant for them. And we hope that the watermark and the field validation will ensure that depositors are entering what you expect.

So I think we should remove the second sentence about entering an absolute URI. And the rest of the changes I suggested below are mostly style changes for consistency:

A URI that points to the web presence of the Keyword Term

jggautier · 2024-04-26T20:22:50Z

It would be cool if we could encourage folks to use a tool that would apply our style guidelines to their text.

The paid versions of Grammarly and a less popular tool called ProWritingAid say they can do this. Maybe there are free tools, too!

stevenferey · 2024-05-02T12:40:42Z

Thank you for this feedback,

I pushed a label update to follow the recommendation.

jggautier · 2024-05-02T13:49:21Z

@qqmyers after you asked me last if this could get on the board to move forward, I asked if it could while we waited to hear back about the tooltip text or if that should be addressed first.

I think the issue with that text's been addressed, so I can't think of any reason why this couldn't get on the board to move forward.

qqmyers · 2024-05-15T10:49:01Z

src/main/java/edu/harvard/iq/dataverse/export/openaire/OpenAireExportUtil.java

                                }
                            }

                            if (StringUtils.isNotBlank(subject)) {
                                subject_check = writeOpenTag(xmlw, "subjects", subject_check);
-                                writeSubjectElement(xmlw, subjectScheme, schemeURI, subject, language);
+                                // we prioritize the keywordTermURI metadata to populate schemeURI
+                                writeSubjectElement(xmlw, subjectScheme, StringUtils.isNotBlank(keywordTermURI) ? keywordTermURI : keywordVocabURI, subject, language);


@stevenferey - The DataCite Schema includes both a schemeURI and valueURI attributes - should this now include both rather than swapping the term URI into the schemeURI attribute?

@qqmyers , Thank you for this feedback
Indeed we can consider using the two metadata as specified in the xsd

With the following example:

We would obtain the following export:

<subjects> <subject>Agricultural Sciences</subject> <subject valueURI="https://term-uri.com/" schemeURI="https://vocabulary-uri.com/" subjectScheme="cvoc name">keyword term</subject> </subjects>

Does this correspond to the proposal?

@qqmyers, The branch is updated with the modification of the OpenAire export. Thanks a lot

…rd-block

coveralls · 2024-05-15T13:44:22Z

coverage: 20.589% (+0.009%) from 20.58%
when pulling a79cdbf on Recherche-Data-Gouv:10288-add-Term-URI-metadata-in-Keyword-block
into 1f9a682 on IQSS:develop.

…rd-block

qqmyers · 2024-05-28T16:25:54Z

@stevenferey - I think the change looks good. There is a (probably unrelated) failing test though that may be related to destroying a dataset - can you merge with develop again to pick up #10566. Details: expected: <404> but was: <500>
Stacktrace
org.opentest4j.AssertionFailedError: expected: <404> but was: <500>
at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:145)
at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:531)
at edu.harvard.iq.dataverse.api.HarvestingServerIT.testSingleRecordOaiSet(HarvestingServerIT.java:630)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

…t():L137)

luddaniel · 2024-05-29T14:00:54Z

Hello @qqmyers, as @stevenferey is in holiday for a full month, I will handle the next exchanges.
This branch is already up to date, and by jumping into debug mode I can tell that the expected should be <500>.

Caused by: jakarta.persistence.NoResultException: getSingleResult() did not retrieve any entities. handled error will throw .add("code", 500) response.
By fast crawling git blame I would say that no code has been modified for the last 3 months so I don't know when this IT code started to fail.
I did go ahead and fixed the error in the last commit a79cdbf, tell me if you want to do something else.

qqmyers · 2024-05-29T16:55:42Z

@luddaniel - thanks. We're seeing the test fail in other branches, one with only docs changes, so definitely unrelated. That said, we're looking to see if we can understand why it started failing recently and will probably want to pick up your try/catch fix. I'll let you know if there's anything else to do.

pdurbin · 2024-05-30T18:06:52Z

will probably want to pick up your try/catch fix

Yes, I cherry-picked it here:

catch exceptions from DatasetServiceBean.findDeep #10601

Thanks, @luddaniel!

pdurbin · 2024-06-03T15:49:15Z

@stevenferey we cherry-picked a commit from this PR into #10601 and merged it. Can you please merge develop into this PR? Thanks.

qqmyers

I think this is done at this point. The last build failed with the AWS instance not starting but probably not related. (Should rerun at some point.)

stevenwinship · 2024-06-14T14:19:11Z

Issue with "Data migration to the new keywordTermURI field"
No validation takes place until someone tries to edit the metadata in the UI

Before update:

After Update:

After trying to update the metadata:

qqmyers · 2024-06-14T14:39:50Z

I think this is how all fields of type url work, i.e. not related to the one field added in the PR.

stevenferey and others added 9 commits February 7, 2024 17:10

New keywordTermURI Metadata in keyword Metadata Block

a5f4bd7

update of the keywordVocabularyURI metadata to make it consistent wit…

9f72845

…h its name Controlled Vocabulary URL

fix description and watermark properties

c103610

Merge branch 'develop' into 10288-add-Term-URI-metadata-in-Keyword-block

ebe2e02

10288 adding documentation

5e9d934

10288 - adaptation of the SolR schema and dataset exports

221311c

10288 - Adjustment of typo and sql

f4efb01

Merge branch 'IQSS:develop' into 10288-add-Term-URI-metadata-in-Keywo…

5e6f37f

…rd-block

Adaptations for Dataverse 6.2

3464212

qqmyers reviewed Apr 3, 2024

View reviewed changes

luddaniel and others added 3 commits April 17, 2024 11:37

10288 - rollback keywordVocabularyURL to keywordVocabularyURI

c54b390

Merge pull request #3 from Recherche-Data-Gouv/10288-add-Term-URI-met…

973e026

…adata-in-Keyword-block_v2 10288 - rollback keywordVocabularyURL to keywordVocabularyURI

Merge branch 'develop' into 10288-add-Term-URI-metadata-in-Keyword-block

ff70b10

qqmyers reviewed Apr 24, 2024

View reviewed changes

10288 - removing obsolete SQL script

7a980ab

10288 - Label modification to follow Dataverse recommendations

05feae2

qqmyers reviewed May 15, 2024

View reviewed changes

Merge branch 'IQSS:develop' into 10288-add-Term-URI-metadata-in-Keywo…

b0a5b23

…rd-block

stevenferey and others added 2 commits May 27, 2024 14:52

Merge branch 'IQSS:develop' into 10288-add-Term-URI-metadata-in-Keywo…

cc3e551

…rd-block

10288 - Added valueURI attribute for OpenAire export

165100d

Fix NoResultException on DatasetServiceBean.findDeep (.getSingleResul…

a79cdbf

…t():L137)

pdurbin mentioned this pull request May 30, 2024

catch exceptions from DatasetServiceBean.findDeep #10601

Merged

Merge branch 'develop' into 10288-add-Term-URI-metadata-in-Keyword-block

d42b388

qqmyers approved these changes Jun 13, 2024

View reviewed changes

qqmyers added the Size: 10 A percentage of a sprint. 7 hours. label Jun 13, 2024

stevenwinship self-assigned this Jun 14, 2024

stevenwinship merged commit 00020e2 into IQSS:develop Jun 14, 2024
15 checks passed

stevenwinship removed their assignment Jun 14, 2024

pdurbin added this to the 6.3 milestone Jun 14, 2024

qqmyers mentioned this pull request Jun 28, 2024

Feature Request/Idea:Implement mechanism to support integration with OntoPortal #9276

Closed

2 tasks

luddaniel deleted the 10288-add-Term-URI-metadata-in-Keyword-block branch September 19, 2024 14:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

10288 Add keywordTermURI metadata in keyword block #10371

10288 Add keywordTermURI metadata in keyword block #10371

stevenferey commented Mar 13, 2024 •

edited

Loading

jggautier commented Apr 3, 2024

qqmyers Apr 3, 2024

jggautier Apr 3, 2024

luddaniel Apr 17, 2024

luddaniel Apr 24, 2024

qqmyers Apr 24, 2024

qqmyers Apr 24, 2024

stevenferey Apr 24, 2024

qqmyers commented Apr 25, 2024

jggautier commented Apr 25, 2024 •

edited

Loading

stevenferey commented Apr 26, 2024

jggautier commented Apr 26, 2024

jggautier commented Apr 26, 2024 •

edited

Loading

stevenferey commented May 2, 2024

jggautier commented May 2, 2024

qqmyers May 15, 2024

stevenferey May 15, 2024

stevenferey May 28, 2024

coveralls commented May 15, 2024 •

edited

Loading

qqmyers commented May 28, 2024

luddaniel commented May 29, 2024

qqmyers commented May 29, 2024

pdurbin commented May 30, 2024

pdurbin commented Jun 3, 2024

qqmyers left a comment

stevenwinship commented Jun 14, 2024

qqmyers commented Jun 14, 2024

		@@ -0,0 +1,5 @@
		-- update of the "keywordVocabularyURI" metadata to make it consistent with its name "Controlled Vocabulary URL"

10288 Add keywordTermURI metadata in keyword block #10371

10288 Add keywordTermURI metadata in keyword block #10371

Conversation

stevenferey commented Mar 13, 2024 • edited Loading

jggautier commented Apr 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qqmyers commented Apr 25, 2024

jggautier commented Apr 25, 2024 • edited Loading

stevenferey commented Apr 26, 2024

jggautier commented Apr 26, 2024

jggautier commented Apr 26, 2024 • edited Loading

stevenferey commented May 2, 2024

jggautier commented May 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented May 15, 2024 • edited Loading

qqmyers commented May 28, 2024

luddaniel commented May 29, 2024

qqmyers commented May 29, 2024

pdurbin commented May 30, 2024

pdurbin commented Jun 3, 2024

qqmyers left a comment

Choose a reason for hiding this comment

stevenwinship commented Jun 14, 2024

qqmyers commented Jun 14, 2024

stevenferey commented Mar 13, 2024 •

edited

Loading

jggautier commented Apr 25, 2024 •

edited

Loading

jggautier commented Apr 26, 2024 •

edited

Loading

coveralls commented May 15, 2024 •

edited

Loading