Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

metadata-added-provenance #730

Open
wants to merge 54 commits into
base: dtq-dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
380fb45
added provenance metadata for update, remove and add
Paurikova2 Aug 13, 2024
ffe630a
checkstyle violations
Paurikova2 Aug 14, 2024
46335a1
added provenance metadata for update and remove bitstream
Paurikova2 Aug 14, 2024
96158a5
item test json added metadata provenance
Paurikova2 Aug 14, 2024
9d6097f
removed unwanted lines
Paurikova2 Aug 14, 2024
a0c500a
control provenance only by tests of item
Paurikova2 Aug 15, 2024
c79c000
update and remove license
Paurikova2 Aug 19, 2024
c55d418
provenance access control and upload bitstream
Paurikova2 Aug 20, 2024
499aac4
provenance move item to different collection
Paurikova2 Aug 20, 2024
9c361d9
provenance make item non-discoverable
Paurikova2 Aug 20, 2024
9f94913
checkstyle violations and test failures
Paurikova2 Aug 21, 2024
2a0d919
make code more readable
Paurikova2 Aug 21, 2024
1922907
provenance for mapped collection
Paurikova2 Aug 22, 2024
d9dd689
grammer check
Paurikova2 Aug 22, 2024
5cddf35
don't write provenance for bitstream without item
Paurikova2 Aug 22, 2024
4a6072a
move addProvenance to parent class
Paurikova2 Aug 22, 2024
e468c90
refactoring
Paurikova2 Aug 22, 2024
9910c90
checkstyle violations and test failures
Paurikova2 Aug 22, 2024
18607a5
removed unnecessary variables, added logs and checkstyle
Paurikova2 Sep 10, 2024
aae82cc
separated class for provenance
Paurikova2 Sep 25, 2024
0b3c0d7
create class for provenance management
Paurikova2 Sep 27, 2024
64d84a1
added metadata item and bitstream tests
Paurikova2 Oct 1, 2024
7c8f7e5
tests for metadata provenance
Paurikova2 Oct 2, 2024
43c5eb2
problem with access control test
Paurikova2 Oct 2, 2024
b67b4b3
tests do not work
Paurikova2 Oct 2, 2024
d7a270b
better service management:
Paurikova2 Oct 7, 2024
e4da4e1
delete clarin license mapping
Paurikova2 Oct 7, 2024
5b6168c
checkstyle violations
Paurikova2 Oct 7, 2024
a4bc467
checkstyle violations
Paurikova2 Oct 7, 2024
0e4c9a2
checkstyle violations
Paurikova2 Oct 7, 2024
556c3d4
checkstyle violations
Paurikova2 Oct 7, 2024
e7caf0d
checkstyle violations
Paurikova2 Oct 7, 2024
9b7fb5c
added json to resources
Paurikova2 Oct 7, 2024
eba6bbd
modified provenance patch messages
Paurikova2 Oct 7, 2024
a0d5d55
messages bags
Paurikova2 Oct 7, 2024
5282d63
refactoring based on git comparison
Paurikova2 Oct 7, 2024
828769f
checkstyle violations
Paurikova2 Oct 7, 2024
a1b95e6
used correct json
Paurikova2 Oct 8, 2024
8621bcd
checkstyle violations
Paurikova2 Oct 8, 2024
2fdc96e
log exception, replace ! by non conds, added doc comments
Paurikova2 Oct 10, 2024
b026ca0
added logs for replacing mtd
Paurikova2 Oct 17, 2024
64d7eae
removed unused services
Paurikova2 Nov 20, 2024
49cae22
make object from provenance service
Paurikova2 Nov 22, 2024
eb2315e
removed interface prom provenance msg provider
Paurikova2 Nov 22, 2024
876a7f2
checkstyle
Paurikova2 Nov 22, 2024
130d78f
Refactored method `removeReadPolicies` - add provenance after removin…
milanmajchrak Dec 12, 2024
419aff3
The message templates are loaded from the Enum instead of json file.
milanmajchrak Dec 12, 2024
a0abf72
The `getMessage` methods was overloaded.
milanmajchrak Dec 13, 2024
a1324ce
Renamed `getItem` to `findItemByBitstream` to clarify the method's pu…
milanmajchrak Dec 13, 2024
8ff24e0
Do not throw exception when adding to provenance
milanmajchrak Dec 13, 2024
51302fa
The exceptions are caught in the ProvenanceProvider and changed order…
milanmajchrak Dec 13, 2024
46ab18c
Renamed ProvenanceProvider to ProvenanceService and refactored it to …
milanmajchrak Dec 13, 2024
5f63dcd
The ProvenanceService wasn't initialized in the BundleAccessControl
milanmajchrak Dec 16, 2024
fb82e94
Removal of unwanted changes
milanmajchrak Dec 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -45,14 +45,20 @@
import org.dspace.app.util.DSpaceObjectUtilsImpl;
import org.dspace.app.util.service.DSpaceObjectUtils;
import org.dspace.authorize.AuthorizeException;
import org.dspace.authorize.ResourcePolicy;
import org.dspace.authorize.factory.AuthorizeServiceFactory;
import org.dspace.authorize.service.ResourcePolicyService;
import org.dspace.content.Bitstream;
import org.dspace.content.Collection;
import org.dspace.content.DCDate;
import org.dspace.content.DSpaceObject;
import org.dspace.content.Item;
import org.dspace.content.MetadataSchemaEnum;
import org.dspace.content.factory.ClarinServiceFactory;
import org.dspace.content.factory.ContentServiceFactory;
import org.dspace.content.service.InstallItemService;
import org.dspace.content.service.ItemService;
import org.dspace.content.service.clarin.ClarinItemService;
import org.dspace.core.Constants;
import org.dspace.core.Context;
import org.dspace.discovery.DiscoverQuery;
Expand Down Expand Up @@ -458,13 +464,26 @@ private void updateItemPolicies(Item item, BulkAccessControlInput accessControl)
private void setItemPolicies(Item item, BulkAccessControlInput accessControl)
throws SQLException, AuthorizeException {

String resPoliciesStr = accessControl
Paurikova2 marked this conversation as resolved.
Show resolved Hide resolved
.getItem()
.getAccessConditions()
.stream()
.map(accessCondition -> accessCondition.getName())
.collect(Collectors.joining(";"));

accessControl
.getItem()
.getAccessConditions()
.forEach(accessCondition -> createResourcePolicy(item, accessCondition,
itemAccessConditions.get(accessCondition.getName())));

itemService.adjustItemPolicies(context, item, item.getOwningCollection(), false);

if (resPoliciesStr.isEmpty()) {
return;
}
String msg = "Access condition (" + resPoliciesStr + ") was added to item";
addProvenanceMetadata(context, item, msg);
}

/**
Expand Down Expand Up @@ -552,12 +571,86 @@ private void updateBitstreamPolicies(Bitstream bitstream, Item item, BulkAccessC
*/
private void removeReadPolicies(DSpaceObject dso, String type) {
try {
List<ResourcePolicy> resPolicies = resourcePolicyService.find(context, dso, type);
if (resPolicies.isEmpty()) {
return;
}

String resPoliciesStr = resPolicies.stream()
.filter(rp -> rp.getAction() == Constants.READ) // Filter out non-READ actions
.map(rp -> {
return "[" + rp.getRpName() + ", "
+ rp.getRpType() + ", "
+ rp.getAction() + ", "
+ (rp.getEPerson() == null ? null : rp.getEPerson().getEmail()) + ", "
+ (rp.getGroup() == null ? null : rp.getGroup().getName()) + ", "
+ (rp.getStartDate() == null ? null : rp.getStartDate().toString()) + ", "
+ (rp.getEndDate() == null ? null : rp.getEndDate().toString()) + "]";
})
.collect(Collectors.joining(";"));

resourcePolicyService.removePolicies(context, dso, type, Constants.READ);

if (dso.getType() != Constants.BITSTREAM && dso.getType() != Constants.ITEM) {
return;
}

if (dso.getType() == Constants.ITEM) {
// Add suitable provenance
Item item = (Item) dso;
// Build some provenance data.
String msg = "Resource policies (" + (Objects.equals(resPoliciesStr, "") ? "empty" : resPoliciesStr)
Paurikova2 marked this conversation as resolved.
Show resolved Hide resolved
+ ") of item (" + item.getID() + ") were removed";
addProvenanceMetadata(context, item, msg);
}

if (dso.getType() == Constants.BITSTREAM) {
// Add suitable provenance
Bitstream bitstream = (Bitstream) dso;
ClarinItemService clarinItemService = ClarinServiceFactory.getInstance().getClarinItemService();
List<Item> items = clarinItemService.findByBitstreamUUID(context, dso.getID());
if (items.isEmpty()) {
return;
}
Item item = items.get(0);
// Build some provenance data while we're at it.
String msg = "Resource policies (" + (Objects.equals(resPoliciesStr, "") ? "empty" : resPoliciesStr)
+ ") of bitstream (" + bitstream.getID() + ") were removed";
milanmajchrak marked this conversation as resolved.
Show resolved Hide resolved
addProvenanceMetadata(context, item, msg);
}
} catch (SQLException | AuthorizeException e) {
throw new BulkAccessControlException(e);
}
}

/**
* Adds provenance metadata to an item, documenting changes made to its resource policies
* and bitstream. This method records the user who performed the action, the action taken,
* and the timestamp of the action. It also appends a bitstream provenance message generated
* by the InstallItemService.
*
* @param context the current DSpace context, which provides details about the current user
* and authorization information.
* @param item the DSpace item to which the provenance metadata should be added.
* @param msg a custom message describing the action taken on the resource policies.
* @throws SQLException if there is a database access error while updating the item.
* @throws AuthorizeException if the current user is not authorized to add metadata to the item.
*/
protected void addProvenanceMetadata(Context context, Item item, String msg)
throws SQLException, AuthorizeException {
InstallItemService installItemService = ContentServiceFactory.getInstance().getInstallItemService();
String timestamp = DCDate.getCurrent().toString();
EPerson e = context.getCurrentUser();
StringBuilder prov = new StringBuilder();
Paurikova2 marked this conversation as resolved.
Show resolved Hide resolved
prov.append(msg).append(" by ").append(e.getFullName()).append(" (").append(e.getEmail()).append(") on ")
.append(timestamp).append("\n");
prov.append(installItemService.getBitstreamProvenanceMessage(context, item));
itemService.addMetadata(context, item, MetadataSchemaEnum.DC.getName(),
"description", "provenance", "en", prov.toString());
//Update item in DB
itemService.update(context, item);
}

/**
* create the new resource policies of bitstream.
* then, call {@link ItemService#adjustItemPolicies(
Expand All @@ -573,13 +666,29 @@ private void removeReadPolicies(DSpaceObject dso, String type) {
private void setBitstreamPolicies(Bitstream bitstream, Item item, BulkAccessControlInput accessControl)
throws SQLException, AuthorizeException {

String accConditionsScr = accessControl
.getBitstream()
.getAccessConditions()
.stream()
.map(accessCondition -> accessCondition.getName())
.collect(Collectors.joining(";"));

accessControl.getBitstream()
.getAccessConditions()
.forEach(accessCondition -> createResourcePolicy(bitstream, accessCondition,
uploadAccessConditions.get(accessCondition.getName())));

itemService.adjustBitstreamPolicies(context, item, item.getOwningCollection(), bitstream);
mediaFilterService.updatePoliciesOfDerivativeBitstreams(context, item, bitstream);

if (accConditionsScr.isEmpty()) {
return;
}

// Add suitable provenance
String msg = "Access condition (" + accConditionsScr + ") was added to bitstream (" +
bitstream.getID() + ") of item";
addProvenanceMetadata(context, item, msg) ;
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,16 @@
import org.dspace.app.rest.repository.BundleRestRepository;
import org.dspace.app.rest.utils.ContextUtil;
import org.dspace.app.rest.utils.Utils;
import org.dspace.authorize.AuthorizeException;
import org.dspace.content.Bundle;
import org.dspace.content.DCDate;
import org.dspace.content.Item;
import org.dspace.content.MetadataSchemaEnum;
import org.dspace.content.service.BundleService;
import org.dspace.content.service.InstallItemService;
import org.dspace.content.service.ItemService;
import org.dspace.core.Context;
import org.dspace.eperson.EPerson;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.rest.webmvc.ControllerUtils;
import org.springframework.data.rest.webmvc.ResourceNotFoundException;
Expand Down Expand Up @@ -71,6 +78,12 @@ public class BundleUploadBitstreamController {
@Autowired
private BundleService bundleService;

@Autowired
private ItemService itemService;

@Autowired
private InstallItemService installItemService;

@Autowired
private BundleRestRepository bundleRestRepository;

Expand Down Expand Up @@ -111,6 +124,30 @@ public ResponseEntity<RepresentationModel<?>> uploadBitstream(
throw new UnprocessableEntityException("The InputStream from the file couldn't be read", e);
}

// We do this before calling `updateBitstream` because that function calls `context.commit`
// Add suitable provenance
EPerson e = context.getCurrentUser();
String timestamp = DCDate.getCurrent().toString();
Item item = bundle.getItems().get(0);
milanmajchrak marked this conversation as resolved.
Show resolved Hide resolved

StringBuilder prov = new StringBuilder();
prov.append("Item was added bitstream to bundle (").append(bundle.getID())
.append(") by ").append(e.getFullName()).append(" (").append(e.getEmail())
.append(") on ").append(timestamp).append("\n");
try {
prov.append(installItemService.getBitstreamProvenanceMessage(context, item));
itemService.addMetadata(context, item, MetadataSchemaEnum.DC.getName(),
"description", "provenance", "en", prov.toString());
//Update item in DB
itemService.update(context, item);
} catch (SQLException ex) {
throw new RuntimeException("SQLException in BundleUploadBitstreamConverter.uploadBitstream when " +
milanmajchrak marked this conversation as resolved.
Show resolved Hide resolved
Paurikova2 marked this conversation as resolved.
Show resolved Hide resolved
"adding new provenance metadata.", ex);
} catch (AuthorizeException ex) {
throw new RuntimeException("AuthorizeException in BundleUploadBitstreamConverter.uploadBitstream " +
milanmajchrak marked this conversation as resolved.
Show resolved Hide resolved
"when adding new provenance metadata.", ex);
}

BitstreamRest bitstreamRest = bundleRestRepository.uploadBitstream(
context, bundle, uploadfile.getOriginalFilename(), fileInputStream, properties);
BitstreamResource bitstreamResource = converter.toResource(bitstreamRest);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,13 +31,18 @@
import org.dspace.authorize.AuthorizeException;
import org.dspace.content.Bitstream;
import org.dspace.content.Bundle;
import org.dspace.content.DCDate;
import org.dspace.content.Item;
import org.dspace.content.MetadataSchemaEnum;
import org.dspace.content.clarin.ClarinLicense;
import org.dspace.content.clarin.ClarinLicenseResourceMapping;
import org.dspace.content.service.InstallItemService;
import org.dspace.content.service.ItemService;
import org.dspace.content.service.clarin.ClarinLicenseResourceMappingService;
import org.dspace.content.service.clarin.ClarinLicenseService;
import org.dspace.core.Constants;
import org.dspace.core.Context;
import org.dspace.eperson.EPerson;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
Expand Down Expand Up @@ -98,6 +103,9 @@ public class ItemAddBundleController {
@Autowired
ClarinLicenseResourceMappingService clarinLicenseResourceMappingService;

@Autowired
InstallItemService installItemService;

/**
* Method to add a Bundle to an Item with the given UUID in the URL. This will create a Bundle with the
* name provided in the request and attach this to the Item that matches the UUID in the URL.
Expand Down Expand Up @@ -166,9 +174,18 @@ public ItemRest updateLicenseForBundle(@PathVariable UUID uuid,
"but the new one will not be attached.");
}
List<Bundle> bundles = item.getBundles(Constants.CONTENT_BUNDLE_NAME);
String oldLicense = null;
for (Bundle clarinBundle : bundles) {
List<Bitstream> bitstreamList = clarinBundle.getBitstreams();
for (Bitstream bundleBitstream : bitstreamList) {
if (Objects.isNull(oldLicense)) {
milanmajchrak marked this conversation as resolved.
Show resolved Hide resolved
List<ClarinLicenseResourceMapping> mappings =
this.clarinLicenseResourceMappingService.findByBitstreamUUID(
context, bundleBitstream.getID());
if (!mappings.isEmpty()) {
oldLicense = mappings.get(0).getLicense().getName();
}
}
// in case bitstream ID exists in license table for some reason .. just remove it
this.clarinLicenseResourceMappingService.detachLicenses(context, bundleBitstream);
if (Objects.nonNull(clarinLicense)) {
Expand All @@ -182,6 +199,21 @@ public ItemRest updateLicenseForBundle(@PathVariable UUID uuid,
clarinLicenseService.addLicenseMetadataToItem(context, clarinLicense, item);
}

// Add suitable provenance
EPerson e = context.getCurrentUser();
String timestamp = DCDate.getCurrent().toString();
StringBuilder prov = new StringBuilder();

prov.append("License (").append(Objects.isNull(oldLicense) ? "empty" : oldLicense).append(") was ")
.append(Objects.isNull(clarinLicense) ? "removed" : Objects.isNull(oldLicense) ? "added" : "updated")
.append(" by ").append(e.getFullName()).append(" (").append(e.getEmail()).append(") on ")
.append(timestamp).append("\n");
prov.append(installItemService.getBitstreamProvenanceMessage(context, item));

itemService.addMetadata(context, item, MetadataSchemaEnum.DC.getName(),
"description", "provenance", "en", prov.toString());

// Update item in DB
itemService.update(context, item);
context.commit();

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,16 @@
import org.dspace.authorize.AuthorizeException;
import org.dspace.authorize.service.AuthorizeService;
import org.dspace.content.Collection;
import org.dspace.content.DCDate;
import org.dspace.content.DSpaceObject;
import org.dspace.content.Item;
import org.dspace.content.MetadataSchemaEnum;
import org.dspace.content.service.CollectionService;
import org.dspace.content.service.InstallItemService;
import org.dspace.content.service.ItemService;
import org.dspace.core.Constants;
import org.dspace.core.Context;
import org.dspace.eperson.EPerson;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.rest.webmvc.ResourceNotFoundException;
import org.springframework.security.access.prepost.PostAuthorize;
Expand All @@ -52,6 +56,8 @@ public class ItemOwningCollectionUpdateRestController {

@Autowired
ItemService itemService;
@Autowired
InstallItemService installItemService;

@Autowired
CollectionService collectionService;
Expand Down Expand Up @@ -125,6 +131,26 @@ private Collection moveItem(final Context context, final Item item, final Collec
final boolean inheritPolicies)
throws SQLException, IOException, AuthorizeException {
itemService.move(context, item, currentCollection, targetCollection, inheritPolicies);

// Add suitable provenance
String timestamp = DCDate.getCurrent().toString();
EPerson e = context.getCurrentUser();
StringBuilder prov = new StringBuilder();

prov.append("Item was moved from collection (")
.append(currentCollection.getID()).append(") to different collection by ").append(e.getFullName())
.append(" (").append(e.getEmail()).append(") on ").append(timestamp).append("\n");

prov.append(installItemService.getBitstreamProvenanceMessage(context, item));

itemService.addMetadata(context, item, MetadataSchemaEnum.DC.getName(),
"description", "provenance", "en", prov.toString());
// Update item in DB
// Because a user can move an item without authorization turn off authorization
context.turnOffAuthorisationSystem();
itemService.update(context, item);
context.restoreAuthSystemState();

// Necessary because Controller does not pass through general RestResourceController, and as such does not do
// its commit in DSpaceRestRepository.createAndReturn() or similar
context.commit();
Expand Down
Loading
Loading