avoid creating duplications

[myrmoteras]

@gsautter we need to figure out how to create duplication by the upload for the same articles. This one has been uploaded twice from the exactly the same place, but it created nevertheless duplicates.

Dcoument Name: zlae009.pdf, or zlae127.pdf or zlae121.pdf

[flsimoes]

The zoological journal has this annoying trend of re-releasing the same article under a different name and different hash, which leaves only the DOI as the single flag we can use to check for duplicates, but that's only done after decoding and running the template

[gsautter]

At least going forward, we could try and track the DOIs of the articles we send to GBIF ... however, this would imply we require a DOI for the GBIF export, so it would get in the way of exporting data that comes without a DOI and cannot get one from Zenodo, either. This mainly concerns data coming in as HTML and the like, like the online Flora Helvetica and InfoFlora, or Species Plantarum Volume 1, which we imported from a Project Gutenberg transcript ...

[gsautter]

The zoological journal has this annoying trend of re-releasing the same article under a different name and different hash, which leaves only the DOI as the single flag we can use to check for duplicates, but that's only done after decoding and running the template

Any idea as to why they are doing this? Or maybe more like how this happens? Could just as well be a shoddy system, after all ...

[flsimoes]

The zoological journal has this annoying trend of re-releasing the same article under a different name and different hash, which leaves only the DOI as the single flag we can use to check for duplicates, but that's only done after decoding and running the template

Any idea as to why they are doing this? Or maybe more like how this happens? Could just as well be a shoddy system, after all ...

They publish the ahead of press version and replace it later with the definitive version. Same DOI, but the hash is slightly different with the page numbers and so on.

[gsautter]

The zoological journal has this annoying trend of re-releasing the same article under a different name and different hash, which leaves only the DOI as the single flag we can use to check for duplicates, but that's only done after decoding and running the template

Any idea as to why they are doing this? Or maybe more like how this happens? Could just as well be a shoddy system, after all ...

They publish the ahead of press version and replace it later with the definitive version. Same DOI, but the hash is slightly different with the page numbers and so on.

This kind of raises the question how if we could somehow limit our imports to the definitive version, I guess? In case of a manual process, that should be a fairly easy thing to do, but even if we use a scraper there, there might be some indications in the file names we might exploit?

[flsimoes]

They publish the ahead of press version and replace it later with the definitive version. Same DOI, but the hash is slightly different with the page numbers and so on.

This kind of raises the question how if we could somehow limit our imports to the definitive version, I guess? In case of a manual process, that should be a fairly easy thing to do, but even if we use a scraper there, there might be some indications in the file names we might exploit?

There doesn't seem to be an indication in the file name, but we can check.

[myrmoteras]

for me it looks as I imported twice the same file. So there is nothing prerelease, but just my mistake importing the same file. This is indicated by the the same file number, ie zlae138, zlae130 https://tb.plazi.org/GgServer/srsStats/stats?outputFields=doc.name+doc.articleUuid+bib.source&groupingFields=doc.name+doc.articleUuid+bib.source&orderingFields=-doc.name&FP-doc.name=zlae%25&FP-bib.source=%22Zoolo%25%22&format=HTML
which are the same, but somehow just erroneously imported by me

[myrmoteras]

https://tb.plazi.org/GgServer/summary/FFBC1516430EFF845F504C732830FFBB vs

https://tb.plazi.org/GgServer/summary/54529F1D566DFF872D4BFFB6FFE4FF98