Split merged ids if they lose orig ids

[jmelot]

Closes #45
Closes #47

This PR does two things:

• Discards merged ids that lose an article, forcing new merged ids to be created. This resolves all the cases you originally identified. It also removes about 4.2% of current merged ids, which is higher than I expected. Note that this method can be a little overzealous and doesn't distinguish between ids that are dropped because they are removed by a vendor and ids that still exist in the corpus but are now part of another match set. I think this is the right call because it will force a rerun of e.g. predictions over such articles, whose metadata may change if they lose a merged id, regardless of downstream implementation details. But I want your take.
• Removes simhash

The latter was the result of my descent into investigating what was going on with your spurious matches. The matches you found are not present in the simhash results, which was my original fear. As I looked into simhash again though, I realized that there are extremely few results found by simhash but not the rest of our method (374K links from 53K orig ids). I think as our vendor composition, vendor data, and linkage method (e.g. incorporating cross-dataset links from our vendors, updating the criteria we use to make a match) have evolved, simhash has become less relevant. I think it's just not worth it at this point between the obscure false positives it can introduce and the complexity it requires to incorporate.

[github-actions]

No need for rebasing 👍
behind_count is 0
ahead_count is 4

[github-actions]

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines	Covered	Coverage	Threshold	Status
299	247	83%	0%	🟢

New Files

No new covered files...

Modified Files

File	Coverage	Status
tests/test_create_merge_ids.py	100%	🟢
utils/create_merge_ids.py	82%	🟢
TOTAL	91%	🟢

updated for commit: c132c82 by action🐍

[jmelot]

Brian, I talked with James about this and he approves the method but doesn't have time to do a detailed code review. I'm sorry to ask you to take a look at this unfamiliar code, but if you'd be able to take a look at this early next week so we can get this merged and the pipeline running again that would be very helpful. Let me know if you'd rather go through it live.

[brianlove]

👍

[jamesdunham]

Discards merged ids that lose an article, forcing new merged ids to be created. [...] I think this is the right call because it will force a rerun of e.g. predictions over such articles, whose metadata may change if they lose a merged id, regardless of downstream implementation details.

@jmelot Strong agree with this reasoning. Thank you for thinking this through and implementing.