Fuzzy matching for journal abbreviations #12477

arraymahdi · 2025-02-10T08:53:43Z

Summary

This PR enhances the Abbreviation class by introducing fuzzy matching to improve abbreviation comparisons. It allows for minor variations, such as punctuation differences, extra spaces, slight typos, and case insensitivity, ensuring a more flexible and accurate comparison system.

Changes Made

✅ Implemented Fuzzy Matching in isSimilar Method

Uses Levenshtein Distance to allow minor typos (up to 2 character differences).
Normalizes text by removing punctuation and converting it to lowercase.

✅ Enhanced compareTo to Use Fuzzy Matching

Abbreviations with slight variations (e.g., "L. N." vs. "L N") are now considered equal if they meet the similarity threshold.
Ensures consistent ordering even when minor differences exist.

✅ Added Unit Tests for Fuzzy Matching

Handles minor typos: "Long Name" vs. "Longn Name".
Ignores punctuation differences: "Long Name" vs. "Long, Name".
Supports case insensitivity: "Long Name" vs. "LONG NAME".
Handles missing spaces: "L. N." vs. "L N".

Mandatory checks

I own the copyright of the code submitted and I licence it under the MIT license
Change in CHANGELOG.md described in a way that is understandable for the average user (if change is visible to the user)
Tests created for changes (if applicable)
Manually tested changed features in running JabRef (always required)
Screenshots added in PR description (for UI changes)
Checked developer's documentation: Is the information available and up to date? If not, I outlined it in this pull request.
Checked documentation: Is the information available and up to date? If not, I created an issue at https://github.com/JabRef/user-documentation/issues or, even better, I submitted a pull request to the documentation repository.

…ve search accuracy

Siedlerchr · 2025-02-10T08:59:27Z

build.gradle

@@ -178,6 +178,7 @@ dependencies {
    implementation "org.apache.lucene:lucene-queries:$luceneVersion"
    implementation "org.apache.lucene:lucene-analysis-common:$luceneVersion"
    implementation "org.apache.lucene:lucene-highlighter:$luceneVersion"
+    implementation 'org.apache.commons:commons-text:1.13.0'


No need for Apache commons text, we already have a Levenstein distance implementation

jabref/src/main/java/org/jabref/logic/util/strings/StringSimilarity.java

Line 8 in c6e317d

private final Levenshtein METRIC_DISTANCE = new Levenshtein();

github-actions

Your code currently does not meet JabRef's code guidelines.
We use OpenRewrite to ensure "modern" Java coding practices.
The issues found can be automatically fixed.
Please execute the gradle task rewriteRun, check the results, commit, and push.

You can check the detailed error output by navigating to your pull request, selecting the tab "Checks", section "Tests" (on the left), subsection "OpenRewrite".

github-actions

Your code currently does not meet JabRef's code guidelines.
We use OpenRewrite to ensure "modern" Java coding practices.
The issues found can be automatically fixed.
Please execute the gradle task rewriteRun, check the results, commit, and push.

You can check the detailed error output by navigating to your pull request, selecting the tab "Checks", section "Tests" (on the left), subsection "OpenRewrite".

github-actions

Your code currently does not meet JabRef's code guidelines.
We use OpenRewrite to ensure "modern" Java coding practices.
The issues found can be automatically fixed.
Please execute the gradle task rewriteRun, check the results, commit, and push.

You can check the detailed error output by navigating to your pull request, selecting the tab "Checks", section "Tests" (on the left), subsection "OpenRewrite".

Siedlerchr · 2025-02-10T09:56:43Z

build.gradle

@@ -73,34 +73,34 @@ application {
    mainModule.set('org.jabref')

    applicationDefaultJvmArgs = [
-        // On a change here, also adapt


please revert the formatting changes

I replaced the file with the original

arraymahdi · 2025-02-10T09:58:14Z

Sure!

github-actions

Your code currently does not meet JabRef's code guidelines.
We use OpenRewrite to ensure "modern" Java coding practices.
The issues found can be automatically fixed.
Please execute the gradle task rewriteRun, check the results, commit, and push.

You can check the detailed error output by navigating to your pull request, selecting the tab "Checks", section "Tests" (on the left), subsection "OpenRewrite".

subhramit

Hey, welcome to JabRef and thank you for your contribution!
I was unable to understand the need of a lot of changes which caused diff, so kindly explain the "why" in the comments below, and revert them.

Furthermore, is the PR description copy-pasted from somewhere? Please do not remove the mandatory checks and link the issue properly.

subhramit · 2025-02-10T12:06:12Z

src/main/java/org/jabref/logic/journals/Abbreviation.java

-    private transient String name;
-    private final String abbreviation;
-    private transient String dotlessAbbreviation;


Is there a specific reason why the order was changed?

No, but I removed some functions and rewrote them, which caused changes to the code structure

I don't see how that should remove class fields, but alright.

subhramit · 2025-02-10T12:06:24Z

src/main/java/org/jabref/logic/journals/Abbreviation.java


-    // Is the empty string if not available


Why are the comments removed?

I’m sorry about that. This is my first open-source contribution! Personally, I often find comments to be annoying!

It's alright. Read on why comments are used in code, and when they should be used. Furthermore, in open-source, whenever you touch existing code - know that it it was written by another developer and had gotten approved to be merged. So by default, any changes made to that (that don't pertain to the issue being solved) should be justified.

subhramit · 2025-02-10T12:06:56Z

src/main/java/org/jabref/logic/journals/Abbreviation.java

@@ -21,12 +25,12 @@ public Abbreviation(String name, String abbreviation) {
    public Abbreviation(String name, String abbreviation, String shortestUniqueAbbreviation) {
        this(name,
                abbreviation,
-                // "L. N." becomes "L  N ", we need to remove the double spaces inbetween


Have you replaced the code directly from somewhere?

No, but I removed some functions and rewrote them more than once, which caused changes to the code structure

src/main/java/org/jabref/logic/journals/Abbreviation.java

subhramit · 2025-02-10T12:15:33Z

src/main/java/org/jabref/logic/journals/Abbreviation.java

-        return Objects.hash(getName(), getAbbreviation(), getShortestUniqueAbbreviation());
+        return Objects.hash(normalize(getName()), normalize(getAbbreviation()), normalize(getShortestUniqueAbbreviation()));


Should the hashcode stay the same if two abbreviations have the same "normalized" name?

I did this to make sure that both values consider the same

From what I understand, there is no difference between the two names except for their case.

Right, but my question is, why does the hashCode need to change in that case? Where do you use/plan to use it? I don't see any direct equality check or use of hashing anywhere in your changes.

Please do not mark discussions resolved until the person who asked the question does so.

Ok, I'll revert the change!

github-actions

Your code currently does not meet JabRef's code guidelines.
We use OpenRewrite to ensure "modern" Java coding practices.
The issues found can be automatically fixed.
Please execute the gradle task rewriteRun, check the results, commit, and push.

You can check the detailed error output by navigating to your pull request, selecting the tab "Checks", section "Tests" (on the left), subsection "OpenRewrite".

github-actions

Your code currently does not meet JabRef's code guidelines.
We use OpenRewrite to ensure "modern" Java coding practices.
The issues found can be automatically fixed.
Please execute the gradle task rewriteRun, check the results, commit, and push.

You can check the detailed error output by navigating to your pull request, selecting the tab "Checks", section "Tests" (on the left), subsection "OpenRewrite".

subhramit · 2025-02-10T13:53:21Z

@arraymahdi please don't resolve the conversations if it involves a question or discussion. If the questioner is satisfied, they'll resolve it themselves or reply that they're fine with your explanation. Otherwise, we have to expand it to see your responses and carry it forward. For simple comments like the "revert" ones, it's fine.

arraymahdi · 2025-02-10T13:57:51Z

@arraymahdi please don't resolve the conversations if it involves a question or discussion. If the questioner is satisfied, they'll resolve it themselves or reply that they're fine with your explanation. Otherwise, we have to expand it to see your responses and carry it forward. For simple comments like the "revert" ones, it's fine.

Noted.

subhramit · 2025-02-10T14:07:19Z

For failing tests refer to https://devdocs.jabref.org/code-howtos/faq.html

koppor

Please add test case for non-latin characters

Chinese Science Bulletin (中国科学通报) → Chin. Sci. Bull.

Full Title (中文)	Abbreviation (中文)
中国物理学报	物理学报
中国科学: 物理学	中科物理
数学学报	数学报
计算机学报	计学报
电子学报	电学报
化学学报	化学报

Also for German

I think, the current implemetnation would consider all chinese journal equal, which is wrong.

For French and German, your approach could work, because the likelyhood that there are only accented letters is low.

Please see the discussion at JabRef#235

Normalization should NOT remove characters, it should really normalize it.

I propose to sue ASCIIFoldingFilter for normalization

I also think that equals should not use normalization, because there could be mistakes made there?!

koppor · 2025-02-10T14:05:43Z

src/main/java/org/jabref/logic/journals/JournalAbbreviationRepository.java

@@ -152,7 +152,7 @@ public void addCustomAbbreviations(Collection<Abbreviation> abbreviationsToAdd)
    }

    public Optional<String> getNextAbbreviation(String text) {
-        return get(text).map(abbreviation -> abbreviation.getNext(text));
+        return get(text).map(abbreviation -> abbreviation.getNext(text.trim()));


Why a trim at the end but no trim at the beginning? Shoulnd't be text be trimmed on method entry?

"Thank you very much for your insightful feedback! I’ve reviewed all your comments and did my best to implement them."

koppor · 2025-02-10T14:05:58Z

src/main/java/org/jabref/logic/journals/Abbreviation.java

+    private static String normalize(String input) {
+        return input.toLowerCase().replaceAll("[^a-z0-9 ]", "").trim();
+    }
+
    @Override
    public int compareTo(Abbreviation toCompare) {


What you described in the PR description should appear as JavaDoc here.

I added DocStrings to the code.

koppor · 2025-02-10T14:13:18Z

src/main/java/org/jabref/logic/journals/Abbreviation.java

+        return distance <= 2;
+    }
+
+    private static String normalize(String input) {


Please add JavaDoc

koppor · 2025-02-10T14:13:51Z

src/main/java/org/jabref/logic/journals/Abbreviation.java

@@ -103,7 +130,9 @@ public boolean equals(Object o) {
            return false;
        }
        Abbreviation that = (Abbreviation) o;
-        return getName().equals(that.getName()) && getAbbreviation().equals(that.getAbbreviation()) && getShortestUniqueAbbreviation().equals(that.getShortestUniqueAbbreviation());
+        return Objects.equals(normalize(name), normalize(that.name)) &&


Add some JavaDoc

I think, there should be no normalization here!

I reverted this change

koppor · 2025-02-10T14:19:43Z

src/test/java/org/jabref/logic/journals/AbbreviationTest.java

    @Test
    void equalAbbrevationsWithFourComponentsAreAlsoCompareZero() {
        Abbreviation abbreviation1 = new Abbreviation("Long Name", "L. N.", "LN");
        Abbreviation abbreviation2 = new Abbreviation("Long Name", "L. N.", "LN");
        assertEquals(0, abbreviation1.compareTo(abbreviation2));
    }
+
+    @Test
+    void testCompareToWithFuzzyMatching() {


Please convert to parameterized Test

I added a parameterized for compareTo

koppor · 2025-02-10T14:20:04Z

src/main/java/org/jabref/logic/journals/Abbreviation.java

+        String normalizedOther = normalize(otherName);
+
+        int distance = LEVENSHTEIN.apply(normalizedThis, normalizedOther);
+        return distance <= 2;


Please add a reasoning why "2" works.

koppor · 2025-02-10T14:21:50Z

src/main/java/org/jabref/logic/journals/Abbreviation.java

@@ -1,12 +1,19 @@
+


No additional empty line at the beginning

koppor · 2025-02-10T14:22:58Z

To ease organizational workflows I have linked the pull-request to the issue with syntax as described in https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue

Linking a pull request to an issue using a keyword

You can link a pull request to an issue by using a supported keyword in the pull request's description or in a commit message. The pull request must be on the default branch.

close

closes

closed

fix

fixes

fixed

resolve

resolves

resolved

If you use a keyword to reference a pull request comment in another pull request, the pull requests will be linked. Merging the referencing pull request also closes the referenced pull request.

The syntax for closing keywords depends on whether the issue is in the same repository as the pull request.

Example:

✅ Fixes #xyz links pull-request to issue. Merging the PR will close the Issue.
✅ Fixes https://github.com/JabRef/jabref/issues/xyz links pull-request to issue. Merging the PR will close the Issue.
✅ Fixes https://github.com/Koppor/jabref/issues/xyz links pull-request to issue. Merging the PR will close the Issue.
❌ Fixes [#xyz](https://github.com/JabRef/jabref/issues/xyz) links pull-request to issue. Merging the PR will NOT close the Issue.

calixtus · 2025-02-10T23:26:00Z

Please do not remove the 'Mandatory checks' from the PR description. I manually readded them. Note that we cannot accept any change if you don't guarantee to own the ip rights of to the code submitted and license them under the terms of the MIT license. In this case, check the checkbox. If not, we will reject your changes and close this PR.

subhramit · 2025-02-11T06:33:41Z

@arraymahdi this is not a UI change, yet I see the box for "added screenshots in PR description" checked. We don't put the checks for the sake of formality. Please read what they say before you tick.

github-actions

Your code currently does not meet JabRef's code guidelines.
We use OpenRewrite to ensure "modern" Java coding practices.
The issues found can be automatically fixed.
Please execute the gradle task rewriteRun, check the results, commit, and push.

You can check the detailed error output by navigating to your pull request, selecting the tab "Checks", section "Tests" (on the left), subsection "OpenRewrite".

koppor · 2025-02-16T16:48:58Z

@arraymahdi Please, please, please, before requesting reviews fix the tests.

koppor

DO NOT REMOVE FILES

Some files of .vscode dir were removed. This is not OK without checking with the devs.

subhramit · 2025-02-16T21:40:14Z

@arraymahdi this is not a UI change, yet I see the box for "added screenshots in PR description" checked. We don't put the checks for the sake of formality. Please read what they say before you tick.

I also don't see any changelog entry, yet you've ticked the check.
Copy above, again. These are basic etiquettes we expect anybody genuinely interested in contributing to follow - to "read" what's written - otherwise all this distracts us from the work done in the PR.

arraymahdi added 4 commits February 10, 2025 09:55

Add Apache Commons Text dependency

9ce1f36

Implement fuzzy matching for journal names and abbreviations to impro…

8c34a3a

…ve search accuracy

Add tests for fuzzy matching of journal names and abbreviations

0435f38

Update deprecated functions to current standards

e241bf9

Siedlerchr requested changes Feb 10, 2025

View reviewed changes

github-actions bot reviewed Feb 10, 2025

View reviewed changes

arraymahdi added 2 commits February 10, 2025 12:26

Update the equal method

14aa1ba

Remove a redundent dependency

acfd019

github-actions bot reviewed Feb 10, 2025

View reviewed changes

Remove a redundent dependency

c6a0d0b

github-actions bot reviewed Feb 10, 2025

View reviewed changes

Siedlerchr reviewed Feb 10, 2025

View reviewed changes

Fix a formatting issue

50a12c2

github-actions bot reviewed Feb 10, 2025

View reviewed changes

arraymahdi requested a review from Siedlerchr February 10, 2025 10:09

subhramit reviewed Feb 10, 2025

View reviewed changes

arraymahdi added 2 commits February 10, 2025 16:06

Fix formatting issue

df16223

Fix formatting issue

bd5b648

github-actions bot reviewed Feb 10, 2025

View reviewed changes

arraymahdi requested a review from subhramit February 10, 2025 13:25

Fix formatting issue

b63c073

github-actions bot reviewed Feb 10, 2025

View reviewed changes

koppor requested changes Feb 10, 2025

View reviewed changes

koppor reviewed Feb 10, 2025

View reviewed changes

subhramit changed the title ~~Feature fuzzy matching for journal abbreviations #12467~~ Fuzzy matching for journal abbreviations Feb 11, 2025

Remove IDE folders

8e62cdc

arraymahdi requested a review from koppor February 12, 2025 14:04

github-actions bot reviewed Feb 12, 2025

View reviewed changes

koppor requested changes Feb 16, 2025

View reviewed changes

koppor marked this pull request as draft February 16, 2025 16:50

		return Objects.hash(getName(), getAbbreviation(), getShortestUniqueAbbreviation());
		return Objects.hash(normalize(getName()), normalize(getAbbreviation()), normalize(getShortestUniqueAbbreviation()));

Fuzzy matching for journal abbreviations #12477

Are you sure you want to change the base?

Fuzzy matching for journal abbreviations #12477

Conversation

arraymahdi commented Feb 10, 2025 • edited Loading

Summary

Changes Made

Mandatory checks

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arraymahdi commented Feb 10, 2025 via email • edited Loading

github-actions bot left a comment

Choose a reason for hiding this comment

subhramit left a comment • edited Loading

Choose a reason for hiding this comment

subhramit Feb 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

subhramit Feb 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

subhramit Feb 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

subhramit Feb 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

subhramit commented Feb 10, 2025

arraymahdi commented Feb 10, 2025

subhramit commented Feb 10, 2025

koppor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

koppor commented Feb 10, 2025

Linking a pull request to an issue using a keyword

Example:

calixtus commented Feb 10, 2025

subhramit commented Feb 11, 2025

github-actions bot left a comment

Choose a reason for hiding this comment

koppor commented Feb 16, 2025

koppor left a comment

Choose a reason for hiding this comment

subhramit commented Feb 16, 2025 • edited Loading

arraymahdi commented Feb 10, 2025 •

edited

Loading

arraymahdi commented Feb 10, 2025 via email •

edited

Loading

subhramit left a comment •

edited

Loading

subhramit Feb 10, 2025 •

edited

Loading

subhramit Feb 10, 2025 •

edited

Loading

subhramit Feb 10, 2025 •

edited

Loading

subhramit Feb 10, 2025 •

edited

Loading

subhramit commented Feb 16, 2025 •

edited

Loading