README.md

Named entitity recognition (NER) and Detecting Hyponym\Hypernym relationship on the dataset of Patents

The main goals of this project are:

• Train NER model with dataset of Patents in the specific domain
• Fine-tune with prodidy
• Implement automatic detection of hyponyms\hypernyms with Hearst patterns
• Validate detection results with several methods, inluding Wikidata

Structure

• project.ipynb - main notebook
• G06K.txt.gz - archive with patent texts
• configs/base_config.cfg - Base config to train NER SpaCy pipeline
• hearst_patterns/patterns.json - Hearst patterns configuration for SpaCy
• extracted_patterns - Extracted patterns (EL) from G06K texts

Setup

1. Install dependencies from requirements.txt
2. Unpack data:
   tar -xvf G06K.txt.gz
3. Open project.ipynb and run first cell to chek that all imports works propperly

Notebook structure

Here is a brief overview of the project.ipynb parts.

Data processing

In this section patent text read and prcessed to extract potential Named entities using curated list of terms manyterms.lower.txt

Training NER model

Next, we are training the model on the created dataset.
Additionaly, if you have access to the Prodiy, you can apply Active Learning to tune the model.

Hearst patterns for hyponym detection

Thise section is dedicated to extracting potential Entity linking (like hypernyms) using Hearst Patterns.

Automatic validation of the results

Afte extraction, we validate results automatically, using Wiki API, WordNet or SpaCy embeddings. Here is an example of validation table after processing: