GROBID

GROBID documentation

Visit the GROBID documentation for more detailed information.

Purpose

GROBID (or Grobid, but not GroBid nor GroBiD) means GeneRation Of BIbliographic Data.

GROBID is a machine learning library for extracting, parsing and re-structuring raw documents such as PDF into structured XML/TEI encoded documents with a particular focus on technical and scientific publications. First developments started in 2008 as a hobby. In 2011 the tool has been made available in open source. Work on GROBID has been steady as a side project since the beginning and is expected to continue until at least 2020 :)

The following functionalities are available:

Header extraction and parsing from article in PDF format. The extraction here covers the usual bibliographical information (e.g. title, abstract, authors, affiliations, keywords, etc.).
References extraction and parsing from articles in PDF format. All the usual publication metadata are covered. The different citation contexts in an article are recognized and linked to the full bibliographical references.
Parsing of references in isolation.
Extraction of patent and non-patent references in patent publications.
Parsing of names (e.g. person title, fornames, middlename, etc.), in particular author names in header, and author names in references (two distinct models).
Parsing of affiliation and address blocks.
Parsing of dates (ISO normalized day, month, year).
Full text extraction from PDF articles, including a model for the the overall document segmentation and a model for the structuring of the text body (paragraph, section titles, reference callout, figure, table, etc.).
In a complete PDF processing, GROBID manages 55 final labels used to build relatively fine-grained structures, from traditional publication metadata (title, author first/last/middlenames, affiliation types, detailed address, journal, volume, issue, pages, etc.) to full text structures (section title, paragraph, reference markers, head/foot notes, figure headers, etc.).
Consolidation/resolution of the extracted bibliographical references using the biblio-glutton service or the CrossRef REST API.

GROBID includes a comprehensive web service API, batch processing, a JAVA API, a Docker image, a relatively generic evaluation framework (precision, recall, etc.) and the semi-automatic generation of training data.

GROBID can be considered as production ready. Deployments in production includes ResearchGate, HAL Research Archive, the European Patent Office, INIST-CNRS, Mendeley, CERN (Invenio), and many more.

GROBID should run properly "out of the box" on Linux (64 bits), MacOS, and Windows (32 and 64 bits).

For more information on how the tool works, on its key features and performance, visit the GROBID documentation.

Demo

For testing purposes, a public GROBID demo server is available at the following address: http://grobid.science-miner.com

The Web services are documented here.

Warning: Some quota and query limitation apply to the demo server! Please be courteous and do not overload the demo server.

Clients

For helping to exploit GROBID service at scale, we provide clients written in Python, Java, node.js using the web services for parallel batch processing:

All these clients will take advantage of the multi-threading for scaling large set of PDF processing. As a consequence, they will be much more efficient than the batch command lines (which use only one thread) and should be prefered.

We have been able recently to run the complete fulltext processing at around 10.6 PDF per second (around 915,000 PDF per day, around 20M pages per day) with the node.js client listed above during one week on a 16 CPU machine (16 threads, 32GB RAM, no SDD, articles from mainstream publishers), see here.

In addition, a Java example project is available to illustrate how to use GROBID as a Java library: https://github.com/kermitt2/grobid-example. The example project is using GROBID Java API for extracting header metadata and citations from a PDF and output the results in BibTeX format.

GROBID Modules

A series of additional modules have been developed for performing structure aware text mining directly on scholar PDF, reusing GROBID's PDF processing and sequence labelling weaponery:

grobid-ner: named entity recognition
grobid-astro: recognition of astronomical entities in scientific papers
grobid-quantities: recognition and normalization of physical quantities/measurements
software-mention: recognition of software mentions and attributes in scientific literature
grobid-bio: a bio-entity tagger using BioNLP/NLPBA 2004 dataset
grobid-dictionaries: structuring dictionaries in raw PDF format
grobid-superconductors: recognition of superconductor material and properties in scientific literature

Latest version

The latest stable release of GROBID is version 0.5.5. This version brings:

Using pdfalto instead of pdf2xml for the first PDF parsing stage, with many improvements in robustness, ICU support, unknown glyph/font normalization
Improvement and full review of the integration of consolidation services, supporting biblio-glutton (additional identifiers and Open Access links) and Crossref REST API (add specific user agent, email and token for Crossref Metadata Plus)
Fix bounding box issues for some PDF #330
Updated lexicon #396

New in previous release 0.5.4:

Transparent usage of DeLFT deep learning models (BidLSTM-CRF) instead of Wapiti CRF models, native integration via JEP
Support of biblio-glutton as DOI/metadata matching service, alternative to crossref REST API
Improvement of citation context identification and matching (+9% recall with similar precision, for PMC sample 1943 articles, from 43.35 correct citation contexts per article to 49.98 correct citation contexts per article)
Citation callout now in abstract, figure and table captions
Structured abstract (including update of TEI schema)
Bug fixes and some more parameters: by default using all available threads when training (thanks @de-code) and possibility to load models at the start of the service