Pyserini provides a simple Python interface to the Anserini IR toolkit via pyjnius.
A low-effort way to try out Pyserini is to look at our online notebooks, which will allow you to get started with just a few clicks. For convenience, we've pre-built a few common indexes, available to download here.
Pyserini versions adopt the convention of X.Y.Z.W, where X.Y.Z tracks the version of Anserini, and W is used to distinguish different releases on the Python end. The current stable release of Pyserini is v0.9.1.0 on PyPI. The current experimental release of Pyserini on TestPyPI is behind the current stable release (i.e., do not use). In general, documentation is kept up to date with the latest code in the repo.
Install via PyPI
pip install pyserini==0.9.1.0
Here's a sample pre-built index on TREC Disks 4 & 5 to play with (used in the TREC 2004 Robust Track):
wget https://git.uwaterloo.ca/jimmylin/anserini-indexes/raw/master/index-robust04-20191213.tar.gz
tar xvfz index-robust04-20191213.tar.gzUse the SimpleSearcher for searching:
from pyserini.search import pysearch
searcher = pysearch.SimpleSearcher('index-robust04-20191213/')
hits = searcher.search('hubble space telescope')
# Print the first 10 hits:
for i in range(0, 10):
print(f'{i+1:2} {hits[i].docid:15} {hits[i].score:.5f}')
# Grab the raw text:
hits[0].raw
# Grab the raw Lucene Document:
hits[0].lucene_documentConfigure BM25 parameters and use RM3 query expansion:
searcher.set_bm25_similarity(0.9, 0.4)
searcher.set_rm3_reranker(10, 10, 0.5)
hits2 = searcher.search('hubble space telescope')
# Print the first 10 hits:
for i in range(0, 10):
print(f'{i+1:2} {hits[i].docid:15} {hits[i].score:.5f}')Pyserini exposes Lucene Analyzers in Python with the Analyzer class.
Below is a demonstration of these functionalities:
from pyserini.analysis import pyanalysis
# Default analyzer for English uses the Porter stemmer:
analyzer = pyanalysis.Analyzer(pyanalysis.get_lucene_analyzer())
tokens = analyzer.analyze('City buses are running on time.')
print(tokens)
# Result is ['citi', 'buse', 'run', 'time']
# We can explictly specify the Porter stemmer as follows:
analyzer = pyanalysis.Analyzer(pyanalysis.get_lucene_analyzer(stemmer='porter'))
tokens = analyzer.analyze('City buses are running on time.')
print(tokens)
# Result is same as above.
# We can explictly specify the Krovetz stemmer as follows:
analyzer = pyanalysis.Analyzer(pyanalysis.get_lucene_analyzer(stemmer='krovetz'))
tokens = analyzer.analyze('City buses are running on time.')
print(tokens)
# Result is ['city', 'bus', 'running', 'time']
# Create an analyzer that doesn't stem, simply tokenizes:
analyzer = pyanalysis.Analyzer(pyanalysis.get_lucene_analyzer(stemming=False))
tokens = analyzer.analyze('City buses are running on time.')
print(tokens)
# Result is ['city', 'buses', 'running', 'time']The IndexReaderUtils class provides functionalities for accessing and manipulating an inverted index.
IMPORTANT NOTE: Be aware whether a method takes or returns analyzed or unanalyzed terms.
"Analysis" refers to processing by a Lucene Analyzer, which typically includes tokenization, stemming, stopword removal, etc.
For example, if a method expects the unanalyzed form and is called with an analyzed form, it'll re-analyze the argument; it is sometimes the case that analysis of an already analyzed term is also a valid term, which means that the method will return incorrect results without triggering any type of warning or error.
Initialize the class as follows:
from pyserini.index import pyutils
from pyserini.analysis import pyanalysis
index_utils = pyutils.IndexReaderUtils('index-robust04-20191213/')Use terms() to grab an iterator over all terms in the collection, i.e., the dictionary.
Note that these terms are analyzed.
Here, we only print out the first 10:
import itertools
for term in itertools.islice(index_utils.terms(), 10):
print(f'{term.term} (df={term.df}, cf={term.cf})')How to fetch term statistics for a particular (unanalyzed) query term, "cities" in this case:
term = 'cities'
# Look up its document frequency (df) and collection frequency (cf).
# Note, we use the unanalyzed form:
df, cf = index_utils.get_term_counts(term)
print(f'term "{term}": df={df}, cf={cf}')What if we want to fetch term statistics for an analyzed term?
This can be accomplished by setting Analyzer to None:
term = 'cities'
# Analyze the term.
analyzed = index_utils.analyze(term)
print(f'The analyzed form of "{term}" is "{analyzed[0]}"')
# Skip term analysis:
df, cf = index_utils.get_term_counts(term, analyzer=None)
print(f'term "{term}": df={df}, cf={cf}')Here's how to fetch and traverse postings:
# Fetch and traverse postings for an unanalyzed term:
postings_list = index_utils.get_postings_list(term)
for posting in postings_list:
print(f'docid={posting.docid}, tf={posting.tf}, pos={posting.positions}')
# Fetch and traverse postings for an analyzed term:
postings_list = index_utils.get_postings_list(analyzed[0], analyzer=None)
for posting in postings_list:
print(f'docid={posting.docid}, tf={posting.tf}, pos={posting.positions}')Here's how to fetch the document vector for a document:
doc_vector = index_utils.get_document_vector('FBIS4-67701')
print(doc_vector)The result is a dictionary where the keys are the analyzed terms and the values are the term frequencies. To compute the tf-idf representation of a document, do something like this:
tf = index_utils.get_document_vector('FBIS4-67701')
df = {term: (index_utils.get_term_counts(term, analyzer=None))[0] for term in tf.keys()}The two dictionaries will hold tf and df statistics; from those it is easy to assemble into the tf-idf representation. However, often the BM25 score is better than tf-idf. To compute the BM25 score for a particular term in a document:
bm25_score = index_utils.compute_bm25_term_weight('FBIS4-67701', 'citi')
# Note that this takes the analyzed form because the common case is to take the term from
# get_document_vector() above.
print(bm25_score)And so, to compute the BM25 vector of a document:
tf = index_utils.get_document_vector('FBIS4-67701')
bm25_vector = {term: index_utils.compute_bm25_term_weight('FBIS4-67701', term) for term in tf.keys()}The collection classes provide interfaces for iterating over a collection and processing documents.
Here's a demonstration on the CACM collection:
wget -O cacm.tar.gz https://github.com/castorini/anserini/blob/master/src/main/resources/cacm/cacm.tar.gz?raw=true
mkdir collection
tar xvfz cacm.tar.gz -C collectionLet's iterate through all documents in the collection:
from pyserini.collection import pycollection
from pyserini.index import pygenerator
collection = pycollection.Collection('HtmlCollection', 'collection/')
generator = pygenerator.Generator('DefaultLuceneDocumentGenerator')
for (i, fs) in enumerate(collection):
for (j, doc) in enumerate(fs):
parsed = generator.create_document(doc)
docid = parsed.get('id') # FIELD_ID
raw = parsed.get('raw') # FIELD_RAW
contents = parsed.get('contents') # FIELD_BODY
print('{} {} -> {} {}...'.format(i, j, docid, contents.strip().replace('\n', ' ')[:50]))Alternatively, for parts of Anserini that have not yet been integrated into the Pyserini interface, you can interact with Anserini's Java classes directly via pyjnius. First, call Pyserini's setup helper for setting up classpath for the JVM:
from pyserini.setup import configure_classpath
configure_classpath('pyserini/resources/jars')Now autoclass can be used to provide direct access to Java classes:
from jnius import autoclass
JString = autoclass('java.lang.String')
JIndexReaderUtils = autoclass('io.anserini.index.IndexReaderUtils')
reader = JIndexReaderUtils.getReader(JString('index-robust04-20191213/'))
# Fetch raw document contents by id:
rawdoc = JIndexReaderUtils.documentRaw(reader, JString('FT934-5418'))Anserini is designed to work with JDK 11. There was a JRE path change above JDK 9 that breaks pyjnius 1.2.0, as documented in this issue, also reported in Anserini here and here. This issue was fixed with pyjnius 1.2.1 (released December 2019). The previous error was documented in this notebook and this notebook documents the fix.
- v0.9.1.0: May 6, 2020 [Release Notes]
- v0.9.0.0: April 18, 2020 [Release Notes]
- v0.8.1.0: March 22, 2020 [Release Notes]
- v0.8.0.0: March 12, 2020 [Release Notes]
- v0.7.2.0: January 25, 2020 [Release Notes]
- v0.7.1.0: January 9, 2020 [Release Notes]
- v0.7.0.0: December 13, 2019 [Release Notes]
- v0.6.0.0: November 2, 2019