dataset

Dataset

This submodule includes helper functions for downloading datasets and formatting them appropriately as well as utilities for splitting data for training / testing.

Data Loading

There are dataloaders for several datasets. For example, the snli module will allow you to load a dataframe in pandas from the SNLI dataset, with the option to set the number of rows to load in order to test algorithms and evaluate performance benchmarks. Most datasets may be split into train, dev, and test, for example:

from utils_nlp.dataset.snli import load_pandas_df

df = load_pandas_df(DATA_FOLDER, file_split ="train", nrows = 1000)

Dataset List

Dataset	Dataloader script
Microsoft Research Paraphrase Corpus	msrpc.py
The Multi-Genre NLI (MultiNLI) Corpus	multinli.py
The Stanford Natural Language Inference (SNLI) Corpus	snli.py
Wikigold NER	wikigold.py
The Cross-Lingual NLI (XNLI) Corpus	xnli.py
The STSbenchmark dataset	stsbenchmark.py
The Stanford Question Answering Dataset (SQuAD)	squad.py
CNN/Daily Mail(CNN/DM) Dataset	cnndm.py
Preprocessed CNN/Daily Mail(CNN/DM) Dataset for Extractive Summarization	cnndm.py

Dataset References

Please see Dataset References for notice and information regarding datasets used.