srsly: Modern high-performance serialization utilities for Python

This package bundles some of the best Python serialization libraries into one standalone package, with a high-level API that makes it easy to write code that's correct across platforms and Pythons. This allows us to provide all the serialization utilities we need in a single binary wheel.

Motivation

Serialization is hard, especially across Python versions and multiple platforms. After dealing with many subtle bugs over the years (encodings, locales, large files) our libraries like spaCy and Prodigy have steadily grown a number of utility functions to wrap the multiple serialization formats we need to support (especially json, msgpack and pickle). These wrapping functions ended up duplicated across our codebases, so we wanted to put them in one place.

At the same time, we noticed that having a lot of small dependencies was making maintenance harder, and making installation slower. To solve this, we've made srsly standalone, by including the component packages directly within it. This way we can provide all the serialization utilities we need in a single binary wheel.

srsly currently includes forks of the following packages:

• ujson
• msgpack
• msgpack-numpy
• cloudpickle

Installation

srsly can be installed from pip:

pip install srsly

Alternatively, you can also comile the library from source. You'll need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler (XCode command-line tools on macOS / OS X or Visual C++ build tools on Windows), pip, virtualenv and git installed.

pip install -r requirements.txt  # install development dependencies
python setup.py build_ext --inplace  # compile the library

API

JSON

📦 The underlying module is exposed via srsly.ujson. However, we normally interact with it via the utility functions only.

function srsly.json_dumps

Serialize an object to a JSON string. Takes care of Python 2/3 compatibility and falls back to json if sort_keys=True is used (until it's fixed in ujson).

data = {"foo": "bar", "baz": 123}
json_string = srsly.json_dumps(data)

Argument	Type	Description
data	-	The JSON-serializable data to output.
indent	int	Number of spaces used to indent JSON. Defaults to 0.
sort_keys	bool	Sort dictionary keys. Defaults to False.
RETURNS	unicode	The serialized string.

function srsly.json_loads

Deserialize unicode or bytes to a Python object.

data = '{"foo": "bar", "baz": 123}'
obj = srsly.json_loads(data)

Argument	Type	Description
data	unicode / bytes	The data to deserialize.
RETURNS	-	The deserialized Python object.

function srsly.write_json

Create a JSON file and dump contents or write to standard output.

data = {"foo": "bar", "baz": 123}
srsly.write_json("/path/to/file.json", data)

Argument	Type	Description
location	unicode / Path	The file path or "-" to write to stdout.
data	-	The JSON-serializable data to output.
indent	int	Number of spaces used to indent JSON. Defaults to 2.

function srsly.read_json

Load JSON from a file or standard input.

data = srsly.read_json("/path/to/file.json")

Argument	Type	Description
location	unicode / Path	The file path or "-" to read from stdin.
RETURNS	dict / list	The loaded JSON content.

function srsly.write_jsonl

Create a JSONL file (newline-delimited JSON) and dump contents line by line, or write to standard output.

data = [{"foo": "bar"}, {"baz": 123}]
srsly.write_jsonl("/path/to/file.jsonl", data)

Argument	Type	Description
location	unicode / Path	The file path or "-" to write to stdout.
lines	iterable	The JSON-serializable lines.

function srsly.read_jsonl

Read a JSONL file (newline-delimited JSON) or from JSONL data from standard input and yield contents line by line. Blank lines will always be skipped.

data = srsly.read_jsonl("/path/to/file.jsonl")

Argument	Type	Description
location	unicode / Path	The file path or "-" to read from stdin.
skip	bool	Skip broken lines and don't raise ValueError. Defaults to False.
YIELDS	-	The loaded JSON contents of each line.

function srsly.is_json_serializable

Check if a Python object is JSON-serializable.

assert srsly.is_json_serializable({"hello": "world"}) is True
assert srsly.is_json_serializable(lambda x: x) is False

Argument	Type	Description
obj	-	The object to check.
RETURNS	bool	Whether the object is JSON-serializable.

msgpack

📦 The underlying module is exposed via srsly.msgpack. However, we normally interact with it via the utility functions only.

function srsly.msgpack_dumps

Serialize an object to a msgpack byte string.

data = {"foo": "bar", "baz": 123}
msg = srsly.msgpack_dumps(data)

Argument	Type	Description
data	-	The data to serialize.
RETURNS	bytes	The serialized bytes.

function srsly.msgpack_loads

Deserialize msgpack bytes to a Python object.

msg = b"\x82\xa3foo\xa3bar\xa3baz{"
data = srsly.msgpack_loads(msg)

Argument	Type	Description
data	bytes	The data to deserialize.
use_list	bool	Don't use tuples instead of lists. Can make deserialization slower. Defaults to True.
RETURNS	-	The deserialized Python object.

function srsly.write_msgpack

Create a msgpack file and dump contents.

data = {"foo": "bar", "baz": 123}
srsly.write_msgpack("/path/to/file.msg", data)

Argument	Type	Description
location	unicode / Path	The file path.
data	-	The data to serialize.

function srsly.read_msgpack

Load a msgpack file.

data = srsly.read_msgpack("/path/to/file.msg")

Argument	Type	Description
location	unicode / Path	The file path.
use_list	bool	Don't use tuples instead of lists. Can make deserialization slower. Defaults to True.
RETURNS	-	The loaded and deserialized content.

pickle

📦 The underlying module is exposed via srsly.cloudpickle. However, we normally interact with it via the utility functions only.

function srsly.pickle_dumps

Serialize a Python object with pickle.

data = {"foo": "bar", "baz": 123}
pickled_data = srsly.pickle_dumps(data)

Argument	Type	Description
data	-	The object to serialize.
protocol	int	Protocol to use. -1 for highest. Defaults to None.
RETURNS	bytes	The serialized object.

function srsly.pickle_loads

Deserialize bytes with pickle.

pickled_data = b"\x80\x04\x95\x19\x00\x00\x00\x00\x00\x00\x00}\x94(\x8c\x03foo\x94\x8c\x03bar\x94\x8c\x03baz\x94K{u."
data = srsly.pickle_loads(pickled_data)

Argument	Type	Description
data	bytes	The data to deserialize.
RETURNS	-	The deserialized Python object.