Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test without numpy #104

Closed
wants to merge 16 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .git-blame-ignore-revs
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
40e5de5211ff6967c6e14f47fd62b3add5997d5f
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ro_json/_static_version.py export-subst
41 changes: 41 additions & 0 deletions .github/workflows/pypi.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/
name: Publish Python 🐍 distributions 📦 to PyPI and TestPyPI

on:
push:
tags:
- '*'

permissions:
id-token: write
contents: read

jobs:
build-n-publish:
name: Build and Publish Python 🐍 Distributions 📦 to PyPI and TestPyPI
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v3
with:
# https://github.com/actions/checkout#fetch-all-history-for-all-tags-and-branches
fetch-depth: 0

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.10"

- name: Install build
run: python -m pip install --upgrade pip build

- name: Build Distributions
run: python -m build --sdist --wheel --outdir dist/

- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
user: __token__
password: ${{ secrets.PYPI_API_TOKEN }}
print-hash: true
verify-metadata: true
92 changes: 37 additions & 55 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@

name: 'pyjson-tricks'
name: Testing

on:
push:
Expand All @@ -9,71 +8,54 @@ on:

jobs:
build:
name: tests
name: Tests on Python ${{ matrix.python-version }}
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ['3.10', '3.11', '3.12']
include:
- python-version: 3.10
with-numpy: true
- python-version: 3.11
with-numpy: false
- python-version: 3.12
with-numpy: true
max-parallel: 8
fail-fast: false
matrix:
libraries: [
'vanilla',
'tz',
'path',
'numpy',
'pandas',
'all'
]
python-version: [
'3.7',
'3.8',
'3.9',
'3.10',
'3.11'
]

steps:
- uses: actions/checkout@v1
- uses: actions/checkout@v3
- uses: proudust/gh-describe@v2
# id needed to generate the outputs
id: ghd
- name: Check outputs
run: |
echo "describe : ${{ steps.ghd.outputs.describe }}"
echo "tag : ${{ steps.ghd.outputs.tag }}"
echo "distance : ${{ steps.ghd.outputs.distance }}"
echo "sha : ${{ steps.ghd.outputs.sha }}"
echo "short-sha : ${{ steps.ghd.outputs.short-sha }}"
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}

- name: Install dependencies
run: |
export RO_JSON_GIT_DESCRIBE=${{ steps.ghd.outputs.describe }}
python -m pip install --upgrade pip
pip install pytest
if [ "${{ matrix.python-version }}" == "2.7" ] ; then
pip install enum34
fi
export LIBS="${{ matrix.libraries }}"
if [ "$LIBS" == "tz" ] || [ "$LIBS" == "all" ] ; then
pip install pytz
fi
if [ "$LIBS" == "path" ] || [ "$LIBS" == "all" ] ; then
pip install pathlib
fi
if [ "$LIBS" == "numpy" ] || [ "$LIBS" == "all" ] ; then
pip install numpy
fi
if [ "$LIBS" == "pandas" ] || [ "$LIBS" == "all" ] ; then
pip install pandas
pip install setuptools pytest pathlib
if [ "${{ matrix.with-numpy }}" = "true" ]; then
pytz numpy pandas
fi

- name: Install package
run: |
export RO_JSON_GIT_DESCRIBE=${{ steps.ghd.outputs.describe }}
python -m pip install . -vvv

- name: Run tests
run: |
export RO_JSON_GIT_DESCRIBE=${{ steps.ghd.outputs.describe }}
python --version
PYTEST_ARGS='-v --strict tests/test_bare.py tests/test_class.py tests/test_meta.py tests/test_enum.py'
export LIBS="${{ matrix.libraries }}"
if [ "$LIBS" == "vanilla" ] ; then
py.test $PYTEST_ARGS
elif [ "$LIBS" == "tz" ] ; then
py.test $PYTEST_ARGS tests/test_tz.py
elif [ "$LIBS" == "path" ] ; then
py.test $PYTEST_ARGS tests/test_pathlib.py
elif [ "$LIBS" == "numpy" ] ; then
py.test $PYTEST_ARGS tests/test_np.py
elif [ "$LIBS" == "pandas" ] ; then
py.test $PYTEST_ARGS tests/test_pandas.py
elif [ "$LIBS" == "all" ] ; then
py.test -v --strict
else
echo "UNKNOWN LIBRARY '$LIBS'"
exit 1
fi
pytest -v --strict
64 changes: 34 additions & 30 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
# JSON tricks (python)
> [!NOTE]

The [pyjson-tricks] package brings several pieces of
>The primary reason for this fork is to enable full round-trip serialization and deserialization of NumPy scalars and 0-dimensional arrays to JSON and back. This feature is essential for applications that require precise data preservation when working with NumPy data types.

Despite contributing this enhancement to the original project (see [Pull Request #99](https://github.com/mverleg/pyjson_tricks/pull/99)), there was a difference in opinion with the maintainer regarding its inclusion. As a result, this fork aims to continue development with this functionality integrated.

# ro_json

The [ro-json] package brings several pieces of
functionality to python handling of json files:

1. **Store and load numpy arrays** in human-readable format.
Expand All @@ -13,9 +19,9 @@ functionality to python handling of json files:

As well as compression and disallowing duplicate keys.

* Code: <https://github.com/mverleg/pyjson_tricks>
* Documentation: <http://json-tricks.readthedocs.org/en/latest/>
* PIP: <https://pypi.python.org/pypi/json_tricks>
* Code: <https://github.com/ramonaoptics/ro-json>
<!-- * Documentation: <http://ro-json.readthedocs.org/en/latest/> -->
* PIP: <https://pypi.python.org/pypi/ro-json>

Several keys of the format `__keyname__` have special meanings, and more
might be added in future releases.
Expand All @@ -31,7 +37,7 @@ Thanks for all the Github stars⭐!
You can install using

``` bash
pip install json-tricks
pip install ro_json
```

Decoding of some data types needs the corresponding package to be
Expand All @@ -42,7 +48,7 @@ You can import the usual json functions dump(s) and load(s), as well as
a separate comment removal function, as follows:

``` bash
from json_tricks import dump, dumps, load, loads, strip_comments
from ro_json import dump, dumps, load, loads, strip_comments
```

The exact signatures of these and other functions are in the [documentation](http://json-tricks.readthedocs.org/en/latest/#main-components).
Expand Down Expand Up @@ -75,7 +81,7 @@ this yields:
```

which will be converted back to a numpy array when using
`json_tricks.loads`. Note that the memory order (`Corder`) is only
`ro_json.loads`. Note that the memory order (`Corder`) is only
stored in v3.1 and later and for arrays with at least 2 dimensions.

As you see, this uses the magic key `__ndarray__`. Don't use
Expand All @@ -87,9 +93,9 @@ closest python primitive type. A special representation was not
feasible, because Python's json implementation serializes some numpy
types as primitives, without consulting custom encoders. If you want to
preserve the exact numpy type, use
[encode_scalars_inplace](https://json-tricks.readthedocs.io/en/latest/#json_tricks.np_utils.encode_scalars_inplace).
[encode_scalars_inplace](https://json-tricks.readthedocs.io/en/latest/#ro_json.np_utils.encode_scalars_inplace).

There is also a compressed format (thanks `claydugo` for fix). From
There is also a compressed format (thanks `claydugo` for fix). From
the next major release, this will be default when using compression.
For now, you can use it as:

Expand Down Expand Up @@ -122,14 +128,14 @@ dumps(data, compression=False, properties={'ndarray_compact': 8})

## Class instances

`json_tricks` can serialize class instances.
`ro_json` can serialize class instances.

If the class behaves normally (not generated dynamic, no `__new__` or
`__metaclass__` magic, etc) *and* all it's attributes are serializable,
then this should work by default.

``` python
# json_tricks/test_class.py
# ro_json/test_class.py
class MyTestCls:
def __init__(self, **kwargs):
for k, v in kwargs.items():
Expand All @@ -146,7 +152,7 @@ You'll get your instance back. Here the json looks like this:
``` javascript
{
"__instance_type__": [
"json_tricks.test_class",
"ro_json.test_class",
"MyTestCls"
],
"attributes": {
Expand Down Expand Up @@ -211,7 +217,7 @@ Date, time, datetime and timedelta objects are stored as dictionaries of
"day", "hour", "millisecond" etc keys, for each nonzero property.

Timezone name is also stored in case it is set, as is DST (thanks `eumir`).
You'll need to have `pytz` installed to use timezone-aware date/times,
You'll need to have `pytz` installed to use timezone-aware date/times,
it's not needed for naive date/times.

``` javascript
Expand Down Expand Up @@ -253,7 +259,7 @@ ordered = OrderedDict((
Converting to json and back will preserve the order:

``` python
from json_tricks import dumps, loads
from ro_json import dumps, loads
json = dumps(ordered)
ordered = loads(json, preserve_order=True)
```
Expand Down Expand Up @@ -303,12 +309,12 @@ Since comments aren't stored in the Python representation of the data,
loading and then saving a json file will remove the comments (it also
likely changes the indentation).

The implementation of comments is a bit crude, which means that there are
The implementation of comments is a bit crude, which means that there are
some exceptional cases that aren't handled correctly ([#57](https://github.com/mverleg/pyjson_tricks/issues/57)).

It is also not very fast. For that reason, if `ignore_comments` wasn't
explicitly set to True, then json-tricks first tries to parge without
ignoring comments. If that fails, then it will automatically re-try
It is also not very fast. For that reason, if `ignore_comments` wasn't
explicitly set to True, then ro_json first tries to parse without
ignoring comments. If that fails, then it will automatically re-try
with comment handling. This makes the no-comment case faster at the cost
of the comment case, so if you are expecting comments make sure to set
`ignore_comments` to True.
Expand All @@ -328,10 +334,10 @@ of the comment case, so if you are expecting comments make sure to set
* Save and load `Enum` (thanks to `Jenselme`), either built-in in
python3.4+, or with the [enum34](https://pypi.org/project/enum34/)
package in earlier versions. `IntEnum` needs
[encode_intenums_inplace](https://json-tricks.readthedocs.io/en/latest/#json_tricks.utils.encode_intenums_inplace).
* `json_tricks` allows for gzip compression using the
[encode_intenums_inplace](https://json-tricks.readthedocs.io/en/latest/#ro_json.utils.encode_intenums_inplace).
* `ro_json` allows for gzip compression using the
`compression=True` argument (off by default).
* `json_tricks` can check for duplicate keys in maps by setting
* `ro_json` can check for duplicate keys in maps by setting
`allow_duplicates` to False. These are [kind of
allowed](http://stackoverflow.com/questions/21832701/does-json-syntax-allow-duplicate-keys-in-an-object),
but are handled inconsistently between json implementations. In
Expand All @@ -340,16 +346,16 @@ of the comment case, so if you are expecting comments make sure to set
* Save and load `pathlib.Path` objects (e.g., the current path,
`Path('.')`, serializes as `{"__pathlib__": "."}`)
(thanks to `bburan`).
* Save and load bytes (python 3+ only), which will be encoded as utf8 if
that is valid, or as base64 otherwise. Base64 is always used if
* Save and load bytes (python 3+ only), which will be encoded as utf8 if
that is valid, or as base64 otherwise. Base64 is always used if
primitives are requested. Serialized as
`[{"__bytes_b64__": "aGVsbG8="}]` vs `[{"__bytes_utf8__": "hello"}]`.
* Save and load slices (thanks to `claydugo`).

# Preserve type vs use primitive

By default, types are encoded such that they can be restored to their
original type when loaded with `json-tricks`. Example encodings in this
original type when loaded with `ro-json`. Example encodings in this
documentation refer to that format.

You can also choose to store things as their closest primitive type
Expand Down Expand Up @@ -455,9 +461,9 @@ print(dumps(data, primitives=True))
]
```

Note that valid json is produced either way: ``json-tricks`` stores meta data as normal json, but other packages probably won't interpret it.
Note that valid json is produced either way: ``ro_json`` stores meta data as normal json, but other packages probably won't interpret it.

Note that valid json is produced either way: `json-tricks` stores meta
Note that valid json is produced either way: `ro_json` stores meta
data as normal json, but other packages probably won't interpret it.

# Usage & contributions
Expand All @@ -477,6 +483,4 @@ Contributors not yet mentioned: `janLo` (performance boost).
Tests are run automatically for commits to the repository for all
supported versions. This is the status:

![image](https://github.com/mverleg/pyjson_tricks/workflows/pyjson-tricks/badge.svg?branch=master)

To run the tests manually for your version, see [this guide](tests/run_locally.md).
To run the tests manually for your version, see [this guide](tests/run_locally.md).
Loading
Loading