[FR] - Standardize JSON representations of Cardano domain types

[klntsky]

Internal/External
External

Area
Other JSON format standardization

Describe the feature you'd like

I am pushing an initiative to standardize JSON representations of Cardano domain types for offchain tooling. See this CPS for the motivation.

Ideally the standard should be based on existing and widely used implementations, which cardano-cli is one of.

There is, however, an ecosystem-wide problem: standard JSON tooling does not support long integer representations, which makes certain values unrepresentable when encoding of long integers to numbers is used. For example, see the comments for JSON machinery in CSL, and this issue.

Would it be possible to avoid encoding long integers as JSON numbers and switch to strings? That would allow tools like CSL (and literally any other offchain library) to be fully compatible.

Additionally, I have a few other questions:

• Is there a way to get JSON schemas of the current encoding without diving into the code?

• Does it make sense to standardize cardano-cli format from your point of view?

Describe alternatives you've considered

The alternative would be to define a standard that is not compatible with cardano-cli.

[gitmachtl]

The "long number" issue, at least for the cli and the widely used tool jq can be avoided by a simple convert like:

Normal output:

./cardano-cli query utxo --address $(cat funds.addr) --out-file /dev/stdout | jq

{
  "040e17a156ebc4417abf03b89ded5cef7b44f67a090ddf65979eb442cb72fdf7#0": {
    "address": "addr_test1vpfwv0ezc5g8a4mkku8hhy3y3vp92t7s3ul8g778g5yegsgalc6gc",
    "datum": null,
    "datumhash": null,
    "inlineDatum": null,
    "referenceScript": null,
    "value": {
      "lovelace": 9467947941
    }
  }
}

Numbers stringified:

./cardano-cli query utxo --address $(cat funds.addr) --out-file /dev/stdout | sed -e 's/": \([0-9]\+\)/": "\1"/g' | jq

{
  "040e17a156ebc4417abf03b89ded5cef7b44f67a090ddf65979eb442cb72fdf7#0": {
    "address": "addr_test1vpfwv0ezc5g8a4mkku8hhy3y3vp92t7s3ul8g778g5yegsgalc6gc",
    "datum": null,
    "datumhash": null,
    "inlineDatum": null,
    "referenceScript": null,
    "value": {
      "lovelace": "9467947941"
    }
  }
}

Its not a JSON problem, its a problem of some tools reading/processing JSON content. cardano-cli is a CLI tool, so that would be a quick solution for CLI apps.

[klntsky]

@gitmachtl It's not that I'm looking for a code solution, the problem is that other tools are unable to parse cardano-cli output or maintain compatibility with it without reinventing the whole JSON encoding stack (like we did)

[smelc]

cc @CarlosLopezDeLara

[gitmachtl]

@gitmachtl It's not that I'm looking for a code solution, the problem is that other tools are unable to parse cardano-cli output or maintain compatibility with it without reinventing the whole JSON encoding stack (like we did)

yea but the output of cardano-cli is fully JSON compliant as it is. so correct me if i am wrong, but the problem lies in the 3rd party tools parsing JSON correctly. do you have examples on 3rd party tools that are taking the output of cardano-cli and having issues with it? because as i posted above, for JQ the solution is simple. also the readout like "jq -r .keyName" returns the correct output also with lovelaces or native-asset amounts stringified. for JS, people can use the json-bigint lib like:

var JSONbig = require('json-bigint');
let result = JSONbig.parse(<your-json>);

if we change the output of cardano-cli to stringified numbers, i am ok with that. i proposed that i think two years ago. the conclusion last time was, that numbers are numbers and should not be stringified. i am not against a stringified number output for cardano-cli.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 120 days.

[smelc]

We probably don't want to change the CLI's handling of long integers, because we rely on aeson for JSON serialization, and aeson is supposed to honor the JSON specification.

However, there is particularly interesting piece here:

Is there a way to get JSON schemas of the current encoding without diving into the code?

This is probably something we can do, if our bandwidth allows, because I think it's indeed interesting for users. Which is why I'm removing the Stale label.

[klntsky]

aeson is supposed to honor the JSON specification.

The JSON specification does not define the ranges for allowed numeric types, leaving them platform-dependent. Do we really want to deal with platform-dependent behaviors in our tech stack? Especially given that the price of avoiding them is so low.

The goal of providing the encodings in the first place is interoperability, but as for now it hasn't been achieved, because the spec in fact contains undefined behavior.

but the problem lies in the 3rd party tools parsing JSON correctly. do you have examples on 3rd party tools that are taking the output of cardano-cli and having issues with it

@gitmachtl yes,

• native JSON.parse (nodejs, the browser) - truncates the numbers silently
• Rust's serde (context)
• probably almost every JSON library in almost every programming language, as long integer arithmetic is not very common to be in the standard library, and is a niche use case - IEEE floats are the default

[gitmachtl]

i am not against a stringified output of numbers for json outputs via cardano-cli like the utxo query. but for nodejs there is the json-bigint lib that i wrote about here #570 (comment)
browser and cardano-cli is a bit of a strange combination. for bash the | sed -e 's/": \([0-9]\+\)/": "\1"/g' method as pipe works pretty fine.

again, i am happy if the decision is made to make the numbers also stringified.

[klntsky]

@gitmachtl

browser and cardano-cli is a bit of a strange combination

The goal is to make everything mutually compatible: cardano-cli <-> nodejs <-> dApp backend <-> browser <-> wallets <-> wallet backends. That would reduce the amount of code needed to simply pass data between these.

but for nodejs there is the json-bigint lib

It implements its own string parser which is too costly for the runtime (in comparison with native JSON.parse). We use an abstraction on top of its fork, and the amount of time it spends in the parse loop is noticeable in the profiler.

again, i am happy if the decision is made to make the numbers also stringified.

OK, I'll provide everything I can to increase the chance of it happening.

Here's a WIP json-schema (branch) that I'm working on - the goal is to define all structural parts of a Transaction. Then there will be a stage of CIP review - but please don't hesitate to look at this in its current state.

[smelc]

@klntsky> thanks for the additional explanations 👍

Two comments though:

Do we really want to deal with platform-dependent behaviors in our tech stack?

Indeed I'm all for being platform agnostic 🙂

Especially given that the price of avoiding them is so low

Not sure about this one 🙂 Staying retro-compatible is important for cardano-cli and this would be a pretty big change! cc @Jimbo4350 @CarlosLopezDeLara

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 120 days.

[github-actions]

This issue was closed because it has been stalled for 120 days with no activity. Remove stale label or comment or this will be closed in 60 days.