JSON-RPC server returns frames as binary unlike most of the ecosystem

[blouflashdb]

Most of the JSON-RPC ecosystem over WebSockets returns JSON over text frames, unlike the Nimiq JSON-RPC server. This creates friction for existing client libraries that expect the responses over text frames.

Original text

Dear Nimiq Development Team,

I am reaching out with a question regarding the implementation of your JSON-RPC 2.0 API, specifically in the context of WebSocket transport.

According to the JSON-RPC 2.0 specification, messages are expected to be JSON-encoded text. However, it seems that your WebSocket implementation sends responses as binary data (e.g., serialized JSON), which deviates from the standard's expectation of JSON text messages.

This has caused issues when using the API in browser environments, where binary messages are received as Blob objects, requiring additional steps to decode them into JSON. While this is manageable with workarounds, it introduces challenges for strict JSON-RPC 2.0 clients and libraries, which are designed to expect text-based JSON responses.

Could you clarify why the WebSocket transport sends binary data instead of JSON text, and if there are any plans to align the implementation with the JSON-RPC 2.0 specification in the future?

Thank you for your time and for the work you’re doing on the Nimiq ecosystem!

Best regards
Daniel Schmitz

[Eligioo]

Hi @blouflashdb looking at the JSON-RPC spec, I can't find anything that enforces whether data frames sent over a WebSocket connection should be in binary or text format. It only defines the format and structure of the JSON-RPC messages (e.g., request, response and notification) in terms of JSON. How these data frames a.k.a. the JSON messages are send over the connection between the server and the client is the responsibility of the underlying transport (WebSockets or HTTP).

Specifically the WebSocket specification states that the interpretation of these data frames can either be in Text or Binary. Therefore I don't feel we are not adhering to a certain specification at the moment. The Blob data type in JS has good support for converting this into UTF-8 within a browser. Therefore I wouldn't consider converting a Blob into an Object to be a workaround.

A WebSocket client that doesn't support Binary data frames would I consider an incomplete implementation. Maybe you could elaborate on it introduces challenges for strict JSON-RPC 2.0 clients and libraries, which are designed to expect text-based JSON responses.

[sisou]

Additionally, one can set the socket.binaryType = 'arraybuffer', which then produces ArrayBuffer instead of Blob, which can be directly passed into a TextDecoder.

[blouflashdb]

The issue arises because the official JSON-RPC client library, @open-rpc/client-js, does not handle WebSocket binary frames in the browser environment. This library is widely used and adheres strictly to the JSON-RPC 2.0 specification, expecting the payload to be JSON-encoded text.

See: https://github.com/open-rpc/client-js/blob/master/src/transports/TransportRequestManager.ts#L58

[blouflashdb]

The JSON-RPC 2.0 specification does not explicitly state that the transport stream (e.g., WebSocket or HTTP) must send data as JSON text or that it cannot use binary formats. However, the specification strongly implies that the payload of any JSON-RPC message must be valid JSON, which is inherently a text-based format. Here’s the reasoning:

---

Key Points in the JSON-RPC 2.0 Specification

1. JSON as the Data Format:
   The specification starts by stating:

   "JSON-RPC is a lightweight remote procedure call (RPC) protocol. It's designed to be simple and to use JSON (RFC 4627) as a data format."

   JSON is explicitly defined as a text-based format encoded in UTF-8. While binary formats like Blob or ArrayBuffer can represent JSON, they are not JSON themselves; they are encoded representations of JSON.

2. Examples of JSON-RPC Messages:
   The spec consistently uses examples of JSON-RPC messages in plain-text JSON format. For example:

   {
     "jsonrpc": "2.0",
     "method": "subtract",
     "params": [42, 23],
     "id": 1
   }

   Nowhere does the spec mention encoding these messages in binary or using any non-JSON representations.

3. Transport-Agnostic, But JSON-Specific:
   While JSON-RPC is transport-agnostic (e.g., it can work over HTTP, WebSocket, etc.), the messages themselves must conform to the JSON format. For WebSockets, the JSON-RPC over WebSocket guidelines (not part of the core JSON-RPC spec but commonly referenced) suggest that the payload must be valid JSON.

   For example:

   • If you send a binary-encoded representation of JSON (e.g., Blob), it is no longer directly JSON until decoded.
   • A strict JSON-RPC client that uses JSON.parse expects text, not binary.

4. Error Handling and Interoperability:
   JSON-RPC assumes that clients and servers can handle JSON messages without needing additional decoding layers. Binary frames add an additional step, which can lead to interoperability issues if the client doesn't know how to handle binary formats.

---

Conclusion from the Spec

While the JSON-RPC 2.0 spec does not explicitly forbid binary encoding, it strongly implies that:

• JSON-RPC messages must be valid JSON.
• JSON itself is inherently text-based, so binary encoding (e.g., Blob or Buffer) is outside the direct scope of the specification.

Sending JSON-RPC messages as binary data is a transport-specific behavior that could be seen as compliant with the JSON-RPC spec only if the client has been designed to decode the binary representation back into valid JSON before processing it. However, this deviates from the standard assumption of JSON-RPC clients, which typically expect JSON text.

---

Reference to the Specification

The key section in the JSON-RPC 2.0 spec is the Introduction, which defines JSON as the data format:

"JSON (RFC 4627) is used to represent method calls and responses."

RFC 4627 (the original JSON specification) explicitly defines JSON as a text-based data interchange format. Thus, any binary transport of JSON-RPC messages inherently requires additional processing to decode it back into valid JSON text.

[hrxi]

Not sure if this is LLM-generated text, but it looks super long form for something that could be said in a lot fewer words.

The issue arises because the official JSON-RPC client library, @open-rpc/client-js, does not handle WebSocket binary frames in the browser environment.

Have we tried asking them why this is the case?

[hrxi]

Ah, there's this issue: open-rpc/client-js#346.

[blouflashdb]

Ah, there's this issue: open-rpc/client-js#346.

Its from me. I am also part of the google group for json-rpc where I also asked to make sure I am correct that the nimiq server should send it as text and not binary.

[blouflashdb]

Right now I am checking other client and server implementations in different languages to see if they strictly send it as text. I checked two server implementations so far and they both send it strictly as text.

[blouflashdb]

Not sure if this is LLM-generated text, but it looks super long form for something that could be said in a lot fewer words.

The issue arises because the official JSON-RPC client library, @open-rpc/client-js, does not handle WebSocket binary frames in the browser environment.

Have we tried asking them why this is the case?

Dont read it if a 2 min text is to long for you. I will include a tl;dr next time.
I hope someone more invested reads it tho.

[hrxi]

Ah, there's this issue: open-rpc/client-js#346.

Its from me. I am also part of the google group for json-rpc where I also asked to make sure I am correct that the nimiq server should send it as text and not binary.

Care to cross-link the thread?

[blouflashdb]

Threads from new members need to be approved. I need to check at home if it has already been posted.

Will share as soon I get the thread/response.

[blouflashdb]

ETHs json rpc api for example sends the stream data as json text as you can see in this client implementation: https://github.com/wevm/viem/blob/main/src/utils/rpc/webSocket.ts

How many proofs will proof to you that you should not send it as binary. I can give many more client/server implemenations. Just need to know how many you need to be convinced.

If its really valid to send binary then thats okay. For me its just a few lines of code. But if its not valid. Many strict client libs in many different languages will simply not recognise it as valid json rpc which would be really bad for Nimiq devs.

[blouflashdb]

So far, I haven't come across a single JavaScript JSON-RPC 2.0 client/server library (with WebSocket support) that can send or expect a binary response.

You might also want to explore libraries in other programming languages for reference. Here’s an excellent list of JSON-RPC-related resources and libraries: awesome-json-rpc.

For proof, here are some specific examples of JavaScript libraries:

1. open-rpc/client-js
2. jsonrpc-bidirectional
3. jayson
4. mqtt-json-rpc + mqtt-json-rpc-websocket

If you'd like, I can also provide sample projects with these libraries. In each case, you’ll notice that the library throws errors because the payload is not a JSON string.

I believe this approach creates challenges for developers who want to work with your API since they can’t use standard JSON-RPC libraries. I hope this feedback resonates, and I’d recommend adjusting the implementation to use JSON strings instead of binary data. From my perspective, the current implementation of Nimiq's JSON-RPC is not fully compliant with the JSON-RPC 2.0 specification and I hope I can soon get the confirmation from the creators of the JSON-RPC spec.

[hrxi]

Care to cross-link the thread?

Did you get an answer?

[hrxi]

I edited the initial post to reflect that this seems to be orthogonal to standards compliance. It's probably still interesting to pursue due to the interaction with existing libraries.

[blouflashdb]

Care to cross-link the thread?

Did you get an answer?

Not yet but I will try to contact more members.