Bytestrings (ay)

[LEW21]

Hi,

I'm the creator of pydbus, DBus bindings for Python. I've stumbled upon the case of bytestrings - ay - which are commonly used to send non-Unicode data over DBus.

Because of the GDBus's API, apps commonly require these strings to be '\0'-terminated. And that's completely unintuitive for high-level languages, and something that requires the user to read API docs.

I'm thinking about solving the problem automatically, and I see two solutions:

1. Auto-append and auto-strip the last character of bytestrings;
2. Decree (how?) that GDBus API is wrong, and strings sent over the wire should not be \0-terminated.

Whatever we choose, the solution should be agreed upon by authors of bindings for all languages. That's why I'm here.

Do you have any experience with "ay"-encoded bytestrings?

Relevant discussion: LEW21/pydbus#27

[jsouthworth]

I don't think that 'ay' should be treated any differently than 'ai' since it is just an array container full of bytes. The D-Bus spec container marshaling section doesn't say anything about NUL termination of arrays, so my inclination is to say the GDBus implementation is wrong (Unless I've missed something).

[alexlarsson]

ay == array of bytes. In general no special magic should happen to these, its what you would use to pass e.g. a png image as data over dbus, which would break if zeros got magically added or removed.

However, on a higher level there is the concept of "bytestrings", which are sometimes used instead of strings in dbus. In particular, its often used for pathnames, because unix pathnames are not strings in the traditional sense (i.e. all byte combinations other than nul bytes are valid, and they have no specified encoding, whereas the dbus "s" type is utf8 only). For ease of use in C these are typically sent with terminating zeros over dbus (just like the "s" type in dbus already guarantees) which means you can take the pointer directly to the dbus message and pass to e.g. the open syscall.

To simplify such use glib has helper functions like g_variant_get_bytestring() that ensures that the array is null terminated (it returns the empty string otherwise) and g_variant_new_bytestring() (creates an array of bytes including the terminating 0 from a "C string"). However, these are just a helper functions, and not something you'd use for non-bytestring byte arrays.

In the particular case of golang, the "string" type is similar to C in that it doesn't specify the encoding (although its not zero terminated). So, a dbus "s" could either map to a "rune[]", or a "string" with extra utf8 guarantees. "ay" however should always map to byte[]. It makes sense to add extra helper apis to make zero terminated "ay" from a gloang "string" though, as well as something that parses "ay", removing the zero and creates a "string".

[omezro]

I have created a DBus service that receives two parameters: a string and a []byte containing JPEG data. The string data works fine, but when passing the JPEG data, I encounter an error: "dbus: wire format error: input has a not-UTF-8 char in string."