Documentation request: clarify padding usage in objects

[davechallis]

I've been working on a velocypack parser (in Rust), and ran into some troubles parsing some output from ArangoDB:

0x0c 0x73 0x01 0x12 0x00 0x00 0x00 0x00 0x00 0x4a 0x61 0x75 0x74 0x68 0x6f 0x72 0x69 0x7a 0x65 0x64 0x1a 0x44 0x75 0x73 0x65 0x72 0x18 0x4b 0x69 0x73 0x41 0x64 0x6d 0x69 0x6e 0x55 0x73 0x65 0x72 0x1a 0x48 0x64 0x61 0x74 0x61 0x62 0x61 0x73 0x65 0x47 0x5f 0x73 0x79 0x73 0x74 0x65 0x6d 0x43 0x75 0x72 0x6c 0x4c 0x2f 0x5f 0x61 0x64 0x6d 0x69 0x6e 0x2f 0x65 0x63 0x68 0x6f 0x48 0x70 0x72 0x6f 0x74 0x6f 0x63 0x6f 0x6c 0x43 0x76 0x73 0x74 0x46 0x73 0x65 0x72 0x76 0x65 0x72 0x0b 0x3b 0x03 0x47 0x61 0x64 0x64 0x72 0x65 0x73 0x73 0x47 0x30 0x2e 0x30 0x2e 0x30 0x2e 0x30 0x44 0x70 0x6f 0x72 0x74 0x29 0x51 0x21 0x48 0x65 0x6e 0x64 0x70 0x6f 0x69 0x6e 0x74 0x53 0x68 0x74 0x74 0x70 0x3a 0x2f 0x2f 0x30 0x2e 0x30 0x2e 0x30 0x2e 0x30 0x3a 0x38 0x35 0x32 0x39 0x03 0x1b 0x13 0x48 0x70 0x6f 0x72 0x74 0x54 0x79 0x70 0x65 0x46 0x74 0x63 0x70 0x2f 0x69 0x70 0x46 0x63 0x6c 0x69 0x65 0x6e 0x74 0x0b 0x26 0x03 0x47 0x61 0x64 0x64 0x72 0x65 0x73 0x73 0x4a 0x31 0x37 0x32 0x2e 0x31 0x37 0x2e 0x30 0x2e 0x31 0x44 0x70 0x6f 0x72 0x74 0x29 0x7a 0x82 0x42 0x69 0x64 0x41 0x30 0x03 0x1e 0x16 0x49 0x69 0x6e 0x74 0x65 0x72 0x6e 0x61 0x6c 0x73 0x0a 0x46 0x70 0x72 0x65 0x66 0x69 0x78 0x41 0x2f 0x47 0x68 0x65 0x61 0x64 0x65 0x72 0x73 0x0a 0x4b 0x72 0x65 0x71 0x75 0x65 0x73 0x74 0x54 0x79 0x70 0x65 0x43 0x47 0x45 0x54 0x4a 0x70 0x61 0x72 0x61 0x6d 0x65 0x74 0x65 0x72 0x73 0x0a 0x46 0x73 0x75 0x66 0x66 0x69 0x78 0x01 0x49 0x72 0x61 0x77 0x53 0x75 0x66 0x66 0x69 0x78 0x01 0x44 0x70 0x61 0x74 0x68 0x41 0x2f 0x4e 0x72 0x61 0x77 0x52 0x65 0x71 0x75 0x65 0x73 0x74 0x42 0x6f 0x64 0x79 0x0b 0x17 0x02 0x44 0x74 0x79 0x70 0x65 0x46 0x42 0x75 0x66 0x66 0x65 0x72 0x44 0x64 0x61 0x74 0x61 0x01 0x0f 0x03 0x09 0x00 0xa9 0x00 0x28 0x00 0xea 0x00 0xd6 0x00 0x1b 0x00 0x03 0x01 0x22 0x01 0x99 0x00 0xe1 0x00 0x4a 0x00 0x29 0x01 0x17 0x01 0xf3 0x00 0x57 0x00 0x0f 0x01 0x39 0x00 0x15 0x00

After some investigation, it looks like this was because my parser encountered 4 bytes of padding after parsing the 0x0c <byte length> <num items> header (i.e. 0x0c 0x73 0x01 0x12 0x00 0x00 0x00 0x00 0x00).

The specification doesn't mention padding in the Objects section, so would be useful to include some details on where padding can appear in objects (would have saved me a lot of debugging time :) ).

[neunhoef]

Sorry for the delay in answering. It is a bit hidden, but lines 332-337 of the specs say:

For all types the offset table describes where the subvalues reside. It
is not necessary for the subvalues to start immediately after the number
of subvalues field. For performance reasons when building the value, it
could be desirable to reserve 8 bytes for the byte length and the number
of subvalues and not fill the gap, even though it turns out later that
offsets and thus the byte length only uses 2 bytes, say.

This is meant to say that the number of bytes between the type byte and the begin of the sub value pairs can be up to 8 bytes long, with padding zero bytes if needed. This is the same as for arrays. The rationale behind this is that it should be possible to write a larger velocypack object linearly, even if it is not known a priori how large it is. One can reserve 8 bytes in this case for this area and does not have to move memory down after the fact when it turns out that the object is in fact smaller.

If you have a good suggestion as to how to clarify this I am happy to amend the specs.

[davechallis]

@neunhoef thanks, that makes things a bit clearer.

My only suggestion could be to add it to the section where non-empty objects are defined, e.g. change from:

Non-empty objects look like this:

  one of 0x0b - 0x12    
  BYTELENGTH
  optional NRITEMS
  sub VPack values as pairs of attribute and value
  optional INDEXTABLE
  NRITEMS for the 8-byte case

to match similar format/wording used in the arrays section, e.g.:

Non-empty objects look like this:

  one of 0x0b - 0x12
  BYTELENGTH
  OPTIONAL NRITEMS
  OPTIONAL UNUSED: padding
  sub VPack values as pairs of attribute and value
  OPTIONAL INDEXTABLE
  NRITEMS for the 8-byte case

[neunhoef]

OK, will do.