Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add packed format support for consumer offset decoding #388

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Add packed format support for consumer offset decoding #388

wants to merge 4 commits into from

Conversation

bge-kernel-panic
Copy link

We needed to be able to decode the exact consumer offset data (and not Kafka's idea of it through its builtin deserializer - it glosses over version differences and we wanted to know the exact data stored).

This PR adds:

  1. Support for various binary formats used internally by Kafka, such as varint and variable length strings
  2. Support for offset data decoding. This should be current up to Kafka 2.6. Note that group data is only partially decoded as it contains an array of records; we were not particularly interested in that data, we only wanted to parse offsets and not have kcat stop because it was seeing data it could not decode in __consumer_offsets.

@bge-kernel-panic
Extra pack formats (somewhat hodgepodge):
ca5e266 
- add : and , as literal characters
- hopefully fix endian swap for 16 bit integers
- Add "S" - count + string, used by Kafka for consumer offsets
- Add "U" - UUID
- Add "v" - varint (used by Kafka for some values)
- Add "C" - varint count + string
- Add "t" - 64 bit millisecond timestamp - note milliseconds are not displayed
            this uses ctime(3) which isn't the best (especially since the newline
            is removed through direct munging of the buffer) but should work well
            enough for this use case
@bge-kernel-panic
Add ability to parse __consuemr_offset data.
133feb1 
The problem with consumer offset is that it's versioned, so we need
to have some special logic for it.

This is a bit of a hack, reusing macros from the regular unpack and
thus creating a bunch of fake variables so the macros work. But it
does work properly.
@bge-kernel-panic
Handle nullable string better.
a88f9d6 
Handle v3 group metadata.
@bge-kernel-panic
Remove UUID support as I have no good way to test it.
2671113 
Reformat help a little bit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@bge-kernel-panic