Parsing big Data blobs where all related bytes aren't sequential

[ryanbooker]

I'm parsing a big chunk of Data from a bluetooth device. The Data has grown with each version of the product, and things you might think of as related or naturally group together are not necessarily sequential in the Data. There are dozens of bytes to take, and a few that are simply markers to skip.

Reading the Data naively into a single type would require quite a few new overrides of take and TakeN parsers.

Any thoughts on parsing strategies?

[stephencelis]

@ryanbooker take ultimately nests OK, but you would have to map to unpack the tuples at the end. We're down to add more take and TakeN parsers, though! How many are we talking about for your use case?

[ryanbooker]

I was thinking to add them locally, to not overburden the lib… honestly, I think I'll run out of alphabet by the time I'm finished. 😆

I'll do it with 5 and mapping, and see where I get. Maybe some structure will become apparent. I'll keep you posted. 😅

[stephencelis]

We took things to 11 in Overture 😉 https://github.com/pointfreeco/swift-overture/blob/97b65147b98cc39e4b4a79a094cc621b44145688/Sources/Overture/ZipOptional.swift#L177

[ryanbooker]

In case it's useful, I've PR'd Take parsers up to 11. Though I'm not sure it's a good idea, given how slow compile times start getting. #24 😬

[ryanbooker]

I'll need to find a better strategy than the naive approach anyway. At around 16 chained parsers (take or skip) the compiler runs screaming into the hills. And after about 6 takes compile times start slowing down.

Perhaps I need some intermediate chunks, and a second pass to create some cleaner more useful groupings/types.

[ryanbooker]

I found the following higher order parser useful for this big messy data blob. Parsing it linearly and then sorting out what's what was a huge pain, but being able to logically group things by repeatedly parsing out just the relevant bits seems to work quite well.

public extension Parser {
    /// Returns a parser that runs a parser without consuming the input.
    ///
    /// - Returns: A parser that runs the parser without consuming the input.
    @inlinable
    func nonConsuming() -> Parsers.NonConsuming<Self> {
        .init(self)
    }
}

public extension Parsers {
    /// A parser that runs a parser without consuming the input.
    struct NonConsuming<P>: Parser
    where P: Parser {
        public let p: P

        @inlinable
        public init(_ p: P) {
            self.p = p
        }

        @inlinable
        public func parse(_ input: inout P.Input) -> (P.Output)? {
            let original = input
            defer { input = original }
            return p.parse(&input)
        }
    }
}

e.g. Each parser operates on the entire blob

let parser = 
    aParser.nonConsuming()
    .bParser.nonConsuming()
    .cParser.nonConsuming()
    .map { a, b, c in // much more manageable }