Merging parsers that consume entire input

[StefanCosminR]

Hello! I have multiple parsers that consume the entire input (using Rest() parser) and I can't find a nice way to merge them.
The general idea is that I have a string that can be split in 3 chunks by a separator, each chunk being parsable with a different parser.

To give my concrete example, the string I'm trying to parse is a connection string for Azure Notifications Hub.

let input = "Endpoint=sb://organization-name.servicebus.windows.net/;SharedAccessKeyName=DefaultListenSharedAccessSignature;SharedAccessKey=3252389randomCharacters43242380=09=/="

// The parsers:
let endpointInputParser = StartsWith("Endpoint=sb")
            .flatMap( { Always("https") } )
            .take(Rest())
            .map { $0 + $1 } // replace Endpoint=sb with https

let sasKeyNameParser = StartsWith("SharedAccessKeyName=")
            .take(Rest())

let sasKeyValueParser = StartsWith("SharedAccessKey=")
            .take(Rest())

// How to I merge them?

Many(Prefix(minLength: 1) { $0 != ";" }, separator: ";")
  .flatMap { components -> AnyParser<Substring, (Substring, Substring, Substring)> in
      guard components.count == 3 else {
          return Fail().eraseToAnyParser()
      }
      
      return Always((components[0], components[1], components[2])).eraseToAnyParser()
  }
  .map({ endpoint, keyName, keyValue in
      let endppointOutput = endpointInputParser
          .parse(endpoint)
       // ... the same for all parsers, manually merge the results, but it's error prone
  })

The solutions I though about was either implementing a MergeN parsers that would act similarly to TakeN, but it seems like there should be an easier way.

Another solution would be to make each parser aware of this separator and to read until they reach it instead of using Rest(), but I feel like they shouldn't know about that.

[ph1ps]

My (naive) approach would be to just pipe the component parsers down a couple of PrefixUpTo parsers like this:

let input = "Endpoint=sb://organization-name.servicebus.windows.net/;SharedAccessKeyName=DefaultListenSharedAccessSignature;SharedAccessKey=3252389randomCharacters43242380=09=/="

let endpointParser = StartsWith("Endpoint=sb")
  .take(Rest())
  .map { "https" + $0 }

let sasKeyNameParser = StartsWith("SharedAccessKeyName=")
  .take(Rest())

let sasKeyValueParser = StartsWith("SharedAccessKey=")
  .take(Rest())

let parser = PrefixUpTo(";")
  .pipe(endpointParser)
  .skip(";")
  .take(
    PrefixUpTo(";")
      .pipe(sasKeyNameParser)
      .skip(";")
  )
  .take(
    sasKeyValueParser
  )

This ensures your .take(Rest()) only consumes the rest of the string in the given domain and not consuming the "whole" rest. But I'm definitely open to other suggestions.

[StefanCosminR]

Thank you so much! It works perfectly and it looks much better.

[tgrapperon]

I think.take(Rest()) is too broad to control what you're going to parse. For example, endpointParser succeeds in parsing the whole input string. It shouldn't. You're asking "Is the input an endpoint?" and the parser says "yes", which is not exact. Parsers should fail if the input is not exactly what they should parse. This otherwise leads to issues like the ones you're encountering down the line.

Because you preferably don't want the endpointParser to be aware of its URL context (for reusability), an alternative way would be to use some CharacterSet instead of Rest(). You build the CharacterSet of all allowed characters for the endPoint (most likely CharacterSet.urlHostAllowed works out of the box here, but it can be completely custom too). You can then check that your parser works correctly by trying endpointParser.skip(Rest()) on the input.

These strict parsers will combine fluently when building the final parser: endpointr.skip(";").take(sasKeyName)….

[StefanCosminR]

That's certainly true for Endpoint, however for SharedAccessKeyName I don't know what characters are allowed. For SharedAccessKey I think the allowed characters are from the standard Base64 set. The only restriction I could be certain of is that ";" is not allowed and I didn't want the individual parsers to know that.

I agree the parsing could be made much safer, but here is the way Microsoft recommends implementing it https://docs.microsoft.com/en-us/rest/api/notificationhubs/common-concepts

I also checked their open source library for Swift and that's exactly how they're doing it, with .startsWith and .prefix, so I just wanted a nicer, more declarative way to write that.

But thank you for the observations, I might apply some of them. You opened my eyes a bit more to just how complex parsing really is.

[tgrapperon]

All right. The small issue with the ad-hoc piping is that its function is not immediately clear. Maybe you can try to define a Component parser like:

public struct ComponentParser<Separator, Component>: Parser where
  Component: Parser,
  Component.Input: Collection,
  Component.Input.SubSequence == Component.Input,
  Separator == Component.Input,
  Separator.Element: Equatable {
  public let component: Component
  public let separator: Separator
  public let isSeparatorRequired: Bool

  public init(_ component: Component, separator: Separator, isSeparatorRequired: Bool = false) {
    self.component = component
    self.separator = separator
    isSeparatorRequired = isSeparatorRequired
  }

  public func parse(_ input: inout Component.Input) -> Component.Output? {
    if isSeparatorRequired {
      return PrefixUpTo(separator)
        .pipe(component)
        .skip(StartsWith(separator))
        .parse(input)
    } else {
      return PrefixUpTo(separator).orElse(Rest())
        .pipe(component)
        .skip(StartsWith(separator).orElse(End()))
        .parse(input)
    }
  }
}

extension Parser {
  public func component<Separator>(separator: Separator) 
  -> ComponentParser<Separator, Self> where
    Input: Collection,
    Input.SubSequence == Input,
    Separator == Input,
    Separator.Element: Equatable {
    ComponentParser(self, separator: separator)
  }
}

With this parsers, the chain would look like:

endpoint.component(separator: ";")
  .take(sasKeyNameParser.component(separator: ";"))
  .take(sasKeyValueParser.component(separator: ";"))
// or the stricter
endpoint.component(separator: ";")
  .take(sasKeyNameParser.component(separator: ";"))
  .take(sasKeyValueParser)

(I didn't test the code Component parser, but you get the idea).
(Edit: Rethreaded the answer in the correct thread)