Skip to content

anemos-io/protobeam

Repository files navigation

Proto-Beam

Warning this is an evolving product, even Pre-Alpha quality. Don't use this in production just yet.

Description

Utilities for handling and converting ProtocolBuffers. Focused but not limited to Apache Beam.

Features

  • Convert ProtoBuf schema to
    • BigQuery proto model
    • BigQuery api model
  • Convert from ProtoBuf to
    • TableRow
  • Convert to ProtoBuf from
    • TableRow
    • GenericRecord (BigQuery dump)
  • Apache Beam Formatter for
    • BigQueryIO.TypedRead
    • BigQueryIO.Write
  • Annotation support for
    • Field description
    • Field deprecation

Why ProtoBuf

  • ProtoBuf has a good DSL, with contract first in mind.
  • ProtoBuf has a binary versioning strategy: A message can even be decoded without a schema.
  • ProtoBuf has a collection of Well Known Types: Native timestamps.
  • ProtoBuf Java implementation has a powerfull schema API.
  • ProtoBuf has options so we can change the behaviour of the pipeline.
  • ProtoBuf has good tooling for generating cross language code.

Known Issues

  • DynamicMessage doesn't seem to be supported by ProtoCodec (in Beam)
  • Only tested happy flow
  • OneOf not supported yet
  • Everything in BigQuery is NULLABLE

Examples

env name example
RUNNER Direct
GOOGLE_PROJECT_ID myproject
BQ_GCS_TEMP gs://my-bucket/tmp
BQ_OUT_DATASET my-project:my-dataset

Create 3 tables in a dataset, converted from Protobuf

transform/src/main/java/io.anemos.protobuf.examples.ProtoToBigQueryPipeline

Read the created tables, convert to protobuf and output to stderr

transform/src/main/java/io.anemos.protobuf.examples.ProtoFromBigQueryPipeline

Roadmap

  • ProtoBuf to BigQuery DDL
  • ProtoBuf to Spanner
  • ProtoBuf to/from BSON
  • ProtoBuf to/from Avro
  • Recursive messages
  • OneOf type
  • Well Known Types
  • Wrapper Types
  • Plugable message type convertions
  • Options support for
    • Description
    • Deprecation
    • Flattening
    • Event time
  • Register standard Options