Add parquet compressor using columnify

[okkez]

How to use:

Install columnify.

The sample configuration for parquet compressor.

<match>
  @id s3-parquet
  @type s3

  s3_region ap-northeast-1
  s3_bucket xxx
  store_as parquet
  <compress>
    parquet_compression_codec snappy
    record_type msgpack
    schema_type avro
    schema_file /path/to/log.avsc
  </compress>

  <format>
    @type msgpack
  </format>
</match>

log.avsc is like following:

{
  "name": "AccessLog",
  "type": "record",
  "fields": [
    { "name": "container_id", "type": "string" },
    { "name": "method", "type": "string" },...
  ]
}

See https://avro.apache.org/docs/current/spec.html for more details about avro schema.

Notice:
columnify's memory usage is proportional to file size. For example, columnify consumes about 750MB memory (RSS) while processing 128MB msgpack.

See also #221

[repeatedly]

Looks good to me.
Need feedback from other users.

[scrwr]

I'd actually really love to test it. But. We use bundler in our build process for installing all plugins and fluentd unfortunately stopped importing bundler installed git sources, so I end up with "Unknown output plugin 's3'" if I add this to our Gemfile

gem 'fluent-plugin-s3', git: 'https://github.com/okkez/fluent-plugin-s3.git', branch: 'add-parquet-compressor'

Can you help with getting this to work without completely workarounding the bundle install.

[lsudo]

thank you for this, I really appreciate that

[okkez]

This parquet compressor has worked fine for the recent 9 months on our system.

Can I merge this PR?

[okkez]

@repeatedly @ganmacs @kenhys @ashie Can you merge this PR and release the new version?

[ashie]

I'll merge & release it after waiting for comments from other maintainers for a few days.

[ashie]

The sample configuration for parquet compressor.

<match>
  @id s3-parquet
  @type s3

  s3_region ap-northeast-1
  s3_bucket xxx
  <compress>
    parquet_compression_codec snappy
    record_type msgpack
    schema_type avro
    schema_file /path/to/log.avsc
  </compress>

  <format>
    @type msgpack
  </format>
</match>

@okkez store_as parquet is also required, isnt it?

[okkez]

@okkez store_as parquet is also required, isnt it?

Yes. It is required.

[ashie]

We'll refine the document at #373

[ashie]

Thank you for your work!

[sunnymeska-w]

the compressor dosnt support list type in schema file. please add support for input type list