Unable to set Compression config when using AvroWriter

[richiesgr]

Hi
I would like to add more on this because I check it now
The problem is that Avro doesn't support Gzip compression by default.
The problem is that Avro and the message writer both use the same configuration params:
secor.compression.codec
So if you put org.apache.hadoop.io.compress.GzipCodec you get an exception because Avro writer try to use it and fail
If you put null (no compression) so the MessageWriter fail with an exception because it try to instance a class with that
If put empty it fails because there you must put value

So I don't know what to put here because I try to Big query to read the files and don't thing it support something else than Gzip or uncompressed. and you can't set neither !!

Originally posted by @richiesgr in #482 (comment)

[HenryCaiHaiying]

Richard, If you need to set those two params separately, you can modify the secor code to introduce two separate params.

…

On Mon, Nov 30, 2020 at 4:14 AM Richard Grossman ***@***.***> wrote: Hi I would like to add more on this because I check it now The problem is that Avro doesn't support Gzip compression by default. The problem is that Avro and the message writer both use the same configuration params: secor.compression.codec So if you put org.apache.hadoop.io.compress.GzipCodec you get an exception because Avro writer try to use it and fail If you put null (no compression) so the MessageWriter fail with an exception because it try to instance a class with that If put empty it fails because there you must put value So I don't know what to put here because I try to Big query to read the files and don't thing it support something else than Gzip or uncompressed. and you can't set neither !! *Originally posted by @richiesgr <https://github.com/richiesgr> in #482 (comment) <#482 (comment)>* — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1725>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABYJP74PTN7GEZNZSGCGKXTSSOECVANCNFSM4UHPPU4Q> .

[richiesgr]

Hi
Yes I've already made the modification.
Don't know if it's something that can interest someone for a PR
If yes I can push a PR

[HenryCaiHaiying]

I don't think it will hurt to introduce that parameter, feel free to submit a PR.

…

On Tue, Dec 1, 2020 at 1:01 AM Richard Grossman ***@***.***> wrote: Hi Yes I've already made the modification. Don't know if it's something that can interest someone for a PR If yes I can push a PR — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1725 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABYJP74AZISWO3HE2JTZGR3SSSWINANCNFSM4UHPPU4Q> .