Panic if input files does not have fields defined by avro schema

[okkez]

Panic occured when execute the following command.

$ columnify -schemaType avro -schemaFile ~/wc/src/github.com/reproio/columnify/examples/schema/primitives.avsc =(echo '{"boolean": false, "string": "foobar"}') > a
panic: interface conversion: interface {} is nil, not string [recovered]
        panic: send on closed channel

goroutine 11 [running]:
github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs.func1.1(0xc00007dd20, 0xc00002c900)
        /home/kenji/go/pkg/mod/github.com/xitongsys/[email protected]/writer/writer.go:208 +0xc8
panic(0x9ef140, 0xc00018fc20)
        /usr/lib/go-1.14/src/runtime/panic.go:969 +0x166
github.com/xitongsys/parquet-go/encoding.WritePlainBYTE_ARRAY(0xc00007de40, 0x1, 0x1, 0x0, 0x0, 0x0)
        /home/kenji/go/pkg/mod/github.com/xitongsys/[email protected]/encoding/encodingwrite.go:107 +0x1bc
github.com/xitongsys/parquet-go/encoding.WritePlain(0xc00007de40, 0x1, 0x1, 0x6, 0xc0002f5460, 0x0, 0xc00007de40)
        /home/kenji/go/pkg/mod/github.com/xitongsys/[email protected]/encoding/encodingwrite.go:51 +0x10a
github.com/xitongsys/parquet-go/layout.(*Page).EncodingValues(0xc000038600, 0xc00007de40, 0x1, 0x1, 0x1, 0xc00007de40, 0x0)
        /home/kenji/go/pkg/mod/github.com/xitongsys/[email protected]/layout/page.go:166 +0xd9
github.com/xitongsys/parquet-go/layout.(*Page).DataPageCompress(0xc000038600, 0x1, 0x0, 0x0, 0xc0002f5460)
        /home/kenji/go/pkg/mod/github.com/xitongsys/[email protected]/layout/page.go:183 +0x187
github.com/xitongsys/parquet-go/layout.TableToDataPages(0xc000038500, 0xc000002000, 0x1, 0x0, 0x1, 0xc0000e8070, 0xc00031e250)
        /home/kenji/go/pkg/mod/github.com/xitongsys/[email protected]/layout/page.go:116 +0x812
github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs.func1(0xc00007dd20, 0xc00002c900, 0xc0000b0280, 0xc0002f5320, 0xc00031e248, 0x1, 0x1, 0x0, 0x1, 0x0)
        /home/kenji/go/pkg/mod/github.com/xitongsys/[email protected]/writer/writer.go:230 +0x44d
created by github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs
        /home/kenji/go/pkg/mod/github.com/xitongsys/[email protected]/writer/writer.go:195 +0x242

And this will create an invalid file as the parquet format.
The invalid file causes an error on Athena if we use this command as a compressor of fluent-plugin-s3.

But the following command does not panic:

$ columnify -schemaType avro -schemaFile ~/wc/src/github.com/reproio/columnify/examples/schema/primitives.avsc =(echo '{"boolean": false, "string": "foobar"}') 
PAR12020/05/01 14:53:35 Failed to write: interface conversion: interface {} is nil, not string

[syucream]

The panic is caused by xitongsys/parquet-go#236 , parquet-go will be released in near future.
For more ideal way to handle the problem, we should check records with a given schema at our record formatter to detect errors earlier.

[syucream]

Let's handle the panic problem in this issue, and separate a better schema handling at #27 🙏

[okkez]

Thanks!

[syucream]

It should be fixed by #31