Panic if provide a valid jsonl file

[okkez]

$ columnify -schemaType avro -schemaFile rails-log.avsc -recordType jsonl crash.json.log
PAR1panic: reflect: call of reflect.Value.Type on zero Value [recovered]
        panic: send on closed channel

goroutine 11 [running]:
github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs.func1.1(0xc002ebe4f0, 0xc00002c900)
        /home/kenji/go/pkg/mod/github.com/xitongsys/[email protected]/writer/writer.go:208 +0xc8
panic(0x9dd420, 0xc002b18ba0)
        /usr/lib/go-1.14/src/runtime/panic.go:969 +0x166
reflect.Value.Type(0x0, 0x0, 0x0, 0x4, 0x9b5420)
        /usr/lib/go-1.14/src/reflect/value.go:1872 +0x183
github.com/xitongsys/parquet-go/common.SizeOf(0x0, 0x0, 0x0, 0xc00329ce30)
        /home/kenji/go/pkg/mod/github.com/xitongsys/[email protected]/common/common.go:503 +0x5a
github.com/xitongsys/parquet-go/layout.TableToDataPages(0xc000038680, 0xc000002000, 0x1, 0x22, 0xc0000ce0e8, 0x2, 0x48b)
        /home/kenji/go/pkg/mod/github.com/xitongsys/[email protected]/layout/page.go:89 +0x259
github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs.func1(0xc002ebe4f0, 0xc00002c900, 0xc0000203c0, 0xc002e76cc8, 0xc000189ae8, 0x1, 0x1, 0x0, 0x5b6, 0x0)
        /home/kenji/go/pkg/mod/github.com/xitongsys/[email protected]/writer/writer.go:230 +0x44d
created by github.com/xitongsys/parquet-go/writer.(*ParquetWriter).flushObjs
        /home/kenji/go/pkg/mod/github.com/xitongsys/[email protected]/writer/writer.go:195 +0x242

I could not understand the error logs above.

crash.json.log

I use the following avro schema:

{
  "name": "RailsAccessLog",
  "type": "record",
  "fields": [
    {
      "name": "container_id",
      "type": "string"
    },
    {
      "name": "container_name",
      "type": "string"
    },
    {
      "name": "source",
      "type": "string"
    },
    {
      "name": "log",
      "type": "string"
    },
    {
      "name": "__fluentd_address__",
      "type": "string"
    },
    {
      "name": "__fluentd_host__",
      "type": "string"
    },
    {
      "name": "role",
      "type": "string"
    },
    {
      "name": "host",
      "type": "string"
    },
    {
      "name": "severity",
      "type": "string"
    },
    {
      "name": "status",
      "type": "string"
    },
    {
      "name": "db",
      "type": "float"
    },
    {
      "name": "view",
      "type": "float"
    },
    {
      "name": "duration",
      "type": "float"
    },
    {
      "name": "method",
      "type": "string"
    },
    {
      "name": "path",
      "type": "string"
    },
    {
      "name": "remote_ip",
      "type": "string"
    },
    {
      "name": "agent",
      "type": "string"
    },
    {
      "name": "params",
      "type": "string"
    },
    {
      "name": "tag",
      "type": "string"
    },
    {
      "name": "time",
      "type": "string"
    }
  ]
}

BTW, the following command does not panic:

$ columnify -recordType jsonl -schemaType avro -schemaFile rails-log.avsc =(head -n 9 crash.json.log) 

Any help?

[syucream]

@okkez It's caused by schema mismatching, the last log entry doesn't have severity required field:

$ jq .severity crash.json.log
"INFO"
"INFO"
"INFO"
"INFO"
"INFO"
"INFO"
"INFO"
"INFO"
"INFO"
null

I modified severity type in the Avro schema like below:

...
   {
      "name": "severity",
      "type": ["null", "string"]
    },
...

Then it didn't fail.

$ ./columnify -recordType jsonl -schemaType avro -schemaFile rails-log.avsc =(head -n 10 crash.json.log)  > /dev/null
$ echo $?
0

The cause of this panic looks same as #24

[okkez]

Thanks!