Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow "brimcap load" of log files in non-auto-detected formats #80

Closed
philrz opened this issue May 6, 2021 · 2 comments
Closed

Allow "brimcap load" of log files in non-auto-detected formats #80

philrz opened this issue May 6, 2021 · 2 comments

Comments

@philrz
Copy link
Contributor

philrz commented May 6, 2021

Right now brimcap load seems dependent on the log outputs of analyzers being auto-detect-able. This works fine for the main cases of Zeek and Suricata since their default outputs are Zeek TSV and NDJSON, respectively. However, while working on #72 I happened to try working with a NetFlow analyzer's CSV output. Using these configs with Brimcap v0.0.3, we can see how it fails because we don't currently auto-detect CSV.

$ cat nfdump-wrapper-csv.sh 
#!/bin/bash
TMPFILE=$(mktemp)
cat - > "$TMPFILE"
nfpcapd -r "$TMPFILE" -l .
rm "$TMPFILE"
for file in nfcapd.*
do
  nfdump -r $file -o csv | ghead -n -3 > ${file}.csv
done

$ cat nfdump-csv.yml
analyzers:
  - cmd: nfdump-wrapper-csv.sh
    globs: ["*.csv"]
    
$ brimcap load -root "$HOME/Library/Application Support/Brim/data/brimcap-root" -config nfdump-csv.yml -p testpool ~/pcap/wrccdc.pcap 
100.0% 500.0MB/500.0MB records=0 
Post "http://localhost:9867/pool/1sArPVhV4gBbiH5B1E8NytXJ34G/log": format detection error
	tzng: line 1: bad format
	zeek: line 1: bad types/fields definition in zeek header
	zjson: line 1: invalid character 's' in literal true (expecting 'r')
	zson: identifier "ts" must be enum and requires decorator
	zng: zng type ID out of range
	parquet: auto-detection not supported
	zst: auto-detection not supported

If we wanted to support this, I can think of two ways to proceed.

  1. Allow the specification of input format in brimcap load (similar to the zq -i options)
  2. Add auto-detection support for all formats at the Zed layer (Auto-detect all input formats for files super#2517)
@philrz
Copy link
Contributor Author

philrz commented Jun 8, 2021

Note to self: For now I've added a comment in the "Custom Brimcap Configuration" article linking to this as an open issue, so that way the reader understands the current limitation is temporary. If/when we address this issue, I should update the wiki article to remove the workaround and the comment.

@nwt
Copy link
Member

nwt commented May 19, 2023

We removed brimcap load in #117.

@nwt nwt closed this as completed May 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants