Brimcap wiki updates

docs/Custom-Brimcap-Config.md

@@ -2,11 +2,10 @@
 
 - [Summary](#summary)
 - [Custom Zeek/Suricata Analyzers](#custom-zeeksuricata-analyzers)
-  * [Background](#background)
   * [Base Zeek/Suricata Installation](#base-zeeksuricata-installation)
   * [Example Configuration](#example-configuration)
 - [Custom NetFlow Analyzer](#custom-netflow-analyzer)
-  * [Background](#background-1)
+  * [Background](#background)
   * [Base nfdump Installation](#base-nfdump-installation)
   * [Example Configuration](#example-configuration-1)
 - [Debug](#debug)
@@ -20,46 +19,11 @@ richly-typed summary logs from pcap files. However, one of Brimcap's strengths
 is its ability to invoke any arbitrary combination of such "analyzers" that
 generate logs from pcaps. These analyzers could be alternate Zeek or Suricata
 installations that you've customized or other pcap-processing tools of your
-choosing.
-
-This article describes such example custom configurations. The section on
-[Custom Zeek/Suricata Analyzers](#custom-zeeksuricata-analyzers) focuses on
-recreating Brimcap configurations equivalent to those created with the
-customization options in Brim app release `v0.24.0` and older. The
-[Custom NetFlow Analyzer](#custom-netflow-analyzer) section builds on the
-approach to create a Brimcap configuration that generates NetFlow records from
-pcaps. The article then closes with some [Debug](#debug) tips.
+choosing. This article includes examples of both configurations along with
+[Debug](#debug) tips.
 
 # Custom Zeek/Suricata Analyzers
 
-## Background
-
-Starting with Brim release `v0.25.0`, Zeek and Suricata binaries are bundled
-differently than they were in release `v0.24.0` and older. A brief summary:
-
-* **`v0.24.0` and older** - The "backend" `zqd` process that ran behind the
-Brim app handled the launching of Zeek and Suricata binaries each time a pcap
-was imported. The launch of these processes was done via "runner" scripts that
-ensured appropriate pathnames and environment variables were set for them to
-run successfully. A [Zeek Customization](https://github.com/brimdata/brim/wiki/Zeek-Customization)
-article described how an alternate "Zeek Runner" script could be
-created/invoked such that Brim would launch Zeek from an install different than
-the one bundled with Brim. A similar option existed to invoke an alternate
-"Suricata Runner" (as well as a "Suricata Updater" for updating
-[rules](https://suricata.readthedocs.io/en/latest/rules/)) however
-these were never fully documented.
-
-* **`v0.25.0` and newer** - Brimcap is now bundled as part of Brim. Brimcap
-includes Zeek and Suricata binaries equivalent to those that were bundled with
-Brim `v0.24.0` and older, however they are no longer launched by the Zed
-backend. Instead Brim now runs `brimcap analyze` for each pcap imported into the
-app. In a default configuration, `brimcap analyze` invokes runner scripts
-that point to the Zeek/Suricata binaries that are bundled with Brimcap,
-effectively providing an out-of-the-box experience equivalent to what is
-possible in Brim `v0.24.0`. However, for custom configurations, Brimcap
-provides a YAML-based configuration that allows for flexibly invoking alternate
-Zeek/Suricata installations and/or alternate pcap processing tools.
-
 ## Base Zeek/Suricata Installation
 
 The goal in our first example customization will be to run Brim with the latest
@@ -121,9 +85,9 @@ outside the app to import a `sample.pcap` like so:
 
 ```
 $ export PATH="/opt/Brim/resources/app.asar.unpacked/zdeps:$PATH"
-$ zed api create testpool
-$ zed api use -p testpool
-$ brimcap analyze -config zeek-suricata.yml sample.pcap | zed api load -
+$ zed create testpool
+$ zed use testpool
+$ brimcap analyze -config zeek-suricata.yml sample.pcap | zed load -
 $ brimcap index -root "$HOME/.config/Brim/data/brimcap-root" -r sample.pcap
 ```
 
@@ -136,12 +100,12 @@ logs and open flows from the pcap via the **Packets** button.
 
 ![NetFlow Pool](media/Custom-Zeek-Suricata-Pool.png)
 
-The same combination of `brimcap` and `zed api` commands can be used to
-incrementally add more logs to the same pool and index for additional pcaps,
-which was not possible pre-Brimcap. The setting in the Brim **Preferences** for
-the **Brimcap YAML Config File** can also be pointed at the path to this
-configuration file, which will cause it to be invoked when you open or drag
-pcap files into Brim.
+The same combination of `brimcap` and `zed` commands can be used to
+incrementally add more logs to the same pool and index for additional pcaps.
+
+The setting in the Brim **Preferences** for the **Brimcap YAML Config File**
+can also be pointed at the path to this configuration file, which will cause it
+to be invoked when you open or drag pcap files into Brim.
 
 ![Brim YAML Config File Preference](media/Brim-Pref-YAML-Config-File.png)
 
@@ -174,8 +138,7 @@ performed on them.
 In our example configuration, the first analyzer invoked by Brimcap is a wrapper
 script as referenced in the YAML. In addition to reading from its standard input, it also
 tells Zeek to ignore checksums (since these are often set incorrectly on pcaps)
-and disables a couple of the excess log outputs. This is very similar to
-the Zeek Runner script that was included with Brim `v0.24.0`.
+and disables a couple of the excess log outputs.
 
 ```
 $ cat zeek-wrapper.sh
@@ -218,47 +181,48 @@ analyzers:
 ...
 ```
 
-What follows below the `globs:` setting is a Zed shaper script. Whereas
-the Zeek TSV logs contain Zed-compatible rich data types (timestamps, IP
+What follows below the `globs:` setting is a Zed shaper script. Whereas the
+Zeek TSV logs contain Zed-compatible rich data types (timestamps, IP
 addresses, etc.), since Suricata's EVE logs are NDJSON, here we use this shaper
 to assign better data types as the NDJSON is being converted for storage
-into the Zed Lake. Out-of-the-box, Brimcap automatically applies this same
-shaper script on the EVE output generated from its bundled Suricata.
+into the Zed lake. Out-of-the-box, Brimcap automatically applies this same
+[shaper script](https://github.com/brimdata/brimcap/blob/main/suricata.zed)
+ on the EVE output generated from its bundled Suricata.
 Here it's broken out and made part of the configuration YAML such that you can
 further modify it to suit your needs.
 ```
 ...
     shaper: |
-      type port=uint16;
+      type port=uint16
       type alert = {
         timestamp: time,
-        event_type: bstring,
+        event_type: string,
         src_ip: ip,
         src_port: port,
         dest_ip: ip,
         dest_port: port,
         vlan: [uint16],
-        proto: bstring,
-        app_proto: bstring,
+        proto: string,
+        app_proto: string,
         alert: {
           severity: uint16,
-          signature: bstring,
-          category: bstring,
-          action: bstring,
+          signature: string,
+          category: string,
+          action: string,
           signature_id: uint64,
           gid: uint64,
           rev: uint64,
           metadata: {
-            signature_severity: [bstring],
-            former_category: [bstring],
-            attack_target: [bstring],
-            deployment: [bstring],
-            affected_product: [bstring],
-            created_at: [bstring],
-            performance_impact: [bstring],
-            updated_at: [bstring],
-            malware_family: [bstring],
-            tag: [bstring]
+            signature_severity: [string],
+            former_category: [string],
+            attack_target: [string],
+            deployment: [string],
+            affected_product: [string],
+            created_at: [string],
+            performance_impact: [string],
+            updated_at: [string],
+            malware_family: [string],
+            tag: [string]
           }
         },
         flow_id: uint64,
@@ -271,15 +235,16 @@ further modify it to suit your needs.
           src_port: port,
           dest_ip: ip,
           dest_port: port,
-          proto: bstring,
+          proto: string,
           depth: uint64
         },
-        community_id: bstring
+        community_id: string
       }
-      filter event_type=="alert" | put this := shape(alert) | rename ts := timestamp
+      where event_type=="alert" | yield shape(alert) | rename ts := timestamp
 ```
 
-A full description of all that's possible with shapers is beyond
+A full description of all that's possible with
+[shapers](https://zed.brimdata.io/docs/language/overview/#9-shaping) is beyond
 the scope of this article. However, this script is quite simple and can
 be described in brief.
 
@@ -289,20 +254,20 @@ defined a single "wide" shape for _all_ alerts we've known Suricata to
 generate, which is convenient because it allows Brim to easily display them in
 columnar format.
 
-2. The `filter event_type=="alert"` trims the processed EVE events to only
+2. The `where event_type=="alert"` trims the processed EVE events to only
 alerts. If you want to let through more Suricata data besides just alerts, you
 could remove this part of the pipeline. If so, you'll likely want to explore
 the additional data and create shapers to apply proper data types to them,
 since this will be a prerequisite for doing certain Zed queries with the data
-(e.g.  a successful CIDR match requires IP addresses to be stored as `ip` type,
-not the string type in which they'd appear in unshaped NDJSON).
+(e.g., a successful CIDR match requires IP addresses to be stored as `ip` type,
+not the `string` type in which they'd appear in unshaped NDJSON).
 
-3. The `put this := shape(alert)` applies the shape of the `alert` type to each
+3. The `yield shape(alert)` applies the shape of the `alert` type to each
 input record. With what's shown here, additional fields that appear beyond
 those specified in the shaper (e.g. as the result of new Suricata features or
 your own customizations) will still be let through this pipeline and stored in
-the Zed Lake. If this is undesirable, add `| put this := crop(alert)` downstream
-of the first `put`, which will trim these additional fields.
+the Zed lake. If this is undesirable, add `| yield crop(alert)` downstream
+of the first `yield`, which will trim these additional fields.
 
 4. The `rename ts := timestamp` changes the name of Suricata's `timestamp`
 field to match the `ts` one used by Zeek, which allows the data from both
@@ -344,8 +309,7 @@ As we did with Zeek and Suricata, we create a [wrapper script](https://github.co
 Brimcap analyzer. It works in two phases, first creating binary NetFlow records
 and then converting them to CSV. `nfpcapd` only accepts a true pcap file input
 (not a device like `/dev/stdin`), so we first store the incoming pcap in a
-temporary file. We also use GNU `head` to trim out some trailing summary lines
-that `nfdump` appends by default which would trip up the Zed CSV reader.
+temporary file.
 
 ```
 $ cat nfdump-wrapper.sh 
@@ -357,55 +321,82 @@ cat - > "$TMPFILE"
 rm "$TMPFILE"
 for file in nfcapd.*
 do
-  /usr/local/bin/nfdump -r $file -o json > ${file}.json
+  /usr/local/bin/nfdump -r $file -o csv | head -n -3 > ${file}.csv
 done
 ```
 
 This script is called from our Brimcap config YAML, which includes a `globs:`
-setting to apply a Zed shaper to only the JSON files that were output from
+setting to apply a Zed shaper to only the CSV files that were output from
 nfdump.
 
 ```
 $ cat nfdump.yml 
 analyzers:
   - cmd: /usr/local/bin/nfdump-wrapper.sh
     name: nfdump
-    globs: ["*.json"]
+    globs: ["*.csv"]
     shaper: |
       type netflow = {
-        type: string,
-        sampled: int64,
-        export_sysid: int64,
-        t_first: time,
-        t_last: time,
-        proto: int64,
-        src4_addr: ip,
-        dst4_addr: ip,
-        icmp_type: int64,
-        icmp_code: int64,
-        src_port: uint16,
-        dst_port: uint16,
-        fwd_status: int64,
-        tcp_flags: string,
-        src_tos: int64,
-        in_packets: int64,
-        in_bytes: int64,
-        cli_latency: float64,
-        srv_latency: float64,
-        app_latency: float64,
-        label: string
+        ts: time,
+        te: time,
+        td: duration,
+        sa: ip,
+        da: ip,
+        sp: uint16,
+        dp: uint16,
+        pr: string,
+        flg: string,
+        fwd: bytes,
+        stos: bytes,
+        ipkt: uint64,
+        ibyt: uint64,
+        opkt: uint64,
+        obyt: uint64,
+        in: uint64,
+        out: uint64,
+        sas: uint64,
+        das: uint64,
+        smk: uint8,
+        dmk: uint8,
+        dtos: bytes,
+        dir: uint8,
+        nh: ip,
+        nhb: ip,
+        svln: uint16,
+        dvln: uint16,
+        ismc: string,
+        odmc: string,
+        idmc: string,
+        osmc: string,
+        mpls1: string,
+        mpls2: string,
+        mpls3: string,
+        mpls4: string,
+        mpls5: string,
+        mpls6: string,
+        mpls7: string,
+        mpls8: string,
+        mpls9: string,
+        mpls10: string,
+        cl: float64,
+        sl: float64,
+        al: float64,
+        ra: ip,
+        eng: string,
+        exid: bytes,
+        tr: time
       }
-      put this := shape(netflow)
+      yield shape(netflow)
 ```
 
 Putting it all together, we can test it by using our command combination to
 create a new pool and import the data for a sample pcap.
 
 ```
 $ export PATH="/opt/Brim/resources/app.asar.unpacked/zdeps:$PATH"
-$ zed api create testpool2
-$ zed api use -p testpool2
-$ brimcap analyze -config nfdump.yml sample.pcap | zed api load -
+$ zed create testpool2
+$ zed use testpool2
+$ brimcap analyze -config nfdump.yml sample.pcap | zed load -
 ```
 
 Our pool is now ready to be queried in Brim.

docs/Geolocation.md

@@ -42,6 +42,7 @@ currently being held open to gather interest:
 | [brim/936](https://github.com/brimdata/brim/issues/936) | Geolocation map visualization                  |
 | [brim/954](https://github.com/brimdata/brim/issues/954) | Look up Geolocation data on-demand             |
 | [brim/955](https://github.com/brimdata/brim/issues/955) | Allow user to replace the Geolocation database |
+| [geoip-conn/39](https://github.com/brimdata/geoip-conn/issues/39) | Include autonomous system info       |
 
 If you're interested in additional geolocation features, please follow the
 links to review these issues and click :+1: below the description on any of

docs/Troubleshooting.md

@@ -71,7 +71,7 @@ doesn't arise again in the future. We understand that concerns regarding
 may make it difficult or impossible to share such data. If the specific
 capture with which you first saw the problem cannot be shared, an alternative
 approach would be to share a special debug capture generated by the same
-system, but made up of traffic known to be innocuous (e.g. performing a
+system, but made up of traffic known to be innocuous (e.g., performing a
 `curl` to a common public web site).
 
 If you're able to reliably reproduce the problem, please
@@ -175,7 +175,7 @@ Before/when [opening a new issue](https://github.com/brimdata/brimcap/issues/new
 you can help us out by doing the following:
 
 * Browse the existing [open issues](https://github.com/brimdata/brimcap/issues?q=is%3Aissue+is%3Aopen). If you confirm you're hitting one of those, please add a comment to it rather than opening a new issue.
-* [Gather info](#gathering-info) that that may help in reproducing the issue and testing a fix, and attach it to your issue.
+* [Gather info](#gathering-info) that may help in reproducing the issue and testing a fix, and attach it to your issue.
 * Feel free to chat with the team on the [public Slack](https://www.brimdata.io/join-slack/) before/after opening an issue.
 * When you open a new issue, its initial text will include a template with standard info that will almost always be needed. Please include detail for all areas of the template.
 

examples/nfdump-wrapper.sh

@@ -6,5 +6,5 @@ cat - > "$TMPFILE"
 rm "$TMPFILE"
 for file in nfcapd.*
 do
-  /usr/local/bin/nfdump -r $file -o json > ${file}.json
+  /usr/local/bin/nfdump -r $file -o csv | head -n -3 > ${file}.csv
 done