From 6b125c08d87d79ff7a1208a51d9f6b6bf36e95ba Mon Sep 17 00:00:00 2001 From: Phil Rzewski Date: Mon, 25 Nov 2024 11:58:28 -0800 Subject: [PATCH] Rename README.md->_index.md to fix broken hyperlinks --- CHANGELOG.md | 34 +++++++++---------- docs/_index.md | 10 +++--- docs/commands/_index.md | 4 +-- docs/commands/super-db.md | 6 ++-- docs/commands/super.md | 18 +++++----- docs/formats/csup.md | 2 +- docs/formats/zjson.md | 2 +- docs/integrations/fluentd.md | 4 +-- .../zeek/data-type-compatibility.md | 2 +- .../zeek/reading-zeek-log-formats.md | 2 +- docs/lake/format.md | 2 +- docs/language/_index.md | 4 +-- docs/language/conventions.md | 2 +- docs/language/data-types.md | 2 +- docs/language/expressions.md | 4 +-- docs/language/lateral-subqueries.md | 2 +- docs/language/operators/summarize.md | 2 +- docs/language/overview.md | 4 +-- docs/language/pipeline-model.md | 6 ++-- docs/language/search-expressions.md | 2 +- docs/language/shaping.md | 2 +- docs/tutorials/zed.md | 2 +- docs/tutorials/zq.md | 8 ++--- 23 files changed, 63 insertions(+), 63 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 120fc2e3a4..a95b819aed 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,10 +3,10 @@ * Improve the performance of the [`load` operator](docs/language/operators/load.md) by removing an unnecessary/inefficient merge (#5200) * Improve the [`sort` operator](docs/language/operators/sort.md) to allow different ordering for each key (#5203, #5262) * Update the [Zeek reference shaper docs](docs/integrations/zeek/shaping-zeek-json.md#reference-shaper-contents) to incorporate changes for [Zeek v7.0.0](https://github.com/zeek/zeek/releases/tag/v7.0.0) logs (#5212) -* Update the [`summarize` operator docs](docs/language/operators/summarize.md) to show the use of `by` without an [aggregate function](docs/language/aggregates/README.md) (#5216) +* Update the [`summarize` operator docs](docs/language/operators/summarize.md) to show the use of `by` without an [aggregate function](docs/language/aggregates/_index.md) (#5216) * Update the [`grok` function docs](docs/language/functions/grok.md) with additional examples and guidance (#5243) * Update the [Lateral Subquery docs](docs/language/lateral-subqueries.md) with an emphasis on when primitive values or arrays are returned by [Lateral Expressions](docs/language/lateral-subqueries.md#lateral-expressions) (#5264) -* The terms "pipeline" and "branch" are now used throughout the [Zed docs](docs/README.md) instead of "dataflow" and "leg" (#5272) +* The terms "pipeline" and "branch" are now used throughout the [Zed docs](docs/_index.md) instead of "dataflow" and "leg" (#5272) * Add docs for [`lake` output format](docs/commands/super.md#superdb-data-lake-metadata-output) and [`zed ls`](docs/commands/super-db.md#ls) (#5187) * Add docs for the [`top` operator](docs/language/operators/top.md) (#5276) * Add [`fluentd` integration docs](docs/integrations/fluentd.md) (#5190, #5195) @@ -22,13 +22,13 @@ * Add the `-pool` flag to [`zed manage`](docs/commands/super-db.md#manage) (#5164) * Fix an issue where the lake API was not providing query descriptions for Zed programs that contain scopes (#5152) * Fix an issue where attempts to use the [`load` operator](docs/language/operators/load.md) in `zq` caused a panic (#5162) -* Fix a parser issue with collisions between the names of [user-defined operators](docs/language/statements.md#operator-statements) and [functions](docs/language/statements.md#func-statements) and some built-in [operators](docs/language/operators/README.md) (#5161) +* Fix a parser issue with collisions between the names of [user-defined operators](docs/language/statements.md#operator-statements) and [functions](docs/language/statements.md#func-statements) and some built-in [operators](docs/language/operators/_index.md) (#5161) * Fix an issue where using `null` values in math caused a panic (#5163) ## v1.16.0 * Improve ZNG scanning performance (#5101, #5103) * Improve the error message shown when `zq` is invoked with a single argument that's not a valid query and doesn't contain a source (#5119) -* Update [Zeek integration docs](docs/integrations/zeek/README.md), including [reference shaper](docs/integrations/zeek/shaping-zeek-json.md) changes for [Zeek v6.2.0](https://github.com/zeek/zeek/releases/tag/v6.2.0) data (#5106) +* Update [Zeek integration docs](docs/integrations/zeek/_index.md), including [reference shaper](docs/integrations/zeek/shaping-zeek-json.md) changes for [Zeek v6.2.0](https://github.com/zeek/zeek/releases/tag/v6.2.0) data (#5106) * [String literals](docs/language/expressions.md#formatted-string-literals) now use the "f-string" format `f"{ }"` instead of the previous `${ }` (#5123) * Prototype SQL support has been dropped from the Zed language (full SQL support is expected at a later date) (#5109) * Empty objects and arrays in JSON output are now consistently printed on a single line (#5127) @@ -241,8 +241,8 @@ ## v1.0.0 -* Comprehensive [documentation](docs/README.md) -* Substantial improvements to the [Zed language](docs/language/README.md) +* Comprehensive [documentation](docs/_index.md) +* Substantial improvements to the [Zed language](docs/language/_index.md) * Revamped [`zed` command](docs/commands/super-db.md) * New Zed lake format (see #3634 for a migration script) * New version of the [ZNG format](docs/formats/bsup.md) (with read-only support for the previous version) @@ -264,7 +264,7 @@ * `zapi`, `zed lake serve`: Add authentication with Auth0 (#3266) * Fix an issue preventing casting from `ip` to `ip` (#3259) * `zed lake serve`: Respect the Accept request header for `GET /events` (#3246) -* Add [function documentation](docs/language/functions/README.md) (#3215) +* Add [function documentation](docs/language/functions/_index.md) (#3215) * `zed lake serve`: Change the default response content encoding to ZSON (#3242) * `zapi load`, `zed lake load`: Add the `-meta` flag to embed custom metadata in commits (#3237) @@ -280,7 +280,7 @@ * Fix a panic when compiling `SELECT ... GROUP BY ...` (#3193) * Fix a bug in which data loaded through the Zed lake service was stored uncompressed (#3198) * Add all lake index commands to Zed lake service (#3181) -* Reorganize [language documentation](docs/language/README.md) (#3187) +* Reorganize [language documentation](docs/language/_index.md) (#3187) * Make `fuse()` output deterministic (#3190) * Use lake indexes to speed up queries (#3158) * Fix bug where constants blocked `from` operator wiring logic (#3185) @@ -329,7 +329,7 @@ As you can see below, there's been many changes since the last Zed GA release! * Enhancements to the Zed language to unify search and expression syntax, introduce new operators and functions for data exploration and shaping, and more! Review the - [Zed language docs](docs/language/README.md) + [Zed language docs](docs/language/_index.md) for details. The exhaustive set of changes is listed below. Come talk to us on @@ -454,7 +454,7 @@ questions. * Fix an issue where `len()` of a `null` array was evaluating to something greater than zero (#2761) * Fix an issue where `sort` with no fields was ignoring alias types and nested fields when picking a sort field (#2762) * Fix an issue where unexpected `cut: no record found` warnings were returned by `zed lake query` but not when the same data was queried via `zq` (#2764) -* Move and extend the [Zeek interoperability docs](docs/integrations/zeek/README.md) (#2770, #2782, #2830) +* Move and extend the [Zeek interoperability docs](docs/integrations/zeek/_index.md) (#2770, #2782, #2830) * Create endpoints in the Zed lake service API that correspond to underlying Zed lake operations, and expose them via `zapi` commands (#2741, #2774, #2786, #2775, #2794, #2795, #2796, #2920, #2925, #2928) * Fix an issue where `zq` would surface a syntax error when reading ZSON it had sent as output (#2792) * Add an `/events` endpoint to the API, which can be used by clients such as the Brim app to be notified of pool updates (#2791) @@ -469,8 +469,8 @@ questions. * Have the [Python client](python) use the `/query` endpoint for the Zed lake (#2869) * Minimize the amount of surrounding context shown when reporting parse errors (#2864) * Field assignments in [`join`](docs/language/operators/join.md) now behave like [`cut`](docs/language/operators/cut.md) instead of `pick` (#2868) -* Add more background/context to Zed top-level language [README](docs/language/README.md) (#2866 #2878, #2901) -* Unify `from`, `split`, and `switch` syntax to the forms shown [here](docs/language/README.md) (#2871, #2896) +* Add more background/context to Zed top-level language [README](docs/language/_index.md) (#2866 #2878, #2901) +* Unify `from`, `split`, and `switch` syntax to the forms shown [here](docs/language/_index.md) (#2871, #2896) * Shapers can now cast values of the `null` type to any type (e.g., arrays or records) (#2882) * Fix an issue where [`join`](docs/language/operators/join.md) was failing to match on values of comparable types (e.g., `string` and `bstring`) (#2880, #2884) * Shapers can now cast a value to a `union` type (#2881) @@ -628,7 +628,7 @@ questions. ## v0.23.0 * zql: Add `week` as a unit for [time grouping with `every`](docs/language/functions/every.md) (#1374) -* zq: Fix an issue where a `null` value in a [JSON type definition](docs/integrations/zeek/README.md) caused a failure without an error message (#1377) +* zq: Fix an issue where a `null` value in a [JSON type definition](docs/integrations/zeek/_index.md) caused a failure without an error message (#1377) * zq: Add [`zst` format](docs/formats/csup.md) to `-i` and `-f` command-line help (#1384) * zq: ZNG spec and `zq` updates to introduce the beta ZNG storage format (#1375, #1415, #1394, #1457, #1512, #1523, #1529), also addressing the following: * New data type `bytes` for storing sequences of bytes encoded as base64 (#1315) @@ -644,11 +644,11 @@ questions. * zqd: Check and convert alpha ZNG filestores to beta ZNG (#1574, #1576) * zq: Fix an issue where spill-to-disk file names could collide (#1391) * zq: Allow the [`fuse` operator](docs/language/operators/fuse.md) to spill-to-disk to avoid memory limitations (#1355, #1402) -* zq: No longer require `_path` as a first column in a [JSON type definition](docs/integrations/zeek/README.md) (#1370) +* zq: No longer require `_path` as a first column in a [JSON type definition](docs/integrations/zeek/_index.md) (#1370) * zql: Improve ZQL docs for [aggregate functions](docs/language/operators/summarize.md) and grouping (#1385) * zql: Point links for developer docs at [pkg.go.dev](https://pkg.go.dev/) instead of [godoc.org](https://godoc.org/) (#1401) * zq: Add support for timestamps with signed timezone offsets (#1389) -* zq: Add a [JSON type definition](docs/integrations/zeek/README.md) for alert events in [Suricata EVE logs](https://suricata.readthedocs.io/en/suricata-5.0.2/output/eve/eve-json-output.html) (#1400) +* zq: Add a [JSON type definition](docs/integrations/zeek/_index.md) for alert events in [Suricata EVE logs](https://suricata.readthedocs.io/en/suricata-5.0.2/output/eve/eve-json-output.html) (#1400) * zq: Update the [ZNG over JSON (ZJSON)](docs/formats/zjson.md) spec and implementation (#1299) * zar: Use buffered streaming for archive import (#1397) * zq: Add an `ast` command that prints parsed ZQL as its underlying JSON object (#1416) @@ -695,7 +695,7 @@ questions. * zql: Fix broken links in documentation (#1321, #1339) * zst: Introduce the [ZST format](docs/formats/csup.md) for columnar data based on ZNG (#1268, #1338) * pcap: Fix an issue where certain pcapng files could fail import with a `bad option length` error (#1341) -* zql: [Document the `**` operator](docs/language/README.md#search-syntax) for type-specific searches that look within nested records (#1337) +* zql: [Document the `**` operator](docs/language/search-expressions.md) for type-specific searches that look within nested records (#1337) * zar: Change the archive data file layout to prepare for handing chunk files with overlapping ranges and improved S3 support (#1330) * zar: Support archive data files with overlapping time spans (#1348) * zqd: Add a page containing guidance for users that directly access the root `zqd` endpoint in a browser (#1350) @@ -789,7 +789,7 @@ questions. * zql: Add a new function `Time.trunc()` (#842) * zql: Support grouping by computed keys (#860) * zq: Change implementation of `every X` to use a computed groupby key (#893) -* zql: Clean up the [ZQL docs](docs/language/README.md) (#884) +* zql: Clean up the [ZQL docs](docs/language/_index.md) (#884) * zql: Change `cut` operator to emit any matching fields (#899) * zq: Allow output to an S3 bucket (#889) diff --git a/docs/_index.md b/docs/_index.md index ff45bd9c84..081d762cb9 100644 --- a/docs/_index.md +++ b/docs/_index.md @@ -5,7 +5,7 @@ heading: SuperDB --- SuperDB offers a new approach that makes it easier to manipulate and manage -your data. With its [super-structured data model](formats/README.md#2-a-super-structured-pattern), +your data. With its [super-structured data model](formats/_index.md#2-a-super-structured-pattern), messy JSON data can easily be given the fully-typed precision of relational tables without giving up JSON's uncanny ability to represent eclectic data. @@ -26,7 +26,7 @@ while for a technical user, SuperDB exposes its technical underpinnings in a gradual slope, providing as much detail as desired, packaged up in the easy-to-understand [Super JSON data format](formats/jsup.md) and -[SuperPipe language](language/README.md). +[SuperPipe language](language/_index.md). While `super` and its accompanying data formats are production quality, the project's [SuperDB data lake](commands/super-db.md) is a bit [earlier in development](commands/super-db.md#status). @@ -37,12 +37,12 @@ While `super` and its accompanying data formats are production quality, the proj a number of different elements of the system: * The [super data model](formats/zed.md) is the abstract definition of the data types and semantics that underlie the super-structured data formats. -* The [super data formats](formats/README.md) are a family of +* The [super data formats](formats/_index.md) are a family of [human-readable (Super JSON, JSUP)](formats/jsup.md), [sequential (Super Binary, BSUP)](formats/bsup.md), and [columnar (Super Columnar, CSUP)](formats/csup.md) formats that all adhere to the same abstract super data model. -* The [SuperPipe language](language/README.md) is the system's pipeline language for performing +* The [SuperPipe language](language/_index.md) is the system's pipeline language for performing queries, searches, analytics, transformations, or any of the above combined together. * A [SuperPipe query](language/overview.md) is a script that performs search and/or analytics. @@ -57,7 +57,7 @@ accessed via a [Git](https://git-scm.com/)-like API. ## Digging Deeper -The [SuperPipe language documentation](language/README.md) +The [SuperPipe language documentation](language/_index.md) is the best way to learn about `super` in depth. All of its examples use `super` commands run on the command line. Run `super -h` for a list of command options and online help. diff --git a/docs/commands/_index.md b/docs/commands/_index.md index 979c5678ca..f618316376 100644 --- a/docs/commands/_index.md +++ b/docs/commands/_index.md @@ -11,7 +11,7 @@ into, querying, and orchestrating SuperDB data lakes. These sub-commands are organized into further subcommands like the familiar command patterns of `docker` or `kubectl`. -All operations with these commands utilize the [super data model](../formats/README.md) -and provide querying via [SuperSQL](../language/README.md). +All operations with these commands utilize the [super data model](../formats/_index.md) +and provide querying via [SuperSQL](../language/_index.md). Built-in help for `super` and all sub-commands is always accessible with the `-h` flag. diff --git a/docs/commands/super-db.md b/docs/commands/super-db.md index a3bf860615..febae54da4 100644 --- a/docs/commands/super-db.md +++ b/docs/commands/super-db.md @@ -5,7 +5,7 @@ title: super db > **TL;DR** `super db` is a sub-command of `super` to manage and query SuperDB data lakes. > You can import data from a variety of formats and it will automatically -> be committed in [super-structured](../formats/README.md) +> be committed in [super-structured](../formats/_index.md) > format, providing full fidelity of the original format and the ability > to reconstruct the original data without loss of information. > @@ -16,7 +16,7 @@ title: super db

:::tip Status -While [`super`](super.md) and its accompanying [formats](../formats/README.md) +While [`super`](super.md) and its accompanying [formats](../formats/_index.md) are production quality, the SuperDB data lake is still fairly early in development and alpha quality. That said, SuperDB data lakes can be utilized quite effectively at small scale, @@ -657,7 +657,7 @@ according to configured policies and logic. ``` super db query [options] ``` -The `query` command runs a [SuperSQL](../language/README.md) query with data from a lake as input. +The `query` command runs a [SuperSQL](../language/_index.md) query with data from a lake as input. A query typically begins with a [`from` operator](../language/operators/from.md) indicating the pool and branch to use as input. diff --git a/docs/commands/super.md b/docs/commands/super.md index bf6790716a..ed3992ff4f 100644 --- a/docs/commands/super.md +++ b/docs/commands/super.md @@ -3,7 +3,7 @@ weight: 1 title: super --- -> **TL;DR** `super` is a command-line tool that uses [SuperSQL](../language/README.md) +> **TL;DR** `super` is a command-line tool that uses [SuperSQL](../language/_index.md) > to query a variety of data formats in files, over HTTP, or in [S3](../integrations/amazon-s3.md) > storage. Best performance is achieved when operating on data in binary formats such as > [Super Binary](../formats/bsup.md), [Super Columnar](../formats/csup.md), @@ -18,7 +18,7 @@ super [ options ] [ -c query ] input [ input ... ] `super` is a command-line tool for processing data in diverse input formats, providing data wrangling, search, analytics, and extensive transformations -using the [SuperSQL](../language/README.md) dialect of SQL. Any SQL query expression +using the [SuperSQL](../language/_index.md) dialect of SQL. Any SQL query expression may be extended with [pipe syntax](https://research.google/pubs/sql-has-problems-we-can-fix-them-pipe-syntax-in-sql/) to filter, transform, and/or analyze input data. Super's SQL pipes dialect is extensive, so much so that it can resemble @@ -28,10 +28,10 @@ The `super` command works with data from ephemeral sources like files and URLs. If you want to persist your data into a data lake for persistent storage, check out the [`super db`](super-db.md) set of commands. -By invoking the `-c` option, a query expressed in the [SuperSQL language](../language/README.md) +By invoking the `-c` option, a query expressed in the [SuperSQL language](../language/_index.md) may be specified and applied to the input stream. -The [super data model](../formats/zed.md) is based on [super-structured data](../formats/README.md#2-a-super-structured-pattern), meaning that all data +The [super data model](../formats/zed.md) is based on [super-structured data](../formats/_index.md#2-a-super-structured-pattern), meaning that all data is both strongly _and_ dynamically typed and need not conform to a homogeneous schema. The type structure is self-describing so it's easy to daisy-chain queries and inspect data at any point in a complex query or data pipeline. @@ -571,15 +571,15 @@ error("divide by zero") ## Examples -As you may have noticed, many examples of the [SuperSQL language](../language/README.md) +As you may have noticed, many examples of the [SuperSQL language](../language/_index.md) are illustrated using this pattern ``` echo | super -c - ``` -which is used throughout the [language documentation](../language/README.md) -and [operator reference](../language/operators/README.md). +which is used throughout the [language documentation](../language/_index.md) +and [operator reference](../language/operators/_index.md). -The language documentation and [tutorials directory](../tutorials/README.md) +The language documentation and [tutorials directory](../tutorials/_index.md) have many examples, but here are a few more simple `super` use cases. _Hello, world_ @@ -613,7 +613,7 @@ produces <[(int64,string)]> <|[string]|> ``` -_A simple [aggregation](../language/aggregates/README.md)_ +_A simple [aggregation](../language/aggregates/_index.md)_ ```mdtest-command echo '{key:"foo",val:1}{key:"bar",val:2}{key:"foo",val:3}' | super -z -c 'sum(val) by key | sort key' - diff --git a/docs/formats/csup.md b/docs/formats/csup.md index 2630b09a2c..133a6e7322 100644 --- a/docs/formats/csup.md +++ b/docs/formats/csup.md @@ -7,7 +7,7 @@ heading: Super Columnar Specification Super Columnar is a file format based on the [super data model](zed.md) where data is stacked to form columns. Its purpose is to provide for efficient analytics and search over -bounded-length sequences of [super-structured data](./README.md#2-a-super-structured-pattern) that is stored in columnar form. +bounded-length sequences of [super-structured data](./_index.md#2-a-super-structured-pattern) that is stored in columnar form. Like [Parquet](https://github.com/apache/parquet-format), Super Columnar provides an efficient representation for semi-structured data, diff --git a/docs/formats/zjson.md b/docs/formats/zjson.md index 4deb1df9ce..f4e8d506fa 100644 --- a/docs/formats/zjson.md +++ b/docs/formats/zjson.md @@ -60,7 +60,7 @@ Also, it is at the whim of a JSON implementation whether or not the order of object keys is preserved. While JSON is well suited for data exchange of generic information, it is not -sufficient for the [super-structured data model](./README.md#2-a-super-structured-pattern). +sufficient for the [super-structured data model](./_index.md#2-a-super-structured-pattern). That said, JSON can be used as an encoding format for super data with another layer of encoding on top of a JSON-based protocol. This allows clients like web apps or Electron apps to receive and understand Super JSON and, with the help of client diff --git a/docs/integrations/fluentd.md b/docs/integrations/fluentd.md index 492a4ee262..9688930bfe 100644 --- a/docs/integrations/fluentd.md +++ b/docs/integrations/fluentd.md @@ -12,7 +12,7 @@ record for archiving and analytics. This guide walks through two simple configurations of Fluentd with a Zed lake that can be used as reference for starting your own production configuration. As it's a data source important to many in the Zed community, log data from -[Zeek](./zeek/README.md) is used in this guide. The approach shown can be +[Zeek](./zeek/_index.md) is used in this guide. The approach shown can be easily adapted to any log data source. ## Software @@ -305,7 +305,7 @@ Example output: Notice quotes are no longer present around the values that contain IP addresses and times, since they are no longer stored as strings. With the data in this -shaped form, we could now invoke [Zed language](../language/README.md) +shaped form, we could now invoke [Zed language](../language/_index.md) functionality that leverages the richer data typing such as filtering `ip` values by CIDR block, e.g., diff --git a/docs/integrations/zeek/data-type-compatibility.md b/docs/integrations/zeek/data-type-compatibility.md index 6c050b1b50..7de64aac74 100644 --- a/docs/integrations/zeek/data-type-compatibility.md +++ b/docs/integrations/zeek/data-type-compatibility.md @@ -19,7 +19,7 @@ the types that may appear in Zeek logs. Zed tools maintain an internal Zed-typed representation of any Zeek data that is read or imported. Therefore, knowing the equivalent types will prove useful when performing operations in the -[Zed language](../../language/README.md) such as +[Zed language](../../language/_index.md) such as [type casting](../../language/shaping.md#cast) or looking at the data when output as Super JSON. diff --git a/docs/integrations/zeek/reading-zeek-log-formats.md b/docs/integrations/zeek/reading-zeek-log-formats.md index cdbac244cf..8d78a9304f 100644 --- a/docs/integrations/zeek/reading-zeek-log-formats.md +++ b/docs/integrations/zeek/reading-zeek-log-formats.md @@ -5,7 +5,7 @@ title: Reading Zeek Log Formats Zed is capable of reading both common Zeek log formats. This document provides guidance for what to expect when reading logs of these formats using -the Zed [command line tools](../../commands/README.md). +the Zed [command line tools](../../commands/_index.md). ## Zeek TSV diff --git a/docs/lake/format.md b/docs/lake/format.md index 3d30c67064..f223ac2700 100644 --- a/docs/lake/format.md +++ b/docs/lake/format.md @@ -31,7 +31,7 @@ to provide a universal data representation for all of these different approaches Also, while we are not currently focused on building a SQL engine for the Zed lake, it is most certainly possible to do so, as a Zed record type -[is analagous to](../formats/README.md#2-a-super-structured-pattern) +[is analagous to](../formats/_index.md#2-a-super-structured-pattern) a SQL table definition. SQL tables can essentially be dynamically projected via a table virtualization layer built on top of the Zed lake model. diff --git a/docs/language/_index.md b/docs/language/_index.md index cb7d545616..303b399ad1 100644 --- a/docs/language/_index.md +++ b/docs/language/_index.md @@ -12,5 +12,5 @@ The language documents: * describe the syntax of [expressions](expressions.md) and [search expressions](search-expressions.md), * explain [lateral subqueries](lateral-subqueries.md), * describe [shaping and type fusion](shaping.md), and -* enumerate the [operators](operators/README.md), [functions](functions/README.md), -and [aggregate functions](aggregates/README.md) in reference format. +* enumerate the [operators](operators/_index.md), [functions](functions/_index.md), +and [aggregate functions](aggregates/_index.md) in reference format. diff --git a/docs/language/conventions.md b/docs/language/conventions.md index 0359014178..5e9823a520 100644 --- a/docs/language/conventions.md +++ b/docs/language/conventions.md @@ -4,7 +4,7 @@ title: Conventions heading: Type Conventions --- -[Function](functions/README.md) arguments and [operator](operators/README.md) input values are all dynamically typed, +[Function](functions/_index.md) arguments and [operator](operators/_index.md) input values are all dynamically typed, yet certain functions expect certain specific [data types](data-types.md) or classes of data types. To this end, the function and operator prototypes in the Zed documentation include several type classes as follows: diff --git a/docs/language/data-types.md b/docs/language/data-types.md index 9283a74b47..ec789b127a 100644 --- a/docs/language/data-types.md +++ b/docs/language/data-types.md @@ -161,7 +161,7 @@ the scope of the Zed data model and language. That said, Zed provides flexible building blocks so systems can define their own schema versioning and schema management policies on top of these Zed primitives. -The [super-structured data model](../formats/README.md#2-a-super-structured-pattern) +The [super-structured data model](../formats/_index.md#2-a-super-structured-pattern) is a superset of relational tables and SuperPipe's type system can easily make this connection. As an example, consider this type definition for "employee": diff --git a/docs/language/expressions.md b/docs/language/expressions.md index c1059a44e7..47d423de70 100644 --- a/docs/language/expressions.md +++ b/docs/language/expressions.md @@ -240,14 +240,14 @@ produces ``` -Zed includes many [built-in functions](functions/README.md), some of which take +Zed includes many [built-in functions](functions/_index.md), some of which take a variable number of arguments. Zed also allows you to create [user-defined functions](statements.md#func-statements). ## Aggregate Function Calls -[Aggregate functions](aggregates/README.md) may be called within an expression. +[Aggregate functions](aggregates/_index.md) may be called within an expression. Unlike the aggregation context provided by a [summarizing group-by](operators/summarize.md), such calls in expression context yield an output value for each input value. diff --git a/docs/language/lateral-subqueries.md b/docs/language/lateral-subqueries.md index 1b7cdd2a7f..86294eafb9 100644 --- a/docs/language/lateral-subqueries.md +++ b/docs/language/lateral-subqueries.md @@ -187,5 +187,5 @@ produces ``` Similarly, a primitive value may be consistently produced by concluding the lateral scope with an operator such as [`head`](operators/head.md) or -[`tail`](operators/tail.md), or by applying certain [aggregate functions](aggregates/README.md) +[`tail`](operators/tail.md), or by applying certain [aggregate functions](aggregates/_index.md) such as done with [`sum`](aggregates/sum.md) above. diff --git a/docs/language/operators/summarize.md b/docs/language/operators/summarize.md index 6d7f4863fc..3edcc78b2a 100644 --- a/docs/language/operators/summarize.md +++ b/docs/language/operators/summarize.md @@ -14,7 +14,7 @@ ### Description In the first four forms, the `summarize` operator consumes all of its input, -applies an [aggregate function](../aggregates/README.md) to each input value +applies an [aggregate function](../aggregates/_index.md) to each input value optionally filtered by a `where` clause and/or organized with the group-by keys specified after the `by` keyword, and at the end of input produces one or more aggregations for each unique set of group-by key values. diff --git a/docs/language/overview.md b/docs/language/overview.md index 1a9daa3b1a..ff5e1d62ca 100644 --- a/docs/language/overview.md +++ b/docs/language/overview.md @@ -14,7 +14,7 @@ by a number of commands: command |> command | command | ... ``` However, in Zed, the entities that transform data are called -"[operators](operators/README.md)" instead of "commands" and unlike Unix pipelines, +"[operators](operators/_index.md)" instead of "commands" and unlike Unix pipelines, the streams of data in a Zed query are typed data sequences that adhere to the [Zed data model](../formats/zed.md). @@ -119,4 +119,4 @@ The following sections continue describing the Zed language. * [Lateral Subqueries](lateral-subqueries.md) * [Shaping and Type Fusion](shaping.md) -You may also be interested in the detailed reference materials on [operators](operators/README.md), [functions](functions/README.md), and [aggregate functions](aggregates/README.md), as well as the [conventions](conventions.md) for how they're described. +You may also be interested in the detailed reference materials on [operators](operators/_index.md), [functions](functions/_index.md), and [aggregate functions](aggregates/_index.md), as well as the [conventions](conventions.md) for how they're described. diff --git a/docs/language/pipeline-model.md b/docs/language/pipeline-model.md index 5bd3c25c9e..a8e1421558 100644 --- a/docs/language/pipeline-model.md +++ b/docs/language/pipeline-model.md @@ -6,7 +6,7 @@ title: Pipeline Model In SuperPipe, each operator takes its input from the output of its upstream operator beginning either with a data source or with an implied source. -All available operators are listed on the [reference page](operators/README.md). +All available operators are listed on the [reference page](operators/_index.md). ## Pipeline Sources @@ -104,7 +104,7 @@ in parallel on multiple threads). To establish a consistent sequence order, a [`merge` operator](operators/merge.md) may be applied at the output of the switch specifying a sort key upon which to order the upstream data. Often such order does not matter (e.g., when the output -of the switch hits an [aggregator](aggregates/README.md)), in which case it is typically more performant +of the switch hits an [aggregator](aggregates/_index.md)), in which case it is typically more performant to omit the merge (though the SuperDB runtime will often delete such unnecessary operations automatically as part optimizing queries when they are compiled). @@ -134,7 +134,7 @@ produces this case-sensitive output: "bar" "foo" ``` -But we can make the sort case-insensitive by applying a [function](functions/README.md) to the +But we can make the sort case-insensitive by applying a [function](functions/_index.md) to the input values with the expression `lower(this)`, which converts each value to lower-case for use in in the sort without actually modifying the input value, e.g., diff --git a/docs/language/search-expressions.md b/docs/language/search-expressions.md index 810f79feee..527d8f7e57 100644 --- a/docs/language/search-expressions.md +++ b/docs/language/search-expressions.md @@ -276,7 +276,7 @@ the "in" operator, e.g., #### Predicate Search Term -Any Boolean-valued [function](functions/README.md) like `is`, `has`, +Any Boolean-valued [function](functions/_index.md) like `is`, `has`, `grep`, etc. and any [comparison expression](expressions.md#comparisons) may be used as a search term and mixed into a search expression. diff --git a/docs/language/shaping.md b/docs/language/shaping.md index f29fb882a7..ed8a2fa033 100644 --- a/docs/language/shaping.md +++ b/docs/language/shaping.md @@ -12,7 +12,7 @@ a well-defined set of schemas, which combines the data into a unified store like a data warehouse. In Zed, this cleansing process is called "shaping" the data, and Zed leverages -its rich, [super-structured](../formats/README.md#2-a-super-structured-pattern) +its rich, [super-structured](../formats/_index.md#2-a-super-structured-pattern) type system to perform core aspects of data transformation. In a data model with nesting and multiple scalar types (such as Zed or JSON), shaping includes converting the type of leaf fields, adding or removing fields diff --git a/docs/tutorials/zed.md b/docs/tutorials/zed.md index bb6151f667..2fa2015b8b 100644 --- a/docs/tutorials/zed.md +++ b/docs/tutorials/zed.md @@ -318,7 +318,7 @@ the `zed` command. Some suggested next steps: 1. Dig deeper into SuperDB data lakes by having a look at the [`super db` command](../commands/super-db.md) documentation. 2. Get a better idea of ways you can query your data by looking at the -[Zed language documentation](../language/README.md). +[Zed language documentation](../language/_index.md). If you have any questions or run into any snags, join the friendly Zed community at the [Brim Data Slack workspace](https://www.brimdata.io/join-slack/). diff --git a/docs/tutorials/zq.md b/docs/tutorials/zq.md index f7b030491c..6561d09b77 100644 --- a/docs/tutorials/zq.md +++ b/docs/tutorials/zq.md @@ -5,10 +5,10 @@ heading: super Tutorial --- This tour provides new users of `super` an overview of the tool and -the [SuperPipe language](../language/README.md) +the [SuperPipe language](../language/_index.md) by walking through a number of examples on the command-line. This should get you started without having to read through all the gory details -of the [SuperPipe language](../language/README.md) or [`super` command-line usage](../commands/super.md). +of the [SuperPipe language](../language/_index.md) or [`super` command-line usage](../commands/super.md). We'll start with some simple one-liners on the command line where we feed some data to `super` with `echo` and specify `-` for `super` input to indicate @@ -410,7 +410,7 @@ preparation, union types are really quite powerful. They allow records with fields of different types or mixed-type arrays to be easily expressed while also having a very precise type definition. This is the essence of Zed's new -[super-structured data model](../formats/README.md#2-a-super-structured-pattern). +[super-structured data model](../formats/_index.md#2-a-super-structured-pattern). ## First-class Types @@ -1253,6 +1253,6 @@ of tricks to: clean data for analysis by `zq` or even export into other systems or for testing. If you'd like to learn more, feel free to read through the -[language docs](../language/README.md) in depth +[language docs](../language/_index.md) in depth or see how you can organize [data into a lake](../commands/super-db.md) using a git-like commit model.