Zed offers a new approach to data that makes it easier to manipulate and manage your data.
With Zed's new super-structured data model, messy JSON data can easily be given the fully-typed precision of relational tables without giving up JSON's uncanny ability to represent eclectic data.
Trying out Zed is easy: just install
the command-line tool zq.
zq is a lot like jq
but is built from the ground up as a search and analytics engine based
on the Zed data model.
Since Zed data is a proper superset of JSON, zq also works natively with JSON.
While zq and the Zed data formats are production quality, the Zed project's
Zed data lake
is a bit earlier in development.
For a non-technical user, Zed is as easy to use as web search while for a technical user, Zed exposes its technical underpinnings in a gradual slope, providing as much detail as desired, packaged up in the easy-to-understand ZSON data format and Zed language.
We think data is hard and it should be much, much easier.
While schemas are a great way to model and organize your data, they often get in the way when you are just trying to store or transmit your semi-structured data.
Also, why should you have to set up one system for search and another completely different system for historical analytics? And the same unified search/analytics system that works at cloud scale should run easily as a lightweight command-line tool on your laptop.
And rather than having to set up complex ETL pipelines with brittle
transformation logic, managing your data lake should be as easy as
git.
Finally, we believe a lightweight data store that provides easy search and analytics would be a great place to store data sets for data science and data engineering experiments running in Python and providing easy integration with your favorite Python libraries.
Zed solves all these problems with a new foundational data format called ZSON, which is a superset of JSON and the relational models. ZSON is syntax-compatible with JSON but it has a comprehensive type system that you can use as little or as much as you like. Zed types can be used as schemas.
The Zed language offers a gentle learning curve, which spans the gamut from simple keyword search to powerful data-transformation operators like lateral sub-queries and shaping.
Zed also has a cloud-based object design that was modeled after
the git design pattern. Commits to the lake are transactional
and consistent. Search index updates are also transactionally
consistent with any ingested data, and searches can run with or
without indexes.
Check out the installation page for a quick and easy install.
Detailed documentation for the entire Zed system and language is available on the Zed docs site.
The Brim app is an electron-based desktop app to explore, query, and shape data in your Zed lake.
We originally developed Brim for security-oriented use cases (having tight integration with Zeek, Suricata, and Wireshark), but we are actively extending Brim with UX for handling generic data sets to support data science, data engineering, and ETL use cases.
See the contributing guide on how you can help improve Zed!
Join our public Slack workspace for announcements, Q&A, and to trade tips!
We modeled this README after
Philip O'Toole's brilliantly succinct
description of rqlite.