Update docs for lateral subqueries and over operator #5264
Conversation
docs/language/lateral-subqueries.md
Outdated
| :::tip | ||
| The parentheses disambiguate a lateral expression from a lateral | ||
| dataflow operator. | ||
| ::: |
There was a problem hiding this comment.
I'd appreciate some input on this text. All I've done with it thus far is reformat it. However, as a user/reader, I struggle to understand what it's trying to clarify. I'd be in favor of expanding the text a bit to make it clearer or drop it until some user's confusion reminds us why we put it there. 😄
There was a problem hiding this comment.
Update: I've spent more time staring at it and I think I at least understand what it's trying to say, but I'm still struggling for the text to propose instead. I think the point being made is that the open parenthesis appearing before the keyword over is what makes it clear that we're in expression context and not dataflow context.
The reason the text was bothering me is because, assuming a new user reading these docs for the first time, we've just showered them with a bunch of related terms on this page: Lateral Subquery, Lateral Scope, Lateral Expression... and it feels like in this one sentence we cooked up "lateral dataflow operator" as yet another, but unlike the others, it's not a specific term that's defined in some place we can link to.
I think I'm still leaning toward just taking it out. I'm feeling like the overall Lateral Subquery topic is one of those growth points as a user learning Zed where there's no shortcuts and they just need to closely read the docs, hack with the examples, apply it to their own use cases, and when they come out the other side it'll hopefully have stuck. And I feel like this particular point about the parens comes through already in the text without this additional note. But I'm open to other thoughts!
There was a problem hiding this comment.
I think this should stay and is fine as is. Rather than adding more text, how about just making "lateral dataflow operator" into a link to the over operator page?
| * a map value generates a sequence of records of the form `{key:<key>,value:<value>}` for each | ||
| entry in the map, and | ||
| * all other values generate a single value equal to itself. | ||
|
|
||
| Records can be converted to maps with the [_flatten_ function](../functions/flatten.md) | ||
| Records can be converted to maps with the [`flatten` function](../functions/flatten.md) |
There was a problem hiding this comment.
Records can be converted to maps...
Strictly speaking, it doesn't seem like flatten converts them to maps. Rather, it generates a sequence of values similar to what over does with maps, but even then, there's still the difference that flatten creates an array for the path of the record keys whereas over on a map outputs the keys as is. Does this bother anyone? It doesn't quite bother me enough to insist on a major expansion of this text. However, user confusion about maps has come up a few times (e.g., the subtle difference between true Zed map types and what I've been calling "JSON-compatible pseudo-maps" (#4565)) which is why I perked up when I saw a possible oversimplified reference to "maps" here.
Anyone have reactions?
Co-authored-by: Noah Treuhaft <[email protected]>
What's Changing
The docs for Lateral Subqueries and the
overoperator are revised to clarify some subtleties revealed during a recent internal discussion. I made other changes along the way such as adding hyperlinks, reducing redundancy, and addressing conflicting text.If you would like to review the content in rendered form, I've published it on a personal staging site at:
Why
The internal discussion was about Lateral Expressions and when they return primitive values or arrays. The discussion created enough debate that once we reached consensus it was clear this needed to be divulged to users.
Details
As I was adding the new text in the Lateral Expressions section, I reviewed the other text at the overall Lateral Subqueries page as well as the related
overdocs and saw many things that could stand to be polished up. For starters, there was no hyperlinking from theoverpage back to the Lateral Subqueries page, which seemed like a big exposure since subtleties like the one covered in this new text represent things a new user ofovermight need to know.I also noticed that there was some text in the
overpage that effectively described Lateral Subqueries, but some of that text used language that conflicted with what's at the Lateral Subqueries page. Rather than end up with redundant text that would need to be kept in sync in future revisions, I took the step of trimming theoverpage to narrow its scope (moving some relevant text that wasn't previously covered at the Lateral Subqueries page) and relying on repeated hyperlinking to the Lateral Subqueries page to ensure the user knows where they can find the additional info.It would be great if I could get an approval on brimdata/zed-docs-site#83 before this merges so we can be spared yet another round of "broken link checker failures" due to a known issue.