Set operations

[philrz]

tl;dr

A user recently asked in a community Slack thread:

does zq have set operations? like union, but I want to compare two sets/arrays, and return values uniq to only one of them … that sort of thing

Right now this can only be done by combining language building blocks, but users could indeed benefit from some purpose-built functionality for the set type.

Details

At the time this issue is being filed, super is at commit a32651f.

The super data model has its set type and the language has some basic functionality that leverages them such as in and the union aggregate function. However, for something like what the user asked above, they'd currently need to combine language building blocks. The user's specific example was:

I’m looking for something like a |[1,2,3]| - |[2,5]| => |[1,3]|

One approach I provided:

$ super -version
Version: v1.18.0-192-ga32651f7

$ echo '{a:|[1,2,3]|,b:|[2,5]|}' | super -c 'over [...a,...b] | count() by val:=this | count < 2 | union(val)' -

|[1,3,5]|

That actually included the 5 in the output because 5 was was "unique to only one of them". If they actually wanted isolate only values that were "in the first but not the second" then, then I proposed:

$ echo '{a:|[1,2,3]|,b:|[2,5]|}' | zq '
fork (
  => over a | yield {left:this}
  => over b | yield {right:this}
) | anti join on left=right
| union(left)' -

|[1,3]|

All that said, asking users to piece this together each time they need it is asking a lot. @mccanne pointed out that purpose-built set functions/operators could indeed be handy. Short of building Go code to put this in the core of the language, once we have modules working (#2599) perhaps one of us could publish some user-defined functions/operators that use the building blocks like above.