Query Performance

[zachstence]

Description

While talking with @ItsMeBrianD about the new Pivot Table component, he raised some concerns about the performance of queries and using pagination to (ideally) make them more performant. The way Query is currently written, pagination would create many queries and might not destroy/free memory when they are unused. This could make the Pivot Table sluggish.

These ideas would benefit Pivot Table performance in particular, but also the whole app in general. They are not a pre-requisite to the Pivot Table, but the performance may be poor without these. Time will tell.

• Mutable queries
  • Don't return new query from .where(), .limit(), etc - limits object creation/destruction
  • Violates a lot of assumptions
    • Components that accept a query shouldn't modify the query because it could be used elsewhere (maybe fixed with a .copy() method?)
    • Currently, query state never goes backwards (e.g. once a query is done, it can never be in progress again)
• Make sure we are loading data lazily
  • length and describe queries should happen on Query creation
  • data query should wait until the value of the query is used
  • This would have a big impact on long chains of query methods (.where().skip().limit()) - avoid executing middle queries
• Better cleanup of unused queries to reduce memory overhead
  • Wouldn't do this and mutable queries, they're mutually exclusive

What problem would this solve?

No response

How should it work?

No response

[ItsMeBrianD]

Mutable queries may make #1803 easier to implement / work with