Flatten ?filters=(...)&labels=(...) to ?f=...&f=...&l=...&l=... #4810

apata · 2024-11-12T12:04:14Z

Changes

Moves current URL params parsing logic to legacy-jsonurl-url-search-params.ts
Adds custom serialization logic for labels and filters in url-search-params.ts and encodeURIComponent based logic for other simple params.
Adds redirect logic from jsonurl search params to new search params.

Tests

Automated tests have been added

Changelog

Entry has been added to changelog

Documentation

This change does not need a documentation update

Dark mode

This PR does not change the UI

github-actions · 2024-11-12T12:04:41Z

Preview environment👷🏼‍♀️🏗️
PR-4810

ruslandoga · 2024-11-20T12:00:54Z

👋

Sorry for a drive-by comment on a draft PR, but if you set the attribute key to f[]= then it would get automatically decoded into a list in Plug / Phoenix. Otherwise a custom decoder for query strings would need to be implemented.

More info: https://hexdocs.pm/plug/Plug.Conn.Query.html

apata · 2024-12-11T15:24:20Z

👋

Sorry for a drive-by comment on a draft PR, but if you set the attribute key to f[]= then it would get automatically decoded into a list in Plug / Phoenix. Otherwise a custom decoder for query strings would need to be implemented.

More info: https://hexdocs.pm/plug/Plug.Conn.Query.html

Thanks for bringing it up @ruslandoga ! At the moment, this format must be recognizable only to the FE and ideally humans, though. I think of it as URL-stored dashboard state.

Currently, the BE receives applied dashboard filters in URL-encoded stringified JSON, and there's an extra transform taking place https://github.com/plausible/analytics/blob/master/assets/js/dashboard/util/filters.js#L231 before that, which we're not planning to get rid of (in the interest of readable URLs).

ukutaht · 2025-01-02T10:26:24Z

The idea with the new filters format was to also support nesting in case we need it in the future. Stuff like:

[
  ["or", [
    ["and", [
      ["is", "visit:source",  "Google", {case_sensitive: false}],
      ["is", "visit:utm_medium", "is", "cpc"]
    ]],
    ["and", [
      ["is", "visit:source", "Facebook", {case_sensitive: false}],
      ["is", "visit:utm_medium", "paid_social"]
    ]]
  ]]
]

This format seems to make that a bit harder. Did you consider that as well @apata ?

apata · 2025-01-06T09:08:11Z

Good question, @ukutaht! I expect there to be a bunch of ways to do this because we know the key words: and, or, not, all operators and all operator options.

[
  [
    "or",
    [
      [
        "and",
        [
          [
            "is",
            "visit:source",
            ["Google", "Bing"],
            { "case_sensitive": false }
          ],
          ["is", "visit:utm_medium", "is", ["cpc"]]
        ]
      ],
      [
        "and",
        [
          ["is", "visit:source", ["Facebook"], { "case_sensitive": false }],
          ["is", "visit:utm_medium", ["paid_social"]]
        ]
      ]
    ]
  ],
  ["not", ["is", "visit:country", ["Estonia"]]],
  ["contains", "event:goal", ["trial"], { "case_sensitive": false }]
]

For this edgy filter tree, an URL in the new style might be

?f=or/2&f=and/2&f=is/i,source,Google,Bing&f=is,utm_medium,cpc&f=and/2&f=is/i,source,Facebook&f=is,utm_medium,paid_social&f=not/1&f=is,country,Estonia&f=contains/i,goal,trial

This would be reconstructable thanks to fixed order of f= in the URL and known group lengths.

// url.split("&").map(i => i.split("=")).filter(i => i[0] === "f").map(i => i[1]) -->
[
  "or/2",
  "and/2",
  "is/i,source,Google,Bing",
  "is,utm_medium,cpc",
  "and/2",
  "is/i,source,Facebook",
  "is,utm_medium,paid_social",
  "not/1",
  "is,country,Estonia",
  "contains/i,goal,trial",
]
// reconstruct recursively

ukutaht · 2025-01-07T15:04:20Z

Cool - the format makes sense to me. While it does diverge a bit form what we initially designed, and comes with the annoyance of managing yet another backwards compatibility redirect, I think it's probably worth it in order to have URLs that can be copy-pasted easily.

ukutaht · 2025-01-07T15:05:42Z

Would be good to get another review/opinion - @aerosol @zoldar @macobo thoughts on this?

assets/js/dashboard/util/url-search-params.ts

macobo · 2025-01-09T12:22:57Z

assets/js/dashboard/util/url-search-params.ts

+  const serializedFilter = [
+    encodeURIComponentPermissive(operator, ':/'),
+    encodeURIComponentPermissive(dimension, ':/'),
+    ...clauses.map((clause) =>


We recently added support for case insensitivity in APIv2 via modifiers (plausible/docs#567).
While we haven't exposed this in the frontend, it's something we should do once we figure out the UX. I'm not sure whether this function is extendable to allow for that structure though.

Note that in general modifiers are not always flags and may contain arbitrary data structures within.

I believe it is: #4810 (comment)

Note that in general modifiers are not always flags and may contain arbitrary data structures within.

The filter modifiers and structures we support is up to us, so I'm not clear on what you mean by arbitrary.

That said, I can imagine something like the following working:

-> `is\non_flag_modifier_name:${encodeURIComponent(JSON.stringify(complex_structure))},<dimension>,<clause1>,<clause2>`

That's stretching the limits of readability though.

I'm not clear on what you mean by arbitrary.

@macobo and I were just bouncing around some ideas about how to extend filtering capabilities in the future which would utilize more modifiers. Note that these new modifiers will not necessarily be built in the short term but having the ability to extend filters with this kind of stuff in the future would be really useful.

Arbitrary date range example:

More flags:

See this discussion: https://3.basecamp.com/5308029/buckets/26383192/card_tables/cards/7793417773

As we were discussing it did cross my mind that this may not work well with the proposed URL structure here.,,

Thanks for the extra examples! What makes these modifiers arbitrary?

We know what keys to look for, and even what the possible values can be.

As I see it, filter clauses are the only arbitrary value currently: their values are unpredictable to us, and may overlap with any of our option names or option configurations.

That's why in this solution, they are expanded to be the last members of the comma separated array: f=<known word at known position>/<optional known words and values>,<known word at known position>,...unpredictable words...

However, from my side, I'm not married to this solution. There's various others that are totally unexplored at the moment, like enriching jsonurl etc. It's just that "working well" will be hard to solve for. Short, easy to implement, linkable, readable are some of the things that I optimized for. I personally think expandable is also one of the qualities, but I see that doubt remains.

I think something analogous to what you proposed in #4810 (comment) should work for the original question (how to handle more complex data structures in filter modifiers and should unblock us for now.

assets/js/dashboard/util/url-search-params.ts

macobo

Behaviorally this is great and I appreciate the extensive tests!

I didn't do extensive testing on the backwards redirect.

I also had some minor suggestions around code structure to avoid the backwards redirects causing issues in the future. Let me know if you'd prefer me to push to the branch to implement the suggestions.

macobo · 2025-01-13T09:56:15Z

assets/js/dashboard/query.ts

-) {
-  const searchRecord = parseSearch(windowLocation.search)
-  const getValue = (k: string) => searchRecord[k]
+export function maybeGetRedirectTargetFromLegacySearchParams(


Let's add a comment detailing the history of this above it - this will help future readers of the code.

Suggested change

export function maybeGetRedirectTargetFromLegacySearchParams(

/*

Calculates whether to redirect from legacy url format(s) to new formats.

We serialize dashboard filters in the URL and maintain old URLs for backwards compatibility reasons to not break old links.

There have been two previous formats:

1. legacy-searchparams - a custom encoding schema was used for filters. This was deemed not flexible enough and diverged from how we represented filters in the code.

2. legacy-jsonurl - jsonurl was used to encode the more complex filter structure via json url. However the links generated via this didn't always work across all platforms (e.g. Twitter).

*/

export function maybeGetRedirectTargetFromLegacySearchParams(

Good idea. Added with slight modifications.

macobo · 2025-01-13T10:01:19Z

assets/js/dashboard/query.ts

-  const getValue = (k: string) => searchRecord[k]
+export function maybeGetRedirectTargetFromLegacySearchParams(
+  windowLocation: Location
+): null | string {


I was struggling reviewing this function. I think it's because the original 'legacy' format and the new 'legacy' format are both called legacy in this, and the code is structured differently, causing friction.

I suggest:

naming the formats: legacy-jsonurl and legacy-searchparams

moving parseLegacyFilter and parseLegacyPropsFilter and parseLegacyFilter to legacy-searchparams.ts

Splitting this function into two parts, function function redirectTarget() and putting them in legacy-searchparams.ts and legacy-jsonurl-....

I think this would increase consistency and simplify keeping this working if/as we iterate this further.

I agree that what you propose is better, but since this the first redirect wasn't properly tested, I opted against it originally: in review, any slight breaking changes could have slipped past. I've now split it out though.

macobo · 2025-01-13T10:03:15Z

assets/js/dashboard/util/legacy-jsonurl-url-search-params.ts

@@ -0,0 +1,80 @@
+/* @format */


I am generally not a fan of utils - they tend to become grab bags of misc code. Given the behavior becomes also more integrated with the rest of the app code, I think some thought is warranted.

How would you feel about moving `url-search-params and the legacy files under assets/js/dashboard/navigation/ or creating a new folder for them?

You're right that utils can get messy, but I think I'd prefer it solved in a separate refactor that tackles code organization only.

There's relations between url, api, url search params, query, filters that our current code organization isn't fully capturing. Just now I had to solve some circular import dependency issue.

apata added the preview label Nov 12, 2024

apata changed the title ~~WIP: flatten ?filters=(...) to ?f=...&f=...~~ [WIP] Flatten ?filters=(...) to ?f=...&f=... Dec 2, 2024

apata force-pushed the fix-links-by-flattening branch from 0f16e70 to caf00fe Compare December 11, 2024 15:07

apata force-pushed the fix-links-by-flattening branch 2 times, most recently from 6f83914 to cd6b053 Compare December 11, 2024 16:20

apata changed the title ~~[WIP] Flatten ?filters=(...) to ?f=...&f=...~~ Flatten ?filters=(...)&labels=(...) to ?f=...&f=...&l=...&l=... Dec 11, 2024

ukutaht approved these changes Jan 7, 2025

View reviewed changes

macobo reviewed Jan 9, 2025

View reviewed changes

assets/js/dashboard/util/url-search-params.ts Show resolved Hide resolved

macobo reviewed Jan 9, 2025

View reviewed changes

assets/js/dashboard/util/url-search-params.ts Outdated Show resolved Hide resolved

apata force-pushed the fix-links-by-flattening branch from c4e5e89 to 7df16a1 Compare January 9, 2025 13:02

macobo reviewed Jan 13, 2025

View reviewed changes

apata force-pushed the fix-links-by-flattening branch from a85f8e6 to 33dc1fd Compare January 14, 2025 10:21

apata added 6 commits January 14, 2025 12:22

Serialize filters and labels as ?f=is,page,/a,/b&f=...

55964a9

Update changelog

bdcfc32

Refactor not to use URLSearchParams, because it led to double decoding

657810c

Handle empty search string and badly formed search strings

2bea9cc

Declare repeating characters as constant

97d72a9

Introduce version names

c20fbd3

apata force-pushed the fix-links-by-flattening branch from 33dc1fd to c20fbd3 Compare January 14, 2025 10:22

macobo approved these changes Jan 14, 2025

View reviewed changes

apata added this pull request to the merge queue Jan 16, 2025

Merged via the queue into master with commit 90a2c5d Jan 16, 2025
8 checks passed

apata mentioned this pull request Jan 16, 2025

Excluding goals in dashboard #4983

Merged

apata deleted the fix-links-by-flattening branch January 23, 2025 13:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flatten ?filters=(...)&labels=(...) to ?f=...&f=...&l=...&l=... #4810

Flatten ?filters=(...)&labels=(...) to ?f=...&f=...&l=...&l=... #4810

apata commented Nov 12, 2024 •

edited

Loading

github-actions bot commented Nov 12, 2024

ruslandoga commented Nov 20, 2024 •

edited

Loading

apata commented Dec 11, 2024

ukutaht commented Jan 2, 2025

apata commented Jan 6, 2025

ukutaht commented Jan 7, 2025

ukutaht commented Jan 7, 2025

macobo Jan 9, 2025 •

edited

Loading

apata Jan 9, 2025

apata Jan 9, 2025

ukutaht Jan 9, 2025

apata Jan 10, 2025

macobo Jan 12, 2025

macobo left a comment

macobo Jan 13, 2025

apata Jan 14, 2025

macobo Jan 13, 2025

apata Jan 14, 2025 •

edited

Loading

macobo Jan 13, 2025

apata Jan 14, 2025

-export function maybeGetRedirectTargetFromLegacySearchParams(
+/*
+Calculates whether to redirect from legacy url format(s) to new formats.
+We serialize dashboard filters in the URL and maintain old URLs for backwards compatibility reasons to not break old links.
+There have been two previous formats:
+. legacy-searchparams - a custom encoding schema was used for filters. This was deemed not flexible enough and diverged from how we represented filters in the code.
+. legacy-jsonurl - jsonurl was used to encode the more complex filter structure via json url. However the links generated via this didn't always work across all platforms (e.g. Twitter).
+*/
+export function maybeGetRedirectTargetFromLegacySearchParams(

Flatten ?filters=(...)&labels=(...) to ?f=...&f=...&l=...&l=... #4810

Flatten ?filters=(...)&labels=(...) to ?f=...&f=...&l=...&l=... #4810

Conversation

apata commented Nov 12, 2024 • edited Loading

Changes

Tests

Changelog

Documentation

Dark mode

github-actions bot commented Nov 12, 2024

ruslandoga commented Nov 20, 2024 • edited Loading

apata commented Dec 11, 2024

ukutaht commented Jan 2, 2025

apata commented Jan 6, 2025

ukutaht commented Jan 7, 2025

ukutaht commented Jan 7, 2025

macobo Jan 9, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

macobo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apata Jan 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apata commented Nov 12, 2024 •

edited

Loading

ruslandoga commented Nov 20, 2024 •

edited

Loading

macobo Jan 9, 2025 •

edited

Loading

apata Jan 14, 2025 •

edited

Loading