fix: Get allowlist answers from both breadcrumb data and answers

[DafyddLlyr]

Follow up from #2835 following comments from @Mike-Heneghan

What does this PR do?

• Captures data from breadcrumbs (e.g. drawBoundary.action)
• Captures answers from breadcrumbs (e.g. proposal.projectType)
• Both values for allow_list_answers column will be in the format [ {nodeFn} : answers ]
  • This matches allow_list_answers for session summary table
  • This is required if any components could set multiple allow list values
  • This is required where the fn field of the component won't match the variable used in the allow list (e.g. drawBoundary.userAction which is set automatically by the component)
  • This might cause some issues with existing captures allow list answers - open to ideas and comments!

I'd love to get some tests written for analytics as my previous PR inadvertently caused a breaking regression which was missed. I think this should be fairly stable going forward so let's wait and see on this one, but it's likely a good idea before the next set of potentially breaking changes 🤞

[github-actions]

Removed vultr server and associated DNS entries

[Mike-Heneghan]

This is an awesome solution @DafyddLlyr 🥳

I think there's just a small tweak required to get the output from getAnswers and getData to match.

I also tried adding some variables to the ALLOW_LIST which are all set by the same component and it worked as anticipated handling setting multiple keys 🙌

I think it's out of scope of this but I think the recent work has shown that node_fn isn't a valuable column although I think it could be useful at some point to add a ticket to rename it to passport_variables and maybe set it? Could be really useful for debugging automations etc.

[Mike-Heneghan]

One other thing which might be worth checking if how this change might impact the analytics_summary view?

[Mike-Heneghan]

Thanks for fixing this @DafyddLlyr

Testing locally this commit has done the trick 🥳

[Mike-Heneghan]

Having a brief look at the analytics_logs I think as the data structure there'll now be a mismatch between current and old data i.e. array of values vs array of objects:

Although seeing as we haven't been tracking allow_list_list I wonder if we can just crack on?

Otherwise I think it would be possible to update historical data using the node_fn value as the key 🤔

[DafyddLlyr]

Yep - good point. First thought would be to ignore this as it's a new column, however having data in different formats in the column would prove tricky to query so we might as well do this properly!

There's more logs populated than expected already on production, so I think clearing older logs is not really a solution here.

I've tested the following SQL locally which will convert the column. Could you please review and manually test the SQL below @Mike-Heneghan before we run on production (once this change has gone to prod).

To select rows with the "old" data format -

SELECT
    id,
    node_fn,
    node_type,
    allow_list_answers
FROM
    analytics_logs
WHERE
    jsonb_typeof(allow_list_answers) = 'array'
    AND jsonb_typeof(allow_list_answers -> 0) = 'string';

And then this to update "old" → "new" -

UPDATE analytics_logs
SET allow_list_answers = (
    SELECT jsonb_agg(jsonb_build_object(key, value))
    FROM (
        SELECT 
            node_fn AS key, 
            jsonb_array_elements_text(allow_list_answers) AS value
        FROM analytics_logs
        WHERE jsonb_typeof(allow_list_answers) = 'array' AND 
              jsonb_typeof(allow_list_answers->0) = 'string'
    ) AS subquery
)
WHERE jsonb_typeof(allow_list_answers) = 'array' AND 
      jsonb_typeof(allow_list_answers->0) = 'string';

[Mike-Heneghan]

Hey Daf, I think there's something not quite right based on my local testing.

The SELECT works as expected:

Although fetching the same records again post UPDATE:

It appears it doesn't group the values against the key and also looks like every record has become an amalgamation of the rest?

[DafyddLlyr]

Ha! That doesn't match my results - I maybe just had a simple use case 😅 Let me take another look!

[Mike-Heneghan]

No worries! This sort of stuff is really fiddly, please let me know when you want me to retest 👍

[Mike-Heneghan]

It seems like this maybe has the same issue based on my testing 😅

[DafyddLlyr]

Ah I get it sorry - I'm just testing on a single record 🤦

Probably just needs and id=id clause. Taking a look and thanks for testing!

[DafyddLlyr]

Hey sorry for the slow turnaround here @Mike-Heneghan

Clause added and highlighted with comment -

UPDATE analytics_logs
SET allow_list_answers = (
    SELECT jsonb_agg(jsonb_build_object(node_fn, subquery.values))
    FROM (
        SELECT jsonb_agg(value) AS values
        FROM (
            SELECT 
              node_fn,
              jsonb_array_elements_text(allow_list_answers) AS value
            FROM analytics_logs inner_log
            WHERE 
              -- New clause to match ids
              inner_log.id = analytics_logs.id AND
              jsonb_typeof(inner_log.allow_list_answers) = 'array' AND 
              jsonb_typeof(inner_log.allow_list_answers->0) = 'string'
        ) AS subquery_inner
    ) AS subquery
)
WHERE 
  jsonb_typeof(allow_list_answers) = 'array' AND 
  jsonb_typeof(allow_list_answers->0) = 'string';

Getting expected results here, but as always would love another pair of eyes 😄

[Mike-Heneghan]

Working as expected for me ✅

I regret not adding an order by to my query used to check results 😅

Although, comparing the updates are made as expected for me and it doesn't affect the data already in the new structure 🥳

Before:

After:

[DafyddLlyr]

Ah perfect - honestly thanks again for being so on-point with testing here, really helped! 🙌