feat: track user resetting flow #2307

Mike-Heneghan · 2023-10-13T15:00:35Z

What

Add a new mutation to update the flow_direction of an analytics_log to a new type reset.
Call this mutation on confirming a reset when clicking the reset icon in the header.
Call this mutation when a user clicks the "reset" of a Notice with a reset.
As per:
- https://trello.com/c/spEYlVk0/2566-track-which-node-users-are-on-when-they-click-to-restart-application-or-start-
- https://trello.com/c/G1bTFyVm/2564-track-notice-reset-status-in-node-metadata

Why

{flow_direction: "reset"} will show when a user has decided to restart.
We can use this to infer data about user behaviour such as how often users are hitting dead ends or from which nodes they're deciding to restart.

Screen Recording

Shows:

Change to reset when resetting via the header.
No change on regular Notice button press.
Change on reset Notice button press.

Screen.Recording.2023-10-13.at.15.49.12.mov

Supersedes

Note

This still shows the bug that when restarting from a notice the first node re-renders unnecessarily.

- Update `analytics_log` record for node which was reset on reset confirmation - Change flow_direction to "reset" to store this user action - Ammend update permission to allow flow_direction to be changed

- On click of a reset button on a notice update analytics_log to `{flow_direction: "reset"}`

github-actions · 2023-10-13T15:01:11Z

🤖 Hasura Change Summary compared a subset of table metadata including permissions:

Updated Tables (1)

public.analytics_logs permissions:

	insert	select	update	delete
public			^➕/_➖

1 added column permissions

	insert	select	update
public			➕ flow_direction

github-actions · 2023-10-13T16:25:37Z

Removed vultr server and associated DNS entries

Mike-Heneghan · 2023-10-13T16:26:37Z

editor.planx.uk/src/pages/FlowEditor/lib/analyticsProvider.tsx

@@ -199,6 +202,31 @@ export const AnalyticsProvider: React.FC<{ children: React.ReactNode }> = ({
    }
  }

+  // Comment to trigger new pizza build.


Vultr isn't behaving itself still 😅

I logged in and killed the instances and DNS records for this PR i.e. 2307 but it seems to be taking a very long time.

I'll remove this comment and troubleshoot further next week.

👀 pizza is looking okay now to me, is this resolved or do you have any outstanding vultr bugs/questions?

I think it's resolved 😅

The situation was:

Pizza failed to build

I deleted the instances and DNS records for it on Vultr

The pizza built but the vultr health check failed.

I pushed a commit

The vultr health check passed 🤷

I'm pretty confident the PR behaves as expected.

If you're happy to review @jessicamcinchak I'll merge / update as per feedback 👍

A couple tips for future Vultru debugging:

If only the healthcheck fails, Vultr might just be slow to warm up - wait a couple minutes and try re-running just the failed GH action job to save yourself commits (it usually passes on the next go!)

Vultr fails to build entirely for a couple common reasons: timeouts (usually around 10m mark), commits pushed close together and an "update" trys to run before the "create" is finished. In these cases, again first trying to re-run just the failed GH action job before doing a full nuke can save a lot of time!

jessicamcinchak

Working as expected for me and database structure looks good 👍

Super easy to understand, really nice analytics-value to number-of-line code change ratio here I think 🙂

editor.planx.uk/src/@planx/components/Notice/Public.tsx

hasura.planx.uk/metadata/tables.yaml

Mike-Heneghan · 2023-10-16T13:42:59Z

editor.planx.uk/src/pages/FlowEditor/lib/analyticsProvider.tsx

@@ -6,7 +6,7 @@ import React, { createContext, useContext, useEffect, useState } from "react";

 import { Store, useStore } from "./store";

-export type AnalyticsType = "init" | "resume";
+export type AnalyticsType = "init" | "resume" | "reset"


Hey @jessicamcinchak something I hadn't realised is that this AnalyticsType is related to the analytics model which tracks the overarching analytics session? It stores whether the user's session was an init or a resume?

I believe the log direction only applies to analytics_log model instances which are tied together with an instance of the analytics model?

Based on the way the "reset" is being recorded against an analytics_log if it make more sense to think of it as a log direction i.e. an extreme case of going backwards where you go back to the the start?

I'm not entirely sure I'm following this one yet, but my instinct here is that "reset" ends the given session and therefore, at least in my mind, it's an analytics type rather than pure "direction" (eg directions like "forwards" and "backwards" are always within a session). But also totally trust your judgement on this!

I think that's a really good point although in this PR the "reset" is only ever applied within a session as a flow_direction

As in:

The last analytics_log for that analytic session would have a flow_direction of "reset".

The type of the aforementioned analytic session would remain whatever type it had been i.e. "init" or "resume"

The new analytic session would default to "init"

So currently the AnalyticType is only applied as a flow_direction.

Does that make sense? I'm maybe over focussing on the semantics but I'm just trying to get my head around who things have been working and what we're looking for with this new functionality.

Thanks for this clarification, this makes a lot more sense now.

I'm thinking:

analytic session type captures how the session starts, it's never captured how it "ends" before now

therefore continuing to just capture reset as a flow_direction on analytics_log seems consistent?

There's an existing chart in metabase about "Where applicants drop-off" - and if we're considering "reset" as similar to "drop-off" - maybe looking at that underlying SQL query might offter confirmation if this approach will be able to be aggregated/summarised/charted similarly or be difficult without carrying over "reset" as a new analytic.type?

That's a very helpful framing of the situation, thanks Jess!

I'll have a look at "Where applicants drop-off" and see how it's being approached there and if there are any implications with this change.

I had a wee look at the SQL query:

SELECT date_trunc('week', analytics_logs.created_at) as "Week", count(*) as "Exits" FROM analytics_logs JOIN analytics ON (analytics.id = analytics_logs.analytics_id) WHERE analytics_logs.user_exit = true AND analytics_logs.node_type = 3 AND analytics.flow_id = {{flow_id}}::uuid GROUP BY "Week" ORDER BY "Week"

The drop-off uses the user_exit property which to my understanding is marked as true when a user stops viewing the page:

planx-new/editor.planx.uk/src/pages/FlowEditor/lib/analyticsProvider.tsx

Lines 60 to 86 in a636268

const onPageExit = () => {

if (lastAnalyticsLogId && shouldTrackAnalytics) {

if (document.visibilityState === "hidden") {

send(

`${

process.env.REACT_APP_API_URL

}/analytics/log-user-exit?analyticsLogId=${lastAnalyticsLogId.toString()}`,

);

}

if (document.visibilityState === "visible") {

send(

`${

process.env.REACT_APP_API_URL

}/analytics/log-user-resume?analyticsLogId=${lastAnalyticsLogId?.toString()}`,

);

}

}

};

useEffect(() => {

if (shouldTrackAnalytics)

document.addEventListener("visibilitychange", onPageExit);

return () => {

if (shouldTrackAnalytics)

document.removeEventListener("visibilitychange", onPageExit);

};

}, []);

I had thought of a user deciding to reset as not actually exiting the page and therefore not being a user_exit: true.

Although in the context of drop off I'm not so sure 🤔

Actually on second thoughts I think we'll be able to handle the reset state in a similar fashion as the query above so we should be okay I think 👍

- Currently, the new "reset" type will only be applied to the flow_direction property - Hence moving to the type for this property i.e. AnalyticsLogDirection - This can be easily changed in the future but it's more consistent with current behaviour - Auto-linting changes

Mike-Heneghan · 2023-10-17T16:01:10Z

Track Notice "reset" status in node metadata

Mike-Heneghan · 2023-10-17T16:01:28Z

Track which node users are on when they click to "Restart application" or "Start over"

Mike-Heneghan added 2 commits October 13, 2023 15:35

feat: track user resetting flow

d7695bc

- Update `analytics_log` record for node which was reset on reset confirmation - Change flow_direction to "reset" to store this user action - Ammend update permission to allow flow_direction to be changed

feat: track flow reset from notice

178d064

- On click of a reset button on a notice update analytics_log to `{flow_direction: "reset"}`

Commit to test PR build

039243a

Mike-Heneghan commented Oct 13, 2023

View reviewed changes

Mike-Heneghan requested a review from a team October 13, 2023 16:26

Mike-Heneghan marked this pull request as ready for review October 13, 2023 16:26

chore: remove unnecessary comment

a8de6b8

jessicamcinchak approved these changes Oct 16, 2023

View reviewed changes

editor.planx.uk/src/@planx/components/Notice/Public.tsx Outdated Show resolved Hide resolved

hasura.planx.uk/metadata/tables.yaml Show resolved Hide resolved

chore: add missing semi colons as per lint

ba12362

Mike-Heneghan commented Oct 16, 2023

View reviewed changes

Mike-Heneghan self-assigned this Oct 17, 2023

Mike-Heneghan merged commit 9515b88 into main Oct 18, 2023
12 checks passed

Mike-Heneghan deleted the mh/analytics-track-reset-via-fe branch October 18, 2023 09:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: track user resetting flow #2307

feat: track user resetting flow #2307

Mike-Heneghan commented Oct 13, 2023

github-actions bot commented Oct 13, 2023 •

edited

Loading

github-actions bot commented Oct 13, 2023 •

edited

Loading

Mike-Heneghan Oct 13, 2023

jessicamcinchak Oct 16, 2023

Mike-Heneghan Oct 16, 2023

Mike-Heneghan Oct 16, 2023

jessicamcinchak Oct 16, 2023

jessicamcinchak left a comment

Mike-Heneghan Oct 16, 2023

jessicamcinchak Oct 16, 2023 •

edited

Loading

Mike-Heneghan Oct 16, 2023

jessicamcinchak Oct 17, 2023 •

edited

Loading

Mike-Heneghan Oct 17, 2023

Mike-Heneghan Oct 17, 2023

Mike-Heneghan Oct 17, 2023

Mike-Heneghan commented Oct 17, 2023

Mike-Heneghan commented Oct 17, 2023

	const onPageExit = () => {
	if (lastAnalyticsLogId && shouldTrackAnalytics) {
	if (document.visibilityState === "hidden") {
	send(
	`${
	process.env.REACT_APP_API_URL
	}/analytics/log-user-exit?analyticsLogId=${lastAnalyticsLogId.toString()}`,
	);
	}
	if (document.visibilityState === "visible") {
	send(
	`${
	process.env.REACT_APP_API_URL
	}/analytics/log-user-resume?analyticsLogId=${lastAnalyticsLogId?.toString()}`,
	);
	}
	}
	};

	useEffect(() => {
	if (shouldTrackAnalytics)
	document.addEventListener("visibilitychange", onPageExit);
	return () => {
	if (shouldTrackAnalytics)
	document.removeEventListener("visibilitychange", onPageExit);
	};
	}, []);

feat: track user resetting flow #2307

feat: track user resetting flow #2307

Conversation

Mike-Heneghan commented Oct 13, 2023

What

Why

Screen Recording

Supersedes

Note

github-actions bot commented Oct 13, 2023 • edited Loading

Updated Tables (1)

github-actions bot commented Oct 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jessicamcinchak left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jessicamcinchak Oct 16, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jessicamcinchak Oct 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mike-Heneghan commented Oct 17, 2023

Mike-Heneghan commented Oct 17, 2023

github-actions bot commented Oct 13, 2023 •

edited

Loading

github-actions bot commented Oct 13, 2023 •

edited

Loading

jessicamcinchak Oct 16, 2023 •

edited

Loading

jessicamcinchak Oct 17, 2023 •

edited

Loading