[HOLD for payment 2024-03-26] [HOLD for payment 2024-03-20] [HOLD for payment 2024-03-13] Investigate a dashboard tool to monitor app performance improvement/degradation after deploys

[muttmuure]

The goal of the #newdot-performance project is achieving, maintaining (and protecting) these benchmarks for actions taken in app:

TTI: 3 seconds
Chat switching: 500ms
Sending a message: 100ms
Switching Active Workspace: 300ms

We have made good progress, in that the app is stable and performant on an anecdotal basis, and the wider company can focus on other areas of the roadmap. We would like to solidify this feeling with hard evidence and protect the app from performance regressions.

We would like to begin formally reporting on the performance benchmarks so we know they are being maintained. And we will use this data to protect the app’s benchmarked performance by “time-stamping” when performance regressions originate.

[adhorodyski]

@rinej started working on this today - he'll investigate the ways we can utilise json outputs from Reassure to showcase historical data and how we can pipe this through to eg. https://grafana.com/

[adhorodyski]

@mountiny I think we can assign @rinej to this one 🙂

[mountiny]

Grafana would be ideal as we use that already!

[rinej]

@mountiny Hello, today I started investigating the approach, I plan to use grafana

[melvin-bot]

📣 @rinej! 📣
Hey, it seems we don’t have your contributor details yet! You'll only have to do this once, and this is how we'll hire you on Upwork.
Please follow these steps:

1. Make sure you've read and understood the contributing guidelines.
2. Get the email address used to login to your Expensify account. If you don't already have an Expensify account, create one here. If you have multiple accounts (e.g. one for testing), please use your main account email.
3. Get the link to your Upwork profile. It's necessary because we only pay via Upwork. You can access it by logging in, and then clicking on your name. It'll look like this. If you don't already have an account, sign up for one here.
4. Copy the format below and paste it in a comment on this issue. Replace the placeholder text with your actual details.
   
   Format:

Contributor details
Your Expensify account email: <REPLACE EMAIL HERE>
Upwork Profile Link: <REPLACE LINK HERE>

[melvin-bot]

Triggered auto assignment to @stephanieelliott (NewFeature), see https://stackoverflowteams.com/c/expensify/questions/14418#:~:text=BugZero%20process%20steps%20for%20feature%20requests for more details.

[rinej]

Hello mountiny,

We created an example approach using the JSON REST API Grafana plugin -> https://grafana.com/grafana/plugins/marcusolsson-json-datasource/

Here is the example dashboard:

In that scenario, we created a simple node server that exposed the endpoints with JSON data, which Grafana consumed. It has some limitations; we will have to maintain a separate instance of the server to deliver data.

We are exploring the other two approaches:

1. Storing just the JSON file on the S3 bucket and updating it when the user runs tests.
2. Storing the SQLite database file in the S3 bucket and updating it when the user runs tests.

This will not require any additional server, just, for example, an S3 bucket to store the data. I will let you know when we have some insights about the approaches.

Meanwhile, do you have any preferences over what needs to be displayed and shown on the graph? I can send you the example output file from Reassure; just let me know! :)

[mountiny]

Thank you @rinej!

Storing just the JSON file on the S3 bucket and updating it when the user runs tests.

I thin kthis sounds good to me, we already have solutions where we save files to S3 bucket from the GH actions so it shoudl also be reasonable easy to achieve

[mountiny]

Meanwhile, do you have any preferences over what needs to be displayed and shown on the graph?

I was imagining that we would output the results of main branch over time, when the results get worse we can see based on timestamp what PR caused it, but we could just keep data about the results on main for various test scenarios and see how they evolved overtime

[rinej]

Thanks mountiny you for your answers!
We created some POC using JSON rest API grafana plugin

JSON api plugin: https://grafana.com/grafana/plugins/marcusolsson-json-datasource/
Plugin needs one endpoint which returns the data

Data used for the POC response (3 test cases run on different dates on different branches):

{"name":"should render Composer with text input interactions","type":"render", "meanDuration":18.244273030757904, "creationDate":"2023-11-17T14:49:26.957Z", "commitHash":"137a4c761ee686bd6e094ec025b0dcc3f5f454e1", "branch":"some-branch" },
{"name":"should press add attachemnt button","type":"render", "meanDuration":24.91762231886387, "creationDate":"2023-11-17T14:49:26.957Z", "commitHash":"137a4c761ee686bd6e094ec025b0dcc3f5f454e1", "branch":"some-branch" },
{"name":"should press add emoji button","type":"render", "meanDuration":12.9931875705718993, "creationDate":"2023-11-17T14:49:26.957Z", "commitHash":"137a4c761ee686bd6e094ec025b0dcc3f5f454e1", "branch":"some-branch" },
{"name":"should render Composer with text input interactions","type":"render", "meanDuration":13.244273030757904, "creationDate":"2023-11-15T12:45:04.146Z", "commitHash":"137a4c761ee686bd6e094ec025b0dcc3f5f454e1", "branch":"branch123" },
{"name":"should press add attachemnt button","type":"render", "meanDuration":16.91762231886387, "creationDate":"2023-11-15T12:45:04.146Z", "commitHash":"137a4c761ee686bd6e094ec025b0dcc3f5f454e1", "branch":"branch123" },
{"name":"should press add emoji button","type":"render", "meanDuration":2.9931875705718993, "creationDate":"2023-11-15T12:45:04.146Z", "commitHash":"137a4c761ee686bd6e094ec025b0dcc3f5f454e1", "branch":"branch123" },
{"name":"should render Composer with text input interactions","type":"render", "meanDuration":14.244273030757904, "creationDate":"2023-11-16T14:49:26.957Z", "commitHash":"137a4c761ee686bd6e094ec025b0dcc3f5f454e1", "branch":"test-branch" },
{"name":"should press add attachemnt button","type":"render", "meanDuration":20.91762231886387, "creationDate":"2023-11-16T14:49:26.957Z", "commitHash":"137a4c761ee686bd6e094ec025b0dcc3f5f454e1", "branch":"test-branch" },
{"name":"should press add emoji button","type":"render", "meanDuration":5.9931875705718993, "creationDate":"2023-11-16T14:49:26.957Z", "commitHash":"137a4c761ee686bd6e094ec025b0dcc3f5f454e1", "branch":"test-branch" }

Example dashboards:

A - table view - we can choose from a dropdown which test data we want to display
B - Tests grouped by name on a timeline - we can preview additional information about each point
C - Table view for all test data
D - Tests grouped by name on one graph

I think dashboards A and B will be the most useful, so we can see on the timeline when performance has dropped

🎯 What needs to be done:

• Add github action to run reassure after merge to main
• Upload output.json file to S3 bucket
  • It can be done either in reassure lib or in the script in Expensify repo
• Create endpoint which gets JSONs files from S3, combines it and returns data in the above format
• Create dashboard in Grafana which consumes the endpoint

❓Questions to mountiny?

• Do you have some infrastructure so we can create and maintain the simple REST get endpoint? The logic in the endpoint:
• Get all json performance files from S3 bucket (or any other storage place)
• Combine and return the final json file which grafana will consume

[mountiny]

@rinej Looks very promising, thanks! I will discuss with out infra team who could help us with the implementation as I do not have access to edit S3 or add this plugin to Graphana if we do not have it yet

[melvin-bot]

Triggered auto assignment to @johnmlee101 (ring0), see https://stackoverflow.com/c/expensify/questions/6102 for more details.

[mountiny]

Thread we were discussing this is here

[rinej]

@mountiny I don't have access to the above link or the link is broken, could you give me the access?

[mountiny]

Sorry cannot give you access to that one. It was Thanksgiving week so @johnmlee101 will be back this week and he is going to be able to help us with this I am sure

[johnmlee101]

Upload output.json file to S3 bucket

Okay might need to figure out some of these details. I think S3 might be overkill if we just need to store and fetch.
Going to check one last thing with Infra

[johnmlee101]

@mountiny @rinej
https://docs.github.com/en/actions/using-workflows/storing-workflow-data-as-artifacts

@justinpersaud brought up using artifacts if we don't care about the data lasting for too long. We have a max 90 day policy.

As for the REST endpoint if we can easily find a way to list all artifacts this way that would be ideal

[rinej]

@johnmlee101 @mountiny
you mentioned that you already use Grafana for some metrics, what kind of data source do you use? Do you use some service like https://prometheus.io/ recommended from Grafana?
Maybe adding integration to the existing flow would be more efficient, instead of creating something new (like S3)?

Do you use Grafana cloud service or standalone app?

[melvin-bot]

Skipping the payment summary for this issue since all the assignees are employees or vendors. If this is incorrect, please manually add the payment summary SO.

[melvin-bot]

Reviewing label has been removed, please complete the "BugZero Checklist".

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 1.4.51-3 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• [NoQA] [Dashboard] add graphite name and port #38061

If no regressions arise, payment will be issued on 2024-03-20. 🎊

For reference, here are some details about the assignees on this issue:

• @rinej does not require payment (Contractor)

[melvin-bot]

BugZero Checklist: The PR adding this new feature has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:

• [@rinej] Please propose regression test steps to ensure the new feature will work correctly on production in further releases.
• [@stephanieelliott] Link the GH issue for creating/updating the regression test once above steps have been agreed upon.

[stephanieelliott]

This was a new feature/tool, no regression test needed here.

[mountiny]

Correct! I think this is done but just checked the dashboard and cant see anything in it

asking if we moved it somewhere else

[mountiny]

It just takes very long to load for anything more than a day

[melvin-bot]

Reviewing label has been removed, please complete the "BugZero Checklist".

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 1.4.54-4 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• [NoQA] Update the format of the string sent for reassure test dashboard #38405

If no regressions arise, payment will be issued on 2024-03-26. 🎊

For reference, here are some details about the assignees on this issue:

• @rinej does not require payment (Contractor)

[melvin-bot]

BugZero Checklist: The PR adding this new feature has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:

• [@rinej] Please propose regression test steps to ensure the new feature will work correctly on production in further releases.
• [@stephanieelliott] Link the GH issue for creating/updating the regression test once above steps have been agreed upon.

[stephanieelliott]

No regression test needed, this was a new feature! Expert contrib so no payment due on this one.