Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Issue #3127] Add a New Relic agent to the API #3130

Merged
merged 20 commits into from
Dec 9, 2024

Conversation

mikehgrantsgov
Copy link
Collaborator

@mikehgrantsgov mikehgrantsgov commented Dec 6, 2024

Summary

Fixes #3127

Time to review: 10 mins

Changes proposed

Add a New Relic ini file, dependency, and lightweight utilities to send custom events.

Context for reviewers

New Relic monitoring is needed for the application and the .ini file is the start place for configuration. Environment-specific configuration is available as well. The NEW_RELIC_LICENSE_KEY will be provided as an environment variable (command line?) to the application in non-local environments.

Additional information

Local env reports to a fake collector. Other envs will report property to NR.
Screenshot 2024-12-06 at 2 32 34 PM

Local env running gunicorn showing NewRelic initializing and sending events to the fake collector:
Screenshot 2024-12-09 at 3 08 45 PM
Screenshot 2024-12-09 at 3 08 38 PM

@mikehgrantsgov mikehgrantsgov marked this pull request as ready for review December 6, 2024 19:34
mdragon
mdragon previously approved these changes Dec 6, 2024
Copy link
Collaborator

@mdragon mdragon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

api/src/util/newrelic/events.py Outdated Show resolved Hide resolved
api/src/app.py Outdated
@@ -67,6 +68,8 @@ def create_app() -> APIFlask:
if os.getenv("ENVIRONMENT") == "local":
initialize_jwt_auth()

init_newrelic()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't this need to happen before anything else? We did that before / the New Relic docs say that needs to happen.

https://docs.newrelic.com/install/python/?python-non-web=non-web-yes#integrating


It might be that this needs to be done for multi-threading purposes, which we only do non-locally unless you reconfigure some commands.

You'll need to swap out the command temporarily at https://github.com/HHS/simpler-grants-gov/blob/main/api/docker-compose.yml#L78 for what we use in prod to run via Gunicorn: https://github.com/HHS/simpler-grants-gov/blob/main/api/Dockerfile#L127

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chouinar What if we just lift init_newrelic() out into app.py, like I did here? Would we still need to change the Docker command?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm suggesting you adjust the docker command to test that it works in multi-threaded/multi-process mode, not as a permanent change

Comment on lines +258 to +268
[newrelic:staging]
app_name = SIMPLER-GRANTS-API-STAGING
monitor_mode = true

[newrelic:prod]
app_name = SIMPLER-GRANTS-API-PROD
monitor_mode = true

[newrelic:dev]
app_name = SIMPLER-GRANTS-API-DEV
monitor_mode = true
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@coilysiren - I think we talked about this a few weeks ago, but what landing scheme did we land on for the API app in New Relic?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks fine to me - the SIMPLER-GRANTS- bit is probably superfluous but if that matches the frontend then I'm down to keep it

api/src/util/newrelic/events.py Outdated Show resolved Hide resolved
api/src/util/newrelic/events.py Outdated Show resolved Hide resolved
mdragon
mdragon previously approved these changes Dec 9, 2024
Copy link
Collaborator

@chouinar chouinar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to verify that this works locally when running via gunicorn before we merge - the process setup is very different and likely why New Relic says it needs to run before everything

@mikehgrantsgov
Copy link
Collaborator Author

I'd like to verify that this works locally when running via gunicorn before we merge - the process setup is very different and likely why New Relic says it needs to run before everything

Thanks, I added screenshots of the docker container running via gunicorn (see gthreads in screenshots) and added evidence of the messages created correctly in the fake NR collector.

Copy link
Collaborator

@chouinar chouinar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@mikehgrantsgov mikehgrantsgov merged commit 21e5c05 into main Dec 9, 2024
8 checks passed
@mikehgrantsgov mikehgrantsgov deleted the mikehgrantsgov/3127-add-new-relic-agent branch December 9, 2024 21:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add a New Relic agent to the API
5 participants