Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[infrastructure] add explicit autoscaling logic to Hasura service #4096

Merged
merged 3 commits into from
Dec 20, 2024

Conversation

freemvmt
Copy link
Contributor

@freemvmt freemvmt commented Dec 19, 2024

Relates to this ticket.

This PR follows hot on the heels of #4080 and #4081, neither of which resolved the issue with Hasura (although I think the latter was necessary because the number of connections the AWS RDS database could handle was a limiting factor).

It implements the following:

  • health checks for both the hasura and hasuraProxy containers, with values based on the docker setup
  • new env variables to give a long timeout for a new Hasura server to load metadata and run migrations at boot, ensure said migrations run sequentially (not concurrently) to minimise chance of overloading CPU, and show debug level logs to easily diagnose whatever's going on
  • according grace periods / timeouts throughout
  • doubles CPU only (to 1GiB) for the hasura container (changed in staging and prod yaml for consistency)
  • explicit scaling policies for the Hasura ECS service, which should trigger a scale out when CPU usage is over 60%, or memory usage over 75%

Those last values are up for grabs/can be adjusted over time. Based on a load test I just ran (with 1000 users hitting the Hasura graphQL endpoint every few seconds), it's CPU utilisation that I would expect to be the trigger.

image

Copy link

github-actions bot commented Dec 19, 2024

Removed vultr server and associated DNS entries

Copy link
Member

@jessicamcinchak jessicamcinchak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for comments throughout here & really getting to grips with all this Pulumi config ! Happy for this to be tested on staging whenever you're ready 🏁

@freemvmt freemvmt marked this pull request as ready for review December 20, 2024 15:11
@freemvmt freemvmt force-pushed the hasura-service-explicit-autoscaling branch from ddc8e93 to 8f15c25 Compare December 20, 2024 15:14
@freemvmt freemvmt merged commit 3421ae5 into main Dec 20, 2024
12 checks passed
@freemvmt freemvmt deleted the hasura-service-explicit-autoscaling branch December 20, 2024 15:54
@freemvmt
Copy link
Contributor Author

See #4103 for minor fix to this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants