Write to Both DynamoDB and RDS

[Douglas-Hong]

Summary

• Make postgres DB on RDS with Drizzle ORM.
• Write to both DDB and RDS as first part of migration.
• Updates patch notes to inform about migration and feedback form.

Migration Procedures

• Start saving to both DBs.
• Copy to new DB, while still writing to both.
  • This needs to be done while few people are using AA. I can set up a chron job to do this at an ungodly hour.
  • Then I'll check in the morning to see if it worked.
• Switch read to new DB.
• Remove old DB.

Action Items

• Auth/user relations.
• Schedules/courses relations.
• Custom events relations.
• Make APIs for new DB.
  • Read/write changes happen here. Don't change API.
• Migration script(s).
  • NOTE: This needs to mangle duplicate schedule names because the (schedule, username) pair is unique.
  • Since we're going to start writing to both first, the copy should ignore schedules that already exist in RDS
• Migration procedures
• Provision dev and prod databases.
  • Just one dev database for non-database PRs.
  • PRs that change the DB just dev locally or will be manually provisioned.
• Check migration and dual-writing locally.
• CI/CD pipelines (ci: pipeline for RDS migration #1029)
  • Only migrate when merging into main?
• Batch copy. Each batch takes 5 minutes, which is good enough.
• Read from RDS in different branch (Read from RDS #1060).
• Execute migration

Test Plan

• Save a schedule.
• See that its data (schedule, courses, custom events) is in pgAdmin or Drizzle Studio.
• Do the same in deployment.

Related Issues

• Part of RFC: Login system #890

Future Followup

• Google auth #1027
• Only update changed schedule(s) instead of all of them.

[MinhxNguyen7]

I'm going to look at and finish up this PR

[MinhxNguyen7]

Now that I'm actually writing the migration scripts, I've realized that this is more work than I gave it credit for.

I'm going to try to get this done in the next week or so, but I don't like my chances

[MinhxNguyen7]

@ecxyzzy I'm gonna need to work with you on this. We've been on this rodeo together, so it's not too crazy, but we need to review the plan, review the migration script, and help with the deployment.

[MinhxNguyen7]

The staging instance isn't working, reporting "DB_URL must be defined" despite #1058 adding the dev DB to the staging deploy.

Works locally as well.

[MinhxNguyen7]

The env is hardcoded in 2 places
Why do you do this to me @ap0nia?

[ap0nia]

I really disliked that too... As far as I can tell, there isn't a good way to fix this because the declarations are used in completely unrelated parts of the deployment pipeline.

The environment variables in the GitHub workflow manually load all the repository secrets into the current environment prior to any scripts running, whereas the CDK script validates environment variables when it actually runs. Neither of these steps can be done in the other one...

A possible strategy might be to upload a base64 encoded env file and decode it to get all the environment variables. This makes the repository secrets more opaque and the deployment scripts more complicated...

[MinhxNguyen7]

@ecxyzzy thanks a ton.

Do you mind talking about what the problem was?

[ecxyzzy]

I think the main issue was that at some point USERDATA_TABLE_NAME stopped getting injected into the Lambda env, since CDK is responsible for that. Other fixes I did included separating the environment variables needed by the backend at runtime and the environment variables needed at build time.

[MinhxNguyen7]

Everything works locally, but, in deployment, writes to the DB are timing out. Seems like there is no regression, though.

[MinhxNguyen7]

Above problem has been fixed. Apparently the problem was the lambda dies after the response is sent, and we don't await the RDS write.