Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write to both DynamoDB and RDS #1064

Merged
merged 92 commits into from
Dec 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
92 commits
Select commit Hold shift + click to select a range
cf5bc89
Scan dynamo for all ids
Douglas-Hong May 13, 2024
49268b2
feat(backend): db drizzle schema
ap0nia May 16, 2024
f92b827
fix(backend): db schema entrypoint
ap0nia May 16, 2024
e196420
Finish local migration
Douglas-Hong Jun 3, 2024
49d723d
chore(ddb.ts): remove unused import
MinhxNguyen7 Oct 24, 2024
60ac1d5
fix(auth): table
MinhxNguyen7 Oct 24, 2024
a572caa
feat(subscription): target status enum
MinhxNguyen7 Oct 24, 2024
da1facb
chore(schema): rename tables to plural nouns
MinhxNguyen7 Oct 24, 2024
dafd70b
feat(schema): move active schedule into users table
MinhxNguyen7 Oct 24, 2024
1117b0c
refactor(schema): schedule dir
MinhxNguyen7 Nov 6, 2024
5464871
feat(schema): export some types
MinhxNguyen7 Nov 6, 2024
46c359c
feat(schema): remove name from user
MinhxNguyen7 Nov 6, 2024
2a013df
feat(migration): proper batching and filtering in getting all data
MinhxNguyen7 Nov 6, 2024
a68b906
feat(schema): clearer account types
MinhxNguyen7 Nov 6, 2024
f3caa71
feat(migration): properly copy users
MinhxNguyen7 Nov 6, 2024
897e647
feat(coursesInSchedule): not null columns
MinhxNguyen7 Nov 10, 2024
a1616c7
fix(ddb): typing for getAddUserDataBatches
MinhxNguyen7 Nov 10, 2024
db8abd0
feat(user): re-add name
MinhxNguyen7 Nov 10, 2024
a5f17d4
feat(migrate): migration script
MinhxNguyen7 Nov 10, 2024
8596734
fix(migrate): return statement positioning
MinhxNguyen7 Nov 10, 2024
c02b374
feat(migrate) error handling
MinhxNguyen7 Nov 10, 2024
d41e2b0
chore(trpc): remove unused endpoint
MinhxNguyen7 Nov 11, 2024
d94c614
feat(schema): add lastUpdated to tables
MinhxNguyen7 Nov 11, 2024
47562f3
feat(schema): add lastUpdated to tables
MinhxNguyen7 Nov 12, 2024
50f454f
feat(schema): user's schedules must be unique
MinhxNguyen7 Nov 12, 2024
9e317a3
feat(api): drizzle db wrapper
MinhxNguyen7 Nov 12, 2024
fd9d6d8
refactor(migrate): use RDS wrapper
MinhxNguyen7 Nov 12, 2024
a764216
feat(api): write to both RDS and Dynamo
MinhxNguyen7 Nov 12, 2024
d65892d
chore: remove unused imports
MinhxNguyen7 Nov 12, 2024
02fbb46
feat(api): mangle duplicate schedule names
MinhxNguyen7 Nov 12, 2024
2ffa976
chore(backend): separate DDB->RDS migration from normal migration
MinhxNguyen7 Nov 14, 2024
691fa90
ci: pipeline for RDS migration (#1029)
ecxyzzy Nov 14, 2024
17fafe7
chore: clean up migration scripts
MinhxNguyen7 Nov 14, 2024
f721579
feat(package.json): ddb to rds command
MinhxNguyen7 Nov 14, 2024
c704230
feat(drizzle): dotenv in config and studio
MinhxNguyen7 Nov 14, 2024
ed8bf5a
feat(drizzle): use arktype instead of dotenv
MinhxNguyen7 Nov 16, 2024
99e7479
refactor(backend): remove mongo
MinhxNguyen7 Nov 16, 2024
68568a4
chore(turbo): rename pipeline to tasks
MinhxNguyen7 Nov 17, 2024
43a9953
chore(deps): update drizzle
MinhxNguyen7 Nov 17, 2024
23e0806
feat(migrate): write one batch at a time
MinhxNguyen7 Nov 17, 2024
6e80fb0
feat(migrate): remove postgres migration from copy script
MinhxNguyen7 Nov 17, 2024
6d8169a
feat(backend): use dotenv in env
MinhxNguyen7 Nov 17, 2024
a124b1e
chore(deps): update drizzle-kit (again)
MinhxNguyen7 Nov 17, 2024
37425f0
fix(ddb): valid user filter with id instead of name
MinhxNguyen7 Nov 17, 2024
0aa4ec3
fix(rds): don't insert courses for schedule with no courses
MinhxNguyen7 Nov 17, 2024
b5af50b
feat(db): import DB_URL from env
MinhxNguyen7 Nov 17, 2024
a0d9362
fix(RDS): transaction param type
MinhxNguyen7 Nov 17, 2024
09cd5b3
feat(migrate): more helpful printout
MinhxNguyen7 Nov 17, 2024
934ba64
feat(RDS): deduplicate courses in schedules
MinhxNguyen7 Nov 17, 2024
4db00fa
feat(migrate): more helpful error message
MinhxNguyen7 Nov 17, 2024
a71b176
deps(package.json): update
MinhxNguyen7 Nov 17, 2024
e07e54b
chore(db): new migrations
MinhxNguyen7 Nov 17, 2024
ba2e1a6
feat(pnpm): more convenenient db scripts
MinhxNguyen7 Nov 17, 2024
3f7f198
chore(schema): new pgTable overload
MinhxNguyen7 Nov 17, 2024
6cee06b
feat(schema): custom events
MinhxNguyen7 Nov 17, 2024
ef13911
fix(schema): custom events
MinhxNguyen7 Nov 17, 2024
96bc950
feat(api): update custom events
MinhxNguyen7 Nov 17, 2024
9d33523
chore(.gitignore): turbo
MinhxNguyen7 Nov 17, 2024
c6d7e7a
Merge branch 'main' into migrate-rds
MinhxNguyen7 Nov 22, 2024
bf9abaa
fix(users): missing imports
MinhxNguyen7 Nov 22, 2024
de96497
fix(schema): proper implementation of constraint array
MinhxNguyen7 Nov 22, 2024
3395b1f
fix(migration): error messages
MinhxNguyen7 Nov 22, 2024
dc63e00
feat(RDS): more specific errors
MinhxNguyen7 Nov 22, 2024
68647bb
feat(RDS): more descriptive messages
MinhxNguyen7 Nov 22, 2024
ae3a27a
feat(migration): always print out success totals
MinhxNguyen7 Nov 22, 2024
6c8255c
Fixed "Error loading grades information" with advanced search (#1043)
IsaacNguyen Nov 22, 2024
ef2ccc5
deps(AA): babel runtime
MinhxNguyen7 Nov 25, 2024
aa051d4
feat(PathNotes): migration warning
MinhxNguyen7 Nov 25, 2024
ac55e11
feat: method for inserting guest user data
ecxyzzy Nov 25, 2024
517c3b7
chore: nuke old searchData
ecxyzzy Nov 25, 2024
f6492df
feat: switch to insert, track skipped users
ecxyzzy Nov 25, 2024
54f75ff
feat(schema): support schedule ordering
MinhxNguyen7 Nov 25, 2024
91e1988
chore(db): schedule index migration
MinhxNguyen7 Nov 25, 2024
51f537f
fix: db => tx (im stupid)
ecxyzzy Nov 25, 2024
74adda3
fix(db): guest user account insert
MinhxNguyen7 Nov 25, 2024
6d4e7bb
feat(migration): time profiling
MinhxNguyen7 Nov 25, 2024
14cc77b
fix(api): drop all schedules before inserting
MinhxNguyen7 Nov 25, 2024
591a9e0
feat(schema): set current schedule to null on delete
MinhxNguyen7 Nov 25, 2024
88956ae
Merge remote-tracking branch 'origin/main' into migrate-rds
MinhxNguyen7 Dec 2, 2024
536d839
feat(cdk): env join from backend
MinhxNguyen7 Dec 2, 2024
7076e2f
ci: hopefully fix deploy?
ecxyzzy Dec 2, 2024
4e182c6
ci: actually fix things???
ecxyzzy Dec 2, 2024
d52aa25
fljdsahfljdsahfs
ecxyzzy Dec 2, 2024
b7dab83
feat(dev): dotenv for backend
MinhxNguyen7 Dec 3, 2024
84b9e32
fix(users router): typo
MinhxNguyen7 Dec 3, 2024
73113ed
feat: parallelize ddb/rds write
ecxyzzy Dec 3, 2024
282d847
fix(ci): use default environment
MinhxNguyen7 Dec 6, 2024
14bc098
feat(saveUserData): only throw if DDB fails to write
MinhxNguyen7 Dec 6, 2024
d7ecd1f
test(ci): (temporarily) run migration
MinhxNguyen7 Dec 6, 2024
b493738
Revert "fix(ci): use default environment"
MinhxNguyen7 Dec 6, 2024
e42fd7f
feat(backend): separate env schema for RDS
MinhxNguyen7 Dec 6, 2024
ed86824
Revert "test(ci): (temporarily) run migration"
MinhxNguyen7 Dec 6, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/workflows/deploy_production.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ env:
VITE_TILES_ENDPOINT: ${{ secrets.VITE_TILES_ENDPOINT}}
GOOGLE_CLIENT_ID: ${{ secrets.GOOGLE_CLIENT_ID }}
GOOGLE_CLIENT_SECRET: ${{ secrets.GOOGLE_CLIENT_SECRET }}
DB_URL: ${{ secrets.PROD_DB_URL }}
ANTEATER_API_KEY: ${{ secrets.ANTEATER_API_KEY }}

# Turborepo credentials.
Expand All @@ -44,6 +45,7 @@ env:
HOSTED_ZONE_ID: ${{ secrets.HOSTED_ZONE_ID }}
CERTIFICATE_ARN: ${{ secrets.CERTIFICATE_ARN }}
PR_NUM: ${{ github.event.pull_request.number }}
STAGE: prod

jobs:
# Build production version of the frontend and upload the artifacts.
Expand Down Expand Up @@ -111,6 +113,9 @@ jobs:
- name: Install dependencies
run: pnpm install --frozen-lockfile --prod false

- name: Run database migrations
run: pnpm --filter "antalmanac-backend" migrate

- name: Build backend
run: pnpm --filter "antalmanac-backend" build
env:
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/deploy_staging.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ env:
HOSTED_ZONE_ID: ${{ secrets.HOSTED_ZONE_ID }}
CERTIFICATE_ARN: ${{ secrets.CERTIFICATE_ARN }}
PR_NUM: ${{ github.event.pull_request.number }}
STAGE: dev

jobs:
get_staging_configuration:
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/destroy_staging.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ env:
HOSTED_ZONE_ID: ${{ secrets.HOSTED_ZONE_ID }}
CERTIFICATE_ARN: ${{ secrets.CERTIFICATE_ARN }}
PR_NUM: ${{ github.event.pull_request.number }}
STAGE: dev

jobs:
destroy_staging_frontend:
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -50,3 +50,6 @@ Desktop.ini

## VSCode settings.
.vscode/

## Turbo
.turbo/
1 change: 1 addition & 0 deletions apps/antalmanac/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
"lint": "eslint src"
},
"dependencies": {
"@babel/runtime": "^7.26.0",
"@date-io/date-fns": "^2.16.0",
"@emotion/react": "^11.10.5",
"@emotion/styled": "^11.10.5",
Expand Down
35 changes: 16 additions & 19 deletions apps/antalmanac/src/components/PatchNotes.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ import { getLocalStoragePatchNotesKey, setLocalStoragePatchNotesKey } from '$lib
*
* @example '20230819'
*/
export const latestPatchNotesUpdate = '20230819';
export const latestPatchNotesUpdate = '20241124';

/**
* Whether the user's last visited patch notes is outdated.
Expand Down Expand Up @@ -53,30 +53,27 @@ function PatchNotes() {
data-testid={dialogTestId}
slots={{ backdrop: PatchNotesBackdrop }}
>
<DialogTitle>{"What's New - October 2023"}</DialogTitle>
<DialogTitle>{"What's New - November 2024"}</DialogTitle>

<DialogContent>
<Typography>Features</Typography>
<Typography>Migration</Typography>
<ul>
<li>We are migrating our database to support exciting features coming this year!</li>
<li>
You can now hover over the Zotistics button to see the Zotistics graph! On mobile, you can still
click the Zotistics button to toggle the graph.
If you experience issues with saving and retrieving schedules, please let us know by filling out
the{' '}
<a href="https://docs.google.com/forms/d/e/1FAIpQLSe0emRHqog-Ctl8tjZfJvewY_CSGXys8ykBkFBy1EEUUUHbUw/viewform">
feedback form
</a>
.
</li>
</ul>
<img
src="https://user-images.githubusercontent.com/78244965/277567417-f9816b9d-ddda-4c0f-80f4-eeac92428612.gif"
alt="(gif of the new feature)"
style={{
maxWidth: '100%',
boxShadow: '4px 4px 4px rgba(0, 0, 0, 0.4)',
}}
/>
<br />
Remember to use the{' '}
<a href="https://docs.google.com/forms/d/e/1FAIpQLSe0emRHqog-Ctl8tjZfJvewY_CSGXys8ykBkFBy1EEUUUHbUw/viewform">
feedback form
</a>{' '}
to let us know what you think!

<Typography>Features</Typography>
<ul>
<li>Search now contains all new classes and will update automatically!</li>
<li>Many bug fixes and quality-of-life improvements</li>
</ul>
</DialogContent>

<DialogActions>
Expand Down
14 changes: 14 additions & 0 deletions apps/backend/drizzle.config.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
import { defineConfig } from "drizzle-kit";

import { rdsEnvSchema } from "./src/env";

const { DB_URL } = rdsEnvSchema.parse(process.env);

export default defineConfig({
dialect: "postgresql",
schema: "./src/db/schema/index.ts",
out: "./drizzle",
dbCredentials: {
url: DB_URL,
}
});
103 changes: 103 additions & 0 deletions apps/backend/drizzle/0000_rapid_speed_demon.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
CREATE TYPE "public"."account_type" AS ENUM('GOOGLE', 'GUEST');--> statement-breakpoint
CREATE TYPE "public"."subscription_target_status" AS ENUM('OPEN', 'WAITLISTED');--> statement-breakpoint
CREATE TABLE IF NOT EXISTS "accounts" (
"user_id" text NOT NULL,
"account_type" "account_type" NOT NULL,
"provider_account_id" text NOT NULL,
CONSTRAINT "accounts_user_id_account_type_pk" PRIMARY KEY("user_id","account_type")
);
--> statement-breakpoint
CREATE TABLE IF NOT EXISTS "sessions" (
"id" text PRIMARY KEY NOT NULL,
"user_id" text NOT NULL,
"expires" timestamp NOT NULL,
"refresh_token" text
);
--> statement-breakpoint
CREATE TABLE IF NOT EXISTS "users" (
"id" text PRIMARY KEY NOT NULL,
"phone" text,
"avatar" text,
"name" text,
"current_schedule_id" text,
"last_updated" timestamp with time zone DEFAULT now()
);
--> statement-breakpoint
CREATE TABLE IF NOT EXISTS "schedules" (
"id" text PRIMARY KEY NOT NULL,
"user_id" text NOT NULL,
"name" text,
"notes" text,
"last_updated" timestamp with time zone NOT NULL,
CONSTRAINT "schedules_user_id_name_unique" UNIQUE("user_id","name")
);
--> statement-breakpoint
CREATE TABLE IF NOT EXISTS "coursesInSchedule" (
"scheduleId" text NOT NULL,
"sectionCode" integer NOT NULL,
"term" text NOT NULL,
"color" text NOT NULL,
"last_updated" timestamp with time zone DEFAULT now(),
CONSTRAINT "coursesInSchedule_scheduleId_sectionCode_term_pk" PRIMARY KEY("scheduleId","sectionCode","term")
);
--> statement-breakpoint
CREATE TABLE IF NOT EXISTS "customEvents" (
"id" text PRIMARY KEY NOT NULL,
"scheduleId" text NOT NULL,
"title" text NOT NULL,
"start" text NOT NULL,
"end" text NOT NULL,
"days" text NOT NULL,
"color" text,
"building" text,
"last_updated" timestamp with time zone DEFAULT now()
);
--> statement-breakpoint
CREATE TABLE IF NOT EXISTS "subscriptions" (
"userId" text,
"sectionCode" integer,
"status" "subscription_target_status",
CONSTRAINT "subscriptions_userId_sectionCode_pk" PRIMARY KEY("userId","sectionCode")
);
--> statement-breakpoint
DO $$ BEGIN
ALTER TABLE "accounts" ADD CONSTRAINT "accounts_user_id_users_id_fk" FOREIGN KEY ("user_id") REFERENCES "public"."users"("id") ON DELETE cascade ON UPDATE no action;
EXCEPTION
WHEN duplicate_object THEN null;
END $$;
--> statement-breakpoint
DO $$ BEGIN
ALTER TABLE "sessions" ADD CONSTRAINT "sessions_user_id_users_id_fk" FOREIGN KEY ("user_id") REFERENCES "public"."users"("id") ON DELETE cascade ON UPDATE no action;
EXCEPTION
WHEN duplicate_object THEN null;
END $$;
--> statement-breakpoint
DO $$ BEGIN
ALTER TABLE "users" ADD CONSTRAINT "users_current_schedule_id_schedules_id_fk" FOREIGN KEY ("current_schedule_id") REFERENCES "public"."schedules"("id") ON DELETE no action ON UPDATE no action;
EXCEPTION
WHEN duplicate_object THEN null;
END $$;
--> statement-breakpoint
DO $$ BEGIN
ALTER TABLE "schedules" ADD CONSTRAINT "schedules_user_id_users_id_fk" FOREIGN KEY ("user_id") REFERENCES "public"."users"("id") ON DELETE cascade ON UPDATE no action;
EXCEPTION
WHEN duplicate_object THEN null;
END $$;
--> statement-breakpoint
DO $$ BEGIN
ALTER TABLE "coursesInSchedule" ADD CONSTRAINT "coursesInSchedule_scheduleId_schedules_id_fk" FOREIGN KEY ("scheduleId") REFERENCES "public"."schedules"("id") ON DELETE cascade ON UPDATE no action;
EXCEPTION
WHEN duplicate_object THEN null;
END $$;
--> statement-breakpoint
DO $$ BEGIN
ALTER TABLE "customEvents" ADD CONSTRAINT "customEvents_scheduleId_schedules_id_fk" FOREIGN KEY ("scheduleId") REFERENCES "public"."schedules"("id") ON DELETE cascade ON UPDATE no action;
EXCEPTION
WHEN duplicate_object THEN null;
END $$;
--> statement-breakpoint
DO $$ BEGIN
ALTER TABLE "subscriptions" ADD CONSTRAINT "subscriptions_userId_users_id_fk" FOREIGN KEY ("userId") REFERENCES "public"."users"("id") ON DELETE cascade ON UPDATE no action;
EXCEPTION
WHEN duplicate_object THEN null;
END $$;
2 changes: 2 additions & 0 deletions apps/backend/drizzle/0001_powerful_tiger_shark.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
ALTER TABLE "schedules" ADD COLUMN "index" integer NOT NULL;--> statement-breakpoint
ALTER TABLE "schedules" ADD CONSTRAINT "schedules_user_id_index_unique" UNIQUE("user_id","index");
7 changes: 7 additions & 0 deletions apps/backend/drizzle/0002_fat_ted_forrester.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
ALTER TABLE "users" DROP CONSTRAINT "users_current_schedule_id_schedules_id_fk";
--> statement-breakpoint
DO $$ BEGIN
ALTER TABLE "users" ADD CONSTRAINT "users_current_schedule_id_schedules_id_fk" FOREIGN KEY ("current_schedule_id") REFERENCES "public"."schedules"("id") ON DELETE set null ON UPDATE no action;
EXCEPTION
WHEN duplicate_object THEN null;
END $$;
Loading
Loading