Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read from RDS #1060

Merged
merged 88 commits into from
Dec 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
88 commits
Select commit Hold shift + click to select a range
cf5bc89
Scan dynamo for all ids
Douglas-Hong May 13, 2024
49268b2
feat(backend): db drizzle schema
ap0nia May 16, 2024
f92b827
fix(backend): db schema entrypoint
ap0nia May 16, 2024
e196420
Finish local migration
Douglas-Hong Jun 3, 2024
49d723d
chore(ddb.ts): remove unused import
MinhxNguyen7 Oct 24, 2024
60ac1d5
fix(auth): table
MinhxNguyen7 Oct 24, 2024
a572caa
feat(subscription): target status enum
MinhxNguyen7 Oct 24, 2024
da1facb
chore(schema): rename tables to plural nouns
MinhxNguyen7 Oct 24, 2024
dafd70b
feat(schema): move active schedule into users table
MinhxNguyen7 Oct 24, 2024
1117b0c
refactor(schema): schedule dir
MinhxNguyen7 Nov 6, 2024
5464871
feat(schema): export some types
MinhxNguyen7 Nov 6, 2024
46c359c
feat(schema): remove name from user
MinhxNguyen7 Nov 6, 2024
2a013df
feat(migration): proper batching and filtering in getting all data
MinhxNguyen7 Nov 6, 2024
a68b906
feat(schema): clearer account types
MinhxNguyen7 Nov 6, 2024
f3caa71
feat(migration): properly copy users
MinhxNguyen7 Nov 6, 2024
897e647
feat(coursesInSchedule): not null columns
MinhxNguyen7 Nov 10, 2024
a1616c7
fix(ddb): typing for getAddUserDataBatches
MinhxNguyen7 Nov 10, 2024
db8abd0
feat(user): re-add name
MinhxNguyen7 Nov 10, 2024
a5f17d4
feat(migrate): migration script
MinhxNguyen7 Nov 10, 2024
8596734
fix(migrate): return statement positioning
MinhxNguyen7 Nov 10, 2024
c02b374
feat(migrate) error handling
MinhxNguyen7 Nov 10, 2024
d41e2b0
chore(trpc): remove unused endpoint
MinhxNguyen7 Nov 11, 2024
d94c614
feat(schema): add lastUpdated to tables
MinhxNguyen7 Nov 11, 2024
47562f3
feat(schema): add lastUpdated to tables
MinhxNguyen7 Nov 12, 2024
50f454f
feat(schema): user's schedules must be unique
MinhxNguyen7 Nov 12, 2024
9e317a3
feat(api): drizzle db wrapper
MinhxNguyen7 Nov 12, 2024
fd9d6d8
refactor(migrate): use RDS wrapper
MinhxNguyen7 Nov 12, 2024
a764216
feat(api): write to both RDS and Dynamo
MinhxNguyen7 Nov 12, 2024
d65892d
chore: remove unused imports
MinhxNguyen7 Nov 12, 2024
02fbb46
feat(api): mangle duplicate schedule names
MinhxNguyen7 Nov 12, 2024
2ffa976
chore(backend): separate DDB->RDS migration from normal migration
MinhxNguyen7 Nov 14, 2024
691fa90
ci: pipeline for RDS migration (#1029)
ecxyzzy Nov 14, 2024
17fafe7
chore: clean up migration scripts
MinhxNguyen7 Nov 14, 2024
f721579
feat(package.json): ddb to rds command
MinhxNguyen7 Nov 14, 2024
c704230
feat(drizzle): dotenv in config and studio
MinhxNguyen7 Nov 14, 2024
ed8bf5a
feat(drizzle): use arktype instead of dotenv
MinhxNguyen7 Nov 16, 2024
99e7479
refactor(backend): remove mongo
MinhxNguyen7 Nov 16, 2024
68568a4
chore(turbo): rename pipeline to tasks
MinhxNguyen7 Nov 17, 2024
43a9953
chore(deps): update drizzle
MinhxNguyen7 Nov 17, 2024
23e0806
feat(migrate): write one batch at a time
MinhxNguyen7 Nov 17, 2024
6e80fb0
feat(migrate): remove postgres migration from copy script
MinhxNguyen7 Nov 17, 2024
6d8169a
feat(backend): use dotenv in env
MinhxNguyen7 Nov 17, 2024
a124b1e
chore(deps): update drizzle-kit (again)
MinhxNguyen7 Nov 17, 2024
37425f0
fix(ddb): valid user filter with id instead of name
MinhxNguyen7 Nov 17, 2024
0aa4ec3
fix(rds): don't insert courses for schedule with no courses
MinhxNguyen7 Nov 17, 2024
b5af50b
feat(db): import DB_URL from env
MinhxNguyen7 Nov 17, 2024
a0d9362
fix(RDS): transaction param type
MinhxNguyen7 Nov 17, 2024
09cd5b3
feat(migrate): more helpful printout
MinhxNguyen7 Nov 17, 2024
934ba64
feat(RDS): deduplicate courses in schedules
MinhxNguyen7 Nov 17, 2024
4db00fa
feat(migrate): more helpful error message
MinhxNguyen7 Nov 17, 2024
a71b176
deps(package.json): update
MinhxNguyen7 Nov 17, 2024
e07e54b
chore(db): new migrations
MinhxNguyen7 Nov 17, 2024
ba2e1a6
feat(pnpm): more convenenient db scripts
MinhxNguyen7 Nov 17, 2024
3f7f198
chore(schema): new pgTable overload
MinhxNguyen7 Nov 17, 2024
6cee06b
feat(schema): custom events
MinhxNguyen7 Nov 17, 2024
ef13911
fix(schema): custom events
MinhxNguyen7 Nov 17, 2024
96bc950
feat(api): update custom events
MinhxNguyen7 Nov 17, 2024
9d33523
chore(.gitignore): turbo
MinhxNguyen7 Nov 17, 2024
c6d7e7a
Merge branch 'main' into migrate-rds
MinhxNguyen7 Nov 22, 2024
bf9abaa
fix(users): missing imports
MinhxNguyen7 Nov 22, 2024
de96497
fix(schema): proper implementation of constraint array
MinhxNguyen7 Nov 22, 2024
3395b1f
fix(migration): error messages
MinhxNguyen7 Nov 22, 2024
dc63e00
feat(RDS): more specific errors
MinhxNguyen7 Nov 22, 2024
68647bb
feat(RDS): more descriptive messages
MinhxNguyen7 Nov 22, 2024
ae3a27a
feat(migration): always print out success totals
MinhxNguyen7 Nov 22, 2024
6c8255c
Fixed "Error loading grades information" with advanced search (#1043)
IsaacNguyen Nov 22, 2024
ef2ccc5
deps(AA): babel runtime
MinhxNguyen7 Nov 25, 2024
aa051d4
feat(PathNotes): migration warning
MinhxNguyen7 Nov 25, 2024
ac55e11
feat: method for inserting guest user data
ecxyzzy Nov 25, 2024
517c3b7
chore: nuke old searchData
ecxyzzy Nov 25, 2024
f6492df
feat: switch to insert, track skipped users
ecxyzzy Nov 25, 2024
54f75ff
feat(schema): support schedule ordering
MinhxNguyen7 Nov 25, 2024
91e1988
chore(db): schedule index migration
MinhxNguyen7 Nov 25, 2024
51f537f
fix: db => tx (im stupid)
ecxyzzy Nov 25, 2024
74adda3
fix(db): guest user account insert
MinhxNguyen7 Nov 25, 2024
6d4e7bb
feat(migration): time profiling
MinhxNguyen7 Nov 25, 2024
14cc77b
fix(api): drop all schedules before inserting
MinhxNguyen7 Nov 25, 2024
591a9e0
feat(schema): set current schedule to null on delete
MinhxNguyen7 Nov 25, 2024
4609aa6
chore(api): remove legacy DDB view schedule endpoint
MinhxNguyen7 Nov 25, 2024
4220744
feat(api): only write to RDS
MinhxNguyen7 Nov 25, 2024
d6c815a
feat(types): change customEventID to string
MinhxNguyen7 Nov 25, 2024
4a1b624
feat(schema): export type CustomEvent
MinhxNguyen7 Nov 25, 2024
d89444a
feat(schema): export AccountType union
MinhxNguyen7 Nov 25, 2024
bce0d81
feat(api): read from RDS
MinhxNguyen7 Nov 25, 2024
7d87792
Merge branch 'main' into migrate-read
MinhxNguyen7 Dec 7, 2024
97153af
feat(users router): write to both but only fail RDS
MinhxNguyen7 Dec 7, 2024
9dbda0d
chore(users router): remove empty line
MinhxNguyen7 Dec 7, 2024
6301c47
fix(types): customEventID can be number or string
MinhxNguyen7 Dec 7, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions apps/backend/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,10 @@
"arktype": "1.0.14-alpha",
"aws-lambda": "^1.0.7",
"cors": "^2.8.5",
"dotenv": "^16.0.3",
"drizzle-orm": "^0.36.3",
"fuzzysort": "3.1.0",
"envalid": "^7.3.1",
"express": "^4.18.2",
"fuzzysort": "3.1.0",
"postgres": "^3.4.4",
"superjson": "^1.12.3",
"zod": "3.23.8"
Expand All @@ -42,6 +41,7 @@
"@typescript-eslint/eslint-plugin": "^5.52.0",
"@typescript-eslint/parser": "^5.52.0",
"concurrently": "^8.0.1",
"dotenv": "^16.0.3",
"drizzle-kit": "^0.28.1",
"esbuild": "^0.17.19",
"eslint": "^8.34.0",
Expand Down
2 changes: 2 additions & 0 deletions apps/backend/src/db/schema/auth/account.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ export const accountTypeEnum = pgEnum(
accountTypes
);

export type AccountType = typeof accountTypes[number];

// Each user can have multiple accounts, each account is associated with a provider.
// A user without an account is a username-only user.
export const accounts = pgTable(
Expand Down
2 changes: 2 additions & 0 deletions apps/backend/src/db/schema/schedule/custom_event.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,3 +32,5 @@ export const customEvents = pgTable(
lastUpdated: timestamp('last_updated', { withTimezone: true }).defaultNow(),
}
);

export type CustomEvent = typeof customEvents.$inferSelect;
122 changes: 121 additions & 1 deletion apps/backend/src/lib/rds.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,10 @@ import { ShortCourse, ShortCourseSchedule, User, RepeatingCustomEvent } from '@p

import { and, eq } from 'drizzle-orm';
import type { Database } from '$db/index';
import { schedules, users, accounts, coursesInSchedule, customEvents } from '$db/schema';
import {
schedules, users, accounts, coursesInSchedule, customEvents,
AccountType, Schedule, CourseInSchedule, CustomEvent
} from '$db/schema';

type DatabaseOrTransaction = Omit<Database, '$client'>;

Expand Down Expand Up @@ -217,4 +220,121 @@ export class RDS {

await db.transaction(async (tx) => await tx.insert(customEvents).values(dbCustomEvents));
}

static async getGuestUserData(
db: DatabaseOrTransaction, guestId: string
): Promise<User | null> {
const userAndAccount = await RDS.getUserAndAccount(db, 'GUEST', guestId);
if (!userAndAccount) {
return null;
}

const userId = userAndAccount.user.id;

const sectionResults = await db
.select()
.from(schedules)
.where(eq(schedules.userId, userId))
.leftJoin(coursesInSchedule, eq(schedules.id, coursesInSchedule.scheduleId))

const customEventResults = await db
.select()
.from(schedules)
.where(eq(schedules.userId, userId))
.leftJoin(customEvents, eq(schedules.id, customEvents.scheduleId));

const userSchedules = RDS.aggregateUserData(sectionResults, customEventResults);

const scheduleIndex = userAndAccount.user.currentScheduleId
? userSchedules.findIndex((schedule) => schedule.id === userAndAccount.user.currentScheduleId)
: userSchedules.length;

return {
id: guestId,
userData: {
schedules: userSchedules,
scheduleIndex,
},
}
}

private static async getUserAndAccount(
db: DatabaseOrTransaction, accountType: AccountType, providerAccountId: string
) {
const res = await db
.select()
.from(accounts)
.where(and(eq(accounts.accountType, accountType), eq(accounts.providerAccountId, providerAccountId)))
.leftJoin(users, eq(accounts.userId, users.id))
.limit(1);

if (res.length === 0 || res[0].users === null || res[0].accounts === null) {
return null;
}

return { user: res[0].users, account: res[0].accounts };
}

/**
* Aggregates the user's schedule data from the results of two queries.
*/
private static aggregateUserData(
sectionResults: {schedules: Schedule, coursesInSchedule: CourseInSchedule | null}[],
customEventResults: {schedules: Schedule, customEvents: CustomEvent | null}[]
): (ShortCourseSchedule & { id: string, index: number })[] {
// Map from schedule ID to schedule data
const schedulesMapping: Record<string, ShortCourseSchedule & { id: string, index: number }> = {};

// Add courses to schedules
sectionResults.forEach(({ schedules: schedule, coursesInSchedule: course }) => {
const scheduleId = schedule.id;

const scheduleAggregate = schedulesMapping[scheduleId] || {
id: scheduleId,
scheduleName: schedule.name,
scheduleNote: schedule.notes,
courses: [],
customEvents: [],
index: schedule.index,
};

if (course) {
scheduleAggregate.courses.push({
sectionCode: course.sectionCode.toString(),
term: course.term,
color: course.color,
});
}

schedulesMapping[scheduleId] = scheduleAggregate;
});

// Add custom events to schedules
customEventResults.forEach(({ schedules: schedule, customEvents: customEvent }) => {
const scheduleId = schedule.id;
const scheduleAggregate = schedulesMapping[scheduleId] || {
scheduleName: schedule.name,
scheduleNote: schedule.notes,
courses: [],
customEvents: [],
};

if (customEvent) {
scheduleAggregate.customEvents.push({
customEventID: customEvent.id,
title: customEvent.title,
start: customEvent.start,
end: customEvent.end,
days: customEvent.days.split('').map((day) => day === '1'),
color: customEvent.color ?? undefined,
building: customEvent.building ?? undefined,
});
}

schedulesMapping[scheduleId] = scheduleAggregate;
});

// Sort schedules by index
return Object.values(schedulesMapping).sort((a, b) => a.index - b.index);
}
}
53 changes: 17 additions & 36 deletions apps/backend/src/routers/users.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,24 +3,14 @@ import { type } from 'arktype';
import { UserSchema } from '@packages/antalmanac-types';

import { db } from 'src/db';
import { ddbClient } from 'src/db/ddb';
import { mangleDupliateScheduleNames as mangleDuplicateScheduleNames } from 'src/lib/formatting';
import { mangleDupliateScheduleNames } from 'src/lib/formatting';
import { RDS } from 'src/lib/rds';
import { TRPCError } from '@trpc/server';
import { procedure, router } from '../trpc';
import { ddbClient } from '$db/ddb';

const userInputSchema = type([{ userId: 'string' }, '|', { googleId: 'string' }]);

const viewInputSchema = type({
/**
* ID of the user who's requesting to view another user's schedule.
*/
requesterId: 'string',

/**
* ID of the user whose schedule is being requested.
*/
requesteeId: 'string',
});
const userInputSchema = type([{ userId: 'string' }, '|', { googleId: 'string' }]);

const saveInputSchema = type({
/**
Expand All @@ -43,9 +33,12 @@ const usersRouter = router({
*/
getUserData: procedure.input(userInputSchema.assert).query(async ({ input }) => {
if ('googleId' in input) {
return await ddbClient.getGoogleUserData(input.googleId);
throw new TRPCError({
code: 'NOT_IMPLEMENTED',
message: 'Google login not implemented',
})
}
return await ddbClient.getUserData(input.userId);
return await RDS.getGuestUserData(db, input.userId);
}),

/**
Expand All @@ -58,33 +51,21 @@ const usersRouter = router({
const data = input.data;

// Mangle duplicate schedule names
data.userData.schedules = mangleDuplicateScheduleNames(data.userData.schedules);
// Await both, but only throw if DDB save fails.
data.userData.schedules = mangleDupliateScheduleNames(data.userData.schedules);

// Await both, but only throw if RDS save fails.
const results = await Promise.allSettled([
ddbClient.insertItem(data),
ddbClient.insertItem(data)
.catch((error) => console.error('DDB Failed to save user data:', error)),
RDS.upsertGuestUserData(db, data)
.catch((error) => console.error('Failed to upsert user data:', error))
.catch((error) => console.error('RDS Failed to upsert user data:', error))
]);

if (results[0].status === 'rejected') {
throw results[0].reason;
if (results[1].status === 'rejected') {
throw results[1].reason;
}
}
),

/**
* Users can view other users' schedules, even anonymously.
* Visibility permissions are used to determine if a user can view another user's schedule.
*
* Visibility values:
* - (default) private: Only the owner can view and edit.
* - public: Other users can view, but can't edit, i.e. "read-only".
* - open: Anybody can view and edit.
*/
viewUserData: procedure.input(viewInputSchema.assert).query(async ({ input }) => {
return await ddbClient.viewUserData(input.requesterId, input.requesteeId);
}),
});

export default usersRouter;
2 changes: 1 addition & 1 deletion packages/types/src/customevent.ts
MinhxNguyen7 marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ export const RepeatingCustomEventSchema = type({
start: 'string',
end: 'string',
days: 'boolean[]',
customEventID: 'number | parsedNumber', // Unique only within the schedule.
customEventID: 'string | number', // Unique only within the schedule.
'color?': 'string',
'building?': 'string | undefined',
});
Expand Down
6 changes: 3 additions & 3 deletions pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading