Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: prepare embedder for Production #9517

Merged
merged 44 commits into from
Mar 29, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
d86eb2d
feat: support ad-hoc metadata inserts
mattkrick Mar 7, 2024
8628e3e
feat: triggers from metadata to queue
mattkrick Mar 8, 2024
285f69b
feat: move job queue to pg
mattkrick Mar 8, 2024
0b024dd
begin retry
mattkrick Mar 13, 2024
ae6126f
feat: support retries after queue goes dry
mattkrick Mar 13, 2024
9fb9c4f
chore: separate abstract models into their own files
mattkrick Mar 13, 2024
20f5d96
feat: support recursive text splitting for chunks
mattkrick Mar 14, 2024
eff8eb9
validate discussions to only include ended meetings
mattkrick Mar 18, 2024
7ec3d6a
support historical info
mattkrick Mar 18, 2024
49c3eae
Merge branch 'master' into feat/embedder1
mattkrick Mar 18, 2024
46dafac
feat: support listening to jobs from app
mattkrick Mar 19, 2024
47e83d9
feat: support calls from app
mattkrick Mar 19, 2024
7cee59e
merge master
mattkrick Mar 19, 2024
5c96c41
fix migration conflict
mattkrick Mar 19, 2024
49b349b
fix: import history
mattkrick Mar 19, 2024
bf9a379
fix: dataloader mem leak
mattkrick Mar 19, 2024
f7ccdc6
feat: handle stalled jobs
mattkrick Mar 19, 2024
b573083
self-review
mattkrick Mar 19, 2024
ba906e0
remove redlock for non-historical jobs
mattkrick Mar 19, 2024
10953b5
feat: clean comments
mattkrick Mar 19, 2024
7953939
remove pubsub
mattkrick Mar 19, 2024
c2b6753
fix lint errors
mattkrick Mar 19, 2024
8852ac2
fix lint
mattkrick Mar 19, 2024
2ba07c1
cast to any for CI
mattkrick Mar 19, 2024
7bcd083
build embeddings table in CI for the typings
mattkrick Mar 19, 2024
368b34c
codecheck after server starts
mattkrick Mar 19, 2024
1158523
use pgvector in CI
mattkrick Mar 19, 2024
ddfc151
POSTGRES_USE_PGVECTOR='true'
mattkrick Mar 19, 2024
07e0d12
lazy Kysely Codegen
mattkrick Mar 20, 2024
8360b95
fix: release-please and CI pgvector image bump
mattkrick Mar 20, 2024
e74754d
chore: rename files for clarity
mattkrick Mar 20, 2024
e7991d3
rename yaml to yml
mattkrick Mar 21, 2024
ef1d8c4
feat: support priority
mattkrick Mar 26, 2024
59b0647
feat: custom text splitter
mattkrick Mar 27, 2024
3faa442
feat: add language to metadata table
mattkrick Mar 27, 2024
2ab9f61
feat: support multiple workers
mattkrick Mar 27, 2024
7fc4136
feat: set workers via env var
mattkrick Mar 27, 2024
ea32be7
fix: handle shutdown and stalled jobs
mattkrick Mar 27, 2024
bc1cf74
update readme for parabol-ubi
mattkrick Mar 28, 2024
7d82b28
remove unused trigger logic
mattkrick Mar 28, 2024
b609f3e
turn on dev servers
mattkrick Mar 28, 2024
8679d2c
Merge branch 'master' into feat/embedder1
mattkrick Mar 28, 2024
0c80dd6
fix: rename migration
mattkrick Mar 28, 2024
826a044
Merge branch 'master' into feat/embedder1
mattkrick Mar 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix lint errors
Signed-off-by: Matt Krick <matt.krick@gmail.com>
  • Loading branch information
mattkrick committed Mar 19, 2024
commit c2b6753fb26b6cbe5dfeb7b6f8584ff084b3599d
2 changes: 1 addition & 1 deletion packages/embedder/JobQueueStream.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ import numberVectorToString from './indexing/numberVectorToString'
import {updateJobState} from './indexing/updateJobState'

type Job = Selectable<DB['EmbeddingsJobQueue']>
export default class JobQueueStream implements AsyncIterator<Job> {
export default class JobQueueStream implements AsyncIterableIterator<Job> {
private dataLoader: RootDataLoader
private modelManager: ModelManager

Expand Down
4 changes: 1 addition & 3 deletions packages/embedder/ai_models/OpenAIGeneration.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,6 @@ import {
GenerationOptions
} from './AbstractGenerationModel'

const MAX_REQUEST_TIME_S = 3 * 60

export type ModelId = 'gpt-3.5-turbo-0125' | 'gpt-4-turbo-preview'

type OpenAIGenerationOptions = Omit<GenerationOptions, 'topK'>
Expand All @@ -27,7 +25,7 @@ function isValidModelId(object: any): object is ModelId {

export class OpenAIGeneration extends AbstractGenerationModel {
private openAIApi: OpenAI | null
private modelId: ModelId
private modelId!: ModelId

constructor(config: GenerationModelConfig) {
super(config)
Expand Down
6 changes: 3 additions & 3 deletions packages/embedder/ai_models/RecursiveCharacterTextSplitter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ which is longer than the specified ${this.chunkSize}`
// - we have a larger chunk than in the chunk overlap
// - or if we still have any chunks and the length is long
while (total > this.chunkOverlap || (total + _len > this.chunkSize && total > 0)) {
total -= currentDoc[0].length
total -= currentDoc[0]!.length
currentDoc.shift()
}
}
Expand All @@ -70,10 +70,10 @@ which is longer than the specified ${this.chunkSize}`
const separators = this.separators

// Get appropriate separator to use
let separator: string = separators[separators.length - 1]
let separator: string = separators[separators.length - 1]!
let newSeparators: string[] | undefined
for (let i = 0; i < separators.length; i += 1) {
const s = separators[i]
const s = separators[i]!
if (s === '') {
separator = s
break
Expand Down
4 changes: 2 additions & 2 deletions packages/embedder/ai_models/TextEmbeddingsInference.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ export class TextEmbeddingsInference extends AbstractEmbeddingsModel {
return new Error(`listOfTokens list length !== 1 (length: ${listOfTokens.length})`)
return listOfTokens[0]
} catch (e) {
return e instanceof Error ? e : new Error(e)
return e instanceof Error ? e : new Error(typeof e === 'string' ? e : 'Unknown error')
}
}
public async getEmbedding(content: string) {
Expand All @@ -73,7 +73,7 @@ export class TextEmbeddingsInference extends AbstractEmbeddingsModel {
return listOfVectors[0]
} catch (e) {
console.log(`TextEmbeddingsInference.getEmbeddings() timeout: `, e)
return e instanceof Error ? e : new Error(e || 'Unknown Error')
return e instanceof Error ? e : new Error(typeof e === 'string' ? e : 'Unknown error')
}
}

Expand Down
4 changes: 2 additions & 2 deletions packages/embedder/mergeAsyncIterators.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ type UnIt<T extends AsyncIterator<any>> = UnYield<Awaited<ReturnType<T['next']>>
// Promise.race has a memory leak
// To avoid: https://github.com/tc39/proposal-async-iterator-helpers/issues/15#issuecomment-1937011820
export function mergeAsyncIterators<
T1 extends AsyncIterator<any>,
T2 extends AsyncIterator<any>,
T1 extends AsyncIterableIterator<any>,
T2 extends AsyncIterableIterator<any>,
K1 = UnIt<T1>,
K2 = UnIt<T2>
>(iterables: [T1, T2]) {
Expand Down
2 changes: 1 addition & 1 deletion packages/gql-executor/RedisStream.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import RedisInstance from 'parabol-server/utils/RedisInstance'
type MessageValue = [prop: string, stringifiedData: string]
type Message = [messageId: string, value: MessageValue]
type XReadGroupRes = [streamName: string, messages: Message[]]
export default class RedisStream implements AsyncIterator<string> {
export default class RedisStream implements AsyncIterableIterator<string> {
private stream: string
private consumerGroup: string
// xreadgroup blocks until a response is received, so this needs its own connection
Expand Down
Loading