Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ingest logs fills up memory with error logs #1164

Closed
2 tasks
MalinAhlberg opened this issue Dec 6, 2024 · 1 comment · Fixed by #1189
Closed
2 tasks

ingest logs fills up memory with error logs #1164

MalinAhlberg opened this issue Dec 6, 2024 · 1 comment · Fixed by #1189
Labels
bug Something isn't working

Comments

@MalinAhlberg
Copy link
Contributor

Describe the bug
When running the data_loader in starter-kit-storage-and-interfaces, no key hash is inserted before ingestion starts. That makes ingest loops infinitely, eating all memory while producing logs like this:

ingest  | {"level":"info","msg":"Received work (corr-id: 5dd81f8c-9c5c-46a1-ae90-09d13da1dac6, filepath: htsnexus_test_NA12878.bam.c4gh, user: [email protected])","time":"2024-12-06T08:04:56Z"}
ingest  | {"level":"error","msg":"Key hash a7db730b021d942d12ac2a8d02ba77c8c91fe4ca61681f9e1450536c546ae72a could not be set for fileID 5dd81f8c-9c5c-46a1-ae90-09d13da1dac6: (pq: insert or update on table \"files\" violates foreign key constraint \"files_key_hash_fkey\")","time":"2024-12-06T08:04:57Z"}

Steps to reproduce
Run docker compose -f docker-compose-demo.yml up -d in starter-kit-storage-and-interfaces, or run the steps manually without the starter kit.

Expected behavior

  • One error is reported
  • Logs are not exploding

Additional context
See also #1037 and #1027

Estimation of size: medium

Estimation of priority: high

@MalinAhlberg MalinAhlberg added the bug Something isn't working label Dec 6, 2024
@jbygdell
Copy link
Collaborator

jbygdell commented Dec 6, 2024

So the reason for this is that ingest will just Nack any DB related error to cope with connection problems with the DB.

The fix for this issue will be to add a Startup check that ensures that the DB has a keyHash registered before it can start reading messages from the MQ.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants