Skip to content
This repository has been archived by the owner on Dec 16, 2019. It is now read-only.

Database IDs and Object Storage IDs may run out of sync #329

Open
dtitov opened this issue Jul 26, 2018 · 2 comments
Open

Database IDs and Object Storage IDs may run out of sync #329

dtitov opened this issue Jul 26, 2018 · 2 comments

Comments

@dtitov
Copy link

dtitov commented Jul 26, 2018

Test case:

  1. Deploy LocalEGA.
  2. Ingest a file. The record about a file appears in the DB with ID 1. Also, this file lands to the S3 as an object with the ID 1, because the ID of an object is taken from the DB.
  3. Recreate DB instance. Not just restart it, but recreate.
  4. Ingest one more file.
    E.R.: File is put to S3 as a separate file.
    A.R.: Existing file with ID 1 is overwritten in S3, because files table is dropped in DB and IDs sequence starts over from 1.

This happens because of several factors.

  1. files table:
CREATE TABLE files (
        id             SERIAL, PRIMARY KEY(id), UNIQUE (id),
...

ID here is a simple auto-increment.
2. Database in our deployments doesn't have persistent volume (like inbox or s3). So each time one recreates the DB service, the tables are erased and numeration starts from the beginning.
3. File are put to the S3 by database IDs.

Possible fixes:

  1. Use something like UUID-generation for IDs (e.g. uuid-ossp Postgress extension).
  2. Introduce a persistent volume for database so the tables survive service recreation.
  3. Don't put files to Object Storage by their database IDs, but better by some randomly generated UUID, along with inserting this UUID to the database to the corresponding file-record.
@silverdaz
Copy link
Contributor

Possible solution is inside #339

@blankdots blankdots modified the milestones: Sprint 33, Sprint 36 Sep 6, 2018
@juhtornr
Copy link
Collaborator

Is this still valid?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants