POC: migration of system_package data

[psegedy]

Dan Kuc helped me with the migration and he suggested to do it in the DB to avoid data copying between server and client and to remove foreign keys to speed it up. I tried migration of 1 partition with generated data (~275GB system_package table) and it took ~ 1 minute to copy data.

My proposal how to migrate the data is:

1. scale down evaluator so we are not writing to system_package or system_package2
2. set IMAGE_TAG_DATABASE_ADMIN to the latest version, DB will be migrated in the init container but manager will run with the old image which will still use system_package table for API response
3. use new image for IMAGE_TAG_ADMIN and run the migration job
4. deploy the latest image tag for all services and scale up evaluator after migration finishes

@MichaelMraka any suggestions? should we try to run the job per partition?

Secure Coding Practices Checklist GitHub Link

• https://github.com/RedHatInsights/secure-coding-checklist

Secure Coding Checklist

• Input Validation
• Output Encoding
• Authentication and Password Management
• Session Management
• Access Control
• Cryptographic Practices
• Error Handling and Logging
• Data Protection
• Communication Security
• System Configuration
• Database Security
• File Management
• Memory Management
• General Coding Practices

[jira-linking]

Commits missing Jira IDs:
6320227
f2d85d1

[MichaelMraka]

That sounds, good. By 'run per partition' you mean sorting accounts by partitions or create a different jobs for individual partitions?

[codecov-commenter]

Codecov Report

Attention: 103 lines in your changes are missing coverage. Please review.

Comparison is base (28f08b2) 62.49% compared to head (f2d85d1) 62.03%.
Report is 88 commits behind head on poc.

Files	Patch %	Lines
manager/middlewares/limits.go	0.00%	16 Missing ⚠️
evaluator/evaluate_packages.go	85.54%	7 Missing and 5 partials ⚠️
manager/controllers/systems_advisories_view.go	77.35%	10 Missing and 2 partials ⚠️
evaluator/evaluate_advisories.go	79.59%	6 Missing and 4 partials ⚠️
base/utils/core.go	37.50%	5 Missing ⚠️
base/database/testing.go	0.00%	4 Missing ⚠️
base/utils/config.go	75.00%	3 Missing and 1 partial ⚠️
listener/upload.go	88.88%	2 Missing and 2 partials ⚠️
manager/controllers/advisory_systems_v3.go	33.33%	2 Missing and 2 partials ⚠️
manager/controllers/package_systems.go	71.42%	2 Missing and 2 partials ⚠️
... and 14 more

Additional details and impacted files

@@            Coverage Diff             @@
##              poc    #1350      +/-   ##
==========================================
- Coverage   62.49%   62.03%   -0.47%     
==========================================
  Files         106      108       +2     
  Lines        6684     6811     +127     
==========================================
+ Hits         4177     4225      +48     
- Misses       1985     2052      +67     
- Partials      522      534      +12     

Flag	Coverage Δ
unittests	62.03% <77.75%> (-0.47%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[psegedy]

I meant sorting by partition, but I'm not sure how would it work with the Dan's logic for incrementing account ids, it would need to be rewritten.