Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Interest Rate and Bearing models #329

Merged

Conversation

MooreTheAnalyst
Copy link
Contributor

@MooreTheAnalyst MooreTheAnalyst commented Nov 26, 2024

closes #309

Hi, there's a failing migration which is affecting alchemy's ability to generate newer migrations.

(data-handler-py3.12) @MooreTheAnalyst ➜ /workspaces/derisk-research/apps (interest-rate-bearing) $ alembic -c data_handler/alembic.ini upgrade head
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> d2fa8201b04a, Add LoanState model
INFO  [alembic.runtime.migration] Running upgrade d2fa8201b04a -> e4c7f75ff173, add liquidable debt model
INFO  [alembic.runtime.migration] Running upgrade e4c7f75ff173 -> 593bb0a7d06b, add order_book and interest_rate
INFO  [alembic.runtime.migration] Running upgrade 593bb0a7d06b -> e813bfbd573f, fix merge conflict
INFO  [alembic.runtime.migration] Running upgrade  -> 509baf9251f2, add health_ratio_level model
INFO  [alembic.runtime.migration] Running upgrade 509baf9251f2 -> efd02f93572b, health ratio model adjusted
INFO  [alembic.runtime.migration] Running upgrade 509baf9251f2, e813bfbd573f -> 31430d42d46a, Merge heads into a single branch
INFO  [alembic.runtime.migration] Running upgrade 31430d42d46a -> 4280b8a75614, add collateral debt zklend model
INFO  [alembic.runtime.migration] Running upgrade 4280b8a75614, efd02f93572b -> 70353336c9ef, empty message
INFO  [alembic.runtime.migration] Running upgrade 70353336c9ef -> fafbe0720bc8, add hashtack collateral debt model
INFO  [alembic.runtime.migration] Running upgrade fafbe0720bc8 -> c65693d53658, add veriosn field for HashtackCollateralDebt
INFO  [alembic.runtime.migration] Running upgrade c65693d53658 -> 64a870953fa5, add constraint for loan_state'
INFO  [alembic.runtime.migration] Running upgrade 64a870953fa5 -> d3691f6e8c5a, add zklend events
INFO  [alembic.runtime.migration] Running upgrade d3691f6e8c5a -> 163f616930fb, migration for BearingCollateralBurnEventModel and DebtTransferEventModel
Traceback (most recent call last):
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/bin/alembic", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/config.py", line 636, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/config.py", line 626, in main
    self.run_cmd(cfg, options)
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/config.py", line 603, in run_cmd
    fn(
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/command.py", line 406, in upgrade
    script.run_env()
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/script/base.py", line 586, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file
    module = load_module_py(module_id, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 994, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/workspaces/derisk-research/apps/data_handler/migrations/env.py", line 66, in <module>
    run_migrations_online()
  File "/workspaces/derisk-research/apps/data_handler/migrations/env.py", line 60, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/runtime/environment.py", line 946, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/runtime/migration.py", line 628, in run_migrations
    step.migration_fn(**kw)
  File "/workspaces/derisk-research/apps/data_handler/migrations/versions/163f616930fb_migration_for_.py", line 28, in upgrade
    sa.Column('protocol_id', sqlalchemy_utils.types.choice.ChoiceType(), nullable=False),
                             ^^^^^^^^^^^^^^^^
NameError: name 'sqlalchemy_utils' is not defined

This is the reason why when I run the migration command, an error occurs.

(data-handler-py3.12) @MooreTheAnalyst ➜ /workspaces/derisk-research/apps (interest-rate-bearing) $ alembic -c data_handler/alembic.ini revision --autogenerate -m "interest rate and bearing mint"
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
ERROR [alembic.util.messaging] Target database is not up to date.
  FAILED: Target database is not up to date.
alembic -c data_handler/alembic.ini revision --autogenerate -m "interest rate and bearing mint"

@djeck1432
Copy link
Collaborator

@MooreTheAnalyst Did you try to google what does it mean this error?

@djeck1432 djeck1432 self-requested a review November 26, 2024 14:37
@MooreTheAnalyst
Copy link
Contributor Author

Hi @djeck1432 for a migration to be generated, the db has to be up to date, this is done by running the alembic upgrade command, alembic stamp head can also be used to achieve that, which I've done now.

@djeck1432
Copy link
Collaborator

alembic upgrade head right command

@MooreTheAnalyst
Copy link
Contributor Author

Hi @djeck1432 this is the result of alembic upgrade head

(data-handler-py3.12) @MooreTheAnalyst ➜ /workspaces/derisk-research/apps (master) $ alembic -c data_handler/alembic.ini upgrade head
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> d2fa8201b04a, Add LoanState model
INFO  [alembic.runtime.migration] Running upgrade d2fa8201b04a -> e4c7f75ff173, add liquidable debt model
INFO  [alembic.runtime.migration] Running upgrade e4c7f75ff173 -> 593bb0a7d06b, add order_book and interest_rate
INFO  [alembic.runtime.migration] Running upgrade 593bb0a7d06b -> e813bfbd573f, fix merge conflict
INFO  [alembic.runtime.migration] Running upgrade  -> 509baf9251f2, add health_ratio_level model
INFO  [alembic.runtime.migration] Running upgrade 509baf9251f2 -> efd02f93572b, health ratio model adjusted
INFO  [alembic.runtime.migration] Running upgrade 509baf9251f2, e813bfbd573f -> 31430d42d46a, Merge heads into a single branch
INFO  [alembic.runtime.migration] Running upgrade 31430d42d46a -> 4280b8a75614, add collateral debt zklend model
INFO  [alembic.runtime.migration] Running upgrade 4280b8a75614, efd02f93572b -> 70353336c9ef, empty message
INFO  [alembic.runtime.migration] Running upgrade 70353336c9ef -> fafbe0720bc8, add hashtack collateral debt model
INFO  [alembic.runtime.migration] Running upgrade fafbe0720bc8 -> c65693d53658, add veriosn field for HashtackCollateralDebt
INFO  [alembic.runtime.migration] Running upgrade c65693d53658 -> 64a870953fa5, add constraint for loan_state'
INFO  [alembic.runtime.migration] Running upgrade 64a870953fa5 -> d3691f6e8c5a, add zklend events
INFO  [alembic.runtime.migration] Running upgrade d3691f6e8c5a -> 163f616930fb, migration for BearingCollateralBurnEventModel and DebtTransferEventModel
Traceback (most recent call last):
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/bin/alembic", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/config.py", line 636, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/config.py", line 626, in main
    self.run_cmd(cfg, options)
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/config.py", line 603, in run_cmd
    fn(
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/command.py", line 406, in upgrade
    script.run_env()
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/script/base.py", line 586, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file
    module = load_module_py(module_id, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 994, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/workspaces/derisk-research/apps/data_handler/migrations/env.py", line 66, in <module>
    run_migrations_online()
  File "/workspaces/derisk-research/apps/data_handler/migrations/env.py", line 60, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/runtime/environment.py", line 946, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/runtime/migration.py", line 628, in run_migrations
    step.migration_fn(**kw)
  File "/workspaces/derisk-research/apps/data_handler/migrations/versions/163f616930fb_migration_for_.py", line 28, in upgrade
    sa.Column('protocol_id', sqlalchemy_utils.types.choice.ChoiceType(), nullable=False),
                             ^^^^^^^^^^^^^^^^
NameError: name 'sqlalchemy_utils' is not defined

@djeck1432
Copy link
Collaborator

did you run poetry shell before running migrations?

@MooreTheAnalyst
Copy link
Contributor Author

did you run poetry shell before running migrations?

Yeah, it's the logs pasted from the terminal includes that I'm in the poetry shell environment.

@MooreTheAnalyst ➜ /workspaces/derisk-research (master) $ docker-compose -f devops/dev/docker-compose.db.yaml up -d --remove-orphans
WARN[0000] /workspaces/derisk-research/devops/dev/docker-compose.db.yaml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
[+] Running 3/3
 ✔ Network dev_default      Created                                                                                   0.1s 
 ✔ Container dev-backend-1  Removed                                                                                  10.4s 
 ✔ Container postgres_dev   Started                                                                                   0.5s 
@MooreTheAnalyst ➜ /workspaces/derisk-research (master) $ cd apps/data_handler/
@MooreTheAnalyst ➜ /workspaces/derisk-research/apps/data_handler (master) $ poetry install
Installing dependencies from lock file

No dependencies to install or update
@MooreTheAnalyst ➜ /workspaces/derisk-research/apps/data_handler (master) $ poetry shell
Spawning shell within /home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12
. /home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/bin/activate
@MooreTheAnalyst ➜ /workspaces/derisk-research/apps/data_handler (master) $ . /home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/bin/activate
(data-handler-py3.12) @MooreTheAnalyst ➜ /workspaces/derisk-research/apps/data_handler (master) $ cd ..
(data-handler-py3.12) @MooreTheAnalyst ➜ /workspaces/derisk-research/apps (master) $ alembic -c data_handler/alembic.ini upgrade head
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> d2fa8201b04a, Add LoanState model
INFO  [alembic.runtime.migration] Running upgrade d2fa8201b04a -> e4c7f75ff173, add liquidable debt model
INFO  [alembic.runtime.migration] Running upgrade e4c7f75ff173 -> 593bb0a7d06b, add order_book and interest_rate
INFO  [alembic.runtime.migration] Running upgrade 593bb0a7d06b -> e813bfbd573f, fix merge conflict
INFO  [alembic.runtime.migration] Running upgrade  -> 509baf9251f2, add health_ratio_level model
INFO  [alembic.runtime.migration] Running upgrade 509baf9251f2 -> efd02f93572b, health ratio model adjusted
INFO  [alembic.runtime.migration] Running upgrade 509baf9251f2, e813bfbd573f -> 31430d42d46a, Merge heads into a single branch
INFO  [alembic.runtime.migration] Running upgrade 31430d42d46a -> 4280b8a75614, add collateral debt zklend model
INFO  [alembic.runtime.migration] Running upgrade 4280b8a75614, efd02f93572b -> 70353336c9ef, empty message
INFO  [alembic.runtime.migration] Running upgrade 70353336c9ef -> fafbe0720bc8, add hashtack collateral debt model
INFO  [alembic.runtime.migration] Running upgrade fafbe0720bc8 -> c65693d53658, add veriosn field for HashtackCollateralDebt
INFO  [alembic.runtime.migration] Running upgrade c65693d53658 -> 64a870953fa5, add constraint for loan_state'
INFO  [alembic.runtime.migration] Running upgrade 64a870953fa5 -> d3691f6e8c5a, add zklend events
INFO  [alembic.runtime.migration] Running upgrade d3691f6e8c5a -> 163f616930fb, migration for BearingCollateralBurnEventModel and DebtTransferEventModel
Traceback (most recent call last):
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/bin/alembic", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/config.py", line 636, in main
    CommandLine(prog=prog).main(argv=argv)
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/config.py", line 626, in main
    self.run_cmd(cfg, options)
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/config.py", line 603, in run_cmd
    fn(
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/command.py", line 406, in upgrade
    script.run_env()
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/script/base.py", line 586, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 95, in load_python_file
    module = load_module_py(module_id, path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/util/pyfiles.py", line 113, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 994, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/workspaces/derisk-research/apps/data_handler/migrations/env.py", line 66, in <module>
    run_migrations_online()
  File "/workspaces/derisk-research/apps/data_handler/migrations/env.py", line 60, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/runtime/environment.py", line 946, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/home/codespace/.cache/pypoetry/virtualenvs/data-handler-qZQs71rz-py3.12/lib/python3.12/site-packages/alembic/runtime/migration.py", line 628, in run_migrations
    step.migration_fn(**kw)
  File "/workspaces/derisk-research/apps/data_handler/migrations/versions/163f616930fb_migration_for_.py", line 28, in upgrade
    sa.Column('protocol_id', sqlalchemy_utils.types.choice.ChoiceType(), nullable=False),
                             ^^^^^^^^^^^^^^^^
NameError: name 'sqlalchemy_utils' is not defined
(data-handler-py3.12) @MooreTheAnalyst ➜ /workspaces/derisk-research/apps (master) $

Notice the (data-handler-py3.12) prefix before my system name @MooreTheAnalyst after I've activated the poetry shell environment.

@djeck1432
Copy link
Collaborator

@MooreTheAnalyst text me in DM in telegram

Copy link
Collaborator

@djeck1432 djeck1432 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

@djeck1432 djeck1432 merged commit 676badd into CarmineOptions:master Nov 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Interest Rate and Bearing] Create db models for nostra events
2 participants