Alembic migrations and mod many methods (#37)

* Added new ModelField to allow global import * Added new insert_many method for bulk inserts. Added delete_many for bulk deletions. Added ModelField to allow for sqlalchemy_type defitions in models. Improved default, unique, primary_key, optional/required detection when building sqlalchemy.Column's using new ModelField / Unique, PrimaryKey arguments * Enabled executate_many for use with insert/delete many querries. Improved automatic migrations of non-sqlite with alembic for removal of columns / adding columns / moding columns preventing full table swaps. Added support for alembic migrations which will strictly require require alembic driven migrations * replaced imp with -> importlib. Improved DatabaseModel.OR class method, capable of grouping conditionals via AND based on list grouping [[This,and,This],or,[This,and,]] * Added database helper method alembic_migrate which orchestrates what alembic env.py would be expected to into single callable. Database.create() can be used outside an event loop when use_alembic=True as migrations should be driven by alembic. Pydbantic driven migrations now leverage alembic to reduce lift-and-shifting operations as much as possible * added .inside() example usage * removed metadata drop at end of test * added delete_many test example * added tests for new OR group conditional support * removed staged models no longer used by migration tests * added docs for alembic setup * added docs examples for insert_many, delete_many methods * updated migration tests - models that migrate are visible in test and are better documented * updated migration test range * added alembic==1.8.1 * Added new ModelField to allow global import * Added new insert_many method for bulk inserts. Added delete_many for bulk deletions. Added ModelField to allow for sqlalchemy_type defitions in models. Improved default, unique, primary_key, optional/required detection when building sqlalchemy.Column's using new ModelField / Unique, PrimaryKey arguments * Enabled executate_many for use with insert/delete many querries. Improved automatic migrations of non-sqlite with alembic for removal of columns / adding columns / moding columns preventing full table swaps. Added support for alembic migrations which will strictly require require alembic driven migrations * replaced imp with -> importlib. Improved DatabaseModel.OR class method, capable of grouping conditionals via AND based on list grouping [[This,and,This],or,[This,and,]] * Added database helper method alembic_migrate which orchestrates what alembic env.py would be expected to into single callable. Database.create() can be used outside an event loop when use_alembic=True as migrations should be driven by alembic. Pydbantic driven migrations now leverage alembic to reduce lift-and-shifting operations as much as possible * added .inside() example usage * removed metadata drop at end of test * added delete_many test example * added tests for new OR group conditional support * removed staged models no longer used by migration tests * added docs for alembic setup * added docs examples for insert_many, delete_many methods * updated migration tests - models that migrate are visible in test and are better documented * updated migration test range * added alembic==0.8.1 * corrected mysql type setting on alembic.alter_column within automigration. Updated mysql / postgres credential config in tests * corrected mysql type setting on alembic.alter_column within automigration. Updated mysql / postgres credential config in tests * corrected mysql type setting on alembic.alter_column within automigration. Updated mysql / postgres credential config in tests * corrected mysql type setting on alembic.alter_column within automigration. Updated mysql / postgres credential config in tests * removed breakpoint Co-authored-by: Codemation <[email protected]>
codemation · Dec 13, 2022 · 22f2908 · 22f2908
1 parent 86e912c
commit 22f2908
Show file tree

Hide file tree

Showing 25 changed files with 657 additions and 446 deletions.
diff --git a/.github/workflows/main.yaml b/.github/workflows/main.yaml
@@ -22,7 +22,8 @@ jobs:
         image: postgres
         # Provide the password for postgres
         env:
-          POSTGRES_PASSWORD: abcd1234
+          POSTGRES_PASSWORD: postgres
+          POSTGRES_USER: postgres
           POSTGRES_DB: database
         ports:
           - 5432:5432
@@ -65,7 +66,7 @@ jobs:
             pytest tests/test_query_caching.py -s -x;
             pytest tests/test_querying.py -s -x;
             pytest tests/test_query_no_caching.py -s -x;
-            for mig_test in {0..9}; do 
+            for mig_test in {0..8}; do 
               pytest tests/migrations/test_model_migration_$mig_test.py;
             done
           done
@@ -91,7 +92,7 @@ jobs:
         image: mysql
         # Provide the password for postgres
         env:
-          MYSQL_USER: josh
+          MYSQL_USER: mysqltestuser
           MYSQL_PASSWORD: abcd1234
           MYSQL_ROOT_PASSWORD: abcd1234
           MYSQL_DATABASE: database
@@ -130,7 +131,7 @@ jobs:
             pytest tests/test_query_caching.py -s -x;
             pytest tests/test_querying.py -s -x;
             pytest tests/test_query_no_caching.py -s -x;
-            for mig_test in {0..9}; do 
+            for mig_test in {0..8}; do 
               pytest tests/migrations/test_model_migration_$mig_test.py;
             done
           done
@@ -188,7 +189,7 @@ jobs:
             pytest tests/test_query_caching.py -s -x;
             pytest tests/test_querying.py -s -x;
             pytest tests/test_query_no_caching.py -s -x;
-            for mig_test in {0..9}; do 
+            for mig_test in {0..8}; do 
               pytest tests/migrations/test_model_migration_$mig_test.py;
             done
           done
diff --git a/.github/workflows/package.yaml b/.github/workflows/package.yaml
@@ -26,7 +26,7 @@ jobs:
         image: mysql
         # Provide the password for postgres
         env:
-          MYSQL_USER: josh
+          MYSQL_USER: mysqltestuser
           MYSQL_PASSWORD: abcd1234
           MYSQL_ROOT_PASSWORD: abcd1234
           MYSQL_DATABASE: database
@@ -65,7 +65,7 @@ jobs:
             pytest tests/test_query_caching.py -s -x;
             pytest tests/test_querying.py -s -x;
             pytest tests/test_query_no_caching.py -s -x;
-            for mig_test in {0..9}; do 
+            for mig_test in {0..8}; do 
               pytest tests/migrations/test_model_migration_$mig_test.py;
             done
           done
@@ -123,7 +123,7 @@ jobs:
             pytest tests/test_query_caching.py -s -x;
             pytest tests/test_querying.py -s -x;
             pytest tests/test_query_no_caching.py -s -x;
-            for mig_test in {0..9}; do 
+            for mig_test in {0..8}; do 
               pytest tests/migrations/test_model_migration_$mig_test.py;
             done
           done
@@ -149,7 +149,8 @@ jobs:
         image: postgres
         # Provide the password for postgres
         env:
-          POSTGRES_PASSWORD: abcd1234
+          POSTGRES_PASSWORD: postgres
+          POSTGRES_USER: postgres
           POSTGRES_DB: database
         ports:
           - 5432:5432
@@ -192,7 +193,7 @@ jobs:
             pytest tests/test_query_caching.py -s -x;
             pytest tests/test_querying.py -s -x;
             pytest tests/test_query_no_caching.py -s -x;
-            for mig_test in {0..9}; do 
+            for mig_test in {0..8}; do 
               pytest tests/migrations/test_model_migration_$mig_test.py;
             done
           done
diff --git a/docs/alembic.md b/docs/alembic.md
@@ -0,0 +1,148 @@
+## Migrations Using Alembic
+Although Pydbantic can handle migrations automatically, alembic can also be used if preferred. Configuration can be done in a few easy steps
+
+### Install alembic
+```bash
+pip install alembic
+```
+### Define Models & DB instance
+```python
+#models.py
+from pydbantic import DataBaseModel, PrimaryKey, Default
+
+def time_now_str():
+    return datetime.now().isoformat()
+
+def stringify_uuid():
+    return str(uuid.uuid4())
+
+class Employee(DataBaseModel):
+    id: str = PrimaryKey(default=stringify_uuid)
+    salary: float
+    is_employed: bool
+    date_employed: str = Default(default=time_now_str)
+```
+
+### Connect with Database
+```python
+#db.py
+from pydbantic import Database
+from models import Employee
+
+db = Database.create(
+    'sqlite:///company.db',
+    tables=[Employee],
+    use_alembic=True
+)
+```
+### Initialze alembic
+```bash
+alembic init migrations
+```
+Within the current directory, alembic will create a `migrations` folder that will store `versions` and its `env.py` that will need to updated. We will remove most of the boiler plate code and simply `import db` and use `db.alembic_migrate()`.
+
+
+Upate `migrations/env.py` file
+```python
+#migrations/env.py
+from alembic import context
+from db import db
+
+def run_migrations_offline() -> None:
+    db.alembic_migrate()
+
+def run_migrations_online() -> None:
+    db.alembic_migrate()
+
+if context.is_offline_mode():
+    run_migrations_offline()
+else:
+    run_migrations_online()
+```
+
+### Creating the first migration file
+Until now, we have just told alembic where to store our migrations, and where our db is configured. We still have not created any database tables to match our `Employee` model. 
+```bash
+alembic revision -m "init_migration" --autogenerate
+```
+
+Alembic is capable of detecting schema changes and often can do most of the work to build your migrations. Notice the new file in `migrations/versions/` matching the `-m "init_migration"` commit message.
+
+
+### Trigger Migration
+The final step once we have created a migration file is to trigger the migration. The instructions defined in the latest `migrations/versions` will be followed. 
+
+```bash
+alembic upgrade head
+```
+
+### Run Application
+Now we are all set, we can Create new persistent instances of our model and trust they are safely stored. 
+
+```python
+#app.py
+import asyncio
+from db import db
+from models import Employee
+
+async def main():
+    await Employee.create(
+        salary=10000,
+        is_employed=True
+    )
+    all_employees = await Employee.all()
+    print(all_employees)
+
+asyncio.run(main())
+```
+
+### Adding a new Model
+
+```python
+# models.py
+import uuid
+from datetime import datetime
+from typing import Optional, Union
+from pydantic import BaseModel
+from pydbantic import DataBaseModel, PrimaryKey, Default
+
+def time_now_str():
+    return datetime.now().isoformat()
+
+def stringify_uuid():
+    return str(uuid.uuid4())
+
+class Positions(DataBaseModel):
+    name: str = PrimaryKey()
+    level: int = 4
+
+class Employee(DataBaseModel):
+    id: str = PrimaryKey(default=stringify_uuid)
+    salary: float
+    is_employed: bool
+    date_employed: str = Default(default=time_now_str)
+    position: Union[Positions, None] = Positions(name='Manager')
+```
+
+Connect to database
+
+```python
+#db.py
+from pydbantic import Database
+from models import Employee, Positions
+
+db = Database.create(
+    'sqlite:///company.db',
+    tables=[Employee, Positions],
+    use_alembic=True
+)
+```
+Create new revision
+
+```bash
+alembic revision -m "added Positions" --autogenerate
+```
+Migrate!
+```bash
+alembic upgrade head
+```
diff --git a/docs/model-usage.md b/docs/model-usage.md
@@ -65,6 +65,20 @@ hr_department = Department(
 
 await hr_department.insert()
 ```
+#### Create Multiple Models 
+```python
+departments = [
+    Department(
+        id=f'{department}_1234',
+        name=f'{department}'
+        company='abc-company',
+        is_sensitive=True
+    ) for department in ['hr','sales', 'marketing']
+]
+
+await Department.insert_many(departments)
+```
+
 
 #### Create Model & Save / Update Later
 
@@ -234,17 +248,25 @@ for employee in all_employees:
     `.save()` can also be used, but first verified object existence before attempting save, while `.update()` does not verify before attempting to update. 
 
 ### Model Usage - Deleting
+
+#### Single 
 Much like updates, `DataBaseModel` objects can only be deleted by directly calling the `.delete()` method of an object instance. 
 
+
 ```python
 all_employees = await Employees.all()
 
 # delete latest employee
 await all_employees[-1].delete()
 ```
 
-!!! WARNING
-    Deleted objects which are depended on by other `DataBaseModel` are <u>NOT</u> deleted, as no strict table relationships exist between `DataBaseModel`. This may be changed later. 
+#### Multiple
+```python
+terminated_employees = await Employees.filter(
+    Employees.contains('name', 'terminated')
+)
+await Employees.delete_many(terminated_employees)
+```
 
 
 ### Models with arrays of Foreign Objects

diff --git a/mkdocs.yml b/mkdocs.yml
@@ -12,4 +12,5 @@ nav:
   - Tutorials: tutorials.md
   - Model Usage: model-usage.md
   - Database Usage: database-usage.md
+  - Migrations with Alembic: alembic.md
   - Redis: redis-usage.md
diff --git a/pydbantic/__init__.py b/pydbantic/__init__.py
@@ -1,2 +1,8 @@
 from .database import Database
-from .core import DataBaseModel, PrimaryKey, Default, Unique
+from .core import (
+    DataBaseModel,
+    PrimaryKey,
+    Default,
+    Unique,
+    ModelField,
+)