Integration/fast agave (#150)

* Update dependencies in requirements files

* Updated __get_value method to convert enum values to string format

* Update GitHub Actions workflow to include Codecov token

* Bump version to 1.0.0.dev1 in version.py

* Refactor project structure

- Refactored import statements in various modules to point to the new chalice_support package

* update import paths in tests and blueprints to reflect new structure

* Enhance generic_query function to accept excluded fields and update decorators to skip private attributes.

* refactor: move tests to dedicated test directory

* feat: Complete integration of fast_agave

* Refactor: Consolidate import statements

* Refactor: Update Makefile and improve description formatting in RestApiBlueprint

* Refactor: Update import paths in resources and blueprints to reflect new structure

* Refactor: Update setup.py to use extras_require for dependencies and add installation validation in agave module

* Enhance setup.py to enforce installation with extras

* Refactor setup.py to improve installation validation by allowing specific commands to skip checks for required extras

* Update version to 1.1.0.dev1 in version.py

* Update cuenca-validations version to 2.0.0.dev7 in setup.py

* Update version to 1.1.0.dev2 in version.py

* Update version to 1.1.0.dev3 and modify installation validation in setup.py to issue a warning instead of an error for missing extras

* Enhance import error handling to guide users on required installation options

* Refactor: Remove chalice_support and fastapi_support module and update imports to use chalice and fastapi directly

* Fix: Rename 'fast_support' to 'fastapi' in setup.py

* Update Python version and dependencies; refactor Makefile and requirements files

* Update Python version in setup.py and workflows

* Update version to 1.1.0.dev5 in version.py

* Update dependencies in requirements and setup files

* Refactor: Replace 'dict()' method with 'model_dump()'

* Fix: Change platform_id in UserQuery to be optional

* Refactor: Change type hints from 'Dict' to 'dict'

* Refactor: Update type hints from 'List' to 'list'

* Update version to 1.1.0.dev7 in version.py

* Add logging to SQS task processing in sqs_tasks.py

* Update version to 1.1.0.dev8 in version.py

* Add logger middlewares and include sensitive fields tracking.

* Update version to 1.1.0.dev9 in version.py

* Refactor: Update type hint for parse_body function to use Union for better clarity

* Enhance SQS task logging by adding detailed task information and result parsing

* Update version to 1.1.0.dev10 in version.py

* Lint

* Update obfuscation logic to include model names for sensitive fields

* Update cuenca-validations version to 2.0.0.dev13 in requirements.txt and setup.py

* Update version to 1.1.0.dev11 in version.py

* Replace mongoengine components with mongoengine_plus

* Refactor error handling: Rename FastAgaveError and related classes to AgaveError

* Remove sensitive data handling from middleware and related components

* Move 'task' package out of FastAPI and update imports

* Move aiobotocore and types-aiobotocore-sqs to 'tasks'

* Refactor dependency versions in requirements.txt and setup.py

* Refactor task remove logging

* Added `asyncio_mode = auto` to `setup.cfg` for improved asyncio handling in tests

* Remove unused import from base.py

* Update README.md to enhance installation instructions

* Update version to 1.1.0.dev12 in version.py

* Rename chalice and fastapi folders for improved structure

* Update GitHub Actions workflows to use latest action versions

* Consolidate models and fixtures for reusability

* rename 'fast' to 'fastapi' in test

* Format

* Remove unnecessary db_alias from models

* Refactor query method in Card class to specify return type as dict

* Promote 'task' folder to top level for clarity

* Update setup.py

* Fix typo in setup.py dependencies list

* Update requirements files to remove commented sections

* Update version to 1.1.0.dev13

* Update dependencies in requirements.txt to stable versions

* Update mongoengine-plus dependency version in setup.py

* Increment version to 1.1.0.dev14

* Remove BaseModel and use mongoengine_plus.BaseModel instead

* Fix formatting in requirements.txt by adding a newline at the end of the file

* Unify endpoint tests for FastAPI and Chalice

* Update dependency version ranges in setup.py

* Enhance README.md with detailed installation and usage instructions

* Version from 1.1.0.dev14 to 1.0.0

* Standardize Chalice client integration in tests

* Replace client fixture with fastapi_client in tests

* Remove redundant assignment of wrong_params

* Import uuid_field from cuenca-validations

* Remove obsolete test files for model helpers, base model, and event handlers

* Refactor ChaliceClient to handle JSON requests and remove helper functions

* Update cuenca-validations dependency version from 2.0.2.dev1 to 2.0.2 in requirements.txt

---------

Co-authored-by: gabino <[email protected]>

.github/workflows/release.yml

@@ -6,18 +6,18 @@ jobs:
   publish-pypi:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@master
-      - name: Set up Python 3.8
-        uses: actions/setup-python@v2.2.1
+      - uses: actions/checkout@v4
+      - name: Set up Python 3.13
+        uses: actions/setup-python@v5
         with:
-          python-version: 3.8
+          python-version: 3.13
       - name: Install dependencies
         run: pip install -qU setuptools wheel twine
       - name: Generating distribution archives
         run: python setup.py sdist bdist_wheel
       - name: Publish distribution 📦 to PyPI
         if: startsWith(github.event.ref, 'refs/tags')
-        uses: pypa/gh-action-pypi-publish@master
+        uses: pypa/gh-action-pypi-publish@release/v1
         with:
           user: __token__
           password: ${{ secrets.pypi_password }}

.github/workflows/test.yml

@@ -6,11 +6,11 @@ jobs:
   lint:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v4
       - name: Set up Python
-        uses: actions/setup-python@v2.2.1
+        uses: actions/setup-python@v5
         with:
-          python-version: 3.8
+          python-version: 3.13
       - name: Install dependencies
         run: make install-test
       - name: Lint
@@ -20,11 +20,11 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: [3.8]
+        python-version: ['3.9', '3.10', '3.11', '3.12', '3.13']
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v4
     - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v2.2.1
+      uses: actions/setup-python@v5
       with:
         python-version: ${{ matrix.python-version }}
     - name: Install dependencies
@@ -35,19 +35,20 @@ jobs:
   coverage:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@master
+      - uses: actions/checkout@v4
       - name: Setup Python
-        uses: actions/setup-python@v2.2.1
+        uses: actions/setup-python@v5
         with:
-          python-version: 3.8
+          python-version: 3.13
       - name: Install dependencies
         run: make install-test
       - name: Generate coverage report
         run: pytest --cov-report=xml
       - name: Upload coverage to Codecov
-        uses: codecov/codecov-action@v1
+        uses: codecov/codecov-action@v5
         with:
           file: ./coverage.xml
           flags: unittests
           name: codecov-umbrella
+          token: ${{ secrets.CODECOV_TOKEN }}
           fail_ci_if_error: true

Makefile

@@ -1,9 +1,9 @@
 SHELL := bash
 PATH := ./venv/bin:${PATH}
-PYTHON = python3.8
+PYTHON = python3.13	
 PROJECT = agave
 isort = isort $(PROJECT) examples tests setup.py
-black = black -S -l 79 --target-version py38 $(PROJECT) examples $(PROJECT)/lib/* tests setup.py
+black = black -S -l 79 --target-version py313 $(PROJECT) tests setup.py examples
 
 
 .PHONY: all

README.md

@@ -3,31 +3,136 @@
 [![codecov](https://codecov.io/gh/cuenca-mx/agave/branch/main/graph/badge.svg)](https://codecov.io/gh/cuenca-mx/agave)
 [![PyPI](https://img.shields.io/pypi/v/agave.svg)](https://pypi.org/project/agave/)
 
-Agave is a library that implement rest_api across the use of Blueprints based on Chalice Aws.
+Agave is a library for building REST APIs using a Blueprint pattern, with support for both AWS Chalice and FastAPI frameworks. It simplifies the creation of JSON-based endpoints for querying, modifying, and creating resources.
 
-this library allow send and receive JSON data to these endpoints to query, modify and create content.
+## Installation
 
-Install agave using pip:
+Choose the installation option based on your framework:
 
+### Chalice Installation
+
+```bash
+pip install agave[chalice]
+```
+
+### FastAPI Installation
+
+```bash
+pip install agave[fastapi]
+```
+
+### SQS task support:
 ```bash
-pip install agave==0.0.2.dev0
+pip install agave[fastapi,tasks]
 ```
 
-You can use agave for blueprint like this:
+## Usage
+
+### Chalice Example
+
+You can then create a REST API blueprint as follows:
 ```python
+from agave.chalice import RestApiBlueprint
+
+app = RestApiBlueprint()
+
+@app.resource('/accounts')
+class Account:
+    model = AccountModel
+    query_validator = AccountQuery
+    update_validator = AccountUpdateRequest
+    get_query_filter = generic_query
 
-from agave.blueprints.rest_api import RestApiBlueprint
+    @staticmethod
+    @app.validate(AccountRequest)
+    def create(request: AccountRequest) -> Response:
+        account = AccountModel(
+            name=request.name,
+            user_id=app.current_user_id,
+            platform_id=app.current_platform_id,
+        )
+        account.save()
+        return Response(account.to_dict(), status_code=201)
 
+    @staticmethod
+    def update(
+        account: AccountModel, request: AccountUpdateRequest
+    ) -> Response:
+        account.name = request.name
+        account.save()
+        return Response(account.to_dict(), status_code=200)
+
+    @staticmethod
+    def delete(account: AccountModel) -> Response:
+        account.deactivated_at = dt.datetime.utcnow().replace(microsecond=0)
+        account.save()
+        return Response(account.to_dict(), status_code=200)
 ```
 
-agave include helpers for mongoengine, for example:
+### FastAPI Example
+
 ```python
+from agave.fastapi import RestApiBlueprint
+
+app = RestApiBlueprint()
+
+@app.resource('/accounts')
+class Account:
+    model = AccountModel
+    query_validator = AccountQuery
+    update_validator = AccountUpdateRequest
+    get_query_filter = generic_query
+    response_model = AccountResponse
+
+    @staticmethod
+    async def create(request: AccountRequest) -> Response:
+        """This is the description for openapi"""
+        account = AccountModel(
+            name=request.name,
+            user_id=app.current_user_id,
+            platform_id=app.current_platform_id,
+        )
+        await account.async_save()
+        return Response(content=account.to_dict(), status_code=201)
+
+    @staticmethod
+    async def update(
+        account: AccountModel,
+        request: AccountUpdateRequest,
+    ) -> Response:
+        account.name = request.name
+        await account.async_save()
+        return Response(content=account.to_dict(), status_code=200)
 
-from agave.models.helpers import (uuid_field, mongo_to_dict, EnumField, updated_at, list_field_to_dict)
+    @staticmethod
+    async def delete(account: AccountModel, _: Request) -> Response:
+        account.deactivated_at = dt.datetime.utcnow().replace(microsecond=0)
+        await account.async_save()
+        return Response(content=account.to_dict(), status_code=200)
+```
+
+### Async Tasks
+
+```python
+from agave.tasks.sqs_tasks import task
 
+QUEUE_URL = 'https://sqs.region.amazonaws.com/account/queue'
+AWS_DEFAULT_REGION = 'us-east-1'
+@task(
+    queue_url=QUEUE_URL,
+    region_name=AWS_DEFAULT_REGION,
+    visibility_timeout=30,
+    max_retries=10,
+)
+async def process_data(data: dict):
+    # Async task processing
+    return {'processed': data}
 ```
 
-Correr tests
+## Running Tests
+
+Run the tests using the following command:
+
 ```bash
 make test
 ```

agave/chalice/__init__.py

@@ -0,0 +1,2 @@
+__all__ = ['RestApiBlueprint']
+from .rest_api import RestApiBlueprint

agave/models/helpers.py → agave/chalice/models/helpers.py

@@ -2,13 +2,6 @@
 from base64 import urlsafe_b64encode
 
 
-def uuid_field(prefix: str = ''):
-    def base64_uuid_func() -> str:
-        return prefix + urlsafe_b64encode(uuid.uuid4().bytes).decode()[:-2]
-
-    return base64_uuid_func
-
-
 # This function is used to generate an id composed of a
 # list of fields in alphabetical order, for example if we want
 # uuid_field_generic('AC', account_number='bla', user_id='ble')

agave/blueprints/rest_api.py → agave/chalice/rest_api.py

@@ -2,12 +2,19 @@
 from typing import Any, Optional, Type, cast
 from urllib.parse import urlencode
 
-from chalice import Blueprint, NotFoundError, Response
+try:
+    from chalice import Blueprint, NotFoundError, Response
+except ImportError:
+    raise ImportError(
+        "You must install agave with [chalice] option.\n"
+        "You can install it with: pip install agave[chalice]"
+    )
+
 from cuenca_validations.types import QueryParams
 from mongoengine import DoesNotExist, Q
 from pydantic import BaseModel, ValidationError
 
-from .decorators import copy_attributes
+from ..core.blueprints.decorators import copy_attributes
 
 
 class RestApiBlueprint(Blueprint):
@@ -71,8 +78,9 @@ def retrieve_object(self, resource_class: Any, resource_id: str) -> Any:
         return data
 
     def validate(self, validation_type: Type[BaseModel]):
-        """This decorator validate the request body using a custom pydantyc model
-        If validation fails return a BadRequest response with details
+        """This decorator validate the request body using a
+        custom pydantyc model. If validation fails return a
+        BadRequest response with details
 
         @app.validate(MyPydanticModel)
         def my_method(request: MyPydanticModel):
@@ -284,7 +292,7 @@ def _all(query: QueryParams, filters: Q):
                 if wants_more and has_more:
                     query.created_before = item_dicts[-1]['created_at']
                     path = self.current_request.context['resourcePath']
-                    params = query.dict()
+                    params = query.model_dump()
                     if self.user_id_filter_required():
                         params.pop('user_id')
                     if self.platform_id_filter_required():

agave/blueprints/decorators.py → agave/core/blueprints/decorators.py

@@ -16,7 +16,8 @@ def wrapper(func: Callable):
             return func
 
         for key, val in original_func.__dict__.items():
-            setattr(func, key, val)
+            if not key.startswith('_'):
+                setattr(func, key, val)
 
         return func
 

agave/core/exc.py

@@ -0,0 +1,63 @@
+from dataclasses import dataclass
+from typing import Optional
+
+
+@dataclass
+class AgaveError(Exception):
+    error: str
+    status_code: int
+
+
+@dataclass
+class BadRequestError(AgaveError):
+    status_code: int = 400
+
+
+@dataclass
+class UnauthorizedError(AgaveError):
+    status_code: int = 401
+
+
+@dataclass
+class ForbiddenError(AgaveError):
+    status_code: int = 403
+
+
+@dataclass
+class NotFoundError(AgaveError):
+    status_code: int = 404
+
+
+@dataclass
+class MethodNotAllowedError(AgaveError):
+    status_code: int = 405
+
+
+@dataclass
+class ConflictError(AgaveError):
+    status_code: int = 409
+
+
+@dataclass
+class UnprocessableEntity(AgaveError):
+    status_code: int = 422
+
+
+@dataclass
+class TooManyRequests(AgaveError):
+    status_code: int = 429
+
+
+@dataclass
+class AgaveViewError(AgaveError):
+    status_code: int = 500
+
+
+@dataclass
+class ServiceUnavailableError(AgaveError):
+    status_code: int = 503
+
+
+@dataclass
+class RetryTask(Exception):
+    countdown: Optional[int] = None

agave/filters.py → agave/core/filters.py

@@ -2,7 +2,7 @@
 from mongoengine import Q
 
 
-def generic_query(query: QueryParams) -> Q:
+def generic_query(query: QueryParams, excluded: list[str] = []) -> Q:
     filters = Q()
     if query.created_before:
         filters &= Q(created_at__lt=query.created_before)
@@ -15,8 +15,9 @@ def generic_query(query: QueryParams) -> Q:
         'limit',
         'page_size',
         'key',
+        *excluded,
     }
-    fields = query.dict(exclude=exclude_fields)
+    fields = query.model_dump(exclude=exclude_fields)
     if 'count' in fields:
         del fields['count']
     return filters & Q(**fields)

agave/blueprints/__init__.py → agave/fastapi/__init__.py

@@ -1,3 +1,3 @@
-__all__ = ['RestApiBlueprint']
-
 from .rest_api import RestApiBlueprint
+
+__all__ = ['RestApiBlueprint']