Merge pull request #43 from anomaly/sql-layout

Major refactor of Shillelagh adapter implementation

.gitignore

@@ -1,3 +1,6 @@
+# SQlite databases
+*.sqlite
+
 # managed by direnv
 .envrc
 

README.md

@@ -12,17 +12,19 @@
 
 Gallagher Security manufacture a variety of [security products](https://security.gallagher.com) all of which are controlled by their [Command Centre](https://products.security.gallagher.com/security/au/en_AU/products/software/command-centre/p/C201311) software. Traditionally Command Centre has been a Windows based server product. Version `8.6` introduced a REST API which allows you to interact with the system via HTTP requests. Gallagher also provide a [Cloud API Gateway](https://gallaghersecurity.github.io/docs/Command%20Centre%20Cloud%20Api%20Gateway%20TIP.pdf) which allows third party integrations to securely communicate with the Command Centre on site.
 
-This project primarily provides the following:
+In principle we provides the following:
 
 - **Python SDK** an idiomatic client (featuring `asyncio`) to build applications for the Command Centre.
 - **Command Line Interface** (CLI) to build powerful pipeline-based workflows.
 - **Terminal User Interface** (TUI) for easy interactions with the Command Centre.
-- **SQL interface** query the REST API like a database or interact with via an ORM.
+- **SQL interface** query the REST API as if it were a database or interact with via an ORM.
 
-> **Note:** this project is **NOT** affiliated with Gallagher Security. All trademarks are the property of their respective owners.
+> [!NOTE]
+> This project is **NOT** affiliated with Gallagher Security. All trademarks are the property of their respective owners.
 
 While Gallagher maintain a set of [Swagger definitions](https://github.com/gallaghersecurity/cc-rest-docs) for their API, they are primarily intended to generate the documentation [published on Github](https://gallaghersecurity.github.io/cc-rest-docs/ref/index.html). They use a tool called [Spectacle](https://github.com/sourcey/spectacle). Gallagher explicitly state that the Swagger definitions are not intended to be used to generate code. Due to this the API client is hand built and not auto-generated.
 
+> [!IMPORTANT]
 > Due to custom annotations the YAML files will not parse with any standard parser.
 
 The client was designed while building products around the Gallagher API. It's design is highly opinionated and does not conform with how Gallagher design software interfaces. If you've worked with [stripe-python](https://github.com/stripe/stripe-python) the syntax may feel familiar.
@@ -39,7 +41,7 @@ from gallagher import (
 from gallagher.dto.summary import (
     CardholderSummary,
 )
-from gallagher.cc.cardholders.cardholders import (
+from gallagher.cc.cardholders import (
     Cardholder
 )
 
@@ -62,6 +64,7 @@ cardholder.href
 cardholder.first_name
 ```
 
+> [!TIP]
 > We pride ourselves in providing a complete test suite as proof of high quality work that you can rely on. These tests constantly run against our _demo_ command centre hosted on the cloud.
 
 ## API Notes
@@ -85,6 +88,7 @@ through to self recursive references with additional attributes:
 }
 ```
 
+> [!NOTE]
 > Above examples have been taken from the Gallagher documentation
 
 Our `schemas` provide a set of `Mixins` that are used to construct the Models. These are repeatable patterns that need not be repeated. The typical patter would be to subclass from the `Mixins` e.g:
@@ -170,6 +174,7 @@ We use [Taskfile](https://taskfile.dev) to automate running tasks.
 
 The project provides a comprehensive set of tests which can be run with `task test`. These tests do create objects in the Command Centre, we advice you to obtain a test license.
 
+> [!IMPORTANT]
 > It's **not recommended** to run tests against a production system.
 
 ### Data Transfer Objects
@@ -185,6 +190,8 @@ In summary:
 - `Refs` are the minimal pathway to an object
 - `Summary` builds on a `Ref` and provides a subset of the attributes
 - `Detail` builds on a `Summary` and provides the full set of attributes
+- `Response` models the response layout that contains summaries
+- `Payload` is used to send a request to the API
 
 ### Resources
 
@@ -224,6 +231,7 @@ To check your API key:
 
 ## Python Libraries
 
+> [!TIP]
 > Following are Python libraries that I have found during the development of the Gallagher tools. They are not necessarily in use at the moment but a reference in case we need the functionality.
 
 - [plotext](https://github.com/piccolomo/plotext?tab=readme-ov-file) - plots directly on your terminal (something I found when I was exploring apps like [dolphie](https://github.com/charles-001/dolphie))

SQL.md

@@ -12,11 +12,13 @@ Copied verbatam from [shillelagh](https://github.com/betodealmeida/shillelagh?ta
 
 [How to pronounce `shillelagh`](https://youtu.be/QHDZtvfTkz4?feature=shared) Shea-lay-lee
 
+### ORM integration
+
 ## Adapter Design
 
 `shillelagh` has appropriate documentation to [build your own adapters](https://shillelagh.readthedocs.io/en/latest/development.html#), the examples on the page go from implementing a simple adapter for the Weather API through to some tips to implement a more complex adapter that reads from an API e.g Google Sheets. It's recommended to study the source code to understand patterns of the how to provide data for more complex implementations.
 
-Couple of things to get note are:
+Few things to address design wise are:
 
 - Methods that allows shillelagh to [calculate the cost of a query](https://shillelagh.readthedocs.io/en/latest/development.html#estimating-query-cost)
 - Strategies in mapping URLs to virtual tables
@@ -30,30 +32,99 @@ The DTOs in particular capture the rules set out by the API documentation, prima
 
 It would thus make sense to extend the `EndpointConfig` class to include SQL specific configuration and the framework to inject the required configuration into the adapter.
 
-**Table names**: The URL of th endpoint maps as the table name e.g:
+The reason for extending `EndpointConfig` is to ensure that all related configuration is maintained in the base classes. The `adapter` amongst other things is an optional extension that:
+
+- depends on the base API client to exist and function in full
+- is optionally installed by the user
+
+### Table names
+
+The URL of th endpoint maps as the table name e.g:
 
 ```sql
 🍀> SELECT * FROM "https://commandcentre-api-au.security.gallagher.cloud/api/cardholders";
 ```
 
 where `https://commandcentre-api-au.security.gallagher.cloud/api/cardholders` is the table. The `adapter` would have acknowledged that it can handle this endpoint. If we support more than one than one endpoint then we would have to validate that the URL is one of the many.
 
-**Field list**
+### Field list
 
 Each DTO is already aware of the fields and their types. If a type is a primitive then it's easy enough to handle.
 
 If a field happens to be a relationship then we would have to return the ID of the object (if this is supported, not sure what to do when all it has is a href) to make it behave more SQL like. I think the child object should determine which field should be returned as the ID and the configuration should be part of the child object.
 
 `SELECT` queries often let you specify while columns you wish to return. This also happens to be the case for the Gallagher API. See [#36](https://github.com/anomaly/gallagher/issues/36) for more information. It would make sense to combine these two patterns.
 
-**Search fields**
+> Shillelagh require us to send back a field called `rowid` which has a unique identifier, we are currently using the `id` of the object as the `rowid`.
+
+For partial queries we should take advantage of the fact that the CC API allows us to send a list of fields to return. This is a good way to reduce the amount of data that is returned.
+
+### Search fields
 
 Search fields are determined by the API per endpoint. These should be defined as `tuples` in the `EndpointConfig` class (note that tuples are immutable hence it's ideal for a configuration).
 
-**Result Order**
+### Result Order
+
+At this stage we support the defaults as defined by Shillelagh and allow the user to order results by any of the fields in the DTO.
+
+The CC API does provide a `sort` query parameter that can be used to sort the results. Where possible we should use this parameter to avoid local computation costs. Note also the use of `-id` the negative sign to sort in descending order.
 
-**Offset and Limit**
+### Offset and Limit
 
 The Gallagher API certain supports pagination, it does return a `next` link, and we might have to filter the results locally on the adapter. We should also cross reference our ticket on performance tuning [#14](https://github.com/anomaly/gallagher/issues/14)
 
+### Query costing
+
+Shillelagh has a method to calculate the cost of a query.
+
+### Mapping pyndatic attributes to shillelagh
+
+Our pyndatic classes use python typing to annotate fields for parsing. Shillelagh provides the `fields` package that allows developers of adapters to provide definition of fields handled by each virtual table.
+
+To make this manageable for developers we need to provide a way to:
+
+- register which endpoints are suited to work with our shillelagh adapters
+- automatically map the fields from the pyndatic classes to the shillelagh fields
+
+## Proposed API syntax
+
+```python
+    dto_list: Optional[any] = None  # DTO to be used for list requests
+    dto_retrieve: Optional[any] = None  # DTO to be used for retrieve requests
+```
+
+`dto_list` object field should be used to determine fields that are returned when a list of objects is requested. The `dto_retrieve` object field should be used to determine fields that are returned when a single object is requested.
+
+```python
+    search_fields: Optional[Tuple[str]] = None  # Fields that can be searched
+```
+
+```python
+    sql = False  # If the endpoint supports SQL queries
+    sql_limit_supported = False  # If the endpoint supports SQL LIMIT
+    sql_offset_supported = False # If the endpoint supports SQL OFFSET
+```
+
+SQL parsing things to check:
+
+- The fields in a select exists in the DTO class
+- Does an endpoint support offsets and limits
+- Examples of joins, group by, order by, where clauses
+
+Each top level package in gallagher.cc will defined a `tuple` called `__shillelagh__` this deliberately mounts classes that are to be used by the shillelagh adapter.
+
+The framework calls the `get_columns` method which is to return a `Dict [str, Field]` where the key is the name of the field and the value is an instance of a `Field` class.
+
+It's advisable to instantiate a list of columns in `init` and not compute them in the `get_columns` method.
+
+### Mutations
+
+Not all endpoints support mutations. The configuration of each DTO endpoint should reflect this. Our adapter will have to reflect this and proxy the operation to the DTO endpoint.
+
+## Static references to Classes
+
+I've decided to ad static references to the classes that support SQL to avoid using introspection.
+
 ## SQLAlchemy `dialect` Design
+
+SQLAlchemy is a popular ORM (one of choice at Anomaly) and is supported by shillelagh. The `dialect` essentially makes available the endpoint as a virtual table to the ORM.

TUI.md

@@ -1,5 +1,19 @@
-# Terminal UI 
+# Terminal UI
 
 > The TUI is developed using the superpowers given to Python developers by [Textual](https://textual.textualize.io)
 
+Cardholder management
 
+- List cardholders
+- Create cardholder
+- Update cardholder
+- Delete cardholder
+- Attach cards to a cardholder
+- Detach cards from a cardholder
+- Visit details for a cardholder
+
+Alarm management
+
+- List alarms
+- Acknowledge alarms
+- Comment on an alarm

Taskfile.yml

@@ -45,7 +45,7 @@ tasks:
   dev:textual:
     desc: runs the textual cli
     cmds:
-      - poetry run textual --
+      - poetry run textual --dev --
   dev:tui:
     desc: runs text gallagher console in dev mode
     cmds:

docs/docs/design.md

@@ -88,6 +88,7 @@ The discovered state of the server is stored in a singleton, that's used by all
 
 For this reason all `APIEndpoint` classes return a configuration as a result of a function called `get_config` (an `async` method that at a `class` scope) as opposed to a statically assigned class variable (otherwise the URLs would always result to be the initial `None` value).
 
+> [!TIP]
 > If you want to force discovery of the endpoints call `expire_discovery` on the `APIEndpoint` before calling the API endpoint.
 
 ```python

docs/docs/index.md

@@ -2,12 +2,12 @@
 
 Gallagher Security's Command Centre provides a REST API to programmatically interact with a large part of the system. This project maintains a set of developer focused tools to make integration experiences a delight. In addition we provide a set of power tools to make automations a breeze. At the moment we maintain:
 
-- a Python [idiomatic](https://www.merriam-webster.com/dictionary/idiomatic) client for developers
-- Command line interface to interact with the command centre
-- Terminal Inteface designed to provide a subset of the command centre functions, with a different viewpoint
-- SQL interface (as a SQLAlchemy dialect) to ease integration into coprorate systems
+- **Python SDK** an [idiomatic](https://www.merriam-webster.com/dictionary/idiomatic) client (featuring `asyncio`) to build applications for the Command Centre.
+- **Command Line Interface** (CLI) to build powerful pipeline-based workflows.
+- **Terminal User Interface** (TUI) for easy interactions with the Command Centre.
+- **SQL interface** query the REST API like a database or interact with via an ORM.
 
-> **Note:** This project is not affiliated with Gallagher in any way.
+> **Note:** While Anomaly is part of Gallagher's Technical Partner Program, this project is not officially affiliated with Gallagher.
 
 ## How to get started?
 
@@ -19,20 +19,20 @@ The offerings of this project are aimed at two significant groups of users:
 If you are a developer, the first things that might excite you are:
 
 - [Python idiomatic client](./python-client.md) that adheres to the standards outlined by Gallagher
-- [SQL interface to the REST API](./sql-interface.md) that allows you to interact with the API using SQL queries
+- [SQL interface to the REST API](./sql.md) that allows you to interact with the API using SQL queries
 
 As an advanced user or integrator, we recommend at looking at:
 
 - [Command line interface](./cli.md) that allows you to interact with the API from the terminal
-- [Terminal interface](./terminal-interface.md) that provides a subset of the command centre functions in a different viewpoint
+- [Terminal interface](./tui.md) that provides a subset of the command centre functions in a different viewpoint
 
 All of our tools are written using the Python programming languages. We provide binaries for most popular operating systems. If you are a developer and would like to contribute to the project, a good place to start is our [design document](./design.md).
 
 ## Motivation
 
 Anomaly has a long history of working with APIs. In the early 2010s, during the early days of JSON APIs, we build a framework in Python called [prestans](https://github.com/anomaly/prestans/), it was designed to encompass the best practices of RESTful API design by providing a programmatic interface in Python. It allowed us to build large scale applications without losing quality as the team scaled up. `prestans` resulted in consistent code across the our applications.
 
-While prestans has had it's day (superseded by projects like pydantic and FastAPI), what we learnt from building it has stayed with us. We've taken our approach of deeply understanding the underlying protocol, design patterns and philosophy of the Gallagher API and encapsulated it as a set of developer products to empower others to build on top of their infrastructure.
+prestans has since been superseded by projects like pydantic and FastAPI, but what we learnt from building it has stayed with us. We've taken our approach of deeply understanding the underlying protocol, design patterns and philosophy of the Gallagher API and encapsulated it as a set of developer products to empower others to build on top of their infrastructure.
 
 Our central theme is that we do the heavily lifting so you can rely on the tools to build reliable integrations. This toolkit is the building block of one of our core business offerings, making it's quality a top priority for us.
 
@@ -42,7 +42,7 @@ Our initial requirements for interacting with Command Centre's API came from a p
 
 Gallagher publish their [API reference](https://gallaghersecurity.github.io/cc-rest-docs/ref/index.html) which is built from an OpenAPI spec with [Spectacle](https://github.com/sourcey/spectacle) documentation generator flavours. The OpenAPI spec is maintained by hand and [can be found on Github](https://github.com/GallagherSecurity/cc-rest-docs/tree/master/swagger). While this is all you need to get started, it can be overwhelming to interact with the API directly.
 
-Initially for the sake of our own product offering, we decided to build a Python client to interact with the API to ensure quality and achieve a great developer experience as our offerings scale. The client side tools started as shortcuts that would assist us to support our customers.
+Born out of solving our client's use cases, we built the Python client to interact with the API to ensure quality and achieve a great developer experience as our offerings scale. As our support requests grew it was evident that we could carve out a set of tools to enhance automation and integrations.
 
 We believe that open sourcing these tools will result in a vibrant community of developers that will be able to leverage Gallagher's technology to build incredibly practical applications.
 

docs/docs/installation.md

@@ -1,4 +1,6 @@
-# Usage
+# Installation & Usage
+
+What you need to install will widely depend on if you are looking to use our tools, building software with our libraries or contributing to the project. This guide will get you set up for each one of the use cases. While the end user tools do not require software development experience, the other development tools are biased towards Python developers.
 
 ## Installation
 
@@ -10,8 +12,6 @@ poetry add gallagher
 
 ## Understanding the interface
 
-
-
 ## Developing the client
 
 This library uses [httpx](https://www.python-httpx.org) as the HTTP transport and [pydantic](https://pydantic.dev) to construct and ingest payloads. We use [taskfile](https://taskfile.dev) to run tasks. Our test suite is setup using `pytest`.
@@ -32,4 +32,3 @@ Some handy commands to get you started:
 - `mkdocs -h` - Print help message and exit.
 
 To start contributing please fork this repository, make the changes you desire and submit a pull request for us to merge your changes in. Alternatively consider [starting a discussion](https://github.com/anomaly/gallagher/discussions) or [raising an issue](https://github.com/anomaly/gallagher/issues). Be kind to our maintainers and check to see if a similar discussion is already in place and join the thread.
-

docs/docs/sql.md

@@ -0,0 +1 @@
+# SQL and SQLAlchemy

docs/mkdocs.yml

@@ -12,8 +12,9 @@ markdown_extensions:
 
 nav:
   - Introduction: index.md
-  - Installation: installation.md
-  - Using the API Client: howto-api-client.md
+  - Installation & Usage: installation.md
+  - SQL and SQLAlchemy: sql.md
   - Command Line Inteface: cli.md
   - Terminal User Inteface: tui.md  
-  - API Client Design: design.md
+  - Using the API Client: howto-api-client.md
+  - Design: design.md

gallagher/cc/alarms/__init__.py

@@ -6,7 +6,11 @@
 
 from typing import Optional
 
-from ..core import APIEndpoint, EndpointConfig, Capabilities
+from ..core import (
+    APIEndpoint,
+    EndpointConfig,
+    Capabilities,
+)
 
 from ...dto.ref import (
     AlarmRef,
@@ -122,3 +126,8 @@ async def mark_as_force_processed(
     ) -> bool:
         """ """
         return False
+
+
+__shillelagh__ = (
+    Alarms,
+)