feat: add additional models, handlers

README.md

@@ -1,6 +1,6 @@
-# GitHub Actions Aggregator Service
+# GitHub Actions Aggregator
 
-A Go-based service to aggregate and analyze data from GitHub Actions workflows across multiple repositories. This application provides insights into workflow runs, success rates, failure rates, and other statistics over customizable time ranges.
+GitHub Actions Aggregator is a Go-based service designed to collect, aggregate, and analyze data from GitHub Actions workflows across multiple repositories. This application provides valuable insights into workflow runs, success rates, failure rates, and other statistics over customizable time ranges.
 
 ## Table of Contents
 
@@ -17,7 +17,7 @@ A Go-based service to aggregate and analyze data from GitHub Actions workflows a
 
 ## Features
 
-- **OAuth 2.0 Authentication**: Securely authenticate users via GitHub OAuth.
+- **OAuth 2.0 Authentication**: Secure user authentication via GitHub OAuth.
 - **Data Collection**:
   - **Webhooks**: Receive real-time updates on workflow events.
   - **Polling**: Periodically poll GitHub API to ensure data completeness.
@@ -29,11 +29,10 @@ A Go-based service to aggregate and analyze data from GitHub Actions workflows a
 
 ## Prerequisites
 
-- **Go**: Version 1.18 or higher.
-- **GitHub Account**: For OAuth authentication and API access.
-- **PostgreSQL**: For storing data.
-- **Redis** (optional): For caching (if implemented).
-- **Docker** (optional): For containerization and deployment.
+- Go (version 1.18 or higher)
+- PostgreSQL
+- GitHub Account (for OAuth authentication and API access)
+- Docker (optional, for containerization)
 
 ## Installation
 
@@ -44,16 +43,15 @@ A Go-based service to aggregate and analyze data from GitHub Actions workflows a
    cd github-actions-aggregator
    ```
 
-2. **Install Dependencies**
-
-   ```bash
+2. Install dependencies:
+   ```
    go mod download
    ```
 
-3. **Set Up Environment Variables**
-
+3. Set up environment variables:
    Create a `.env` file or export the required environment variables:
 
+
    ```bash
    export GITHUB_CLIENT_ID="your_github_client_id"
    export GITHUB_CLIENT_SECRET="your_github_client_secret"
@@ -63,11 +61,12 @@ A Go-based service to aggregate and analyze data from GitHub Actions workflows a
    export SERVER_PORT="8080"
    ```
 
+
 ## Configuration
 
 Configuration can be managed via a `config.yaml` file in the `configs/` directory or through environment variables.
 
-**Example `config.yaml`:**
+Example `config.yaml`:
 
 ```yaml
 server:
@@ -83,126 +82,68 @@ github:
   webhook_secret: "your_webhook_secret"
 ```
 
-**Note:** Environment variables override values in the configuration file.
+Note: Environment variables override values in the configuration file.
 
 ## Usage
 
-### Running the Application
-
-1. **Run Database Migrations**
-
-   ```bash
-   ./scripts/migrate.sh
+1. Run database migrations:
+   ```
+   ./scripts/migrate.sh up
    ```
 
-2. **Start the Application**
-
-   ```bash
+2. Start the application:
+   ```
    go run cmd/server/main.go
    ```
 
-### Accessing the Application
-
-- **Login with GitHub**: Navigate to `http://localhost:8080/login` to authenticate via GitHub.
-- **API Requests**: Use tools like `curl` or Postman to interact with the API endpoints.
+3. Access the application:
+   - Login with GitHub: Navigate to `http://localhost:8080/login`
+   - API Requests: Use tools like `curl` or Postman to interact with the API endpoints
 
 ## API Endpoints
 
-### Authentication
-
 - `GET /login`: Redirects the user to GitHub for OAuth authentication.
 - `GET /callback`: Handles the OAuth callback from GitHub.
-
-### Workflow Statistics
-
 - `GET /workflows/:id/stats`: Retrieves statistics for a specific workflow.
+- `GET /repositories/:id/workflows`: Get all workflows for a repository
+- `GET /workflows/:id/runs`: Get all runs for a workflow
+- `GET /runs/:id`: Get a specific run
+- `GET /jobs/:id`: Get a specific job
+- `GET /jobs/:id/steps`: Get all steps for a job
+- `GET /jobs/:id/stats`: Get stats for a job
 
-  **Query Parameters:**
-
-  - `start_time` (optional): Start of the time range (ISO 8601 format).
-  - `end_time` (optional): End of the time range (ISO 8601 format).
-
-  **Example Request:**
-
-  Explicit time range:
-
-  ```http
-  GET /workflows/123/stats?start_time=2023-09-01T00:00:00Z&end_time=2023-09-30T23:59:59Z
-  ```
-
-  Relative time range:
-
-  ```http
-  GET /workflows/123/stats?start_time=24_hours&end_time=now
-  ```
-
-  **Example Response:**
-
-  ```json
-  {
-    "workflow_id": 123,
-    "workflow_name": "CI Build and Test",
-    "total_runs": 200,
-    "success_count": 150,
-    "failure_count": 30,
-    "cancelled_count": 10,
-    "timed_out_count": 5,
-    "action_required_count": 5,
-    "success_rate": 75.0,
-    "failure_rate": 15.0,
-    "cancelled_rate": 5.0,
-    "timed_out_rate": 2.5,
-    "action_required_rate": 2.5,
-    "start_time": "2023-09-01T00:00:00Z",
-    "end_time": "2023-09-30T23:59:59Z"
-  }
-  ```
+For detailed information on request parameters and response formats, please refer to the API documentation.
 
 ## Authentication
 
-### Setting Up OAuth with GitHub
-
-1. **Register a New OAuth Application**
-
-   - Go to [GitHub Developer Settings](https://github.com/settings/developers).
-   - Click on **"New OAuth App"**.
+1. Register a new OAuth application on GitHub:
+   - Go to GitHub Developer Settings
+   - Click on "New OAuth App"
    - Fill in the application details:
-     - **Application Name**
-     - **Homepage URL**: `http://localhost:8080`
-     - **Authorization Callback URL**: `http://localhost:8080/callback`
-   - Obtain your **Client ID** and **Client Secret**.
-
-2. **Configure Application Credentials**
+     - Application Name
+     - Homepage URL: `http://localhost:8080`
+     - Authorization Callback URL: `http://localhost:8080/callback`
+   - Obtain your Client ID and Client Secret
 
+2. Configure application credentials:
    Set your `GITHUB_CLIENT_ID` and `GITHUB_CLIENT_SECRET` in your environment variables or `config.yaml`.
 
-### Permissions and Scopes
-
-Ensure that your GitHub OAuth application has the necessary scopes:
-
-- `read:user`
-- `repo`
-- `workflow`
+Ensure that your GitHub OAuth application has the necessary scopes: `read:user`, `repo`, and `workflow`.
 
 ## Testing
 
-### Running Unit Tests
-
-```bash
+Run unit tests:
+```
 go test ./tests/unit/...
 ```
 
-### Running Integration Tests
-
-```bash
+Run integration tests:
+```
 go test ./tests/integration/...
 ```
 
-### Test Coverage
-
-You can generate a test coverage report using:
-
-```bash
+Generate a test coverage report:
+```
 go test -coverprofile=coverage.out ./...
 go tool cover -html=coverage.out
 ```
@@ -241,11 +182,11 @@ Contributions are welcome! Please follow these steps:
 
 6. **Create a Pull Request**
 
-   Go to the original repository and open a pull request.
+For more details, please refer to the [CONTRIBUTING.md](CONTRIBUTING.md) file.
 
 ## License
 
-This project is licensed under the [MIT License](LICENSE).
+This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.
 
 ---
 

cmd/server/main.go

@@ -37,11 +37,16 @@ func main() {
 	// Initialize GitHub client
 	githubClient := github.NewClient(cfg.GitHub.AccessToken)
 
-	workerPool := worker.NewWorkerPool(database, cfg.WorkerPoolSize)
-	workerPool.Start()
+	// Initialize worker pool for polling
+	pollingWorkerPool := worker.NewWorkerPool(database, cfg.PollingWorkerPoolSize)
+	pollingWorkerPool.Start()
+
+	// Initialize worker pool for webhooks
+	webhookWorkerPool := worker.NewWorkerPool(database, cfg.WebhookWorkerPoolSize)
+	webhookWorkerPool.Start()
 
 	// Start the API server
-	go api.StartServer(cfg, database, githubClient, workerPool)
+	go api.StartServer(cfg, database, githubClient, webhookWorkerPool)
 
 	// Set up graceful shutdown
 	quit := make(chan os.Signal, 1)
@@ -50,9 +55,9 @@ func main() {
 
 	log.Println("Shutting down server...")
 
-	// Stop the worker pool
-	workerPool.Stop()
-
+	// Stop the worker pools
+	webhookWorkerPool.Stop()
+	pollingWorkerPool.Stop()
 	log.Println("Server exiting")
 }
 

docs/DESIGN.md

@@ -0,0 +1,17 @@
+# Design
+
+Instead of querying the GitHub API for each request, we will query our database. This will be faster and cheaper. At the organization level, a webhook is created that sends events to our server.
+
+## API
+
+### Endpoints
+
+#### GET /repositories/:id/workflows
+
+#### GET /workflows/:id/stats
+
+#### GET /workflows/:id/runs
+
+#### GET /runs/:id
+
+#### GET /jobs/:id