clean up

server/webhook-ingest/README.md

@@ -8,7 +8,7 @@ A service to ingest GitHub webhooks and publish the data to NATS JetStream.
 
 ### Prerequisites
 
-- **Python 3.9+**
+- **Python 3.12**
 - **Poetry** for dependency management
 - **Docker** for containerization
 
@@ -50,14 +50,17 @@ Service ports:
 ## Environment Variables
 
 - `NATS_URL`: NATS server URL
-- `SECRET`: HMAC secret for verifying GitHub webhooks
+- `NATS_AUTH_TOKEN`: Authorization token for NATS server
+- `WEBHOOK_SECRET`: HMAC secret for verifying GitHub webhooks
+- `TLS_CERT_FILE`: Path to the TLS certificate file (used by NATS server)
+- `TLS_KEY_FILE`: Path to the TLS key file (used by NATS server)
 
 ## Usage
 
 Configure your GitHub webhooks to POST to:
 
 ```
-http://<server>:4200/github
+https://<server>:4200/github
 ```
 
 ### Event Handling
@@ -67,3 +70,103 @@ Events are published to NATS with the subject:
 ```
 github.<owner>.<repo>.<event_type>
 ```
+
+## NATS Configuration with TLS
+
+
+
+You're absolutely right. The NATS configuration with TLS and Let's Encrypt, along with the corresponding environment variables, is crucial for ensuring secure communication and should be highlighted in the README. Here’s an updated version:
+
+---
+
+# WebHook Ingest
+
+## Overview
+
+A service to ingest GitHub webhooks and publish the data to NATS JetStream.
+
+## Setup
+
+### Prerequisites
+
+- **Python 3.12**
+- **Poetry** for dependency management
+- **Docker** for containerization
+
+### Installation
+
+Install dependencies using Poetry:
+
+```bash
+pip install poetry
+poetry install
+```
+
+## Running the Service
+
+### Development
+
+```bash
+fastapi dev
+```
+
+### Production
+
+```bash
+fastapi run
+```
+
+## Docker Deployment
+
+Build and run with Docker Compose:
+
+```bash
+docker-compose up --build
+```
+
+Service ports:
+
+- **Webhook Service**: `4200`
+- **NATS Server**: `4222`
+
+## Environment Variables
+
+- `NATS_URL`: NATS server URL
+- `NATS_AUTH_TOKEN`: Authorization token for NATS server
+- `WEBHOOK_SECRET`: HMAC secret for verifying GitHub webhooks
+- `TLS_CERT_FILE`: Path to the TLS certificate file (used by NATS server)
+- `TLS_KEY_FILE`: Path to the TLS key file (used by NATS server)
+
+## NATS Configuration with TLS
+
+For secure communication in production, NATS can be configured with TLS using Let's Encrypt certificates.
+
+### Steps to Create TLS Certificates
+
+1. **Install Certbot** on your server:
+
+```bash
+sudo apt-get install certbot
+```
+
+2. **Obtain a Certificate**:
+
+```bash
+sudo certbot certonly --standalone -d <your.domain.com>
+```
+
+Replace `<your.domain.com>` with your actual domain name.
+
+3. **Configure NATS** to use the certificate and key in the environment variables:
+
+```bash
+TLS_CERT_FILE=/etc/letsencrypt/live/<your.domain.com>/fullchain.pem
+TLS_KEY_FILE=/etc/letsencrypt/live/<your.domain.com>/privkey.pem
+```
+
+For more detailed instructions and options, refer to the [Certbot documentation](https://certbot.eff.org/).
+
+### Important Notes
+
+- The service automatically sets up a NATS JetStream stream named `github` to store events.
+- Ensure your firewall allows traffic on port 4222 (NATS) and ports 80/443 (Let's Encrypt challenge).

server/webhook-ingest/app/config.py

@@ -1,5 +1,4 @@
 from pydantic_settings import BaseSettings
-from app.logger import logger
 
 
 class Settings(BaseSettings):
@@ -11,5 +10,3 @@ class Config:
         env_file = ".env"
 
 settings = Settings()
-
-logger.info(f"Loaded settings: {settings.model_dump_json()}")

server/webhook-ingest/app/nats_client.py

@@ -7,14 +7,28 @@ def __init__(self):
         self.nc = NATS()
 
     async def connect(self):
+        async def error_cb(e):
+            logger.error(f'There was an error: {e}')
+
+        async def disconnected_cb():
+            logger.info('NATS got disconnected!')
+
+        async def reconnected_cb():
+            logger.info(f'NATS got reconnected to {self.nc.connected_url.netloc}')
+
+        async def closed_cb():
+            logger.info('NATS connection is closed')
+
         await self.nc.connect(
             servers=settings.NATS_URL,
             token=settings.NATS_AUTH_TOKEN,
-            verbose=True,
-            pedantic=True,
             max_reconnect_attempts=-1,
             allow_reconnect=True,
             reconnect_time_wait=2,
+            error_cb=error_cb,
+            disconnected_cb=disconnected_cb,
+            reconnected_cb=reconnected_cb,
+            closed_cb=closed_cb,
         )
         self.js = self.nc.jetstream()
         logger.info(f"Connected to NATS at {self.nc.connected_url.netloc}")
@@ -25,7 +39,6 @@ async def publish(self, subject: str, message: bytes):
 
     async def close(self):
         await self.nc.close()
-        logger.info("Closed connection to NATS")
 
 
 nats_client = NATSClient()

server/webhook-ingest/compose.yaml

@@ -17,7 +17,7 @@ services:
     image: nats:latest
     ports:
       - "4222:4222"
-    command: ["--config", "/etc/nats/nats-server.conf", "-DV"]
+    command: ["--config", "/etc/nats/nats-server.conf"]
     environment:
       - TLS_CERT_FILE
       - TLS_KEY_FILE

server/webhook-ingest/pyproject.toml

@@ -8,7 +8,7 @@ package-mode = false
 
 
 [tool.poetry.dependencies]
-python = "^3.9"
+python = "^3.12"
 fastapi = {extras = ["standard"], version = "0.112.1"}
 nats-py = "2.8.0"
 pydantic = "2.8.2"