README.md

Content Watcher Service

The Content Watcher Service is a crucial component of the Gateway suite, providing a familiar callback API to retrieve content and publishing announcements from the Frequency blockchain. This document provides an overview of the service, its architecture, and guides for setup and usage.

📗 Table of Contents

• 📖 About the Project
• 🔍 Architecture Overview
• 🔑 Key Features
• 💻 Getting Started
• 🚀 API Documentation
• 🛠 Development
• 🤝 Contributing
• ❓ FAQ
• 📝 License

📖 About the Project

The Content Watcher Service is part of the Gateway suite that provides a Web2-friendly interface for monitoring and retrieving blockchain announcements through a webhook system.

On Frequency, announcements about content (posts, replies, reactions) are stored on-chain, while the content itself is stored off-chain. The Content Watcher Service simplifies the process of monitoring these announcements by automatically watching the blockchain, processing new announcements as they appear, and delivering them to your application in a familiar format through webhooks.

🔍 Architecture Overview

The Content Watcher Service provides a webhook-based system for receiving notifications about new content.

🔑 Key Features

Content Monitoring

• Parse DSNP Messages: Monitor and parse messages on Frequency
• Webhook Integration: Send content to registered webhooks
• Flexible Filtering: Simple Schema and MSA Id based filtering

Scanner Operations

• Start/Stop Control: Manage scanning operations
• Restart Capability: Reset and restart scanning processes
• Block Range Scanning: Specify custom block ranges for content retrieval

💻 Getting Started

This section guides you through setting up the Content Watcher Service for both development and deployment.

Prerequisites

Ensure you have the following installed:

• Node.js
• Docker

Quick Start with Docker

1. Clone the repository and navigate to the project directory:

   git clone https://github.com/ProjectLibertyLabs/gateway
   cd gateway

2. Install dependencies:

   npm install

3. Start auxiliary services:

   docker compose up -d frequency redis ipfs

4. Start the Content Watcher Service:

   docker compose up -d content-watcher-service

Local Development Setup

1. Start required services:

   docker compose up -d frequency redis ipfs

2. Set up environment variables:

   cp env-files/content-watcher.template .env.content-watcher

   Configure the environment variables according to your needs.

3. Start Content-Watcher

   npm run start:content-watcher:dev

Optional Setup Steps

1. Set up publishing services:

   docker compose up -d content-publishing-service-api content-publishing-service-worker

2. Initialize basic publisher:

   npm run setup:content-publishing:chain

3. Start test webhook:

   npm run setup:content-watcher:webhook

4. Generate test content:

   npm run setup:content-watcher:publish

Verification

Docker Setup

• Access Swagger UI: http://localhost:3011/api/docs/swagger
• View and manage queues: http://localhost:3011/queues

Local Development Setup

• Access Swagger UI: http://localhost:3000/api/docs/swagger
• View and manage queues: http://localhost:3000/queues

🚀 API Documentation

• Live API Documentation

🛠 Development

Testing

Run unit tests:

npm run test:content-watcher

Run E2E tests:

make test-content-watcher-services-start

npm run test:e2e:content-watcher

Linting and Formatting

Run linter:

npm run lint

Auto-format code:

npm run format

Built With

• Server Framework: NestJS, Node.js, TypeScript
• Data Store: Redis (ioredis)
• Queue System: BullMQ
• Blockchain Integration: Polkadot API, DSNP
• Testing: Jest, Supertest
• Documentation: Swagger
• Containerization: Docker, Docker Compose

🤝 Contributing

We welcome contributions! Please check our Contributing Guidelines and open issues.

❓ FAQ

Q: Can I use this service in my production social app?

Yes, Gateway Services are designed to be ready-to-use out of the box as part of your social media app using DSNP on Frequency.

Q: Does this service index content?

No. This can be used by your own indexing service to get access to the content, but the service is intentionally limited to getting the content and further customization is open to you.

Q: Does this service filter content?

No. This can be used by your own content filtering service to get new content and then have your custom service process them.

📝 License

This project is licensed under the Apache 2.0 License.

(back to top)