diff --git a/Watcher/README.md b/Watcher/README.md index c7b15d9..c4e2d52 100755 --- a/Watcher/README.md +++ b/Watcher/README.md @@ -5,12 +5,11 @@ Developed by [Thales Group CERT](https://github.com/thalesgroup-cert). ## Prerequisites - [Install docker](https://docs.docker.com/install/) -- [Install docker-compose](https://docs.docker.com/compose/install/) ## Launch watcher -- Grab the `docker-compose.yml`, `.env` files and `Searx`, `Rss-bridge` directories (keep the directory structure). +- Grab the `docker-compose.yml`, `.env` files and `Searx` directory (keep the directory structure). - According to your existing infrastructure you may need to configure **Watcher settings** using the `.env` file ([Static configuration](#static-configuration)). -- `docker-compose up` +- `docker compose up` This should run Docker containers. @@ -27,21 +26,21 @@ Please wait until you see: - Try to access Watcher on http://0.0.0.0:9002 or http://yourserverip:9002. - `CONTROL-C` -- `docker-compose down` to stop all containers. +- `docker compose down` to stop all containers. ### Migrate Updates the state of the database in accordance with all current models and migrations. Migrations, their relationships with applications... - docker-compose down - docker-compose run watcher bash + docker compose down + docker compose run watcher bash python manage.py migrate ### Create admin user You will need to create the first superuser to access the `/admin` page. - docker-compose down - docker-compose run watcher bash + docker compose down + docker compose run watcher bash python manage.py createsuperuser ### Populate your database @@ -49,8 +48,8 @@ Populate your database with hundred of banned words and RSS sources related to C Use `populate_db` script: - docker-compose down - docker-compose run watcher bash + docker compose down + docker compose run watcher bash python manage.py populate_db ### Good to know @@ -81,8 +80,8 @@ Time Zone settings in the `.env` file: If you have modified some of these parameters, don't forget to restart all containers: - docker-compose down - docker-compose up + docker compose down + docker compose up ### Access Watcher remotely within your server instance In case of **"Bad Request" Error** when accessing Watcher web interface, fill `ALLOWED_HOST` variable (in `.env` file) with your Watcher Server Instance **IP** / or your **FQDN**. @@ -95,8 +94,8 @@ Please use this syntax: Now, you can restart your instance and the parameters will be taken into account: - docker-compose down - docker-compose up + docker compose down + docker compose up ### SMTP Server Settings (Email Notifications) In the `.env` file: @@ -110,8 +109,8 @@ Website url, which will be the link in the email notifications body: Now, you can restart your instance and the parameters will be taken into account: - docker-compose down - docker-compose up + docker compose down + docker compose up ### TheHive Settings If you want to use **TheHive export**, please fill the **IP** of your TheHive instance and a **generated API key**. @@ -125,8 +124,8 @@ In the `.env` file: Now, you can restart your instance and the parameters will be taken into account: - docker-compose down - docker-compose up + docker compose down + docker compose up ### MISP Settings If you want to use **MISP export**, please fill the **IP** of your MISP instance and an **API key**. @@ -140,8 +139,8 @@ In the `.env` file: Now, you can restart your instance and the parameters will be taken into account: - docker-compose down - docker-compose up + docker compose down + docker compose up ### LDAP Settings You can configure an LDAP authentication within Watcher: @@ -157,31 +156,31 @@ In the `.env` file: Now, you can restart your instance and the parameters will be taken into account: - docker-compose down - docker-compose up + docker compose down + docker compose up ## Troubleshooting ### Remove the database You may want to **reset** your database entirely, in case of troubleshooting or other. To do this, you need to remove the database stored in your host system and restart the instance: - docker-compose down + docker compose down docker volume rm watcher-project_db_data docker volume rm watcher-project_db_log Now, you can rebuild the image and the parameters will be taken into account: - docker-compose up + docker compose up Don't forget to [migrate](#migrate). ### Useful commands -Use `docker-compose up -d` if you want to run it in background. +Use `docker compose up -d` if you want to run it in background. Run interactive shell session on the Watcher container: - docker-compose run watcher bash + docker compose run watcher bash # Use Watcher ## User enrollment @@ -226,18 +225,175 @@ Connect to the `/admin` page: - Fill **Url** text input. - Click on **SAVE**. -## How to use RSS-Bridge to add more sources from Facebook, GoogleSearch, YouTube… -RSS-Bridge is, by default, configured with Twitter only, but users can use it for all other sources like Facebook, DuckDuckGo, GoogleSearch… -To do so, you need to add the new bridge needed in the `Watcher/Rss-bridge/whitelist.txt` file. +## API Key Creation & Management -An RSS-Bridge source URL looks like this: `http://10.10.10.7/?action=display&bridge=Twitter&context=By+username&u=tomchop_&norep=on&nopic=on&noimg=on&noimgscaling=on&format=Mrss` +Connect to the `/admin` page: + +- Click on **API Keys** in **Authentication and Authorization** part. +- Click on **ADD API KEY**. +- Select the **expiration** date. +- Click on **SAVE**. -To add your own custom url, simply change the bridge, if necessary, with the associated parameters (just keep `http://10.10.10.7/` & `format=Mrss`). -You can test RSS-Bridge API with a public instance like this one: [https://wtf.roflcopter.fr/rss-bridge/](https://wtf.roflcopter.fr/rss-bridge/) +You must pass your API keys via the Authorization header. It should be formatted as follows: **`Authorization: Token `** where **** refers to the full generated API key. + + +Below, you will find our 4 modules with their API functions: + +
+ +Threats Watcher
+ +`^api/threats_watcher/trendyword/$` +- **HTTP Method:** GET, POST, PATCH, DELETE +- **Description:** + - **GET:** Returns a list of trending words monitored by the application. + - **POST:** Adds a new trending word to the monitored list. + - **PATCH:** Updates an existing trending word in the monitored list. + - **DELETE:** Removes a trending word from the monitored list. +- **Usage:** Used to get, add, update, or delete currently monitored trending keywords. + +`^api/threats_watcher/bannedword/$` +- **HTTP Method:** GET, POST, PATCH, DELETE +- **Description:** + - **GET:** Returns a list of banned words monitored by the application. + - **POST:** Adds a new banned word to the monitored list. + - **PATCH:** Updates an existing banned word in the monitored list. + - **DELETE:** Removes a banned word from the monitored list. +- **Usage:** Used to get, add, update, or delete currently monitored banned keywords. + +

+ +
+ +Data Leak
+ +`^api/data_leak/keyword/$` +- **HTTP Method:** GET, POST, PATCH, DELETE +- **Description:** + - **GET:** Returns a list of keywords monitored for data leaks. + - **POST:** Adds a new keyword to the monitored list for data leaks. + - **PATCH:** Updates an existing keyword in the monitored list for data leaks. + - **DELETE:** Removes a keyword from the monitored list for data leaks. +- **Usage:** Used to get, add, update, or delete keywords monitored for data leaks. + +`^api/data_leak/alert/$` +- **HTTP Method:** GET, POST, PATCH, DELETE +- **Description:** + - **GET:** Returns a list of alerts related to data leaks. + - **POST:** Adds a new alert for data leaks. + - **PATCH:** Updates an existing alert for data leaks. + - **DELETE:** Removes an alert related to data leaks. +- **Usage:** Used to get, add, update, or delete current alerts regarding data leaks. + +

+ +
+ +Website Monitoring
+ +`^api/site_monitoring/site/$` +- **HTTP Method:** GET, POST, PATCH, DELETE +- **Description:** + - **GET:** Returns a list of sites monitored by the application. + - **POST:** Adds a new site to the monitored list. + - **PATCH:** Updates an existing site in the monitored list. + - **DELETE:** Removes a site from the monitored list. +- **Usage:** Used to get, add, update, or delete currently monitored sites. + +`^api/site_monitoring/alert/$` +- **HTTP Method:** GET, POST, PATCH, DELETE +- **Description:** + - **GET:** Returns a list of alerts related to site monitoring. + - **POST:** Adds a new alert related to site monitoring. + - **PATCH:** Updates an existing alert related to site monitoring. + - **DELETE:** Removes an alert related to site monitoring. +- **Usage:** Used to get, add, update, or delete current alerts regarding site monitoring. + +`^api/site_monitoring/thehive/$` +- **HTTP Method:** POST +- **Description:** + - **POST:** Adds a new integration with TheHive. +- **Usage:** Used to get, add, update, or delete current integrations with TheHive. + +`^api/site_monitoring/misp/$` +- **HTTP Method:** POST +- **Description:** + - **POST:** Adds a new integration with MISP. +- **Usage:** Used to get, add, update, or delete current integrations with MISP. + +

+ +
+ +DNS Finder
+ +`^api/dns_finder/dns_monitored/$` +- **HTTP Method:** GET, POST, PATCH, DELETE +- **Description:** + - **GET:** Returns a list of monitored DNS domains. + - **POST:** Adds a new DNS domain to the monitored list. + - **PATCH:** Updates an existing monitored DNS domain. + - **DELETE:** Removes a DNS domain from the monitored list. +- **Usage:** Used to get, add, update, or delete currently monitored DNS domains. + +`^api/dns_finder/keyword_monitored/$` +- **HTTP Method:** GET, POST, PATCH, DELETE +- **Description:** + - **GET:** Returns a list of monitored DNS keywords. + - **POST:** Adds a new keyword to the monitored list. + - **PATCH:** Updates an existing monitored keyword. + - **DELETE:** Removes a keyword from the monitored list. +- **Usage:** Used to get, add, update, or delete currently monitored DNS keywords. + +`^api/dns_finder/dns_twisted/$` +- **HTTP Method:** GET, POST, PATCH, DELETE +- **Description:** + - **GET:** Returns a list of twisted DNS domains (typosquatting). + - **POST:** Adds a new twisted DNS domain to the monitored list. + - **PATCH:** Updates an existing twisted DNS domain in the monitored list. + - **DELETE:** Removes a twisted DNS domain from the monitored list. +- **Usage:** Used to get, add, update, or delete currently monitored twisted DNS domains. + +`^api/dns_finder/alert/$` +- **HTTP Method:** GET, POST, PATCH, DELETE +- **Description:** + - **GET:** Returns a list of DNS alerts. + - **POST:** Adds a new DNS alert. + - **PATCH:** Updates an existing DNS alert. + - **DELETE:** Removes a DNS alert. +- **Usage:** Used to get, add, update, or delete current DNS alerts. + +`^api/dns_finder/thehive/$` +- **HTTP Method:** POST, PATCH, DELETE +- **Description:** + - **POST:** Adds a new integration with TheHive for DNS. + - **PATCH:** Updates an existing integration with TheHive for DNS. + - **DELETE:** Removes an integration with TheHive for DNS. +- **Usage:** Used to get, add, update, or delete current integrations with TheHive for DNS. + +`^api/dns_finder/misp/$` +- **HTTP Method:** POST, PATCH, DELETE +- **Description:** + - **POST:** Adds a new integration with MISP for DNS. + - **PATCH:** Updates an existing integration with MISP for DNS. + - **DELETE:** Removes an integration with MISP for DNS. +- **Usage:** Used to get, add, update, or delete current integrations with MISP for DNS. + +

+ + +### Specific Details + +To obtain detailed information about a specific item, such as an alert, a monitored site, or any other entity in the system, you can access its details by appending /(?P[^/.]+)/$ to the end of the corresponding API URL. + +For instance, let's say you want to retrieve information about an alert with the ID "1". You would construct the URL as follows: `http://0.0.0.0:9002/api/site_monitoring/alert/1` + +By making a GET request to this URL using your web browser, CURL, or any HTTP client, you will receive comprehensive details about the alert with the ID "1". + +Following this pattern, you can easily navigate and retrieve specific information for any item in the system, ensuring efficient use of the available API endpoints. -RSS API request example: [https://wtf.roflcopter.fr/rss-bridge/?action=display&bridge=Twitter&context=By+username&u=tomchop_&norep=on&nopic=on&noimg=on&noimgscaling=on&format=Mrss](https://wtf.roflcopter.fr/rss-bridge/?action=display&bridge=Twitter&context=By+username&u=tomchop_&norep=on&nopic=on&noimg=on&noimgscaling=on&format=Mrss) ## Thehive & MISP Export You can export **monitored DNS** to [TheHive](https://thehive-project.org/) or [MISP](https://www.misp-project.org/): @@ -289,18 +445,43 @@ To archived **several** alerts: - Select "**Disable selected alerts**". # Update Watcher + +Verify that your local files `/.env`, `/docker-compose.yml` and `/Searx/` are **up-to-date**. + To update Watcher image please follow the instructions below: -- Stop all containers: `docker-compose down` -- Remove the old docker images: `docker rmi felix83000/watcher searx/searx rssbridge/rss-bridge` -- Pull the newer docker images: `docker-compose up` +- Stop all containers: `docker compose down` +- Remove the old docker images: `docker rmi felix83000/watcher searx/searx` +- Pull the newer docker images: `docker compose up` -This will update the all project **Watcher**, **Rss-bridge** and **Searx**. +This will update the Watcher project. -Verify that your local files `/.env`, `/docker-compose.yml` and `/Searx/` are **up-to-date**. +## Migrate & Populate the database (mandatory) + +Updates the state of the database in accordance with all current models and migrations. Populate your database with hundred of banned words and RSS sources related to Cyber Security. + + docker compose down + docker compose run watcher bash + python manage.py migrate + python manage.py populate_db + +## Managing Breaking Changes (optional) + +If the release includes breaking changes, additional steps may be required. Here are the general steps to manage breaking changes: + +1. **Review release notes or documentation**: Check the release notes or documentation for any information about breaking changes and specific instructions on how to handle them. + +2. **Backup data**: Before proceeding with the update, it's advisable to backup any critical data to prevent loss in case of unexpected issues. + +3. **Test in a staging environment**: If possible, test the update in a staging environment to identify any potential issues or conflicts with existing configurations or customizations. + +4. **Update configurations**: Review and update any configurations or settings that may be affected by the breaking changes. + +5. **Modify custom code**: If you have any custom code or scripts that rely on the previous version's functionality, you may need to modify them to work with the new version. + +6. **Run additional migration scripts**: If provided, run any additional migration scripts or commands provided by the developers to handle specific breaking changes. -Sometimes you will see in Watcher logs a **database migration request in red**. -If so, please follow the migration [process](#migrate). +Always refer to the specific instructions provided with the update. # Developers If you want to modify the project and Pull Request (PR) your work, you will need to setup your development environment. @@ -325,7 +506,7 @@ Then, follow the steps below: - **Install Python and Node.js:** `sudo apt install python3 python3-pip -y` **&** `sudo apt install nodejs -y` - **Pull Watcher code:** `git clone ` - **Move to the following directory:** `cd Watcher/Watcher` -- **Install** `python-ldap` **dependencies:** `sudo apt install -y libsasl2-dev python-dev libldap2-dev libssl-dev` +- **Install** `python-ldap` **dependencies:** `sudo apt install _y libsasl2-dev python-dev-is-python3 libldap2-dev libssl-dev` - **Install** `mysqlclient` **dependency:** `sudo apt install default-libmysqlclient-dev` - **Install Python dependencies:** `pip3 install -r requirements.txt` - **Install NLTK/punkt dependency:** `python3 ./nltk_dependencies.py` @@ -377,7 +558,7 @@ In `settings.py` change `HOST` variable to `localhost`: ## Deploy a simple SMTP server to test the email notifications If you are working on a test environment and willing to have email alerts, here is a simple way to configure the SMTP settings to make it work. - Grab the docker-compose file: [here](https://github.com/rnwood/smtp4dev/blob/master/docker-compose.yml). -- Run the command: `docker-compose up` +- Run the command: `docker compose up` - The mails will be available here by default: `localhost:5000` - Modify the mail settings in the environment variables: `vi /.env` - `EMAIL_FROM=from@from.com` @@ -427,4 +608,4 @@ Go to `/Watcher/docs` and run: When commit please add the all `/Watcher/docs` folder and the `README.md` file: - git add ../docs ../README.md + git add ../docs ../README.md \ No newline at end of file