Re-write documentation

.github/workflows/release.yml

@@ -16,6 +16,9 @@ jobs:
       contents: read
       packages: write
 
+    outputs:
+      extract-version: ${{steps.extract-version.outputs.VERSION}}
+
     steps:
       - name: Checkout repository
         uses: actions/checkout@v3
@@ -35,9 +38,11 @@ jobs:
 
       # Store the version, stripping any v-prefix
       - name: Write release version
+        id: extract-version
         run: |
           TAG=${{ github.event.release.tag_name }}
           echo "VERSION=${TAG#v}" >> $GITHUB_ENV
+          echo "::set-output name=VERSION::$VERSION"
 
       - name: Build and push Docker image
         uses: docker/[email protected]
@@ -47,7 +52,7 @@ jobs:
           push: true
           tags: |
             ghcr.io/cern-cert/pdnssoc:latest
-            ghcr.io/cern-cert/pdnssoc:${VERSION}
+            ghcr.io/cern-cert/pdnssoc:${{ env.VERSION }}
           labels: ${{ steps.meta.outputs.labels }}
           build-args: |
             GO_DNSCOLLECTOR_VERSION=0.36.0

FAQ.md

@@ -0,0 +1,29 @@
+## FAQ
+
+Q: What happens if a malicious domain is added to MISP _after_ a victim resolved that domain?
+
+_A: The pDNSSOC instance automatically reprocesses all the DNS resolutions on a daily basis. The limit is set by configuration or the disc space available on the pDNSSOC instance._
+
+Q: What can you infer from DNS traffic?
+
+_A: The correlation with MISP events can provide accurate contextual information regarding the nature of the threat, its purpose and other indicators of compromise that can be leveraged to investigate the victim system._
+
+Q: Is it possible to connect pDNSSOC to multiple MISP instances?
+
+_A: Yes, pDNSSOC will combine the intel from several MISP instances and allow for specific follow up in the resulting alerts._
+
+Q: An external team has their own "private" MISP instance and pDNSSOC deployment. How could we benefit from it without breaching TLP or privacy?
+
+_A: pDNSSOC can send incoming DNS data to be reprocessed by another pDNSSOC instance operated by a different team.
+
+**Privacy**: DNS collection can be configured to hide the client IP and use either the DNS server IP or the pDNSSOC instance IP instead. As a result, the client IP or originating DNS server is not exposed to the external team.
+
+**TLP**: The external team can alert the originating pDNSSOC or DNS source without revealing specific data regarding the MISP event._
+
+Q: Are there performance or scalability concerns regarding pDNSSOC?
+
+_A: pDNSSOC deployments follow a scale-out approach. Tests showed that a single Raspberry Pi was sufficient to handle data from dozens health organizations._
+
+Q: Is pDNSSOC aimed at prevention?
+
+_A: No. A primary objective of pDNSSOC is to improve incident response capabilities. With sufficient pDNSSOC coverage, CSIRTs should simply add malicious IPs/Domains in MISP to obtain a near-realtime view of the ongoing attack against their community. Connecting pDNSSOC instances as well as MISP instances together allows a community to response together to security incidents._

LICENSE.md

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2023 Christos Arvanitis
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,206 +1,39 @@
-# pDNSSOC
-
-*Correlating DNS logs with threat intel from MISP as a poor man’s SOC.*
+<img alt="GitHub contributors" src="https://img.shields.io/github/contributors/CERN-CERT/pDNSSOC"> <img alt="GitHub release (with filter)" src="https://img.shields.io/github/v/release/CERN-CERT/pDNSSOC"><img alt="GitHub Discussions" src="https://img.shields.io/github/discussions/CERN-CERT/pDNSSOC">
+<br>
+<br>
+<p align="center">
+  <img src="https://github.com/CERN-CERT/pDNSSOC/assets/1295367/a4173633-820e-4da3-9d81-19f222b67ae3" width="30%" height="30%" />
+  <br>For CIRTs with deadlines
 
-## Background & motivation
+</p>
 
-In the Research & Education (R&E) sector, "defending as a global community" is a crucial strategy.
-This involves having the hability to produce and share relevant threat intelligence, and the capability to leverage it accross the entire sector.
 
-Large campuses and research labs typically have dedicated security teams and adequate skills. But even there, deploying and operating a Security Operations Center (SOC) is a hard problem to solve.
-In addition, a significant part of the R&E sector has a relatively low maturity level on threat intel, and no or limited security staff and expertise.
 
-pDNSSOC provides a turn-key solution to both detect and respond to security incidents.
 
 
-## What is pDNSSOC?
 
-**pDNSSOC is aimed at providing SOC capabilities in under 10 sec on a fresh VM.**
+# pDNSSOC
 
-pDNSSOC is a lightweight component correlating DNS logs with threat intelligence aiming at communities with mixed levels of security expertise.
+pDNSSOC is a minimalistic toolset allowing DNS data to be centrally collected, and correlated with malicious domains / IPs from a MISP instance.
 
 Basically:
-1. A central security team is available
-2. The central team operate or has access to one or more MISP instances
-3. The central team deploys and operates a pDNSSOC server
-4. Community members send DNS logs to the pDNSSOC server
-5. The central team receives security alerts about matches between DNS queries made by the client and malicious domains sources from MISP
-
-pDNSSOC logs can be used to:
-- **Detect new intrusions from known malicious domains**
-- **Identify victims in near-realtime as part of incident response**
-
-
-![](docs/images/overview.png)
-
-
-### Client side
-
-In order to get quality and well-formatted data for correlation, the DNS log shipping implementation is based on [go-dnscollector](https://github.com/dmachard/go-dnscollector), a lightweight DNS logging collector supporting various log sources (e.g. AF_PACKET, dnstap, log files).
-
-### Server side
-
-Deploying and operating pDNSSOC is expected to require little/no specific expertise beyond basic sysadmin work.
-A pDNSSOC instance typically runs on a small VM and can handle large volume of DNS data -- many deployment scenarios are possible.
-
-
-## Deploying a pDNSSOC server
-
-The intent is to allow pDNSSOC to be used in environment where:
-- The clients have very limited time and expertise
-- The pDNSSOC operator has security expertise and existing trust relationships with the clients
-
-```mermaid
-flowchart LR
-    A[DNS Logs] --> C(pDNSSOC)
-    B[Threat intelligence\n IOCs] --> C(pDNSSOC)
-    C(pDNSSOC) -->|alerts| D[Clients]
-
-```
-
-### Who should deploy a pDNSSOC server?
-
-pDNSSOC operators are typically security teams at NREN CERTs, e-infratructure security teams, regional or national security teams, central teams of distributed projects/organizations, Managed Security Service Providers, etc.
-
-
-### Deployment
-
-#### Client
-A specific collector needs to be installed on the client side to ensure proper formatting and filtering of DNS logs:
-
-##### Steps
-1. Install `go-dnscollector`
-    ```sh
-    curl -LO  "https://github.com/dmachard/go-dnscollector/releases/download/v${GO_DNSCOLLECTOR_VERSION}/go-dnscollector_${GO_DNSCOLLECTOR_VERSION}_linux_amd64.tar.gz" && \
-    tar xvf "go-dnscollector_${GO_DNSCOLLECTOR_VERSION}_linux_amd64.tar.gz" && \
-    mv go-dnscollector /usr/bin/
-    ```
-2. Modify the collector config file based on the [client template](files/configuration/dnscollector/client.yml).
-3. Start the collector
-    ```sh
-    go-dnscollector -c client.yml
-    ```
-
-#### pDNSSOC server
-
-The main DNS log collection point correlating data with IOCs and producing JSON files ready to be ingested in alerting systems.
-
-##### Requirements
-
-* python3
-* supervisor
-* superfsmon
-
-
-##### Steps
-
-1. Install `go-dnscollector`
-    ```sh
-    curl -LO  "https://github.com/dmachard/go-dnscollector/releases/download/v${GO_DNSCOLLECTOR_VERSION}/go-dnscollector_${GO_DNSCOLLECTOR_VERSION}_linux_amd64.tar.gz" && \
-    tar xvf "go-dnscollector_${GO_DNSCOLLECTOR_VERSION}_linux_amd64.tar.gz" && \
-    mv go-dnscollector /usr/bin/
-    ```
-2. Modify the collector config file based on the [server template](files/configuration/dnscollector/server.yml) and place under `/etc/dnscollector/config.yml`
-3. Create the required files for storing incoming DNS logs:
-    ```sh
-    mkdir -p /var/dnscollector/matches
-    ```
-4. Copy the [postrotate script](files/configuration/dnscollector/postrotate_query.sh) to `/var/dnscollector/postrotate_query.sh`
-5. Ensure that the collecting port set in the previous step is accessible.
-6. Install `pdnssoc-cli`
-    ```sh
-    pip install pdnssoc-cli
-    ```
-7. Modify the pdnssoccli config file based on the [pdnssoccli template](files/configuration/pdnssoccli/pdnssoccli.yml) by adding MISP details.
-8. Install `supervisor` tooling
-    ```sh
-    pip install supervisor superfsmon
-    ```
-9.  Launch `supervisord` with the config file of the [template](files/configuration/supervisor/supervisord.conf).
-    ```ssh
-    supervisord -c supervisord.conf
-    ```
-10. Schedule the following recurring commands (e.g. with crond, timers):
-    * Refresh of IOCs:
-        ```sh
-        pdnssoc-cli -c pdnssoccli.yml fetch-iocs
-        ```
-    * Run correlation:
-        ```sh
-        pdnssoc-cli -c pdnssoccli.yml correlate /var/dnscollector/matches/
-        ```
-    * Run correlation for archived data:
-        ```sh
-        # Pick a date that covers all of your dataset entries
-        pdnssoc-cli -c /etc/pdnssoccli.yml correlate /var/dnscollector/queries/ --start-date 1996-12-10T00:00:00
-        ```
-11. That's it. Alerts can be found at `/var/dnscollector/alerts`.
-
-##### Docker
-
-All of the previous steps for the server part can be automated using the dockerized pdnssoc version:
-
-1. Install `docker`, `docker-compose` and `git`.
-2. Fetch pdnssoc:
-    ```sh
-    git clone --depth 1 --branch v0.0.2 https://github.com/CERN-CERT/pDNSSOC.git
-    ```
-3. Go to the `files/docker` directory
-4. Populate `pdnssoccli.yml` with MISP server details.
-5. Run `docker-compose up`
-
-In case you need to re-build the image, please use the following (after adjusting the tags to match the latest dependency versions):
-```sh
-cd files/docker
-docker build . -t pdnssoc --build-arg GO_DNSCOLLECTOR_VERSION="0.35.0" --build-arg PDNSSOC_CLI_VERSION="v0.0.2"
-```
-
-
-## Important directories
-* `/var/dnscollector/queries/`: One of the default flows for the receiving dns collector is compressing and storing the unfiltered DNS traffic for future correlation. In this directory you will find multiple archives per day hosting the aggregated received DNS traffic.
-* `/var/dnscollector/matches/`: DNS log entries matching the IOC lists defined during ingestion. These files are afterwards enriched with threat intelligence data and stored in `/var/dnscollector/alerts/`.
-* `/var/dnscollector/alerts/`: JSON files ready to be ingested in logging and alerting systems (e.g. Opensearch) or consumed by tools to generate notifications.
-* `/var/dnscollector/misp_domains.txt`: New-line separated file with malicious domains fetched by MISP servers defined in `/etc/dnscollector/pdnssoccli.yml`. Those are used for DNS request correlation.
-* `/var/dnscollector/misp_domains.txt`: New-line separated file with malicious IPs fetched by MISP servers defined in `/etc/dnscollector/pdnssoccli.yml`. Those are used for DNS response correlation.
-
-
-## FAQ
-
-### Deployment questions and support
-
-#### Can I operate a pDNSSOC server for my local community and pass on the data flow to another security team with access to a better MISP instance?
-
-Yes, **providing trust and privacy issues** have been addressed with all parties.  It is possible to configure a pDNSSOC server as a client for another pDNSSOC server.
-
-Forwarding requires adding another logger in the server dnscollector side:
-```yaml
-  loggers:
-    - name: pdnssoc_forwarder
-      dnstap:
-        remote-address: pdnssoc_second_server_ip # Modify with the target pdnssoc destination
-        remote-port: pdnssoc_server_port
-        connect-timeout: 5
-        retry-interval: 10
-        flush-interval: 10
-        tls-support: false
-        tls-insecure: false
-        server-id: "SERVER_ID" @
-        buffer-size: 100
-        chan-buffer-size: 65535
-    - .... Other loggers....
-
-  routes:
-    - from: [ dnstap ]
-      to: [ filelogdomains, filelogips, fileall, pdnssoc_forwarder ]
-```
+- A collector runs on the DNS servers
+- A dedicated pDNSSOC instance collects, correlates and generates alerts.
 
-#### Is it possible to use multiple MISP instance?
-Yes. Just add them in the pdnssoc-cli configuration file.
+The goal is to identify signs of infection on the clients making the DNS requests.
 
-#### Is it possible to add other DNS server types or input format?
+A typical use case would be universities deploying a pDNSSOC client on their DNS server, and sending DNS data to a pDNSSOC server operated by a central CSIRT (NREN, campus, etc.).
 
-Yes. All of the input implementations listed at the `go-dnscollector` [documentation](https://github.com/dmachard/go-dnscollector/blob/main/docs/collectors.md) are supported by pdnssoc-cli
+## Getting started
+* [:bookmark_tabs: Project documentation](../../wiki)
+* [:beetle: Issue tracker](../../issues)
+* [:loudspeaker: Community discussions](../../discussions)
+* [:question: Frequently asked questions](./FAQ.md)
 
-## Bug reports
+## Acknowledgments
+pDNSSOC would not exist without:
+* [go-dnscollector](https://github.com/dmachard/go-dnscollector)
+* [MISP](https://github.com/MISP/MISP/)
 
-Please report issues on [Github](https://github.com/CERN-CERT/pDNSSOC/issues).
+## License
+Distributed under the MIT License. See [LICENSE.md](./LICENSE.md) for more information.