-
Notifications
You must be signed in to change notification settings - Fork 434
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[VulnCheck] Initial Implementation of VulnCheck Connector
- Loading branch information
Showing
28 changed files
with
2,278 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
src/config.yml | ||
src/__pycache__ | ||
src/logs | ||
src/*.gql | ||
src/.venv |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
FROM python:3.12.5-alpine | ||
ENV CONNECTOR_TYPE=EXTERNAL_IMPORT | ||
|
||
# Copy the connector | ||
COPY src /opt/opencti-connector-vulncheck | ||
|
||
# Install Python modules | ||
# hadolint ignore=DL3003 | ||
RUN apk update && apk upgrade && \ | ||
apk --no-cache add git build-base libmagic libffi-dev libxml2-dev libxslt-dev | ||
|
||
RUN cd /opt/opencti-connector-vulncheck && \ | ||
pip3 install --no-cache-dir -r requirements.txt && \ | ||
apk del git build-base | ||
|
||
# Expose and entrypoint | ||
COPY entrypoint.sh / | ||
RUN chmod +x /entrypoint.sh | ||
ENTRYPOINT ["/entrypoint.sh"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,227 @@ | ||
# OpenCTI VulnCheck Connector | ||
|
||
The VulnCheck Connector for OpenCTI is a standalone Python process designed to | ||
integrate VulnCheck's extensive threat intelligence into the OpenCTI platform. | ||
VulnCheck aggregates data from a wide range of sources, providing actionable | ||
insights into vulnerabilities, exploits, ransomware activity, threat actors, | ||
and more. | ||
|
||
Table of Contents | ||
|
||
<!--toc:start--> | ||
- [OpenCTI VulnCheck Connector](#opencti-vulncheck-connector) | ||
- [Introduction](#introduction) | ||
- [Installation](#installation) | ||
- [Requirements](#requirements) | ||
- [Configuration](#configuration) | ||
- [OpenCTI Configuration](#opencti-configuration) | ||
- [VulnCheck Connector Configuration](#vulncheck-connector-configuration) | ||
- [Deployment](#deployment) | ||
- [Docker Deployment](#docker-deployment) | ||
- [Manual Deployment](#manual-deployment) | ||
- [Usage](#usage) | ||
- [Behavior](#behavior) | ||
- [Data Ingestion](#data-ingestion) | ||
- [Data Volume](#data-volume) | ||
- [NIST and VulnCheck NVD](#nist-and-vulncheck-nvd) | ||
- [Debugging](#debugging) | ||
- [Useful Resources](#useful-resources) | ||
<!--toc:end--> | ||
|
||
## Introduction | ||
|
||
The VulnCheck Connector retrieves and translates data from VulnCheck into STIX | ||
objects. It supports feeds for Known Exploited Vulnerabilities (KEVs), NVD-2, | ||
ransomware, threat actors, exploits, botnets, initial access indicators, and IP | ||
intelligence. Each data source is processed to create structured, meaningful | ||
objects like vulnerabilities, malware, infrastructure, and relationships. | ||
|
||
The VulnCheck Connector helps organizations enrich their threat intelligence | ||
within OpenCTI by automating the ingestion of curated vulnerability and threat | ||
actor data. This enables: | ||
|
||
- **Proactive Risk Management**: Identify and prioritize vulnerabilities | ||
actively exploited in the wild, improving patch management and defensive | ||
measures. | ||
- **Threat Actor Profiling**: Gain insights into ransomware groups, botnets, | ||
and advanced persistent threat (APT) groups with detailed metadata and | ||
associated tactics. | ||
- **Contextual Analysis**: Link vulnerabilities to threat actors, exploits, and | ||
malicious infrastructure to build a comprehensive understanding of potential | ||
threats. | ||
- **Enhanced Situational Awareness**: Integrate multi-source threat | ||
intelligence into a unified platform, enabling more informed decision-making | ||
for security operations and incident response teams. | ||
|
||
## Installation | ||
|
||
### Requirements | ||
|
||
- OpenCTI Platform >= 6.4.5 | ||
- A Valid [VulnCheck](https://www.vulncheck.com/) API Key | ||
|
||
## Configuration | ||
|
||
There are several configuration options, which can be set either in | ||
`docker-compose.yml` (for Docker deployments) or in `config.yml` (for manual | ||
deployments). | ||
|
||
### OpenCTI Configuration | ||
|
||
Below are the parameters you'll need to set for OpenCTI: | ||
|
||
| Parameter | config.yml | Docker Environment Variable | Mandatory | Description | | ||
|-------------------|------------|-----------------------------|-----------|----------------------------------------------| | ||
| OpenCTI URL | `url` | `OPENCTI_URL` | Yes | The URL of the OpenCTI platform. | | ||
| OpenCTI Token | `token` | `OPENCTI_TOKEN` | Yes | The API token for authenticating with OpenCTI. | | ||
|
||
### VulnCheck Connector Configuration | ||
|
||
Below are the parameters you'll need to set for running the connector: | ||
|
||
| Parameter | config.yml | Docker Environment Variable | Default | Mandatory | Description | | ||
|-----------------|----------------|------------------------------------|------------------------------------------------------------------------------------------------------------------------|-----------|-----------------------------------------------------------------------------| | ||
| Connector ID | `id` | `CONNECTOR_ID` | / | Yes | A unique `UUIDv4` identifier for this connector. | | ||
| Connector Type | `type` | `CONNECTOR_TYPE` | EXTERNAL_IMPORT | Yes | Specifies the type of connector. Should always be set to `EXTERNAL_IMPORT`. | | ||
| Connector Name | `name` | `CONNECTOR_NAME` | VulnCheck | Yes | The name of the connector as it will appear in OpenCTI. | | ||
| Connector Scope | `scope` | `CONNECTOR_SCOPE` | VulnCheck | Yes | Must be set as `VulnCheck` | | ||
| Log Level | `log_level` | `CONNECTOR_LOG_LEVEL` | info | Yes | Sets the verbosity of logs. Options: `debug`, `info`, `warn`, `error`. | | ||
| API Base URL | `api_base_url` | `CONNECTOR_VULNCHECK_API_BASE_URL` | None | Yes | The base URL for the VulnCheck API (e.g., `https://api.vulncheck.com/v3`). | | ||
| API Key | `api_key` | `CONNECTOR_VULNCHECK_API_KEY` | None | Yes | The API key for authenticating with VulnCheck's API. | | ||
| Data Sources | `data_sources` | `CONNECTOR_VULNCHECK_DATA_SOURCES` | botnets,epss,exploits,initial-access,ipintel,nist-nvd2,ransomware,snort,suricata,threat-actors,vulncheck-kev,vulncheck-nvd2 | Yes | List of data sources to collect intelligence from. | | ||
|
||
## Deployment | ||
|
||
### Docker Deployment | ||
|
||
Before building the Docker container, you need to set the version of pycti in | ||
`requirements.txt` equal to whatever version of OpenCTI you're running. | ||
Example, `pycti==5.12.20`. If you don't, it will take the latest version, but | ||
sometimes the OpenCTI SDK fails to initialize. | ||
|
||
Build a Docker Image using the provided `Dockerfile`. | ||
|
||
Example: | ||
|
||
```shell | ||
# Replace the IMAGE NAME with the appropriate value | ||
docker build . -t [IMAGE NAME]:latest | ||
``` | ||
|
||
Make sure to replace the environment variables in `docker-compose.yml` with the | ||
appropriate configurations for your environment. Then, start the docker | ||
container with the provided docker-compose.yml | ||
|
||
```shell | ||
docker compose up -d | ||
# -d for detached | ||
``` | ||
|
||
### Manual Deployment | ||
|
||
Create a file `config.yml` based on the provided `config.yml.sample`. | ||
|
||
Replace the configuration variables (especially the "**ChangeMe**" variables) | ||
with the appropriate configurations for you environment. | ||
|
||
Install the required python dependencies (preferably in a virtual environment): | ||
|
||
```shell | ||
pip3 install -r requirements.txt | ||
``` | ||
|
||
Then, start the connector from `vulncheck/src`: | ||
|
||
```shell | ||
python3 main.py | ||
``` | ||
|
||
## Usage | ||
|
||
After Installation, the connector should require minimal interaction to use, | ||
and should update automatically at a regular interval specified in your | ||
`docker-compose.yml` or `config.yml` in `duration_period`. | ||
|
||
However, if you would like to force an immediate download of a new batch of | ||
entities, navigate to: | ||
|
||
`Data management` -> `Ingestion` -> `Connectors` in the OpenCTI platform. | ||
|
||
Find the connector, and click on the refresh button to reset the connector's | ||
state and force a new download of data by re-running the connector. | ||
|
||
> [!NOTE] | ||
> The VulnCheck Connector is designed to fetch data once every 24 hours. This | ||
> approach ensures the connector remains efficient and minimizes the risk of | ||
> overloading the VulnCheck API or the OpenCTI platform. | ||
## Behavior | ||
|
||
The VulnCheck Connector integrates VulnCheck's threat intelligence data into | ||
the OpenCTI platform, converting raw data from various feeds into structured | ||
STIX objects. Below is an overview of its behavior and functionality: | ||
|
||
### Data Ingestion | ||
|
||
The connector retrieves data from VulnCheck's API and imports it into OpenCTI | ||
as STIX objects. The following types of data are processed: | ||
|
||
- **VulnCheck KEV**: Populates OpenCTI with vulnerabilities actively exploited | ||
in the wild, focusing on high-priority risks. | ||
- **NVD-2**: Imports vulnerability information enriched with CVSS scores, | ||
descriptions, and associated CPEs. (VulnCheck NVD-2 is available for | ||
subscribers, NIST NVD-2 is available for other users) | ||
- **Exploits**: Maps exploits to vulnerabilities and generates corresponding | ||
Malware objects in OpenCTI. | ||
- EPSS Enrichment: Adds vulnerabilities along with their EPSS scores and | ||
percentiles, helping prioritize remediation efforts based on exploit | ||
probability. | ||
- **Ransomware**: Creates Malware objects for ransomware families, linking them | ||
to associated vulnerabilities. | ||
- **Threat Actors**: Adds Threat Actor objects with external references, | ||
relationships to targeted vulnerabilities, and descriptive metadata. | ||
- **Botnets**: Ingests infrastructure data associated with botnet activities | ||
and links them to targeted vulnerabilities. | ||
- **Initial Access Indicators**: Maps CPEs and vulnerabilities leveraged for | ||
initial access tactics. | ||
- **IP Intelligence**: Adds infrastructure and IP-related intelligence, | ||
including countries and related vulnerabilities. | ||
- **Snort/Suricata Rules**: Ingests Snort and Suricata rules as Indicators. | ||
|
||
### Data Volume | ||
|
||
> [!WARNING] | ||
> Users should be aware of the significant resource impact when enabling | ||
> certain data sources within the VulnCheck Connector. Enabling any of these | ||
> data sources will require significant time and resources (primarily RAM) to | ||
> process. After the initial ingestion, operations will become more efficient | ||
> with reduced daily data pulls. | ||
#### NIST and VulnCheck NVD | ||
|
||
On the initial run, these feeds ingest every vulnerability available, creating | ||
a massive number of STIX Objects and Relationships (e.g., Software `has` | ||
Vulnerability). | ||
|
||
Subsequent runs will only pull vulnerabilities published on the previous day, | ||
significantly reducing the data volume after the initial ingestion. The first | ||
run requires substantial RAM and processing time due to the volume of data | ||
being processed. Please ensure your environment is prepared to support this | ||
before enabling the source. | ||
|
||
## Debugging | ||
|
||
The connector can be debugged by setting the appropiate log level. Note that | ||
logging messages can be added using | ||
`self.helper.connector_logger,{LOG_LEVEL}("Sample message")`, i. e., | ||
`self.helper.connector_logger.error("An error message")`. | ||
|
||
--- | ||
|
||
## Useful Resources | ||
|
||
OpenCTI documentation for connectors: | ||
|
||
- [OpenCTI Ecosystem](https://filigran.notion.site/OpenCTI-Ecosystem-868329e9fb734fca89692b2ed6087e76) | ||
- [Connectors Deployment](https://docs.opencti.io/latest/deployment/connectors/) | ||
- [Connectors Development](https://docs.opencti.io/latest/development/connectors/) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
version: "3" | ||
services: | ||
connector-vulncheck: | ||
image: opencti/connector-vulncheck:6.2.4 | ||
environment: | ||
# Connector's generic execution parameters | ||
- OPENCTI_URL=http://localhost | ||
- OPENCTI_TOKEN=CHANGEME | ||
# Connector's definition parameters REQUIRED | ||
- CONNECTOR_ID=CHANGEME | ||
- CONNECTOR_NAME=VulnCheck | ||
- CONNECTOR_SCOPE=VulnCheck | ||
- CONNECTOR_LOG_LEVEL=error | ||
- CONNECTOR_DURATION_PERIOD=PT5M # ISO8601 format in String, start with 'P...' for Period | ||
|
||
# Connector's definition parameters OPTIONAL | ||
# - CONNECTOR_QUEUE_THRESHOLD=500 # Default 500Mo, Float accepted | ||
# - CONNECTOR_RUN_AND_TERMINATE=False # Default False, True run connector once | ||
# - CONNECTOR_SEND_TO_QUEUE=True # Default True | ||
# - CONNECTOR_SEND_TO_DIRECTORY=False # Default False | ||
# - CONNECTOR_SEND_TO_DIRECTORY_PATH=CHANGEME # if CONNECTOR_SEND_TO_DIRECTORY is True, you must specify a path | ||
# - CONNECTOR_SEND_TO_DIRECTORY_RETENTION=7 # Default 7, in days | ||
|
||
# Connector's custom execution parameters | ||
- CONNECTOR_VULNCHECK_API_BASE_URL=https://api.vulncheck.com/v3 | ||
- CONNECTOR_VULNCHECK_API_KEY=CHANGEME | ||
- CONNECTOR_VULNCHECK_DATA_SOURCES=botnets,epss,exploits,initial-access,ipintel,nist-nvd2,ransomware,snort,suricata,threat-actors,vulncheck-kev,vulncheck-nvd2 | ||
|
||
# Add proxy parameters below if needed | ||
# - HTTP_PROXY=CHANGEME | ||
# - HTTPS_PROXY=CHANGEME | ||
# - NO_PROXY=CHANGEME | ||
restart: always | ||
# networks: | ||
# - docker_default | ||
# networks: | ||
# default: | ||
# external: true | ||
# name: docker_default |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
#!/bin/sh | ||
|
||
cd /opt/opencti-connector-vulncheck | ||
|
||
# Launch the worker | ||
python3 main.py |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
opencti: | ||
url: 'http://localhost:PORT' | ||
token: 'ChangeMe' | ||
|
||
connector: | ||
id: 'ChangeMe' | ||
type: 'EXTERNAL_IMPORT' | ||
name: 'External Import Connector VulnCheck' | ||
scope: 'VulnCheck' | ||
log_level: 'info' | ||
duration_period: 'PT5M' # Interval given for scheduler process in ISO-8601 format | ||
#============================================# | ||
# Optional connector's definition parameters # | ||
#============================================# | ||
#queue_threshold: 500 | ||
#run_and_terminate: 'False' | ||
#send_to_queue: 'True' | ||
#send_to_directory: 'False' | ||
#send_to_directory_path: 'ChangeMe' | ||
#send_to_directory_retention: 7 | ||
|
||
connector_vulncheck: | ||
api_base_url: 'ChangeMe' | ||
api_key: 'ChangeMe' | ||
data_sources: 'botnets,epss,exploits,initial-access,ipintel,nist-nvd2,snort,suricata,ransomware,threat-actors,vulncheck-kev,vulncheck-nvd2' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
import traceback | ||
|
||
from vclib import ConnectorVulnCheck | ||
|
||
if __name__ == "__main__": | ||
""" | ||
Entry point of the script | ||
- traceback.print_exc(): This function prints the traceback of the exception to the standard error (stderr). | ||
The traceback includes information about the point in the program where the exception occurred, | ||
which is very useful for debugging purposes. | ||
- exit(1): effective way to terminate a Python program when an error is encountered. | ||
It signals to the operating system and any calling processes that the program did not complete successfully. | ||
""" | ||
try: | ||
connector = ConnectorVulnCheck() | ||
connector.run() | ||
except Exception: | ||
traceback.print_exc() | ||
exit(1) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
vulncheck-sdk==0.0.7 | ||
pycti==6.2.18 | ||
pydantic==2.8.2 | ||
validators==0.33.0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
from .connector import ConnectorVulnCheck | ||
|
||
__all__ = ["ConnectorVulnCheck"] |
Oops, something went wrong.