webapp cve processor

This is a go application which downloads cve records and sends them to a topic on kafka.

To run this Application use the following command :

make run

The application downloads the cve records from the nvd database and sends them to a topic on kafka. The application is a producer microservice. Async kafka producer is used to speed up the process of sending the records to the topic.

If you are using a local kafka cluster it is recommended to start the confluent services using the following command:

confluent local services start

then create a topic using the following command:

kafka-topics --create --topic cve --bootstrap-server localhost:9092 --partitions 10 --replication-factor 1

Then set the following environment variables in the .env file:

CVE_URL=https://github.com/CVEProject/cvelistV5/archive/refs/heads/main.zip
ZIP_FILE_PATH=internal/data/cvelistV5.zip
UNZIP_DIR=internal/data
CVE_DIR=internal/data/cvelistV5-main

KAFKA_BOOTSTRAP_SERVERS=localhost:61231
KAFKA_CLIENT_ID=goapp
KAFKA_ACKS=all
KAFKA_MESSAGE_MAX_BYTES=5242880
KAFKA_QUEUE_BUFFERING_MAX_MESSAGES=300000
KAFKA_TOPIC=cve