Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add kafka topic with basic sample #680

Merged
merged 5 commits into from
Sep 1, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ This is the **first** of my DevOps trio repositories: [devops-basics](https://gi
DevOps combines development (Dev) and operations (Ops) to increase the efficiency, speed, and security of software development and delivery compared to traditional processes. A more nimble software development lifecycle results in a competitive advantage for businesses and their customers (source: GitLab)

### Overview

- ➡️ [getting-started](./getting-started/)
- ➡️ [DevOps toolchain](https://en.wikipedia.org/wiki/DevOps_toolchain)
- ➡️ [roadmap.sh/devops](https://roadmap.sh/devops)
Expand Down Expand Up @@ -230,22 +231,27 @@ We cover a wide range of DevOps topics in our content library, explore them unde
<td>📖 <a href="https://developer.hashicorp.com/vault/docs">hashicorp.com/vault</a></td>
<td>⏩ coming-soon</td>
</tr>
<tr>
<tr>
<td><img width="32" src="https://upload.wikimedia.org/wikipedia/commons/8/87/Sql_data_base_with_logo.png"></td>
<td>SQL</td>
<td><a href="./topics/sql/">sql</a></td>
<td>📖 <a href="https://github.com/tungbq/devops-basic/blob/main/topics/sql/README.md">sql/README.md</a></td>
<td>✔️ <a href="./topics/sql/mysql-basics.md">mysql-basics</a></td>
</tr>

<tr>
<tr>
<td><img height="28" src="https://upload.wikimedia.org/wikipedia/commons/a/ab/Haproxy-logo.png" /></td>
<td>HAProxy</td>
<td><a href="./topics/haproxy/">haproxy</a></td>
<td>📖 <a href="https://www.haproxy.org/">www.haproxy.org</a></td>
<td>✔️ <a href="./topics/haproxy/basic/">HAProxy basics</a></td>
</tr>

<tr>
<td><img height="28" src="https://upload.wikimedia.org/wikipedia/commons/thumb/0/05/Apache_kafka.svg/800px-Apache_kafka.svg.png" /></td>
<td>Kafka</td>
<td><a href="./topics/kafka/">kafka</a></td>
<td>📖 <a href="https://kafka.apache.org/">kafka.apache.org/</a></td>
<td>✔️ <a href="./topics/kafka/basic/">Kafka basics</a></td>
</tr>
</table>

- And **more upcoming topics...⏩** you can star/follow this repository to get more up-to-dated content ⭐
Expand Down
53 changes: 53 additions & 0 deletions topics/kafka/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
## 1. What is Apache Kafka?

- [Introduction to Apache Kafka](https://kafka.apache.org/documentation/)
- [Youtube - What is Apache Kafka?](https://youtu.be/vHbvbwSEYGo?si=SbouSV-0NZzigsXV)

### Overview

- Apache Kafka is a distributed event streaming platform capable of handling trillions of events a day. It is used for building real-time data pipelines and streaming applications. Kafka is horizontally scalable, fault-tolerant, and fast.

- Kafka allows you to publish, subscribe to, store, and process streams of records in real-time. It is often used in scenarios where data needs to be processed or moved between systems efficiently, such as log aggregation, real-time analytics, or as a backbone for microservices.

### Kafka Architecture

- [Understanding Kafka Architecture](https://kafka.apache.org/10/documentation/streams/architecture)

### Official Website Documentation for Apache Kafka

- [Apache Kafka Documentation](https://kafka.apache.org/documentation/)

## 2. Prerequisites

- Basic Linux command line skills
- Understanding of distributed systems and event streaming concepts

## 3. Installation

### How to install Apache Kafka?

- [Kafka Quickstart Guide](https://kafka.apache.org/quickstart)

## 4. Basics of Apache Kafka

### Getting Started with Kafka

- [Kafka 101: Getting Started with Kafka](https://kafka.apache.org/quickstart)

### Kafka Basics 👋

- See: [**basic**](./basic/)

## 5. Beyond the Basics

- TODO

## 6. More...

### Kafka cheatsheet

- https://www.redpanda.com/guides/kafka-tutorial-kafka-cheat-sheet

### Recommended Books

- None
57 changes: 57 additions & 0 deletions topics/kafka/basic/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
## Kafka Basics

Here's a basic "Hello World" example for Apache Kafka using Docker and Docker Compose. This will set up a Kafka broker and a Zookeeper instance, allowing you to produce and consume messages.

### 1. Create a `docker-compose.yml` File

Create a [docker-compose.yml](./docker-compose.yml) file that defines the services for Zookeeper and Kafka.

### 2. Start Kafka and Zookeeper

Run the following command to start the Kafka and Zookeeper containers:

```bash
cd devops-basics/topics/kafka/basic
docker-compose up -d
```

This command will start Zookeeper and Kafka in the background.

### 3. Create a Kafka Topic

Once the containers are running, create a Kafka topic named `helloworld`.

```bash
docker exec kafka kafka-topics.sh --create --topic helloworld --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1
```

### 4. Produce Messages to the Kafka Topic

To send a message to the `helloworld` topic:

```bash
docker exec -it kafka kafka-console-producer.sh --topic helloworld --bootstrap-server localhost:9092
```

Type a message (e.g., `Hello, Kafka!`) and press Enter. This sends the message to the Kafka topic.

### 5. Consume Messages from the Kafka Topic

To read the message from the `helloworld` topic:

```bash
docker exec -it kafka kafka-console-consumer.sh --topic helloworld --bootstrap-server localhost:9092 --from-beginning
```

You should see the message you produced earlier.

### 6. Cleanup

To stop and remove the Kafka and Zookeeper containers, run:

```bash
cd devops-basics/topics/kafka/basic
docker-compose down
```

This basic setup allows you to get hands-on experience with Kafka using Docker and Docker Compose. You can extend this setup to explore more advanced Kafka features.
23 changes: 23 additions & 0 deletions topics/kafka/basic/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
version: '3.8'
services:
zookeeper:
image: bitnami/zookeeper:latest
container_name: zookeeper
environment:
- ZOO_ENABLE_AUTH=no
- ALLOW_ANONYMOUS_LOGIN=yes
ports:
- '2181:2181'

kafka:
image: bitnami/kafka:latest
container_name: kafka
ports:
- '9092:9092'
environment:
- KAFKA_BROKER_ID=1
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- KAFKA_LISTENERS=PLAINTEXT://:9092
- KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092
depends_on:
- zookeeper