-
Notifications
You must be signed in to change notification settings - Fork 13
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* wip: first setup * feat: working kafka listener * feat: working kafka source in admin * fix: update bean name in tests * feat: working kafka source in admin * feat: add integration test * feat: add integration test * fix test * fix test * chore: add kafka jar * sonar * add assembly * docs: documentation effort * chore: pr remarks * ci: prepare release 3.6.0 --------- Co-authored-by: Yalz <[email protected]>
- Loading branch information
Showing
102 changed files
with
1,370 additions
and
169 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| version: "3.3" | ||
| services: | ||
| ldes-server: | ||
| container_name: basic_ldes-server | ||
| image: ldes/ldes-server | ||
| environment: | ||
| - SPRING_CONFIG_LOCATION=/config/ | ||
| - PYROSCOPE_CONFIGURATION_FILE=/config/pyroscope.properties | ||
| volumes: | ||
| - ./docker-compose/server.config.yml:/config/application.yml:ro | ||
| - ./docker-compose/pyroscope.properties:/config/pyroscope.properties:ro | ||
| ports: | ||
| - "8080:8080" | ||
| - "8087:8087" | ||
| - "8088:8088" | ||
| - "8089:8089" | ||
| networks: | ||
| - ldes | ||
| depends_on: | ||
| - postgres | ||
| postgres: | ||
| container_name: ldes-postgres | ||
| image: postgres:14-alpine | ||
| ports: | ||
| - 5432:5432 | ||
| environment: | ||
| - POSTGRES_PASSWORD=admin | ||
| - POSTGRES_USER=admin | ||
| - POSTGRES_DB=test | ||
| networks: | ||
| - ldes | ||
| pyroscope: | ||
| image: grafana/pyroscope:latest | ||
| ports: | ||
| - 4040:4040 | ||
| networks: | ||
| - ldes | ||
|
|
||
| zookeeper: | ||
| image: zookeeper:3.7.0 | ||
| ports: | ||
| - "2181:2181" | ||
|
|
||
| kafka: | ||
| image: confluentinc/cp-kafka:7.3.2 | ||
| ports: | ||
| - "9092:9092" | ||
| - "29092:29092" | ||
| depends_on: | ||
| - zookeeper | ||
| environment: | ||
| KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 | ||
| KAFKA_LISTENERS: INTERNAL://:9092,EXTERNAL://0.0.0.0:29092 | ||
| KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,EXTERNAL://localhost:29092 | ||
| KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT | ||
| KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL | ||
| ALLOW_PLAINTEXT_LISTENER: 'yes' | ||
| KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 | ||
|
|
||
| kafka-ui: | ||
| image: provectuslabs/kafka-ui:latest | ||
| depends_on: | ||
| - kafka | ||
| ports: | ||
| - "9200:8080" | ||
| environment: | ||
| KAFKA_CLUSTERS_0_NAME: local | ||
| KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:9092 | ||
| KAFKA_CLUSTERS_0_ZOOKEEPER: zookeeper:2181 | ||
| networks: | ||
| ldes: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| --- | ||
| layout: default | ||
| title: Ingest Members With Kafka | ||
| nav_order: 0 | ||
| --- | ||
|
|
||
| # Ingest Members With Kafka | ||
|
|
||
| Ingesting members into an Event Stream without too much overhead? | ||
| That's now possible via ingestion over Apache Kafka. | ||
|
|
||
| ## Getting Started | ||
|
|
||
| To get started with ingesting members via Kafka, you need to have the following: | ||
| * Kafka consumer configuration (in the Application Properties) | ||
| * Event Stream configuration pointing to a Kafka topic (in the Admin API) | ||
|
|
||
| ### Application Properties | ||
|
|
||
| The Kafka consumer configuration can be set in the `application.properties` file. | ||
|
|
||
| The most basic properties that are needed are: | ||
| ````yaml | ||
| spring.kafka.consumer.bootstrap-servers=localhost:9092 | ||
| spring.kafka.consumer.group-id=my-group | ||
| ```` | ||
|
|
||
| To guarantee that the Kafka consumer will always read from the beginning of the topic, you can add the following property: | ||
| ````yaml | ||
| spring.kafka.consumer.auto-offset-reset=earliest | ||
| ```` | ||
|
|
||
| For more advanced options to configure advanced Kafka connections, please refer to the [Spring Kafka documentation](https://docs.spring.io/spring-boot/appendix/application-properties/index.html). | ||
|
|
||
| ### Event Stream Configuration | ||
|
|
||
| To configure a new Event Stream that uses Kafka as the ingestion method, | ||
| you need to create an Event Stream configuration that points to a Kafka topic. \ | ||
| This can be done by adding a `https://w3id.org/ldes#KafkaEventStream` object to the Event Stream configuration. | ||
|
|
||
| This object should contain the following properties: | ||
| * `ldes:topic` - The Kafka topic to which the members should be ingested. | ||
| * `ldes:mimeType` - The mime type in which the data of your topic will be. This is used to parse your member to a model. \ | ||
| This can be `application/ld+json`, `application/json`, `text/turtle`, ... \ | ||
| All members in your topic need to therefor be in one mime type. | ||
|
|
||
| #### Example | ||
|
|
||
| Creating a generic Event Stream named "event-stream" that uses Kafka as the ingestion method. | ||
|
|
||
| ````turtle | ||
| @prefix ldes: <https://w3id.org/ldes#> . | ||
| @prefix dcterms: <http://purl.org/dc/terms/> . | ||
| @prefix prov: <http://www.w3.org/ns/prov#> . | ||
| @prefix tree: <https://w3id.org/tree#>. | ||
| @prefix sh: <http://www.w3.org/ns/shacl#> . | ||
| @prefix server: <http://localhost:8080/> . | ||
| @prefix xsd: <http://www.w3.org/2001/XMLSchema#> . | ||
| @prefix event-stream: <http://localhost:8080/event-stream/> . | ||
| server:event-stream a ldes:EventStream ; | ||
| ldes:timestampPath dcterms:created ; | ||
| ldes:versionOfPath dcterms:isVersionOf ; | ||
| tree:shape [ a sh:NodeShape ] ; | ||
| ldes:kafkaSource [ | ||
| ldes:topic "testTopic" ; | ||
| ldes:mimeType "application/n-quads" ; | ||
| ] . | ||
| ```` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.