The library provides a pipeline engine for stream processing of data. It is based on the concept of EIP (enterprise integration patterns).
The motivation for this project is simple: to get an easy and clear way of coding ETL-like programs for parallel processing of data. In my case it was a BFS crawler tuned for extraction of specific metadata, (see a basic version in example folder).
The library provides the following primitives:
- route (chain of nodes processing messages)
- node:
- filter -- filter a stream of exchanges
- processor -- processes a stream of exchanges (tasks) in parallel
- wire tap: sends a copy of msg to another route (referenced by name)
- to: enrich msg on another route (request-reply / enrichment pattern)
All the primitives are accessible through DSL.
| Method | Signature | Args |
|---|---|---|
| Source | f func(n *Node) | function f, used for generation of exchanges |
| Filter | f func(e Exchange, n *Node) | function f, intercepts exchanges |
| Process | workers int | Number of workers, beeing used to process exchanges |
| To | route string | Name of the route, where to redirect an exchange for execution of request-reply |
| WireTap | route string | Name of the route, where to copy an exchange |
| Sink | f func(e Exchange) error | Function f, used for consumption exchanges |
- a simplistic bfs crawler:
go run examples/crawler/main.go
- each node is connected with the next one (if exists) only with 1 channel
- node owns an input channel
- output is just a reference to the input of next node
- node does not close the output channel, instead it just sends a Stop msg to a next node
- if a node is the last in a chain than an output message being sent is discarded unless it's not a RequestReply
make code-coverage and see the coverage.html for details.
github.com/Zensey/go-data-routing/node.go (100.0%)
github.com/Zensey/go-data-routing/nodetype_string.go (75.0%)
github.com/Zensey/go-data-routing/pool.go (100.0%)
github.com/Zensey/go-data-routing/route_builder.go (71.8%)
github.com/Zensey/go-data-routing/router_context.go (100.0%)
github.com/Zensey/go-data-routing/worker.go (81.8%)