Skip to content

Latest commit

 

History

History
29 lines (23 loc) · 1.06 KB

buffer.md

File metadata and controls

29 lines (23 loc) · 1.06 KB
description
Data processing with reliability

Buffer

Previously defined in the Buffering concept section, the buffer phase in the pipeline aims to provide a unified and persistent mechanism to store your data, either using the primary in-memory model or using the filesystem based mode.

The buffer phase already contains the data in an immutable state, meaning, no other filter can be applied.

graph LR
    accTitle: Fluent Bit data pipeline
    accDescr: The Fluent Bit data pipeline includes input, a parser, a filter, a buffer, routing, and various outputs.
    A[Input] --> B[Parser]
    B --> C[Filter]
    C --> D[Buffer]
    D --> E((Routing))
    E --> F[Output 1]
    E --> G[Output 2]
    E --> H[Output 3]
    style D stroke:darkred,stroke-width:2px;
Loading

{% hint style="info" %} Note that buffered data is not raw text, it's in Fluent Bit's internal binary representation. {% endhint %}

Fluent Bit offers a buffering mechanism in the file system that acts as a backup system to avoid data loss in case of system failures.