GitBook: [1.6] 18 pages and 18 assets modified

SUMMARY.md

@@ -161,3 +161,4 @@
 * [Ingest Records Manually](development/ingest-records-manually.md)
 * [Golang Output Plugins](development/golang-output-plugins.md)
 * [Developer guide for beginners on contributing to Fluent Bit](development/developer-guide.md)
+

administration/buffering-and-storage.md

@@ -4,49 +4,49 @@ The end-goal of [Fluent Bit](https://fluentbit.io) is to collect, parse, filter
 
 By default when Fluent Bit process data, it uses Memory as a primary and temporary place to store the records, but there are certain scenarios where would be ideal to have a persistent buffering mechanism based in the filesystem to provide aggregation and data safety capabilities.
 
-Choosing the right configuration is critical and the behavior of the service can be conditioned based in the backpressure settings. Before to jump into the configuration properties let's understand the relationship between *Chunks*, *Memory*, *Filesystem* and *Backpressure*.
+Choosing the right configuration is critical and the behavior of the service can be conditioned based in the backpressure settings. Before to jump into the configuration properties let's understand the relationship between _Chunks_, _Memory_, _Filesystem_ and _Backpressure_.
 
 ## Chunks, Memory, Filesystem and Backpressure
 
 Understanding the chunks, buffering and backpressure concepts is critical for a proper configuration. Let's do a recap of the meaning of these concepts.
 
 #### Chunks
 
-When an input plugin (source) emit records, the engine group the records together in a *Chunk*. A Chunk size usually is around 2MB. By configuration, the engine decide where to place this Chunk, the default is that all chunks are created only in memory.
+When an input plugin \(source\) emit records, the engine group the records together in a _Chunk_. A Chunk size usually is around 2MB. By configuration, the engine decide where to place this Chunk, the default is that all chunks are created only in memory.
 
 #### Buffering and Memory
 
 As mentioned above, the Chunks generated by the engine are placed in memory but this is configurable.
 
-If memory is the only mechanism set for the input plugin, it will just store data as much as it can there (memory). This is the fastest mechanism with less system overhead, but if the service is not able to deliver the records fast enough because of a slow network or an unresponsive remote service, Fluent Bit memory usage will increase since it will accumulate more data than it can deliver. 
+If memory is the only mechanism set for the input plugin, it will just store data as much as it can there \(memory\). This is the fastest mechanism with less system overhead, but if the service is not able to deliver the records fast enough because of a slow network or an unresponsive remote service, Fluent Bit memory usage will increase since it will accumulate more data than it can deliver.
 
-On a high load environment with backpressure the risks of having high memory usage is the chance to get killed by the Kernel (OOM Killer). A workaround for this backpressure scenario is to limit the amount of memory in records that an input plugin can register, this configuration property is called ```mem_buf_limit```: if a plugin have enqueued more than ```mem_buf_limit```, it won't be able to ingest more until it data can be delivered or flushed properly. On this scenario the input plugin in question is paused.  
+On a high load environment with backpressure the risks of having high memory usage is the chance to get killed by the Kernel \(OOM Killer\). A workaround for this backpressure scenario is to limit the amount of memory in records that an input plugin can register, this configuration property is called `mem_buf_limit`: if a plugin have enqueued more than `mem_buf_limit`, it won't be able to ingest more until it data can be delivered or flushed properly. On this scenario the input plugin in question is paused.
 
-The workaround of ```mem_buf_limit``` is good for certain scenarios and environments, it helps to control the memory usage of the service, but at the costs that if a file gets rotated while paused, you might lose that data since it won't be able to register new records. This can happen with any input source plugin. The goal of ```mem_buf_limit``` is memory control and survival of the service.
+The workaround of `mem_buf_limit` is good for certain scenarios and environments, it helps to control the memory usage of the service, but at the costs that if a file gets rotated while paused, you might lose that data since it won't be able to register new records. This can happen with any input source plugin. The goal of `mem_buf_limit` is memory control and survival of the service.
 
 For full data safety guarantee, use filesystem buffering.
 
 #### Filesystem buffering to the rescue
 
 Filesystem buffering enabled helps with backpressure and overall memory control.
 
-Behind the scenes, Memory and Filesystem buffering mechanisms are **not** mutual exclusive, indeed when enabling filesystem buffering for your input plugin (source) you are getting the best of the two worlds: performance and data safety. 
+Behind the scenes, Memory and Filesystem buffering mechanisms are **not** mutual exclusive, indeed when enabling filesystem buffering for your input plugin \(source\) you are getting the best of the two worlds: performance and data safety.
 
-When the Filesystem buffering is enabled, the behavior of the engine is different, upon Chunk creation, it stores the content in memory but also it maps a copy on disk (through [mmap(2)](https://man7.org/linux/man-pages/man2/mmap.2.html)), this Chunk is active in memory and backed up in disk is called to be ```up``` which means "the chunk content is up in memory".  
+When the Filesystem buffering is enabled, the behavior of the engine is different, upon Chunk creation, it stores the content in memory but also it maps a copy on disk \(through [mmap\(2\)](https://man7.org/linux/man-pages/man2/mmap.2.html)\), this Chunk is active in memory and backed up in disk is called to be `up` which means "the chunk content is up in memory".
 
-How this Filesystem buffering mechanism deals with high memory usage and backpressure ?: Fluent Bit controls the number of Chunks that are ```up``` in memory.
+How this Filesystem buffering mechanism deals with high memory usage and backpressure ?: Fluent Bit controls the number of Chunks that are `up` in memory.
 
-By default, the engine allows to have 128 Chunks ```up``` in memory in total (considering all Chunks), this value is controlled by service property ```storage.max_chunks_up```. The active Chunks that are ```up```  are ready for delivery and the ones that still are receiving records. Any other remaining Chunk is in a ```down``` state, which means that's only in the filesystem and won't be  ```up``` in memory unless is ready to be delivered.
+By default, the engine allows to have 128 Chunks `up` in memory in total \(considering all Chunks\), this value is controlled by service property `storage.max_chunks_up`. The active Chunks that are `up` are ready for delivery and the ones that still are receiving records. Any other remaining Chunk is in a `down` state, which means that's only in the filesystem and won't be `up` in memory unless is ready to be delivered.
 
-If the input plugin has enabled ```mem_buf_limit``` and ```storage.type``` as ```filesystem```, when reaching the ```mem_buf_limit``` threshold, instead of the plugin being paused, all new data will go to Chunks that are ```down``` in the filesystem. This allows to control the memory usage by the service but also providing a a guarantee that the service won't lose any data.
+If the input plugin has enabled `mem_buf_limit` and `storage.type` as `filesystem`, when reaching the `mem_buf_limit` threshold, instead of the plugin being paused, all new data will go to Chunks that are `down` in the filesystem. This allows to control the memory usage by the service but also providing a a guarantee that the service won't lose any data.
 
-##### Limiting Filesystem  space for Chunks
+**Limiting Filesystem  space for Chunks**
 
 Fluent Bit implements the concept of logical queues: a Chunk based on its Tag, can be routed to multiple destinations, so internally we keep a reference from where a Chunk was created and where it needs to go.
 
 It's common to find cases that if we have multiple destinations for a Chunk, one of the destination might be slower than the other, and maybe one of the destinations is generating backpressure and not all of them. On this scenario how do we limit the amount of filesystem Chunks that we are logically queueing ?.
 
-Starting from Fluent Bit v1.6, we introduced the new configuration property for output plugins called ```storage.total_limit_size``` which limits the number of Chunks that exists in the file system for a certain logical output destination.  If one destinations reaches the ```storage.total_limit_size``` limit, the oldest Chunk from it queue for that logical output destination will be discarded.
+Starting from Fluent Bit v1.6, we introduced the new configuration property for output plugins called `storage.total_limit_size` which limits the number of Chunks that exists in the file system for a certain logical output destination. If one destinations reaches the `storage.total_limit_size` limit, the oldest Chunk from it queue for that logical output destination will be discarded.
 
 ## Configuration
 
@@ -67,7 +67,7 @@ The Service section refers to the section defined in the main [configuration fil
 | storage.path | Set an optional location in the file system to store streams and chunks of data. If this parameter is not set, Input plugins can only use in-memory buffering. |  |
 | storage.sync | Configure the synchronization mode used to store the data into the file system. It can take the values _normal_ or _full_. | normal |
 | storage.checksum | Enable the data integrity check when writing and reading data from the filesystem. The storage layer uses the CRC32 algorithm. | Off |
-| storage.max_chunks_up | If the input plugin has enabled ```filesystem``` storage type, this property sets the maximum number of Chunks that can be ```up``` in memory. This helps to control memory usage. | 128 |
+| storage.max\_chunks\_up | If the input plugin has enabled `filesystem` storage type, this property sets the maximum number of Chunks that can be `up` in memory. This helps to control memory usage. | 128 |
 | storage.backlog.mem\_limit | If _storage.path_ is set, Fluent Bit will look for data chunks that were not delivered and are still in the storage layer, these are called _backlog_ data. This option configure a hint of maximum value of memory to use when processing these records. | 5M |
 | storage.metrics | If `http_server` option has been enable in the main `[SERVICE]` section, this option registers a new endpoint where internal metrics of the storage layer can be consumed. For more details refer to the [Monitoring](monitoring.md) section. | off |
 
@@ -117,13 +117,13 @@ The following example configure a service that offers filesystem buffering capab
 
 If certain chunks are filesystem storage.type based, it's possible to control the size of the logical queue for an output plugin. The following table describe the options available:
 
-| Key                      | Description                                                  | Default |
-| ------------------------ | ------------------------------------------------------------ | ------- |
-| storage.total_limit_size | Limit the maximum number of Chunks in the filesystem for the current output logical destination. |         |
+| Key | Description | Default |
+| :--- | :--- | :--- |
+| storage.total\_limit\_size | Limit the maximum number of Chunks in the filesystem for the current output logical destination. |  |
 
-The following example create records with CPU usage samples in the filesystem and then they are delivered to Google Stackdriver service limiting the logical queue (buffering) to 5M:
+The following example create records with CPU usage samples in the filesystem and then they are delivered to Google Stackdriver service limiting the logical queue \(buffering\) to 5M:
 
-```
+```text
 [SERVICE]
     flush                     1
     log_Level                 info
@@ -142,4 +142,5 @@ The following example create records with CPU usage samples in the filesystem an
     storage.total_limit_size  5M
 ```
 
-If for some reason Fluent Bit get's offline because of a network issue, it will continuing buffering CPU samples but just keeping a maximum of 5M of the newest data.
+If for some reason Fluent Bit get's offline because of a network issue, it will continuing buffering CPU samples but just keeping a maximum of 5M of the newest data.
+

concepts/data-pipeline/buffer.md

@@ -8,7 +8,7 @@ Previously defined in the [Buffering](../buffering.md) concept section, the `buf
 
 The `buffer` phase already contains the data in an immutable state, meaning, no other filter can be applied.
 
-![](../../.gitbook/assets/logging_pipeline_buffer%20%281%29.png)
+![](../../.gitbook/assets/logging_pipeline_buffer%20%281%29%20%281%29.png)
 
 {% hint style="info" %}
 Note that buffered data is not raw text, it's in Fluent Bit's internal binary representation.

concepts/data-pipeline/filter.md

@@ -6,7 +6,7 @@ description: 'Modify, Enrich or Drop your records'
 
 In production environments we want to have full control of the data we are collecting, filtering is an important feature that allows us to **alter** the data before delivering it to some destination.
 
-![](../../.gitbook/assets/logging_pipeline_filter%20%281%29.png)
+![](../../.gitbook/assets/logging_pipeline_filter%20%281%29%20%281%29.png)
 
 Filtering is implemented through plugins, so each filter available could be used to match, exclude or enrich your logs with some specific metadata.
 

concepts/data-pipeline/input.md

@@ -6,7 +6,7 @@ description: The way to gather data from your sources
 
 [Fluent Bit](http://fluentbit.io) provides different _Input Plugins_ to gather information from different sources, some of them just collect data from log files while others can gather metrics information from the operating system. There are many plugins for different needs.
 
-![](../../.gitbook/assets/logging_pipeline_input.png)
+![](../../.gitbook/assets/logging_pipeline_input%20%281%29.png)
 
 When an input plugin is loaded, an internal _instance_ is created. Every instance has its own and independent configuration. Configuration keys are often called **properties**.
 

concepts/data-pipeline/parser.md

@@ -6,7 +6,7 @@ description: Convert Unstructured to Structured messages
 
 Dealing with raw strings or unstructured messages is a constant pain; having a structure is highly desired. Ideally we want to set a structure to the incoming data by the Input Plugins as soon as they are collected:
 
-![](../../.gitbook/assets/logging_pipeline_parser.png)
+![](../../.gitbook/assets/logging_pipeline_parser%20%281%29.png)
 
 The Parser allows you to convert from unstructured to structured data. As a demonstrative example consider the following Apache \(HTTP Server\) log entry:
 

concepts/data-pipeline/router.md

@@ -6,7 +6,7 @@ description: Create flexible routing rules
 
 Routing is a core feature that allows to **route** your data through Filters and finally to one or multiple destinations. The router relies on the concept of [Tags](../key-concepts.md) and [Matching](../key-concepts.md) rules
 
-![](../../.gitbook/assets/logging_pipeline_routing%20%281%29.png)
+![](../../.gitbook/assets/logging_pipeline_routing%20%281%29%20%281%29.png)
 
 There are two important concepts in Routing:
 

installation/docker.md

@@ -82,3 +82,4 @@ Alpine Linux uses Musl C library instead of Glibc. Musl is not fully compatible
 Our Docker containers images are deployed thousands of times per day, we take security and stability very seriously.
 
 The _latest_ tag _most of the time_ points to the latest stable image. When we release a major update to Fluent Bit like for example from v1.3.x to v1.4.0, we don't move _latest_ tag until 2 weeks after the release. That give us extra time to verify with our community that everything works as expected.
+

installation/kubernetes.md

@@ -4,7 +4,7 @@ description: Kubernetes Production Grade Log Processor
 
 # Kubernetes
 
-![](../.gitbook/assets/fluentbit_kube_logging%20%281%29.png)
+![](../.gitbook/assets/fluentbit_kube_logging%20%283%29.png)
 
 [Fluent Bit](http://fluentbit.io) is a lightweight and extensible **Log Processor** that comes with full support for Kubernetes:
 
@@ -92,6 +92,33 @@ The default configuration of Fluent Bit makes sure of the following:
 * The default backend in the configuration is Elasticsearch set by the [Elasticsearch Ouput Plugin](https://docs.fluentbit.io/manual/v/1.0/output/elasticsearch). It uses the Logstash format to ingest the logs. If you need a different Index and Type, please refer to the plugin option and do your own adjustments.
 * There is an option called **Retry\_Limit** set to False, that means if Fluent Bit cannot flush the records to Elasticsearch it will re-try indefinitely until it succeed.
 
+## Container Runtime Interface \(CRI\) parser
+
+Fluent Bit by default assumes that logs are formatted by the Docker interface standard. However, when using CRI you can run into issues with malformed JSON if you do not modify the parser used. Fluent Bit includes a CRI log parser that can be used instead. An example of the parser is seen below:
+
+```text
+# CRI Parser
+[PARSER]
+    # http://rubular.com/r/tjUt3Awgg4
+    Name cri
+    Format regex
+    Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
+    Time_Key    time
+    Time_Format %Y-%m-%dT%H:%M:%S.%L%z
+```
+
+To use this parser change the Input section for your configuration from `docker` to `cri`
+
+```text
+[INPUT]
+        Name tail
+        Path /var/log/containers/*.log
+        Parser cri
+        Tag kube.*
+        Mem_Buf_Limit 5MB
+        Skip_Long_Lines On
+```
+
 ## Windows Deployment
 
 Since v1.5.0, Fluent Bit supports deployment to Windows pods.

installation/upgrade-notes.md

@@ -4,23 +4,21 @@ The following article cover the relevant notes for users upgrading from previous
 
 For more details about changes on each release please refer to the [Official Release Notes](https://fluentbit.io/announcements/).
 
-
-
 ## Fluent Bit v1.6
 
 If you are migrating from previous version of Fluent Bit please review the following important changes:
 
 #### Tail Input Plugin
 
-Now by default the plugin follows a file from the end once the service starts (old behavior was always read from the beginning). For every file found at start, its followed from it last position, for new files discovered at runtime or rotated, they are read from the beginning.
+Now by default the plugin follows a file from the end once the service starts \(old behavior was always read from the beginning\). For every file found at start, its followed from it last position, for new files discovered at runtime or rotated, they are read from the beginning.
 
-If you desire to keep the old behavior you can set the option ```read_from_head``` to true. 
+If you desire to keep the old behavior you can set the option `read_from_head` to true.
 
 ### Stackdriver Output Plugin
 
-The project_id of [resource](https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource) in [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) sent to Google Cloud Logging would be set to the project ID rather than the project number. To learn the difference between Project ID and project number, see [this](https://cloud.google.com/resource-manager/docs/creating-managing-projects#before_you_begin) for more details.
+The project\_id of [resource](https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource) in [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) sent to Google Cloud Logging would be set to the project ID rather than the project number. To learn the difference between Project ID and project number, see [this](https://cloud.google.com/resource-manager/docs/creating-managing-projects#before_you_begin) for more details.
 
-If you have any existing queries based on the resource's project_id, please update your query accordingly.
+If you have any existing queries based on the resource's project\_id, please update your query accordingly.
 
 ## Fluent Bit v1.5
 

installation/windows.md

@@ -128,3 +128,4 @@ To halt the Fluent Bit service, just execute the "stop" command.
 ```text
 % sc.exe stop fluent-bit
 ```
+