Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add gzip support #223

Merged
merged 6 commits into from
Aug 2, 2024
Merged

Add gzip support #223

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
0a62415
Added encoding support for exposed metrics
Athishpranav2003 Jul 29, 2024
87189d9
Updated readme
Athishpranav2003 Jul 29, 2024
29d0dca
Addressed comments
Athishpranav2003 Aug 1, 2024
08389fd
Added UTs
Athishpranav2003 Aug 1, 2024
d1a545b
Addressed comments
Athishpranav2003 Aug 2, 2024
72e2483
Removed dep
Athishpranav2003 Aug 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ More configuration parameters:
- `port`: listen port (default: 24231)
- `metrics_path`: metrics HTTP endpoint (default: /metrics)
- `aggregated_metrics_path`: metrics HTTP endpoint (default: /aggregated_metrics)
- `content_encoding`: encoding format for the exposed metrics (default: identity)
Copy link

@Lusitaniae Lusitaniae Jul 29, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

usually you'd read Accept-Encoding header from the request to determine how to encode the response

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Encoding

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is true
But in our case Prometheus server accepts both gzip and identity (it's not specific) so the request header has both in accept encoding. We cannot determine whether we only need compressed or identity. That's why I configured it in server side


When using multiple workers, each worker binds to port + `fluent_worker_id`.
To scrape metrics from all workers at once, you can access http://localhost:24231/aggregated_metrics.
Expand Down
1 change: 1 addition & 0 deletions fluent-plugin-prometheus.gemspec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ Gem::Specification.new do |spec|

spec.add_dependency "fluentd", ">= 1.9.1", "< 2"
spec.add_dependency "prometheus-client", ">= 2.1.0"
spec.add_dependency "zlib", ">=3.0.0"
ashie marked this conversation as resolved.
Show resolved Hide resolved
spec.add_development_dependency "bundler"
spec.add_development_dependency "rake"
spec.add_development_dependency "rspec"
Expand Down
29 changes: 26 additions & 3 deletions lib/fluent/plugin/in_prometheus.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
require 'fluent/plugin/prometheus_metrics'
require 'net/http'
require 'openssl'
require 'zlib'

module Fluent::Plugin
class PrometheusInput < Fluent::Plugin::Input
Expand Down Expand Up @@ -32,6 +33,9 @@ class PrometheusInput < Fluent::Plugin::Input
config_param :extra_conf, :hash, default: nil, symbolize_keys: true, deprecated: 'See http helper config'
end

desc 'Content encoding of the exposed metrics, Currently supported encoding is identity, gzip. Ref: https://prometheus.io/docs/instrumenting/exposition_formats/#basic-info'
config_param :content_encoding, :string, default: "identity"
Athishpranav2003 marked this conversation as resolved.
Show resolved Hide resolved

def initialize
super
@registry = ::Prometheus::Client.registry
Expand All @@ -54,6 +58,8 @@ def configure(conf)

@base_port = @port
@port += fluentd_worker_id

raise "Invalid content encoding for the exposed metrics endpoint" unless @content_encoding="identity" || @content_encoding="gzip"
Athishpranav2003 marked this conversation as resolved.
Show resolved Hide resolved
Athishpranav2003 marked this conversation as resolved.
Show resolved Hide resolved
end

def multi_workers_ready?
Expand Down Expand Up @@ -184,7 +190,16 @@ def start_webrick
end

def all_metrics
[200, { 'Content-Type' => ::Prometheus::Client::Formats::Text::CONTENT_TYPE }, ::Prometheus::Client::Formats::Text.marshal(@registry)]
body = nil
case @content_encoding
when 'gzip'
Athishpranav2003 marked this conversation as resolved.
Show resolved Hide resolved
gzip = Zlib::GzipWriter.new(StringIO.new)
gzip << ::Prometheus::Client::Formats::Text.marshal(@registry)
body = gzip.close.string
when 'identity'
Athishpranav2003 marked this conversation as resolved.
Show resolved Hide resolved
body = ::Prometheus::Client::Formats::Text.marshal(@registry)
end
[200, { 'Content-Type' => ::Prometheus::Client::Formats::Text::CONTENT_TYPE, 'Content-Encoding' => @content_encoding }, body]
Athishpranav2003 marked this conversation as resolved.
Show resolved Hide resolved
rescue => e
[500, { 'Content-Type' => 'text/plain' }, e.to_s]
end
Expand All @@ -197,8 +212,16 @@ def all_workers_metrics
full_result.add_metrics(resp.body)
end
end

[200, { 'Content-Type' => ::Prometheus::Client::Formats::Text::CONTENT_TYPE }, full_result.get_metrics]
body = nil
case @content_encoding
when 'gzip'
Athishpranav2003 marked this conversation as resolved.
Show resolved Hide resolved
gzip = Zlib::GzipWriter.new(StringIO.new)
gzip << full_result.get_metrics
body = gzip.close.string
when 'identity'
Athishpranav2003 marked this conversation as resolved.
Show resolved Hide resolved
body = full_result.get_metrics
end
[200, { 'Content-Type' => ::Prometheus::Client::Formats::Text::CONTENT_TYPE, 'Content-Encoding' => @content_encoding }, body]
rescue => e
[500, { 'Content-Type' => 'text/plain' }, e.to_s]
end
Expand Down
Loading