Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add message transformer interface #452

Merged
merged 1 commit into from
Nov 19, 2024

Conversation

jfzunigac
Copy link
Contributor

Introducing a message transformer interface for LogFileReaders, enabling the transformation of messages before they are forwarded to LogStreamProcessors. The desired transformer along its configuration can be defined within a LogStream configuration file as follows:

# Configuration for transformer
transformer.type=regex_based_modifier
transformer.regex_based_modifier.regex=(?s)^(.+?) (INFO|ERROR|OFF|DEBUG|SEVERE|WARN) (null) (.*)$
transformer.regex_based_modifier.modifiedMessageFormat={log_level: $2, message: $4, timestamp: $1}
transformer.regex_based_modifier.appendNewLine=true

Please note that configuring message transformers is optional, and only one transformer can be applied per log stream. Currently, we are introducing a single transformer: RegexBasedModifier. This transformer uses a regular expression to extract fields from a message and reconstructs it based on the modifiedMessageFormat specification. Additionally, transformers are intentionally disabled for ThriftReaders, as altering Thrift messages would be too expensive.

For example, with the configuration shown above, the following message:

2024-11-15 23:59:37 OFF null 7301524e-6de0-4b01-87b8

would be transformed to:

{log_level: OFF, message: 7301524e-6de0-4b01-87b8, timestamp: 2024-11-15 23:59:37}

@jfzunigac jfzunigac requested a review from a team as a code owner November 19, 2024 01:21
@jfzunigac jfzunigac merged commit 650880f into pinterest:master Nov 19, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants