| layout | title | categories | author_picture | author_github | seo-title | seo-description | blog_description |
|---|---|---|---|---|---|---|---|
post |
Adding custom fields to JSON logs in Open Liberty |
blog |
Adding custom fields to JSON logs - OpenLiberty.io |
Add custom fields to your application's JSON logs to assist diagnosis of problems. For example, detecting problems occurring in a series of requests from a single user in a session. |
Add custom fields to your application's JSON logs to assist diagnosis of problems. For example, detecting problems occurring in a series of requests from a single user in a session. |
When monitoring applications in Open Liberty servers, you can view the logs to determine what is happening in your application. However, with many incoming logs, you may want want to sort and filter your logs in a certain way to better understand what is occurring in your application.
In Liberty, you can configure your server to log in JSON format. You can send your logs to a log analysis solution that understands JSON such that it can consume the server’s logs. When you want to analyze your server’s application logs, being able to add extra fields to your JSON log records can be very useful.
For example, most real-world applications handle traffic from multiple users. If you want to check the requests made from a specific user, it would be useful to have an extra field to filter application logs by the user’s ID. Without the user ID field, it makes it much more difficult to determine which requests are made by which user, especially if you are trying to diagnose issues occurring with a specific user. Furthermore, it might be useful to have another field for session ID, so that you can analyze and filter application logs for a specific session. This means you can easily diagnose issues occurring with a specific session associated with the user. Extra fields in log records can be very useful in various use cases.
But how do you add fields to log and trace records?
Open Liberty has a little-known API, LogRecordContext. LogRecordContext provides the ability to add extra fields to your log and trace records in Open Liberty. LogRecordContext provides a similar ability to Log4j MDC (Mapped Diagnostic Context).
When the Liberty server is logging in JSON format, application logs include any fields specified in LogRecordContext.
To enable Liberty to log in JSON format, add the following properties to your bootstrap.properties file:
com.ibm.ws.logging.console.format=json
com.ibm.ws.logging.console.log.level=info
com.ibm.ws.logging.console.source=message,trace,accessLog,ffdc,auditThe following examples illustrate how you can add extra fields to your application’s log and trace records.
To use LogRecordContext, import the com.ibm.websphere.logging.hpel.LogRecordContext class.
To add a string-valued extension (e.g. a field called userName) to your application logs, include the following line in your application:
LogRecordContext.addExtension("userName","bob");To add other extensions with boolean, float, int, or long values, the extension name must include the suffixes _bool, _float, _int, _long so that the value is formatted correctly in the JSON output:
LogRecordContext.addExtension("extensionName_bool","true");
LogRecordContext.addExtension("extensionName_int","112233");
LogRecordContext.addExtension("extensionName_float","1.2");
LogRecordContext.addExtension("extensionName_long","132");After including the extensions all logs and trace made on the same thread include the field extensions. Notice that only string-valued extensions have quotes around the JSON values:
Trace statement:
logger.fine("Logging in Liberty");Output:
{"type":"liberty_trace","host":"yushans-mbp-2.war.can.ibm.com","ibm_userDir":"\/Users\/[email protected]\/Documents\/libertyGit\/open-liberty\/dev\/build.image\/wlp\/usr\/","ibm_serverName":"sampleServer","message":"Logging in Liberty","ibm_threadId":"00000047","ibm_datetime":"2019-11-19T13:15:53.254-0500","module":"com.ibm.sample.LoggingServiceJUL","loglevel":"FINE","ibm_sequence":"1574187353254_0000000000001","ext_extensionName_bool":true,"ext_extensionName_float":1.2,"ext_extensionName_int":112233,"ext_thread":"Default Executor-thread-21","ext_userName":"bob","ext_extensionName_long":132}Log statement:
logger.info("Logging with LogRecordContext");Output:
{"type":"liberty_message","host":"yushans-mbp-2.war.can.ibm.com","ibm_userDir":"\/Users\/[email protected]\/Documents\/libertyGit\/open-liberty\/dev\/build.image\/wlp\/usr\/","ibm_serverName":"sampleServer","message":"Logging with LogRecordContext","ibm_threadId":"0000003e","ibm_datetime":"2019-11-19T13:15:53.517-0500","module":"com.ibm.sample.LoggingServiceJUL","loglevel":"INFO","ibm_sequence":"1574187353517_0000000000029","ext_extensionName_bool":true,"ext_extensionName_float":1.2,"ext_extensionName_int":112233,"ext_thread":"Default Executor-thread-15","ext_userName":"bob","ext_extensionName_long":132}If using an ELK (ElasticSearch, Logstash, Kibana) stack to view logs, you can see the extension fields there as well. You might need to refresh the Kibana index so you can use the new fields in your visualizations:
For more information about analyzing logs in ELK, see: https://www.ibm.com/support/knowledgecenter/en/SSEQTP_liberty/com.ibm.websphere.wlp.doc/ae/twlp_elk_stack.html.
Extensions can be removed using the following method:
LogRecordContext.removeExtension(extensionName);After removing an extension, JSON output for subsequent logs and trace made on the same thread do not include that field.
Overall, the LogRecordContext provides the useful ability to add additional fields to your log and trace records so you can organize, analyze, and filter logs more effectively.
Try it out in Open Liberty.