Logging in Pravega

Logging and Configuration

Logging is a fundamental aspect of a distributed system to allow developers understanding the activity of the system and debug problems. To this end, Pravega uses Simple Logging Facade for Java (SLF4J) for logging.

Gradle gradle.build file in Pravega project:

{
qosLogbackVersion=1.2.3 
slf4jApiVersion=1.7.25
testCompile group: 'org.slf4j', name: 'log4j-over-slf4j', version: slf4jApiVersion
testCompile group: 'ch.qos.logback', name: 'logback-classic', version: qosLogbackVersion 
}

The configuration of the logging service in Pravega is done via the logback.xml configuration file. Out of the box, Pravega provides the following configuration:

• Pravega logs both to ConsoleAppender and RollingFileAppender (rotated logs are 10MB in size and are retained for 30 days).
• Log level for both appenders is set to INFO.

Moreover, a user may want to set different log levels to messages related with specific system components libraries (Zookeeper, Netty, Bookkeeper, etc.). This is possible by overriding the logger configuration:

<logger name="org.apache.zookeeper" level="DEBUG"/>
<logger name="io.grpc.netty" level="ERROR"/>

Pravega Logging in a Kubernetes Environment

While the default logging configuration may be suitable in certain types of deployments, Pravega can run in a variety of environments that may have certain constrains. For instance, if we deploy Pravega in a Kubernetes environment, we may need to avoid storing rotated logs in the "local storage" space of a pod, as it could get evacuated due to lack of resources. In such a containerized environment, there are two main alternatives:

• Use a logging node agent (recommended): In the Kubernetes world, it is becoming a trend to delegate the collection and storage of logs to an “logging agent”, such as FluentD. In a nutshell, Pravega pods (as any other pod in the cluster) just need to write logs to the standard_out/err. If we chose to deploy a logging agent pod per cluster node, they will collect and store logs from all the pods running on the same node. There is a variety of options to choose the eventual storage for logs, ranging from the local storage of pods (e.g., syslog) to external storage services (e.g., logstash, s3).

• Store logs in a volume pointing to external storage: This approach consists of mounting the directory where logs are being stored to an external storage volume. By doing this, Pravega services writing logs in a local directory (/opt/pravega/logs/) would be actually storing data to an external storage volume, thus avoiding running out of storage resources.

To make it easier the collection of logs in both modes, we provide log collection utilities in the pravega-tools repository.