Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

leverage rsyslog for data usage and server issues #30

Open
JessyBarrette opened this issue Apr 11, 2023 · 5 comments
Open

leverage rsyslog for data usage and server issues #30

JessyBarrette opened this issue Apr 11, 2023 · 5 comments
Assignees
@fostermh @JessyBarrette

Comments

@JessyBarrette
Copy link
Collaborator

The latest changes brought in a new erddap-log service based on the docker-erddap

This service combines together the three different logs from tomcat, erddap and access logs to the stdout output, while adding a timestamp and a tag that relates to the source of each entries. See example below:

cioos-pacific-erddap-public-erddap-logs-1  | [TOMCAT] 11-Apr-2023 18:44:49.706 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded Apache Tomcat Native library [1.2.32] using APR version [1.7.0].
cioos-pacific-erddap-public-erddap-logs-1  | [TOMCAT] 11-Apr-2023 18:44:49.706 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true], UDS [true].
cioos-pacific-erddap-public-erddap-logs-1  | [TOMCAT] 11-Apr-2023 18:44:49.717 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.1.1n  15 Mar 2022]
erddap                                     | 11-Apr-2023 18:44:50.175 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
cioos-pacific-erddap-public-erddap-logs-1  | [TOMCAT] 11-Apr-2023 18:44:50.175 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
erddap                                     | 11-Apr-2023 18:44:50.206 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [817] milliseconds
cioos-pacific-erddap-public-erddap-logs-1  | [TOMCAT] 11-Apr-2023 18:44:50.206 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [817] milliseconds
erddap                                     | 11-Apr-2023 18:44:50.261 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
erddap                                     | 11-Apr-2023 18:44:50.262 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat]
cioos-pacific-erddap-public-erddap-logs-1  | [TOMCAT] 11-Apr-2023 18:44:50.261 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
cioos-pacific-erddap-public-erddap-logs-1  | [TOMCAT] 11-Apr-2023 18:44:50.262 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat]
erddap                                     | 11-Apr-2023 18:44:50.273 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/usr/local/tomcat/webapps/ROOT]
cioos-pacific-erddap-public-erddap-logs-1  | [TOMCAT] 11-Apr-2023 18:44:50.273 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/usr/local/tomcat/webapps/ROOT]
erddap                                     | 11-Apr-2023 18:44:50.764 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/usr/local/tomcat/webapps/ROOT] has finished in [490] ms
erddap                                     | 11-Apr-2023 18:44:50.764 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/usr/local/tomcat/webapps/erddap]
cioos-pacific-erddap-public-erddap-logs-1  | [TOMCAT] 11-Apr-2023 18:44:50.764 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/usr/local/tomcat/webapps/ROOT] has finished in [490] ms
cioos-pacific-erddap-public-erddap-logs-1  | [TOMCAT] 11-Apr-2023 18:44:50.764 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/usr/local/tomcat/webapps/erddap]
erddap                                     | 11-Apr-2023 18:44:56.939 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
cioos-pacific-erddap-public-erddap-logs-1  | [TOMCAT] 11-Apr-2023 18:44:56.939 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
cioos-pacific-erddap-public-erddap-logs-1  | [TOMCAT] 11-Apr-2023 18:44:56.940 SEVERE [main] org.apache.tomcat.util.descriptor.web.SecurityConstraint.findUncoveredHttpMethods For security constraints with URL pattern [/*] only the HTTP methods [TRACE DELETE OPTIONS PUT] are covered. All other methods are uncovered.
erddap                                     | 11-Apr-2023 18:44:56.959 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/usr/local/tomcat/webapps/erddap] has finished in [6,194] ms
cioos-pacific-erddap-public-erddap-logs-1  | [TOMCAT] 11-Apr-2023 18:44:56.959 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/usr/local/tomcat/webapps/erddap] has finished in [6,194] ms
erddap                                     | 11-Apr-2023 18:44:56.966 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
cioos-pacific-erddap-public-erddap-logs-1  | [TOMCAT] 11-Apr-2023 18:44:56.966 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
erddap                                     | 11-Apr-2023 18:44:56.985 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [6778] milliseconds
cioos-pacific-erddap-public-erddap-logs-1  | [TOMCAT] 11-Apr-2023 18:44:56.985 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [6778] milliseconds
erddap                                     | 
erddap                                     | ////**** EDStatic Low Level Startup
erddap                                     | localTime=2023-04-11T18:44:57+00:00
erddap                                     | erddapVersion=2.22
erddap                                     | Java 17.0.2 (64 bit, Oracle Corporation) on Linux (3.10.0-957.1.3.el7.x86_64).
erddap                                     | MemoryInUse=   225 MB (highWaterMark=   225 MB) (Xmx ~= 4096 MB)
erddap                                     | got logLevel from ERDDAP_logLevel
erddap                                     | SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
erddap                                     | SLF4J: Defaulting to no-operation (NOP) logger implementation
erddap                                     | SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
....
cioos-pacific-erddap-public-erddap-logs-1  | [ACCESS] 127.0.0.1 - - [11/Apr/2023:18:45:50 +0000] "GET /erddap/index.html HTTP/1.1" 200 22949
cioos-pacific-erddap-public-erddap-logs-1  | [ACCESS] 127.0.0.1 - - [11/Apr/2023:18:46:00 +0000] "GET /erddap/index.html HTTP/1.1" 200 22949

This seems pretty useful and could be handy for tracking the usage of ERDDAP and/or many other aspects.

Right now, rsyslog outputs stdout. We could potentially output to files.
@JessyBarrette
Copy link
Collaborator Author

@steviewanders @n-a-t-e this may interest you

@fostermh
Copy link
Member

I had heard of this, thanks for writing up an issue. I wonder if the combined logs give us anything useful over the regular Apache/Nginx logs from outside the container. tomcat for example is notoriously verbose and uninformative at the same time.

@JessyBarrette
Copy link
Collaborator Author

Thanks @fostermh I'm not sure from which log comes from those ACCESS lines but those are each url request made to erddap. Not sure if that more helpful than Apache/Nginx.

FIY this is replicating the erddap-docker feature presented here:
https://github.com/axiom-data-science/docker-erddap#log-consolidation---experimental

@fostermh
Copy link
Member

Getting usage stats from erddap is a sprint topic coming up for the cioos development team. I have done some work on this already so it would be good to collaborate on this. We can definitely try scraping the combined erddap logs and compare to the apache/nginx only method. I have some software in place already to do this but need to remember where I got to with it.

@JessyBarrette
Copy link
Collaborator Author

@fostermh sounds good this all that have done so far (essentially copied what's suggested by erddap-docker

Happy to help figure out the rest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fostermh fostermh

@JessyBarrette JessyBarrette

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@fostermh @JessyBarrette