Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove access logs older than 3 months #45

Closed
Tracked by #1465
mahalakshme opened this issue Oct 24, 2024 · 5 comments
Closed
Tracked by #1465

Remove access logs older than 3 months #45

mahalakshme opened this issue Oct 24, 2024 · 5 comments
Assignees
@1t5j0y

Comments

@mahalakshme
Copy link
Contributor

mahalakshme commented Oct 24, 2024 •
edited by 1t5j0y
Loading

Issue:

Not clearing the old access logs is leading to GBs of uncleared old data - which can be cleared regularly to avoid out of disk space issue.

AC:

Clear access-logs older than 3 months from /var/log/avni_server/log folder.
@mahalakshme mahalakshme converted this from a draft issue Oct 24, 2024
@mahalakshme mahalakshme moved this from In Analysis to In Analysis Review in Avni Product Oct 24, 2024
@mahalakshme mahalakshme moved this from In Analysis Review to Analysis Complete in Avni Product Oct 24, 2024
@1t5j0y 1t5j0y changed the title Remove access logs older than 3 months via ansible Remove access logs older than 3 months Oct 25, 2024
@1t5j0y 1t5j0y moved this from Analysis Complete to Ready in Avni Product Oct 25, 2024
@1t5j0y 1t5j0y moved this from Ready to In Progress in Avni Product Oct 25, 2024
@1t5j0y 1t5j0y self-assigned this Oct 25, 2024
1t5j0y added a commit to avniproject/avni-server that referenced this issue Oct 25, 2024
@1t5j0y
avniproject/avni-infra#45 | Disable console logging for live profile …
135c9a0 
…(causes bloated syslog)
1t5j0y added a commit to avniproject/avni-server that referenced this issue Oct 25, 2024
@1t5j0y
Revert "avniproject/avni-infra#45 | Disable console logging for live …
0a34c4c 
…profile (causes bloated syslog)"

This reverts commit 135c9a0.
1t5j0y added a commit to avniproject/avni-server that referenced this issue Oct 28, 2024
@1t5j0y
avniproject/avni-infra#45 | Change log level for ehcache DefaultSizeO…
d78a58f 
…fEngine to ERROR
1t5j0y added a commit to avniproject/avni-server that referenced this issue Oct 28, 2024
@1t5j0y
avniproject/avni-infra#45 | Configure tomcat access log max retention
1c89414
@1t5j0y
Copy link
Contributor

1t5j0y commented Oct 28, 2024 •
edited
Loading

/var/log/syslog and corresponding rotated files had the largest disk space usage. This was due to systemd journal forwarding logs to syslog.

This forwarding has been temporarily disabled in avni cloud staging, prerelease and prod.
set ForwardToSyslog=no in /etc/systemd/journald.conf
sudo systemctl restart systemd-journald

application logs -> journal logs -> syslogs were getting bloated due to repeated warning logs from ehcache DefaultSizeOfEngine since release 10.0.x. Log level has been changed to error for this. Syslog itself is being rotated by logrotate but we were writing too fast for the rotation to help.

Deletion of older tomcat access logs being tested with tomcat config server.tomcat.accesslog.max-days since we are using tomcat 9. If this does not work, will add a cron/logrotate solution for this.

1t5j0y added a commit to avniproject/avni-server that referenced this issue Oct 29, 2024
@1t5j0y
avniproject/avni-infra#45 | Customize tomcat to respect access log ma…
9f2c164 
…x-days config
1t5j0y added a commit to avniproject/avni-server that referenced this issue Oct 29, 2024
@1t5j0y
avniproject/avni-infra#45 | Include server.tomcat.accesslog.max-days …
9abb067 
…in test/application.properties
@1t5j0y
Copy link
Contributor

1t5j0y commented Oct 29, 2024

Staging access logs

Before: 
ubuntu@ip-10-20-1-175:/var/log/avni_server/log$ ls
access_log.2023-11-15.log  access_log.2024-01-04.log  access_log.2024-02-23.log  access_log.2024-04-13.log  access_log.2024-06-02.log  access_log.2024-07-22.log  access_log.2024-09-10.log
access_log.2023-11-16.log  access_log.2024-01-05.log  access_log.2024-02-24.log  access_log.2024-04-14.log  access_log.2024-06-03.log  access_log.2024-07-23.log  access_log.2024-09-11.log
access_log.2023-11-17.log  access_log.2024-01-06.log  access_log.2024-02-25.log  access_log.2024-04-15.log  access_log.2024-06-04.log  access_log.2024-07-24.log  access_log.2024-09-12.log
access_log.2023-11-18.log  access_log.2024-01-07.log  access_log.2024-02-26.log  access_log.2024-04-16.log  access_log.2024-06-05.log  access_log.2024-07-25.log  access_log.2024-09-13.log
access_log.2023-11-19.log  access_log.2024-01-08.log  access_log.2024-02-27.log  access_log.2024-04-17.log  access_log.2024-06-06.log  access_log.2024-07-26.log  access_log.2024-09-14.log
access_log.2023-11-20.log  access_log.2024-01-09.log  access_log.2024-02-28.log  access_log.2024-04-18.log  access_log.2024-06-07.log  access_log.2024-07-27.log  access_log.2024-09-15.log
access_log.2023-11-21.log  access_log.2024-01-10.log  access_log.2024-02-29.log  access_log.2024-04-19.log  access_log.2024-06-08.log  access_log.2024-07-28.log  access_log.2024-09-16.log
access_log.2023-11-22.log  access_log.2024-01-11.log  access_log.2024-03-01.log  access_log.2024-04-20.log  access_log.2024-06-09.log  access_log.2024-07-29.log  access_log.2024-09-17.log
access_log.2023-11-23.log  access_log.2024-01-12.log  access_log.2024-03-02.log  access_log.2024-04-21.log  access_log.2024-06-10.log  access_log.2024-07-30.log  access_log.2024-09-18.log
access_log.2023-11-24.log  access_log.2024-01-13.log  access_log.2024-03-03.log  access_log.2024-04-22.log  access_log.2024-06-11.log  access_log.2024-07-31.log  access_log.2024-09-19.log
access_log.2023-11-25.log  access_log.2024-01-14.log  access_log.2024-03-04.log  access_log.2024-04-23.log  access_log.2024-06-12.log  access_log.2024-08-01.log  access_log.2024-09-20.log
access_log.2023-11-26.log  access_log.2024-01-15.log  access_log.2024-03-05.log  access_log.2024-04-24.log  access_log.2024-06-13.log  access_log.2024-08-02.log  access_log.2024-09-21.log
access_log.2023-11-27.log  access_log.2024-01-16.log  access_log.2024-03-06.log  access_log.2024-04-25.log  access_log.2024-06-14.log  access_log.2024-08-03.log  access_log.2024-09-22.log
access_log.2023-11-28.log  access_log.2024-01-17.log  access_log.2024-03-07.log  access_log.2024-04-26.log  access_log.2024-06-15.log  access_log.2024-08-04.log  access_log.2024-09-23.log
access_log.2023-11-29.log  access_log.2024-01-18.log  access_log.2024-03-08.log  access_log.2024-04-27.log  access_log.2024-06-16.log  access_log.2024-08-05.log  access_log.2024-09-24.log
access_log.2023-11-30.log  access_log.2024-01-19.log  access_log.2024-03-09.log  access_log.2024-04-28.log  access_log.2024-06-17.log  access_log.2024-08-06.log  access_log.2024-09-25.log
access_log.2023-12-01.log  access_log.2024-01-20.log  access_log.2024-03-10.log  access_log.2024-04-29.log  access_log.2024-06-18.log  access_log.2024-08-07.log  access_log.2024-09-26.log
access_log.2023-12-02.log  access_log.2024-01-21.log  access_log.2024-03-11.log  access_log.2024-04-30.log  access_log.2024-06-19.log  access_log.2024-08-08.log  access_log.2024-09-27.log
access_log.2023-12-03.log  access_log.2024-01-22.log  access_log.2024-03-12.log  access_log.2024-05-01.log  access_log.2024-06-20.log  access_log.2024-08-09.log  access_log.2024-09-28.log
access_log.2023-12-04.log  access_log.2024-01-23.log  access_log.2024-03-13.log  access_log.2024-05-02.log  access_log.2024-06-21.log  access_log.2024-08-10.log  access_log.2024-09-29.log
access_log.2023-12-05.log  access_log.2024-01-24.log  access_log.2024-03-14.log  access_log.2024-05-03.log  access_log.2024-06-22.log  access_log.2024-08-11.log  access_log.2024-09-30.log
access_log.2023-12-06.log  access_log.2024-01-25.log  access_log.2024-03-15.log  access_log.2024-05-04.log  access_log.2024-06-23.log  access_log.2024-08-12.log  access_log.2024-10-01.log
access_log.2023-12-07.log  access_log.2024-01-26.log  access_log.2024-03-16.log  access_log.2024-05-05.log  access_log.2024-06-24.log  access_log.2024-08-13.log  access_log.2024-10-02.log
access_log.2023-12-08.log  access_log.2024-01-27.log  access_log.2024-03-17.log  access_log.2024-05-06.log  access_log.2024-06-25.log  access_log.2024-08-14.log  access_log.2024-10-03.log
access_log.2023-12-09.log  access_log.2024-01-28.log  access_log.2024-03-18.log  access_log.2024-05-07.log  access_log.2024-06-26.log  access_log.2024-08-15.log  access_log.2024-10-04.log
access_log.2023-12-10.log  access_log.2024-01-29.log  access_log.2024-03-19.log  access_log.2024-05-08.log  access_log.2024-06-27.log  access_log.2024-08-16.log  access_log.2024-10-05.log
access_log.2023-12-11.log  access_log.2024-01-30.log  access_log.2024-03-20.log  access_log.2024-05-09.log  access_log.2024-06-28.log  access_log.2024-08-17.log  access_log.2024-10-06.log
access_log.2023-12-12.log  access_log.2024-01-31.log  access_log.2024-03-21.log  access_log.2024-05-10.log  access_log.2024-06-29.log  access_log.2024-08-18.log  access_log.2024-10-07.log
access_log.2023-12-13.log  access_log.2024-02-01.log  access_log.2024-03-22.log  access_log.2024-05-11.log  access_log.2024-06-30.log  access_log.2024-08-19.log  access_log.2024-10-08.log
access_log.2023-12-14.log  access_log.2024-02-02.log  access_log.2024-03-23.log  access_log.2024-05-12.log  access_log.2024-07-01.log  access_log.2024-08-20.log  access_log.2024-10-09.log
access_log.2023-12-15.log  access_log.2024-02-03.log  access_log.2024-03-24.log  access_log.2024-05-13.log  access_log.2024-07-02.log  access_log.2024-08-21.log  access_log.2024-10-10.log
access_log.2023-12-16.log  access_log.2024-02-04.log  access_log.2024-03-25.log  access_log.2024-05-14.log  access_log.2024-07-03.log  access_log.2024-08-22.log  access_log.2024-10-11.log
access_log.2023-12-17.log  access_log.2024-02-05.log  access_log.2024-03-26.log  access_log.2024-05-15.log  access_log.2024-07-04.log  access_log.2024-08-23.log  access_log.2024-10-12.log
access_log.2023-12-18.log  access_log.2024-02-06.log  access_log.2024-03-27.log  access_log.2024-05-16.log  access_log.2024-07-05.log  access_log.2024-08-24.log  access_log.2024-10-13.log
access_log.2023-12-19.log  access_log.2024-02-07.log  access_log.2024-03-28.log  access_log.2024-05-17.log  access_log.2024-07-06.log  access_log.2024-08-25.log  access_log.2024-10-14.log
access_log.2023-12-20.log  access_log.2024-02-08.log  access_log.2024-03-29.log  access_log.2024-05-18.log  access_log.2024-07-07.log  access_log.2024-08-26.log  access_log.2024-10-15.log
access_log.2023-12-21.log  access_log.2024-02-09.log  access_log.2024-03-30.log  access_log.2024-05-19.log  access_log.2024-07-08.log  access_log.2024-08-27.log  access_log.2024-10-16.log
access_log.2023-12-22.log  access_log.2024-02-10.log  access_log.2024-03-31.log  access_log.2024-05-20.log  access_log.2024-07-09.log  access_log.2024-08-28.log  access_log.2024-10-17.log
access_log.2023-12-23.log  access_log.2024-02-11.log  access_log.2024-04-01.log  access_log.2024-05-21.log  access_log.2024-07-10.log  access_log.2024-08-29.log  access_log.2024-10-18.log
access_log.2023-12-24.log  access_log.2024-02-12.log  access_log.2024-04-02.log  access_log.2024-05-22.log  access_log.2024-07-11.log  access_log.2024-08-30.log  access_log.2024-10-19.log
access_log.2023-12-25.log  access_log.2024-02-13.log  access_log.2024-04-03.log  access_log.2024-05-23.log  access_log.2024-07-12.log  access_log.2024-08-31.log  access_log.2024-10-20.log
access_log.2023-12-26.log  access_log.2024-02-14.log  access_log.2024-04-04.log  access_log.2024-05-24.log  access_log.2024-07-13.log  access_log.2024-09-01.log  access_log.2024-10-21.log
access_log.2023-12-27.log  access_log.2024-02-15.log  access_log.2024-04-05.log  access_log.2024-05-25.log  access_log.2024-07-14.log  access_log.2024-09-02.log  access_log.2024-10-22.log
access_log.2023-12-28.log  access_log.2024-02-16.log  access_log.2024-04-06.log  access_log.2024-05-26.log  access_log.2024-07-15.log  access_log.2024-09-03.log  access_log.2024-10-23.log
access_log.2023-12-29.log  access_log.2024-02-17.log  access_log.2024-04-07.log  access_log.2024-05-27.log  access_log.2024-07-16.log  access_log.2024-09-04.log  access_log.2024-10-24.log
access_log.2023-12-30.log  access_log.2024-02-18.log  access_log.2024-04-08.log  access_log.2024-05-28.log  access_log.2024-07-17.log  access_log.2024-09-05.log  access_log.2024-10-25.log
access_log.2023-12-31.log  access_log.2024-02-19.log  access_log.2024-04-09.log  access_log.2024-05-29.log  access_log.2024-07-18.log  access_log.2024-09-06.log  access_log.2024-10-26.log
access_log.2024-01-01.log  access_log.2024-02-20.log  access_log.2024-04-10.log  access_log.2024-05-30.log  access_log.2024-07-19.log  access_log.2024-09-07.log  access_log.2024-10-27.log
access_log.2024-01-02.log  access_log.2024-02-21.log  access_log.2024-04-11.log  access_log.2024-05-31.log  access_log.2024-07-20.log  access_log.2024-09-08.log  access_log.2024-10-28.log
access_log.2024-01-03.log  access_log.2024-02-22.log  access_log.2024-04-12.log  access_log.2024-06-01.log  access_log.2024-07-21.log  access_log.2024-09-09.log  access_log.2024-10-29.log
After: 
ubuntu@ip-10-20-1-175:/var/log/avni_server/log$ ls
access_log.2024-07-31.log  access_log.2024-08-13.log  access_log.2024-08-26.log  access_log.2024-09-08.log  access_log.2024-09-21.log  access_log.2024-10-04.log  access_log.2024-10-17.log
access_log.2024-08-01.log  access_log.2024-08-14.log  access_log.2024-08-27.log  access_log.2024-09-09.log  access_log.2024-09-22.log  access_log.2024-10-05.log  access_log.2024-10-18.log
access_log.2024-08-02.log  access_log.2024-08-15.log  access_log.2024-08-28.log  access_log.2024-09-10.log  access_log.2024-09-23.log  access_log.2024-10-06.log  access_log.2024-10-19.log
access_log.2024-08-03.log  access_log.2024-08-16.log  access_log.2024-08-29.log  access_log.2024-09-11.log  access_log.2024-09-24.log  access_log.2024-10-07.log  access_log.2024-10-20.log
access_log.2024-08-04.log  access_log.2024-08-17.log  access_log.2024-08-30.log  access_log.2024-09-12.log  access_log.2024-09-25.log  access_log.2024-10-08.log  access_log.2024-10-21.log
access_log.2024-08-05.log  access_log.2024-08-18.log  access_log.2024-08-31.log  access_log.2024-09-13.log  access_log.2024-09-26.log  access_log.2024-10-09.log  access_log.2024-10-22.log
access_log.2024-08-06.log  access_log.2024-08-19.log  access_log.2024-09-01.log  access_log.2024-09-14.log  access_log.2024-09-27.log  access_log.2024-10-10.log  access_log.2024-10-23.log
access_log.2024-08-07.log  access_log.2024-08-20.log  access_log.2024-09-02.log  access_log.2024-09-15.log  access_log.2024-09-28.log  access_log.2024-10-11.log  access_log.2024-10-24.log
access_log.2024-08-08.log  access_log.2024-08-21.log  access_log.2024-09-03.log  access_log.2024-09-16.log  access_log.2024-09-29.log  access_log.2024-10-12.log  access_log.2024-10-25.log
access_log.2024-08-09.log  access_log.2024-08-22.log  access_log.2024-09-04.log  access_log.2024-09-17.log  access_log.2024-09-30.log  access_log.2024-10-13.log  access_log.2024-10-26.log
access_log.2024-08-10.log  access_log.2024-08-23.log  access_log.2024-09-05.log  access_log.2024-09-18.log  access_log.2024-10-01.log  access_log.2024-10-14.log  access_log.2024-10-27.log
access_log.2024-08-11.log  access_log.2024-08-24.log  access_log.2024-09-06.log  access_log.2024-09-19.log  access_log.2024-10-02.log  access_log.2024-10-15.log  access_log.2024-10-28.log
access_log.2024-08-12.log  access_log.2024-08-25.log  access_log.2024-09-07.log  access_log.2024-09-20.log  access_log.2024-10-03.log  access_log.2024-10-16.log  access_log.2024-10-29.log

@himeshr himeshr moved this from Code Review Ready to In Code Review in Avni Product Oct 30, 2024
@himeshr himeshr moved this from In Code Review to QA Ready in Avni Product Oct 30, 2024
@himeshr
Copy link
Contributor

himeshr commented Oct 30, 2024

Marking it as Tech type QA, to ensure we check the server state before the deploy and validate all the fixes after deployment.

Test cases

  • Access logs retention period of 90 days (existing clean up and new logs roll-over)
  • EhCache DefaultSizeOfEngine logging only at Error level

@mahalakshme mahalakshme mentioned this issue Nov 4, 2024
Closed
@himeshr himeshr moved this from In QA to Done in Avni Product Nov 12, 2024
@himeshr
Copy link
Contributor

himeshr commented Nov 12, 2024

This forwarding has been temporarily disabled in avni cloud staging, prerelease and prod.
set ForwardToSyslog=no in /etc/systemd/journald.conf
sudo systemctl restart systemd-journald

@1t5j0y Should we enable "ForwardToSyslog" on deployment of 10.1 to prod server.?

@himeshr himeshr moved this from Done to Further Action Required in Avni Product Nov 12, 2024
@1t5j0y 1t5j0y mentioned this issue Nov 12, 2024
Open
@1t5j0y
Copy link
Contributor

1t5j0y commented Nov 12, 2024

This forwarding has been temporarily disabled in avni cloud staging, prerelease and prod.
set ForwardToSyslog=no in /etc/systemd/journald.conf
sudo systemctl restart systemd-journald

@1t5j0y Should we enable "ForwardToSyslog" on deployment of 10.1 to prod server.?

Yes. Updated avniproject/avni-product#1657

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@1t5j0y 1t5j0y

Labels
None yet
Projects
Avni Product
Archived in project
Milestone
No milestone
Development

No branches or pull requests
4 participants
@vinayvenu @mahalakshme @1t5j0y @himeshr