update from bleepblop

cottrell · Sep 12, 2024 · ae62932 · ae62932
1 parent f842171
commit ae62932
Show file tree

Hide file tree

Showing 3 changed files with 41 additions and 6 deletions.
diff --git a/my-gym b/my-gym
diff --git a/mybin b/mybin
diff --git a/system_monitoring/README.md b/system_monitoring/README.md
@@ -1,3 +1,32 @@
+# 2024-09-09
+
+More OOM.
+
+Found PID via
+
+    grep -5 -i "killed process" /var/log/syslog
+
+(or perhaps other means). Then
+
+    atop -r
+
+    <I> <PID> to search for PID
+    <t> <T> forward back until you find it in history
+    <c> to show command line view and hopefully you see something more informative than just "python" or whatever. I think the first entry is the full command.
+
+
+No way to track PID to process (other than python).
+
+Try to enable this for next time:
+
+    sudo apt install auditd
+    sudo auditctl -a exit,always -F arch=b64 -S execve
+
+Then (maybe) something like
+
+    ausearch -k python_exec
+
+
 # 2024-09-06
 
     sudo apt-get install atop
@@ -19,17 +48,23 @@
 
 # 2024
 
+
+Reminders (for post mortem)
+
+    sudo dmesg -T ... this might get you a PID before the OOM
+
+    sar -r   # Shows memory usage over time
+    sar -u   # Shows CPU usage over time
+    sar -S   # Shows swap usage over time
+
+
 * https://tomscii.sig7.se/2022/07/uMon-stupid-simple-monitoring
 * glances
 * systar (sar):
 
     sudo apt install sysstat  # On Ubuntu/Debian
     sudo systemctl enable --now sysstat
 
-    sar -r   # Shows memory usage over time
-    sar -u   # Shows CPU usage over time
-    sar -S   # Shows swap usage over time
-
     sar -r 10 100   # Logs memory usage every 10 seconds for 100 intervals
 
     # post event debug