This repository has been archived by the owner on Jul 22, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 7
/
Copy pathIBM_RB_LNXBasic_ProcessHighCPU.json
50 lines (44 loc) · 17.3 KB
/
IBM_RB_LNXBasic_ProcessHighCPU.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
{
"name": "Handle high CPU usage on Linux",
"description": "This Procedure describes how to handle the situation that a Linux server has to handle high CPU usage.",
"steps": [
{
"number": 1,
"description": "\n<p dir=\"ltr\" style=\"box-sizing: border-box; margin: 0px 0px 10px; color: rgb(34, 34, 34); font-family: Arial, Verdana, sans-serif; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17.142858505249px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;\">Login to <span class=\"span-grp rba-param\" style=\"box-sizing: border-box; color: rgb(255, 255, 255); padding: 4px 2px; white-space: nowrap; border-bottom-style: dotted; border-width: 1px; background: rgb(238, 61, 150);\" contenteditable=\"false\">HOSTNAME</span> using a terminal emulator, e.g. putty.</p>\n\n"
},
{
"number": 2,
"description": "\n<p dir=\"ltr\" style=\"box-sizing: border-box; margin: 0px 0px 10px; color: rgb(34, 34, 34); font-family: Arial, Verdana, sans-serif; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17.142858505249px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;\">Identify the process(es) consuming a high amount of CPU.<br>\nType the following commands to list the top 10 CPU consuming processes every 5 seconds. After each command, press the enter key in your terminal session.<br style=\"box-sizing: border-box;\">\n<br>\n<br>\n<span class=\"span-grp rba-command\" style=\"box-sizing: border-box; color: rgb(51, 51, 51); padding: 6px 3px; font-family: 'Courier New'; white-space: nowrap; border-bottom-style: dashed; border-width: 1px; background: rgb(180, 224, 81);\"><span style=\"box-sizing: border-box; font-family: 'Courier New';\">while true</span></span><br>\n<br>\n<span class=\"span-grp rba-command\" style=\"box-sizing: border-box; color: rgb(51, 51, 51); padding: 6px 3px; font-family: 'Courier New'; white-space: nowrap; border-bottom-style: dashed; border-width: 1px; background: rgb(180, 224, 81);\"><span style=\"box-sizing: border-box; font-family: 'Courier New';\"> do</span> </span></p>\n\n<p dir=\"ltr\" style=\"box-sizing: border-box; margin: 0px 0px 10px; color: rgb(34, 34, 34); font-family: Arial, Verdana, sans-serif; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17.142858505249px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;\"><span class=\"span-grp rba-command\" style=\"box-sizing: border-box; color: rgb(51, 51, 51); padding: 6px 3px; font-family: 'Courier New'; white-space: nowrap; border-bottom-style: dashed; border-width: 1px; background: rgb(180, 224, 81);\"><span style=\"box-sizing: border-box; font-family: 'Courier New';\"> clear</span></span></p>\n\n<p dir=\"ltr\" style=\"box-sizing: border-box; margin: 0px 0px 10px; color: rgb(34, 34, 34); font-family: Arial, Verdana, sans-serif; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17.142858505249px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;\"><span class=\"span-grp rba-command\" style=\"box-sizing: border-box; color: rgb(51, 51, 51); padding: 6px 3px; font-family: 'Courier New'; white-space: nowrap; border-bottom-style: dashed; border-width: 1px; background: rgb(180, 224, 81);\"> ps -eo %cpu,pid,comm | sort -nr | head -n 10</span> </p>\n\n<p dir=\"ltr\" style=\"box-sizing: border-box; margin: 0px 0px 10px; color: rgb(34, 34, 34); font-family: Arial, Verdana, sans-serif; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17.142858505249px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;\"><span class=\"span-grp rba-command\" style=\"box-sizing: border-box; color: rgb(51, 51, 51); padding: 6px 3px; font-family: 'Courier New'; white-space: nowrap; border-bottom-style: dashed; border-width: 1px; background: rgb(180, 224, 81);\"><span style=\"box-sizing: border-box; font-family: 'Courier New';\"> sleep 5</span></span></p>\n\n<p dir=\"ltr\" style=\"box-sizing: border-box; margin: 0px 0px 10px; color: rgb(34, 34, 34); font-family: Arial, Verdana, sans-serif; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17.142858505249px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;\"><span class=\"span-grp rba-command\" style=\"box-sizing: border-box; color: rgb(51, 51, 51); padding: 6px 3px; font-family: 'Courier New'; white-space: nowrap; border-bottom-style: dashed; border-width: 1px; background: rgb(180, 224, 81);\"><span style=\"box-sizing: border-box; font-family: 'Courier New';\">done</span> </span><br style=\"box-sizing: border-box;\">\n<br>\n<br>\n<br>\nIf the result looks like below (%CPU utilization, Process ID, Process Executable):<br style=\"box-sizing: border-box;\">\n<br>\n<br>\n<span style=\"box-sizing: border-box; font-family: 'Courier New';\"><span style=\"box-sizing: border-box; color: rgb(0, 128, 0);\"><strong style=\"box-sizing: border-box; font-weight: bold;\">105 52593 java<br>\n12.6 52576 java<br>\n 8.1 18130 java<br>\n 2.7 25064 main<br>\n 2.1 52476 gnome-system-mo<br>\n 1.4 18009 KfwServices<br>\n 0.7 16971 kdsmain</strong></span></span><br>\n<span style=\"box-sizing: border-box; color: rgb(0, 128, 0);\"><strong style=\"box-sizing: border-box; font-weight: bold;\"><span style=\"box-sizing: border-box; font-family: 'Courier New';\"> 0.5 3701 db2sysc<br>\n 0.4 17911 java<br>\n 0.3 41211 main</span></strong></span><br style=\"box-sizing: border-box;\">\n<br>\n<br>\nhere is a java process running at 105% CPU utilization with the process ID of 52593.</p>\n\n<p dir=\"ltr\" style=\"box-sizing: border-box; margin: 0px 0px 10px; color: rgb(34, 34, 34); font-family: Arial, Verdana, sans-serif; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17.142858505249px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;\">If you find a process like this, go back to your runbook parameters and change the PROCESSID parameter to the number identified - in our example, the value would be 52593.<br style=\"box-sizing: border-box;\">\n<br>\n<br>\nIf there are no processes appearing in this list with more than 95% CPU utilization, cancel this runbook, indicating that you found the problem and no longer need this runbook.</p>\n\n<p dir=\"ltr\" style=\"box-sizing: border-box; margin: 0px 0px 10px; color: rgb(34, 34, 34); font-family: Arial, Verdana, sans-serif; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17.142858505249px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;\"><span class=\"span-grp rba-goto\" data-stepid=\"-1\" contenteditable=\"false\">END</span></p>\n\n"
},
{
"number": 3,
"description": "\n<p dir=\"ltr\">Attempt to stop the identified process(es). In the runbook parameters add now the process ID which is listed to consume the highest CPU.</p>\n\n<p dir=\"ltr\" style=\"box-sizing: border-box; margin: 0px 0px 10px; color: rgb(34, 34, 34); font-family: Arial, Verdana, sans-serif; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17.142858505249px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;\">Run the following command to attempt to gracefully stop the process:<br style=\"box-sizing: border-box;\">\n<br>\n<br>\n<span class=\"span-grp rba-command\" style=\"box-sizing: border-box; color: rgb(51, 51, 51); padding: 6px 3px; font-family: 'Courier New'; white-space: nowrap; border-bottom-style: dashed; border-width: 1px; background: rgb(180, 224, 81);\"><span style=\"box-sizing: border-box; font-family: 'Courier New';\">kill -HUP <span class=\"span-grp rba-param\" style=\"box-sizing: border-box; color: rgb(255, 255, 255); padding: 4px 2px; white-space: nowrap; border-bottom-style: dotted; border-width: 1px; background: rgb(238, 61, 150);\" contenteditable=\"false\">PROCESSID</span></span></span><br style=\"box-sizing: border-box;\">\n<br>\n<br>\nWait for a few seconds and run this command to check that the process has ended accordingly. Repeat this step for the remaining processes identified.<br style=\"box-sizing: border-box;\">\n<br>\n<br>\n<span class=\"span-grp rba-command\" style=\"box-sizing: border-box; color: rgb(51, 51, 51); padding: 6px 3px; font-family: 'Courier New'; white-space: nowrap; border-bottom-style: dashed; border-width: 1px; background: rgb(180, 224, 81);\"><span style=\"box-sizing: border-box; font-family: 'Courier New';\">ps -ef | grep <span class=\"span-grp rba-param\" style=\"box-sizing: border-box; color: rgb(255, 255, 255); padding: 4px 2px; white-space: nowrap; border-bottom-style: dotted; border-width: 1px; background: rgb(238, 61, 150);\" contenteditable=\"false\">PROCESSID</span> | grep -v grep</span></span><br style=\"box-sizing: border-box;\">\n<br>\n<br>\n<br>\nIf the output is empty proceed to the end of the runbook and ignore the next steps. If the process is still running, proceed to the next step.</p>\n\n"
},
{
"number": 4,
"description": "\n<p dir=\"ltr\" style=\"box-sizing: border-box; margin: 0px 0px 10px; color: rgb(34, 34, 34); font-family: Arial, Verdana, sans-serif; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17.142858505249px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;\">Run the following command:</p>\n\n<p dir=\"ltr\" style=\"box-sizing: border-box; margin: 0px 0px 10px; color: rgb(34, 34, 34); font-family: Arial, Verdana, sans-serif; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17.142858505249px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;\"><span class=\"span-grp rba-command\" style=\"box-sizing: border-box; color: rgb(51, 51, 51); padding: 6px 3px; font-family: 'Courier New'; white-space: nowrap; border-bottom-style: dashed; border-width: 1px; background: rgb(180, 224, 81);\">kill -TERM <span class=\"span-grp rba-param\" style=\"box-sizing: border-box; color: rgb(255, 255, 255); padding: 4px 2px; white-space: nowrap; border-bottom-style: dotted; border-width: 1px; background: rgb(238, 61, 150);\" contenteditable=\"false\">PROCESSID</span></span><br style=\"box-sizing: border-box;\">\n<br>\n<br>\nWait for a few seconds and run this command to check that the process has ended accordingly. Repeat this step for the remaining processes identified.<br style=\"box-sizing: border-box;\">\n<br>\n<br>\n<span class=\"span-grp rba-command\" style=\"box-sizing: border-box; color: rgb(51, 51, 51); padding: 6px 3px; font-family: 'Courier New'; white-space: nowrap; border-bottom-style: dashed; border-width: 1px; background: rgb(180, 224, 81);\"><span style=\"box-sizing: border-box; font-family: 'Courier New';\">ps -ef | grep <span class=\"span-grp rba-param\" style=\"box-sizing: border-box; color: rgb(255, 255, 255); padding: 4px 2px; white-space: nowrap; border-bottom-style: dotted; border-width: 1px; background: rgb(238, 61, 150);\" contenteditable=\"false\">PROCESSID</span></span><span style=\"box-sizing: border-box; font-family: 'Courier New';\">| grep -v grep</span></span><br style=\"box-sizing: border-box;\">\n<br>\n<br>\nIf the output is empty you can <span class=\"span-grp rba-goto\" data-stepid=\"-1\" contenteditable=\"false\">END</span> the runbook here. If the process is still running continue with the next step.</p>\n\n"
},
{
"number": 5,
"description": "\n<p dir=\"ltr\" style=\"box-sizing: border-box; margin: 0px 0px 10px; color: rgb(34, 34, 34); font-family: Arial, Verdana, sans-serif; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17.142858505249px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;\">Force the process(es) to stop<br>\n<br>\nIf you have reached this step of the runbook, the process is most likely out of control and need to be stopped by force. Any data held in memory or data not yet written to disk will be lost. Run the following command to force the process to stop:<br>\n<br>\n<br>\n<span class=\"span-grp rba-command\" style=\"box-sizing: border-box; color: rgb(51, 51, 51); padding: 6px 3px; font-family: 'Courier New'; white-space: nowrap; border-bottom-style: dashed; border-width: 1px; background: rgb(180, 224, 81);\"><span style=\"box-sizing: border-box; font-family: 'Courier New';\">kill -KILL <span class=\"span-grp rba-param\" style=\"box-sizing: border-box; color: rgb(255, 255, 255); padding: 4px 2px; white-space: nowrap; border-bottom-style: dotted; border-width: 1px; background: rgb(238, 61, 150);\" contenteditable=\"false\">PROCESSID</span></span></span><br>\n<br>\n<br>\nWait for a few seconds and run this command to establish that the process has ended accordingly. Repeat this step for the remaining processes identified.<br style=\"box-sizing: border-box;\">\n<br>\n<br>\n<span class=\"span-grp rba-command\" style=\"box-sizing: border-box; color: rgb(51, 51, 51); padding: 6px 3px; font-family: 'Courier New'; white-space: nowrap; border-bottom-style: dashed; border-width: 1px; background: rgb(180, 224, 81);\"><span style=\"box-sizing: border-box; font-family: 'Courier New';\">ps -ef | grep <span class=\"span-grp rba-param\" style=\"box-sizing: border-box; color: rgb(255, 255, 255); padding: 4px 2px; white-space: nowrap; border-bottom-style: dotted; border-width: 1px; background: rgb(238, 61, 150);\" contenteditable=\"false\">PROCESSID</span> </span><span style=\"box-sizing: border-box; font-family: 'Courier New';\">| grep -v grep</span></span><br style=\"box-sizing: border-box;\">\n<br>\n<br>\n<br>\nIf the output is empty <span style=\"box-sizing: border-box; line-height: 17.142858505249px;\">proceed to the end of the runbook and ignore the next steps.</span></p>\n\n<p dir=\"ltr\" style=\"box-sizing: border-box; margin: 0px 0px 10px; color: rgb(34, 34, 34); font-family: Arial, Verdana, sans-serif; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17.142858505249px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;\">If the process is still running continue to last / escalation step</p>\n\n"
},
{
"number": 6,
"description": "\n<p dir=\"ltr\" style=\"box-sizing: border-box; margin: 0px 0px 10px; color: rgb(34, 34, 34); font-family: Arial, Verdana, sans-serif; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17.142858505249px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;\">Escalate the issue to the next support level</p>\n\n<p dir=\"ltr\" style=\"box-sizing: border-box; margin: 0px 0px 10px; color: rgb(34, 34, 34); font-family: Arial, Verdana, sans-serif; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17.142858505249px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;\">There are other issues with the system and need attention by an engineer. Escalate to the next support level with the information gathered in the steps above and complete this runbook. </p>\n"
}
],
"parameters": [
{
"name": "HOSTNAME"
},
{
"name": "PROCESSID"
}
],
"tags": [
"usage",
"high",
"CPU",
"Linux"
]
}