added plugin support, added ssh plugin

CHANGELOG

@@ -4,6 +4,14 @@ All notable changes to this project will be documented in this file.
 ## [unreleased]
 ### Changed
 
+## [1.1.0] - 2019.04.24
+### Changed
+- added plugin support
+- added SSH plugin
+- added shell script atop_parser.sh for gather atop data
+- added scatter charts support for custom charts
+- removed command line arguments support
+
 ## [1.0.1] - 2018.10.09
 ### Changed
 - added date to workbook file name

README.md

@@ -1,141 +1,72 @@
-oracle-awr-report.py is easy-to-use data exporter from Oracle database to xls (Excel)
+oracle-awr-report.py is easy-to-use data exporter from different data sources (Oracle Database, ssh) to xlsx (Excel)
 
 # Key features
-* Export data from Oracle database to xls worksheet
+* Export data from Oracle database to xlsx worksheet
+* Export data by shell script to xlsx worksheet
 * Charts drawing
 * Customizing charts
 * Adding computed columns
 
 AWR SQL queries by [MaksimIvanovPerm](https://github.com/MaksimIvanovPerm)
 
 # Prerequisite
-* python 2.7 or higher
-* cx_Oracle 5.3 or higher
-* xlswriter 1.0.5 or higher
-
-## Tested on
-* python 2.7.12
-  * cx_Oracle 6.4.1
-  * xlsxwriter 1.0.9
-* python 2.7.12
-  * cx_Oracle 5.3
-  * xlswriter 1.1.1
-* python 2.7.15
-  * cx_Oracle 5.3
-  * xlsxwriter 1.0.5
-* python 3.6.5
-  * cx_Oracle 6.3.1 
-  * xlsxwriter 1.0.5
+* python
+* cx_Oracle
+* xlswriter
+* paramiko
+* scp
 
 # Usage
 
-**Minimal run with cli arguments:**
-```
-$ python3 oracle-awr-report.py --host test1.exampledomain.loc --service_name testdb1 --general_conf NONE --username cade --report_conf conf.d/report.conf 
-```
-**Run with main configuration file:**
-```
-$ python3 oracle-awr-report.py
-```
-
-## Command line arguments
-```
-$ python oracle-awr-report.py -h
-usage: oracle-awr-report.py [-h] [--author AUTHOR] [--begin_snap BEGIN_SNAP]
-                            [--company COMPANY] [--db_name DB_NAME]
-                            [--dbid DBID] [--end_snap END_SNAP]
-                            [--general_conf GENERAL_CONF] [--host HOST]
-                            [--logging LOGGING] [--password PASSWORD]
-                            [--port PORT] [--prefix PREFIX]
-                            [--report_conf REPORT_CONF]
-                            [--service_name SERVICE_NAME] [--sid SID]
-                            [--tnsalias TNS_ALIAS] [--username USERNAME]
-
-optional arguments:
-  -h, --help            show this help message and exit
-  --author AUTHOR       Report Author
-  --begin_snap BEGIN_SNAP
-                        For AWR: start snap_id
-  --company COMPANY     Report Company
-  --db_name DB_NAME     Database name, printed in report
-  --dbid DBID           For AWR: database dbid
-  --end_snap END_SNAP   For AWR: end snap_id
-  --general_conf GENERAL_CONF
-                        Path to main configuration file. If set in NONE, don't
-                        use configuration file.
-  --host HOST           Database host address
-  --logging LOGGING     Logging level. Can be one of
-                        DEBUG/INFO/WARNING/ERROR/CRITICAL
-  --password PASSWORD   Database password
-  --port PORT           Database port
-  --prefix PREFIX       Path to report
-  --report_conf REPORT_CONF
-                        Path to report configration file
-  --service_name SERVICE_NAME
-                        Database SERVICE_NAME.
-  --sid SID             Database SID. If not set - use service_name
-  --tnsalias TNS_ALIAS  tnslias for db connection. If not set - use host,
-                        port, sid/service_name
-  --username USERNAME   Database username
+```
+$ python oracle-awr-report.py 
 ```
 
 ## Main configuration **conf.d/general.conf**
 _All config files openes with utf-8 encoding_
 
-| Parameter name | Description                              		    | Mandatory | Not null
-| -------------- | -------------------------------------------------------- | --------- | --------
-| TNS_ALIAS      | alias from tnsnames.ora to db connection 		    | +-        | +
-| HOST           | database server hostname/ipaddr			    | +-        | +
-| PORT		 | database port (default: 1521)                            | -         | +
-| SID            | database SID                                             | +-        | +
-| SERVICE_NAME   | database SERVICE_NAME                                    | +-        | +
-| USERNAME       | database login                            		    | +         | +
-| PASSWORD       | database password                        		    | -         | +
-| DBID		 | database id						    | -         | +
-| BEGIN_SNAP     | awr begin snap_id  			                    | -         | +
-| END_SNAP       | awr end snap_id                 		            | -         | +
-| PREFIX         | path to report file                  		    | -         | -
-| DB_NAME        | database title     			                    | -         | -
-| AUTHOR         | report author                            		    | -         | -
-| COMPANY        | report company            		                    | -         | -
-| LOGGING        | log level, one of: DEBUG, INFO, WARNING, ERROR, CRITICAL | -         | +
-| REPORT_CONF    | report configuration file 				    | +         | +
-
-* When TNS_ALIAS exist, we using it. Otherwise we using HOST, PORT and SID/SERVICE_NAME
-* When SID exist, we using it. Otherwise we using SERVICE_NAME
-* When PASSWORD not present in general.conf, read it from input
-
 ## Report configuration file
 Format:
 ```
 id:sql script:chart title:chart config file:columns config file
+id:plugin:source file:worksheet title:chart title:charts config file:columns config file
 ```
-where chart config file is one of following:
+where charts config file is one of following:
 1. **none** - not print chart
 2. **default** - use all data to print one chart only
 3. **custom charts configuration file name**
 
+where columns config file is one of following:
+1. **default** - not use custom columns
+2. **custom columns configuration file name**
+
 Example:
 ```
-1:RDBMSServiceTime.sql:DB service time structure:default
-2:WaitTimeStructure.sql:DB wait time structure:default
-3:RedoStat.sql::conf.d/redostat.conf
+1:oradb:oradb/RDBMSServiceTime.sql:RDBMSServiceTime:DB service time structure:default:default
+2:oradb:oradb/WaitTimeStructure.sql:WaitTimeStructure:DB wait time structure:default:default
+3:oradb:oradb/RedoStat.sql:RedoStat::conf.d/redostat.conf:default
+4:ssh:ssh/atop_parser.sh --loglevel SILENT2 --swap:atop_swap:atopSWP:default:default
+5:ssh:ssh/atop_parser.sh --loglevel SILENT2 --dsk sdc:atop_sdc::conf.d/atop_dsk_charts.conf:conf.d/atop_dsk_columns.conf
 ```
 
 ## Custom charts configuration file
 Format:
 ```
-# id:column number in char:column title
+# id:column number in char:column title:chart type
 
 ```
 
+where chart type is one of following:
+1. **line**
+2. **scatter**
+
 Example:
 ```
-1:25:Redo writes per hour
-1:27:
-2:34:Redo write latency, ms
-3:27:Redo write structure
-3:33:
+1:25:Redo writes per hour:line
+1:27::
+2:34:Redo write latency, ms:line
+3:27:Redo write structure:line
+3:33::
 ```
 
 ## Custom columns configuration file
@@ -155,3 +86,27 @@ Example:
 
 ## Report worksheet example
 ![DB wait time structure](DOC/oracle-awr-report.png)
+
+# Plugins
+
+## ORADB
+Plugin for gathering data from Oracle Database
+
+## SSH
+Plugin for gathering data by shell script via ssh.
+
+Shell script must output one line with format:
+```
+File_with_data unixtimestamp column2_title column2_title columnt3_title...
+```
+
+Example:
+```
+/tmp/atop_parser_CPU_20190424_2018_24444.log unixtimestamp sys user irq idle wait
+```
+
+So, first column in data file must be unixtimestamp!
+
+# ATOP
+For gathering atop data from host prepared shell script **atop_parser.sh**. It work with atop log files /var/log/atop/atop_* created with interval=600 seconds
+

conf.d/atop_dsk_charts.conf

@@ -0,0 +1,12 @@
+# id:номер столбца в график(отсчет начинается с 0):название графика:graph type
+# supported graph types:
+#    line: Creates a Line style chart.
+#    scatter: Creates a Scatter style chart.
+1:2:dsk device:line
+1:3:dsk device:
+2:4:dsk device:line
+2:7:dsk device:
+3:6:зависимость io-latency от iops:scatter
+3:5::
+4:7:зависимость длины io-очереди от iops:scatter
+4:5::

conf.d/atop_dsk_columns.conf

@@ -0,0 +1,5 @@
+# id столбца:название:формула
+# _ROWID_ - служебное слово для вставки номера текущей строки
+
+6:IOPS:=(B_ROWID_+C_ROWID_)/600
+7:avio*10ms:=E_ROWID_*10

conf.d/general.conf

@@ -1,35 +1,57 @@
-# Oracle tnsalias from tnsnames.ora
-#TNS_ALIAS="testdb"
-HOST="example.com"
-PORT=1521
-#SID="testdb"
-SERVICE_NAME="testdb.example.com"
+# target host
+HOST="oracle-db.example.com"
 
-# Database credentials
-USERNAME="username"
-PASSWORD="password"
+# report store path
+#PATH=""
+# Report (and graphs) title
+TITLE="oracle-db"
 
-# Database id
-#DBID=3235167506
-
-# Minimal awr-snapshot id including in report
-#BEGIN_SNAP=90721
-
-# Maximun awr-snapshot id including in report
-#END_SNAP=91072
-
-# Path to xls file
-#PREFIX=""
-# Database name
-DB_NAME="testdb"
-
-# Report Author
+# Report author
 AUTHOR="Denis Vodopyanov"
-# Report Company
+# Report company
 COMPANY="example.com"
 
-# Logging level: DEBUG|INFO|WARNING|ERROR|CRITICAL
-LOGGING="WARNING"
+# Logging level DEBUG|INFO|WARNING|ERROR|CRITICAL
+LOGGING="INFO"
+#LOGGING="DEBUG"
 
-# Report configuration file
+# Report config file
 REPORT_CONF="conf.d/report.conf"
+
+# ==============================================================
+# ORADDB plugin parameters
+# ==============================================================
+# if TNS_ALIAS exist - it in use, otherwise using HOST, ORADB_PORT, ORADB_SID/ORADB_SERVICE_NAME
+# oracle database tns alias
+#ORADB_TNS_ALIAS="oracle-db"
+# oracle database port, default value 1521
+#ORADB_PORT=1521
+
+# if ORADB_SID exist - it in use, otherwise using ORADB_SERVICE_NAME
+# oracle database sid
+ORADB_SID="example"
+# oracle database service name
+#ORADB_SERVICE_NAME="example.example.com"
+
+# oracle database login
+ORADB_USERNAME="username"
+# oracle database password
+ORADB_PASSWORD="password"
+
+# oracle database id
+#ORADB_DBID=3453375711
+
+# oracle database first awr stapshot
+#ORADB_BEGIN_SNAP=40936
+
+# oracle database last awr snapshot
+#ORADB_END_SNAP=41283
+
+# ==============================================================
+# SSH plugin parameters
+# ==============================================================
+# ssh port default value 22
+#SSH_PORT=22
+SSH_USERNAME="username"
+#SSH_PASSWORD="a1w8eGs"
+SSH_KEY="/home/username/.ssh/id_rsa"

conf.d/ioeventexplanation_charts.conf

@@ -1,5 +1,8 @@
-# id:column number in char:column title
-1:11:major User I/O event explanation
-1:12:major User I/O event explanation
-2:13:random-read latency
-2:14:random-read latency
+# id:номер столбца в график(отсчет начинается с 0):название графика:graph type
+# supported graph types:
+#    line: Creates a Line style chart.
+#    scatter: Creates a Scatter style chart.
+1:11:major User I/O event explanation:line
+1:12:major User I/O event explanation:
+2:13:random-read latency, ms:line
+#2:14:random-read latency

conf.d/ioeventexplanation_columns.conf

@@ -1,5 +1,6 @@
-# column number:column title:formula
-# _ROWID_ - service variable
-11:db_file_sequential_read:=UserIOWaitsTime!W_ROWID_/10000
+# id столбца:название:формула
+# _ROWID_ - служебное слово для вставки номера текущей строки 
+
+11:db_file_sequential_read:=UserIOWaitsStructure!W_ROWID_/10000
 12:userio:=WaitTimeStructure!D_ROWID_/10000
-13:rr-latency:=C_ROWID_/E_ROWID_
+13:rr-latency:=10*C_ROWID_/E_ROWID_

conf.d/redostat.conf

@@ -1,6 +1,9 @@
-# id:column number in char:column title
-1:25:Redo writes per hour
-1:27:
-2:34:Redo write latency, ms
-3:27:Redo write structure
-3:33:
+# id:номер столбца в график(отсчет начинается с 0):название графика:graph type
+# supported graph types:
+#    line: Creates a Line style chart.
+#    scatter: Creates a Scatter style chart.
+1:25:Редо-врайтов/час, временные затраты на редо-записи.:line
+1:27::
+2:34:Латенси по редо записи, мсек;:line
+3:27:Стр-ра редо-врайтов:line
+3:33::