Merge Engaged and Stored alerts

[cardigliano]

Currently ntopng visualize alerts in different tabs according to the status (engaged or stored). It is requested to merge them in a single view, which should allow also sorting and filtering them. In order to achieve this, it is required to store engaged alerts in an in-memory database table, and create a view (union) which includes both in-memory and stored records.

Current implementation details:

• Database schema is defined in the files below:
  httpdocs/misc/alert_store_schema.sql (SQLite)
  httpdocs/misc/db_schema_clickhouse.sql (CH)
  httpdocs/misc/db_schema_clickhouse_cluster.sql (CH Cluster)

• NetworkInterface allocates AlertStore (SQLiteAlertStore, or ClickHouseAlertStore when enabled) into alertStore (class AlertStore) for storing alerts. SQLiteAlertStore opens the database on the provided file path, while ClickHouseAlertStore executes queries (e.g. in ClickHouseAlertStore::query()) through the db (class DB) instance in NetworkInterface (e.g. iface->exec_sql_query()), where db is an instance of ClickHouseFlowDB.

• Alerts (one-shot or engaged when they are released) are currently stored to the persistent database from Lua by calling interface.alert_store_query() mapped to C++ which is calling alertStore->query()

• Engaged (e.g. Host) alerts are triggered in Host::triggerAlert and released in Host::releaseAlert, those methods use addEngagedAlert(alert) and removeEngagedAlert(alert) to add and remove them from the active alerts in memory.

It is requested to:

• Extend SQLiteAlertStore to create an in-memory database for storing engaged alerts
• Extend ClickHouseAlertStore/ClickHouseFlowDB to create an in-memory database for storing engaged alerts
• Add alerts to the in-memory table when trigger is called, remove them from the in-memory table when release is called. This can be done in alert_store_db.lua / host_alert_store:insert based on the alert status (see alert.action)

Nice to have:

• It would be nice to be able to filter on the alert status (engaged, acknowledge, etc)
• For consistency, implement the same for all other alert families (exception made for flow alerts)
• Consider removing the Engaged tab

Open issues:

• Disable actions related to the historical alerts from the Alerts Explorer when the alert is engaged (e.g. the "Remove" action)
• The chart in the 'all' tab does not show alerts engaged before the visualized time interval (and still engage)
• The table in the 'all' tab only shows engaged alerts for the last 24h to avoid a full db scan (only begin time is indexed)
• The 'Require Attention' tab should also include engaged alerts (that require attention)
• Engaged alerts in the 'Require Attention' should handle the "Acknowledge" action as a "Release and Acknowledge", or to simplify just disable "Acknowledge" or add a "Release" action.

[cardigliano]

SQLite in-memory schema implemented in b48c0e0

[cardigliano]

Add/remove engaged host alerts from the in-memory table implemented in b235b24 64028e3
Currently this is working in SQLite only as the CH schema has not yet been implemented

[cardigliano]

CH in-memory schema implemented in 7dedfa1 009288b

[cardigliano]

Temporary added constraint in 6bde300 to avoid a full db scan when visualizing engaged alerts. Showing alerts engaged up to 1 day before.

[cardigliano]

Implemented aggregated views for all alert families f3963c9 2da5925 6f9d069 12407d6 f412a7d 173e0aa 1257879 2aa46f9