jupyter/Notebooks/blogs/playbooks-scirius/JupyterPlaybooksForScirius.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "856b0e94-09bd-41c9-ac0f-604889d9dfc2",
   "metadata": {},
   "source": [
    "# Jupyter Playbooks for Scirius\n",
    "\n",
    "Author: Markus Kont"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0449234c-7004-4829-a895-8893a27060ea",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "Back in 2022, I did a Suricon presentation titled [Jupyter Playbooks for Suricata](https://youtu.be/hevTFubjlDQ). [This led into a blog series](https://www.stamus-networks.com/jupyter-playbooks-for-suricata), [also available as a notebook](https://github.com/StamusNetworks/suricata-analytics/blob/main/jupyter/Notebooks/JupyterPlaybooksForSuricata.ipynb), meant to expand on the topic and provide more context to those interested in notebooks who lack experience to get started.\n",
    "\n",
    "This post can be thought of as continuation of that blog series, though taking on a slightly new direction. Whereas the original Jupyter series was focused on core [Suricata](https://github.com/OISF/suricata), then now we look toward [Scirius](https://github.com/StamusNetworks/scirius). Note that I do not mean [SELKS](https://www.stamus-networks.com/selks), our open-source IDS/NSM threat hunting system, nor [Stamus Security Platform](https://www.stamus-networks.com/stamus-security-platform), our commercial appliance. Instead, the focus is on a software component that is shared between the two solutions. Our goal is to extend the functionalities of these products beyond what is already implemented in the user interface. Scirius, which is the Django web application that implements our management and backend functionalities, was chosen for this task. \n",
    "\n",
    "Interfacing our Jupyter data connectors to open-source Scirius means we can contribute to the Suricata community while also enhancing the product to our customers. Suricata is still seen as nothing more than rule-based IDS engine, whereas in reality it produces a ton of useful NSM events. Our goal is to expose this data to users in meaningful manner. And Jupyter notebooks are the tool we chose for this task."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90ae5587-e4a1-4e54-b304-4afe51829d17",
   "metadata": {},
   "source": [
    "# The problem\n",
    "\n",
    "So far we've talked about processing simple EVE log files with pandas and Jupyter notebooks. However, this approach does not scale. Pandas is designed to be simple to use, not for ingesting and transforming vast amount of data. All processing is done in-memory, and it's not conservative in using it. On the other hand, Suricata can produce vast amounts of NSM events. Does it mean pandas is not fit for processing Suricata EVE JSON logs at scale? No, not at all!\n",
    "\n",
    "Pandas is an amazing tool for interacting with data and for gaining quick insights. Problem is filtering and transforming large datasets. Scirius is already able to do the former by relaying queries to backend Elasticsearch database. Our commercial Stamus Security Platform also has a powerful streaming pipeline to enhance core Suricata events, which addresses the data preparation. But even without it, the core Suricata EVE logs have so much on offer that most user never dig into."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44ff27de-3da0-4f0f-8b7e-6a6acb34db3a",
   "metadata": {},
   "source": [
    "# The REST API\n",
    "\n",
    "A little known feature in our products is the ability to query the REST API. REST, which stands for REpresentational State Transfer, is a standard paradigm for building web applications whereby backend server is responsible for frontend components via API requests. In our case, most frontend components simply fetch and display data from backend URLs. Important part being that we have already implemented a number of useful API endpoints to fetch useful data. It's also fairly simple to add new endpoints.\n",
    "\n",
    "But before we can discuss newly added endpoints or even how anyone could contribute to adding them, we must first explore how API queries work. In short, anyone with proper *API token* is able to issue authenticated requests to endpoints. To generate that token, we must first navigate to `Account Settings` section which is available at the top right corner of the title menu.\n",
    "\n",
    "![Account Settings](account-settings.png)\n",
    "\n",
    "Then on the left hand side, choose `Edit Token`.\n",
    "\n",
    "![Edit Token](img/edit-token.png)\n",
    "\n",
    "Finally, the token will be visible in the `Token` field. If empty, then simply click `Regenerate` button to create a new one. Then copy the value to a keychain or password safe of your choice.\n",
    "\n",
    "![Generate Token](generate-token.png)\n",
    "\n",
    "Once we have found our token, we can start issuing queries to Scirius REST API. We can even fetch data from the command line! Simply point you web client to the appliance IP or fully qualified domain name with API endpoint in the *URI path*. API token must be defined within the `Authorization` header.\n",
    "\n",
    "```bash\n",
    "curl -XGET \"https://$SELKS_OR_SSP/rest/rules/es/alerts_count/\" \\\n",
    "    -H \"Authorization: Token $TOKEN\" \\\n",
    "    -H 'Content-Type: application/json'\n",
    "```\n",
    "\n",
    "This very simple endpoint returns the number of alerts that match within given time period. If left undefined, it will default to 30 days in the past to now.\n",
    "\n",
    "```json\n",
    "{\n",
    "  \"prev_doc_count\": 0,\n",
    "  \"doc_count\": 810605\n",
    "}\n",
    "```\n",
    "\n",
    "We can pull data directly from any SELKS or SSP instance. Directly from command line. That's pretty cool! But let's look at something more powerful."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8a808bb-ea4a-44a7-b748-4ad2ad4b9ccb",
   "metadata": {},
   "source": [
    "## Suricata Analytics project\n",
    "\n",
    "The most difficult aspect of working with notebooks is data ingestion. Most notebooks become unusable over time since they depend on CSV or JSON files for input. Even worse, those files might be preprocessed and notebook assumes existence of some fields that are not actually present in raw data. Jupyter notebooks are often used as references when working on new notebooks simply because they cannot be used without being shipped with exact data they were originally developed with. This clearly diminishes their usefulness. By using Scirius as our data ingestion point, we're able to mitigate that problem. We can make *assumptions* about what data is present and how it's formatted without shipping it with notebooks.\n",
    "\n",
    "This was one of the critical factors that motivated us to start the [Suricata Analytics project](https://github.com/StamusNetworks/suricata-analytics). If the REST API is the *server* component, then Suricata Analytics notebooks are the clients. Those notebooks use Python to interact with Scirius REST API. Next section will explain how it works.\n",
    "\n",
    "## Scirius REST API with Python\n",
    "\n",
    "Firstly, we need to point our notebooks to the right host. We also need to store the authentication token along with any parameters that might alter the connection. After all, hard coding variables like this into each notebook will severely diminish their usability. And to make matters worse, committing and pushing API tokens is a security breach. To keep things simple, we decided to use `.env` files. In fact, our SELKS on docker setup uses the same method, so it was only natural to use it for notebooks as well. It can be set up as described in [Suricata Analytics main README file](https://github.com/StamusNetworks/suricata-analytics/tree/main#jupyter).\n",
    "\n",
    "```bash\n",
    "SCIRIUS_TOKEN=<TOKEN VALUE>\n",
    "SCIRIUS_HOST=<IP or Hostname>\n",
    "SCIRIUS_TLS_VERIFY=yes\n",
    "```\n",
    "\n",
    "For now we handle a very limited set of options. Those being the token value itself, server IP or hostname, and an option to disable TLS verification if using self-signed certificates. Latter being the default for most lab setups and out of the box SELKS installations.\n",
    "\n",
    "Python has`dotenv` package to import variables in this file into python session. Once imported, `dotenv_values` allows us to use variables in environment file like any other python dictionary. Note that Suricata Analytics project includes a reference docker container which mounts the environment file from project root directory into the home folder of container. Subsequent example is written with this in mind."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62267f6c-c6bd-4bde-836c-3e4cddcd04b8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from dotenv import dotenv_values\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd404db4-4d64-405a-9461-b33b558aeedd",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "CONFIG = dotenv_values(os.path.join(os.path.expanduser(\"~\"), \".env\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f951961-cbfc-4a71-86b2-3aa6464280a5",
   "metadata": {},
   "source": [
    "We can use Python `requests` package to interact with Scirius REST API. But before we do, we need to set up some parameters. Like before, the API token is passed with `Authorization` header. Though this time it's more structured. We can also use the environment dictionary to dynamically build the URL and authentication."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5783b96f-f3a2-4a8d-bffc-7a3e2214a7b2",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "URL = \"https://{host}/rest/rules/es/events_tail\".format(host=CONFIG[\"SCIRIUS_HOST\"])\n",
    "HEADERS = {\n",
    "    \"Authorization\": \"Token {token}\".format(token=CONFIG[\"SCIRIUS_TOKEN\"])\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1978950e-19b8-4c30-87d3-881b8e4d8f8e",
   "metadata": {},
   "source": [
    "Each API endpoint usually defines it's own parameters. But some are common for most. The important ones being:\n",
    "* `qfilter` for passing a KQL style query to the endpoint;\n",
    "* `from_date` unix epoch to define point in time from which we want to retrieve the events;\n",
    "* `to_date` unix epoch to define point in time to which the data should be retrieved;\n",
    "* `page_size` how many documents should be fetched;\n",
    "\n",
    "Note that we can pass any Kibana style query to the endpoint using the `qfilter` parameter. Essentially allowing us to fetch any data we want. We can also modify the query the query period. The default is to fetch data from last 30 days. This is something to be careful with since many queries might match more documents than what's returned by Elasticsearch. A wide query over past 30 days with default page size would return a tiny sample of overall data, and would thus not be very useful.\n",
    "\n",
    "Ideally, we would need to fetch something specific. For example, we might be interested in `http` events where HTTP URI contains a command injection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff869493-e208-4cb5-8e51-467a416c64f8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "GET_PARAMS = {\n",
    "    \"qfilter\": \"event_type: http AND http.url: *wget*\",\n",
    "    \"page_size\": 100\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4471a981-9a9b-4239-bf73-12a1e2b82c96",
   "metadata": {},
   "source": [
    "Most data can simply be fetched with HTTP GET requests. A very powerful API endpoint to get started with is `events_tail` which allows the user to query raw EVE events."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "688d4365-6e2c-4170-8fb5-01885de7d6cf",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import urllib3\n",
    "urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b753bb9-edcb-4b64-9fd7-3f4d50794b23",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "resp = requests.get(URL,\n",
    "                    headers=HEADERS,\n",
    "                    verify=False if CONFIG[\"SCIRIUS_TLS_VERIFY\"] == \"no\" else True,\n",
    "                    params=GET_PARAMS)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6278eb2-b4c8-4230-9175-520cf1e21e2d",
   "metadata": {},
   "source": [
    "Once the data is retrieved, we can simply load the values from `results` JSON key and pass them to Pandas `json_normalize` helper to build a flat dataframe of EVE events. Once done, we can interact with the data as described in previous posts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84277dce-726f-40e6-a0ec-b5d028c276dc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e3085fb-9b9a-4735-afb9-2003252f01d7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "DF = pd.json_normalize(json.loads(resp.text)[\"results\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30953681-8927-493e-b9bd-25704a55543c",
   "metadata": {},
   "source": [
    "We can simply measure how many events were fetched."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b98f9b4a-b111-4200-ac57-56c32dc7b927",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "len(DF)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c9b88fb-7463-4954-b603-29f6b17c28c6",
   "metadata": {},
   "source": [
    "Or we could subset the data frame for a quick glance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b41f3b1e-8024-4bf5-87e8-64f8e697ce26",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(\n",
    "    DF\n",
    "    [[\"timestamp\", \"src_ip\", \"dest_ip\", \"event_type\", \"flow_id\", \"http.hostname\"]]\n",
    "    .head(3)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73c70fc9-90ff-40cc-9565-21ef77f69b1f",
   "metadata": {},
   "source": [
    "Naturally, a more useful interaction would be some kind of aggregate report. For example, we could see what URL-s were accessed, what user agents were used for individual HTTP hosts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22dc3c83-b4d9-42fe-9eb4-3b5229a9320f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "DF.groupby([\"http.hostname\"]).agg({\"src_ip\": \"nunique\",\n",
    "                                   \"dest_ip\": \"nunique\",\n",
    "                                   \"http.hostname\": \"unique\",\n",
    "                                   \"http.url\": \"unique\",\n",
    "                                   \"http.http_user_agent\": \"unique\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da6098e3-1e5a-4ac4-a44f-bfeec43fd6f8",
   "metadata": {},
   "source": [
    "This is really powerful but involves some some boilerplate. In the next section we'll see how Suricata Analytics improves on this."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07b63132-cfc3-47bf-95b5-a6750e566e1d",
   "metadata": {},
   "source": [
    "## Suricata Analytics data connector\n",
    "\n",
    "Boilerplate refers to code that repeats in many parts of the code with little variation. But it must be there to set up some other functionality. In our case, user would need to import the API token and Scirius server address in every notebook using `dotenv`. If we ever changed how they are stored, then every notebook would break. Secondly, we would need to import requests and set up HTTP query parameters all the time.\n",
    "\n",
    "Notebooks can become really complex. Especially when weighed down with code that's actually not relevant for exploring data. Having discarded many notebooks for that reason, we decided to write a Python *data connector* to move this complexity from notebooks to importable library. This connector is also part of the Suricata Analytics project and can simply be installed with `pip install .` while in the project root directory. This idea was very much inspired by [MSTIC Jupyter and Python Security Tools](https://msticpy.readthedocs.io/en/latest/), developed by [Microsoft Threat Intelligence team (MSTIC)](https://www.microsoft.com/en-us/security/blog/topic/threat-intelligence/?sort-by=newest-oldest&date=any). Like our project, it provides data connectors to quickly import and analyze security data into Jupyter Notebooks.\n",
    "\n",
    "Once installed, the connector can be imported into any notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba986b7f-3f5e-4d6c-8ccf-a94c7b3e4deb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from surianalytics.connectors import RESTSciriusConnector"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53e54e1a-8deb-4dad-87dc-e8935fe268c8",
   "metadata": {},
   "source": [
    "Then we create new connector object. Environment file is automatically detected on object initialization, though the user can override the parameters with object arguments as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cc4a33b-544b-4684-a82f-eda6e2f8aa71",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "CONNECTOR = RESTSciriusConnector()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1c3d674-faf9-4a63-a5bb-c99d196b72fa",
   "metadata": {},
   "source": [
    "The object maintains persistent state so the user only needs to set certain parameters once. Page size parameter is one that could be easily overlooked. User might execute one query with modified page size yet forget to pass that argument in the next. That could skew the results since the second data fetch might be partial, due to more documents matching the query than would be returned by Elasticsearch.\n",
    "\n",
    "The object allows user to simply set the parameter once. All subsequent queries would then use the value until it's once again updated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90d82f9b-adb5-49fa-911a-6f67e2c51890",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "CONNECTOR.set_page_size(100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd2a34fe-f5dc-42dd-9e6c-4f6d9c3afceb",
   "metadata": {},
   "source": [
    "Same is true for defining the query time period. Relative time queries are very common when working with NSM data. Most users simply need to know what happened X amount of time ago in the past, and might not really care for setting exact timestamps.\n",
    "\n",
    "We provided a helper method that handles this calculation automatically. Likewise, the time frame will apply to all subsequent queries once set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dca66b46-4a21-49ab-877b-e1e1c5c22b48",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "CONNECTOR.set_query_delta(hours=1, minutes=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a24b5a2b-9c05-49fc-b92e-c67b078b92b8",
   "metadata": {},
   "source": [
    "Naturally, the user could also explicitly set from and to timestamps as RFC3339 formatted strings, a unix Epochs, or parsed Python timestamp objects. Our library handles basic validation such as ensuring that timestamps are not in reverse. \n",
    "\n",
    "These are just some of the ways how we can easily prepare the following method call. That call would then be functionally identical to `requests` example that was shown in prior section, albeit with less lines of code. We also do not need to worry about parsing the results. Our library automatically converts the resulting JSON into a normalized pandas data frame, further reducing redundant code. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4427698-8584-4d3c-896f-369be14d617a",
   "metadata": {},
   "outputs": [],
   "source": [
    "DF = CONNECTOR.get_events_df(qfilter=\"event_type: http AND http.url: *wget*\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04e94f17-ee42-45ad-974f-6d286cd4a2fd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(\n",
    "    DF\n",
    "    [[\"timestamp\", \"src_ip\", \"dest_ip\", \"event_type\", \"flow_id\", \"http.hostname\"]]\n",
    "    .head(3)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72d06c92-d741-438f-95e0-d6e298ced1f0",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}