0.1.0 Project Architecture

A Brief Overview of the Universe... The GREASE Universe

Why Python

We chose Python because most of GRE has had exposure to Python, and its fairly simple to onboard new developers. Additionally with careful programming Python is easily cross-platform and containerized. Also deploying GREASE as a group of Python PIP Modules ensures scripted installation is easy and dependency management is done for us via PIP.

Why Python2.7

Python2.7 is still the default shipped with most mainstream Linux distributions, and is widely available on a range of platforms. There are well established tool chains for Python2 and build pipelines still broadly suport it. The maintainers of GREASE really, REALLY want to be Python3 ready/compliant when the time arises to use Python3.

The Overall Concept / Goal

GREASE is designed as a system to enable generalized and large scale automation efforts. Rather than approaching automation via manual scripts, scheduled runs in a task runner system, GREASE is designed to be completely hands free. This enables true and actual automation and reduction of toil. GREASE is designed to stop the decades old mentality of operations and operators from break/fix work, and begin having time to interface with their solutions portfolios more effectively, empower guests with more reliable systems to service them, and enable red shirts to stay on the floor. A common simile for GREASE is: "Let GRE and Operations STOP fighting all these fires (automated preventable incidents), install the sprinkler system (GREASE) and begin performing arson investigation (helping solutions portfolio working on & prioritizing their backlog/Designing better processes and infrastructure for stores)

The Basic Idea behind the Beast

GREASE is by default built to be a distributed system. This means many moving pieces, all with their own small differences. It is always a highly sought goal with the maintainers of GREASE to maintain as much consistancy as possible with the way the pieces of GREASE work.

Process 1: Sourcing (Data Ingestion)

GREASE relies on external systems to provide information to it. It is not in scope of GREASE to handle things like watching a file or aggregating logs, this is what tools like NxLog and ELK Stack are excellent at. Instead GREASE can read from/recieve data from these systems and process it for possible jobs to run.

NOTE: while GREASE is not made to be a file watcher or log aggregator, its not out of scope to build a source in GREASE to do such a thing ;)

1. Data is fed into the engine via HTTP, or any other way you can dream up
2. DeDuplication (Check the wiki for how the DeDuplication Engine works) removes data the cluster has already seen
3. Unique Data is then persisted to the central cluster database and scheduled for detection

This is what we call sourcing. We source information from external sources. These Sources are then processed. Each source document in a source is then processed in a system we call detection.

Helpful Iluistration:

Process 2: Detection (Detecting Unfit Environments)

The second tier of data moving through GREASE is the detection system. This is where we apply the logic from user contributed Configuration. If we find a source document that meets the logic of then we will update the source document to let scheduling know a job needs to be executed

1. Detection Server picks up the sources that need to be processed
2. For each source it loops through each document operating each set of logical blocks per config searching for possible matches
3. Updates each source document with the results of step 2
4. Schedules a scheduling server to process detected jobs

This is what we call detection. We detect information from sources and report jobs to be scheduled

Helpful Illustration:

Process 3: Scheduling (Assigning Jobs to Execution Servers)

The third tier of data processing in GREASE is the scheduling system. Yeah I know the other two steps have scheduling too, but this is like, actual scheduling okay? Calm down. This will process sources that Detection has already operated on and are ready for scheduling.

1. Scheduling Server picks up the sources that need to be processed
2. Loop through each source and process documents to schedule load-balanced execution across available servers in each environment

This is scheduling where we schedule jobs for execution

Helpful Illustration:

Process 4: Scheduling (Executing Jobs)

The Final tier of data processing in GREASE is job execution. This is where the engine actually performs automated recoveries, sends emails, alerts people, or really anything you could possibly want a computer to do. Some more outlandish things to can do in an execution environment: build & publish software, email distro groups change logs based on git history, use the dominos pizza api to order the GREASE maintainers pizza (one cheese, one peperoni, and one meat lovers please)

1. Execution Server queries the database for jobs it has been assigned (Only 15 per loop _roughly 120 loops per second based on processor / execution time per-job)
2. Attempt to execute any jobs scheduled to the server from Process 3 * Report results of execution back to the Database
   • Execution Success: There were no execptions raised executing the code
   • Command Success: The command returned a boolean evaluated True value
3. Attempt to execute and persistent jobs * Persistent jobs are roles assigned to the server EX: sourcing, detection, scheduling * These are configured to only run once per minute

This is execution where we execute jobs and report telemetry from their execution as well as execute persistent jobs

Helpful Illustration:

That's the idea of GREASE Check all the pages below here for more information or reach out to the maintainers for anything that needs to be added here/additional questions!