Summer 2019 update of analytics

.github/ISSUE_TEMPLATE/analytic.md

@@ -0,0 +1,13 @@
+---
+name: Analytic Request
+about: Suggest a new EQL analytic
+labels: analytic
+
+---
+
+## Analytic Description
+* *What attacker technique are you looking to detect?*
+* *Was there a recent blogpost, Tweet, etc. about the technique?*
+* *Is this a high-fidelity detection, or a query with data stacking for hunting?*
+* *Do you have any thoughts how to structure your query?*
+* *Would you expect this analytic to be noisy? High false positive rates?*

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,22 @@
+---
+name: Bug report
+about: File a bug
+labels: bug
+
+---
+
+### Describe the bug
+A clear and concise description of what the bug is.
+
+#### To Reproduce
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+#### Expected behavior
+A clear and concise description of what you expected to happen.
+
+#### Screenshots
+If applicable, add screenshots to help explain your problem.

.github/ISSUE_TEMPLATE/schema.md

@@ -0,0 +1,14 @@
+---
+name: Schema Request
+about: Suggest a new data source and mapping
+labels: schema
+
+---
+
+## Schema Description
+* *What data source does this schema describe?*
+* *What OS, hardware, device, etc. does this schema represent?*
+* *What are the specific fields you would expect in the data, e.g. process_name, file_name, ip_address, etc.?*
+* *Are there many different event types in this query?*
+* *Here is a Sysmon [example](https://github.com/endgameinc/eqllib/blob/master/eqllib/sources/sysmon.toml).*
+* *Do you intend to keep or drop fields that are not mapped?*

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,7 @@
+<!--    Please read the Contribution Guidelines for more information about contributing    -->
+
+## Issues
+
+
+## Details
+*Describe details about your requested changes*

.gitignore

@@ -118,3 +118,5 @@ eqllib-env
 docs/analytics
 /docs/links.rst
 /docs/matrices/
+
+junit.xml

CONTRIBUTING.md

@@ -0,0 +1,75 @@
+# Contribution Guide
+Welcome to the Event Query Language (EQL) Analytics Library contribution guide and thank you for expressing an interest in contributing to EQL!
+
+As a quick refresher, the Event Query Language (EQL) was built by Endgame to express relationships between events. The language contains all the conditional matching you need, is data source and platform agnostic, includes the ability to ask stateful questions, and enables the hunter by including data pipes to sift and stack data. If you have structured data, you can start asking questions now. 
+
+The EQL community consists of two main components
+* The core [language and evaluation engine](https://eql.readthedocs.io) 
+* [Analytics Library](https://eqllib.readthedocs.io/) as a community for analytics and hunting strategies.
+
+Contributions to extend core capabilities of the language are directed to [``eql``](https://github.com/endgameinc/eql). For new detections, hunts, data sources, or knowledge sharing please read the guidelines below before contributing.
+
+We are all in this together and hope you join us in expanding behavior based analytics, sharing data models, and more.
+
+# Table of Contents
+1. [Contribution Process](#contribution-process)
+2. [Ways to Contribute](#ways-to-contribute)
+3. [Resources](#resources)
+4. [Licenses](#licenses)
+
+## Contribution Process
+Contributing to the analytics library is a simple process facilitated by Git:
+
+1. Create an [issue](https://github.com/endgameinc/eqllib/issues) to track and discuss the work
+2. Create a [branch](https://help.github.com/en/articles/about-branches)
+3. Submit a [pull request](https://help.github.com/en/articles/about-pull-requests)
+4. Update according to the code review
+5. [Merge](https://help.github.com/en/articles/merging-a-pull-request) after approval.
+
+### Additional Notes
+
+* If you are accustomed to git, then great! If you aren't, don't fear, the command line tools are easy to use, but GitHub also has a straightforward process within your web browser to create branches and subsequent merging
+* Use the Issues and PR templates! Git [Issues](https://github.com/endgameinc/eqllib/issues) are a great place to collaborate, discuss, or just track a request before development begins.
+* There is plenty of literature and resources out there to help you. A great place to start is [GitHub guides](https://guides.github.com/).
+
+## Ways to contribute
+
+### Analytics
+
+With EQL, we can write analytics for hunts, detections, and even enrichments. When writing a rule, be certain to think about the suspicious behavior rather than a single indicator. Here are some questions to help:
+
+ - How much noise will this most likely match?
+ - Does it make more sense as an enrichment or high fidelity detection? Or does it use stacking that is relevant for hunting?
+ - How precise are the ATT&CK mappings? For instance, if tagged with *Privilege Escalation*, does the query specifically have logic to look for jumps in integrity levels or privileges?
+ - Do you want a [docs/templates/links.rst](https://github.com/endgameinc/eqllib/blob/master/docs/_templates/links.rst) for your contributor handle?
+ - Was the UUID autogenerated? Copy and pasted UUIDs often cause confusion when large parts look the same, so completely random UUIDs has the best chance of minimizing confusion.
+
+### Schema
+
+The Event Query Language isn't inherently bound to specific schemas, but within our Analytics Library, we're currently more focused on [security](https://eqllib.readthedocs.io/en/latest/schemas.html#security-events) analytics. If there is a data source that you find useful and you would like to add it to the EQL framework, create an [issue](https://github.com/endgameinc/eqllib/issues) or pull request!
+
+
+## Resources
+For additional resources on EQL, check [here](https://eql.readthedocs.io/en/latest/resources.html)
+
+* Press Releases
+  * [Public Availability of EQL](https://www.endgame.com/news/press-releases/endgame-announces-public-availability-eql)
+* Blogs
+  * [Introducing EQL](https://www.endgame.com/blog/technical-blog/introducing-event-query-language)
+  * [EQL For the Masses](https://www.endgame.com/blog/technical-blog/eql-for-the-masses)
+  * [Getting Started with EQL](https://www.endgame.com/blog/technical-blog/getting-started-eql)
+* Conferences and Webinars
+  * BlackHat 2019: Fantastic Red-Team Attacks and How to Find Them [(abstract)](https://www.blackhat.com/us-19/briefings/schedule/index.html#fantastic-red-team-attacks-and-how-to-find-them-16540)
+  * BSIDES SATX 2019: The Hunter Games [(abstract)](https://www.bsidessatx.com/presentations-2019.html)
+  * Circle City Con 2019: The Hunter Games [(video)](https://www.youtube.com/watch?v=K47gX3WHcm8)
+  * Atomic Friday ([slides](https://eql.readthedocs.io/en/latest/_static/eql-crash-course.pdf)) ([video](https://www.youtube.com/watch?v=yvqxS5Bjc-s))
+  * MITRE(TM) ATT&CK Con 2018: *From Technique to Detection*  [(video)](https://www.youtube.com/watch?v=a3hIIzJrH14)
+* Read the Docs
+  * [EQL](https://eql.readthedocs.io/)
+  * [EQL Analytics Library](https://eqllib.readthedocs.io/)
+* GitHub
+  * [EQL](https://github.com/endgameinc/eql)
+  * [EQL Analytics Library](https://github.com/endgameinc/eqllib)
+
+## License
+The EQL Analytics Library is licensed under the [MIT License](https://github.com/endgameinc/eqllib/blob/master/LICENSE)

Makefile

@@ -2,38 +2,62 @@
 ### EQL
 #################
 
-VENV := ./eqllib-env
+VENV := ./env/eqllib-build
 VENV_BIN := $(VENV)/bin
 PYTHON := $(VENV_BIN)/python
-PIP := $(VENV_BIN)/pip
+PIP := $(PYTHON) -m pip
+SPHINXBUILD ?= $(VENV_BIN)/sphinx-build
 
-init:
+
+$(VENV):
 	pip install virtualenv
 	virtualenv $(VENV)
+	$(PIP) install setuptools -U
+	$(PIP) install .
+	$(PIP) install eqllib[test]
 
-install:
-	$(PYTHON) setup.py install
-
+.PHONY: clean
 clean:
 	rm -rf $(VENV) *.egg-info .eggs *.egg htmlcov build dist .build .tmp .tox
 
-test:
+.PHONY: pytest
+pytest: $(VENV)
 	$(PYTHON) setup.py -q test
 
-lint:
-	$(PYTHON) setup.py -q lint
 
-sdist:
+.PHONY: test
+test: $(VENV) pytest
+
+
+.PHONY: sdist
+sdist: $(VENV)
 	$(PYTHON) setup.py sdist
 
-bdist_egg:
+
+.PHONY: bdist_egg
+bdist_egg: $(VENV)
 	$(PYTHON) setup.py bdist_egg
 
-bdist_wheel:
+
+.PHONY: bdist_wheel
+bdist_wheel: $(VENV)
 	$(PYTHON) setup.py bdist_wheel
 
-.PHONY: docs
-docs:
-	$(PIP) install sphinx sphinx_rtd_theme
+
+.PHONY: install
+install: $(VENV) sdist
 	$(PYTHON) setup.py install
-	$(VENV_BIN)/activate; cd docs; make html
+
+.PHONY: all
+all: sdist
+
+.PHONY: docs
+docs: $(VENV) install
+	$(PIP) install eqllib[docs]
+	cd docs && ../$(SPHINXBUILD) -M html . _build
+
+
+.PHONY: upload
+upload: $(VENV)
+	$(PIP) install twine~=1.13
+	$(VENV_BIN)/twine upload dist/*

README.md

@@ -1,7 +1,8 @@
 # EQL Analytics Library
-See https://eqllib.readthedocs.io for documentation
+[![Library Link](https://readthedocs.org/projects/eqllib/badge/?version=latest)](https://eqllib.readthedocs.io/en/latest/?badge=latest)
+[![Gitter](https://badges.gitter.im/eventquerylang/community.svg)](https://gitter.im/eventquerylang/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
 
-![alt text](docs/_static/eql-whoami.jpg "What is EQL")
+[![alt text](docs/_static/eql-whoami.jpg "What is EQL")](https://eqllib.readthedocs.io)
 
 # Getting Started
 

docs/_templates/analytic.rst

@@ -1,8 +1,10 @@
 .. include:: ../links.rst
-
-========================================================
+{% if display_github and github_user and github_repo and github_version and repo_path %}
+:github_url: https://{{ github_host|default("github.com") }}/{{ github_user }}/{{ github_repo }}/{{ theme_vcs_pageview_mode|default("blob") }}/{{ github_version }}/{{ repo_path }}
+{% endif %}
+=======================================================================
 {{analytic.name}}
-========================================================
+=======================================================================
 
 {{analytic.metadata.description}}
 

docs/_templates/matrix.rst

@@ -15,7 +15,7 @@
     {% for row in matrix_cells %}
     * {% for tactic, technique in zip(tactics, row) %}- {% if technique %}`{{technique.name}}`_
         {% for analytic in coverage[tactic.name][technique.external_references[0].external_id] %}
-        {% if not os or platform.lower() in analytic.metadata.os %}- :doc:`../analytics/{{analytic.id}}`{% endif %}
+        {% if not os or platform.lower() in analytic.metadata.os %}- :doc:`../analytics/{{analytic.id}}`{% if show_os %} ({{analytic.metadata.os | sort|join(", ", 0)}}){% endif %}{% endif %}
         {% endfor %}
         {% endif %}
       {% endfor %}