Merge pull request #17 from avilum/feature/new_cli_and_interactive_qu…

…ickstart Feature/new cli and interactive quickstart
avilum · Apr 9, 2023 · b0c8b35 · b0c8b35
2 parents e7e65ac + 372ef0e
commit b0c8b35
Show file tree

Hide file tree

Showing 17 changed files with 195 additions and 168 deletions.
diff --git a/.gitignore b/.gitignore
@@ -128,3 +128,7 @@ dmypy.json
 
 # Pyre type checker
 .pyre/
+
+# Secimport files
+test_traced_modules.*
+sandbox.*
diff --git a/README.md b/README.md
@@ -1,178 +1,109 @@
-<!-- START doctoc generated TOC please keep comment here to allow auto update -->
-<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
-**Table of Contents**  *generated with [DocToc](https://github.com/thlorenz/doctoc)*
-
-- [secimport](#secimport)
-  - [Why It's Awesome](#why-its-awesome)
-  - [Examples](#examples)
-  - [Quick Start - Using the CLI](#quick-start---using-the-cli)
-  - [Alternative Usage: Python Imports](#alternative-usage-python-imports)
-  - [Docker](#docker)
-  - [Installation](#installation)
-  - [Documentation](#documentation)
-  - [References](#references)
-- [Contributing](#contributing)
-    - [Roadmap](#roadmap)
-
-<!-- END doctoc generated TOC please keep comment here to allow auto update -->
-
 # secimport
-`secimport` is a sandbox toolkit that traces your application and enforces privileges per module in your code in runtime.<br>
-It uses backends like bpftrace (eBPF) and dtrace under the hood, making it cross-platform.
 
-[Medium Article](https://infosecwriteups.com/sandboxing-python-modules-in-your-code-1e590d71fc26?source=friends_link&sk=5e9a2fa4d4921af0ec94f175f7ee49f9)
+`secimport` is a cross-platform sandbox toolkit that traces your Python application and enforces privileges per module in your code in runtime. It uses backends like bpftrace (eBPF) and dtrace under the hood.
 
 ## Why It's Awesome
+
 - Trace which syscalls are called by each module in your code.
-  - Like seccomp, apparmor, firejail, etc.
-  - Audit the flow of your python application at user-space/os/kernel level.
-- Reduce supply chain attack and RCE vectors.
-  - Restrict modules/packages inside your production environment
-  - It is hard to keep 3rd party and open source up-to-date.
-  - Vulnerabilities are inevitable.
-- No performance impact (see )
+  - `secimport` uses USDT (Userland Statically Defined Tracing) probes in the runtime (Python interpreter for example) using eBPF and dtrace instrumentation scripts.
+  - Audit the flow of your application at user-space/os/kernel level
+- Reduce supply chain attack and RCE vectors by restricting modules/packages inside your production environment.
+- No performance impact (see [Performance](https://github.com/avilum/secimport/wiki/Performance-Benchmarks)).
 - Don't change the way you code!
+- Supports `Python` at the moment
+  -  `Go` is under development
 
-## Examples
-<a href="https://github.com/avilum/secimport/wiki/Sandbox-Examples">Sandbox Examples</a> contains basic and advanced usage with many interactive examples.
-
-## Quick Start - Using the <a href="https://github.com/avilum/secimport/wiki/Command-Line-Usage">CLI</a><br>
-  - `secimport` or `python -m secimport.cli --help`
-Let's trace some logic we would like to trace.
-```shell
-root@e28bf0ec63d4:/workspace# secimport trace
- >>> secimport trace
-
-TRACING: ['/workspace/secimport/profiles/trace.bt', '-c', '/workspace/Python-3.10.0/python', '-o', 'trace.log']
 
-                        Press CTRL+D to stop the trace;
+## Installation
+For evaluation, we highly recommend the QuickStart with <a href="#Docker">Docker</a> instead of self-installing.<br>
+If you are not using Docker, follow <a href="https://github.com/avilum/secimport/wiki/Installation">Installation</a> to install eBPF or DTrace.
+- To install secimport from git clone: `python3 -m pip install -e .`
+- To install secimport from pypi (latest stable release): `python3 -m pip install secimport`
 
-Python 3.10.0 (default, Mar 19 2023, 08:34:46) [GCC 9.4.0] on linux
-Type "help", "copyright", "credits" or "license" for more information.
->>> import os
->>> os.cpu_count()
 
-CTRL+D
- TRACING DONE;
+## Docker
+The quickest way to evaluate `secimport` is to use our [Docker container](docker/README.md), which includes `bpftrace` (`ebpf`) and other plug-and-play examples.
+
+## Quick Start - Using the CLI
+To run an end-to-end interactive example:
+1. Build a docker with custom kernel that matches your existing OS kernel version
+    ```
+    ➜  secimport ✗ cd docker/
+    ➜  docker ✗ ./build.sh
+    ```
+2. Run the container
+    ```
+    ➜  docker ✗ ./run.sh
+    Running temporary container...
+
+    root@f05d2c33b0b3:/workspace#
+    ```
+3. Use the CLI
+    ```
+    root@f05d2c33b0b3:/workspace# secimport interactive
+
+    Let's create our first tailor-made sandbox with secimport!
+    - A python shell will be opened
+    - The behavior will be recorded.
+
+    OK? (y): y
+    >>> secimport trace
+
+    TRACING: ['/workspace/secimport/profiles/trace.bt', '-c', '/workspace/Python-3.10.0/python', '-o', 'trace.log']
+
+                            Press CTRL+D/CTRL+C to stop the trace;
+
+    Python 3.10.0 (default, Apr  9 2023, 17:19:17) [GCC 9.4.0] on linux
+    Type "help", "copyright", "credits" or "license" for more information.
+    >>> import ...
+
+
+    ```
+## Sandbox Your Program (Using the CLI)
+`secimport trace` will start a bpftrace program that will log all the syscalls for all the modules in your application into a file.<br>
+Once you covered the logic you would like to or sandbox and you're satisfied, hit `CTRL+C` or `CTRL+D` or wait for the program to finish.
+```
+$ secimport trace
+$ secimport trace -h
+$ secimport trace_pid 123
+$ secimport trace_pid -h
 ```
-Create a sandbox from the trace, it should allow only running os.num_cpus() and no syscalls;
-```shell
-root@e28bf0ec63d4:/workspace# secimport build
- >>> secimport build
-
-SECIMPORT COMPILING...
-
-CREATED JSON TEMPLATE:  sandbox.json
-CREATED YAML TEMPLATE:  sandbox.yaml
-
 
-compiling template sandbox.yaml
-...
-[debug] adding syscall openat to allowlist for module /workspace/Python-3.10.0/Lib/ast.py
-[debug] adding syscall close to allowlist for module /workspace/Python-3.10.0/Lib/codecs.py
-...
+Then, build a sandbox from the trace using the `build` command:
+```
+# secimport build
+$ secimport build -h
+```
 
- SANDBOX READY: sandbox.bt
+Finally, run the sandbox with the `run` command:
 ```
-Try to violate the behavior by running os.system, we expect it to be logged / kill the process:
-```shell
-root@e28bf0ec63d4:/workspace# secimport run
- >>> secimport run
- RUNNING SANDBOX... ['./sandbox.bt', '--unsafe', ' -c ', '/workspace/Python-3.10.0/python']
-Attaching 5 probes...
-REGISTERING SYSCALLS...
-STARTED
-Python 3.10.0 (default, Mar 19 2023, 08:34:46) [GCC 9.4.0] on linux
-Type "help", "copyright", "credits" or "license" for more information.
->>> import os
->>> os.cpu_count()
-6
->>> os.system('echo pwned')
-[SECIMPORT VIOLATION]: <stdin> called syscall clone at depth 1
-[SECIMPORT VIOLATION]: <stdin> called syscall execve at depth 1
-[SECIMPORT VIOLATION]: <stdin> called syscall wait4 at depth 1
-pwned
-0
+$ secimport run
+$ secimport run --entrypoint my_custom_main.py
+$ secimport run --entrypoint my_custom_main.py --stop_on_violation=true
+$ secimport run --entrypoint my_custom_main.py --kill_on_violation=true
+$ secimport run --sandbox_executable /path/to/my_sandbox.bt --pid 2884
+$ secimport run --sandbox_executable /path/to/my_sandbox.bt --sandbox_logfile my_log.log
+$ secimport run -h
 ```
 
-## Alternative Usage: <a href="examples/python_imports/">Python Imports</a>
-  - Works by replacing `import` with `secimort.secure_import` for selected modules.
+For more detailed usage instructions, see the [Command-Line Usage](https://github.com/avilum/secimport/wiki/Command-Line-Usage) page.
 
-## Docker
-The quickest method to evaluate secimport is using our [Docker for MacOS and Linux](docker/README.md). The container includes `bpftrace`, python and secimport.<br> `dtrace` can be used in Mac OS, Windows, Solaris, Unix, some Linux distributions.<br>
+## Python API
+
+You can also use `secimport` by replacing `import` with `secimport.secure_import` for selected modules. See the [Python Imports](examples/python_imports/) example for more details.
 
-```python
-root@d57458518cbf:/workspace$ ./run_sandbox.sh
-🚀 Starting secimport sandbox with bpftrace backend, the sandbox should kill the python process...
+## Examples
 
-  PID TTY          TIME CMD
-    1 pts/0    00:00:00 sh
-   18 pts/0    00:00:00 bash
-   19 pts/0    00:00:00 bpftrace
-   23 pts/0    00:00:00 python
+The [Sandbox Examples](https://github.com/avilum/secimport/wiki/Sandbox-Examples) page contains basic and advanced real-world examples.
 
+## Contributing
 
-🛑 The process was killed, as expected.
-🚀 The sandbox bpftrace code is at sandbox.bt
-🚀 The sandbox log is at sandbox.log.
-```
-## Installation
-For evaluation, we highly recommend the QuickStart with <a href="#Docker">Docker</a> instead of self-installing.
+For information on how to contribute to `secimport`, see the [Contributing](https://github.com/avilum/secimport/blob/master/docs/CONTRIBUTING.md) guide.
 
-```shell
-# To install secimport from git clone:
-python3 -m pip install -e .
+## Roadmap
 
-# To install secimport from pypi (latest stable release):
-python3 -m pip install secimport
-```
+See the [Roadmap](https://github.com/avilum/secimport/blob/master/docs/ROADMAP.md) for the planned features and development milestones.
+
+## Changelog
 
-This does not install the backends that secimport needs in order to run, but only the python package.
-To install one of the backends yourself (eBPF or DTrace), please see <a href="https://github.com/avilum/secimport/wiki/Installation">Installation</a>.
-
-## Documentation
-[Our documentation center is Wiki on GitHub](https://github.com/avilum/secimport/wiki)
-
-## References
-- Read more about the primitives of secimport:
-  - `bpftrace` - https://github.com/iovisor/bpftrace
-  - `dtrace` - [DTrace Cheatsheet](https://www.brendangregg.com/DTrace/DTrace-cheatsheet.pdf)
-    - [DTrace for Linux (2018)](https://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html)
-- <a href="https://github.com/avilum/secimport/wiki/Sandbox-Examples">Sandbox Examples</a>
-- Guides
-  - <a href="https://github.com/avilum/secimport/wiki/Tracing-Processes">Tracing Processes Guide</a>
-  - <a href="https://github.com/avilum/secimport/wiki/Installation">Installation</a>
-  - <a href="https://github.com/avilum/secimport/wiki/YAML-Profiles">Create a Sandbox from YAML file</a>
-  - <a href="https://github.com/avilum/secimport/wiki/MacOS-Users">Mac OS Users</a>
-  - <a href="https://github.com/avilum/secimport/wiki/F.A.Q">F.A.Q</a>
-
-# Contributing
-1. Fork this repo ^
-2. Install `poetry`, `pre-commit`, `doctoc` (Run `python3 -m pip install poetry pre-commit doctoc`)
-3. Run `poetry install`
-4. Add your feature/bugfixes/changes (see [Roadmap](#roadmap) if your are looking for Ideas)
-5. Run ./scripts/lint to correct the code styling and lint using pre-commit hooks
-6. Create a pull request with a desriptive title :)
-
-### Roadmap
-- <b>Extandible Language Template</b>
-  - Create a miminal template that will support instrumenting different artifacts for different languages easily.
-    - <b>JS support</b> (bpftrace/dtrace hooks)
-      - Implement a template for JS event loop (V8 or alt.)
-    - <b>Go support</b>
-      - Implement a template for golang's call stack
-    - <b>Node support</b>
-      - Implement a template for Node's call stack and event loop
-
-Changelog:
-- ✔️ Added Allow/Block list configuration
-- ✔️ Created a .yaml configuration per module in the code
-  - ✔️ Use secimport to compile that yml
-  - ✔️ Create a single dcript policy
-  - ✔️ Run an application with that policy using dtrace, without using `secure_import`
-- ✔️ Added eBPF basic support using bpftrace
-  - ✔️ bpftrace backend tests
-- ✔️ Implemented python USDT probes template
-- ✔️ Added CLI for bpftrace backend usage
-- ✔️ Updated documentation and improved CLI
-- ✔️ Added GIFs
+See the [Changelog](https://github.com/avilum/secimport/blob/master/docs/CHANGELOG.md) for development progress and existing features.
diff --git a/docker/docker/Dockerfile b/docker/docker/Dockerfile
@@ -16,7 +16,7 @@ ARG SECIMPORT_VERSION="0.6.2"
 
 # TODO: add openssl (longer build time, but pip will work for our interpreter)
 RUN echo "Installing prerequisites" && \
-    apt-get update && apt-get install nano sudo build-essential libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev curl wget auditd vim tmux git binutils unzip gcc systemtap-sdt-dev cmake zlib1g-dev -y
+    apt-get update && apt-get install nano sudo build-essential libsqlite3-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev curl wget auditd vim tmux git binutils unzip gcc systemtap-sdt-dev cmake zlib1g-dev -y
 RUN echo "Installing python with dtrace" && \
     curl -o Python-${PYTHON_VERSION}.tgz https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz && tar -xzf Python-${PYTHON_VERSION}.tgz && \
     cd Python-${PYTHON_VERSION} && ./configure --with-dtrace --prefix=/usr/local/openssl --prefix=$(pwd) --with-ensurepip=install && make && make install && \
@@ -32,12 +32,13 @@ RUN Python-3.10.0/python -m pip install secimport==${SECIMPORT_VERSION} fastapi
 
 ENV BPFTRACE_KERNEL_SOURCE=/kernel/usr/src/linux-headers
 COPY setup.sh .
+COPY run_tests.sh .
 COPY run_sandbox.sh .
 COPY processing_sandbox.bt .
 COPY fastapi_example.sh .
 COPY fastapi_main.py .
 RUN chmod 755 run_sandbox.sh
-RUN chmod +x fastapi_example.sh fastapi_main.py
+RUN chmod +x run_tests.sh fastapi_example.sh fastapi_main.py
 RUN \
     echo 'export PYTHONPATH="/workspace/Python-3.10.0/python:$PYTHONPATH"' >> /root/.bashrc && \
     echo 'alias python="/workspace/Python-3.10.0/python"' >> /root/.bashrc && \

diff --git a/docker/docker/run_tests.sh b/docker/docker/run_tests.sh
@@ -0,0 +1,8 @@
+#! /bin/bash
+echo "Running tests..."
+echo "#! /bin/bash \\n\
+/workspace/Python-3.10.0/python -m secimport.cli" > /tmp/secimport && chmod +x /tmp/secimport
+export PATH=/tmp/:$PATH
+export alias secimport="/workspace/Python-3.10.0/python -m secimport.cli"
+cd /workspace
+/workspace/Python-3.10.0/python -m pip install coverage pytest && /workspace/Python-3.10.0/python -m coverage run -m pytest tests && /workspace/Python-3.10.0/python -m coverage report -m --skip-empty --omit=\"*/tests/*\"
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
@@ -0,0 +1,25 @@
+# Changelog
+- ✔️ Added Allow/Block list configuration
+- ✔️ Created a .yaml configuration per module in the code
+  - ✔️ Use secimport to compile that yml
+  - ✔️ Create a single dcript policy
+  - ✔️ Run an application with that policy using dtrace, without using `secure_import`
+- ✔️ Added eBPF basic support using bpftrace
+  - ✔️ bpftrace backend tests
+- ✔️ Implemented python USDT probes template
+- ✔️ Added CLI for bpftrace backend usage
+- ✔️ Updated documentation and improved CLI
+- ✔️ Added GIFs
+
+## References
+- Read more about the primitives of secimport:
+  - `bpftrace` - https://github.com/iovisor/bpftrace
+  - `dtrace` - [DTrace Cheatsheet](https://www.brendangregg.com/DTrace/DTrace-cheatsheet.pdf)
+    - [DTrace for Linux (2018)](https://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html)
+- <a href="https://github.com/avilum/secimport/wiki/Sandbox-Examples">Sandbox Examples</a>
+- Guides
+  - <a href="https://github.com/avilum/secimport/wiki/Tracing-Processes">Tracing Processes Guide</a>
+  - <a href="https://github.com/avilum/secimport/wiki/Installation">Installation</a>
+  - <a href="https://github.com/avilum/secimport/wiki/YAML-Profiles">Create a Sandbox from YAML file</a>
+  - <a href="https://github.com/avilum/secimport/wiki/MacOS-Users">Mac OS Users</a>
+  - <a href="https://github.com/avilum/secimport/wiki/F.A.Q">F.A.Q</a>
diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md
@@ -0,0 +1,9 @@
+# Contributing
+1. Fork this repo ^
+2. Install `poetry`, `pre-commit`, `doctoc` (Run `python3 -m pip install poetry pre-commit doctoc`)
+3. Run `poetry install` and `pre-commit install` from the root directory of the project.
+4. Add your feature/bugfixes/changes (see [Roadmap](#roadmap) if your are looking for Ideas)
+5. Run `./scripts/check.sh` to correct the code styling and lint using pre-commit hooks.
+   1. run `git add .` if the script modified any files.
+   2. commit the changes using `git commit `
+6. Create a pull request with a desriptive title :)
diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md
@@ -0,0 +1,9 @@
+# Roadmap
+- <b>Extandible Language Template</b>
+  - Create a miminal template that will support instrumenting different artifacts for different languages easily.
+    - <b>JS support</b> (bpftrace/dtrace hooks)
+      - Implement a template for JS event loop (V8 or alt.)
+    - <b>Go support</b>
+      - Implement a template for golang's call stack
+    - <b>Node support</b>
+      - Implement a template for Node's call stack and event loop
diff --git a/requirements.txt b/requirements.txt
diff --git a/sandbox.d b/sandbox.d
diff --git a/sandbox.json b/sandbox.json
diff --git a/sandbox.yaml b/sandbox.yaml
diff --git a/scripts/check.sh b/scripts/check.sh
@@ -10,7 +10,12 @@ fi
 
 python3 -m ruff --fix .
 doctoc
-git add .
 pre-commit
 export PYTHONPATH=$(pwd):$PYTHONPATH
-python3 -m pytest tests/
+cd docker/
+./build.sh
+
+# Run unit tests inside container
+# cd ..
+# export KERNEL_VERSION=`docker run --rm -it alpine uname -r | cut -d'-' -f1`
+# docker run --rm --name=secimport --privileged -v "$(pwd)/secimport":"/workspace/secimport/" -v "$(pwd)/tests":"/workspace/tests/" -it secimport:${KERNEL_VERSION} "/workspace/run_tests.sh"