Skip to content

Commit

Permalink
Updating tutorials for new CC3 release
Browse files Browse the repository at this point in the history
  • Loading branch information
@maxstanden
maxstanden committed Dec 15, 2022
1 parent 94ab9d2 commit d0d7fa1
Show file tree
Hide file tree
Showing 6 changed files with 998 additions and 968 deletions.
240 changes: 55 additions & 185 deletions CybORG/Tutorial/1. Introduction.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,12 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"To use the CybORG environment, it is necessary to import the CybORG class. CybORG stands for __Cyb__er __O__perations __R__esearch __G__ym, so remember to capitalise correctly when importing. "
"To use the CybORG environment, it is necessary to import the CybORG class. CybORG stands for **Cyb**er **O**perations **R**esearch **G**ym, so remember to capitalise correctly when importing. "
]
},
{
"cell_type": "code",
"execution_count": 16,
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -34,39 +34,45 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"ALthough CybORG uses an OpenAI gym API, it is not run by calling gym.make(). Instead it has to be manually instantiated. The constructor has two mandatory string parameters: a mode-type which specifies which engine will be used under the hood and the path to a .yaml scenario file which defines the network layout and agent action spaces.\n",
"\n",
"The only currently supported mode is simulation, while this challenge uses Scenario 1b."
"CybORG has to be manually instantiated by calling the class constructor. This must be passed a ScenarioGenerator class, which contains the details of the scenario.\n",
"For Challenge 3 we will be using the DroneSwarmScenarioGenerator which creates the correct scenario."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/david/anaconda3/envs/ai/lib/python3.9/site-packages/gym/utils/seeding.py:47: DeprecationWarning: \u001b[33mWARN: Function `rng.randint(low, [high, size, dtype])` is marked as deprecated and will be removed in the future. Please use `rng.integers(low, [high, size, dtype])` instead.\u001b[0m\n",
" deprecation(\n"
]
}
],
"source": [
"import inspect\n",
"from CybORG.Simulator.Scenarios import DroneSwarmScenarioGenerator\n",
"\n",
"path = str(inspect.getfile(CybORG))\n",
"path = path[:-7] + f'/Shared/Scenarios/Scenario1b.yaml'\n",
"\n",
"env = CybORG(path, 'sim')"
"sg = DroneSwarmScenarioGenerator()\n",
"cyborg = CybORG(scenario_generator=sg)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Using an Agent with CybORG"
"### The PettingZoo Interface"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Scenario 1b has multiple actors involved: Red team will be attacking the network, Blue team will be defending the network, while Green team represents noise generated by users. Normally the roles of Red and Green will be handled by internal rules-based agents, while Blue team interacts with the external API. However, for demonstration purposes, it will be easier to first examine a Red agent.\n",
"Challenge 3 is a multi-agent scenario consisting of several teams of agents. Red team will be attacking the network, Blue team will be defending the network, while Green team represents the network users who will be passing messages to each other via the drone network. For this challenge, the roles of Red and Green will be handled by internal rules-based agents, while your task is to use the external API to train Blue Team.\n",
"\n",
"CybORG uses an OpenAI Gym interface to interact with agents. Thus, we can begin the scenario by calling the reset method. It is necessary to specify which team you are on as a string parameter. Without using any wrappers, CybORG will return a results object which contains various bits of data. We can get the observation by accessing the corresponding attribute."
"CybORG supports single and multi-agent reinforcement learning. The single agent API is based on OpenAI gym, while the multi-agent API is based on PettingZoo. Since Challenge 3 is a multi-agent scenario, we will thus use the PettingZooParallelWrapper. We do this by instantiating the wrapper and pass in CybORG as the env parameter."
]
},
{
Expand All @@ -78,262 +84,126 @@
"name": "stdout",
"output_type": "stream",
"text": [
"{'success': <TrinaryEnum.UNKNOWN: 2>, 'User0': {'Interface': [{'Interface Name': 'eth0', 'IP Address': IPv4Address('10.0.189.193'), 'Subnet': IPv4Network('10.0.189.192/28')}], 'Sessions': [{'Username': 'SYSTEM', 'ID': 0, 'Timeout': 0, 'PID': 7580, 'Type': <SessionType.RED_ABSTRACT_SESSION: 10>, 'Agent': 'Red'}], 'Processes': [{'PID': 7580, 'Username': 'SYSTEM'}], 'System info': {'Hostname': 'User0', 'OSType': <OperatingSystemType.WINDOWS: 2>, 'OSDistribution': <OperatingSystemDistribution.WINDOWS_SVR_2008: 4>, 'OSVersion': <OperatingSystemVersion.W6_1_7601: 13>, 'Architecture': <Architecture.x64: 2>}}}\n"
"dict_keys(['blue_agent_0', 'blue_agent_1', 'blue_agent_2', 'blue_agent_3', 'blue_agent_4', 'blue_agent_5', 'blue_agent_6', 'blue_agent_7', 'blue_agent_8', 'blue_agent_9', 'blue_agent_10', 'blue_agent_11', 'blue_agent_12', 'blue_agent_13', 'blue_agent_14', 'blue_agent_15', 'blue_agent_16', 'blue_agent_17'])\n"
]
}
],
"source": [
"results = env.reset(agent='Red')\n",
"obs = results.observation\n",
"\n",
"print(obs)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that the above observation outputs a messy dictionary. In order to understand raw CybORG observations, please go to the observation tutorial. We will show how to train a neural network-based agent with CybORG down below.\n",
"\n",
"Because of the complexties of Cybersecurity, the action space in CybORG is generated on the fly. For Scenario 1b, this only needs to be extracted at the beginning of the scenario. This can also be found in the results object. It is another messy dictionary, so we will only print out the keys. You can learn more in the action_space tutorial."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
},
{
"name": "stdout",
"name": "stderr",
"output_type": "stream",
"text": [
"['action', 'subnet', 'ip_address', 'session', 'username', 'password', 'process', 'port', 'target_session', 'agent', 'hostname']\n"
"/home/david/CybORG/CybORG/Agents/Wrappers/PettingZooParallelWrapper.py:241: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.\n",
"Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
" new_obs = np.zeros(obs_length, dtype=np.int)\n"
]
}
],
"source": [
"action_space = results.action_space\n",
"\n",
"print(list(action_space.keys()))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Just like OpenAI gym, CybORG uses a step function to input actions and return results. The method itself requires two string parameters: agent is the name of the team that is taking the action and action is the action being performed."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"from CybORG.Agents import B_lineAgent\n",
"from CybORG.Agents.Wrappers import PettingZooParallelWrapper\n",
"\n",
"agent = B_lineAgent()\n",
"\n",
"action = agent.get_action(obs,action_space)\n",
"env = PettingZooParallelWrapper(env=cyborg)\n",
"obs = env.reset()\n",
"\n",
"results = env.step(agent='Red',action=action)"
"print(obs.keys())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The results object contains the new observation, reward and done attributes."
"We obtain the initial observation by resetting the environment. This ouputs a dictionary whose keys are the names of the various blue agents and whose values are the corresponding agent's observation. We can see below the observation of 'blue_agent_0'. This is in vector form designed for input into a neural network. See the README page for an explanation of what the values in this vector mean."
]
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'success': <TrinaryEnum.TRUE: 1>, '10.0.189.193': {'Interface': [{'IP Address': IPv4Address('10.0.189.193'), 'Subnet': IPv4Network('10.0.189.192/28')}]}, '10.0.189.204': {'Interface': [{'IP Address': IPv4Address('10.0.189.204'), 'Subnet': IPv4Network('10.0.189.192/28')}]}, '10.0.189.197': {'Interface': [{'IP Address': IPv4Address('10.0.189.197'), 'Subnet': IPv4Network('10.0.189.192/28')}]}, '10.0.189.198': {'Interface': [{'IP Address': IPv4Address('10.0.189.198'), 'Subnet': IPv4Network('10.0.189.192/28')}]}, '10.0.189.195': {'Interface': [{'IP Address': IPv4Address('10.0.189.195'), 'Subnet': IPv4Network('10.0.189.192/28')}]}}\n",
"----------------------------------------------------------------------------\n",
"DiscoverRemoteSystems 10.0.189.192/28\n",
"----------------------------------------------------------------------------\n",
"False\n"
"[ 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0\n",
" 0 0 0 0 0 0 0 0 0 0 0 0 0 0 29 18 1 33 20 0 2 90 73 0\n",
" 3 83 11 0 4 18 5 0 5 73 23 0 6 42 19 0 7 65 68 0 8 46 90 0\n",
" 9 32 54 0 10 34 18 0 11 89 3 0 12 66 50 0 13 17 90 0 14 68 28 0\n",
" 15 29 64 0 16 61 99 0 17 43 98 0]\n"
]
}
],
"source": [
"print(results.observation)\n",
"print(76*'-')\n",
"print(results.action)\n",
"print(76*'-')\n",
"print(results.done)"
"print(obs['blue_agent_0'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Adding Opponents"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the example above, only Red Team performed any actions. In order to setup CybORG for the full challenge, we want a Blue Agent to be interacting with the external API while Red and Green take their actions automatically in the background.\n",
"\n",
"We can achieve this by specifying an agents dictionary to pass into CybORG when instantiating the class. Now, whenever the step function is called, the agents will take turn to perform their actions. Note that the turn order is Blue, Green then Red."
"To see the action space for a specific agent, the action_space method is called, passing in the agent name as a string. This returns a Discrete object from OpenAI gym."
]
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0.0\n",
"0.0\n",
"-0.1\n",
"-0.1\n",
"-0.1\n",
"-0.1\n",
"-1.1\n",
"-1.1\n",
"-1.1\n",
"-1.1\n",
"-2.1\n",
"-2.1\n",
"-2.1\n",
"-3.1\n",
"-13.1\n",
"-13.1\n",
"-13.1\n",
"-13.1\n",
"-13.1\n",
"-13.1\n"
"Discrete(56)\n"
]
}
],
"source": [
"from CybORG.Agents import B_lineAgent, GreenAgent, BlueMonitorAgent\n",
"\n",
"agents = {\n",
" 'Red': B_lineAgent,\n",
" 'Green': GreenAgent\n",
"}\n",
"\n",
"env = CybORG(path,'sim',agents=agents)\n",
"\n",
"results = env.reset(agent='Blue')\n",
"obs = results.observation\n",
"action_space = results.action_space\n",
"agent = BlueMonitorAgent()\n",
"\n",
"for step in range(20):\n",
" action = agent.get_action(obs,action_space=action_space)\n",
" results = env.step(agent='Blue',action=action)\n",
" obs = results.observation\n",
" reward = results.reward\n",
" print(reward)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Using a Neural Network with CybORG"
"action_space = env.action_space('blue_agent_0')\n",
"print(action_space)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In order to use a Neural network with CybORG, we will need to import a wrapper. The Challenge Wrapper provides all the functionality needed for the Scenario1b challenge and makes the external api much more in line with OpenAI gym."
"Actions are given to CybORG via the step method. Just like PettingZoo, this is given a dictionary whose keys are the agent names and whose values are the corresponding actions. This function returns the next observation, rewards for the agents, the done signal for each agent and the info dictionary, which is usually empty."
]
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
" 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n"
"0.0\n"
]
}
],
"source": [
"from CybORG.Agents.Wrappers import ChallengeWrapper\n",
"cyborg = CybORG(path,'sim',agents=agents)\n",
"env = ChallengeWrapper(env=cyborg,agent_name='Blue')\n",
"\n",
"obs = env.reset()\n",
"actions = {k:action_space.sample() for k in obs}\n",
"obs, reward, done, info = env.step(actions)\n",
"\n",
"print(obs)"
"print(reward['blue_agent_0'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Not only is the observation now a vector, but we can extract the action and observation spaces exactly like a gym environment."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The step function also behaves like that of OpenAI gym. The info parameter contains a dictionary form of the standard results object for debugging purposes."
"This should be enough to get you started training agents."
]
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
" 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]\n",
"----------------------------------------------------------------------------\n",
"0.0\n",
"----------------------------------------------------------------------------\n",
"False\n",
"----------------------------------------------------------------------------\n",
"{'observation': array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
" 0, 0, 0, 0, 0, 0, 0, 0]), 'next_observation': None, 'done': False, 'reward': 0.0, 'action': <CybORG.Shared.Actions.Action.Sleep object at 0x7f3a314d7750>, 'info': None, 'parameter_mask': None, 'action_space': 41, 'error': None, 'error_msg': None, 'action_name': None, 'selection_masks': None}\n"
]
}
],
"source": [
"action = 0\n",
"\n",
"obs, reward, done, info = env.step(action)\n",
"\n",
"print(obs)\n",
"print(76*'-')\n",
"print(reward)\n",
"print(76*'-')\n",
"print(done)\n",
"print(76*'-')\n",
"print(info)"
]
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
Expand All @@ -347,7 +217,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.5"
"version": "3.9.15"
}
},
"nbformat": 4,
Expand Down
Loading

0 comments on commit d0d7fa1

Please sign in to comment.