Layers directory cleanup (#649)

* fix link

* move outdated layers to old_versions directory

* remove outdated layer samples

* rename dir

* move samples

* moved script to mitreattack-python

* removed update_layers, superseded by our diffStix script

* rename dir

* Revert "removed update_layers, superseded by our diffStix script"

This reverts commit a5d9517.

* remove update-layers.py

* removed update_layers

* update readme

* rename dir

* modify directory scheme

* add spec links for v4.3 and v3.0

* update changelog

CHANGELOG.md

@@ -12,6 +12,7 @@
 # Changes Staged on Develop
 
 ## Improvements
+- Refactored the `layers/` directory structure to organize Layer File Formats into versioned subdirectories and removed outdated layer samples. See pull request [#649](https://github.com/mitre-attack/attack-navigator/pull/649).
 - Updated Angular from v14 to v17.
 
 # 5.0.1 - 9 May 2024
@@ -94,7 +95,7 @@ Adds support for ATT&CK v14.0.
 
 ## Layer File Format Changes
 
-Layer file format updated to version 4.5. See [layers/LAYERFORMATv4_5.md](layers/LAYERFORMATv4_5.md) for the full specification.
+Layer file format updated to version 4.5. See [layer format v4.5](layers/spec/v4.5/layerformat.md) for the full specification.
 
 - Added support for selecting only visible techniques. The `selectVisibleTechniques` field specifies whether or not hidden techniques will be included in the different select behaviors.
 - Added support for configuring how sub-techniques are displayed in the layer with the `expandedSubtechniques` field. This property can be set to `all`, `annotated`, or `none` to expand all sub-techniques, expand only annotated sub-techniques, or collapse all sub-techniques, respectively.
@@ -128,7 +129,7 @@ Adds support for ATT&CK v13.
 
 ## Layer File Format Changes
 
-Layer file format updated to version 4.4. This update adds support for layers created with a custom collection or STIX bundle; the optional `customDataURL` field contains the URL from which custom data was loaded. This update is fully backwards compatible with layer format v4.3 since the added field is optional. See [layers/LAYERFORMATv4_4.md](layers/LAYERFORMATv4_4.md) for the full specification.
+Layer file format updated to version 4.4. This update adds support for layers created with a custom collection or STIX bundle; the optional `customDataURL` field contains the URL from which custom data was loaded. This update is fully backwards compatible with layer format v4.3 since the added field is optional. See [layer format v4.4](layers/spec/v4.4/layerformat.md) for the full specification.
 
 # 4.7.1 - 8 November 2022
 
@@ -213,7 +214,7 @@ Adds support for ATT&CK v11.
 
 ## Layer File Format Changes
 
-Updated the Layer File Format to v4.3 which adds a `links` array field to technique objects and to layers. This supports the assignment of hyperlinks to techniques which are accessed in the context menu and to layers which are accessed in the layer information dropdown menu. Link objects must conform to the schema `{"label": string, "url": string}` or `{"divider": boolean}`. A separator is displayed in the technique context menu where the `divider` property occurs in the list of hyperlinks.
+Updated the Layer File Format to v4.3 which adds a `links` array field to technique objects and to layers. This supports the assignment of hyperlinks to techniques which are accessed in the context menu and to layers which are accessed in the layer information dropdown menu. Link objects must conform to the schema `{"label": string, "url": string}` or `{"divider": boolean}`. A separator is displayed in the technique context menu where the `divider` property occurs in the list of hyperlinks. See [layer format v4.3](layers/spec/v4.3/layerformat.md) for the full specification.
 
 # v4.5.4 - 15 November 2021
 
@@ -315,7 +316,7 @@ Version 4.4 of the Navigator restores Safari support provided you are using Safa
 
 ## Layer File Format Changes
 
-Layer file format updated to version 4.2. This update is fully backwards compatible with the layer format v4.1 since the added fields are optional. See [layers/LAYERFORMATv4_2.md](layers/LAYERFORMATv4_2.md) for the full specification.
+Layer file format updated to version 4.2. This update is fully backwards compatible with the layer format v4.1 since the added fields are optional. See [layer format v4.2](layers/spec/v4.2/layerformat.md) for the full specification.
 
 This update adds settings for aggregate scores to the layout object of the layer:
 
@@ -356,7 +357,7 @@ Refactored the implementation of tabs to reduce performance issues when opening
 
 ## Layer File Format Changes
 
-Layer file format updated to version 4.1. This update is fully backwards compatible with layer format v4.0 since the added field is optional. See [layers/LAYERFORMATv4_1.md](layers/LAYERFORMATv4_1.md) for the full specification.
+Layer file format updated to version 4.1. This update is fully backwards compatible with layer format v4.0 since the added field is optional. See [layer format v4.1](layers/spec/v4.1/layerformat.md) for the full specification.
 
 This update adds an optional `divider` object to the `metadata` format on technique objects. Each object in the metadata array must either be of the schema `{"name": string, "value": string}` or `{"divider": boolean}`. A separator will be displayed in the metadata tooltip where the `divider` property occurs in the list of metadata.
 
@@ -391,7 +392,7 @@ This update adds an optional `divider` object to the `metadata` format on techni
 
 ## Layer File Format Changes
 
-Layer file format updated to version 4.0. Older versions can still be loaded in the Navigator, but will no longer display the Pre-ATT&CK domain. See [layers/LAYERFORMATv4.md](layers/LAYERFORMATv4.md) for the full specification.
+Layer file format updated to version 4.0. Older versions can still be loaded in the Navigator, but will no longer display the Pre-ATT&CK domain. See [layer format v4.0](layers/spec/v4.0/layerformat.md) for the full specification.
 
 - ATT&CK version 8.0 removed the pre-ATT&CK domain, which became two tactics tagged with the `PRE` platform in the Enterprise domain. The `stages` section of filters have been removed to reflect this migration.
 - Replaced `version` field with `versions` object which specifies the layer format, Navigator, and ATT&CK content versions in support of the mixed domains and versions update.
@@ -508,7 +509,7 @@ If you want to continue using the non-sub-techniques Navigator, please use the [
 
 ## Layer File Format Changes
 
-Layer file format updated to version 3.0. Older versions can still be loaded in the Navigator, but may have degraded functionality.
+Layer file format updated to version 3.0. Older versions can still be loaded in the Navigator, but may have degraded functionality. See [layer format v3.0](layers/spec/v3.0/layerformat.md) for the full specification.
 
 - Removed "viewMode" enumeration in favor of "layout" object. viewMode will get parsed into a layout configuration automatically, but the conversion is not perfect since the layouts have changed.
 - Added "showSubtechniques" field to technique objects.
@@ -569,7 +570,7 @@ The "features" structure is used to enable/disable specific Navigator features.
 
 ## Layer File Format Changes
 
-Layer file format updated to version 2.2. Older versions can still be loaded in the Navigator, and this update is fully backwards compatible with Version 2.1. See [layers/LAYERFORMATv2_2md](layers/LAYERFORMATv2_2.md) for the full v2.2 specification.
+Layer file format updated to version 2.2. Older versions can still be loaded in the Navigator, and this update is fully backwards compatible with Version 2.1. See [layer format v2.2](layers/spec/v2.2/layerformat.md) for the full specification.
 
 - Added the following cloud platforms to the set of acceptable enterprise platforms: "AWS", "GCP", "Azure", "Azure AD", "Office 365", "SaaS".
 - Updated Enterprise and Mobile platforms to match their format as seen elsewhere in ATT&CK. This change is fully backwards compatible, and if the old format is detected it will automatically be updated to the new format.
@@ -654,7 +655,7 @@ Also, please note that multiple matrices are only supported for `mitre-mobile`,
 
 ## Layer File Format Changes
 
-Layer file format updated to version 2.1. This update is fully backwards compatible with layer format v2.0 since all the added fields are optional. See [layers/LAYERFORMATv2_1.md](layers/LAYERFORMATv2_1.md) for the full v2.1 specification.
+Layer file format updated to version 2.1. This update is fully backwards compatible with layer format v2.0 since all the added fields are optional. See [layer format v2.1](layers/spec/v2.1/layerformat.md) for the full specification.
 
 This update constitutes the addition of `metadata` fields to the layer and technique objects. Metadata can be used to support other applications using the layer format, or to add additional descriptive fields to layers or techniques. Metadata is formatted as an array, and each piece of metadata in the array must conform to the schema `{"name": string, "value": string}`.  
 
@@ -714,7 +715,7 @@ This update constitutes the addition of `metadata` fields to the layer and techn
 
 ## Layer File Format Changes
 
-Layer file format updated to version 2.0. Older layer versions can still be loaded by the Navigator, however some fields may no longer be supported. See [layers/LAYERFORMATv2.md](layers/LAYERFORMATv2.md) for the full v2.0 specification.
+Layer file format updated to version 2.0. Older layer versions can still be loaded by the Navigator, however some fields may no longer be supported. See [layer format v2.0](layers/spec/v2.0/layerformat.md) for the full specification.
 
 - Replaced the `viewFullTable` field (boolean) with the `viewMode` field (number) in order to support the "super compact" view option. See issue [#11](https://github.com/mitre-attack/attack-navigator/issues/11).
   - If `viewFullTable` is present in a layer file uploaded to the v2.0 Navigator it will be ignored.

Dockerfile

@@ -14,9 +14,11 @@ RUN npm install
 # copy over needed files
 COPY ./nav-app/ ./
 
+# copy layers directory
 WORKDIR /src
-COPY layers/*.md ./layers/
+COPY layers/ ./layers/
 
+# copy markdown files from root
 COPY *.md ./
 
 WORKDIR /src/nav-app

README.md

@@ -238,7 +238,7 @@ Local files to load should be placed in the `nav-app/src/assets/` directory.
         "enabled": true,
         "urls": [
             "assets/example.json", 
-            "https://raw.githubusercontent.com/mitre-attack/attack-navigator/master/layers/data/samples/Bear_APT.json"
+            "https://raw.githubusercontent.com/mitre-attack/attack-navigator/master/layers/samples/Bear_APT.json"
         ]
     }
    ```
@@ -276,7 +276,7 @@ If you want to embed the Navigator in a webpage, use an iframe:
 
 If you want to embed a version of the Navigator with specific features removed (e.g tabs, adding annotations), or with a default layer, we recommend using the _create customized Navigator_ feature. We highly recommend disabling the "leave site dialog" via this means when embedding the Navigator since otherwise you will be warned whenever you try to leave the embedding page. Refer to the in-application help page section "Customizing the Navigator" for more details.
 
-The following is an example iframe which embeds our [*Bear APTs](layers/data/samples/Bear_APT.json) layer with tabs and the ability to add annotations removed:
+The following is an example iframe which embeds our [*Bear APTs](layers/samples/Bear_APT.json) layer with tabs and the ability to add annotations removed:
 
 ```HTML
 <iframe src="https://mitre-attack.github.io/attack-navigator/enterprise/#layerURL=https%3A%2F%2Fraw.githubusercontent.com%2Fmitre%2Fattack-navigator%2Fmaster%2Flayers%2Fdata%2Fsamples%2FBear_APT.json&tabs=false&selecting_techniques=false" width="1000" height="500"></iframe>

USAGE.md

@@ -45,8 +45,9 @@ Each layer created is independent of other layers. However, layers can be
 <a href="#creating-layers-from-other-layers">combined</a> in ways to support analysis, or
 <a href="#saving-and-loading-layers">saved locally</a>. Layer files are saved in easy to parse and easy to generate JSON
 file so that ATT&CK data can be used in other applications, analyzed beyond the capability of the ATT&CK Navigator, and
-generated by tools for import into the Navigator. The Layer file format is
-described <a href="layers/">here</a>.
+generated by tools for import into the Navigator.
+
+*See the latest <a href="layers/spec/">Layer File Format Definition</a> for the full specification.*
 
 ## Creating New Layers
 

layers/README.md

@@ -2,23 +2,8 @@
 
 A layer constitutes a set of annotations on the ATT&CK matrix for a specific technology domain. Layers can also store a default configuration of the view such as sorting, visible platforms, and more. The ATT&CK Navigator includes functionalities for exporting annotations into layer files, as well as the ability to import layer files for viewing.
 
-See the [layer format specification](LAYERFORMATv4.md) for more information about Layer files.
+See the latest [layer format specification](spec/v4.5/layerformat.md) for more information about Layer files.
 
 ## Sample Layers
 
-This repository includes [several layers demonstrating example use cases of layers and the ATT&CK Navigator](data/samples). The scripts used to generate these layer files can be found on our [attack-scripts repository here](https://github.com/mitre-attack/attack-scripts/tree/master/scripts/layers/samples) to serve as an example on how to access and work with the [the source data on our MITRE/CTI repo](https://github.com/mitre/cti).
-
-Lastly, we've included [a tutorial on the programmatic generation of layers from CSV](attack_layers).
-
-Feel free to come up with your own ideas for layer file generation, and contribute them to the community by making a pull request to the ATT&CK Navigator!
-
-## Layers showing updates to the ATT&CK knowledge base
-
-[Updates to the ATT&CK knowledge base](https://attack.mitre.org/resources/updates/) are typically accompanied by layer files showing changes to techniques. Layers for relevant updates can be found in the [data/update_layers](data/update_layers) folder. The script used to generate these update layers [can be found in our attack-scripts repository](https://github.com/mitre-attack/attack-scripts/blob/master/scripts/diff_stix.py).
-
-## Updating outdated layers
-
-The sub-techniques update of ATT&CK caused many techniques to be replaced by sub-techniques. Since the replacing sub-techniques have different IDs, many layers created before the sub-technques release will still be using IDs for the replaced techniques and therefore won't work properly in the new version even if the annotation format is correct. [update-layers.py](update-layers.py) is a conversion script which both updates layers to the most recent format and also updates technique IDs that of their replacers where possible. There are however a few cases which won't be caught:
-1. Cases where techniques which have been replaced by multiple sub-techniques are ignored entirely due to limitations in the remapping data.
-2. Cases where the `tactic` field was present but the replacing technique is not in that tactic.
-Run `python3 update-layers.py -h` for usage instructions.
+This repository includes a couple of [sample layers](samples/) demonstrating example use cases of layers and the ATT&CK Navigator. The scripts used to generate these layer files can be found in the [mitreattack-python repository](https://github.com/mitre-attack/attack-scripts/tree/master/scripts/layers/samples). These scripts may serve as examples on how to access and work with [ATT&CK data](https://github.com/mitre/cti).