commit changes that got mucked up in PR GoogleCloudPlatform#97

kardiff18 · Nov 9, 2018 · bff43d3 · bff43d3
1 parent 65426a7
commit bff43d3
Show file tree

Hide file tree

Showing 5 changed files with 355 additions and 0 deletions.
diff --git a/README.md b/README.md
@@ -24,6 +24,7 @@ The tools folder contains ready-made utilities which can simpilfy Google Cloud P
 * [DNS Sync](tools/dns-sync) - Sync a Cloud DNS zone with GCE resources. Instances and load balancers are added to the cloud DNS zone as they start from compute_engine_activity log events sent from a pub/sub push subscription. Can sync multiple projects to a single Cloud DNS zone.
 * [GCE Quota Sync](tools/gce-quota-sync) - A tool that fetches resource quota usage from the GCE API and synchronizes it to Stackdriver as a custom metric, where it can be used to define automated alerts.
 * [GCS Bucket Mover](tools/gcs-bucket-mover) - A tool to move user's bucket, including objects, metadata, and ACL, from one project to another.
+* [GCP Architecture Visualizer](/tools/gcp-arch-viz) - A tool that takes CSV output from a Forseti Inventory scan and draws out a dynamic hierarchical tree diagram of org -> folders -> projects -> gcp_resources using the D3.js javascript library.
 * [GKE Billing Export](tools/gke-billing-export) - Google Kubernetes Engine fine grained billing export.
 * [GSuite Exporter](tools/gsuite-exporter/) - A Python package that automates syncing Admin SDK APIs activity reports to a GCP destination. The module takes entries from the chosen Admin SDK API, converts them into the appropriate format for the destination, and exports them to a destination (e.g: Stackdriver Logging).
 * [LabelMaker](tools/labelmaker) - A tool that reads key:value pairs from a json file and labels the running instance and all attached drives accordingly.

diff --git a/tools/gcp-arch-viz/README.md b/tools/gcp-arch-viz/README.md
@@ -0,0 +1,42 @@
+# GCP Architecture Visualizer
+
+This tool serves a single, simple purpose: visualizing a user's GCP architecture and environment, so that they have a dynamic and simple-to-use way to view their world in GCP.
+
+It does this by:
+
+* Ingesting a strictly-formatted CSV input file, which is generally output from a [Forseti Security](http://forsetisecurity.org) Inventory scan.
+* Dynamically building the parent->child tree of inventory resources in GCP.
+* Drawing an interactive tree structure using D3.js, containing useful GCP info and per-resource icons.
+
+
+## Usage
+
+The tool has been tested with [Forseti Security](http://forsetisecurity.org) 2.0, and can be used to draw out any CSV input (currently stored in gcp-data.csv) that conforms to the following schema:
+
+```
+id, resource_type, category, resource_id, parent_id, resource_name
+```
+
+CSV generation currently performed using Google Cloud SQL export from Forseti Security Inventory tables, using the query below:
+
+```
+SELECT id, resource_type, category, resource_id, parent_id, IFNULL(resource_data->>'$.displayName', '') as resource_data_displayname, IFNULL(resource_data->>'$.name', '') as resource_data_name FROM gcp_inventory WHERE inventory_index_id = (SELECT id FROM inventory_index ORDER BY completed_at_datetime DESC LIMIT 1) AND (category='resource') AND  (resource_type='organization' OR resource_type='project' OR resource_type='folder' OR resource_type='appengine_app' OR resource_type='kubernetes_cluster' OR resource_type='cloudsqlinstance');
+```
+
+Other useful queries:
+
+__Get id of latest inventory scan (timestamp):__
+```
+SELECT id FROM inventory_index ORDER BY completed_at_datetime DESC LIMIT 1;
+```
+
+__Check schema of gcp_inventory table (in case schema changes, and query needs to be updated):__
+```
+Describe forseti_security.gcp_inventory;
+```
+
+## Examples
+
+Fully functional example available [here](https://storage.googleapis.com/strike3-gcp-arch-viz/gcp-arch-viz.html). Looks like this:
+
+![gcp-arch-viz animation](https://storage.googleapis.com/strike3-gcp-arch-viz/gcp-arch-viz.gif)
diff --git a/tools/gcp-arch-viz/gcp-arch-viz.gif b/tools/gcp-arch-viz/gcp-arch-viz.gif
diff --git a/tools/gcp-arch-viz/gcp-arch-viz.html b/tools/gcp-arch-viz/gcp-arch-viz.html
@@ -0,0 +1,294 @@
+<!DOCTYPE html>
+<!--
+/*
+ * Copyright (C) 2018 Google Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not
+ * use this file except in compliance with the License. You may obtain a copy of
+ * the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+-->
+<meta charset="utf-8">
+<style>
+  /* set the CSS */
+
+  .node circle {
+    fill: #fff;
+    stroke: steelblue;
+    stroke-width: 3px;
+  }
+
+  .node text {
+    font: 12px sans-serif;
+  }
+
+  .node--internal text {
+    text-shadow: 0 1px 0 #fff, 0 -1px 0 #fff, 1px 0 0 #fff, -1px 0 0 #fff;
+  }
+
+  .link {
+    fill: none;
+    stroke: #ccc;
+    stroke-width: 2px;
+  }
+</style>
+
+<body>
+  <script src="//d3js.org/d3.v5.min.js"></script>
+  <script>
+
+    var csv_data;
+    var treeData, margin, tree, table, svg, g;
+
+    // FILENAME is static, but add a random variable to the end to prevent browser caching,
+    // aka - "I changed the content of the file, but why isn't that stuff drawing?!"
+    var FILENAME = "gcp-data.csv" + "?nocache=" + Date.now(); 
+
+    var FILE_LOAD_START_TIME = new Date().toLocaleTimeString();
+    console.log ("Loading file: " + FILENAME + ", start time: " + FILE_LOAD_START_TIME);
+
+    d3.text(FILENAME).then(function(text) {
+      var FILE_LOAD_END_TIME = new Date().toLocaleTimeString();
+      console.log ("CSV loaded: " + FILE_LOAD_END_TIME);
+
+      csv_data = text;
+      console.log(csv_data);
+
+      i = 0, duration = 750;
+
+      // set the dimensions and margins of the diagram
+      var margin = { top: 20, right: 45, bottom: 30, left: 150 };
+
+      // Use the parsed tree data to dynamically create height & width
+      var width = 1500 - margin.left - margin.right,
+        height = 540 - margin.top - margin.bottom;
+
+      tree = d3.tree().size([height, width]);
+
+      table = d3.csvParseRows(csv_data, function (d, i) {
+        return {
+          // NOTE: This stuff is VERY tightly coupled to the format of the inventory data export. 
+          //       Because there are no headers in the CSV export file, the column order is
+          //       extremely important and needs to be carefully paid attention to.        
+          id: d[0],
+          resource_type: d[1],
+          category: d[2],
+          resource_id: d[3],
+          parent_id: (d[1] == 'organization' ? "" : d[4]),
+          resource_name: (d[5] != '' ? d[5] : d[6]),
+          image: getImageURL(d[1])
+        };
+      });
+
+      console.log(table);
+
+      treeData = d3.stratify()
+        .id(function (d) { return d.id; })
+        .parentId(function (d) { return d.parent_id; })
+        (table);
+
+      // assign the name to each node
+      treeData.each(function (d) {
+        d.name = d.data.resource_name;
+      });
+
+      // treeData is the root of the tree,
+      // and the tree has all the data we need in it now.
+      // let's draw that thing...
+      console.log(treeData);
+
+      // append the svg object to the body of the page
+      // appends a 'group' element to 'svg'
+      // moves the 'group' element to the top left margin
+      svg = d3.select("body").append("svg")
+        .attr("width", width + margin.left + margin.right)
+        .attr("height", height + margin.top + margin.bottom);
+
+      g = svg.append("g")
+        .attr("transform",
+          "translate(" + margin.left + "," + margin.top + ")");
+
+
+      // Collapse after the second level
+      // TODO(mzinni):  enable this after folders & projects have color-coded indicators 
+      //                of children/no-children
+      treeData.children.forEach(collapse);
+      update(treeData);
+    });
+
+    var i = 0;
+    function update(source) {
+      tree(treeData);
+
+      treeData.each(function(d) { d.y = d.depth * 180; });
+
+      var node = g.selectAll('.node')
+        .data(treeData.descendants(), function(d) { return d.id || (d.id = ++i); });
+
+      var nodeEnter = node
+        .enter()
+        .append("g")
+        .attr("class", "node")
+        .attr("transform", function(d) { return "translate(" + source.y + "," + source.x + ")"; })
+        .on("click", function(d) {
+          toggle(d);
+          update(d);
+        });
+
+      nodeEnter.append("circle")
+        .attr("r", 22)
+        .style("fill", function(d) { return d._children ? "lightsteelblue" : "#fff"; })
+        .style("fill-opacity", function(d) { return d._children ? 1 : 0; })
+        .style("stroke", "white")
+        .style("stroke-opacity", 0);
+
+      // adds the image to the node
+      nodeEnter.append("image")
+        .attr("xlink:href", function (d) { return d.data.image; })
+        .attr("x", function (d) { return -16; })
+        .attr("y", function (d) { return -16; })
+        .attr("height", 35)
+        .attr("width", 35);
+
+      // adds the text to the node
+      nodeEnter.append("text")
+        .attr("x", function (d) { return d.children ? -25 : 25; })
+        .attr("dy", ".35em")
+        .style("text-anchor", function (d) {
+          return d.children ? "end" : "start";
+        })
+        .text(function (d) { return d.name; });
+
+      var nodeUpdate = nodeEnter.merge(node);
+      var ANIMATION_DURATION_MS = 500;
+
+      nodeUpdate.transition()
+        .duration(ANIMATION_DURATION_MS)
+        .attr("transform", function(d) { return "translate(" + d.y + "," + d.x + ")"; });
+
+      nodeUpdate.select("circle")
+      .attr("r", 22)
+      .style("fill", function(d) { return d._children ? "lightsteelblue" : "#fff"; })
+      .style("fill-opacity", function(d) { return d._children ? 1 : 0; })
+      .style("stroke-opacity", 0);
+
+      nodeUpdate.select("text")
+        .style("fill-opacity", 1);
+
+      var nodeExit = node
+        .exit()
+        .transition()
+        .duration(ANIMATION_DURATION_MS)
+        .attr("transform", function(d) { return "translate(" + source.y + "," + source.x + ")"; })
+        .remove();
+
+      nodeExit.select("circle")
+        .attr("r", 1e-6);
+
+      nodeExit.select("text")
+        .style("fill-opacity", 1e-6);
+
+      var link = g.selectAll(".link")
+        .data(treeData.links(), function(d) { return d.target.id; });
+
+      var linkEnter = link.enter().insert('path', "g")
+        .attr("class", "link")
+        .attr("d", d3.linkHorizontal()
+          .x(function(d) { return source.y; })
+          .y(function(d) { return source.x; }));
+
+      var linkUpdate = linkEnter.merge(link);
+      linkUpdate
+        .transition()
+        .duration(ANIMATION_DURATION_MS)
+        .attr("d", d3.linkHorizontal()
+          .x(function(d) { return d.y; })
+          .y(function(d) { return d.x; }));
+
+      link
+        .exit()
+        .transition()
+        .duration(ANIMATION_DURATION_MS)
+        .attr("d", d3.linkHorizontal()
+          .x(function(d) { return source.y; })
+          .y(function(d) { return source.x; })
+        )
+        .remove();
+
+      node.each(function(d) {
+        d.x0 = d.x;
+        d.y0 = d.y;
+      });
+    }
+
+    // Collapse the node and all it's children
+    function collapse(node) {
+      if (node.children) {
+        node._children = node.children
+        node._children.forEach(collapse)
+        node.children = null
+      }
+    }
+
+    // Toggle children on click.
+    function toggle(node) {
+      if (node.children) {
+        node._children = node.children;
+        node.children = null;
+      } else {
+        node.children = node._children;
+        node._children = null;
+      }
+      update(node);
+    }
+
+    function getImageURL(resource_type) {
+      var URL_BASE = "https://storage.googleapis.com/mps-storage/mzinni/external/gcp-arch-viz-images/";
+      var imageFilename = "gcp-logo.png";
+
+      switch (resource_type) {
+        case "organization":
+          imageFilename = "cloud_logo.png";
+          break;
+
+        case "folder":
+          imageFilename = "folder_logo.png";
+          break;
+
+        case "project":
+          imageFilename = "project_logo.png";
+          break;
+
+        case "appengine_app":
+          imageFilename = "app_engine_logo.png";
+          break;
+
+        case "kubernetes_cluster":
+          imageFilename = "container_engine_logo.png";
+          break;
+
+        case "cloudsqlinstance":
+          imageFilename = "cloud_sql_logo.png";
+          break;
+
+        case "bucket":
+          imageFilename = "cloud_storage_logo.png";
+          break;
+
+        default:
+          imageFilename = "gcp-logo.png";
+          break;
+      }
+
+      return URL_BASE + imageFilename;
+    }
+  </script>
+</body>
diff --git a/tools/gcp-arch-viz/gcp-data.csv b/tools/gcp-arch-viz/gcp-data.csv
@@ -0,0 +1,18 @@
+1,"organization","resource","180703686097",,"strike3software.com","organizations/180703686097"
+2,"folder","resource","1037858470618",1,"vendors","folders/1037858470618"
+3,"folder","resource","447738853093",1,"products","folders/447738853093"
+4,"folder","resource","259115136721",1,"demo","folders/259115136721"
+5,"folder","resource","776007621624",1,"migrate","folders/776007621624"
+6,"folder","resource","93375575187",1,"experimental","folders/93375575187"
+7,"folder","resource","830051761309",1,"admin","folders/830051761309"
+8,"folder","resource","98635841550",1,"acquisitions","folders/98635841550"
+9,"folder","resource","898988358812",6,"mikez","folders/898988358812"
+10,"folder","resource","45281402264",9,"forseti","folders/45281402264"
+35,"project","resource","mikez-test-project",5,"","mikez-test-project"
+50,"project","resource","strike3-helloworld",4,"","strike3-helloworld"
+55,"project","resource","strike3-billing-project",5,"","strike3-billing-project"
+60,"project","resource","mikez-strike3project-test",5,"","mikez-strike3project-test"
+73,"project","resource","strike3-forseti",10,"","strike3-forseti"
+252,"cloudsqlinstance","resource","forseti-server-db-ff5789a",73,"","forseti-server-db-ff5789a"
+354,"appengine_app","resource","11804977894556867746",50,"","apps/strike3-helloworld"
+361,"appengine_app","resource","1330910310474806251",73,"","apps/strike3-forseti"