Skip to content

Commit

Permalink
Restructure the dashboards to leverage mosaicLayout (#18)
Browse files Browse the repository at this point in the history
This layout is more cumbersome because of the requirement to explicitly
lay things out, however, this change introduces a number of helper
modules to make this a bit more tolerable.

At a high-level, "sections" (our own intermediate concept) produce
groupings of tiles laid out relative to `(0, 0)` with a width of
`module.width.size` (see below).

The modules in `./dashboard/sections` group our "widgets" into
arrangements of "tiles" (e.g. logs, http, resources) with several
special helper modules:
1. `width`: This module exists to define a global constant for the
dashboard width,
1. `collapsible`: This wraps a list of tiles in a "collapsible" region
with a title (it automagically figures out the bounding box, to avoid
human error here),
1. `layout`: This takes a list of "sections" (defined above) and rebases
them so that the `yPos` of each section starts after the preceding
section.


This also incorporates a change to give our networking module's DNS
constructs unique names, so that we can provision multiple distinct
private networks (I hit this creating a small example to test with, and
it conflicted with my dev environment).

Signed-off-by: Matt Moore <[email protected]>
  • Loading branch information
mattmoor authored Dec 20, 2023
1 parent 28937b3 commit 763b08d
Show file tree
Hide file tree
Showing 12 changed files with 328 additions and 146 deletions.
10 changes: 4 additions & 6 deletions dashboard/job/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,12 +42,10 @@ No requirements.

| Name | Source | Version |
|------|--------|---------|
| <a name="module_cpu_utilization"></a> [cpu\_utilization](#module\_cpu\_utilization) | ../widgets/xy | n/a |
| <a name="module_logs"></a> [logs](#module\_logs) | ../widgets/logs | n/a |
| <a name="module_memory_utilization"></a> [memory\_utilization](#module\_memory\_utilization) | ../widgets/xy | n/a |
| <a name="module_received_bytes"></a> [received\_bytes](#module\_received\_bytes) | ../widgets/xy | n/a |
| <a name="module_sent_bytes"></a> [sent\_bytes](#module\_sent\_bytes) | ../widgets/xy | n/a |
| <a name="module_startup_latency"></a> [startup\_latency](#module\_startup\_latency) | ../widgets/xy | n/a |
| <a name="module_layout"></a> [layout](#module\_layout) | ../sections/layout | n/a |
| <a name="module_logs"></a> [logs](#module\_logs) | ../sections/logs | n/a |
| <a name="module_resources"></a> [resources](#module\_resources) | ../sections/resources | n/a |
| <a name="module_width"></a> [width](#module\_width) | ../sections/width | n/a |

## Resources

Expand Down
70 changes: 19 additions & 51 deletions dashboard/job/dashboard.tf
Original file line number Diff line number Diff line change
@@ -1,49 +1,23 @@
locals { common_filter = ["resource.type=\"cloud_run_job\""] }

module "logs" {
source = "../widgets/logs"
title = "Service Logs"
filter = local.common_filter
}

module "cpu_utilization" {
source = "../widgets/xy"
title = "CPU utilization"
filter = concat(local.common_filter, ["metric.type=\"run.googleapis.com/container/cpu/utilizations\""])
primary_align = "ALIGN_DELTA"
primary_reduce = "REDUCE_MEAN"
source = "../sections/logs"
title = "Job Logs"
filter = ["resource.type=\"cloud_run_job\""]
}

module "memory_utilization" {
source = "../widgets/xy"
title = "Memory utilization"
filter = concat(local.common_filter, ["metric.type=\"run.googleapis.com/container/memory/utilizations\""])
primary_align = "ALIGN_DELTA"
primary_reduce = "REDUCE_MEAN"
module "resources" {
source = "../sections/resources"
title = "Resources"
filter = ["resource.type=\"cloud_run_job\""]
}

module "startup_latency" {
source = "../widgets/xy"
title = "Startup latency"
filter = concat(local.common_filter, ["metric.type=\"run.googleapis.com/container/startup_latencies\""])
primary_align = "ALIGN_DELTA"
primary_reduce = "REDUCE_MEAN"
}
module "width" { source = "../sections/width" }

module "sent_bytes" {
source = "../widgets/xy"
title = "Sent bytes"
filter = concat(local.common_filter, ["metric.type=\"run.googleapis.com/container/network/sent_bytes_count\""])
primary_align = "ALIGN_MEAN"
primary_reduce = "REDUCE_NONE"
}

module "received_bytes" {
source = "../widgets/xy"
title = "Received bytes"
filter = concat(local.common_filter, ["metric.type=\"run.googleapis.com/container/network/received_bytes_count\""])
primary_align = "ALIGN_MEAN"
primary_reduce = "REDUCE_NONE"
module "layout" {
source = "../sections/layout"
sections = [
module.logs.section,
module.resources.section,
]
}

resource "google_monitoring_dashboard" "dashboard" {
Expand All @@ -54,17 +28,11 @@ resource "google_monitoring_dashboard" "dashboard" {
stringValue = var.job_name
labelKey = "job_name"
}]
// https://cloud.google.com/monitoring/api/ref_v3/rest/v1/projects.dashboards#GridLayout
gridLayout = {
columns = 3
widgets = [
module.logs.widget,
module.cpu_utilization.widget,
module.memory_utilization.widget,
module.startup_latency.widget,
module.sent_bytes.widget,
module.received_bytes.widget,
]

// https://cloud.google.com/monitoring/api/ref_v3/rest/v1/projects.dashboards#mosaiclayout
mosaicLayout = {
columns = module.width.size
tiles = module.layout.tiles,
}
})
}
24 changes: 24 additions & 0 deletions dashboard/sections/collapsible/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
variable "title" { type = string }
variable "tiles" {}
variable "collapsed" { default = false }

locals {
start_row = min([for s in var.tiles : s.yPos]...)
}

module "width" { source = "../width" }

output "section" {
value = concat([{
yPos = local.start_row
xPos = 0,
height = max([for s in var.tiles : s.yPos + s.height - local.start_row]...),
width = module.width.size,
widget = {
title = var.title
collapsibleGroup = {
collapsed = var.collapsed
}
},
}], var.tiles)
}
60 changes: 60 additions & 0 deletions dashboard/sections/http/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
variable "title" { type = string }
variable "filter" { type = list(string) }
variable "collapsed" { default = false }

module "width" { source = "../width" }

module "request_count" {
source = "../../widgets/xy"
title = "Request count"
filter = concat(var.filter, ["metric.type=\"run.googleapis.com/request_count\""])
group_by_fields = ["metric.label.\"response_code_class\""]
primary_align = "ALIGN_RATE"
primary_reduce = "REDUCE_NONE"
secondary_align = "ALIGN_NONE"
secondary_reduce = "REDUCE_SUM"
}

module "incoming_latency" {
source = "../../widgets/latency"
title = "Incoming request latency"
filter = concat(var.filter, ["metric.type=\"run.googleapis.com/request_latencies\""])
}

// TODO(mattmoor): output HTTP charts.

locals {
columns = 2
unit = module.width.size / local.columns

// https://www.terraform.io/language/functions/range
// N columns, unit width each ([0, unit, 2 * unit, ...])
col = range(0, local.columns * local.unit, local.unit)

tiles = [{
yPos = 0
xPos = local.col[0],
height = local.unit,
width = local.unit,
widget = module.request_count.widget,
},
{
yPos = 0
xPos = local.col[1],
height = local.unit,
width = local.unit,
widget = module.incoming_latency.widget,
}]
}

module "collapsible" {
source = "../collapsible"

title = var.title
tiles = local.tiles
collapsed = var.collapsed
}

output "section" {
value = module.collapsible.section
}
28 changes: 28 additions & 0 deletions dashboard/sections/layout/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
variable "sections" {}

module "width" { source = "../width" }

locals {
// The maximum height of a tile in each section.
max_heights = [for s in var.sections : max([for t in s : t.yPos + t.height]...)]

// The sum of the maximum tile heights in all of the sections prior to this one.
// Note: sum doesn't work on an empty list, so we concatenate with 0 for the base case.
sum_heights = [for s in var.sections : sum(concat([0], slice(local.max_heights, 0, index(var.sections, s))))]

// Rebase the yPos of each tile in each section to be relative to the top of
// the section, which starts after the topmost tile of the preceding section.
rebased = [for s in var.sections : [
for t in s : {
yPos = t.yPos + local.sum_heights[index(var.sections, s)]
xPos = t.xPos
height = t.height
width = t.width
widget = t.widget
}]
]
}

output "tiles" {
value = concat(local.rebased...)
}
33 changes: 33 additions & 0 deletions dashboard/sections/logs/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
variable "title" { type = string }
variable "filter" { type = list(string) }
variable "collapsed" { default = true }

module "width" { source = "../width" }

module "logs" {
source = "../../widgets/logs"
title = var.title
filter = var.filter
}

locals {
tiles = [{
yPos = 0
xPos = 0,
height = module.width.size,
width = module.width.size,
widget = module.logs.widget,
}]
}

module "collapsible" {
source = "../collapsible"

title = var.title
tiles = local.tiles
collapsed = var.collapsed
}

output "section" {
value = module.collapsible.section
}
119 changes: 119 additions & 0 deletions dashboard/sections/resources/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
variable "title" { type = string }
variable "filter" { type = list(string) }
variable "collapsed" { default = false }

module "width" { source = "../width" }

module "instance_count" {
source = "../../widgets/xy"
title = "Instance count + revisions"
filter = concat(var.filter, ["metric.type=\"run.googleapis.com/container/instance_count\""])
group_by_fields = ["resource.label.\"revision_name\""]
primary_align = "ALIGN_MEAN"
primary_reduce = "REDUCE_SUM"
plot_type = "STACKED_AREA"
}

module "cpu_utilization" {
source = "../../widgets/xy"
title = "CPU utilization"
filter = concat(var.filter, ["metric.type=\"run.googleapis.com/container/cpu/utilizations\""])
primary_align = "ALIGN_DELTA"
primary_reduce = "REDUCE_MEAN"
}

module "memory_utilization" {
source = "../../widgets/xy"
title = "Memory utilization"
filter = concat(var.filter, ["metric.type=\"run.googleapis.com/container/memory/utilizations\""])
primary_align = "ALIGN_DELTA"
primary_reduce = "REDUCE_MEAN"
}

module "startup_latency" {
source = "../../widgets/xy"
title = "Startup latency"
filter = concat(var.filter, ["metric.type=\"run.googleapis.com/container/startup_latencies\""])
primary_align = "ALIGN_DELTA"
primary_reduce = "REDUCE_MEAN"
}

module "sent_bytes" {
source = "../../widgets/xy"
title = "Sent bytes"
filter = concat(var.filter, ["metric.type=\"run.googleapis.com/container/network/sent_bytes_count\""])
primary_align = "ALIGN_MEAN"
primary_reduce = "REDUCE_NONE"
}

module "received_bytes" {
source = "../../widgets/xy"
title = "Received bytes"
filter = concat(var.filter, ["metric.type=\"run.googleapis.com/container/network/received_bytes_count\""])
primary_align = "ALIGN_MEAN"
primary_reduce = "REDUCE_NONE"
}

locals {
columns = 3
unit = module.width.size / local.columns

// https://www.terraform.io/language/functions/range
// N columns, unit width each ([0, unit, 2 * unit, ...])
col = range(0, local.columns * local.unit, local.unit)

tiles = [{
yPos = 0,
xPos = local.col[0],
height = local.unit,
width = local.unit,
widget = module.cpu_utilization.widget,
},
{
yPos = 0,
xPos = local.col[1],
height = local.unit,
width = local.unit,
widget = module.memory_utilization.widget,
},
{
yPos = 0,
xPos = local.col[2],
height = local.unit,
width = local.unit,
widget = module.instance_count.widget,
},
{
yPos = local.unit,
xPos = local.col[0],
height = local.unit,
width = local.unit,
widget = module.startup_latency.widget,
},
{
yPos = local.unit,
xPos = local.col[1],
height = local.unit,
width = local.unit,
widget = module.sent_bytes.widget,
},
{
yPos = local.unit,
xPos = local.col[2],
height = local.unit,
width = local.unit,
widget = module.received_bytes.widget,
}]
}

module "collapsible" {
source = "../collapsible"

title = var.title
tiles = local.tiles
collapsed = var.collapsed
}

output "section" {
value = module.collapsible.section
}
3 changes: 3 additions & 0 deletions dashboard/sections/width/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
output "size" {
value = 12
}
14 changes: 5 additions & 9 deletions dashboard/service/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,15 +52,11 @@ No requirements.

| Name | Source | Version |
|------|--------|---------|
| <a name="module_cpu_utilization"></a> [cpu\_utilization](#module\_cpu\_utilization) | ../widgets/xy | n/a |
| <a name="module_incoming_latency"></a> [incoming\_latency](#module\_incoming\_latency) | ../widgets/latency | n/a |
| <a name="module_instance_count"></a> [instance\_count](#module\_instance\_count) | ../widgets/xy | n/a |
| <a name="module_logs"></a> [logs](#module\_logs) | ../widgets/logs | n/a |
| <a name="module_memory_utilization"></a> [memory\_utilization](#module\_memory\_utilization) | ../widgets/xy | n/a |
| <a name="module_received_bytes"></a> [received\_bytes](#module\_received\_bytes) | ../widgets/xy | n/a |
| <a name="module_request_count"></a> [request\_count](#module\_request\_count) | ../widgets/xy | n/a |
| <a name="module_sent_bytes"></a> [sent\_bytes](#module\_sent\_bytes) | ../widgets/xy | n/a |
| <a name="module_startup_latency"></a> [startup\_latency](#module\_startup\_latency) | ../widgets/xy | n/a |
| <a name="module_http"></a> [http](#module\_http) | ../sections/http | n/a |
| <a name="module_layout"></a> [layout](#module\_layout) | ../sections/layout | n/a |
| <a name="module_logs"></a> [logs](#module\_logs) | ../sections/logs | n/a |
| <a name="module_resources"></a> [resources](#module\_resources) | ../sections/resources | n/a |
| <a name="module_width"></a> [width](#module\_width) | ../sections/width | n/a |

## Resources

Expand Down
Loading

0 comments on commit 763b08d

Please sign in to comment.