From 264c699458c571441b8e23088fa95e0c28b92044 Mon Sep 17 00:00:00 2001 From: Nicholas Shaddox Date: Mon, 18 Sep 2023 09:24:13 -0700 Subject: [PATCH 1/8] fix: link in README.md goes to website now --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 6526367..69c61ea 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # openo11y.io -Welcome to Open O11y! This repository hosts the website files for [openo11y.dev](openo11y.dev), +Welcome to Open O11y! This repository hosts the website files for [openo11y.dev](https://openo11y.dev), an open-source public website with guidance and information on O11y. If you're looking to contribute, please see the [contributing](./CONTRIBUTING.md) documentation. From 59f38c115100cffb9606fd5c27298b0f6a632654 Mon Sep 17 00:00:00 2001 From: Nicholas Shaddox Date: Mon, 18 Sep 2023 09:26:16 -0700 Subject: [PATCH 2/8] fix: link to contributing.md works now --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 69c61ea..2d0df28 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ Welcome to Open O11y! This repository hosts the website files for [openo11y.dev](https://openo11y.dev), an open-source public website with guidance and information on O11y. -If you're looking to contribute, please see the [contributing](./CONTRIBUTING.md) documentation. +If you're looking to contribute, please see the [contributing](./docs/contributing.md) documentation. # Local Development From 57bfb596231a6510f82f9bb552bf8a72c0445e77 Mon Sep 17 00:00:00 2001 From: Nicholas Shaddox Date: Mon, 18 Sep 2023 12:25:55 -0700 Subject: [PATCH 3/8] add: delivery metrics markdown page --- docs/delivery-metrics.md | 256 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 256 insertions(+) create mode 100644 docs/delivery-metrics.md diff --git a/docs/delivery-metrics.md b/docs/delivery-metrics.md new file mode 100644 index 0000000..b052844 --- /dev/null +++ b/docs/delivery-metrics.md @@ -0,0 +1,256 @@ +# Delivery Metrics + +## Executive Summary: + +Below defines the approach for measuring product teams. The measuring approach will be uniform across product delivery. There is no one single metric that rules them all. Focusing on one metric will cause over-rotation in one direction and will negatively impact the others. The outcome is not a “big brother is watching” nor is it a way to measure individual performance rather it’s a mechanism to provide product **teams** signals (quantitative and qualitative) as to how they are delivering their product against an opinionated set of engineering practices and principles (sensible defaults) as well as a measure for overall team culture and joy. + +### Assumptions and Approach + +- Metrics will be gathered real-time or as near real-time as possible and will not impede delivery. (i.e. if the metrics collection system is down product delivery will not be impacted.) + +## 4 Key Metrics + +These metrics have become the industry defacto for measuring teams and we at Liatrio agree these are a meaningful way to measure teams. However, these 4 key metrics are lagging indicators on team performance and we believe in order to drive change the engineering practices and principles need to be followed to impact change. We are leveraging Sensible Defaults and Joy Index to provide teams a mechanism to gauge how they are performing. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MetricsWhat is MeasuredConsiderations
+ Deployment Frequency
+ Source(s): CircleCI, DataDog		 + Deployment frequency, to production, is a good indicator for how small of batches teams are working.

+ IN PROGRESS - Implemented but the solution is not sustainable. + 		+
    +
  • + For teams who are not fully in control of their production deployments showing deploy frequency to the lower environments is beneficial (Dev, QA, etc.) +
    • +
  • + Teams should set a minimum target a production deployments per day. +
    • +
+
Lead Time for Changes + The amount of time it takes a commit to get into production +
Mean Time to Restore Service (MTTR) + How long it takes an organization to recover from a failure in production +
Change Failure Rate + The percentage of deployments causing a failure in production +
+ +## Sensible Defaults + +The sensible defaults are the engineering practices & standards for product delivery. The below metrics are signals for product teams to see how they are tracking against the engineering practices. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Sensible DefaultsWhat is MeasuredConsiderations
+ Trunk Based Development
+ Source(s): GitHub Insights		 +
    +
  • + DONE Number of active branches - should be far less than the number of engineers on the team. +
    • + Once a change is merged into main delete topic branches locally and on remote. +
    +
    • +
  • + DONE Age of open branches - other than main - branch life should be measured in minutes and hours not days. +
    • +
  • + DONE PR Duration - life cycle should be measured in minutes and hours not days +
    • +
+ 		+
    +
  • + How to leverage network insights to show commits to main over time like GitHub Insights does? +
    • +
+
+ Small Stories
+ Source(s): JIRA + 		+ Cycle time in JIRA from In Progress (work started) to Done (work complete) - start out by targeting 1 story per engineer per business day (3 days max) until you find the right pace for your team. +
    +
  • + Track average and standard deviation of cycle time. +
    • +
  • + Log the WIP +
    • +
+
+ Team Ownership of Quality
+ Source(s): CodeClimate + 		+ Unit Testing (TDD) +
    +
  • + DONE Code Climate - Unit Test code coverage - target 80% for net new code +
    • +
  • + Cyclomatic Complexity - I’m not sure this is being calculated by CodeClimate, rather this is more of a feature of SonarQube +
    • +
  • + Technical Debt Ratio +
    • +
+ 		+
  • + For existing code bases (mono-repos) where multiple teams are pushing code changes, start by measuring and gating on net new code changes. +
+
+ Build in Security
+ Source(s): Synk + 		+ Snyk - Number of vulnerabilities over time - Critical & High. +
    +
  • MTTR on Critical and High - Identified to remediated
    • +
  • Critical should be resolved in ~ 14 days
    • +
  • High should be resolved in 30 - 45 days.
    • +
  • Date of discovery of CVE not date of identified by Snyk
    • +
  • Fixed vs opened ratios of issues.
    • +
  • Running summary from the last 30 days, fixed vulns, (newly) reported, ignored (what are they choosing not to fix)
    • +
+
+ Pair Programming
+ Source(s): GitHub + 		+
    +
  • Co-Authored commits
    • +
  • Huddles
    • +
  • Open channel communication
    • +
+
+ Automated Deployment
+ Source(s): CircleCI + 		+
    + From pushing a code change, an automated build process provides feedback that the build is production-deployable in minutes. +
  • + Pipeline MTTR - when you have a failed build what’s the time to recover +
    • + The main branch is your production candidate branch and should be in good running order to facilitate frequent production deployments. +
    +
    • +
+
+ Fast Automated Builds
+ Source(s): CircleCI + 		+
    +
  • + From pushing a code change, an automated build process provides feedback that the build is production-deployable in minutes. +
    • +
  • + Pipeline Duration by Stage - Can be pulled from CircleCI api target < 10 from commit to deploy anything more is not fast feedback. +
    • +
+
+ Build for Production
+ Source(s): DataDog? + 		+
    +
  • + Definition of done for a feature includes at minimum: +
      +
    • + ensuring that the application is observable and supportable in production +
      • +
    • + logs, traces, and metrics are available and useful +
      • +
    • + monitoring and alerts are configured +
      • +
    • + on-call team members are prepared to respond quickly to any failures or outages (e.g. production access, runbooks in place) +
      • +
    +
    • +
+
+ + +| Sensible Defaults | What is Measured | Considerations | +|:---|:---|:---| +| **Trunk Based Development**
Source(s): GitHub Insights |
  • DONE Number of active branches - should be far less than the number of engineers on the team.
    • Once a change is merged into main delete topic branches locally and on remote.
  • DONE Age of open branches - other than main - branch life should be measured in minutes and hours not days.
  • DONE PR Duration - life cycle should be measured in minutes and hours not days
|
  • How to leverage network insights to show commits to main over time like GitHub Insights does?
| +| **Small Stories**
Source(s): JIRA |
  • Cycle time in JIRA from In Progress (work started) to Done (work complete) - start out by targeting 1 story per engineer per business day (3 days max) until you find the right pace for your team.
  • Track average and standard deviation of cycle time.
  • Log the WIP
| | +| **Team Ownership of Quality**
Source(s): CodeClimate |
  • Unit Testing (TDD)
  • DONE Code Climate - Unit Test code coverage - target 80% for net new code
  • Cyclomatic Complexity - I’m not sure this is being calculated by CodeClimate, rather this is more of a feature of SonarQube
  • Technical Debt Ratio
|
  • For existing code bases (mono-repos) where multiple teams are pushing code changes, start by measuring and gating on net new code changes.
| +| **Build in Security**
Source(s): Synk |
  • Snyk - Number of vulnerabilities over time - Critical & High.
  • MTTR on Critical and High - Identified to remediated
  • Critical should be resolved in ~ 14 days
  • High should be resolved in 30 - 45 days.
  • Date of discovery of CVE not date of identified by Snyk
  • Fixed vs opened ratios of issues.
  • Running summary from the last 30 days, fixed vulns, (newly) reported, ignored (what are they choosing not to fix)
| | +| **Pair Programming**
Source(s): GitHub |
  • Co-Authored commits
  • Huddles
  • Open channel communication
| | +| **Automated Deployment**
Source(s): CircleCI |
  • From pushing a code change, an automated build process provides feedback that the build is production-deployable in minutes.
  • Pipeline MTTR - when you have a failed build what’s the time to recover
    • The main branch is your production candidate branch and should be in good running order to facilitate frequent production deployments.
| | +| **Fast Automated Builds**
Source(s): CircleCI |
  • From pushing a code change, an automated build process provides feedback that the build is production-deployable in minutes.
  • Pipeline Duration by Stage - Can be pulled from CircleCI api target < 10 from commit to deploy anything more is not fast feedback.
| | +| **Build for Production**
Source(s): DataDog? |
  • Definition of done for a feature includes at minimum:
    • ensuring that the application is observable and supportable in production
    • logs, traces, and metrics are available and useful
    • monitoring and alerts are configured
    • on-call team members are prepared to respond quickly to any failures or outages (e.g. production access, runbooks in place)
| | From 63a7b11e7e9241763c7f6f5dbc17105f167f2d0c Mon Sep 17 00:00:00 2001 From: Nicholas Shaddox Date: Mon, 18 Sep 2023 13:23:34 -0700 Subject: [PATCH 4/8] add: delivery metrics markdown page w/ newer info --- docs/delivery-metrics.md | 727 ++++++++++++++++++++++++++------------- 1 file changed, 487 insertions(+), 240 deletions(-) diff --git a/docs/delivery-metrics.md b/docs/delivery-metrics.md index b052844..23cda6d 100644 --- a/docs/delivery-metrics.md +++ b/docs/delivery-metrics.md @@ -1,256 +1,503 @@ -# Delivery Metrics + + + + + + +#### Our recommendation is to collect the following metrics for team effectiveness & productivity. + +**Delivery Metrics (Product & Squad level)** + +1. Lead Time - Tracks the time it takes for a code change to go from commit (branch or main) to production +2. Deployment Frequency - Measures how often code is deployed +3. Change Failure Rate* - Percentage of changes that result in failures after it is in production or released to end user +4. Minutes of Impact - Calculates the down time from a service disruption over set time + +**Engineering Metrics (Squad level)** + +5. Branch & PR Age - Evaluates development best practices on repos +6. Code Coverage - Provides percent of unit testing covering the lines of code +7. Code Quality - Assesses high or low quality of code as well as security related defects + +**Joy Index (all level)** + +8. Team & Leadership Joy Index - Benchmarks individual developer experience, satisfaction, and happiness + +**Product Metrics (Product level)** + +9. Business OKRs defined & measurable - Verify the squads have established high quality OKRs with clearly defined measures - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MetricsWhat is MeasuredConsiderations
- Deployment Frequency
- Source(s): CircleCI, DataDog		 - Deployment frequency, to production, is a good indicator for how small of batches teams are working.

- IN PROGRESS - Implemented but the solution is not sustainable. - 		-
    -
  • - For teams who are not fully in control of their production deployments showing deploy frequency to the lower environments is beneficial (Dev, QA, etc.) -
    • -
  • - Teams should set a minimum target a production deployments per day. -
    • -
-
Lead Time for Changes - The amount of time it takes a commit to get into production -
Mean Time to Restore Service (MTTR) - How long it takes an organization to recover from a failure in production -
Change Failure Rate - The percentage of deployments causing a failure in production -
+Theme + Metrics + Focus Relentlessly + Excel Technically + High Performing Team +
Delivery Metrics +

+(Product & Squad level) +

Lead Time + ✔️ + ✔️ + ✔️ +
Deployment Frequency + + ✔️ + ✔️ +
Change Failure Rate* + + ✔️ + ✔️ +
Minutes of Impact + + ✔️ + ✔️ +
Engineering Metrics +

+(Squad level) +

Branch & PR Age + + ✔️ + ✔️ +
Code Coverage + + ✔️ + ✔️ +
Code Quality + + ✔️ + ✔️ +
Joy Index + + + ✔️ +
Product Metrics (Product level) + Business OKRs defined & measurable + ✔️ + + ✔️ +
-## Sensible Defaults -The sensible defaults are the engineering practices & standards for product delivery. The below metrics are signals for product teams to see how they are tracking against the engineering practices. +_* Currently **Change Failure Rate** is not measurable both at Product or Squad level in AA_ + + +# Performance Pulse Check + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Sensible DefaultsWhat is MeasuredConsiderations
- Trunk Based Development
- Source(s): GitHub Insights		 -
    -
  • - DONE Number of active branches - should be far less than the number of engineers on the team. -
    • - Once a change is merged into main delete topic branches locally and on remote. -
    -
    • -
  • - DONE Age of open branches - other than main - branch life should be measured in minutes and hours not days. -
    • -
  • - DONE PR Duration - life cycle should be measured in minutes and hours not days -
    • -
- 		-
    -
  • - How to leverage network insights to show commits to main over time like GitHub Insights does? -
    • -
-
- Small Stories
- Source(s): JIRA - 		- Cycle time in JIRA from In Progress (work started) to Done (work complete) - start out by targeting 1 story per engineer per business day (3 days max) until you find the right pace for your team. -
    -
  • - Track average and standard deviation of cycle time. -
    • -
  • - Log the WIP -
    • -
-
- Team Ownership of Quality
- Source(s): CodeClimate - 		- Unit Testing (TDD) -
    -
  • - DONE Code Climate - Unit Test code coverage - target 80% for net new code -
    • -
  • - Cyclomatic Complexity - I’m not sure this is being calculated by CodeClimate, rather this is more of a feature of SonarQube -
    • -
  • - Technical Debt Ratio -
    • -
- 		-
  • - For existing code bases (mono-repos) where multiple teams are pushing code changes, start by measuring and gating on net new code changes. -
-
- Build in Security
- Source(s): Synk - 		- Snyk - Number of vulnerabilities over time - Critical & High. -
    -
  • MTTR on Critical and High - Identified to remediated
    • -
  • Critical should be resolved in ~ 14 days
    • -
  • High should be resolved in 30 - 45 days.
    • -
  • Date of discovery of CVE not date of identified by Snyk
    • -
  • Fixed vs opened ratios of issues.
    • -
  • Running summary from the last 30 days, fixed vulns, (newly) reported, ignored (what are they choosing not to fix)
    • -
-
- Pair Programming
- Source(s): GitHub - 		-
    -
  • Co-Authored commits
    • -
  • Huddles
    • -
  • Open channel communication
    • -
-
- Automated Deployment
- Source(s): CircleCI - 		-
    - From pushing a code change, an automated build process provides feedback that the build is production-deployable in minutes. -
  • - Pipeline MTTR - when you have a failed build what’s the time to recover -
    • - The main branch is your production candidate branch and should be in good running order to facilitate frequent production deployments. -
    -
    • -
-
- Fast Automated Builds
- Source(s): CircleCI - 		-
    -
  • - From pushing a code change, an automated build process provides feedback that the build is production-deployable in minutes. -
    • -
  • - Pipeline Duration by Stage - Can be pulled from CircleCI api target < 10 from commit to deploy anything more is not fast feedback. -
    • -
-
- Build for Production
- Source(s): DataDog? - 		-
    -
  • - Definition of done for a feature includes at minimum: -
      -
    • - ensuring that the application is observable and supportable in production -
      • -
    • - logs, traces, and metrics are available and useful -
      • -
    • - monitoring and alerts are configured -
      • -
    • - on-call team members are prepared to respond quickly to any failures or outages (e.g. production access, runbooks in place) -
      • -
    -
    • -
-
Metrics + Run + Walk + Crawl +
Lead Time + Less than 1 hour + Between one +

+week and +

+one month +

Between one +

+month and +

+six months +

Deployment Frequency + On Demand/Multiple times per day + Between once per day to once per week + Between 2 weeks and a month +
+ + + +
Change Failure Rate + 0-7% (<15 min) + 7-15% (1+ hours) + 16-30% (2+ hours) +
Minutes of Impact + Less than 15 minutes + More than 1 hour + More than 2 hours +
Branch & PR Age + Less than 1 day + Less than 3 days + 1 week to 1 month +
Code Coverage + 80-90%+ for existing & new code + 20-50% for existing code +

+80%+ for new code +

0-50% for existing code +

+80%+ for new code +

Code Quality +

+(vulnerabilities) +

0 critical +

+0 high +

+0 medium +

+0 low +

0 critical +

+ 0 high +

0 critical +
Joy Index + eNPS 30 to 80+ + eNPS -10 to 30+ + eNPS -30 to 10+ +
Business OKRs defined & measurable + Defined with clear and inspiring objectives that align with the company's vision + Defined but room for improvement regarding alignment with company's vision + Defined but challenging for squads to understand how their work contributes to the company's vision +
-| Sensible Defaults | What is Measured | Considerations | -|:---|:---|:---| -| **Trunk Based Development**
Source(s): GitHub Insights |
  • DONE Number of active branches - should be far less than the number of engineers on the team.
    • Once a change is merged into main delete topic branches locally and on remote.
  • DONE Age of open branches - other than main - branch life should be measured in minutes and hours not days.
  • DONE PR Duration - life cycle should be measured in minutes and hours not days
|
  • How to leverage network insights to show commits to main over time like GitHub Insights does?
| -| **Small Stories**
Source(s): JIRA |
  • Cycle time in JIRA from In Progress (work started) to Done (work complete) - start out by targeting 1 story per engineer per business day (3 days max) until you find the right pace for your team.
  • Track average and standard deviation of cycle time.
  • Log the WIP
| | -| **Team Ownership of Quality**
Source(s): CodeClimate |
  • Unit Testing (TDD)
  • DONE Code Climate - Unit Test code coverage - target 80% for net new code
  • Cyclomatic Complexity - I’m not sure this is being calculated by CodeClimate, rather this is more of a feature of SonarQube
  • Technical Debt Ratio
|
  • For existing code bases (mono-repos) where multiple teams are pushing code changes, start by measuring and gating on net new code changes.
| -| **Build in Security**
Source(s): Synk |
  • Snyk - Number of vulnerabilities over time - Critical & High.
  • MTTR on Critical and High - Identified to remediated
  • Critical should be resolved in ~ 14 days
  • High should be resolved in 30 - 45 days.
  • Date of discovery of CVE not date of identified by Snyk
  • Fixed vs opened ratios of issues.
  • Running summary from the last 30 days, fixed vulns, (newly) reported, ignored (what are they choosing not to fix)
| | -| **Pair Programming**
Source(s): GitHub |
  • Co-Authored commits
  • Huddles
  • Open channel communication
| | -| **Automated Deployment**
Source(s): CircleCI |
  • From pushing a code change, an automated build process provides feedback that the build is production-deployable in minutes.
  • Pipeline MTTR - when you have a failed build what’s the time to recover
    • The main branch is your production candidate branch and should be in good running order to facilitate frequent production deployments.
| | -| **Fast Automated Builds**
Source(s): CircleCI |
  • From pushing a code change, an automated build process provides feedback that the build is production-deployable in minutes.
  • Pipeline Duration by Stage - Can be pulled from CircleCI api target < 10 from commit to deploy anything more is not fast feedback.
| | -| **Build for Production**
Source(s): DataDog? |
  • Definition of done for a feature includes at minimum:
    • ensuring that the application is observable and supportable in production
    • logs, traces, and metrics are available and useful
    • monitoring and alerts are configured
    • on-call team members are prepared to respond quickly to any failures or outages (e.g. production access, runbooks in place)
| | + +# + + +# Why Metrics Matter + + +### Quantifiable Performance + + + +* Metrics provide objective measurements of squad performance, fostering a data-driven culture. They enable squads to track progress, set benchmarks, and identify areas for improvement. + + +### Continuous Improvement + + + +* By identifying bottlenecks and inefficiencies, metrics drive continuous improvement efforts. Squads can refine processes, optimize workflows, and enhance collaboration to accelerate delivery. + + +### Risk Mitigation + + + +* Lower change failure rates and shorter Minutes of Impact indicate increased software stability and reliability. This minimizes business risks associated with service disruptions and costly downtimes. + + +### Informed Decision-Making + + + +* Metrics facilitate informed decision-making by offering a comprehensive view of the software delivery pipeline. Squads can make strategic choices to align with business goals. + + +### Cross-Functional Collaboration + + + +* These metrics encourage collaboration between development, operations, and quality assurance teams. Silos are broken down, fostering a holistic approach to software delivery. + +Software powers innovation and competitive advantage and metrics have emerged as the compass guiding software delivery squads toward success. They provide a roadmap to accelerated delivery, enhanced quality, and collaborative excellence. By leveraging the insights gleaned from these metrics, organizations can transform their software delivery processes and stay at the forefront of their industries. + + +# + + +# Delivery Metrics (Product & Squad level) + + +## Lead Time + +Measures the time it takes for a code change to go from commit (branch or main) to production. Provides insights into workflow efficiency and bottlenecks. Shorter lead times indicate smoother processes and quicker value delivery. + + + +* _How to measure:_ Conduct squad level VSM to gather time code change goes from commit to production +* _Example: _Squad’s lead time is 509.15h (~64 days). Working time is 163.85h (~20 days) + + +## Deploy Frequency + +Measures how often code is deployed. Enables rapid iteration and faster time-to-market. Encourages small, incremental changes, reducing the risk of failures. + + + +* _How to measure:_ Divide the total number of deployments made in a given time period (e.g., a month) by the total number of days in that period +* _Example:_ If a squad deployed code 10 times in a month with 31 days, the deployment frequency would be 10/31 = 0.32 deployments per day + + +## Change Failure Rate + +Gauges the percentage of deployments that result in failures. Offers visibility into code quality and stability. Low failure rates signify robust testing and higher software reliability. + + + +* _How to measure:_ Percentage of changes that were made to code that then resulted in incident/rollback/or any type of prod failure. Calculated by counting the number of deployment failures and then dividing by number of total deployments. +* _Example:_ If your squad deployed five times this week and one of them resulted in a failure, your change failure rate is 20% + + +## Minutes of Impact (MoI) + +Calculates the down time from a service disruption over set time. Highlights the squad’s ability to detect and resolve issues swiftly. Shorter MoI reflects strong incident response and system resilience. + + + +* _How to measure:_ Measures time it takes for service to recover from failure. Calculated by tracking the average time between a bug report and the moment a bug fix is deployed. +* _Example: _If your squad had four incidents in a 40-hour workweek and spent one total hour on them (from alert to fix), your MoI for that week would be 15 minutes + + +# Engineering Metrics (Squad level) + + +## Branch & PR Age + +Evaluates development best practices on repos. + + + +* _How to measure: _Calculated by the overall average age of branches & pull requests across each repo +* _Example: _Number of commits a branch is behind or ahead of main. Hours or days a PR has existed before merging into main. + + +## Code Coverage + +Provides percent of unit testing covering the lines of code. Assesses the test performance. + + + +* _How to measure: _3rd party tooling which runs in your automated CI/CD builds +* _Example: _If the code were testing has a 100 lines of code and the number of lines of code that is actually validated in the same software is 50, then the code coverage percentage of this software will be 50% + + +## Code Quality + +Assesses high or low quality of code as well as security related defects. Identifies potential risks of compromising code and indicated where to fix software defects + + + +* _How to measure: _3rd party tooling which runs in your automated CI/CD builds +* _Example: _1 aspect of code quality is reusability. This is best measured by counting the number of interdependencies. The more there are, generally the less reusable the code. + + +## Joy Index + +Benchmarks individual developer experience, satisfaction, and happiness. + + + +* _How to measure:_ Employee net promoter score tracks employee engagement and measures the impact of their people and culture initiatives. The eNPS shows how workers feel about your organization. +* _Example: _Question -_ _“How likely are you to recommend us as a place to work for your family, friends, co-workers?” + + +# + + +# Product Metrics (Product level) + + +## Business OKRs defined & measurable + +OKRs are defined, with clear and inspiring Objectives that align with the company's overall mission and vision. Key Results are specific, measurable, and quantifiable, providing a clear path towards achieving the Objectives. OKRs are regularly reviewed and updated as needed, with a strong commitment to achieving them. + + + +* _How to measure:_ All team members understand the OKRs and how their work contributes to their achievement. The OKRs are logged in the company's OKR tracker. +* _Example:_ Baggage Services Product; Reduce the percentage in manual edits to optimizer suggested solutions from 86% to 43% by Q3, 23 inorder to increase reliance on the TLink optimizer + + From 7f511ccd35fe9938eba9b0379d4380a516f98217 Mon Sep 17 00:00:00 2001 From: Nicholas Shaddox Date: Mon, 18 Sep 2023 14:28:53 -0700 Subject: [PATCH 5/8] add: css styling for the new page --- docs/delivery-metrics.md | 114 +++++++++++++++++---------------------- docs/styles/table.css | 24 +++++++++ mkdocs.yml | 1 + 3 files changed, 75 insertions(+), 64 deletions(-) create mode 100644 docs/styles/table.css diff --git a/docs/delivery-metrics.md b/docs/delivery-metrics.md index 23cda6d..3e9333c 100644 --- a/docs/delivery-metrics.md +++ b/docs/delivery-metrics.md @@ -52,25 +52,11 @@ You have 8 H1 headings. You may want to use the "H1 -> H2" option to demote all ![alt_text](images/image2.png "image_tooltip") - - - - - - - - - - - - -
Team Effectiveness and Productivity Metrics -
August 2023 -
--> +# Team Effectiveness and Productivity Metrics -#### Our recommendation is to collect the following metrics for team effectiveness & productivity. +#### Our recommendation is to collect the following metrics for team effectiveness & productivity. **Delivery Metrics (Product & Squad level)** @@ -95,16 +81,15 @@ You have 8 H1 headings. You may want to use the "H1 -> H2" option to demote all - - - - - @@ -114,11 +99,11 @@ Theme - - - @@ -126,9 +111,9 @@ Theme - - @@ -136,9 +121,9 @@ Theme - - @@ -146,9 +131,9 @@ Theme - - @@ -160,9 +145,9 @@ Theme - - @@ -170,9 +155,9 @@ Theme - - @@ -180,9 +165,9 @@ Theme - - @@ -192,7 +177,7 @@ Theme - @@ -200,17 +185,19 @@ Theme - -
-Theme + Theme Metrics + Metrics Focus Relentlessly + Focus Relentlessly Excel Technically + Excel Technically High Performing Team + High Performing Team
Lead Time ✔️ + ✔️ ✔️ + ✔️ ✔️ + ✔️
✔️ + ✔️ ✔️ + ✔️
✔️ + ✔️ ✔️ + ✔️
✔️ + ✔️ ✔️ + ✔️
✔️ + ✔️ ✔️ + ✔️
✔️ + ✔️ ✔️ + ✔️
✔️ + ✔️ ✔️ + ✔️
✔️ + ✔️
Business OKRs defined & measurable ✔️ + ✔️ ✔️ + ✔️
+_Currently **Change Failure Rate** is not measurable both at Product or Squad level in AA_ -_* Currently **Change Failure Rate** is not measurable both at Product or Squad level in AA_ + +# # Performance Pulse Check @@ -218,13 +205,13 @@ _* Currently **Change Failure Rate** is not measurable both at Product or Squad - - - - @@ -352,42 +339,41 @@ six months
Metrics + Metrics Run + Run Walk + Walk Crawl + Crawl
- # -# Why Metrics Matter +## Why Metrics Matter -### Quantifiable Performance +#### Quantifiable Performance * Metrics provide objective measurements of squad performance, fostering a data-driven culture. They enable squads to track progress, set benchmarks, and identify areas for improvement. -### Continuous Improvement +#### Continuous Improvement * By identifying bottlenecks and inefficiencies, metrics drive continuous improvement efforts. Squads can refine processes, optimize workflows, and enhance collaboration to accelerate delivery. -### Risk Mitigation +#### Risk Mitigation * Lower change failure rates and shorter Minutes of Impact indicate increased software stability and reliability. This minimizes business risks associated with service disruptions and costly downtimes. -### Informed Decision-Making +#### Informed Decision-Making * Metrics facilitate informed decision-making by offering a comprehensive view of the software delivery pipeline. Squads can make strategic choices to align with business goals. -### Cross-Functional Collaboration +#### Cross-Functional Collaboration @@ -399,10 +385,10 @@ Software powers innovation and competitive advantage and metrics have emerged as # -# Delivery Metrics (Product & Squad level) +## Delivery Metrics (Product & Squad level) -## Lead Time +### **Lead Time** Measures the time it takes for a code change to go from commit (branch or main) to production. Provides insights into workflow efficiency and bottlenecks. Shorter lead times indicate smoother processes and quicker value delivery. @@ -412,7 +398,7 @@ Measures the time it takes for a code change to go from commit (branch or main) * _Example: _Squad’s lead time is 509.15h (~64 days). Working time is 163.85h (~20 days) -## Deploy Frequency +### **Deploy Frequency** Measures how often code is deployed. Enables rapid iteration and faster time-to-market. Encourages small, incremental changes, reducing the risk of failures. @@ -422,7 +408,7 @@ Measures how often code is deployed. Enables rapid iteration and faster time-to- * _Example:_ If a squad deployed code 10 times in a month with 31 days, the deployment frequency would be 10/31 = 0.32 deployments per day -## Change Failure Rate +### **Change Failure Rate** Gauges the percentage of deployments that result in failures. Offers visibility into code quality and stability. Low failure rates signify robust testing and higher software reliability. @@ -432,7 +418,7 @@ Gauges the percentage of deployments that result in failures. Offers visibility * _Example:_ If your squad deployed five times this week and one of them resulted in a failure, your change failure rate is 20% -## Minutes of Impact (MoI) +### **Minutes of Impact (MoI)** Calculates the down time from a service disruption over set time. Highlights the squad’s ability to detect and resolve issues swiftly. Shorter MoI reflects strong incident response and system resilience. @@ -442,10 +428,10 @@ Calculates the down time from a service disruption over set time. Highlights the * _Example: _If your squad had four incidents in a 40-hour workweek and spent one total hour on them (from alert to fix), your MoI for that week would be 15 minutes -# Engineering Metrics (Squad level) +## Engineering Metrics (Squad level) -## Branch & PR Age +### **Branch & PR Age** Evaluates development best practices on repos. @@ -455,7 +441,7 @@ Evaluates development best practices on repos. * _Example: _Number of commits a branch is behind or ahead of main. Hours or days a PR has existed before merging into main. -## Code Coverage +### **Code Coverage** Provides percent of unit testing covering the lines of code. Assesses the test performance. @@ -465,7 +451,7 @@ Provides percent of unit testing covering the lines of code. Assesses the test p * _Example: _If the code were testing has a 100 lines of code and the number of lines of code that is actually validated in the same software is 50, then the code coverage percentage of this software will be 50% -## Code Quality +### **Code Quality** Assesses high or low quality of code as well as security related defects. Identifies potential risks of compromising code and indicated where to fix software defects @@ -475,7 +461,7 @@ Assesses high or low quality of code as well as security related defects. Identi * _Example: _1 aspect of code quality is reusability. This is best measured by counting the number of interdependencies. The more there are, generally the less reusable the code. -## Joy Index +### **Joy Index** Benchmarks individual developer experience, satisfaction, and happiness. @@ -488,10 +474,10 @@ Benchmarks individual developer experience, satisfaction, and happiness. # -# Product Metrics (Product level) +## Product Metrics (Product level) -## Business OKRs defined & measurable +### **Business OKRs defined & measurable** OKRs are defined, with clear and inspiring Objectives that align with the company's overall mission and vision. Key Results are specific, measurable, and quantifiable, providing a clear path towards achieving the Objectives. OKRs are regularly reviewed and updated as needed, with a strong commitment to achieving them. diff --git a/docs/styles/table.css b/docs/styles/table.css new file mode 100644 index 0000000..815e434 --- /dev/null +++ b/docs/styles/table.css @@ -0,0 +1,24 @@ +.highlight-bg { + background-color: #2396bc; + color: #ffffff; + text-align: center; /* Doesn't work */ + width: 100%; /* make the background color span the width of the page */ + display: block; +} + +.highlight-hdr { + background-color: #117b9e; + text-align: center; /* Doesn't work */ + color: #ffffff; + vertical-align: middle; /* Doesn't work */ + font-weight: bold; /* make the text bold */ + font-size: 1.1em; /* adjust the value as needed to increase/decrease font size */ +} + +.highlight-cell{ + background-color: #40bae3; + text-align: center !important; /* Doesn't work */ + color: #000000; + vertical-align: middle; /* center the content vertically */ +} + diff --git a/mkdocs.yml b/mkdocs.yml index 6c8290c..73efa32 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -14,6 +14,7 @@ markdown_extensions: extra_css: - "styles/extra.css" + - "styles/table.css" nav: - 'Welcome to Open O11y': 'index.md' From 775caf36acbb788383f741880c72df60236887a5 Mon Sep 17 00:00:00 2001 From: Nicholas Shaddox Date: Wed, 20 Sep 2023 11:54:12 -0700 Subject: [PATCH 6/8] fix: format metrics page --- docs/delivery-metrics.md | 129 ++++++++++----------------------------- 1 file changed, 31 insertions(+), 98 deletions(-) diff --git a/docs/delivery-metrics.md b/docs/delivery-metrics.md index 3e9333c..c7bfc4f 100644 --- a/docs/delivery-metrics.md +++ b/docs/delivery-metrics.md @@ -1,83 +1,29 @@ - - - - - -# Team Effectiveness and Productivity Metrics - -#### Our recommendation is to collect the following metrics for team effectiveness & productivity. +### Recommended Metrics **Delivery Metrics (Product & Squad level)** -1. Lead Time - Tracks the time it takes for a code change to go from commit (branch or main) to production -2. Deployment Frequency - Measures how often code is deployed -3. Change Failure Rate* - Percentage of changes that result in failures after it is in production or released to end user -4. Minutes of Impact - Calculates the down time from a service disruption over set time +* Lead Time - Tracks the time it takes for a code change to go from commit (branch or main) to production +* Deployment Frequency - Measures how often code is deployed +* Change Failure Rate* - Percentage of changes that result in failures after it is in production or released to end user +* Minutes of Impact - Calculates the down time from a service disruption over set time **Engineering Metrics (Squad level)** -5. Branch & PR Age - Evaluates development best practices on repos -6. Code Coverage - Provides percent of unit testing covering the lines of code -7. Code Quality - Assesses high or low quality of code as well as security related defects +* Branch & PR Age - Evaluates development best practices on repos +* Code Coverage - Provides percent of unit testing covering the lines of code +* Code Quality - Assesses high or low quality of code as well as security related defects **Joy Index (all level)** -8. Team & Leadership Joy Index - Benchmarks individual developer experience, satisfaction, and happiness +* Team & Leadership Joy Index - Benchmarks individual developer experience, satisfaction, and happiness **Product Metrics (Product level)** -9. Business OKRs defined & measurable - Verify the squads have established high quality OKRs with clearly defined measures +* Business OKRs defined & measurable - Verify the squads have established high quality OKRs with clearly defined measures @@ -197,10 +143,7 @@ You have 8 H1 headings. You may want to use the "H1 -> H2" option to demote all _Currently **Change Failure Rate** is not measurable both at Product or Squad level in AA_ -# - - -# Performance Pulse Check +### Performance Pulse Check
@@ -339,41 +282,38 @@ six months
-# +### Why Metrics Matter -## Why Metrics Matter - - -#### Quantifiable Performance +**Quantifiable Performance** * Metrics provide objective measurements of squad performance, fostering a data-driven culture. They enable squads to track progress, set benchmarks, and identify areas for improvement. -#### Continuous Improvement +**Continuous Improvement** * By identifying bottlenecks and inefficiencies, metrics drive continuous improvement efforts. Squads can refine processes, optimize workflows, and enhance collaboration to accelerate delivery. -#### Risk Mitigation +**Risk Mitigation** * Lower change failure rates and shorter Minutes of Impact indicate increased software stability and reliability. This minimizes business risks associated with service disruptions and costly downtimes. -#### Informed Decision-Making +**Informed Decision-Making** * Metrics facilitate informed decision-making by offering a comprehensive view of the software delivery pipeline. Squads can make strategic choices to align with business goals. -#### Cross-Functional Collaboration +**Cross-Functional Collaboration** @@ -382,13 +322,10 @@ six months Software powers innovation and competitive advantage and metrics have emerged as the compass guiding software delivery squads toward success. They provide a roadmap to accelerated delivery, enhanced quality, and collaborative excellence. By leveraging the insights gleaned from these metrics, organizations can transform their software delivery processes and stay at the forefront of their industries. -# - - -## Delivery Metrics (Product & Squad level) +### Delivery Metrics (Product & Squad level) -### **Lead Time** +**Lead Time** Measures the time it takes for a code change to go from commit (branch or main) to production. Provides insights into workflow efficiency and bottlenecks. Shorter lead times indicate smoother processes and quicker value delivery. @@ -398,7 +335,7 @@ Measures the time it takes for a code change to go from commit (branch or main) * _Example: _Squad’s lead time is 509.15h (~64 days). Working time is 163.85h (~20 days) -### **Deploy Frequency** +**Deploy Frequency** Measures how often code is deployed. Enables rapid iteration and faster time-to-market. Encourages small, incremental changes, reducing the risk of failures. @@ -408,7 +345,7 @@ Measures how often code is deployed. Enables rapid iteration and faster time-to- * _Example:_ If a squad deployed code 10 times in a month with 31 days, the deployment frequency would be 10/31 = 0.32 deployments per day -### **Change Failure Rate** +**Change Failure Rate** Gauges the percentage of deployments that result in failures. Offers visibility into code quality and stability. Low failure rates signify robust testing and higher software reliability. @@ -418,7 +355,7 @@ Gauges the percentage of deployments that result in failures. Offers visibility * _Example:_ If your squad deployed five times this week and one of them resulted in a failure, your change failure rate is 20% -### **Minutes of Impact (MoI)** +**Minutes of Impact (MoI)** Calculates the down time from a service disruption over set time. Highlights the squad’s ability to detect and resolve issues swiftly. Shorter MoI reflects strong incident response and system resilience. @@ -428,10 +365,10 @@ Calculates the down time from a service disruption over set time. Highlights the * _Example: _If your squad had four incidents in a 40-hour workweek and spent one total hour on them (from alert to fix), your MoI for that week would be 15 minutes -## Engineering Metrics (Squad level) +### Engineering Metrics (Squad level) -### **Branch & PR Age** +**Branch & PR Age** Evaluates development best practices on repos. @@ -441,17 +378,16 @@ Evaluates development best practices on repos. * _Example: _Number of commits a branch is behind or ahead of main. Hours or days a PR has existed before merging into main. -### **Code Coverage** +**Code Coverage** Provides percent of unit testing covering the lines of code. Assesses the test performance. - * _How to measure: _3rd party tooling which runs in your automated CI/CD builds * _Example: _If the code were testing has a 100 lines of code and the number of lines of code that is actually validated in the same software is 50, then the code coverage percentage of this software will be 50% -### **Code Quality** +**Code Quality** Assesses high or low quality of code as well as security related defects. Identifies potential risks of compromising code and indicated where to fix software defects @@ -461,7 +397,7 @@ Assesses high or low quality of code as well as security related defects. Identi * _Example: _1 aspect of code quality is reusability. This is best measured by counting the number of interdependencies. The more there are, generally the less reusable the code. -### **Joy Index** +**Joy Index** Benchmarks individual developer experience, satisfaction, and happiness. @@ -471,13 +407,10 @@ Benchmarks individual developer experience, satisfaction, and happiness. * _Example: _Question -_ _“How likely are you to recommend us as a place to work for your family, friends, co-workers?” -# - - -## Product Metrics (Product level) +### Product Metrics (Product level) -### **Business OKRs defined & measurable** +**Business OKRs defined & measurable** OKRs are defined, with clear and inspiring Objectives that align with the company's overall mission and vision. Key Results are specific, measurable, and quantifiable, providing a clear path towards achieving the Objectives. OKRs are regularly reviewed and updated as needed, with a strong commitment to achieving them. From 298d3134d78f293c860a1e33fb0fa7fefe2bc7a6 Mon Sep 17 00:00:00 2001 From: Nicholas Shaddox Date: Wed, 20 Sep 2023 12:34:32 -0700 Subject: [PATCH 7/8] fix: broken italics --- docs/delivery-metrics.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/delivery-metrics.md b/docs/delivery-metrics.md index c7bfc4f..240572c 100644 --- a/docs/delivery-metrics.md +++ b/docs/delivery-metrics.md @@ -332,7 +332,7 @@ Measures the time it takes for a code change to go from commit (branch or main) * _How to measure:_ Conduct squad level VSM to gather time code change goes from commit to production -* _Example: _Squad’s lead time is 509.15h (~64 days). Working time is 163.85h (~20 days) +* _Example:_ Squad’s lead time is 509.15h (~64 days). Working time is 163.85h (~20 days) **Deploy Frequency** @@ -362,7 +362,7 @@ Calculates the down time from a service disruption over set time. Highlights the * _How to measure:_ Measures time it takes for service to recover from failure. Calculated by tracking the average time between a bug report and the moment a bug fix is deployed. -* _Example: _If your squad had four incidents in a 40-hour workweek and spent one total hour on them (from alert to fix), your MoI for that week would be 15 minutes +* _Example:_ If your squad had four incidents in a 40-hour workweek and spent one total hour on them (from alert to fix), your MoI for that week would be 15 minutes ### Engineering Metrics (Squad level) @@ -374,8 +374,8 @@ Evaluates development best practices on repos. -* _How to measure: _Calculated by the overall average age of branches & pull requests across each repo -* _Example: _Number of commits a branch is behind or ahead of main. Hours or days a PR has existed before merging into main. +* _How to measure:_ Calculated by the overall average age of branches & pull requests across each repo +* _Example:_ Number of commits a branch is behind or ahead of main. Hours or days a PR has existed before merging into main. **Code Coverage** @@ -383,8 +383,8 @@ Evaluates development best practices on repos. Provides percent of unit testing covering the lines of code. Assesses the test performance. -* _How to measure: _3rd party tooling which runs in your automated CI/CD builds -* _Example: _If the code were testing has a 100 lines of code and the number of lines of code that is actually validated in the same software is 50, then the code coverage percentage of this software will be 50% +* _How to measure:_ 3rd party tooling which runs in your automated CI/CD builds +* _Example:_ If the code were testing has a 100 lines of code and the number of lines of code that is actually validated in the same software is 50, then the code coverage percentage of this software will be 50% **Code Quality** @@ -393,8 +393,8 @@ Assesses high or low quality of code as well as security related defects. Identi -* _How to measure: _3rd party tooling which runs in your automated CI/CD builds -* _Example: _1 aspect of code quality is reusability. This is best measured by counting the number of interdependencies. The more there are, generally the less reusable the code. +* _How to measure:_ 3rd party tooling which runs in your automated CI/CD builds +* _Example:_ 1 aspect of code quality is reusability. This is best measured by counting the number of interdependencies. The more there are, generally the less reusable the code. **Joy Index** @@ -404,7 +404,7 @@ Benchmarks individual developer experience, satisfaction, and happiness. * _How to measure:_ Employee net promoter score tracks employee engagement and measures the impact of their people and culture initiatives. The eNPS shows how workers feel about your organization. -* _Example: _Question -_ _“How likely are you to recommend us as a place to work for your family, friends, co-workers?” +* _Example: Question_ - “How likely are you to recommend us as a place to work for your family, friends, co-workers?” ### Product Metrics (Product level) From 14fe41fff0ae232319a30e2ad55d74ad379cc49a Mon Sep 17 00:00:00 2001 From: Nicholas Shaddox Date: Tue, 3 Oct 2023 11:44:18 -0700 Subject: [PATCH 8/8] chore: update delivery metrics with new doc --- docs/delivery-metrics.md | 464 ++++++++++++--------------------------- 1 file changed, 146 insertions(+), 318 deletions(-) diff --git a/docs/delivery-metrics.md b/docs/delivery-metrics.md index 240572c..af10a59 100644 --- a/docs/delivery-metrics.md +++ b/docs/delivery-metrics.md @@ -1,422 +1,250 @@ -# Leading Indicators for Delivery Effectiveness - Our recommendation is to collect the following metrics for delivery effectiveness. +# Engineering Effectiveness and Delivery Metrics -### Recommended Metrics +--- -**Delivery Metrics (Product & Squad level)** +## Summary -* Lead Time - Tracks the time it takes for a code change to go from commit (branch or main) to production -* Deployment Frequency - Measures how often code is deployed -* Change Failure Rate* - Percentage of changes that result in failures after it is in production or released to end user -* Minutes of Impact - Calculates the down time from a service disruption over set time +Below defines the approach for measuring product teams. The measuring approach will be uniform across product delivery. There is no one single metric that rules them all. Focusing on one metric will cause over-rotation in one direction and will negatively impact the others. The outcome is not a “big brother is watching” nor is it a way to measure individual performance rather it’s a mechanism to provide product teams signals (quantitative and qualitative) as to how they are delivering their product against an opinionated set of engineering practices and principles as well as a measure for overall team culture and joy. -**Engineering Metrics (Squad level)** +Metrics will be gathered real-time or as near real-time as possible and will not impede delivery. (i.e. if the metrics collection system is down product delivery will not be impacted.) -* Branch & PR Age - Evaluates development best practices on repos -* Code Coverage - Provides percent of unit testing covering the lines of code -* Code Quality - Assesses high or low quality of code as well as security related defects -**Joy Index (all level)** -* Team & Leadership Joy Index - Benchmarks individual developer experience, satisfaction, and happiness +### Key Metrics -**Product Metrics (Product level)** +**Delivery Metrics** -* Business OKRs defined & measurable - Verify the squads have established high quality OKRs with clearly defined measures +* Lead Time for Changes - Tracks the time it takes for a code change to go from commit (branch or main) to production +* Deployment Frequency - Measures how often code is deployed to Production +* Change Failure Rate - Percentage of changes that result in failures after it is in production or released to end user +* Mean Time to Restore (MTTR) - Calculates the time needed to recover from a service disruption; highlights the team's ability to detect and resolve issues swiftly. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Theme - Metrics - Focus Relentlessly - Excel Technically - High Performing Team -
Delivery Metrics -

-(Product & Squad level) -

Lead Time - ✔️ - ✔️ - ✔️ -
Deployment Frequency - - ✔️ - ✔️ -
Change Failure Rate* - - ✔️ - ✔️ -
Minutes of Impact - - ✔️ - ✔️ -
Engineering Metrics -

-(Squad level) -

Branch & PR Age - - ✔️ - ✔️ -
Code Coverage - - ✔️ - ✔️ -
Code Quality - - ✔️ - ✔️ -
Joy Index - - - ✔️ -
Product Metrics (Product level) - Business OKRs defined & measurable - ✔️ - - ✔️ -
-_Currently **Change Failure Rate** is not measurable both at Product or Squad level in AA_ +**Engineering Metrics** -### Performance Pulse Check +* Branch & PR Age - Evaluates the lifespan and efficiency of code integration in repos +* Code Coverage - Percentage of code statements exercised during unit test runs +* Code Quality - Assesses high or low quality of code across three tenets - Security (Vulnerability), Reliability (Bug) and Maintainability (Code Smell) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Metrics - Run - Walk - Crawl -
Lead Time - Less than 1 hour - Between one -

-week and -

-one month -

Between one -

-month and -

-six months -

Deployment Frequency - On Demand/Multiple times per day - Between once per day to once per week - Between 2 weeks and a month -
- - - -
Change Failure Rate - 0-7% (<15 min) - 7-15% (1+ hours) - 16-30% (2+ hours) -
Minutes of Impact - Less than 15 minutes - More than 1 hour - More than 2 hours -
Branch & PR Age - Less than 1 day - Less than 3 days - 1 week to 1 month -
Code Coverage - 80-90%+ for existing & new code - 20-50% for existing code -

-80%+ for new code -

0-50% for existing code -

-80%+ for new code -

Code Quality -

-(vulnerabilities) -

0 critical -

-0 high -

-0 medium -

-0 low -

0 critical -

- 0 high -

0 critical -
Joy Index - eNPS 30 to 80+ - eNPS -10 to 30+ - eNPS -30 to 10+ -
Business OKRs defined & measurable - Defined with clear and inspiring objectives that align with the company's vision - Defined but room for improvement regarding alignment with company's vision - Defined but challenging for squads to understand how their work contributes to the company's vision -
+**Joy Index (all level)** -### Why Metrics Matter +* Team & Leadership Joy Index - Benchmarks individual developer experience, satisfaction, and happiness using NPS(Net Promoter Score) and surveys -**Quantifiable Performance** +**Product Metrics** +* Business Objectives & Key Results are defined & measurable - Verify the teams have established high quality OKRs with clearly defined measures -* Metrics provide objective measurements of squad performance, fostering a data-driven culture. They enable squads to track progress, set benchmarks, and identify areas for improvement. +--- -**Continuous Improvement** +## Performance Pulse Check + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MetricsRun Walk Crawl
Lead TimeLess than 1 hourBetween one week andone monthBetween one month and six months
Deployment FrequencyOn Demand/Multiple times per dayBetween once per day to once per weekBetween 2 weeks and a month
Change Failure Rate0-7% (<15 min)7-15% (1+ hours)16-30% (2+ hours)
MTTR< 1 hour1-6 hours1 business day
Branch & PR AgeLess than 1 dayLess than 3 days1 week to 1 month
Code Coverage1Exemplary

Ex: 80-90%+ for existing & new code

Commendable

Ex: 20-50% for existing code

80%+ for new code

Acceptable

Example: 0-50% for existing code

80%+ for new code

Code Quality2

(SonarQube Rating)

Security: A

Reliability: A

Maintainability: B

Security: B

Reliability: B

Maintainability: C

Security: C

Reliability: C

Maintainability: D

Joy IndexeNPS 30 to 80+eNPS -10 to 30+eNPS -30 to 10+ +
Business OKRs defined & measurableDefined with clear and inspiring objectives that align with the company's visionDefined but room for improvement regarding alignment with company's visionDefined but challenging for teams to understand how their work contributes to the company's vision
-* By identifying bottlenecks and inefficiencies, metrics drive continuous improvement efforts. Squads can refine processes, optimize workflows, and enhance collaboration to accelerate delivery. + + 1Code coverage percentage baseline and target will be need to determined during charter/assessment phase for teams + + + 2Security - Vulnerability | Reliability- Bug | Maintainability - Code Smell + -**Risk Mitigation** +## Why Metrics Matter -* Lower change failure rates and shorter Minutes of Impact indicate increased software stability and reliability. This minimizes business risks associated with service disruptions and costly downtimes. +**Quantifiable Performance** +* Metrics provide objective measurements of team performance, fostering a data-driven culture. They enable teams to track progress, set benchmarks, and identify areas for improvement. -**Informed Decision-Making** +**Continuous Improvement** -* Metrics facilitate informed decision-making by offering a comprehensive view of the software delivery pipeline. Squads can make strategic choices to align with business goals. +* By identifying bottlenecks and inefficiencies, metrics drive continuous improvement efforts. Teams can refine processes, optimize workflows, and enhance collaboration to accelerate delivery. -**Cross-Functional Collaboration** +**Risk Mitigation** +* Lower change failure rates and shorter Minutes of Impact indicate increased software stability and reliability. This minimizes business risks associated with service disruptions and costly downtimes. -* These metrics encourage collaboration between development, operations, and quality assurance teams. Silos are broken down, fostering a holistic approach to software delivery. -Software powers innovation and competitive advantage and metrics have emerged as the compass guiding software delivery squads toward success. They provide a roadmap to accelerated delivery, enhanced quality, and collaborative excellence. By leveraging the insights gleaned from these metrics, organizations can transform their software delivery processes and stay at the forefront of their industries. +**Informed Decision-Making** -### Delivery Metrics (Product & Squad level) +* Metrics facilitate informed decision-making by offering a comprehensive view of the software delivery pipeline. Teams can make strategic choices to improve alignment with business goals. -**Lead Time** -Measures the time it takes for a code change to go from commit (branch or main) to production. Provides insights into workflow efficiency and bottlenecks. Shorter lead times indicate smoother processes and quicker value delivery. +**Cross-Functional Collaboration** +* These metrics encourage collaboration between development, operations, and quality assurance teams. Silos are broken down, fostering a holistic approach to software delivery. +Software powers innovation and competitive advantage and metrics have emerged as the compass guiding software delivery teams toward success. They provide a roadmap to accelerated delivery, enhanced quality, and collaborative excellence. By leveraging the insights gleaned from these metrics, organizations can transform their software delivery processes and stay at the forefront of their industries. -* _How to measure:_ Conduct squad level VSM to gather time code change goes from commit to production -* _Example:_ Squad’s lead time is 509.15h (~64 days). Working time is 163.85h (~20 days) -**Deploy Frequency** +## Delivery Metrics -Measures how often code is deployed. Enables rapid iteration and faster time-to-market. Encourages small, incremental changes, reducing the risk of failures. +### Lead Time for Change +Measures the time it takes for a code change to go from commit (branch or main) to production. Provides insights into workflow efficiency and bottlenecks. Shorter lead times indicate smoother processes and quicker value delivery. +* _How to measure:_ Conduct team level Value Stream Map ( VSM) to gather time code change goes from commit to production +* _Example: _team’s lead time is 509.15h (~64 days). Working time is 163.85h (~20 days) -* _How to measure:_ Divide the total number of deployments made in a given time period (e.g., a month) by the total number of days in that period -* _Example:_ If a squad deployed code 10 times in a month with 31 days, the deployment frequency would be 10/31 = 0.32 deployments per day -**Change Failure Rate** +### Deploy Frequency -Gauges the percentage of deployments that result in failures. Offers visibility into code quality and stability. Low failure rates signify robust testing and higher software reliability. +Measures how often code is deployed. Enables rapid iteration and faster time-to-market. Encourages small, incremental changes, reducing the risk of failures. +* _How to measure:_ Divide the total number of deployments made in a given time period (e.g., a month) by the total number of days in that period +* _Example:_ If a team deployed code 10 times in a month with 31 days, the deployment frequency would be 10/31 = an average of _0.32 deployments per day_ over the month -* _How to measure:_ Percentage of changes that were made to code that then resulted in incident/rollback/or any type of prod failure. Calculated by counting the number of deployment failures and then dividing by number of total deployments. -* _Example:_ If your squad deployed five times this week and one of them resulted in a failure, your change failure rate is 20% +### Change Failure Rate -**Minutes of Impact (MoI)** +Gauges the percentage of deployments that result in failures. Offers visibility into code quality and stability. Low failure rates signify robust testing and higher software reliability. -Calculates the down time from a service disruption over set time. Highlights the squad’s ability to detect and resolve issues swiftly. Shorter MoI reflects strong incident response and system resilience. +* _How to measure:_ Percentage of code changes that resulted in an incident, rollback, or any type of prod failure. Calculated by counting the number of deployment failures and then dividing by the number of total deployments in a given time period. +* _Example:_ If your team deployed five times this week and one of them resulted in a failure, your change failure rate is 20% -* _How to measure:_ Measures time it takes for service to recover from failure. Calculated by tracking the average time between a bug report and the moment a bug fix is deployed. -* _Example:_ If your squad had four incidents in a 40-hour workweek and spent one total hour on them (from alert to fix), your MoI for that week would be 15 minutes +### Mean Time to Restore (MTTR) +Calculates the time needed to recover from a service disruption and highlights the team's ability to detect and resolve issues swiftly. Shorter MTTR reflects strong incident response and system resilience. -### Engineering Metrics (Squad level) +* _How to measure:_ Measures time it takes for service to recover from failure. Calculated by tracking the average time between a time of service disruption and the moment a fix is deployed. +* _Example: _A team's average time from problem detection to full recovery is 90 minutes over the course of 6 months. -**Branch & PR Age** -Evaluates development best practices on repos. +## Engineering Metrics +### Branch & PR Age +Evaluates development best practices on repos. -* _How to measure:_ Calculated by the overall average age of branches & pull requests across each repo -* _Example:_ Number of commits a branch is behind or ahead of main. Hours or days a PR has existed before merging into main. +* _How to measure: _Calculated by the overall average age of branches & pull requests across each repo +* _Example: _Number of commits a branch is behind or ahead of main. Hours or days a PR has existed before merging into main. -**Code Coverage** -Provides percent of unit testing covering the lines of code. Assesses the test performance. +### Code Coverage +Provides percentage of code statements exercised during unit test runs. Assesses the amount of code logic invoked during unit testing. -* _How to measure:_ 3rd party tooling which runs in your automated CI/CD builds -* _Example:_ If the code were testing has a 100 lines of code and the number of lines of code that is actually validated in the same software is 50, then the code coverage percentage of this software will be 50% +* _How to measure: _3rd party tooling which runs in your automated CI/CD builds +* _Example: _If the code we're testing has 100 lines of code and 50 of those are executed by unit test code, then the code coverage percentage of this software is 50% -**Code Quality** -Assesses high or low quality of code as well as security related defects. Identifies potential risks of compromising code and indicated where to fix software defects +### Code Quality +Grades the quality of code as well as detecting security related defects. Identifies potential vulnerabilities in code and indicates where to resolve these and other software defects +* _How to measure: _3rd party tooling which runs in your automated CI/CD builds +* _Example: _One aspect of code quality is reusability, which can be measured by counting the number of interdependencies. The more tightly-coupled the code is with surrounding code and dependencies, the less reusable the code tends to be. -* _How to measure:_ 3rd party tooling which runs in your automated CI/CD builds -* _Example:_ 1 aspect of code quality is reusability. This is best measured by counting the number of interdependencies. The more there are, generally the less reusable the code. -**Joy Index** +## Joy Index Benchmarks individual developer experience, satisfaction, and happiness. +* _How to measure:_ Employee Net Promoter Score tracks employee engagement and measures the impact of their people and culture initiatives. The eNPS shows how workers feel about your organization. +* _Example: _Question -_ _“How likely are you to recommend us as a place to work for your family, friends, co-workers?” -* _How to measure:_ Employee net promoter score tracks employee engagement and measures the impact of their people and culture initiatives. The eNPS shows how workers feel about your organization. -* _Example: Question_ - “How likely are you to recommend us as a place to work for your family, friends, co-workers?” +## Product Metrics -### Product Metrics (Product level) - - -**Business OKRs defined & measurable** +### Business OKRs defined & measurable OKRs are defined, with clear and inspiring Objectives that align with the company's overall mission and vision. Key Results are specific, measurable, and quantifiable, providing a clear path towards achieving the Objectives. OKRs are regularly reviewed and updated as needed, with a strong commitment to achieving them. - - * _How to measure:_ All team members understand the OKRs and how their work contributes to their achievement. The OKRs are logged in the company's OKR tracker. -* _Example:_ Baggage Services Product; Reduce the percentage in manual edits to optimizer suggested solutions from 86% to 43% by Q3, 23 inorder to increase reliance on the TLink optimizer