From cfb0b6b534a365ab269b1154c37fb40e1e3e1073 Mon Sep 17 00:00:00 2001 From: Alex Ramsay Date: Thu, 25 Jan 2024 15:50:09 -0500 Subject: [PATCH 1/5] Reorganizing content --- docs/business-systems.md | 5 +++ docs/guidance/metrics/overview.md | 39 ------------------- .../delivery-metrics}/devex-platform.md | 0 .../lagging-delivery-indicators.md | 0 .../leading-delivery-indicators.md | 0 .../delivery-metrics/overview.md | 6 +++ .../satisfaction-well-being.md | 0 .../observing-human-systems.md | 0 docs/{guidance => }/why-metrics-matter.md | 0 mkdocs.yml | 18 +++++---- 10 files changed, 21 insertions(+), 47 deletions(-) create mode 100644 docs/business-systems.md delete mode 100644 docs/guidance/metrics/overview.md rename docs/{guidance/metrics => human-systems/delivery-metrics}/devex-platform.md (100%) rename docs/{guidance/metrics => human-systems/delivery-metrics}/lagging-delivery-indicators.md (100%) rename docs/{guidance/metrics => human-systems/delivery-metrics}/leading-delivery-indicators.md (100%) create mode 100644 docs/human-systems/delivery-metrics/overview.md rename docs/{guidance/metrics => human-systems/delivery-metrics}/satisfaction-well-being.md (100%) rename docs/{guidance => human-systems}/observing-human-systems.md (100%) rename docs/{guidance => }/why-metrics-matter.md (100%) diff --git a/docs/business-systems.md b/docs/business-systems.md new file mode 100644 index 0000000..861b92f --- /dev/null +++ b/docs/business-systems.md @@ -0,0 +1,5 @@ +# Business OKRs defined & measurable + +Objectives & Key Results (OKRs) are defined, with clear and inspiring Objectives that align with the company's overall mission and vision. Key Results are specific, measurable, and quantifiable, providing a clear path towards achieving the Objectives. OKRs are regularly reviewed and updated as needed, with a strong commitment to achieving them. + +- _How to Measure:_ All team members understand the OKRs and how their work contributes to their achievement. The OKRs are logged in the company's OKR tracker. diff --git a/docs/guidance/metrics/overview.md b/docs/guidance/metrics/overview.md deleted file mode 100644 index c5409d5..0000000 --- a/docs/guidance/metrics/overview.md +++ /dev/null @@ -1,39 +0,0 @@ - -# Engineering Effectiveness and Delivery Metrics - -Below defines the approach for measuring engineering teams. The measuring approach will be uniform across engineering delivery. There is no one single metric that rules them all. Focusing on one metric will cause overemphasis in one direction and will negatively impact the others. The outcome is not “big brother is watching” nor is it a way to measure individual performance. Rather, it is a mechanism to provide quantitative and qualitative [signals](https://github.com/cncf/tag-observability/blob/main/whitepaper.md#observability-signals) to product teams that measure their delivery, culture, and joy against an opinionated set of engineering practices and principles. - -Metrics will be gathered real-time or as near real-time as possible. Product delivery will not be impacted if the observability platform is unavailable. - -## Business OKRs defined & measurable - -Objectives & Key Results (OKRs) are defined, with clear and inspiring Objectives that align with the company's overall mission and vision. Key Results are specific, measurable, and quantifiable, providing a clear path towards achieving the Objectives. OKRs are regularly reviewed and updated as needed, with a strong commitment to achieving them. - -- _How to Measure:_ All team members understand the OKRs and how their work contributes to their achievement. The OKRs are logged in the company's OKR tracker. - ---- - -## Performance Pulse Check - -|Metrics|Run|Walk|Crawl| -|-------|---|----|-----| -|Lead Time|Less than 1 hour|Between one week andone month|Between one month and six months| -|Deployment Frequency|On Demand/Multiple times per day|Between once per day to once per week|Between 2 weeks and a month| -|Change Failure Rate|0-7% (<15 min)|7-15% (1+ hours)|16-30% (2+ hours)| -|MTTR|< 1 hour|1-6 hours|1 business day| -|Branch & PR Age|Less than 4 hours|Less than 3 days|Greater than 1 week| -|Code Coverage (New Code)|> 90%|80 - 90%|< 80%| -|Code Coverage (Legacy Code)|> 80%|20% - 80%|< 50%| -|Code Quality (SonarQube Security)|A|B|C| -|Code Quality (SonarQube Reliability)|A|B|C| -|Code Quality (SonarQube Maintainability)|B|C|D| -|Joy Index|eNPS 30 to 80+|eNPS -10 to 30+|eNPS -30 to 10+| -|Business OKRs defined & measurable|Defined with clear and inspiring objectives that align with the company's vision|Defined but room for improvement regarding alignment with company's vision|Defined but challenging for teams to understand how their work contributes to the company's vision| - - - 1Code coverage percentage baseline and target will be need to determined during charter/assessment phase for teams - - - - 2Security - Vulnerability | Reliability- Bug | Maintainability - Code Smell - diff --git a/docs/guidance/metrics/devex-platform.md b/docs/human-systems/delivery-metrics/devex-platform.md similarity index 100% rename from docs/guidance/metrics/devex-platform.md rename to docs/human-systems/delivery-metrics/devex-platform.md diff --git a/docs/guidance/metrics/lagging-delivery-indicators.md b/docs/human-systems/delivery-metrics/lagging-delivery-indicators.md similarity index 100% rename from docs/guidance/metrics/lagging-delivery-indicators.md rename to docs/human-systems/delivery-metrics/lagging-delivery-indicators.md diff --git a/docs/guidance/metrics/leading-delivery-indicators.md b/docs/human-systems/delivery-metrics/leading-delivery-indicators.md similarity index 100% rename from docs/guidance/metrics/leading-delivery-indicators.md rename to docs/human-systems/delivery-metrics/leading-delivery-indicators.md diff --git a/docs/human-systems/delivery-metrics/overview.md b/docs/human-systems/delivery-metrics/overview.md new file mode 100644 index 0000000..592af79 --- /dev/null +++ b/docs/human-systems/delivery-metrics/overview.md @@ -0,0 +1,6 @@ + +# Engineering Effectiveness and Delivery Metrics + +Below defines the approach for measuring engineering teams. The measuring approach will be uniform across engineering delivery. There is no one single metric that rules them all. Focusing on one metric will cause overemphasis in one direction and will negatively impact the others. The outcome is not “big brother is watching” nor is it a way to measure individual performance. Rather, it is a mechanism to provide quantitative and qualitative [signals](https://github.com/cncf/tag-observability/blob/main/whitepaper.md#observability-signals) to product teams that measure their delivery, culture, and joy against an opinionated set of engineering practices and principles. + +Metrics will be gathered real-time or as near real-time as possible. Product delivery will not be impacted if the observability platform is unavailable. diff --git a/docs/guidance/metrics/satisfaction-well-being.md b/docs/human-systems/delivery-metrics/satisfaction-well-being.md similarity index 100% rename from docs/guidance/metrics/satisfaction-well-being.md rename to docs/human-systems/delivery-metrics/satisfaction-well-being.md diff --git a/docs/guidance/observing-human-systems.md b/docs/human-systems/observing-human-systems.md similarity index 100% rename from docs/guidance/observing-human-systems.md rename to docs/human-systems/observing-human-systems.md diff --git a/docs/guidance/why-metrics-matter.md b/docs/why-metrics-matter.md similarity index 100% rename from docs/guidance/why-metrics-matter.md rename to docs/why-metrics-matter.md diff --git a/mkdocs.yml b/mkdocs.yml index d2ed377..d89bf6a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -60,18 +60,20 @@ repo_url: https://github.com/liatrio/openo11y.dev nav: - "Welcome to Open O11y": "index.md" - - "Guidance": - - "Why Metrics Matter": "./guidance/why-metrics-matter.md" - - "Observing Human Systems": "./guidance/observing-human-systems.md" + - "Why Metrics Matter": "./why-metrics-matter.md" + - "Human Systems": + - "Observing Human Systems": "./human-systems/observing-human-systems.md" - "Delivery Metrics": - - "Overview": "./guidance/metrics/overview.md" - - "Lagging Inidicators": "./guidance/metrics/lagging-delivery-indicators.md" - - "Leading Inidicators": "./guidance/metrics/leading-delivery-indicators.md" - - "DevEx & Platform": "./guidance/metrics/devex-platform.md" - - "Satisfaction & Well-Being": "./guidance/metrics/satisfaction-well-being.md" + - "Overview": "./human-systems/delivery-metrics/overview.md" + - "Lagging Inidicators": "./human-systems/delivery-metrics/lagging-delivery-indicators.md" + - "Leading Inidicators": "./human-systems/delivery-metrics/leading-delivery-indicators.md" + - "DevEx & Platform": "./human-systems/delivery-metrics/devex-platform.md" + - "Satisfaction & Well-Being": "./human-systems/delivery-metrics/satisfaction-well-being.md" + - "Business Systems": "./business-systems.md" # - "O11y Quick Start Platform": # - "Overview": "./platform/README.md" # - "Architecture": "./platform/architecture.md" # - "Recommended Implementations": # - "AWS": "./platform/aws.md" + - "External Resources": "./useful-resources.md" - "Contributing": "./contributing.md" From 7ca4ccdc789a99aa073e62478fe06734baa39c7e Mon Sep 17 00:00:00 2001 From: Alex Ramsay Date: Thu, 25 Jan 2024 16:33:43 -0500 Subject: [PATCH 2/5] Small updates based on current feedback --- docs/human-systems/observing-human-systems.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/human-systems/observing-human-systems.md b/docs/human-systems/observing-human-systems.md index 72e2b68..c98421d 100644 --- a/docs/human-systems/observing-human-systems.md +++ b/docs/human-systems/observing-human-systems.md @@ -1,10 +1,10 @@ # Observing Human Systems Many of us typically think of Observability in the context of observing a system of -software (e.g. a collection of microservices that form a single product). But some of the -advice provided by Open O11y describes how to observe a system of humans like a -development organization. This is a much more sensitive activity. Open O11y offers these -guiding principles to observing human systems ethically. +software (e.g. a collection of microservices that form a single product). Some of the +advice provided herein describes how to observe a human system like teams or development +organizations. This is a much more sensitive activity. Open O11y offers these guiding +principles to observing human systems ethically. ## Building A Culture of Safely Being Observed From a5d5709cb4b3538939b49477247d53c19f09ba61 Mon Sep 17 00:00:00 2001 From: Alex Ramsay Date: Fri, 26 Jan 2024 16:43:15 -0500 Subject: [PATCH 3/5] Update Observing Human Systems based on feedback --- docs/human-systems/observing-human-systems.md | 58 ++++++++++--------- 1 file changed, 31 insertions(+), 27 deletions(-) diff --git a/docs/human-systems/observing-human-systems.md b/docs/human-systems/observing-human-systems.md index c98421d..96125d7 100644 --- a/docs/human-systems/observing-human-systems.md +++ b/docs/human-systems/observing-human-systems.md @@ -6,40 +6,40 @@ advice provided herein describes how to observe a human system like teams or dev organizations. This is a much more sensitive activity. Open O11y offers these guiding principles to observing human systems ethically. -## Building A Culture of Safely Being Observed +## Building A Culture of Being Safely Observed -Many people don't like being watched. The act of being watched by another person can make -us feel uncomfortable, nervous, or many other emotions. It can cause us to wonder why we -are being watched and what the watcher is going to do. Observing human systems is subject -the same side effects. This is especially true when observations are misused. +Humans don't enjoy being watched, especially when observations are misused or directly +used for measuring performance. Misusing telemetry can have a net negative effect on human +systems and the individuals they are comprised of. + +> Tell me how you measure me and I will tell you how I will behave. If you measure me in +> an illogical way, do not complain about illogical behavior. +> \- Eli Goldratt *When observing human systems, it is critical to build a culture where individuals can -feel and **be** safe while being observed.* +both feel and **be** safe while being observed.* ### Employee Evaluations and Compensation -Being safely observed implies that employers should *not* use a specific set or subset of -telemetry for reviews, compensation adjustments, etc. If this is done, then the subset of -telemetry becomes very important to employees which inherently makes other telemetry less -important. This is a problem that can be avoided. - -Similarly, telemetry should be a minimal or irrelevant factor in these events. Because -reviews and compensation are so impactful to employees, the use of telemetry data can -dramatically shift an employee's focus to optimizing telemetry data. While it may seem +Being safely observed implies that employers must *not* use telemetry for reviews, +compensation adjustments, employment decisions, etc. Because reviews and compensation are +so impactful to employees, the use of telemetry data can dramatically shift an employee's +focus to optimizing telemetry data. If employers misuse telemetry, then employees are +negatively impacted and using telemetry to improve becomes unimportant. While it may seem like a positive change, optimizing telemetry often results in employees being less -effective contributors. +effective contributors. This is a problem that can be avoided. -### Personal Telemetry +### Individual Telemetry -Telemetry data should only contain aggregated data for small or larger groups. -Observability over human systems should *not* gather telemetry from specific individuals. +Telemetry data must only contain aggregated data for systems (groups) of people. +Observability over human systems must *not* gather telemetry from specific individuals. This is a foundational requirement of being safely observed. ### Comparing Disparate Groups Being safely observed also extends to cross-group evaluations. Every team has distinct -members, work responsibilities, personal responsibilities, opinions, etc. In other words, -every team is unique. Thus, some of the telemetry produced by observing any two teams will +members, work responsibilities, personal responsibilities, opinions, etc. Every human +system is unique. Thus, some of the telemetry produced by observing any two teams will differ naturally. The Agile Framework provides a useful example with the nature of `story points` in @@ -53,14 +53,18 @@ it may be reasonable to compare the `story points planned / story points complet between multiple teams as an unusually low or high ratio may indicate a planning and/or capacity issue. +> Don’t ask people to collaborate if they know that, in the end, there will be a winner +> and a loser. +> \- Eli Goldratt + ## Using Telemetry In Different Roles ### Teams and Individual Contributors -These are the ideal candidates for using telemetry data. Teams and individuals should -observe themselves in to identify trends and establish their working norms. Once a -baseline is established, teams can then use telemetry to identify issues and strive to -improve individual and team performance. +These are the ideal candidates for using telemetry data. Teams and individuals must +observe themselves to identify trends and establish their working norms. Once a baseline +is established, teams can then use telemetry to identify issues and strive to improve +individual and team performance based on an established set of engineering principles. ### Secondary Contributors @@ -68,7 +72,7 @@ Open O11y coined this term to describe roles that exist to organize, improve, or facilitate the work of others. In the world of software development, this includes SCRUM Masters, Project Managers, middle and upper management, and others. -Secondary contributors should generally *not* use telemetry to observe the teams they +Generally, secondary contributors must *not* use telemetry to observe the teams they support. Doing so offers limited benefits but can significantly damage the the organizations culture of being safely observed. @@ -80,5 +84,5 @@ contributor, or both. This also varies heavily within a single organization. The must be especially careful with how they use telemetry data. These team members must constantly evaluate if they are observing their team or themselves as a part of that team. -These leaders should pay careful attention to reactions, team feedback, and changes in -team behaviors. +These leaders must pay careful attention to reactions, team feedback, and changes in team +behaviors. From 5f9876d7f9c203d5ba6980f1156bfb9867cf1773 Mon Sep 17 00:00:00 2001 From: Alex Ramsay Date: Mon, 29 Jan 2024 12:52:14 -0500 Subject: [PATCH 4/5] Include feedback for "observing human systems" --- docs/business-systems.md | 9 +++- docs/human-systems/observing-human-systems.md | 41 ++++++++++--------- 2 files changed, 28 insertions(+), 22 deletions(-) diff --git a/docs/business-systems.md b/docs/business-systems.md index 861b92f..3da3db9 100644 --- a/docs/business-systems.md +++ b/docs/business-systems.md @@ -1,5 +1,10 @@ # Business OKRs defined & measurable -Objectives & Key Results (OKRs) are defined, with clear and inspiring Objectives that align with the company's overall mission and vision. Key Results are specific, measurable, and quantifiable, providing a clear path towards achieving the Objectives. OKRs are regularly reviewed and updated as needed, with a strong commitment to achieving them. +[Objectives & Key Results(OKRs)](https://www.productboard.com/blog/defining-objectives-and-key-results-for-your-product-team/) +are defined, with clear and inspiring Objectives that align with the company's overall +mission and vision. Key Results are specific, measurable, and quantifiable, providing a +clear path towards achieving the Objectives. OKRs are regularly reviewed and updated as +needed, with a strong commitment to achieving them. -- _How to Measure:_ All team members understand the OKRs and how their work contributes to their achievement. The OKRs are logged in the company's OKR tracker. +- _How to Measure:_ All team members understand the OKRs and how their work contributes to + their achievement. The OKRs are logged in the company's OKR tracker. diff --git a/docs/human-systems/observing-human-systems.md b/docs/human-systems/observing-human-systems.md index 96125d7..97f1dac 100644 --- a/docs/human-systems/observing-human-systems.md +++ b/docs/human-systems/observing-human-systems.md @@ -2,7 +2,7 @@ Many of us typically think of Observability in the context of observing a system of software (e.g. a collection of microservices that form a single product). Some of the -advice provided herein describes how to observe a human system like teams or development +advice provided herein describes how to observe human systems like teams or development organizations. This is a much more sensitive activity. Open O11y offers these guiding principles to observing human systems ethically. @@ -14,14 +14,14 @@ systems and the individuals they are comprised of. > Tell me how you measure me and I will tell you how I will behave. If you measure me in > an illogical way, do not complain about illogical behavior. -> \- Eli Goldratt +> \- [Eli Goldratt](https://en.wikiquote.org/wiki/Eliyahu_M._Goldratt#The_Haystack_Syndrome_(1990)) *When observing human systems, it is critical to build a culture where individuals can -both feel and **be** safe while being observed.* +both feel and be safe while being observed.* ### Employee Evaluations and Compensation -Being safely observed implies that employers must *not* use telemetry for reviews, +Being safely observed implies that employers ***must not*** use telemetry for reviews, compensation adjustments, employment decisions, etc. Because reviews and compensation are so impactful to employees, the use of telemetry data can dramatically shift an employee's focus to optimizing telemetry data. If employers misuse telemetry, then employees are @@ -31,9 +31,9 @@ effective contributors. This is a problem that can be avoided. ### Individual Telemetry -Telemetry data must only contain aggregated data for systems (groups) of people. -Observability over human systems must *not* gather telemetry from specific individuals. -This is a foundational requirement of being safely observed. +Telemetry data ***must*** only contain aggregated data for systems (groups) of people. +Observability over human systems ***must not*** gather telemetry from specific +individuals. This is a foundational requirement of being safely observed. ### Comparing Disparate Groups @@ -53,36 +53,37 @@ it may be reasonable to compare the `story points planned / story points complet between multiple teams as an unusually low or high ratio may indicate a planning and/or capacity issue. -> Don’t ask people to collaborate if they know that, in the end, there will be a winner +> Don't ask people to collaborate if they know that, in the end, there will be a winner > and a loser. -> \- Eli Goldratt +> \- [Bertrand Duperrin](https://www.duperrin.com/english/2014/06/23/quote-tell-how-you-measure-and-i-will-tell-you-how-i-will-behave/) ## Using Telemetry In Different Roles ### Teams and Individual Contributors -These are the ideal candidates for using telemetry data. Teams and individuals must +These are the ideal candidates for using telemetry data. Teams and individuals ***must*** observe themselves to identify trends and establish their working norms. Once a baseline is established, teams can then use telemetry to identify issues and strive to improve individual and team performance based on an established set of engineering principles. ### Secondary Contributors -Open O11y coined this term to describe roles that exist to organize, improve, or -facilitate the work of others. In the world of software development, this includes SCRUM -Masters, Project Managers, middle and upper management, and others. +Open O11y uses this term to describe roles that exist to organize, improve, or facilitate +the work of others. In the world of software development, this includes SCRUM Masters, +Project Managers, middle and upper management, and others. -Generally, secondary contributors must *not* use telemetry to observe the teams they -support. Doing so offers limited benefits but can significantly damage the the -organizations culture of being safely observed. +Secondary contributors ***must not*** use telemetry to observe the teams they support. +Doing so offers limited benefits but can significantly damage the the organizations +culture of being safely observed. ### Direct Managers In this context, a "direct manager" refers to any managers whose direct reports are individual contributors. A direct manager can be an individual contributor, a secondary contributor, or both. This also varies heavily within a single organization. These leaders -must be especially careful with how they use telemetry data. These team members must -constantly evaluate if they are observing their team or themselves as a part of that team. +***must*** be especially careful with how they use telemetry data. These team members +***must*** constantly evaluate if they are observing their team or themselves as a part of +that team. -These leaders must pay careful attention to reactions, team feedback, and changes in team -behaviors. +These leaders ***must*** pay careful attention to reactions, team feedback, and changes in +team behaviors. From 3d2c3c540bf3f4ed9a12a376fd1ffb94f76a9980 Mon Sep 17 00:00:00 2001 From: Alex Ramsay Date: Mon, 29 Jan 2024 17:56:47 -0500 Subject: [PATCH 5/5] Feature: Awesome Content Co-authored-by: Adriel Perkins Co-authored-by: Eric Chapman --- docs/human-systems/observing-human-systems.md | 133 ++++++++++++++---- 1 file changed, 102 insertions(+), 31 deletions(-) diff --git a/docs/human-systems/observing-human-systems.md b/docs/human-systems/observing-human-systems.md index 97f1dac..7fb056e 100644 --- a/docs/human-systems/observing-human-systems.md +++ b/docs/human-systems/observing-human-systems.md @@ -1,4 +1,12 @@ -# Observing Human Systems +# Safely Observing Human Systems + +## RFC 2119 + +Open O11y has very strong opinions on how to observe human systems ethically. While +acknowledging it is social advice, this guidance is considered a specification. Thus, we +use certain keywords in accordance with [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt). + +## Guiding Principles Many of us typically think of Observability in the context of observing a system of software (e.g. a collection of microservices that form a single product). Some of the @@ -6,33 +14,53 @@ advice provided herein describes how to observe human systems like teams or deve organizations. This is a much more sensitive activity. Open O11y offers these guiding principles to observing human systems ethically. +1. When observing human systems, individuals ***MUST*** feel and be safe while being + observed. +2. Telemetry data ***MUST*** be anonymized and/or aggregated to protect the privacy of the + individuals. +3. Leadership, and the organization as a whole, ***MUST*** proactively protect against the + misuse of telemetry. +4. The gathering and use of telemetry ***MUST*** be excessively transparent[^1] to all + human systems impacted by the telemetry. +5. Human evaluations and compensation ***MUST NOT*** rely on the telemetry data listed + within Open o11y. +6. When observing human systems, observers ***MUST*** continuously reevaluate the health + and safety of the people and culture. +7. Secondary contributors ***MUST NOT*** use telemetry to measure the teams they support + in any way that violates these principles. +8. Groups within a human system ***MUST NOT*** be compared to others in any way that + violates these principles. +9. You ***MUST NOT*** be an A$$hole[^2] + +---- + ## Building A Culture of Being Safely Observed Humans don't enjoy being watched, especially when observations are misused or directly -used for measuring performance. Misusing telemetry can have a net negative effect on human -systems and the individuals they are comprised of. +used for measuring performance. Misusing telemetry can have a net negative effect, eroding +trust in human systems and the individuals they are comprised of. > Tell me how you measure me and I will tell you how I will behave. If you measure me in -> an illogical way, do not complain about illogical behavior. -> \- [Eli Goldratt](https://en.wikiquote.org/wiki/Eliyahu_M._Goldratt#The_Haystack_Syndrome_(1990)) +> an illogical way, do not complain about illogical behavior +> \- Eli Goldratt[^as81] *When observing human systems, it is critical to build a culture where individuals can both feel and be safe while being observed.* ### Employee Evaluations and Compensation -Being safely observed implies that employers ***must not*** use telemetry for reviews, +Being safely observed requires that employers ***MUST NOT*** use telemetry for reviews, compensation adjustments, employment decisions, etc. Because reviews and compensation are so impactful to employees, the use of telemetry data can dramatically shift an employee's focus to optimizing telemetry data. If employers misuse telemetry, then employees are -negatively impacted and using telemetry to improve becomes unimportant. While it may seem -like a positive change, optimizing telemetry often results in employees being less +negatively impacted and using telemetry to improve becomes unimportant. While it ***MAY*** +seem like a positive change, optimizing telemetry often results in employees being less effective contributors. This is a problem that can be avoided. ### Individual Telemetry -Telemetry data ***must*** only contain aggregated data for systems (groups) of people. -Observability over human systems ***must not*** gather telemetry from specific +Telemetry data ***MUST*** only contain aggregated data for systems (groups) of people. +Observability over human systems ***MUST NOT*** gather telemetry from specific individuals. This is a foundational requirement of being safely observed. ### Comparing Disparate Groups @@ -44,46 +72,89 @@ differ naturally. The Agile Framework provides a useful example with the nature of `story points` in software development. As a unit of measurement, the value of a story point is defined by -each team. Thus, comparing story points across teams isn't valuable. +each team. Thus, giving story points are relative, comparing story points across teams +isn't valuable. This is not inherently a problem. Yet, it does require us to be selective when comparing data across disparate groups. Considering the story points example, it is unreasonable to compare story points "completed" over a given time period between multiple teams. However, -it may be reasonable to compare the `story points planned / story points completed` -between multiple teams as an unusually low or high ratio may indicate a planning and/or +it ***MAY*** be reasonable to compare the `story points planned / story points completed` +between multiple teams as an unusually low or high ratio could indicate a planning and/or capacity issue. > Don't ask people to collaborate if they know that, in the end, there will be a winner -> and a loser. -> \- [Bertrand Duperrin](https://www.duperrin.com/english/2014/06/23/quote-tell-how-you-measure-and-i-will-tell-you-how-i-will-behave/) +> and a loser +> \- Bertrand Duperrin[^5] + +---- ## Using Telemetry In Different Roles ### Teams and Individual Contributors -These are the ideal candidates for using telemetry data. Teams and individuals ***must*** -observe themselves to identify trends and establish their working norms. Once a baseline -is established, teams can then use telemetry to identify issues and strive to improve -individual and team performance based on an established set of engineering principles. +These are the ideal candidates for using telemetry data. Teams and individuals +***SHOULD*** observe themselves to identify trends and establish their working norms. Once +a baseline is established, teams ***SHOULD*** then use telemetry to identify issues and +strive to improve individual and team performance based on an established set of +engineering principles. ### Secondary Contributors Open O11y uses this term to describe roles that exist to organize, improve, or facilitate -the work of others. In the world of software development, this includes SCRUM Masters, -Project Managers, middle and upper management, and others. - -Secondary contributors ***must not*** use telemetry to observe the teams they support. +the work of others. In the world of software development, these roles commonly include, +but are not limited to, SCRUM Masters, Project Managers, Product Owners, and Management. +Secondary contributors ***SHOULD NOT*** use telemetry to observe the teams they support. Doing so offers limited benefits but can significantly damage the the organizations culture of being safely observed. -### Direct Managers +Senior Leadership ***MAY*** use this telemetry to solve a particular business case in the +context of organizational transformation. Senior Leadership ***MUST*** ensure the use of +this telemetry is done in a manner that promotes a safe working environment and culture. +See [Gathering and Using Telemetry](#gathering-and-using-telemetry) for specifications on +cultural use of telemetry. -In this context, a "direct manager" refers to any managers whose direct reports are -individual contributors. A direct manager can be an individual contributor, a secondary -contributor, or both. This also varies heavily within a single organization. These leaders -***must*** be especially careful with how they use telemetry data. These team members -***must*** constantly evaluate if they are observing their team or themselves as a part of -that team. +Senior Leadership ***MAY*** involve lower levels of leadership and management in the above +exception, particularly with respect to Direct Managers whose direct reports are +individual contributors. These leaders ***MUST*** be especially careful with how they use +telemetry data. These team members ***MUST*** constantly evaluate if they are observing +their team or themselves as a part of that team. -These leaders ***must*** pay careful attention to reactions, team feedback, and changes in +All leaders ***MUST*** pay careful attention to reactions, team feedback, and changes in team behaviors. + +---- + +## Gathering And Using Telemetry + +When gathering and using telemetry, organizations ***MUST*** must respect the [Guiding +Principles](#guiding-principles). The following list of actions can help ensure that +leaders do not compromise the trust and safety of their organization while gathering and +using telemetry data. + +1. **Transparency in Data Collection:** Leadership ***MUST*** clearly advertise what + telemetry data is being collected and why. The purpose and benefits of the data + collection ***MUST*** be communicated openly to all groups. +2. **Purposeful Use:** Telemetry data ***MUST*** only be used for the openly communicated + purposes. These purposes ***MAY*** be aligned with organizational goals such as + improving productivity, identifying bottlenecks, enhancing efficiency, and fostering a + culture of continuous improvement. However, the use of this telemetry data ***MUST + NOT*** violate any [Guiding Principles](#guiding-principles). +3. **Data Privacy and Security:** The implementing organization ***MUST*** implement + robust data security measures to protect telemetry data and respect privacy. This data + ***MUST*** be anonymized and ***MUST NOT*** contain any PII[^3]. +4. **No Misuse:** Telemetry data ***MUST NOT*** be used for punitive measures. Instead, it + ***SHOULD*** be used for constructive feedback, learning, and growth. +5. **Feedback Mechanism:** Leadership ***MUST*** provide opportunities for team members to + share their concerns or suggestions about how telemetry is collected and used. This + helps in building trust and making improvements based on feedback. + +[^1]: Transparent - Characterized by visibility or accessibility of information especially +concerning business practices. [Merriam Webster - Transparent](https://www.merriam-webster.com/dictionary/transparent) + +[^2]: [The No A$$hole Rule](https://www.merriam-webster.com/dictionary/transparent) + +[^3]: [Personal Identifiable Information](https://www.dol.gov/general/ppii) + +[^as81]: [Eliyahu Goldratt](https://en.wikiquote.org/wiki/Eliyahu_M._Goldratt#The_Haystack_Syndrome_(1990)) + +[^5]: [Bertrand Duperrin](https://www.duperrin.com/english/2014/06/23/quote-tell-how-you-measure-and-i-will-tell-you-how-i-will-behave/)