From 568290d1f5b616791502b2694ffcde83f46347b7 Mon Sep 17 00:00:00 2001
From: Katie Wetstone <klwetstone@gmail.com>
Date: Wed, 12 Jun 2024 11:04:10 -0400
Subject: [PATCH 01/20] update readme in _common_body

---
 docs/md_templates/_common_body.tpl | 26 +++++++++++---------------
 1 file changed, 11 insertions(+), 15 deletions(-)

diff --git a/docs/md_templates/_common_body.tpl b/docs/md_templates/_common_body.tpl
index c46a089..9f34e9f 100644
--- a/docs/md_templates/_common_body.tpl
+++ b/docs/md_templates/_common_body.tpl
@@ -36,25 +36,21 @@ For more configuration details, see the sections on [command line options](#comm
 
 # Background and perspective
 
-We have a particular perspective with this package that we will use to make decisions about contributions, issues, PRs, and other maintenance and support activities.
+We created `deon` with the goal of helping data scientists across the sector to be more intentional in their choices, and more aware of the ethical implications of their work. We use that perspective to make decisions about contributions, issues, PRs, and other maintenance and support activities.
 
-First and foremost, our goal is not to be arbitrators of what ethical concerns merit inclusion. We have a [process for changing the default checklist](#changing-the-checklist), but we believe that many domain-specific concerns are not included and teams will benefit from developing [custom checklists](#custom-checklists). Not every checklist item will be relevant. We encourage teams to remove items, sections, or mark items as `N/A` as the concerns of their projects dictate.
 
-Second, we built our initial list from a set of proposed items on [multiple checklists that we referenced](#checklist-citations). This checklist was heavily inspired by an article written by Mike Loukides, Hilary Mason, and DJ Patil and published by O'Reilly: ["Of Oaths and Checklists"](https://www.oreilly.com/ideas/of-oaths-and-checklists). We owe a great debt to the thinking that proceeded this, and we look forward to thoughtful engagement with the ongoing discussion about checklists for data science ethics.
+1. 🔓 First and foremost, **our goal is not to be arbitrators of what ethical concerns merit inclusion**. We have a [process for changing the default checklist](#changing-the-checklist), but we believe that many domain-specific concerns are not included and teams will benefit from developing [custom checklists](#custom-checklists). Not every checklist item will be relevant. We encourage teams to remove items, sections, or mark items as `N/A` as the concerns of their projects dictate.
+2. 📊 It's not up to data scientists alone to decide what the ethical course of action is. This has always been a responsibility of organizations that are part of civil society. This checklist is designed to provoke conversations around **issues where data scientists have particular responsibility and perspective**. This conversation should be part of a larger organizational commitment to doing what is right.
+3. 💬 We are working at a level of abstraction that cannot concretely recommend a specific action (e.g., "remove variable X from your model"). Nearly all of the items on the checklist are **meant to provoke discussion** among good-faith actors who take their ethical responsibilities seriously. Because of this, most of the items are framed as prompts to discuss or consider. Teams will want to document these discussions and decisions for posterity.
+4. 🌎 We believe in the **power of examples** to bring the principles of data ethics to bear on human experience. This repository includes a [list of real-world examples](http://deon.drivendata.org/examples/) connected with each item in the default checklist. We encourage you to contribute relevant use cases that you believe can benefit the community by their example. In addition, if you have a topic, idea, or comment that doesn't seem right for the documentation, please add it to the [wiki page](https://github.com/drivendataorg/deon/wiki) for this project!
+5. 🔍 We believe the primary benefit of a checklist is **ensuring that we don't overlook important work**. Sometimes it is difficult with pressing deadlines and a demand to multitask to make sure we do the hard work to think about the big picture. This package is meant to help ensure that those discussions happen, even in fast-moving environments. Ethics is hard, and we expect some of the conversations that arise from this checklist may also be hard.
+6. ❓ We can't define exhaustively every term that appears in the checklist. Some of these **terms are open to interpretation** or mean different things in different contexts. We recommend that when relevant, users create their own glossary for reference.
+7. ✨ We want to avoid any items that strictly fall into the realm of statistical best practices. Instead, we want to highlight the areas where we need to pay particular attention **above and beyond best practices**.
+8. ✅ We want all the checklist items to be **as simple as possible** (but no simpler), and to be actionable.
 
-Third, we believe in the power of examples to bring the principles of data ethics to bear on human experience. This repository includes a [list of real-world examples](http://deon.drivendata.org/examples/) connected with each item in the default checklist. We encourage you to contribute relevant use cases that you believe can benefit the community by their example. In addition, if you have a topic, idea, or comment that doesn't seem right for the documentation, please add it to the [wiki page](https://github.com/drivendataorg/deon/wiki) for this project!
+## Sources
 
-Fourth, it's not up to data scientists alone to decide what the ethical course of action is. This has always been a responsibility of organizations that are part of civil society. This checklist is designed to provoke conversations around issues where data scientists have particular responsibility and perspective. This conversation should be part of a larger organizational commitment to doing what is right.
-
-Fifth, we believe the primary benefit of a checklist is ensuring that we don't overlook important work. Sometimes it is difficult with pressing deadlines and a demand to multitask to make sure we do the hard work to think about the big picture. This package is meant to help ensure that those discussions happen, even in fast-moving environments. Ethics is hard, and we expect some of the conversations that arise from this checklist may also be hard.
-
-Sixth, we are working at a level of abstraction that cannot concretely recommend a specific action (e.g., "remove variable X from your model"). Nearly all of the items on the checklist are meant to provoke discussion among good-faith actors who take their ethical responsibilities seriously. Because of this, most of the items are framed as prompts to discuss or consider. Teams will want to document these discussions and decisions for posterity.
-
-Seventh, we can't define exhaustively every term that appears in the checklist. Some of these terms are open to interpretation or mean different things in different contexts. We recommend that when relevant, users create their own glossary for reference.
-
-Eighth, we want to avoid any items that strictly fall into the realm of statistical best practices. Instead, we want to highlight the areas where we need to pay particular attention above and beyond best practices.
-
-Ninth, we want all the checklist items to be as simple as possible (but no simpler), and to be actionable.
+We built our initial list from a set of proposed items on [multiple checklists that we referenced](#checklist-citations). This checklist was heavily inspired by an article written by Mike Loukides, Hilary Mason, and DJ Patil and published by O'Reilly: ["Of Oaths and Checklists"](https://www.oreilly.com/ideas/of-oaths-and-checklists). We owe a great debt to the thinking that proceeded this, and we look forward to thoughtful engagement with the ongoing discussion about checklists for data science ethics.
 
 # Using this tool
 

From a487c369e42cea6ad6aea1668b98d364dfa10644 Mon Sep 17 00:00:00 2001
From: Katie Wetstone <klwetstone@gmail.com>
Date: Wed, 12 Jun 2024 11:04:23 -0400
Subject: [PATCH 02/20] update readme and index.md to preview

---
 README.md          | 26 +++++++++++---------------
 docs/docs/index.md | 26 +++++++++++---------------
 2 files changed, 22 insertions(+), 30 deletions(-)

diff --git a/README.md b/README.md
index 99ae008..7bad827 100644
--- a/README.md
+++ b/README.md
@@ -45,25 +45,21 @@ For more configuration details, see the sections on [command line options](#comm
 
 # Background and perspective
 
-We have a particular perspective with this package that we will use to make decisions about contributions, issues, PRs, and other maintenance and support activities.
+We created `deon` with the goal of helping data scientists across the sector to be more intentional in their choices, and more aware of the ethical implications of their work. We use that perspective to make decisions about contributions, issues, PRs, and other maintenance and support activities.
 
-First and foremost, our goal is not to be arbitrators of what ethical concerns merit inclusion. We have a [process for changing the default checklist](#changing-the-checklist), but we believe that many domain-specific concerns are not included and teams will benefit from developing [custom checklists](#custom-checklists). Not every checklist item will be relevant. We encourage teams to remove items, sections, or mark items as `N/A` as the concerns of their projects dictate.
 
-Second, we built our initial list from a set of proposed items on [multiple checklists that we referenced](#checklist-citations). This checklist was heavily inspired by an article written by Mike Loukides, Hilary Mason, and DJ Patil and published by O'Reilly: ["Of Oaths and Checklists"](https://www.oreilly.com/ideas/of-oaths-and-checklists). We owe a great debt to the thinking that proceeded this, and we look forward to thoughtful engagement with the ongoing discussion about checklists for data science ethics.
+1. 🔓 First and foremost, **our goal is not to be arbitrators of what ethical concerns merit inclusion**. We have a [process for changing the default checklist](#changing-the-checklist), but we believe that many domain-specific concerns are not included and teams will benefit from developing [custom checklists](#custom-checklists). Not every checklist item will be relevant. We encourage teams to remove items, sections, or mark items as `N/A` as the concerns of their projects dictate.
+2. 📊 It's not up to data scientists alone to decide what the ethical course of action is. This has always been a responsibility of organizations that are part of civil society. This checklist is designed to provoke conversations around **issues where data scientists have particular responsibility and perspective**. This conversation should be part of a larger organizational commitment to doing what is right.
+3. 💬 We are working at a level of abstraction that cannot concretely recommend a specific action (e.g., "remove variable X from your model"). Nearly all of the items on the checklist are **meant to provoke discussion** among good-faith actors who take their ethical responsibilities seriously. Because of this, most of the items are framed as prompts to discuss or consider. Teams will want to document these discussions and decisions for posterity.
+4. 🌎 We believe in the **power of examples** to bring the principles of data ethics to bear on human experience. This repository includes a [list of real-world examples](http://deon.drivendata.org/examples/) connected with each item in the default checklist. We encourage you to contribute relevant use cases that you believe can benefit the community by their example. In addition, if you have a topic, idea, or comment that doesn't seem right for the documentation, please add it to the [wiki page](https://github.com/drivendataorg/deon/wiki) for this project!
+5. 🔍 We believe the primary benefit of a checklist is **ensuring that we don't overlook important work**. Sometimes it is difficult with pressing deadlines and a demand to multitask to make sure we do the hard work to think about the big picture. This package is meant to help ensure that those discussions happen, even in fast-moving environments. Ethics is hard, and we expect some of the conversations that arise from this checklist may also be hard.
+6. ❓ We can't define exhaustively every term that appears in the checklist. Some of these **terms are open to interpretation** or mean different things in different contexts. We recommend that when relevant, users create their own glossary for reference.
+7. ✨ We want to avoid any items that strictly fall into the realm of statistical best practices. Instead, we want to highlight the areas where we need to pay particular attention **above and beyond best practices**.
+8. ✅ We want all the checklist items to be **as simple as possible** (but no simpler), and to be actionable.
 
-Third, we believe in the power of examples to bring the principles of data ethics to bear on human experience. This repository includes a [list of real-world examples](http://deon.drivendata.org/examples/) connected with each item in the default checklist. We encourage you to contribute relevant use cases that you believe can benefit the community by their example. In addition, if you have a topic, idea, or comment that doesn't seem right for the documentation, please add it to the [wiki page](https://github.com/drivendataorg/deon/wiki) for this project!
+## Sources
 
-Fourth, it's not up to data scientists alone to decide what the ethical course of action is. This has always been a responsibility of organizations that are part of civil society. This checklist is designed to provoke conversations around issues where data scientists have particular responsibility and perspective. This conversation should be part of a larger organizational commitment to doing what is right.
-
-Fifth, we believe the primary benefit of a checklist is ensuring that we don't overlook important work. Sometimes it is difficult with pressing deadlines and a demand to multitask to make sure we do the hard work to think about the big picture. This package is meant to help ensure that those discussions happen, even in fast-moving environments. Ethics is hard, and we expect some of the conversations that arise from this checklist may also be hard.
-
-Sixth, we are working at a level of abstraction that cannot concretely recommend a specific action (e.g., "remove variable X from your model"). Nearly all of the items on the checklist are meant to provoke discussion among good-faith actors who take their ethical responsibilities seriously. Because of this, most of the items are framed as prompts to discuss or consider. Teams will want to document these discussions and decisions for posterity.
-
-Seventh, we can't define exhaustively every term that appears in the checklist. Some of these terms are open to interpretation or mean different things in different contexts. We recommend that when relevant, users create their own glossary for reference.
-
-Eighth, we want to avoid any items that strictly fall into the realm of statistical best practices. Instead, we want to highlight the areas where we need to pay particular attention above and beyond best practices.
-
-Ninth, we want all the checklist items to be as simple as possible (but no simpler), and to be actionable.
+We built our initial list from a set of proposed items on [multiple checklists that we referenced](#checklist-citations). This checklist was heavily inspired by an article written by Mike Loukides, Hilary Mason, and DJ Patil and published by O'Reilly: ["Of Oaths and Checklists"](https://www.oreilly.com/ideas/of-oaths-and-checklists). We owe a great debt to the thinking that proceeded this, and we look forward to thoughtful engagement with the ongoing discussion about checklists for data science ethics.
 
 # Using this tool
 
diff --git a/docs/docs/index.md b/docs/docs/index.md
index c597afe..03fbd80 100644
--- a/docs/docs/index.md
+++ b/docs/docs/index.md
@@ -38,25 +38,21 @@ For more configuration details, see the sections on [command line options](#comm
 
 # Background and perspective
 
-We have a particular perspective with this package that we will use to make decisions about contributions, issues, PRs, and other maintenance and support activities.
+We created `deon` with the goal of helping data scientists across the sector to be more intentional in their choices, and more aware of the ethical implications of their work. We use that perspective to make decisions about contributions, issues, PRs, and other maintenance and support activities.
 
-First and foremost, our goal is not to be arbitrators of what ethical concerns merit inclusion. We have a [process for changing the default checklist](#changing-the-checklist), but we believe that many domain-specific concerns are not included and teams will benefit from developing [custom checklists](#custom-checklists). Not every checklist item will be relevant. We encourage teams to remove items, sections, or mark items as `N/A` as the concerns of their projects dictate.
 
-Second, we built our initial list from a set of proposed items on [multiple checklists that we referenced](#checklist-citations). This checklist was heavily inspired by an article written by Mike Loukides, Hilary Mason, and DJ Patil and published by O'Reilly: ["Of Oaths and Checklists"](https://www.oreilly.com/ideas/of-oaths-and-checklists). We owe a great debt to the thinking that proceeded this, and we look forward to thoughtful engagement with the ongoing discussion about checklists for data science ethics.
+1. 🔓 First and foremost, **our goal is not to be arbitrators of what ethical concerns merit inclusion**. We have a [process for changing the default checklist](#changing-the-checklist), but we believe that many domain-specific concerns are not included and teams will benefit from developing [custom checklists](#custom-checklists). Not every checklist item will be relevant. We encourage teams to remove items, sections, or mark items as `N/A` as the concerns of their projects dictate.
+2. 📊 It's not up to data scientists alone to decide what the ethical course of action is. This has always been a responsibility of organizations that are part of civil society. This checklist is designed to provoke conversations around **issues where data scientists have particular responsibility and perspective**. This conversation should be part of a larger organizational commitment to doing what is right.
+3. 💬 We are working at a level of abstraction that cannot concretely recommend a specific action (e.g., "remove variable X from your model"). Nearly all of the items on the checklist are **meant to provoke discussion** among good-faith actors who take their ethical responsibilities seriously. Because of this, most of the items are framed as prompts to discuss or consider. Teams will want to document these discussions and decisions for posterity.
+4. 🌎 We believe in the **power of examples** to bring the principles of data ethics to bear on human experience. This repository includes a [list of real-world examples](http://deon.drivendata.org/examples/) connected with each item in the default checklist. We encourage you to contribute relevant use cases that you believe can benefit the community by their example. In addition, if you have a topic, idea, or comment that doesn't seem right for the documentation, please add it to the [wiki page](https://github.com/drivendataorg/deon/wiki) for this project!
+5. 🔍 We believe the primary benefit of a checklist is **ensuring that we don't overlook important work**. Sometimes it is difficult with pressing deadlines and a demand to multitask to make sure we do the hard work to think about the big picture. This package is meant to help ensure that those discussions happen, even in fast-moving environments. Ethics is hard, and we expect some of the conversations that arise from this checklist may also be hard.
+6. ❓ We can't define exhaustively every term that appears in the checklist. Some of these **terms are open to interpretation** or mean different things in different contexts. We recommend that when relevant, users create their own glossary for reference.
+7. ✨ We want to avoid any items that strictly fall into the realm of statistical best practices. Instead, we want to highlight the areas where we need to pay particular attention **above and beyond best practices**.
+8. ✅ We want all the checklist items to be **as simple as possible** (but no simpler), and to be actionable.
 
-Third, we believe in the power of examples to bring the principles of data ethics to bear on human experience. This repository includes a [list of real-world examples](http://deon.drivendata.org/examples/) connected with each item in the default checklist. We encourage you to contribute relevant use cases that you believe can benefit the community by their example. In addition, if you have a topic, idea, or comment that doesn't seem right for the documentation, please add it to the [wiki page](https://github.com/drivendataorg/deon/wiki) for this project!
+## Sources
 
-Fourth, it's not up to data scientists alone to decide what the ethical course of action is. This has always been a responsibility of organizations that are part of civil society. This checklist is designed to provoke conversations around issues where data scientists have particular responsibility and perspective. This conversation should be part of a larger organizational commitment to doing what is right.
-
-Fifth, we believe the primary benefit of a checklist is ensuring that we don't overlook important work. Sometimes it is difficult with pressing deadlines and a demand to multitask to make sure we do the hard work to think about the big picture. This package is meant to help ensure that those discussions happen, even in fast-moving environments. Ethics is hard, and we expect some of the conversations that arise from this checklist may also be hard.
-
-Sixth, we are working at a level of abstraction that cannot concretely recommend a specific action (e.g., "remove variable X from your model"). Nearly all of the items on the checklist are meant to provoke discussion among good-faith actors who take their ethical responsibilities seriously. Because of this, most of the items are framed as prompts to discuss or consider. Teams will want to document these discussions and decisions for posterity.
-
-Seventh, we can't define exhaustively every term that appears in the checklist. Some of these terms are open to interpretation or mean different things in different contexts. We recommend that when relevant, users create their own glossary for reference.
-
-Eighth, we want to avoid any items that strictly fall into the realm of statistical best practices. Instead, we want to highlight the areas where we need to pay particular attention above and beyond best practices.
-
-Ninth, we want all the checklist items to be as simple as possible (but no simpler), and to be actionable.
+We built our initial list from a set of proposed items on [multiple checklists that we referenced](#checklist-citations). This checklist was heavily inspired by an article written by Mike Loukides, Hilary Mason, and DJ Patil and published by O'Reilly: ["Of Oaths and Checklists"](https://www.oreilly.com/ideas/of-oaths-and-checklists). We owe a great debt to the thinking that proceeded this, and we look forward to thoughtful engagement with the ongoing discussion about checklists for data science ethics.
 
 # Using this tool
 

From 3c417cd708b3d38111cb0475d43ac7337a4e5aa6 Mon Sep 17 00:00:00 2001
From: Katie Wetstone <klwetstone@gmail.com>
Date: Wed, 12 Jun 2024 11:05:29 -0400
Subject: [PATCH 03/20] list spacing

---
 README.md | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/README.md b/README.md
index 7bad827..017727a 100644
--- a/README.md
+++ b/README.md
@@ -49,12 +49,19 @@ We created `deon` with the goal of helping data scientists across the sector to
 
 
 1. 🔓 First and foremost, **our goal is not to be arbitrators of what ethical concerns merit inclusion**. We have a [process for changing the default checklist](#changing-the-checklist), but we believe that many domain-specific concerns are not included and teams will benefit from developing [custom checklists](#custom-checklists). Not every checklist item will be relevant. We encourage teams to remove items, sections, or mark items as `N/A` as the concerns of their projects dictate.
+
 2. 📊 It's not up to data scientists alone to decide what the ethical course of action is. This has always been a responsibility of organizations that are part of civil society. This checklist is designed to provoke conversations around **issues where data scientists have particular responsibility and perspective**. This conversation should be part of a larger organizational commitment to doing what is right.
+
 3. 💬 We are working at a level of abstraction that cannot concretely recommend a specific action (e.g., "remove variable X from your model"). Nearly all of the items on the checklist are **meant to provoke discussion** among good-faith actors who take their ethical responsibilities seriously. Because of this, most of the items are framed as prompts to discuss or consider. Teams will want to document these discussions and decisions for posterity.
+
 4. 🌎 We believe in the **power of examples** to bring the principles of data ethics to bear on human experience. This repository includes a [list of real-world examples](http://deon.drivendata.org/examples/) connected with each item in the default checklist. We encourage you to contribute relevant use cases that you believe can benefit the community by their example. In addition, if you have a topic, idea, or comment that doesn't seem right for the documentation, please add it to the [wiki page](https://github.com/drivendataorg/deon/wiki) for this project!
+
 5. 🔍 We believe the primary benefit of a checklist is **ensuring that we don't overlook important work**. Sometimes it is difficult with pressing deadlines and a demand to multitask to make sure we do the hard work to think about the big picture. This package is meant to help ensure that those discussions happen, even in fast-moving environments. Ethics is hard, and we expect some of the conversations that arise from this checklist may also be hard.
+
 6. ❓ We can't define exhaustively every term that appears in the checklist. Some of these **terms are open to interpretation** or mean different things in different contexts. We recommend that when relevant, users create their own glossary for reference.
+
 7. ✨ We want to avoid any items that strictly fall into the realm of statistical best practices. Instead, we want to highlight the areas where we need to pay particular attention **above and beyond best practices**.
+
 8. ✅ We want all the checklist items to be **as simple as possible** (but no simpler), and to be actionable.
 
 ## Sources

From 101a1ca08f1fe4aecbc213f486bdf57945a9be46 Mon Sep 17 00:00:00 2001
From: Katie Wetstone <klwetstone@gmail.com>
Date: Wed, 12 Jun 2024 11:09:24 -0400
Subject: [PATCH 04/20] final README updates

---
 README.md                          |  6 +++---
 docs/docs/index.md                 | 13 ++++++++++---
 docs/md_templates/_common_body.tpl | 13 ++++++++++---
 3 files changed, 23 insertions(+), 9 deletions(-)

diff --git a/README.md b/README.md
index 017727a..d7e11f1 100644
--- a/README.md
+++ b/README.md
@@ -43,16 +43,16 @@ Dig into the checklist questions to identify and navigate the ethical considerat
 
 For more configuration details, see the sections on [command line options](#command-line-options), [supported output file types](#supported-file-types), and [custom checklists](#custom-checklists).
 
-# Background and perspective
+# What is `deon` designed to do?
 
 We created `deon` with the goal of helping data scientists across the sector to be more intentional in their choices, and more aware of the ethical implications of their work. We use that perspective to make decisions about contributions, issues, PRs, and other maintenance and support activities.
 
 
 1. 🔓 First and foremost, **our goal is not to be arbitrators of what ethical concerns merit inclusion**. We have a [process for changing the default checklist](#changing-the-checklist), but we believe that many domain-specific concerns are not included and teams will benefit from developing [custom checklists](#custom-checklists). Not every checklist item will be relevant. We encourage teams to remove items, sections, or mark items as `N/A` as the concerns of their projects dictate.
 
-2. 📊 It's not up to data scientists alone to decide what the ethical course of action is. This has always been a responsibility of organizations that are part of civil society. This checklist is designed to provoke conversations around **issues where data scientists have particular responsibility and perspective**. This conversation should be part of a larger organizational commitment to doing what is right.
+2. 📊 This checklist is designed to provoke conversations around **issues where data scientists have particular responsibility and perspective**. It's not up to data scientists alone to decide what the ethical course of action is. This has always been a responsibility of organizations that are part of civil society. Conversations should be part of a larger organizational commitment to doing what is right.
 
-3. 💬 We are working at a level of abstraction that cannot concretely recommend a specific action (e.g., "remove variable X from your model"). Nearly all of the items on the checklist are **meant to provoke discussion** among good-faith actors who take their ethical responsibilities seriously. Because of this, most of the items are framed as prompts to discuss or consider. Teams will want to document these discussions and decisions for posterity.
+3. 💬 Items on the checklist are **meant to provoke discussion** among good-faith actors who take their ethical responsibilities seriously. We are working at a level of abstraction that cannot concretely recommend a specific action (e.g., "remove variable X from your model"). Because of this, most of the items are framed as prompts to discuss or consider. Teams will want to document these discussions and decisions for posterity.
 
 4. 🌎 We believe in the **power of examples** to bring the principles of data ethics to bear on human experience. This repository includes a [list of real-world examples](http://deon.drivendata.org/examples/) connected with each item in the default checklist. We encourage you to contribute relevant use cases that you believe can benefit the community by their example. In addition, if you have a topic, idea, or comment that doesn't seem right for the documentation, please add it to the [wiki page](https://github.com/drivendataorg/deon/wiki) for this project!
 
diff --git a/docs/docs/index.md b/docs/docs/index.md
index 03fbd80..daf1334 100644
--- a/docs/docs/index.md
+++ b/docs/docs/index.md
@@ -36,18 +36,25 @@ Dig into the checklist questions to identify and navigate the ethical considerat
 
 For more configuration details, see the sections on [command line options](#command-line-options), [supported output file types](#supported-file-types), and [custom checklists](#custom-checklists).
 
-# Background and perspective
+# What is `deon` designed to do?
 
 We created `deon` with the goal of helping data scientists across the sector to be more intentional in their choices, and more aware of the ethical implications of their work. We use that perspective to make decisions about contributions, issues, PRs, and other maintenance and support activities.
 
 
 1. 🔓 First and foremost, **our goal is not to be arbitrators of what ethical concerns merit inclusion**. We have a [process for changing the default checklist](#changing-the-checklist), but we believe that many domain-specific concerns are not included and teams will benefit from developing [custom checklists](#custom-checklists). Not every checklist item will be relevant. We encourage teams to remove items, sections, or mark items as `N/A` as the concerns of their projects dictate.
-2. 📊 It's not up to data scientists alone to decide what the ethical course of action is. This has always been a responsibility of organizations that are part of civil society. This checklist is designed to provoke conversations around **issues where data scientists have particular responsibility and perspective**. This conversation should be part of a larger organizational commitment to doing what is right.
-3. 💬 We are working at a level of abstraction that cannot concretely recommend a specific action (e.g., "remove variable X from your model"). Nearly all of the items on the checklist are **meant to provoke discussion** among good-faith actors who take their ethical responsibilities seriously. Because of this, most of the items are framed as prompts to discuss or consider. Teams will want to document these discussions and decisions for posterity.
+
+2. 📊 This checklist is designed to provoke conversations around **issues where data scientists have particular responsibility and perspective**. It's not up to data scientists alone to decide what the ethical course of action is. This has always been a responsibility of organizations that are part of civil society. Conversations should be part of a larger organizational commitment to doing what is right.
+
+3. 💬 Items on the checklist are **meant to provoke discussion** among good-faith actors who take their ethical responsibilities seriously. We are working at a level of abstraction that cannot concretely recommend a specific action (e.g., "remove variable X from your model"). Because of this, most of the items are framed as prompts to discuss or consider. Teams will want to document these discussions and decisions for posterity.
+
 4. 🌎 We believe in the **power of examples** to bring the principles of data ethics to bear on human experience. This repository includes a [list of real-world examples](http://deon.drivendata.org/examples/) connected with each item in the default checklist. We encourage you to contribute relevant use cases that you believe can benefit the community by their example. In addition, if you have a topic, idea, or comment that doesn't seem right for the documentation, please add it to the [wiki page](https://github.com/drivendataorg/deon/wiki) for this project!
+
 5. 🔍 We believe the primary benefit of a checklist is **ensuring that we don't overlook important work**. Sometimes it is difficult with pressing deadlines and a demand to multitask to make sure we do the hard work to think about the big picture. This package is meant to help ensure that those discussions happen, even in fast-moving environments. Ethics is hard, and we expect some of the conversations that arise from this checklist may also be hard.
+
 6. ❓ We can't define exhaustively every term that appears in the checklist. Some of these **terms are open to interpretation** or mean different things in different contexts. We recommend that when relevant, users create their own glossary for reference.
+
 7. ✨ We want to avoid any items that strictly fall into the realm of statistical best practices. Instead, we want to highlight the areas where we need to pay particular attention **above and beyond best practices**.
+
 8. ✅ We want all the checklist items to be **as simple as possible** (but no simpler), and to be actionable.
 
 ## Sources
diff --git a/docs/md_templates/_common_body.tpl b/docs/md_templates/_common_body.tpl
index 9f34e9f..1a2ed45 100644
--- a/docs/md_templates/_common_body.tpl
+++ b/docs/md_templates/_common_body.tpl
@@ -34,18 +34,25 @@ Dig into the checklist questions to identify and navigate the ethical considerat
 
 For more configuration details, see the sections on [command line options](#command-line-options), [supported output file types](#supported-file-types), and [custom checklists](#custom-checklists).
 
-# Background and perspective
+# What is `deon` designed to do?
 
 We created `deon` with the goal of helping data scientists across the sector to be more intentional in their choices, and more aware of the ethical implications of their work. We use that perspective to make decisions about contributions, issues, PRs, and other maintenance and support activities.
 
 
 1. 🔓 First and foremost, **our goal is not to be arbitrators of what ethical concerns merit inclusion**. We have a [process for changing the default checklist](#changing-the-checklist), but we believe that many domain-specific concerns are not included and teams will benefit from developing [custom checklists](#custom-checklists). Not every checklist item will be relevant. We encourage teams to remove items, sections, or mark items as `N/A` as the concerns of their projects dictate.
-2. 📊 It's not up to data scientists alone to decide what the ethical course of action is. This has always been a responsibility of organizations that are part of civil society. This checklist is designed to provoke conversations around **issues where data scientists have particular responsibility and perspective**. This conversation should be part of a larger organizational commitment to doing what is right.
-3. 💬 We are working at a level of abstraction that cannot concretely recommend a specific action (e.g., "remove variable X from your model"). Nearly all of the items on the checklist are **meant to provoke discussion** among good-faith actors who take their ethical responsibilities seriously. Because of this, most of the items are framed as prompts to discuss or consider. Teams will want to document these discussions and decisions for posterity.
+
+2. 📊 This checklist is designed to provoke conversations around **issues where data scientists have particular responsibility and perspective**. It's not up to data scientists alone to decide what the ethical course of action is. This has always been a responsibility of organizations that are part of civil society. Conversations should be part of a larger organizational commitment to doing what is right.
+
+3. 💬 Items on the checklist are **meant to provoke discussion** among good-faith actors who take their ethical responsibilities seriously. We are working at a level of abstraction that cannot concretely recommend a specific action (e.g., "remove variable X from your model"). Because of this, most of the items are framed as prompts to discuss or consider. Teams will want to document these discussions and decisions for posterity.
+
 4. 🌎 We believe in the **power of examples** to bring the principles of data ethics to bear on human experience. This repository includes a [list of real-world examples](http://deon.drivendata.org/examples/) connected with each item in the default checklist. We encourage you to contribute relevant use cases that you believe can benefit the community by their example. In addition, if you have a topic, idea, or comment that doesn't seem right for the documentation, please add it to the [wiki page](https://github.com/drivendataorg/deon/wiki) for this project!
+
 5. 🔍 We believe the primary benefit of a checklist is **ensuring that we don't overlook important work**. Sometimes it is difficult with pressing deadlines and a demand to multitask to make sure we do the hard work to think about the big picture. This package is meant to help ensure that those discussions happen, even in fast-moving environments. Ethics is hard, and we expect some of the conversations that arise from this checklist may also be hard.
+
 6. ❓ We can't define exhaustively every term that appears in the checklist. Some of these **terms are open to interpretation** or mean different things in different contexts. We recommend that when relevant, users create their own glossary for reference.
+
 7. ✨ We want to avoid any items that strictly fall into the realm of statistical best practices. Instead, we want to highlight the areas where we need to pay particular attention **above and beyond best practices**.
+
 8. ✅ We want all the checklist items to be **as simple as possible** (but no simpler), and to be actionable.
 
 ## Sources

From 331174eeaf53e67d0838664d928f0ac422456998 Mon Sep 17 00:00:00 2001
From: Katie Wetstone <klwetstone@gmail.com>
Date: Wed, 12 Jun 2024 11:26:30 -0400
Subject: [PATCH 05/20] test bullet formatting

---
 README.md                     | 5 +++++
 docs/docs/extra_css/extra.css | 5 +++++
 2 files changed, 10 insertions(+)

diff --git a/README.md b/README.md
index d7e11f1..9d88c2f 100644
--- a/README.md
+++ b/README.md
@@ -6,6 +6,11 @@
 
 ------
 
+<ul class="bad">
+  <li>item 1</li>
+  <li>item 2</li>
+</ul>
+
 
 <h1><b>An ethics checklist for data scientists</b></h1>
 
diff --git a/docs/docs/extra_css/extra.css b/docs/docs/extra_css/extra.css
index 7ef5f78..570b4aa 100644
--- a/docs/docs/extra_css/extra.css
+++ b/docs/docs/extra_css/extra.css
@@ -59,4 +59,9 @@ hr.checklist-buffer {
 	 margin-top: 3em;
 	 border: none;
 	 border-top: medium double #888;
+}
+
+ul.bad {
+	list-style-type: upper-roman;
+	color: red;
 }
\ No newline at end of file

From 64693013fbb1e8e9c98157e2e3d30ea8842e6ec7 Mon Sep 17 00:00:00 2001
From: Katie Wetstone <klwetstone@gmail.com>
Date: Wed, 12 Jun 2024 11:28:04 -0400
Subject: [PATCH 06/20] test css changes

---
 docs/docs/extra_css/extra.css | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/docs/docs/extra_css/extra.css b/docs/docs/extra_css/extra.css
index 570b4aa..9785902 100644
--- a/docs/docs/extra_css/extra.css
+++ b/docs/docs/extra_css/extra.css
@@ -56,9 +56,10 @@ div.document {
 }
 
 hr.checklist-buffer {
-	 margin-top: 3em;
+	 margin-top: 10em;
 	 border: none;
 	 border-top: medium double #888;
+	 color: red;
 }
 
 ul.bad {

From 33d77e4a718084ad61e541238ecf9941cfa11042 Mon Sep 17 00:00:00 2001
From: Katie Wetstone <klwetstone@gmail.com>
Date: Wed, 12 Jun 2024 11:28:50 -0400
Subject: [PATCH 07/20] undo css testing

---
 README.md                     | 5 -----
 docs/docs/extra_css/extra.css | 3 +--
 2 files changed, 1 insertion(+), 7 deletions(-)

diff --git a/README.md b/README.md
index 9d88c2f..d7e11f1 100644
--- a/README.md
+++ b/README.md
@@ -6,11 +6,6 @@
 
 ------
 
-<ul class="bad">
-  <li>item 1</li>
-  <li>item 2</li>
-</ul>
-
 
 <h1><b>An ethics checklist for data scientists</b></h1>
 
diff --git a/docs/docs/extra_css/extra.css b/docs/docs/extra_css/extra.css
index 9785902..570b4aa 100644
--- a/docs/docs/extra_css/extra.css
+++ b/docs/docs/extra_css/extra.css
@@ -56,10 +56,9 @@ div.document {
 }
 
 hr.checklist-buffer {
-	 margin-top: 10em;
+	 margin-top: 3em;
 	 border: none;
 	 border-top: medium double #888;
-	 color: red;
 }
 
 ul.bad {

From 8c2cb59d9fbd6123ba625f35457e39cf0b82bcc9 Mon Sep 17 00:00:00 2001
From: Katie Wetstone <klwetstone@gmail.com>
Date: Wed, 12 Jun 2024 11:50:08 -0400
Subject: [PATCH 08/20] add positive examples

---
 deon/assets/examples_of_ethical_issues.yml | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/deon/assets/examples_of_ethical_issues.yml b/deon/assets/examples_of_ethical_issues.yml
index a9fad18..2f695f9 100644
--- a/deon/assets/examples_of_ethical_issues.yml
+++ b/deon/assets/examples_of_ethical_issues.yml
@@ -26,6 +26,8 @@
       url: https://www.theregister.co.uk/2018/02/13/facial_recognition_software_is_better_at_white_men_than_black_women/
 - line_id: B.1
   links:
+    - text: MediCapt, which documents forensic evidence in conflict regions, effectively protects sensitive information using encryption, limited access, and security audits.
+      url: https://phr.org/issues/sexual-violence/medicapt/
     - text: Personal and financial data for more than 146 million people was stolen in Equifax data breach.
       url: https://www.nbcnews.com/news/us-news/equifax-breaks-down-just-how-bad-last-year-s-data-n872496
     - text: Cambridge Analytica harvested private information from over 50 million Facebook profiles without users' permission.
@@ -112,10 +114,14 @@
       url: hhttps://academic.oup.com/idpl/article/7/4/233/4762325
 - line_id: D.5
   links:
+    - text: OpenAI posted an explanation of how ChatGPT is trained to behave, its limitations, and future directions for improvement.
+      url:  https://openai.com/index/how-should-ai-systems-behave/
     - text: Google Flu claims to accurately predict weekly influenza activity and then misses the 2009 swine flu pandemic.
       url: https://www.forbes.com/sites/stevensalzberg/2014/03/23/why-google-flu-is-a-failure/#6fa6a1925535
 - line_id: E.1
   links:
+    - text: RobotsMali uses AI to create children's books in Mali's native languages, and incorporates human review to ensure that all AI-generated content is accurate and culturally sensitive.
+      url: https://restofworld.org/2024/mali-ai-translate-local-language-education/
     - text: Dutch Prime Minister and entire cabinet resign after investigations reveal that 26,000 innocent families were wrongly accused of social benefits fraud partially due to a discriminatory algorithm.
       url: https://www.vice.com/en/article/jgq35d/how-a-discriminatory-algorithm-wrongly-accused-thousands-of-families-of-fraud
     - text: Sending police officers to areas of high predicted crime skews future training data collection as police are repeatedly sent back to the same neighborhoods regardless of the true crime rate.

From f62bd44e4d093e743073760b33746ac097bedbec Mon Sep 17 00:00:00 2001
From: Katie Wetstone <klwetstone@gmail.com>
Date: Wed, 12 Jun 2024 16:19:40 -0400
Subject: [PATCH 09/20] categorize examples as positive or negative

---
 deon/assets/examples_of_ethical_issues.yml | 94 +++++++++++-----------
 docs/docs/examples.md                      | 42 +++++-----
 2 files changed, 68 insertions(+), 68 deletions(-)

diff --git a/deon/assets/examples_of_ethical_issues.yml b/deon/assets/examples_of_ethical_issues.yml
index 2f695f9..44fe09e 100644
--- a/deon/assets/examples_of_ethical_issues.yml
+++ b/deon/assets/examples_of_ethical_issues.yml
@@ -1,144 +1,144 @@
 - line_id: A.1
   links:
-    - text: Facebook uses phone numbers provided for two-factor authentication to target users with ads.
+    - text: ⛔ Facebook uses phone numbers provided for two-factor authentication to target users with ads.
       url: https://techcrunch.com/2018/09/27/yes-facebook-is-using-your-2fa-phone-number-to-target-you-with-ads/
-    - text: African-American men were enrolled in the Tuskegee Study on the progression of syphilis without being told the true purpose of the study or that treatment for syphilis was being withheld.
+    - text: ⛔ African-American men were enrolled in the Tuskegee Study on the progression of syphilis without being told the true purpose of the study or that treatment for syphilis was being withheld.
       url: https://en.wikipedia.org/wiki/Tuskegee_syphilis_experiment
-    - text: OpenAI's ChatGPT memorized and regurgitated entire poems without checking for copyright permissions.
+    - text: ⛔ OpenAI's ChatGPT memorized and regurgitated entire poems without checking for copyright permissions.
       url: https://news.cornell.edu/stories/2024/01/chatgpt-memorizes-and-spits-out-entire-poems
 - line_id: A.2
   links:
-    - text: StreetBump, a smartphone app to passively detect potholes, may fail to direct public resources to areas where smartphone penetration is lower, such as lower income areas or areas with a larger elderly population.
+    - text: ⛔ StreetBump, a smartphone app to passively detect potholes, may fail to direct public resources to areas where smartphone penetration is lower, such as lower income areas or areas with a larger elderly population.
       url: https://hbr.org/2013/04/the-hidden-biases-in-big-data
-    - text: Facial recognition cameras used for passport control register Asian's eyes as closed.
+    - text: ⛔ Facial recognition cameras used for passport control register Asian's eyes as closed.
       url: http://content.time.com/time/business/article/0,8599,1954643,00.html
 - line_id: A.3
   links:
-    - text: Personal information on taxi drivers can be accessed in poorly anonymized taxi trips dataset released by New York City.
+    - text: ⛔ Personal information on taxi drivers can be accessed in poorly anonymized taxi trips dataset released by New York City.
       url: https://www.theguardian.com/technology/2014/jun/27/new-york-taxi-details-anonymised-data-researchers-warn
-    - text: Netflix prize dataset of movie rankings by 500,000 customers is easily de-anonymized through cross referencing with other publicly available datasets.
+    - text: ⛔ Netflix prize dataset of movie rankings by 500,000 customers is easily de-anonymized through cross referencing with other publicly available datasets.
       url: https://www.wired.com/2007/12/why-anonymous-data-sometimes-isnt/
 - line_id: A.4
   links:
-    - text: In six major cities, Amazon's same day delivery service excludes many predominantly black neighborhoods.
+    - text: ⛔ In six major cities, Amazon's same day delivery service excludes many predominantly black neighborhoods.
       url: https://www.bloomberg.com/graphics/2016-amazon-same-day/
-    - text: Facial recognition software is significanty worse at identifying people with darker skin.
+    - text: ⛔ Facial recognition software is significanty worse at identifying people with darker skin.
       url: https://www.theregister.co.uk/2018/02/13/facial_recognition_software_is_better_at_white_men_than_black_women/
 - line_id: B.1
   links:
-    - text: MediCapt, which documents forensic evidence in conflict regions, effectively protects sensitive information using encryption, limited access, and security audits.
+    - text: ✅ MediCapt, which documents forensic evidence in conflict regions, effectively protects sensitive information using encryption, limited access, and security audits.
       url: https://phr.org/issues/sexual-violence/medicapt/
-    - text: Personal and financial data for more than 146 million people was stolen in Equifax data breach.
+    - text: ⛔ Personal and financial data for more than 146 million people was stolen in Equifax data breach.
       url: https://www.nbcnews.com/news/us-news/equifax-breaks-down-just-how-bad-last-year-s-data-n872496
-    - text: Cambridge Analytica harvested private information from over 50 million Facebook profiles without users' permission.
+    - text: ⛔ Cambridge Analytica harvested private information from over 50 million Facebook profiles without users' permission.
       url: https://www.nytimes.com/2018/03/17/us/politics/cambridge-analytica-trump-campaign.html
-    - text: AOL accidentally released 20 million search queries from 658,000 customers.
+    - text: ⛔ AOL accidentally released 20 million search queries from 658,000 customers.
       url: https://www.wired.com/2006/08/faq-aols-search-gaffe-and-you/
 - line_id: B.2
   links:
-    - text: The EU's General Data Protection Regulation (GDPR) includes the "right to be forgotten."
+    - text: ✅ The EU's General Data Protection Regulation (GDPR) includes the "right to be forgotten."
       url: https://www.eugdpr.org/the-regulation.html
 - line_id: B.3
   links:
-    - text: FedEx exposes private information of thousands of customers after a legacy s3 server was left open without a password.
+    - text: ⛔ FedEx exposes private information of thousands of customers after a legacy s3 server was left open without a password.
       url: https://www.zdnet.com/article/unsecured-server-exposes-fedex-customer-records/
 - line_id: C.1
   links:
-    - text: When Apple's HealthKit came out in 2014, women couldn't track menstruation.
+    - text: ⛔ When Apple's HealthKit came out in 2014, women couldn't track menstruation.
       url: https://www.theverge.com/2014/9/25/6844021/apple-promised-an-expansive-health-app-so-why-cant-i-track
 - line_id: C.2
   links:
-    - text: word2vec, trained on Google News corpus, reinforces gender stereotypes.
+    - text: ⛔ word2vec, trained on Google News corpus, reinforces gender stereotypes.
       url: https://www.technologyreview.com/s/602025/how-vector-space-mathematics-reveals-the-hidden-sexism-in-language/
-    - text: Women are more likely to be shown lower-paying jobs than men in Google ads.
+    - text: ⛔ Women are more likely to be shown lower-paying jobs than men in Google ads.
       url: https://www.theguardian.com/technology/2015/jul/08/women-less-likely-ads-high-paid-jobs-google-study
 - line_id: C.3
   links:
-    - text: Misleading chart shown at Planned Parenthood hearing distorts actual trends of abortions vs. cancer screenings and preventative services.
+    - text: ⛔ Misleading chart shown at Planned Parenthood hearing distorts actual trends of abortions vs. cancer screenings and preventative services.
       url: https://www.politifact.com/truth-o-meter/statements/2015/oct/01/jason-chaffetz/chart-shown-planned-parenthood-hearing-misleading-/
-    - text: Georgia Dept. of Health graph of COVID-19 cases falsely suggests a steeper decline when dates are ordered by total cases rather than chronologically.
+    - text: ⛔ Georgia Dept. of Health graph of COVID-19 cases falsely suggests a steeper decline when dates are ordered by total cases rather than chronologically.
       url: https://www.vox.com/covid-19-coronavirus-us-response-trump/2020/5/18/21262265/georgia-covid-19-cases-declining-reopening
 - line_id: C.4
   links:
-    - text: Strava heatmap of exercise routes reveals sensitive information on military bases and spy outposts.
+    - text: ⛔ Strava heatmap of exercise routes reveals sensitive information on military bases and spy outposts.
       url: https://www.theguardian.com/world/2018/jan/28/fitness-tracking-app-gives-away-location-of-secret-us-army-bases
 - line_id: C.5
   links:
-    - text: Excel error in well-known economics paper undermines justification of austerity measures.
+    - text: ⛔ Excel error in well-known economics paper undermines justification of austerity measures.
       url: https://www.bbc.com/news/magazine-22223190
 - line_id: D.1
   links:
-    - text: In hypothetical trials, language models assign the death penalty more frequently to defendants who use African American dialects.    
+    - text: ⛔ In hypothetical trials, language models assign the death penalty more frequently to defendants who use African American dialects.    
       url: https://arxiv.org/abs/2403.00742
-    - text: Variables used to predict child abuse and neglect are direct measurements of poverty, unfairly targeting low-income families for child welfare scrutiny.
+    - text: ⛔ Variables used to predict child abuse and neglect are direct measurements of poverty, unfairly targeting low-income families for child welfare scrutiny.
       url: https://www.wired.com/story/excerpt-from-automating-inequality/
-    - text: Amazon scraps AI recruiting tool that showed bias against women.
+    - text: ⛔ Amazon scraps AI recruiting tool that showed bias against women.
       url: https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G
-    - text: Criminal sentencing risk asessments don't ask directly about race or income, but other demographic factors can end up being proxies.
+    - text: ⛔ Criminal sentencing risk asessments don't ask directly about race or income, but other demographic factors can end up being proxies.
       url: https://www.themarshallproject.org/2015/08/04/the-new-science-of-sentencing
-    - text: Creditworthiness algorithms based on nontraditional criteria such as grammatic habits, preferred grocery stores, and friends' credit scores can perpetuate systemic bias.
+    - text: ⛔ Creditworthiness algorithms based on nontraditional criteria such as grammatic habits, preferred grocery stores, and friends' credit scores can perpetuate systemic bias.
       url: https://www.whitecase.com/publications/insight/algorithms-and-bias-what-lenders-need-know
 - line_id: D.2
   links:
-    - text: Apple credit card offers smaller lines of credit to women than men.
+    - text: ⛔ Apple credit card offers smaller lines of credit to women than men.
       url: https://www.wired.com/story/the-apple-card-didnt-see-genderand-thats-the-problem/
-    - text: Google Photos tags two African-Americans as gorillas.
+    - text: ⛔ Google Photos tags two African-Americans as gorillas.
       url: https://www.forbes.com/sites/mzhang/2015/07/01/google-photos-tags-two-african-americans-as-gorillas-through-facial-recognition-software/#12bdb1fd713d
-    - text: With COMPAS, a risk-assessment algorithm used in criminal sentencing, black defendants are almost twice as likely as white defendants to be mislabeled as likely to reoffend.
+    - text: ⛔ With COMPAS, a risk-assessment algorithm used in criminal sentencing, black defendants are almost twice as likely as white defendants to be mislabeled as likely to reoffend.
       url: https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
     - text: -- Northpointe's rebuttal to ProPublica article.
       url: https://www.documentcloud.org/documents/2998391-ProPublica-Commentary-Final-070616.html
     - text: -- Related academic study.
       url: https://www.liebertpub.com/doi/pdf/10.1089/big.2016.0047
-    - text: Google's speech recognition software doesn't recognize women's voices as well as men's.
+    - text: ⛔ Google's speech recognition software doesn't recognize women's voices as well as men's.
       url: https://www.dailydot.com/debug/google-voice-recognition-gender-bias/
-    - text: Google searches involving black-sounding names are more likely to serve up ads suggestive of a criminal record than white-sounding names.
+    - text: ⛔ Google searches involving black-sounding names are more likely to serve up ads suggestive of a criminal record than white-sounding names.
       url: https://www.technologyreview.com/s/510646/racism-is-poisoning-online-ad-delivery-says-harvard-professor/
     - text: -- Related academic study.
       url: https://arxiv.org/abs/1301.6822
-    - text: OpenAI's GPT models show racial bias in ranking job applications based on candidate names.
+    - text: ⛔ OpenAI's GPT models show racial bias in ranking job applications based on candidate names.
       url: https://www.bloomberg.com/graphics/2024-openai-gpt-hiring-racial-discrimination/
 - line_id: D.3
   links:
-    - text: Facebook seeks to optimize "time well spent", prioritizing interaction over popularity.
+    - text: ✅ Facebook seeks to optimize "time well spent", prioritizing interaction over popularity.
       url: https://www.wired.com/story/facebook-tweaks-newsfeed-to-favor-content-from-friends-family/
-    - text: YouTube's search autofill suggests pedophiliac phrases due to high viewership of related videos.
+    - text: ⛔ YouTube's search autofill suggests pedophiliac phrases due to high viewership of related videos.
       url: https://gizmodo.com/youtubes-creepy-kid-problem-was-worse-than-we-thought-1820763240
-    - text: A widely used commercial algorithm in the healthcare industry underestimates the care needs of black patients because it optimizes for spending as a proxy for need, introducing racial bias due to unequal access to care.
+    - text: ⛔ A widely used commercial algorithm in the healthcare industry underestimates the care needs of black patients because it optimizes for spending as a proxy for need, introducing racial bias due to unequal access to care.
       url: https://www.science.org/doi/10.1126/science.aax2342
 - line_id: D.4
   links:
-    - text: Patients with pneumonia with a history of asthma are usually admitted to the intensive care unit as they have a high risk of dying from pneumonia. Given the success of the intensive care, neural networks predicted asthmatics had a low risk of dying and could therefore be sent home. Without explanatory models to identify this issue, patients may have been sent home to die.
+    - text: ⛔ Patients with pneumonia with a history of asthma are usually admitted to the intensive care unit as they have a high risk of dying from pneumonia. Given the success of the intensive care, neural networks predicted asthmatics had a low risk of dying and could therefore be sent home. Without explanatory models to identify this issue, patients may have been sent home to die.
       url: http://people.dbmi.columbia.edu/noemie/papers/15kdd.pdf
     - text: GDPR includes a "right to explanation," i.e. meaningful information on the logic underlying automated decisions.
       url: hhttps://academic.oup.com/idpl/article/7/4/233/4762325
 - line_id: D.5
   links:
-    - text: OpenAI posted an explanation of how ChatGPT is trained to behave, its limitations, and future directions for improvement.
+    - text: ✅ OpenAI posted an explanation of how ChatGPT is trained to behave, its limitations, and future directions for improvement.
       url:  https://openai.com/index/how-should-ai-systems-behave/
-    - text: Google Flu claims to accurately predict weekly influenza activity and then misses the 2009 swine flu pandemic.
+    - text: ⛔ Google Flu claims to accurately predict weekly influenza activity and then misses the 2009 swine flu pandemic.
       url: https://www.forbes.com/sites/stevensalzberg/2014/03/23/why-google-flu-is-a-failure/#6fa6a1925535
 - line_id: E.1
   links:
-    - text: RobotsMali uses AI to create children's books in Mali's native languages, and incorporates human review to ensure that all AI-generated content is accurate and culturally sensitive.
+    - text: ✅ RobotsMali uses AI to create children's books in Mali's native languages, and incorporates human review to ensure that all AI-generated content is accurate and culturally sensitive.
       url: https://restofworld.org/2024/mali-ai-translate-local-language-education/
-    - text: Dutch Prime Minister and entire cabinet resign after investigations reveal that 26,000 innocent families were wrongly accused of social benefits fraud partially due to a discriminatory algorithm.
+    - text: ⛔ Dutch Prime Minister and entire cabinet resign after investigations reveal that 26,000 innocent families were wrongly accused of social benefits fraud partially due to a discriminatory algorithm.
       url: https://www.vice.com/en/article/jgq35d/how-a-discriminatory-algorithm-wrongly-accused-thousands-of-families-of-fraud
-    - text: Sending police officers to areas of high predicted crime skews future training data collection as police are repeatedly sent back to the same neighborhoods regardless of the true crime rate.
+    - text: ⛔ Sending police officers to areas of high predicted crime skews future training data collection as police are repeatedly sent back to the same neighborhoods regardless of the true crime rate.
       url: https://www.smithsonianmag.com/innovation/artificial-intelligence-is-now-used-predict-crime-is-it-biased-180968337/
 - line_id: E.2
   links:
-    - text: Software mistakes result in healthcare cuts for people with diabetes or cerebral palsy.
+    - text: ⛔ Software mistakes result in healthcare cuts for people with diabetes or cerebral palsy.
       url: https://www.theverge.com/2018/3/21/17144260/healthcare-medicaid-algorithm-arkansas-cerebral-palsy
 - line_id: E.3
   links:
-    - text: Google "fixes" racist algorithm by removing gorillas from image-labeling technology.
+    - text: ⛔ Google "fixes" racist algorithm by removing gorillas from image-labeling technology.
       url: https://www.theverge.com/2018/1/12/16882408/google-racist-gorillas-photo-recognition-algorithm-ai
-    - text: Microsoft's Twitter chatbot Tay quickly becomes racist.
+    - text: ⛔ Microsoft's Twitter chatbot Tay quickly becomes racist.
       url: https://www.theguardian.com/technology/2016/mar/24/microsoft-scrambles-limit-pr-damage-over-abusive-ai-bot-tay
 - line_id: E.4
   links:
-    - text: Generative AI can be exploited to create convincing scams like "virtual kidnapping".
+    - text: ⛔ Generative AI can be exploited to create convincing scams like "virtual kidnapping".
       url: https://www.trendmicro.com/vinfo/us/security/news/cybercrime-and-digital-threats/how-cybercriminals-can-perform-virtual-kidnapping-scams-using-ai-voice-cloning-tools-and-chatgpt
-    - text: Deepfakes—realistic but fake videos generated with AI—span the gamut from celebrity porn to presidential statements.
+    - text: ⛔ Deepfakes—realistic but fake videos generated with AI—span the gamut from celebrity porn to presidential statements.
       url: http://theweek.com/articles/777592/rise-deepfakes
diff --git a/docs/docs/examples.md b/docs/docs/examples.md
index 621f332..36ae1f5 100644
--- a/docs/docs/examples.md
+++ b/docs/docs/examples.md
@@ -7,28 +7,28 @@ To make the ideas contained in the checklist more concrete, we've compiled examp
 <center>Checklist Question</center> | <center>Examples of Ethical Issues</center>
 --- | ---
  | <center>**Data Collection**</center>
-**A.1 Informed consent**: If there are human subjects, have they given informed consent, where subjects affirmatively opt-in and have a clear understanding of the data uses to which they consent? | <ul><li>[Facebook uses phone numbers provided for two-factor authentication to target users with ads.](https://techcrunch.com/2018/09/27/yes-facebook-is-using-your-2fa-phone-number-to-target-you-with-ads/)</li><li>[African-American men were enrolled in the Tuskegee Study on the progression of syphilis without being told the true purpose of the study or that treatment for syphilis was being withheld.](https://en.wikipedia.org/wiki/Tuskegee_syphilis_experiment)</li><li>[OpenAI's ChatGPT memorized and regurgitated entire poems without checking for copyright permissions.](https://news.cornell.edu/stories/2024/01/chatgpt-memorizes-and-spits-out-entire-poems)</li></ul>
-**A.2 Collection bias**: Have we considered sources of bias that could be introduced during data collection and survey design and taken steps to mitigate those? | <ul><li>[StreetBump, a smartphone app to passively detect potholes, may fail to direct public resources to areas where smartphone penetration is lower, such as lower income areas or areas with a larger elderly population.](https://hbr.org/2013/04/the-hidden-biases-in-big-data)</li><li>[Facial recognition cameras used for passport control register Asian's eyes as closed.](http://content.time.com/time/business/article/0,8599,1954643,00.html)</li></ul>
-**A.3 Limit PII exposure**: Have we considered ways to minimize exposure of personally identifiable information (PII) for example through anonymization or not collecting information that isn't relevant for analysis? | <ul><li>[Personal information on taxi drivers can be accessed in poorly anonymized taxi trips dataset released by New York City.](https://www.theguardian.com/technology/2014/jun/27/new-york-taxi-details-anonymised-data-researchers-warn)</li><li>[Netflix prize dataset of movie rankings by 500,000 customers is easily de-anonymized through cross referencing with other publicly available datasets.](https://www.wired.com/2007/12/why-anonymous-data-sometimes-isnt/)</li></ul>
-**A.4 Downstream bias mitigation**: Have we considered ways to enable testing downstream results for biased outcomes (e.g., collecting data on protected group status like race or gender)? | <ul><li>[In six major cities, Amazon's same day delivery service excludes many predominantly black neighborhoods.](https://www.bloomberg.com/graphics/2016-amazon-same-day/)</li><li>[Facial recognition software is significanty worse at identifying people with darker skin.](https://www.theregister.co.uk/2018/02/13/facial_recognition_software_is_better_at_white_men_than_black_women/)</li></ul>
+**A.1 Informed consent**: If there are human subjects, have they given informed consent, where subjects affirmatively opt-in and have a clear understanding of the data uses to which they consent? | <ul><li>[⛔ Facebook uses phone numbers provided for two-factor authentication to target users with ads.](https://techcrunch.com/2018/09/27/yes-facebook-is-using-your-2fa-phone-number-to-target-you-with-ads/)</li><li>[⛔ African-American men were enrolled in the Tuskegee Study on the progression of syphilis without being told the true purpose of the study or that treatment for syphilis was being withheld.](https://en.wikipedia.org/wiki/Tuskegee_syphilis_experiment)</li><li>[⛔ OpenAI's ChatGPT memorized and regurgitated entire poems without checking for copyright permissions.](https://news.cornell.edu/stories/2024/01/chatgpt-memorizes-and-spits-out-entire-poems)</li></ul>
+**A.2 Collection bias**: Have we considered sources of bias that could be introduced during data collection and survey design and taken steps to mitigate those? | <ul><li>[⛔ StreetBump, a smartphone app to passively detect potholes, may fail to direct public resources to areas where smartphone penetration is lower, such as lower income areas or areas with a larger elderly population.](https://hbr.org/2013/04/the-hidden-biases-in-big-data)</li><li>[⛔ Facial recognition cameras used for passport control register Asian's eyes as closed.](http://content.time.com/time/business/article/0,8599,1954643,00.html)</li></ul>
+**A.3 Limit PII exposure**: Have we considered ways to minimize exposure of personally identifiable information (PII) for example through anonymization or not collecting information that isn't relevant for analysis? | <ul><li>[⛔ Personal information on taxi drivers can be accessed in poorly anonymized taxi trips dataset released by New York City.](https://www.theguardian.com/technology/2014/jun/27/new-york-taxi-details-anonymised-data-researchers-warn)</li><li>[⛔ Netflix prize dataset of movie rankings by 500,000 customers is easily de-anonymized through cross referencing with other publicly available datasets.](https://www.wired.com/2007/12/why-anonymous-data-sometimes-isnt/)</li></ul>
+**A.4 Downstream bias mitigation**: Have we considered ways to enable testing downstream results for biased outcomes (e.g., collecting data on protected group status like race or gender)? | <ul><li>[⛔ In six major cities, Amazon's same day delivery service excludes many predominantly black neighborhoods.](https://www.bloomberg.com/graphics/2016-amazon-same-day/)</li><li>[⛔ Facial recognition software is significanty worse at identifying people with darker skin.](https://www.theregister.co.uk/2018/02/13/facial_recognition_software_is_better_at_white_men_than_black_women/)</li></ul>
  | <center>**Data Storage**</center>
-**B.1 Data security**: Do we have a plan to protect and secure data (e.g., encryption at rest and in transit, access controls on internal users and third parties, access logs, and up-to-date software)? | <ul><li>[Personal and financial data for more than 146 million people was stolen in Equifax data breach.](https://www.nbcnews.com/news/us-news/equifax-breaks-down-just-how-bad-last-year-s-data-n872496)</li><li>[Cambridge Analytica harvested private information from over 50 million Facebook profiles without users' permission.](https://www.nytimes.com/2018/03/17/us/politics/cambridge-analytica-trump-campaign.html)</li><li>[AOL accidentally released 20 million search queries from 658,000 customers.](https://www.wired.com/2006/08/faq-aols-search-gaffe-and-you/)</li></ul>
-**B.2 Right to be forgotten**: Do we have a mechanism through which an individual can request their personal information be removed? | <ul><li>[The EU's General Data Protection Regulation (GDPR) includes the "right to be forgotten."](https://www.eugdpr.org/the-regulation.html)</li></ul>
-**B.3 Data retention plan**: Is there a schedule or plan to delete the data after it is no longer needed? | <ul><li>[FedEx exposes private information of thousands of customers after a legacy s3 server was left open without a password.](https://www.zdnet.com/article/unsecured-server-exposes-fedex-customer-records/)</li></ul>
+**B.1 Data security**: Do we have a plan to protect and secure data (e.g., encryption at rest and in transit, access controls on internal users and third parties, access logs, and up-to-date software)? | <ul><li>[✅ MediCapt, which documents forensic evidence in conflict regions, effectively protects sensitive information using encryption, limited access, and security audits.](https://phr.org/issues/sexual-violence/medicapt/)</li><li>[⛔ Personal and financial data for more than 146 million people was stolen in Equifax data breach.](https://www.nbcnews.com/news/us-news/equifax-breaks-down-just-how-bad-last-year-s-data-n872496)</li><li>[⛔ Cambridge Analytica harvested private information from over 50 million Facebook profiles without users' permission.](https://www.nytimes.com/2018/03/17/us/politics/cambridge-analytica-trump-campaign.html)</li><li>[⛔ AOL accidentally released 20 million search queries from 658,000 customers.](https://www.wired.com/2006/08/faq-aols-search-gaffe-and-you/)</li></ul>
+**B.2 Right to be forgotten**: Do we have a mechanism through which an individual can request their personal information be removed? | <ul><li>[✅ The EU's General Data Protection Regulation (GDPR) includes the "right to be forgotten."](https://www.eugdpr.org/the-regulation.html)</li></ul>
+**B.3 Data retention plan**: Is there a schedule or plan to delete the data after it is no longer needed? | <ul><li>[⛔ FedEx exposes private information of thousands of customers after a legacy s3 server was left open without a password.](https://www.zdnet.com/article/unsecured-server-exposes-fedex-customer-records/)</li></ul>
  | <center>**Analysis**</center>
-**C.1 Missing perspectives**: Have we sought to address blindspots in the analysis through engagement with relevant stakeholders (e.g., checking assumptions and discussing implications with affected communities and subject matter experts)? | <ul><li>[When Apple's HealthKit came out in 2014, women couldn't track menstruation.](https://www.theverge.com/2014/9/25/6844021/apple-promised-an-expansive-health-app-so-why-cant-i-track)</li></ul>
-**C.2 Dataset bias**: Have we examined the data for possible sources of bias and taken steps to mitigate or address these biases (e.g., stereotype perpetuation, confirmation bias, imbalanced classes, or omitted confounding variables)? | <ul><li>[word2vec, trained on Google News corpus, reinforces gender stereotypes.](https://www.technologyreview.com/s/602025/how-vector-space-mathematics-reveals-the-hidden-sexism-in-language/)</li><li>[Women are more likely to be shown lower-paying jobs than men in Google ads.](https://www.theguardian.com/technology/2015/jul/08/women-less-likely-ads-high-paid-jobs-google-study)</li></ul>
-**C.3 Honest representation**: Are our visualizations, summary statistics, and reports designed to honestly represent the underlying data? | <ul><li>[Misleading chart shown at Planned Parenthood hearing distorts actual trends of abortions vs. cancer screenings and preventative services.](https://www.politifact.com/truth-o-meter/statements/2015/oct/01/jason-chaffetz/chart-shown-planned-parenthood-hearing-misleading-/)</li><li>[Georgia Dept. of Health graph of COVID-19 cases falsely suggests a steeper decline when dates are ordered by total cases rather than chronologically.](https://www.vox.com/covid-19-coronavirus-us-response-trump/2020/5/18/21262265/georgia-covid-19-cases-declining-reopening)</li></ul>
-**C.4 Privacy in analysis**: Have we ensured that data with PII are not used or displayed unless necessary for the analysis? | <ul><li>[Strava heatmap of exercise routes reveals sensitive information on military bases and spy outposts.](https://www.theguardian.com/world/2018/jan/28/fitness-tracking-app-gives-away-location-of-secret-us-army-bases)</li></ul>
-**C.5 Auditability**: Is the process of generating the analysis well documented and reproducible if we discover issues in the future? | <ul><li>[Excel error in well-known economics paper undermines justification of austerity measures.](https://www.bbc.com/news/magazine-22223190)</li></ul>
+**C.1 Missing perspectives**: Have we sought to address blindspots in the analysis through engagement with relevant stakeholders (e.g., checking assumptions and discussing implications with affected communities and subject matter experts)? | <ul><li>[⛔ When Apple's HealthKit came out in 2014, women couldn't track menstruation.](https://www.theverge.com/2014/9/25/6844021/apple-promised-an-expansive-health-app-so-why-cant-i-track)</li></ul>
+**C.2 Dataset bias**: Have we examined the data for possible sources of bias and taken steps to mitigate or address these biases (e.g., stereotype perpetuation, confirmation bias, imbalanced classes, or omitted confounding variables)? | <ul><li>[⛔ word2vec, trained on Google News corpus, reinforces gender stereotypes.](https://www.technologyreview.com/s/602025/how-vector-space-mathematics-reveals-the-hidden-sexism-in-language/)</li><li>[⛔ Women are more likely to be shown lower-paying jobs than men in Google ads.](https://www.theguardian.com/technology/2015/jul/08/women-less-likely-ads-high-paid-jobs-google-study)</li></ul>
+**C.3 Honest representation**: Are our visualizations, summary statistics, and reports designed to honestly represent the underlying data? | <ul><li>[⛔ Misleading chart shown at Planned Parenthood hearing distorts actual trends of abortions vs. cancer screenings and preventative services.](https://www.politifact.com/truth-o-meter/statements/2015/oct/01/jason-chaffetz/chart-shown-planned-parenthood-hearing-misleading-/)</li><li>[⛔ Georgia Dept. of Health graph of COVID-19 cases falsely suggests a steeper decline when dates are ordered by total cases rather than chronologically.](https://www.vox.com/covid-19-coronavirus-us-response-trump/2020/5/18/21262265/georgia-covid-19-cases-declining-reopening)</li></ul>
+**C.4 Privacy in analysis**: Have we ensured that data with PII are not used or displayed unless necessary for the analysis? | <ul><li>[⛔ Strava heatmap of exercise routes reveals sensitive information on military bases and spy outposts.](https://www.theguardian.com/world/2018/jan/28/fitness-tracking-app-gives-away-location-of-secret-us-army-bases)</li></ul>
+**C.5 Auditability**: Is the process of generating the analysis well documented and reproducible if we discover issues in the future? | <ul><li>[⛔ Excel error in well-known economics paper undermines justification of austerity measures.](https://www.bbc.com/news/magazine-22223190)</li></ul>
  | <center>**Modeling**</center>
-**D.1 Proxy discrimination**: Have we ensured that the model does not rely on variables or proxies for variables that are unfairly discriminatory? | <ul><li>[In hypothetical trials, language models assign the death penalty more frequently to defendants who use African American dialects.](https://arxiv.org/abs/2403.00742)</li><li>[Variables used to predict child abuse and neglect are direct measurements of poverty, unfairly targeting low-income families for child welfare scrutiny.](https://www.wired.com/story/excerpt-from-automating-inequality/)</li><li>[Amazon scraps AI recruiting tool that showed bias against women.](https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G)</li><li>[Criminal sentencing risk asessments don't ask directly about race or income, but other demographic factors can end up being proxies.](https://www.themarshallproject.org/2015/08/04/the-new-science-of-sentencing)</li><li>[Creditworthiness algorithms based on nontraditional criteria such as grammatic habits, preferred grocery stores, and friends' credit scores can perpetuate systemic bias.](https://www.whitecase.com/publications/insight/algorithms-and-bias-what-lenders-need-know)</li></ul>
-**D.2 Fairness across groups**: Have we tested model results for fairness with respect to different affected groups (e.g., tested for disparate error rates)? | <ul><li>[Apple credit card offers smaller lines of credit to women than men.](https://www.wired.com/story/the-apple-card-didnt-see-genderand-thats-the-problem/)</li><li>[Google Photos tags two African-Americans as gorillas.](https://www.forbes.com/sites/mzhang/2015/07/01/google-photos-tags-two-african-americans-as-gorillas-through-facial-recognition-software/#12bdb1fd713d)</li><li>[With COMPAS, a risk-assessment algorithm used in criminal sentencing, black defendants are almost twice as likely as white defendants to be mislabeled as likely to reoffend.](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing)</li><li>[-- Northpointe's rebuttal to ProPublica article.](https://www.documentcloud.org/documents/2998391-ProPublica-Commentary-Final-070616.html)</li><li>[-- Related academic study.](https://www.liebertpub.com/doi/pdf/10.1089/big.2016.0047)</li><li>[Google's speech recognition software doesn't recognize women's voices as well as men's.](https://www.dailydot.com/debug/google-voice-recognition-gender-bias/)</li><li>[Google searches involving black-sounding names are more likely to serve up ads suggestive of a criminal record than white-sounding names.](https://www.technologyreview.com/s/510646/racism-is-poisoning-online-ad-delivery-says-harvard-professor/)</li><li>[-- Related academic study.](https://arxiv.org/abs/1301.6822)</li><li>[OpenAI's GPT models show racial bias in ranking job applications based on candidate names.](https://www.bloomberg.com/graphics/2024-openai-gpt-hiring-racial-discrimination/)</li></ul>
-**D.3 Metric selection**: Have we considered the effects of optimizing for our defined metrics and considered additional metrics? | <ul><li>[Facebook seeks to optimize "time well spent", prioritizing interaction over popularity.](https://www.wired.com/story/facebook-tweaks-newsfeed-to-favor-content-from-friends-family/)</li><li>[YouTube's search autofill suggests pedophiliac phrases due to high viewership of related videos.](https://gizmodo.com/youtubes-creepy-kid-problem-was-worse-than-we-thought-1820763240)</li><li>[A widely used commercial algorithm in the healthcare industry underestimates the care needs of black patients because it optimizes for spending as a proxy for need, introducing racial bias due to unequal access to care.](https://www.science.org/doi/10.1126/science.aax2342)</li></ul>
-**D.4 Explainability**: Can we explain in understandable terms a decision the model made in cases where a justification is needed? | <ul><li>[Patients with pneumonia with a history of asthma are usually admitted to the intensive care unit as they have a high risk of dying from pneumonia. Given the success of the intensive care, neural networks predicted asthmatics had a low risk of dying and could therefore be sent home. Without explanatory models to identify this issue, patients may have been sent home to die.](http://people.dbmi.columbia.edu/noemie/papers/15kdd.pdf)</li><li>[GDPR includes a "right to explanation," i.e. meaningful information on the logic underlying automated decisions.](hhttps://academic.oup.com/idpl/article/7/4/233/4762325)</li></ul>
-**D.5 Communicate limitations**: Have we communicated the shortcomings, limitations, and biases of the model to relevant stakeholders in ways that can be generally understood? | <ul><li>[Google Flu claims to accurately predict weekly influenza activity and then misses the 2009 swine flu pandemic.](https://www.forbes.com/sites/stevensalzberg/2014/03/23/why-google-flu-is-a-failure/#6fa6a1925535)</li></ul>
+**D.1 Proxy discrimination**: Have we ensured that the model does not rely on variables or proxies for variables that are unfairly discriminatory? | <ul><li>[⛔ In hypothetical trials, language models assign the death penalty more frequently to defendants who use African American dialects.](https://arxiv.org/abs/2403.00742)</li><li>[⛔ Variables used to predict child abuse and neglect are direct measurements of poverty, unfairly targeting low-income families for child welfare scrutiny.](https://www.wired.com/story/excerpt-from-automating-inequality/)</li><li>[⛔ Amazon scraps AI recruiting tool that showed bias against women.](https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G)</li><li>[⛔ Criminal sentencing risk asessments don't ask directly about race or income, but other demographic factors can end up being proxies.](https://www.themarshallproject.org/2015/08/04/the-new-science-of-sentencing)</li><li>[⛔ Creditworthiness algorithms based on nontraditional criteria such as grammatic habits, preferred grocery stores, and friends' credit scores can perpetuate systemic bias.](https://www.whitecase.com/publications/insight/algorithms-and-bias-what-lenders-need-know)</li></ul>
+**D.2 Fairness across groups**: Have we tested model results for fairness with respect to different affected groups (e.g., tested for disparate error rates)? | <ul><li>[⛔ Apple credit card offers smaller lines of credit to women than men.](https://www.wired.com/story/the-apple-card-didnt-see-genderand-thats-the-problem/)</li><li>[⛔ Google Photos tags two African-Americans as gorillas.](https://www.forbes.com/sites/mzhang/2015/07/01/google-photos-tags-two-african-americans-as-gorillas-through-facial-recognition-software/#12bdb1fd713d)</li><li>[⛔ With COMPAS, a risk-assessment algorithm used in criminal sentencing, black defendants are almost twice as likely as white defendants to be mislabeled as likely to reoffend.](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing)</li><li>[-- Northpointe's rebuttal to ProPublica article.](https://www.documentcloud.org/documents/2998391-ProPublica-Commentary-Final-070616.html)</li><li>[-- Related academic study.](https://www.liebertpub.com/doi/pdf/10.1089/big.2016.0047)</li><li>[⛔ Google's speech recognition software doesn't recognize women's voices as well as men's.](https://www.dailydot.com/debug/google-voice-recognition-gender-bias/)</li><li>[⛔ Google searches involving black-sounding names are more likely to serve up ads suggestive of a criminal record than white-sounding names.](https://www.technologyreview.com/s/510646/racism-is-poisoning-online-ad-delivery-says-harvard-professor/)</li><li>[-- Related academic study.](https://arxiv.org/abs/1301.6822)</li><li>[⛔ OpenAI's GPT models show racial bias in ranking job applications based on candidate names.](https://www.bloomberg.com/graphics/2024-openai-gpt-hiring-racial-discrimination/)</li></ul>
+**D.3 Metric selection**: Have we considered the effects of optimizing for our defined metrics and considered additional metrics? | <ul><li>[✅ Facebook seeks to optimize "time well spent", prioritizing interaction over popularity.](https://www.wired.com/story/facebook-tweaks-newsfeed-to-favor-content-from-friends-family/)</li><li>[⛔ YouTube's search autofill suggests pedophiliac phrases due to high viewership of related videos.](https://gizmodo.com/youtubes-creepy-kid-problem-was-worse-than-we-thought-1820763240)</li><li>[⛔ A widely used commercial algorithm in the healthcare industry underestimates the care needs of black patients because it optimizes for spending as a proxy for need, introducing racial bias due to unequal access to care.](https://www.science.org/doi/10.1126/science.aax2342)</li></ul>
+**D.4 Explainability**: Can we explain in understandable terms a decision the model made in cases where a justification is needed? | <ul><li>[⛔ Patients with pneumonia with a history of asthma are usually admitted to the intensive care unit as they have a high risk of dying from pneumonia. Given the success of the intensive care, neural networks predicted asthmatics had a low risk of dying and could therefore be sent home. Without explanatory models to identify this issue, patients may have been sent home to die.](http://people.dbmi.columbia.edu/noemie/papers/15kdd.pdf)</li><li>[GDPR includes a "right to explanation," i.e. meaningful information on the logic underlying automated decisions.](hhttps://academic.oup.com/idpl/article/7/4/233/4762325)</li></ul>
+**D.5 Communicate limitations**: Have we communicated the shortcomings, limitations, and biases of the model to relevant stakeholders in ways that can be generally understood? | <ul><li>[✅ OpenAI posted an explanation of how ChatGPT is trained to behave, its limitations, and future directions for improvement.](https://openai.com/index/how-should-ai-systems-behave/)</li><li>[⛔ Google Flu claims to accurately predict weekly influenza activity and then misses the 2009 swine flu pandemic.](https://www.forbes.com/sites/stevensalzberg/2014/03/23/why-google-flu-is-a-failure/#6fa6a1925535)</li></ul>
  | <center>**Deployment**</center>
-**E.1 Monitoring and evaluation**: Do we have a clear plan to monitor the model and its impacts after it is deployed (e.g., performance monitoring, regular audit of sample predictions, human review of high-stakes decisions, reviewing downstream impacts of errors or low-confidence decisions, testing for concept drift)? | <ul><li>[Dutch Prime Minister and entire cabinet resign after investigations reveal that 26,000 innocent families were wrongly accused of social benefits fraud partially due to a discriminatory algorithm.](https://www.vice.com/en/article/jgq35d/how-a-discriminatory-algorithm-wrongly-accused-thousands-of-families-of-fraud)</li><li>[Sending police officers to areas of high predicted crime skews future training data collection as police are repeatedly sent back to the same neighborhoods regardless of the true crime rate.](https://www.smithsonianmag.com/innovation/artificial-intelligence-is-now-used-predict-crime-is-it-biased-180968337/)</li></ul>
-**E.2 Redress**: Have we discussed with our organization a plan for response if users are harmed by the results (e.g., how does the data science team evaluate these cases and update analysis and models to prevent future harm)? | <ul><li>[Software mistakes result in healthcare cuts for people with diabetes or cerebral palsy.](https://www.theverge.com/2018/3/21/17144260/healthcare-medicaid-algorithm-arkansas-cerebral-palsy)</li></ul>
-**E.3 Roll back**: Is there a way to turn off or roll back the model in production if necessary? | <ul><li>[Google "fixes" racist algorithm by removing gorillas from image-labeling technology.](https://www.theverge.com/2018/1/12/16882408/google-racist-gorillas-photo-recognition-algorithm-ai)</li><li>[Microsoft's Twitter chatbot Tay quickly becomes racist.](https://www.theguardian.com/technology/2016/mar/24/microsoft-scrambles-limit-pr-damage-over-abusive-ai-bot-tay)</li></ul>
-**E.4 Unintended use**: Have we taken steps to identify and prevent unintended uses and abuse of the model and do we have a plan to monitor these once the model is deployed? | <ul><li>[Generative AI can be exploited to create convincing scams like "virtual kidnapping".](https://www.trendmicro.com/vinfo/us/security/news/cybercrime-and-digital-threats/how-cybercriminals-can-perform-virtual-kidnapping-scams-using-ai-voice-cloning-tools-and-chatgpt)</li><li>[Deepfakes—realistic but fake videos generated with AI—span the gamut from celebrity porn to presidential statements.](http://theweek.com/articles/777592/rise-deepfakes)</li></ul>
+**E.1 Monitoring and evaluation**: Do we have a clear plan to monitor the model and its impacts after it is deployed (e.g., performance monitoring, regular audit of sample predictions, human review of high-stakes decisions, reviewing downstream impacts of errors or low-confidence decisions, testing for concept drift)? | <ul><li>[✅ RobotsMali uses AI to create children's books in Mali's native languages, and incorporates human review to ensure that all AI-generated content is accurate and culturally sensitive.](https://restofworld.org/2024/mali-ai-translate-local-language-education/)</li><li>[⛔ Dutch Prime Minister and entire cabinet resign after investigations reveal that 26,000 innocent families were wrongly accused of social benefits fraud partially due to a discriminatory algorithm.](https://www.vice.com/en/article/jgq35d/how-a-discriminatory-algorithm-wrongly-accused-thousands-of-families-of-fraud)</li><li>[⛔ Sending police officers to areas of high predicted crime skews future training data collection as police are repeatedly sent back to the same neighborhoods regardless of the true crime rate.](https://www.smithsonianmag.com/innovation/artificial-intelligence-is-now-used-predict-crime-is-it-biased-180968337/)</li></ul>
+**E.2 Redress**: Have we discussed with our organization a plan for response if users are harmed by the results (e.g., how does the data science team evaluate these cases and update analysis and models to prevent future harm)? | <ul><li>[⛔ Software mistakes result in healthcare cuts for people with diabetes or cerebral palsy.](https://www.theverge.com/2018/3/21/17144260/healthcare-medicaid-algorithm-arkansas-cerebral-palsy)</li></ul>
+**E.3 Roll back**: Is there a way to turn off or roll back the model in production if necessary? | <ul><li>[⛔ Google "fixes" racist algorithm by removing gorillas from image-labeling technology.](https://www.theverge.com/2018/1/12/16882408/google-racist-gorillas-photo-recognition-algorithm-ai)</li><li>[⛔ Microsoft's Twitter chatbot Tay quickly becomes racist.](https://www.theguardian.com/technology/2016/mar/24/microsoft-scrambles-limit-pr-damage-over-abusive-ai-bot-tay)</li></ul>
+**E.4 Unintended use**: Have we taken steps to identify and prevent unintended uses and abuse of the model and do we have a plan to monitor these once the model is deployed? | <ul><li>[⛔ Generative AI can be exploited to create convincing scams like "virtual kidnapping".](https://www.trendmicro.com/vinfo/us/security/news/cybercrime-and-digital-threats/how-cybercriminals-can-perform-virtual-kidnapping-scams-using-ai-voice-cloning-tools-and-chatgpt)</li><li>[⛔ Deepfakes—realistic but fake videos generated with AI—span the gamut from celebrity porn to presidential statements.](http://theweek.com/articles/777592/rise-deepfakes)</li></ul>

From d62745e4b9edf28ec3ea87bcd9ac5c1e4a7195af Mon Sep 17 00:00:00 2001
From: Katie Wetstone <klwetstone@gmail.com>
Date: Wed, 12 Jun 2024 16:26:09 -0400
Subject: [PATCH 10/20] remove repeat gorilla story

---
 deon/assets/examples_of_ethical_issues.yml | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/deon/assets/examples_of_ethical_issues.yml b/deon/assets/examples_of_ethical_issues.yml
index 44fe09e..871258b 100644
--- a/deon/assets/examples_of_ethical_issues.yml
+++ b/deon/assets/examples_of_ethical_issues.yml
@@ -82,8 +82,6 @@
   links:
     - text: ⛔ Apple credit card offers smaller lines of credit to women than men.
       url: https://www.wired.com/story/the-apple-card-didnt-see-genderand-thats-the-problem/
-    - text: ⛔ Google Photos tags two African-Americans as gorillas.
-      url: https://www.forbes.com/sites/mzhang/2015/07/01/google-photos-tags-two-african-americans-as-gorillas-through-facial-recognition-software/#12bdb1fd713d
     - text: ⛔ With COMPAS, a risk-assessment algorithm used in criminal sentencing, black defendants are almost twice as likely as white defendants to be mislabeled as likely to reoffend.
       url: https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
     - text: -- Northpointe's rebuttal to ProPublica article.
@@ -108,10 +106,10 @@
       url: https://www.science.org/doi/10.1126/science.aax2342
 - line_id: D.4
   links:
+    - text: ✅ GDPR includes a "right to explanation," i.e. meaningful information on the logic underlying automated decisions.
+      url: hhttps://academic.oup.com/idpl/article/7/4/233/4762325
     - text: ⛔ Patients with pneumonia with a history of asthma are usually admitted to the intensive care unit as they have a high risk of dying from pneumonia. Given the success of the intensive care, neural networks predicted asthmatics had a low risk of dying and could therefore be sent home. Without explanatory models to identify this issue, patients may have been sent home to die.
       url: http://people.dbmi.columbia.edu/noemie/papers/15kdd.pdf
-    - text: GDPR includes a "right to explanation," i.e. meaningful information on the logic underlying automated decisions.
-      url: hhttps://academic.oup.com/idpl/article/7/4/233/4762325
 - line_id: D.5
   links:
     - text: ✅ OpenAI posted an explanation of how ChatGPT is trained to behave, its limitations, and future directions for improvement.

From 8c37cfa005c0b9973c0aa002c4a037634889bccb Mon Sep 17 00:00:00 2001
From: Katie Wetstone <klwetstone@gmail.com>
Date: Wed, 12 Jun 2024 16:30:32 -0400
Subject: [PATCH 11/20] adjust section description

---
 docs/md_templates/_common_body.tpl | 6 +++---
 docs/md_templates/examples.tpl     | 4 ++--
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/docs/md_templates/_common_body.tpl b/docs/md_templates/_common_body.tpl
index 1a2ed45..07260fc 100644
--- a/docs/md_templates/_common_body.tpl
+++ b/docs/md_templates/_common_body.tpl
@@ -198,9 +198,9 @@ We're excited to see so many articles popping up on data ethics! The short list
 - [Technology is biased too. How do we fix it?](https://fivethirtyeight.com/features/technology-is-biased-too-how-do-we-fix-it/)
 - [The dark secret at the heart of AI](https://www.technologyreview.com/s/604087/the-dark-secret-at-the-heart-of-ai/)
 
-## Where things have gone wrong
+## Data ethics in the real world
 
-To make the ideas contained in the checklist more concrete, we've compiled [examples](http://deon.drivendata.org/examples/) of times when things have gone wrong. They're paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction.
+To make the ideas contained in the checklist more concrete, we've compiled [examples](http://deon.drivendata.org/examples/) of times when tradoffs were handled well, and times when things have gone wrong. They're paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction.
 
 We welcome contributions! Follow [these instructions](https://github.com/drivendataorg/deon/blob/main/CONTRIBUTING.md) to add an example.
 
@@ -211,4 +211,4 @@ There are other groups working on data ethics and thinking about how tools can h
 - [Aequitas](https://dsapp.uchicago.edu/aequitas/) ([github](https://github.com/dssg/aequitas))
 - [Ethical OS Toolkit](https://ethicalos.org/)
 - [Ethics & Algorithms Toolkit: A risk management framework for governments](http://ethicstoolkit.ai/)
-- Ethics and Data Science ([free ebook](https://www.amazon.com/dp/B07GTC8ZN7/ref=cm_sw_r_cp_ep_dp_klAOBb4Z72B4G)) and ([write-up](https://medium.com/@sjgadler/care-about-ai-ethics-what-you-can-do-starting-today-882a0e63d828))
+- Ethics and Data Science ([free ebook](https://www.amazon.com/dp/B07GTC8ZN7/ref=cm_sw_r_cp_ep_dp_klAOBb4Z72B4G)) and ([write-up](https://medium.com/@sjgadler/care-about-ai-ethics-what-you-can-do-starting-today-882a0e63d828))
\ No newline at end of file
diff --git a/docs/md_templates/examples.tpl b/docs/md_templates/examples.tpl
index 31808e3..b17a19f 100644
--- a/docs/md_templates/examples.tpl
+++ b/docs/md_templates/examples.tpl
@@ -1,7 +1,7 @@
 <div style="display: inline-block; line-height: 2.25em">&nbsp;</div>
 
-# Where things have gone wrong
+# Data ethics in the real world
 
-To make the ideas contained in the checklist more concrete, we've compiled examples of times when things have gone wrong. They're paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction.
+To make the ideas contained in the checklist more concrete, we've compiled **examples** of times when tradoffs were handled well, and times when things have gone wrong. Examples are paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction. The positive examples show how principle's of `deon` can be followed in the real world.
 
 {{ links_table }}
\ No newline at end of file

From a6d22afd00dfcc6d0e79a927dcd05e7974d783c4 Mon Sep 17 00:00:00 2001
From: Katie Wetstone <klwetstone@gmail.com>
Date: Wed, 12 Jun 2024 17:08:07 -0400
Subject: [PATCH 12/20] add a few more positive examples

---
 deon/assets/examples_of_ethical_issues.yml | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/deon/assets/examples_of_ethical_issues.yml b/deon/assets/examples_of_ethical_issues.yml
index 871258b..672ad93 100644
--- a/deon/assets/examples_of_ethical_issues.yml
+++ b/deon/assets/examples_of_ethical_issues.yml
@@ -44,10 +44,14 @@
       url: https://www.zdnet.com/article/unsecured-server-exposes-fedex-customer-records/
 - line_id: C.1
   links:
+    - text: ✅ Code for America programmatically cleared >140,000 eligible criminal records by collaborating with multiple relevant stakeholders like policymakers, advocacy groups, and courts.
+      url: https://codeforamerica.org/programs/criminal-justice/automatic-record-clearance/
     - text: ⛔ When Apple's HealthKit came out in 2014, women couldn't track menstruation.
       url: https://www.theverge.com/2014/9/25/6844021/apple-promised-an-expansive-health-app-so-why-cant-i-track
 - line_id: C.2
   links:
+    - text: ✅ A study by Park et al shows how reweighting can effectively mitigate algorithmic racial bias when predicting risk of postpartum depression.
+      url: https://doi.org/10.1001/jamanetworkopen.2021.3909 
     - text: ⛔ word2vec, trained on Google News corpus, reinforces gender stereotypes.
       url: https://www.technologyreview.com/s/602025/how-vector-space-mathematics-reveals-the-hidden-sexism-in-language/
     - text: ⛔ Women are more likely to be shown lower-paying jobs than men in Google ads.
@@ -80,6 +84,8 @@
       url: https://www.whitecase.com/publications/insight/algorithms-and-bias-what-lenders-need-know
 - line_id: D.2
   links:
+    - text: ✅ A study by Garriga et al uses ML best practices to test for and communicate fairness across racial groups for a model that predicts mental health crises. 
+      url: https://www.nature.com/articles/s41591-022-01811-5
     - text: ⛔ Apple credit card offers smaller lines of credit to women than men.
       url: https://www.wired.com/story/the-apple-card-didnt-see-genderand-thats-the-problem/
     - text: ⛔ With COMPAS, a risk-assessment algorithm used in criminal sentencing, black defendants are almost twice as likely as white defendants to be mislabeled as likely to reoffend.

From e8794d1682ad1daf7974de1b1e834c2033d23baa Mon Sep 17 00:00:00 2001
From: Katie Wetstone <klwetstone@gmail.com>
Date: Wed, 12 Jun 2024 17:08:53 -0400
Subject: [PATCH 13/20] update md files

---
 README.md             |  4 ++--
 docs/docs/examples.md | 12 ++++++------
 docs/docs/index.md    |  4 ++--
 3 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/README.md b/README.md
index d7e11f1..4c9c374 100644
--- a/README.md
+++ b/README.md
@@ -269,9 +269,9 @@ We're excited to see so many articles popping up on data ethics! The short list
 - [Technology is biased too. How do we fix it?](https://fivethirtyeight.com/features/technology-is-biased-too-how-do-we-fix-it/)
 - [The dark secret at the heart of AI](https://www.technologyreview.com/s/604087/the-dark-secret-at-the-heart-of-ai/)
 
-## Where things have gone wrong
+## Data ethics in the real world
 
-To make the ideas contained in the checklist more concrete, we've compiled [examples](http://deon.drivendata.org/examples/) of times when things have gone wrong. They're paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction.
+To make the ideas contained in the checklist more concrete, we've compiled [examples](http://deon.drivendata.org/examples/) of times when tradoffs were handled well, and times when things have gone wrong. They're paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction.
 
 We welcome contributions! Follow [these instructions](https://github.com/drivendataorg/deon/blob/main/CONTRIBUTING.md) to add an example.
 
diff --git a/docs/docs/examples.md b/docs/docs/examples.md
index 36ae1f5..b69569f 100644
--- a/docs/docs/examples.md
+++ b/docs/docs/examples.md
@@ -1,8 +1,8 @@
 <div style="display: inline-block; line-height: 2.25em">&nbsp;</div>
 
-# Where things have gone wrong
+# Data ethics in the real world
 
-To make the ideas contained in the checklist more concrete, we've compiled examples of times when things have gone wrong. They're paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction.
+To make the ideas contained in the checklist more concrete, we've compiled **examples** of times when tradoffs were handled well, and times when things have gone wrong. Examples are paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction. The positive examples show how principle's of `deon` can be followed in the real world.
 
 <center>Checklist Question</center> | <center>Examples of Ethical Issues</center>
 --- | ---
@@ -16,16 +16,16 @@ To make the ideas contained in the checklist more concrete, we've compiled examp
 **B.2 Right to be forgotten**: Do we have a mechanism through which an individual can request their personal information be removed? | <ul><li>[✅ The EU's General Data Protection Regulation (GDPR) includes the "right to be forgotten."](https://www.eugdpr.org/the-regulation.html)</li></ul>
 **B.3 Data retention plan**: Is there a schedule or plan to delete the data after it is no longer needed? | <ul><li>[⛔ FedEx exposes private information of thousands of customers after a legacy s3 server was left open without a password.](https://www.zdnet.com/article/unsecured-server-exposes-fedex-customer-records/)</li></ul>
  | <center>**Analysis**</center>
-**C.1 Missing perspectives**: Have we sought to address blindspots in the analysis through engagement with relevant stakeholders (e.g., checking assumptions and discussing implications with affected communities and subject matter experts)? | <ul><li>[⛔ When Apple's HealthKit came out in 2014, women couldn't track menstruation.](https://www.theverge.com/2014/9/25/6844021/apple-promised-an-expansive-health-app-so-why-cant-i-track)</li></ul>
-**C.2 Dataset bias**: Have we examined the data for possible sources of bias and taken steps to mitigate or address these biases (e.g., stereotype perpetuation, confirmation bias, imbalanced classes, or omitted confounding variables)? | <ul><li>[⛔ word2vec, trained on Google News corpus, reinforces gender stereotypes.](https://www.technologyreview.com/s/602025/how-vector-space-mathematics-reveals-the-hidden-sexism-in-language/)</li><li>[⛔ Women are more likely to be shown lower-paying jobs than men in Google ads.](https://www.theguardian.com/technology/2015/jul/08/women-less-likely-ads-high-paid-jobs-google-study)</li></ul>
+**C.1 Missing perspectives**: Have we sought to address blindspots in the analysis through engagement with relevant stakeholders (e.g., checking assumptions and discussing implications with affected communities and subject matter experts)? | <ul><li>[✅ Code for America programmatically cleared >140,000 eligible criminal records by collaborating with multiple relevant stakeholders like policymakers, advocacy groups, and courts.](https://codeforamerica.org/programs/criminal-justice/automatic-record-clearance/)</li><li>[⛔ When Apple's HealthKit came out in 2014, women couldn't track menstruation.](https://www.theverge.com/2014/9/25/6844021/apple-promised-an-expansive-health-app-so-why-cant-i-track)</li></ul>
+**C.2 Dataset bias**: Have we examined the data for possible sources of bias and taken steps to mitigate or address these biases (e.g., stereotype perpetuation, confirmation bias, imbalanced classes, or omitted confounding variables)? | <ul><li>[✅ A study by Park et al shows how reweighting can effectively mitigate algorithmic racial bias when predicting risk of postpartum depression.](https://doi.org/10.1001/jamanetworkopen.2021.3909)</li><li>[⛔ word2vec, trained on Google News corpus, reinforces gender stereotypes.](https://www.technologyreview.com/s/602025/how-vector-space-mathematics-reveals-the-hidden-sexism-in-language/)</li><li>[⛔ Women are more likely to be shown lower-paying jobs than men in Google ads.](https://www.theguardian.com/technology/2015/jul/08/women-less-likely-ads-high-paid-jobs-google-study)</li></ul>
 **C.3 Honest representation**: Are our visualizations, summary statistics, and reports designed to honestly represent the underlying data? | <ul><li>[⛔ Misleading chart shown at Planned Parenthood hearing distorts actual trends of abortions vs. cancer screenings and preventative services.](https://www.politifact.com/truth-o-meter/statements/2015/oct/01/jason-chaffetz/chart-shown-planned-parenthood-hearing-misleading-/)</li><li>[⛔ Georgia Dept. of Health graph of COVID-19 cases falsely suggests a steeper decline when dates are ordered by total cases rather than chronologically.](https://www.vox.com/covid-19-coronavirus-us-response-trump/2020/5/18/21262265/georgia-covid-19-cases-declining-reopening)</li></ul>
 **C.4 Privacy in analysis**: Have we ensured that data with PII are not used or displayed unless necessary for the analysis? | <ul><li>[⛔ Strava heatmap of exercise routes reveals sensitive information on military bases and spy outposts.](https://www.theguardian.com/world/2018/jan/28/fitness-tracking-app-gives-away-location-of-secret-us-army-bases)</li></ul>
 **C.5 Auditability**: Is the process of generating the analysis well documented and reproducible if we discover issues in the future? | <ul><li>[⛔ Excel error in well-known economics paper undermines justification of austerity measures.](https://www.bbc.com/news/magazine-22223190)</li></ul>
  | <center>**Modeling**</center>
 **D.1 Proxy discrimination**: Have we ensured that the model does not rely on variables or proxies for variables that are unfairly discriminatory? | <ul><li>[⛔ In hypothetical trials, language models assign the death penalty more frequently to defendants who use African American dialects.](https://arxiv.org/abs/2403.00742)</li><li>[⛔ Variables used to predict child abuse and neglect are direct measurements of poverty, unfairly targeting low-income families for child welfare scrutiny.](https://www.wired.com/story/excerpt-from-automating-inequality/)</li><li>[⛔ Amazon scraps AI recruiting tool that showed bias against women.](https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G)</li><li>[⛔ Criminal sentencing risk asessments don't ask directly about race or income, but other demographic factors can end up being proxies.](https://www.themarshallproject.org/2015/08/04/the-new-science-of-sentencing)</li><li>[⛔ Creditworthiness algorithms based on nontraditional criteria such as grammatic habits, preferred grocery stores, and friends' credit scores can perpetuate systemic bias.](https://www.whitecase.com/publications/insight/algorithms-and-bias-what-lenders-need-know)</li></ul>
-**D.2 Fairness across groups**: Have we tested model results for fairness with respect to different affected groups (e.g., tested for disparate error rates)? | <ul><li>[⛔ Apple credit card offers smaller lines of credit to women than men.](https://www.wired.com/story/the-apple-card-didnt-see-genderand-thats-the-problem/)</li><li>[⛔ Google Photos tags two African-Americans as gorillas.](https://www.forbes.com/sites/mzhang/2015/07/01/google-photos-tags-two-african-americans-as-gorillas-through-facial-recognition-software/#12bdb1fd713d)</li><li>[⛔ With COMPAS, a risk-assessment algorithm used in criminal sentencing, black defendants are almost twice as likely as white defendants to be mislabeled as likely to reoffend.](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing)</li><li>[-- Northpointe's rebuttal to ProPublica article.](https://www.documentcloud.org/documents/2998391-ProPublica-Commentary-Final-070616.html)</li><li>[-- Related academic study.](https://www.liebertpub.com/doi/pdf/10.1089/big.2016.0047)</li><li>[⛔ Google's speech recognition software doesn't recognize women's voices as well as men's.](https://www.dailydot.com/debug/google-voice-recognition-gender-bias/)</li><li>[⛔ Google searches involving black-sounding names are more likely to serve up ads suggestive of a criminal record than white-sounding names.](https://www.technologyreview.com/s/510646/racism-is-poisoning-online-ad-delivery-says-harvard-professor/)</li><li>[-- Related academic study.](https://arxiv.org/abs/1301.6822)</li><li>[⛔ OpenAI's GPT models show racial bias in ranking job applications based on candidate names.](https://www.bloomberg.com/graphics/2024-openai-gpt-hiring-racial-discrimination/)</li></ul>
+**D.2 Fairness across groups**: Have we tested model results for fairness with respect to different affected groups (e.g., tested for disparate error rates)? | <ul><li>[✅ A study by Garriga et al uses ML best practices to test for and communicate fairness across racial groups for a model that predicts mental health crises.](https://www.nature.com/articles/s41591-022-01811-5)</li><li>[⛔ Apple credit card offers smaller lines of credit to women than men.](https://www.wired.com/story/the-apple-card-didnt-see-genderand-thats-the-problem/)</li><li>[⛔ With COMPAS, a risk-assessment algorithm used in criminal sentencing, black defendants are almost twice as likely as white defendants to be mislabeled as likely to reoffend.](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing)</li><li>[-- Northpointe's rebuttal to ProPublica article.](https://www.documentcloud.org/documents/2998391-ProPublica-Commentary-Final-070616.html)</li><li>[-- Related academic study.](https://www.liebertpub.com/doi/pdf/10.1089/big.2016.0047)</li><li>[⛔ Google's speech recognition software doesn't recognize women's voices as well as men's.](https://www.dailydot.com/debug/google-voice-recognition-gender-bias/)</li><li>[⛔ Google searches involving black-sounding names are more likely to serve up ads suggestive of a criminal record than white-sounding names.](https://www.technologyreview.com/s/510646/racism-is-poisoning-online-ad-delivery-says-harvard-professor/)</li><li>[-- Related academic study.](https://arxiv.org/abs/1301.6822)</li><li>[⛔ OpenAI's GPT models show racial bias in ranking job applications based on candidate names.](https://www.bloomberg.com/graphics/2024-openai-gpt-hiring-racial-discrimination/)</li></ul>
 **D.3 Metric selection**: Have we considered the effects of optimizing for our defined metrics and considered additional metrics? | <ul><li>[✅ Facebook seeks to optimize "time well spent", prioritizing interaction over popularity.](https://www.wired.com/story/facebook-tweaks-newsfeed-to-favor-content-from-friends-family/)</li><li>[⛔ YouTube's search autofill suggests pedophiliac phrases due to high viewership of related videos.](https://gizmodo.com/youtubes-creepy-kid-problem-was-worse-than-we-thought-1820763240)</li><li>[⛔ A widely used commercial algorithm in the healthcare industry underestimates the care needs of black patients because it optimizes for spending as a proxy for need, introducing racial bias due to unequal access to care.](https://www.science.org/doi/10.1126/science.aax2342)</li></ul>
-**D.4 Explainability**: Can we explain in understandable terms a decision the model made in cases where a justification is needed? | <ul><li>[⛔ Patients with pneumonia with a history of asthma are usually admitted to the intensive care unit as they have a high risk of dying from pneumonia. Given the success of the intensive care, neural networks predicted asthmatics had a low risk of dying and could therefore be sent home. Without explanatory models to identify this issue, patients may have been sent home to die.](http://people.dbmi.columbia.edu/noemie/papers/15kdd.pdf)</li><li>[GDPR includes a "right to explanation," i.e. meaningful information on the logic underlying automated decisions.](hhttps://academic.oup.com/idpl/article/7/4/233/4762325)</li></ul>
+**D.4 Explainability**: Can we explain in understandable terms a decision the model made in cases where a justification is needed? | <ul><li>[✅ GDPR includes a "right to explanation," i.e. meaningful information on the logic underlying automated decisions.](hhttps://academic.oup.com/idpl/article/7/4/233/4762325)</li><li>[⛔ Patients with pneumonia with a history of asthma are usually admitted to the intensive care unit as they have a high risk of dying from pneumonia. Given the success of the intensive care, neural networks predicted asthmatics had a low risk of dying and could therefore be sent home. Without explanatory models to identify this issue, patients may have been sent home to die.](http://people.dbmi.columbia.edu/noemie/papers/15kdd.pdf)</li></ul>
 **D.5 Communicate limitations**: Have we communicated the shortcomings, limitations, and biases of the model to relevant stakeholders in ways that can be generally understood? | <ul><li>[✅ OpenAI posted an explanation of how ChatGPT is trained to behave, its limitations, and future directions for improvement.](https://openai.com/index/how-should-ai-systems-behave/)</li><li>[⛔ Google Flu claims to accurately predict weekly influenza activity and then misses the 2009 swine flu pandemic.](https://www.forbes.com/sites/stevensalzberg/2014/03/23/why-google-flu-is-a-failure/#6fa6a1925535)</li></ul>
  | <center>**Deployment**</center>
 **E.1 Monitoring and evaluation**: Do we have a clear plan to monitor the model and its impacts after it is deployed (e.g., performance monitoring, regular audit of sample predictions, human review of high-stakes decisions, reviewing downstream impacts of errors or low-confidence decisions, testing for concept drift)? | <ul><li>[✅ RobotsMali uses AI to create children's books in Mali's native languages, and incorporates human review to ensure that all AI-generated content is accurate and culturally sensitive.](https://restofworld.org/2024/mali-ai-translate-local-language-education/)</li><li>[⛔ Dutch Prime Minister and entire cabinet resign after investigations reveal that 26,000 innocent families were wrongly accused of social benefits fraud partially due to a discriminatory algorithm.](https://www.vice.com/en/article/jgq35d/how-a-discriminatory-algorithm-wrongly-accused-thousands-of-families-of-fraud)</li><li>[⛔ Sending police officers to areas of high predicted crime skews future training data collection as police are repeatedly sent back to the same neighborhoods regardless of the true crime rate.](https://www.smithsonianmag.com/innovation/artificial-intelligence-is-now-used-predict-crime-is-it-biased-180968337/)</li></ul>
diff --git a/docs/docs/index.md b/docs/docs/index.md
index daf1334..0a18a2f 100644
--- a/docs/docs/index.md
+++ b/docs/docs/index.md
@@ -262,9 +262,9 @@ We're excited to see so many articles popping up on data ethics! The short list
 - [Technology is biased too. How do we fix it?](https://fivethirtyeight.com/features/technology-is-biased-too-how-do-we-fix-it/)
 - [The dark secret at the heart of AI](https://www.technologyreview.com/s/604087/the-dark-secret-at-the-heart-of-ai/)
 
-## Where things have gone wrong
+## Data ethics in the real world
 
-To make the ideas contained in the checklist more concrete, we've compiled [examples](http://deon.drivendata.org/examples/) of times when things have gone wrong. They're paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction.
+To make the ideas contained in the checklist more concrete, we've compiled [examples](http://deon.drivendata.org/examples/) of times when tradoffs were handled well, and times when things have gone wrong. They're paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction.
 
 We welcome contributions! Follow [these instructions](https://github.com/drivendataorg/deon/blob/main/CONTRIBUTING.md) to add an example.
 

From f7f36ea73a21a0870e0fb54c27fa3a3e2ee1be85 Mon Sep 17 00:00:00 2001
From: Katie Wetstone <klwetstone@gmail.com>
Date: Thu, 13 Jun 2024 08:57:48 -0400
Subject: [PATCH 14/20] tweak README

---
 README.md                          | 4 ++--
 docs/docs/index.md                 | 4 ++--
 docs/md_templates/_common_body.tpl | 4 ++--
 3 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/README.md b/README.md
index 4c9c374..3b309b1 100644
--- a/README.md
+++ b/README.md
@@ -48,7 +48,7 @@ For more configuration details, see the sections on [command line options](#comm
 We created `deon` with the goal of helping data scientists across the sector to be more intentional in their choices, and more aware of the ethical implications of their work. We use that perspective to make decisions about contributions, issues, PRs, and other maintenance and support activities.
 
 
-1. 🔓 First and foremost, **our goal is not to be arbitrators of what ethical concerns merit inclusion**. We have a [process for changing the default checklist](#changing-the-checklist), but we believe that many domain-specific concerns are not included and teams will benefit from developing [custom checklists](#custom-checklists). Not every checklist item will be relevant. We encourage teams to remove items, sections, or mark items as `N/A` as the concerns of their projects dictate.
+1. 🔓 **Our goal is not to be arbitrators of what ethical concerns merit inclusion**. We have a [process for changing the default checklist](#changing-the-checklist), but we believe that many domain-specific concerns are not included and teams will benefit from developing [custom checklists](#custom-checklists). Not every checklist item will be relevant. We encourage teams to remove items, sections, or mark items as `N/A` as the concerns of their projects dictate.
 
 2. 📊 This checklist is designed to provoke conversations around **issues where data scientists have particular responsibility and perspective**. It's not up to data scientists alone to decide what the ethical course of action is. This has always been a responsibility of organizations that are part of civil society. Conversations should be part of a larger organizational commitment to doing what is right.
 
@@ -56,7 +56,7 @@ We created `deon` with the goal of helping data scientists across the sector to
 
 4. 🌎 We believe in the **power of examples** to bring the principles of data ethics to bear on human experience. This repository includes a [list of real-world examples](http://deon.drivendata.org/examples/) connected with each item in the default checklist. We encourage you to contribute relevant use cases that you believe can benefit the community by their example. In addition, if you have a topic, idea, or comment that doesn't seem right for the documentation, please add it to the [wiki page](https://github.com/drivendataorg/deon/wiki) for this project!
 
-5. 🔍 We believe the primary benefit of a checklist is **ensuring that we don't overlook important work**. Sometimes it is difficult with pressing deadlines and a demand to multitask to make sure we do the hard work to think about the big picture. This package is meant to help ensure that those discussions happen, even in fast-moving environments. Ethics is hard, and we expect some of the conversations that arise from this checklist may also be hard.
+5. 🔍 We believe the primary benefit of a checklist is **ensuring that we don't overlook important work**. Sometimes it is difficult with pressing deadlines and a demand to multitask to make sure we do the hard work to think about the big picture. This package is meant to help ensure that those discussions happen, even in fast-moving environments.
 
 6. ❓ We can't define exhaustively every term that appears in the checklist. Some of these **terms are open to interpretation** or mean different things in different contexts. We recommend that when relevant, users create their own glossary for reference.
 
diff --git a/docs/docs/index.md b/docs/docs/index.md
index 0a18a2f..5648d58 100644
--- a/docs/docs/index.md
+++ b/docs/docs/index.md
@@ -41,7 +41,7 @@ For more configuration details, see the sections on [command line options](#comm
 We created `deon` with the goal of helping data scientists across the sector to be more intentional in their choices, and more aware of the ethical implications of their work. We use that perspective to make decisions about contributions, issues, PRs, and other maintenance and support activities.
 
 
-1. 🔓 First and foremost, **our goal is not to be arbitrators of what ethical concerns merit inclusion**. We have a [process for changing the default checklist](#changing-the-checklist), but we believe that many domain-specific concerns are not included and teams will benefit from developing [custom checklists](#custom-checklists). Not every checklist item will be relevant. We encourage teams to remove items, sections, or mark items as `N/A` as the concerns of their projects dictate.
+1. 🔓 **Our goal is not to be arbitrators of what ethical concerns merit inclusion**. We have a [process for changing the default checklist](#changing-the-checklist), but we believe that many domain-specific concerns are not included and teams will benefit from developing [custom checklists](#custom-checklists). Not every checklist item will be relevant. We encourage teams to remove items, sections, or mark items as `N/A` as the concerns of their projects dictate.
 
 2. 📊 This checklist is designed to provoke conversations around **issues where data scientists have particular responsibility and perspective**. It's not up to data scientists alone to decide what the ethical course of action is. This has always been a responsibility of organizations that are part of civil society. Conversations should be part of a larger organizational commitment to doing what is right.
 
@@ -49,7 +49,7 @@ We created `deon` with the goal of helping data scientists across the sector to
 
 4. 🌎 We believe in the **power of examples** to bring the principles of data ethics to bear on human experience. This repository includes a [list of real-world examples](http://deon.drivendata.org/examples/) connected with each item in the default checklist. We encourage you to contribute relevant use cases that you believe can benefit the community by their example. In addition, if you have a topic, idea, or comment that doesn't seem right for the documentation, please add it to the [wiki page](https://github.com/drivendataorg/deon/wiki) for this project!
 
-5. 🔍 We believe the primary benefit of a checklist is **ensuring that we don't overlook important work**. Sometimes it is difficult with pressing deadlines and a demand to multitask to make sure we do the hard work to think about the big picture. This package is meant to help ensure that those discussions happen, even in fast-moving environments. Ethics is hard, and we expect some of the conversations that arise from this checklist may also be hard.
+5. 🔍 We believe the primary benefit of a checklist is **ensuring that we don't overlook important work**. Sometimes it is difficult with pressing deadlines and a demand to multitask to make sure we do the hard work to think about the big picture. This package is meant to help ensure that those discussions happen, even in fast-moving environments.
 
 6. ❓ We can't define exhaustively every term that appears in the checklist. Some of these **terms are open to interpretation** or mean different things in different contexts. We recommend that when relevant, users create their own glossary for reference.
 
diff --git a/docs/md_templates/_common_body.tpl b/docs/md_templates/_common_body.tpl
index 07260fc..718574b 100644
--- a/docs/md_templates/_common_body.tpl
+++ b/docs/md_templates/_common_body.tpl
@@ -39,7 +39,7 @@ For more configuration details, see the sections on [command line options](#comm
 We created `deon` with the goal of helping data scientists across the sector to be more intentional in their choices, and more aware of the ethical implications of their work. We use that perspective to make decisions about contributions, issues, PRs, and other maintenance and support activities.
 
 
-1. 🔓 First and foremost, **our goal is not to be arbitrators of what ethical concerns merit inclusion**. We have a [process for changing the default checklist](#changing-the-checklist), but we believe that many domain-specific concerns are not included and teams will benefit from developing [custom checklists](#custom-checklists). Not every checklist item will be relevant. We encourage teams to remove items, sections, or mark items as `N/A` as the concerns of their projects dictate.
+1. 🔓 **Our goal is not to be arbitrators of what ethical concerns merit inclusion**. We have a [process for changing the default checklist](#changing-the-checklist), but we believe that many domain-specific concerns are not included and teams will benefit from developing [custom checklists](#custom-checklists). Not every checklist item will be relevant. We encourage teams to remove items, sections, or mark items as `N/A` as the concerns of their projects dictate.
 
 2. 📊 This checklist is designed to provoke conversations around **issues where data scientists have particular responsibility and perspective**. It's not up to data scientists alone to decide what the ethical course of action is. This has always been a responsibility of organizations that are part of civil society. Conversations should be part of a larger organizational commitment to doing what is right.
 
@@ -47,7 +47,7 @@ We created `deon` with the goal of helping data scientists across the sector to
 
 4. 🌎 We believe in the **power of examples** to bring the principles of data ethics to bear on human experience. This repository includes a [list of real-world examples](http://deon.drivendata.org/examples/) connected with each item in the default checklist. We encourage you to contribute relevant use cases that you believe can benefit the community by their example. In addition, if you have a topic, idea, or comment that doesn't seem right for the documentation, please add it to the [wiki page](https://github.com/drivendataorg/deon/wiki) for this project!
 
-5. 🔍 We believe the primary benefit of a checklist is **ensuring that we don't overlook important work**. Sometimes it is difficult with pressing deadlines and a demand to multitask to make sure we do the hard work to think about the big picture. This package is meant to help ensure that those discussions happen, even in fast-moving environments. Ethics is hard, and we expect some of the conversations that arise from this checklist may also be hard.
+5. 🔍 We believe the primary benefit of a checklist is **ensuring that we don't overlook important work**. Sometimes it is difficult with pressing deadlines and a demand to multitask to make sure we do the hard work to think about the big picture. This package is meant to help ensure that those discussions happen, even in fast-moving environments.
 
 6. ❓ We can't define exhaustively every term that appears in the checklist. Some of these **terms are open to interpretation** or mean different things in different contexts. We recommend that when relevant, users create their own glossary for reference.
 

From 3afd31053b80b8c89f0cb8f69a7651553fddb613 Mon Sep 17 00:00:00 2001
From: Katie Wetstone <klwetstone@gmail.com>
Date: Thu, 13 Jun 2024 08:58:28 -0400
Subject: [PATCH 15/20] add more positive examples

---
 deon/assets/examples_of_ethical_issues.yml | 6 +++++-
 docs/docs/examples.md                      | 8 ++++----
 docs/md_templates/examples.tpl             | 2 +-
 3 files changed, 10 insertions(+), 6 deletions(-)

diff --git a/deon/assets/examples_of_ethical_issues.yml b/deon/assets/examples_of_ethical_issues.yml
index 672ad93..2e5ff5a 100644
--- a/deon/assets/examples_of_ethical_issues.yml
+++ b/deon/assets/examples_of_ethical_issues.yml
@@ -1,5 +1,7 @@
 - line_id: A.1
   links:
+    - text: ✅ A voiceover studio is now required to get informed consent from a performer before using their likeness in AI-generated content.
+      url: https://variety.com/2024/biz/news/sag-aftra-ai-voiceover-studio-video-games-1235866313/
     - text: ⛔ Facebook uses phone numbers provided for two-factor authentication to target users with ads.
       url: https://techcrunch.com/2018/09/27/yes-facebook-is-using-your-2fa-phone-number-to-target-you-with-ads/
     - text: ⛔ African-American men were enrolled in the Tuskegee Study on the progression of syphilis without being told the true purpose of the study or that treatment for syphilis was being withheld.
@@ -14,6 +16,8 @@
       url: http://content.time.com/time/business/article/0,8599,1954643,00.html
 - line_id: A.3
   links:
+    - text: ✅ DuckDuckGo enables users to anonymously access ChatGPT by *not* collecting user IP addresses along with queries.
+      url: https://www.theverge.com/2024/6/6/24172719/duckduckgo-private-ai-chats-anonymous-gpt-3-5
     - text: ⛔ Personal information on taxi drivers can be accessed in poorly anonymized taxi trips dataset released by New York City.
       url: https://www.theguardian.com/technology/2014/jun/27/new-york-taxi-details-anonymised-data-researchers-warn
     - text: ⛔ Netflix prize dataset of movie rankings by 500,000 customers is easily de-anonymized through cross referencing with other publicly available datasets.
@@ -50,7 +54,7 @@
       url: https://www.theverge.com/2014/9/25/6844021/apple-promised-an-expansive-health-app-so-why-cant-i-track
 - line_id: C.2
   links:
-    - text: ✅ A study by Park et al shows how reweighting can effectively mitigate algorithmic racial bias when predicting risk of postpartum depression.
+    - text: ✅ A study by Park et al shows how reweighting can effectively mitigate racial bias when predicting risk of postpartum depression.
       url: https://doi.org/10.1001/jamanetworkopen.2021.3909 
     - text: ⛔ word2vec, trained on Google News corpus, reinforces gender stereotypes.
       url: https://www.technologyreview.com/s/602025/how-vector-space-mathematics-reveals-the-hidden-sexism-in-language/
diff --git a/docs/docs/examples.md b/docs/docs/examples.md
index b69569f..e00f044 100644
--- a/docs/docs/examples.md
+++ b/docs/docs/examples.md
@@ -2,14 +2,14 @@
 
 # Data ethics in the real world
 
-To make the ideas contained in the checklist more concrete, we've compiled **examples** of times when tradoffs were handled well, and times when things have gone wrong. Examples are paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction. The positive examples show how principle's of `deon` can be followed in the real world.
+To make the ideas contained in the checklist more concrete, we've compiled **examples** of times when tradoffs were handled well, and times when things have gone wrong. Examples are paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction. Positive examples show how principles of `deon` can be followed in the real world.
 
 <center>Checklist Question</center> | <center>Examples of Ethical Issues</center>
 --- | ---
  | <center>**Data Collection**</center>
-**A.1 Informed consent**: If there are human subjects, have they given informed consent, where subjects affirmatively opt-in and have a clear understanding of the data uses to which they consent? | <ul><li>[⛔ Facebook uses phone numbers provided for two-factor authentication to target users with ads.](https://techcrunch.com/2018/09/27/yes-facebook-is-using-your-2fa-phone-number-to-target-you-with-ads/)</li><li>[⛔ African-American men were enrolled in the Tuskegee Study on the progression of syphilis without being told the true purpose of the study or that treatment for syphilis was being withheld.](https://en.wikipedia.org/wiki/Tuskegee_syphilis_experiment)</li><li>[⛔ OpenAI's ChatGPT memorized and regurgitated entire poems without checking for copyright permissions.](https://news.cornell.edu/stories/2024/01/chatgpt-memorizes-and-spits-out-entire-poems)</li></ul>
+**A.1 Informed consent**: If there are human subjects, have they given informed consent, where subjects affirmatively opt-in and have a clear understanding of the data uses to which they consent? | <ul><li>[✅ A voiceover studio is now required to get informed consent from a performer before using their likeness in AI-generated content.](https://variety.com/2024/biz/news/sag-aftra-ai-voiceover-studio-video-games-1235866313/)</li><li>[⛔ Facebook uses phone numbers provided for two-factor authentication to target users with ads.](https://techcrunch.com/2018/09/27/yes-facebook-is-using-your-2fa-phone-number-to-target-you-with-ads/)</li><li>[⛔ African-American men were enrolled in the Tuskegee Study on the progression of syphilis without being told the true purpose of the study or that treatment for syphilis was being withheld.](https://en.wikipedia.org/wiki/Tuskegee_syphilis_experiment)</li><li>[⛔ OpenAI's ChatGPT memorized and regurgitated entire poems without checking for copyright permissions.](https://news.cornell.edu/stories/2024/01/chatgpt-memorizes-and-spits-out-entire-poems)</li></ul>
 **A.2 Collection bias**: Have we considered sources of bias that could be introduced during data collection and survey design and taken steps to mitigate those? | <ul><li>[⛔ StreetBump, a smartphone app to passively detect potholes, may fail to direct public resources to areas where smartphone penetration is lower, such as lower income areas or areas with a larger elderly population.](https://hbr.org/2013/04/the-hidden-biases-in-big-data)</li><li>[⛔ Facial recognition cameras used for passport control register Asian's eyes as closed.](http://content.time.com/time/business/article/0,8599,1954643,00.html)</li></ul>
-**A.3 Limit PII exposure**: Have we considered ways to minimize exposure of personally identifiable information (PII) for example through anonymization or not collecting information that isn't relevant for analysis? | <ul><li>[⛔ Personal information on taxi drivers can be accessed in poorly anonymized taxi trips dataset released by New York City.](https://www.theguardian.com/technology/2014/jun/27/new-york-taxi-details-anonymised-data-researchers-warn)</li><li>[⛔ Netflix prize dataset of movie rankings by 500,000 customers is easily de-anonymized through cross referencing with other publicly available datasets.](https://www.wired.com/2007/12/why-anonymous-data-sometimes-isnt/)</li></ul>
+**A.3 Limit PII exposure**: Have we considered ways to minimize exposure of personally identifiable information (PII) for example through anonymization or not collecting information that isn't relevant for analysis? | <ul><li>[✅ DuckDuckGo enables users to anonymously access ChatGPT by *not* collecting user IP addresses along with queries.](https://www.theverge.com/2024/6/6/24172719/duckduckgo-private-ai-chats-anonymous-gpt-3-5)</li><li>[⛔ Personal information on taxi drivers can be accessed in poorly anonymized taxi trips dataset released by New York City.](https://www.theguardian.com/technology/2014/jun/27/new-york-taxi-details-anonymised-data-researchers-warn)</li><li>[⛔ Netflix prize dataset of movie rankings by 500,000 customers is easily de-anonymized through cross referencing with other publicly available datasets.](https://www.wired.com/2007/12/why-anonymous-data-sometimes-isnt/)</li></ul>
 **A.4 Downstream bias mitigation**: Have we considered ways to enable testing downstream results for biased outcomes (e.g., collecting data on protected group status like race or gender)? | <ul><li>[⛔ In six major cities, Amazon's same day delivery service excludes many predominantly black neighborhoods.](https://www.bloomberg.com/graphics/2016-amazon-same-day/)</li><li>[⛔ Facial recognition software is significanty worse at identifying people with darker skin.](https://www.theregister.co.uk/2018/02/13/facial_recognition_software_is_better_at_white_men_than_black_women/)</li></ul>
  | <center>**Data Storage**</center>
 **B.1 Data security**: Do we have a plan to protect and secure data (e.g., encryption at rest and in transit, access controls on internal users and third parties, access logs, and up-to-date software)? | <ul><li>[✅ MediCapt, which documents forensic evidence in conflict regions, effectively protects sensitive information using encryption, limited access, and security audits.](https://phr.org/issues/sexual-violence/medicapt/)</li><li>[⛔ Personal and financial data for more than 146 million people was stolen in Equifax data breach.](https://www.nbcnews.com/news/us-news/equifax-breaks-down-just-how-bad-last-year-s-data-n872496)</li><li>[⛔ Cambridge Analytica harvested private information from over 50 million Facebook profiles without users' permission.](https://www.nytimes.com/2018/03/17/us/politics/cambridge-analytica-trump-campaign.html)</li><li>[⛔ AOL accidentally released 20 million search queries from 658,000 customers.](https://www.wired.com/2006/08/faq-aols-search-gaffe-and-you/)</li></ul>
@@ -17,7 +17,7 @@ To make the ideas contained in the checklist more concrete, we've compiled **exa
 **B.3 Data retention plan**: Is there a schedule or plan to delete the data after it is no longer needed? | <ul><li>[⛔ FedEx exposes private information of thousands of customers after a legacy s3 server was left open without a password.](https://www.zdnet.com/article/unsecured-server-exposes-fedex-customer-records/)</li></ul>
  | <center>**Analysis**</center>
 **C.1 Missing perspectives**: Have we sought to address blindspots in the analysis through engagement with relevant stakeholders (e.g., checking assumptions and discussing implications with affected communities and subject matter experts)? | <ul><li>[✅ Code for America programmatically cleared >140,000 eligible criminal records by collaborating with multiple relevant stakeholders like policymakers, advocacy groups, and courts.](https://codeforamerica.org/programs/criminal-justice/automatic-record-clearance/)</li><li>[⛔ When Apple's HealthKit came out in 2014, women couldn't track menstruation.](https://www.theverge.com/2014/9/25/6844021/apple-promised-an-expansive-health-app-so-why-cant-i-track)</li></ul>
-**C.2 Dataset bias**: Have we examined the data for possible sources of bias and taken steps to mitigate or address these biases (e.g., stereotype perpetuation, confirmation bias, imbalanced classes, or omitted confounding variables)? | <ul><li>[✅ A study by Park et al shows how reweighting can effectively mitigate algorithmic racial bias when predicting risk of postpartum depression.](https://doi.org/10.1001/jamanetworkopen.2021.3909)</li><li>[⛔ word2vec, trained on Google News corpus, reinforces gender stereotypes.](https://www.technologyreview.com/s/602025/how-vector-space-mathematics-reveals-the-hidden-sexism-in-language/)</li><li>[⛔ Women are more likely to be shown lower-paying jobs than men in Google ads.](https://www.theguardian.com/technology/2015/jul/08/women-less-likely-ads-high-paid-jobs-google-study)</li></ul>
+**C.2 Dataset bias**: Have we examined the data for possible sources of bias and taken steps to mitigate or address these biases (e.g., stereotype perpetuation, confirmation bias, imbalanced classes, or omitted confounding variables)? | <ul><li>[✅ A study by Park et al shows how reweighting can effectively mitigate racial bias when predicting risk of postpartum depression.](https://doi.org/10.1001/jamanetworkopen.2021.3909)</li><li>[⛔ word2vec, trained on Google News corpus, reinforces gender stereotypes.](https://www.technologyreview.com/s/602025/how-vector-space-mathematics-reveals-the-hidden-sexism-in-language/)</li><li>[⛔ Women are more likely to be shown lower-paying jobs than men in Google ads.](https://www.theguardian.com/technology/2015/jul/08/women-less-likely-ads-high-paid-jobs-google-study)</li></ul>
 **C.3 Honest representation**: Are our visualizations, summary statistics, and reports designed to honestly represent the underlying data? | <ul><li>[⛔ Misleading chart shown at Planned Parenthood hearing distorts actual trends of abortions vs. cancer screenings and preventative services.](https://www.politifact.com/truth-o-meter/statements/2015/oct/01/jason-chaffetz/chart-shown-planned-parenthood-hearing-misleading-/)</li><li>[⛔ Georgia Dept. of Health graph of COVID-19 cases falsely suggests a steeper decline when dates are ordered by total cases rather than chronologically.](https://www.vox.com/covid-19-coronavirus-us-response-trump/2020/5/18/21262265/georgia-covid-19-cases-declining-reopening)</li></ul>
 **C.4 Privacy in analysis**: Have we ensured that data with PII are not used or displayed unless necessary for the analysis? | <ul><li>[⛔ Strava heatmap of exercise routes reveals sensitive information on military bases and spy outposts.](https://www.theguardian.com/world/2018/jan/28/fitness-tracking-app-gives-away-location-of-secret-us-army-bases)</li></ul>
 **C.5 Auditability**: Is the process of generating the analysis well documented and reproducible if we discover issues in the future? | <ul><li>[⛔ Excel error in well-known economics paper undermines justification of austerity measures.](https://www.bbc.com/news/magazine-22223190)</li></ul>
diff --git a/docs/md_templates/examples.tpl b/docs/md_templates/examples.tpl
index b17a19f..c330f5c 100644
--- a/docs/md_templates/examples.tpl
+++ b/docs/md_templates/examples.tpl
@@ -2,6 +2,6 @@
 
 # Data ethics in the real world
 
-To make the ideas contained in the checklist more concrete, we've compiled **examples** of times when tradoffs were handled well, and times when things have gone wrong. Examples are paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction. The positive examples show how principle's of `deon` can be followed in the real world.
+To make the ideas contained in the checklist more concrete, we've compiled **examples** of times when tradoffs were handled well, and times when things have gone wrong. Examples are paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction. Positive examples show how principles of `deon` can be followed in the real world.
 
 {{ links_table }}
\ No newline at end of file

From a0fb376d818d8f54a486ba6ae0fbace379297580 Mon Sep 17 00:00:00 2001
From: Katie Wetstone <klwetstone@gmail.com>
Date: Thu, 13 Jun 2024 10:34:57 -0400
Subject: [PATCH 16/20] more examples

---
 deon/assets/examples_of_ethical_issues.yml | 14 +++++++++-----
 docs/docs/examples.md                      | 12 ++++++------
 docs/render_templates.py                   |  2 +-
 3 files changed, 16 insertions(+), 12 deletions(-)

diff --git a/deon/assets/examples_of_ethical_issues.yml b/deon/assets/examples_of_ethical_issues.yml
index 2e5ff5a..4c03b61 100644
--- a/deon/assets/examples_of_ethical_issues.yml
+++ b/deon/assets/examples_of_ethical_issues.yml
@@ -54,7 +54,7 @@
       url: https://www.theverge.com/2014/9/25/6844021/apple-promised-an-expansive-health-app-so-why-cant-i-track
 - line_id: C.2
   links:
-    - text: ✅ A study by Park et al shows how reweighting can effectively mitigate racial bias when predicting risk of postpartum depression.
+    - text: ✅ A study by Park et al shows how reweighting can mitigate racial bias when predicting risk of postpartum depression.
       url: https://doi.org/10.1001/jamanetworkopen.2021.3909 
     - text: ⛔ word2vec, trained on Google News corpus, reinforces gender stereotypes.
       url: https://www.technologyreview.com/s/602025/how-vector-space-mathematics-reveals-the-hidden-sexism-in-language/
@@ -72,16 +72,20 @@
       url: https://www.theguardian.com/world/2018/jan/28/fitness-tracking-app-gives-away-location-of-secret-us-army-bases
 - line_id: C.5
   links:
+    - text: ✅ NASA's Transform to Open Science initiative is working to make research more reproducible and accessible.
+      url: https://nasa.github.io/Transform-to-Open-Science/
+    - text: ✅ Medic's Community Health Tooklit supports health workers in hard-to-reach areas. The toolkit is fully open source on Github for anyone to view or collaborate.
+      url: https://communityhealthtoolkit.org/
     - text: ⛔ Excel error in well-known economics paper undermines justification of austerity measures.
       url: https://www.bbc.com/news/magazine-22223190
 - line_id: D.1
   links:
+    - text: ✅ Amazon developed an experimental AI recruiting tool, but did not deploy it because it learned to perpetuate bias against women.
+      url: https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G
     - text: ⛔ In hypothetical trials, language models assign the death penalty more frequently to defendants who use African American dialects.    
       url: https://arxiv.org/abs/2403.00742
     - text: ⛔ Variables used to predict child abuse and neglect are direct measurements of poverty, unfairly targeting low-income families for child welfare scrutiny.
       url: https://www.wired.com/story/excerpt-from-automating-inequality/
-    - text: ⛔ Amazon scraps AI recruiting tool that showed bias against women.
-      url: https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G
     - text: ⛔ Criminal sentencing risk asessments don't ask directly about race or income, but other demographic factors can end up being proxies.
       url: https://www.themarshallproject.org/2015/08/04/the-new-science-of-sentencing
     - text: ⛔ Creditworthiness algorithms based on nontraditional criteria such as grammatic habits, preferred grocery stores, and friends' credit scores can perpetuate systemic bias.
@@ -102,8 +106,6 @@
       url: https://www.dailydot.com/debug/google-voice-recognition-gender-bias/
     - text: ⛔ Google searches involving black-sounding names are more likely to serve up ads suggestive of a criminal record than white-sounding names.
       url: https://www.technologyreview.com/s/510646/racism-is-poisoning-online-ad-delivery-says-harvard-professor/
-    - text: -- Related academic study.
-      url: https://arxiv.org/abs/1301.6822
     - text: ⛔ OpenAI's GPT models show racial bias in ranking job applications based on candidate names.
       url: https://www.bloomberg.com/graphics/2024-openai-gpt-hiring-racial-discrimination/
 - line_id: D.3
@@ -136,6 +138,8 @@
       url: https://www.smithsonianmag.com/innovation/artificial-intelligence-is-now-used-predict-crime-is-it-biased-180968337/
 - line_id: E.2
   links:
+    - text: ✅ Healing ARC uses a targeted, race-conscious algorithm to counteract documented inequities in access to heart failure care for Black and Latinx patients.
+      url: https://catalyst.nejm.org/doi/full/10.1056/CAT.22.0076
     - text: ⛔ Software mistakes result in healthcare cuts for people with diabetes or cerebral palsy.
       url: https://www.theverge.com/2018/3/21/17144260/healthcare-medicaid-algorithm-arkansas-cerebral-palsy
 - line_id: E.3
diff --git a/docs/docs/examples.md b/docs/docs/examples.md
index e00f044..c1ee2fc 100644
--- a/docs/docs/examples.md
+++ b/docs/docs/examples.md
@@ -4,7 +4,7 @@
 
 To make the ideas contained in the checklist more concrete, we've compiled **examples** of times when tradoffs were handled well, and times when things have gone wrong. Examples are paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction. Positive examples show how principles of `deon` can be followed in the real world.
 
-<center>Checklist Question</center> | <center>Examples of Ethical Issues</center>
+<center>Checklist Question</center> | <center>Examples</center>
 --- | ---
  | <center>**Data Collection**</center>
 **A.1 Informed consent**: If there are human subjects, have they given informed consent, where subjects affirmatively opt-in and have a clear understanding of the data uses to which they consent? | <ul><li>[✅ A voiceover studio is now required to get informed consent from a performer before using their likeness in AI-generated content.](https://variety.com/2024/biz/news/sag-aftra-ai-voiceover-studio-video-games-1235866313/)</li><li>[⛔ Facebook uses phone numbers provided for two-factor authentication to target users with ads.](https://techcrunch.com/2018/09/27/yes-facebook-is-using-your-2fa-phone-number-to-target-you-with-ads/)</li><li>[⛔ African-American men were enrolled in the Tuskegee Study on the progression of syphilis without being told the true purpose of the study or that treatment for syphilis was being withheld.](https://en.wikipedia.org/wiki/Tuskegee_syphilis_experiment)</li><li>[⛔ OpenAI's ChatGPT memorized and regurgitated entire poems without checking for copyright permissions.](https://news.cornell.edu/stories/2024/01/chatgpt-memorizes-and-spits-out-entire-poems)</li></ul>
@@ -17,18 +17,18 @@ To make the ideas contained in the checklist more concrete, we've compiled **exa
 **B.3 Data retention plan**: Is there a schedule or plan to delete the data after it is no longer needed? | <ul><li>[⛔ FedEx exposes private information of thousands of customers after a legacy s3 server was left open without a password.](https://www.zdnet.com/article/unsecured-server-exposes-fedex-customer-records/)</li></ul>
  | <center>**Analysis**</center>
 **C.1 Missing perspectives**: Have we sought to address blindspots in the analysis through engagement with relevant stakeholders (e.g., checking assumptions and discussing implications with affected communities and subject matter experts)? | <ul><li>[✅ Code for America programmatically cleared >140,000 eligible criminal records by collaborating with multiple relevant stakeholders like policymakers, advocacy groups, and courts.](https://codeforamerica.org/programs/criminal-justice/automatic-record-clearance/)</li><li>[⛔ When Apple's HealthKit came out in 2014, women couldn't track menstruation.](https://www.theverge.com/2014/9/25/6844021/apple-promised-an-expansive-health-app-so-why-cant-i-track)</li></ul>
-**C.2 Dataset bias**: Have we examined the data for possible sources of bias and taken steps to mitigate or address these biases (e.g., stereotype perpetuation, confirmation bias, imbalanced classes, or omitted confounding variables)? | <ul><li>[✅ A study by Park et al shows how reweighting can effectively mitigate racial bias when predicting risk of postpartum depression.](https://doi.org/10.1001/jamanetworkopen.2021.3909)</li><li>[⛔ word2vec, trained on Google News corpus, reinforces gender stereotypes.](https://www.technologyreview.com/s/602025/how-vector-space-mathematics-reveals-the-hidden-sexism-in-language/)</li><li>[⛔ Women are more likely to be shown lower-paying jobs than men in Google ads.](https://www.theguardian.com/technology/2015/jul/08/women-less-likely-ads-high-paid-jobs-google-study)</li></ul>
+**C.2 Dataset bias**: Have we examined the data for possible sources of bias and taken steps to mitigate or address these biases (e.g., stereotype perpetuation, confirmation bias, imbalanced classes, or omitted confounding variables)? | <ul><li>[✅ A study by Park et al shows how reweighting can mitigate racial bias when predicting risk of postpartum depression.](https://doi.org/10.1001/jamanetworkopen.2021.3909)</li><li>[⛔ word2vec, trained on Google News corpus, reinforces gender stereotypes.](https://www.technologyreview.com/s/602025/how-vector-space-mathematics-reveals-the-hidden-sexism-in-language/)</li><li>[⛔ Women are more likely to be shown lower-paying jobs than men in Google ads.](https://www.theguardian.com/technology/2015/jul/08/women-less-likely-ads-high-paid-jobs-google-study)</li></ul>
 **C.3 Honest representation**: Are our visualizations, summary statistics, and reports designed to honestly represent the underlying data? | <ul><li>[⛔ Misleading chart shown at Planned Parenthood hearing distorts actual trends of abortions vs. cancer screenings and preventative services.](https://www.politifact.com/truth-o-meter/statements/2015/oct/01/jason-chaffetz/chart-shown-planned-parenthood-hearing-misleading-/)</li><li>[⛔ Georgia Dept. of Health graph of COVID-19 cases falsely suggests a steeper decline when dates are ordered by total cases rather than chronologically.](https://www.vox.com/covid-19-coronavirus-us-response-trump/2020/5/18/21262265/georgia-covid-19-cases-declining-reopening)</li></ul>
 **C.4 Privacy in analysis**: Have we ensured that data with PII are not used or displayed unless necessary for the analysis? | <ul><li>[⛔ Strava heatmap of exercise routes reveals sensitive information on military bases and spy outposts.](https://www.theguardian.com/world/2018/jan/28/fitness-tracking-app-gives-away-location-of-secret-us-army-bases)</li></ul>
-**C.5 Auditability**: Is the process of generating the analysis well documented and reproducible if we discover issues in the future? | <ul><li>[⛔ Excel error in well-known economics paper undermines justification of austerity measures.](https://www.bbc.com/news/magazine-22223190)</li></ul>
+**C.5 Auditability**: Is the process of generating the analysis well documented and reproducible if we discover issues in the future? | <ul><li>[✅ NASA's Transform to Open Science initiative is working to make research more reproducible and accessible.](https://nasa.github.io/Transform-to-Open-Science/)</li><li>[✅ Medic's Community Health Tooklit supports health workers in hard-to-reach areas. The toolkit is fully open source on Github for anyone to view or collaborate.](https://communityhealthtoolkit.org/)</li><li>[⛔ Excel error in well-known economics paper undermines justification of austerity measures.](https://www.bbc.com/news/magazine-22223190)</li></ul>
  | <center>**Modeling**</center>
-**D.1 Proxy discrimination**: Have we ensured that the model does not rely on variables or proxies for variables that are unfairly discriminatory? | <ul><li>[⛔ In hypothetical trials, language models assign the death penalty more frequently to defendants who use African American dialects.](https://arxiv.org/abs/2403.00742)</li><li>[⛔ Variables used to predict child abuse and neglect are direct measurements of poverty, unfairly targeting low-income families for child welfare scrutiny.](https://www.wired.com/story/excerpt-from-automating-inequality/)</li><li>[⛔ Amazon scraps AI recruiting tool that showed bias against women.](https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G)</li><li>[⛔ Criminal sentencing risk asessments don't ask directly about race or income, but other demographic factors can end up being proxies.](https://www.themarshallproject.org/2015/08/04/the-new-science-of-sentencing)</li><li>[⛔ Creditworthiness algorithms based on nontraditional criteria such as grammatic habits, preferred grocery stores, and friends' credit scores can perpetuate systemic bias.](https://www.whitecase.com/publications/insight/algorithms-and-bias-what-lenders-need-know)</li></ul>
-**D.2 Fairness across groups**: Have we tested model results for fairness with respect to different affected groups (e.g., tested for disparate error rates)? | <ul><li>[✅ A study by Garriga et al uses ML best practices to test for and communicate fairness across racial groups for a model that predicts mental health crises.](https://www.nature.com/articles/s41591-022-01811-5)</li><li>[⛔ Apple credit card offers smaller lines of credit to women than men.](https://www.wired.com/story/the-apple-card-didnt-see-genderand-thats-the-problem/)</li><li>[⛔ With COMPAS, a risk-assessment algorithm used in criminal sentencing, black defendants are almost twice as likely as white defendants to be mislabeled as likely to reoffend.](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing)</li><li>[-- Northpointe's rebuttal to ProPublica article.](https://www.documentcloud.org/documents/2998391-ProPublica-Commentary-Final-070616.html)</li><li>[-- Related academic study.](https://www.liebertpub.com/doi/pdf/10.1089/big.2016.0047)</li><li>[⛔ Google's speech recognition software doesn't recognize women's voices as well as men's.](https://www.dailydot.com/debug/google-voice-recognition-gender-bias/)</li><li>[⛔ Google searches involving black-sounding names are more likely to serve up ads suggestive of a criminal record than white-sounding names.](https://www.technologyreview.com/s/510646/racism-is-poisoning-online-ad-delivery-says-harvard-professor/)</li><li>[-- Related academic study.](https://arxiv.org/abs/1301.6822)</li><li>[⛔ OpenAI's GPT models show racial bias in ranking job applications based on candidate names.](https://www.bloomberg.com/graphics/2024-openai-gpt-hiring-racial-discrimination/)</li></ul>
+**D.1 Proxy discrimination**: Have we ensured that the model does not rely on variables or proxies for variables that are unfairly discriminatory? | <ul><li>[✅ Amazon developed an experimental AI recruiting tool, but did not deploy it because it learned to perpetuate bias against women.](https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G)</li><li>[⛔ In hypothetical trials, language models assign the death penalty more frequently to defendants who use African American dialects.](https://arxiv.org/abs/2403.00742)</li><li>[⛔ Variables used to predict child abuse and neglect are direct measurements of poverty, unfairly targeting low-income families for child welfare scrutiny.](https://www.wired.com/story/excerpt-from-automating-inequality/)</li><li>[⛔ Criminal sentencing risk asessments don't ask directly about race or income, but other demographic factors can end up being proxies.](https://www.themarshallproject.org/2015/08/04/the-new-science-of-sentencing)</li><li>[⛔ Creditworthiness algorithms based on nontraditional criteria such as grammatic habits, preferred grocery stores, and friends' credit scores can perpetuate systemic bias.](https://www.whitecase.com/publications/insight/algorithms-and-bias-what-lenders-need-know)</li></ul>
+**D.2 Fairness across groups**: Have we tested model results for fairness with respect to different affected groups (e.g., tested for disparate error rates)? | <ul><li>[✅ A study by Garriga et al uses ML best practices to test for and communicate fairness across racial groups for a model that predicts mental health crises.](https://www.nature.com/articles/s41591-022-01811-5)</li><li>[⛔ Apple credit card offers smaller lines of credit to women than men.](https://www.wired.com/story/the-apple-card-didnt-see-genderand-thats-the-problem/)</li><li>[⛔ With COMPAS, a risk-assessment algorithm used in criminal sentencing, black defendants are almost twice as likely as white defendants to be mislabeled as likely to reoffend.](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing)</li><li>[-- Northpointe's rebuttal to ProPublica article.](https://www.documentcloud.org/documents/2998391-ProPublica-Commentary-Final-070616.html)</li><li>[-- Related academic study.](https://www.liebertpub.com/doi/pdf/10.1089/big.2016.0047)</li><li>[⛔ Google's speech recognition software doesn't recognize women's voices as well as men's.](https://www.dailydot.com/debug/google-voice-recognition-gender-bias/)</li><li>[⛔ Google searches involving black-sounding names are more likely to serve up ads suggestive of a criminal record than white-sounding names.](https://www.technologyreview.com/s/510646/racism-is-poisoning-online-ad-delivery-says-harvard-professor/)</li><li>[⛔ OpenAI's GPT models show racial bias in ranking job applications based on candidate names.](https://www.bloomberg.com/graphics/2024-openai-gpt-hiring-racial-discrimination/)</li></ul>
 **D.3 Metric selection**: Have we considered the effects of optimizing for our defined metrics and considered additional metrics? | <ul><li>[✅ Facebook seeks to optimize "time well spent", prioritizing interaction over popularity.](https://www.wired.com/story/facebook-tweaks-newsfeed-to-favor-content-from-friends-family/)</li><li>[⛔ YouTube's search autofill suggests pedophiliac phrases due to high viewership of related videos.](https://gizmodo.com/youtubes-creepy-kid-problem-was-worse-than-we-thought-1820763240)</li><li>[⛔ A widely used commercial algorithm in the healthcare industry underestimates the care needs of black patients because it optimizes for spending as a proxy for need, introducing racial bias due to unequal access to care.](https://www.science.org/doi/10.1126/science.aax2342)</li></ul>
 **D.4 Explainability**: Can we explain in understandable terms a decision the model made in cases where a justification is needed? | <ul><li>[✅ GDPR includes a "right to explanation," i.e. meaningful information on the logic underlying automated decisions.](hhttps://academic.oup.com/idpl/article/7/4/233/4762325)</li><li>[⛔ Patients with pneumonia with a history of asthma are usually admitted to the intensive care unit as they have a high risk of dying from pneumonia. Given the success of the intensive care, neural networks predicted asthmatics had a low risk of dying and could therefore be sent home. Without explanatory models to identify this issue, patients may have been sent home to die.](http://people.dbmi.columbia.edu/noemie/papers/15kdd.pdf)</li></ul>
 **D.5 Communicate limitations**: Have we communicated the shortcomings, limitations, and biases of the model to relevant stakeholders in ways that can be generally understood? | <ul><li>[✅ OpenAI posted an explanation of how ChatGPT is trained to behave, its limitations, and future directions for improvement.](https://openai.com/index/how-should-ai-systems-behave/)</li><li>[⛔ Google Flu claims to accurately predict weekly influenza activity and then misses the 2009 swine flu pandemic.](https://www.forbes.com/sites/stevensalzberg/2014/03/23/why-google-flu-is-a-failure/#6fa6a1925535)</li></ul>
  | <center>**Deployment**</center>
 **E.1 Monitoring and evaluation**: Do we have a clear plan to monitor the model and its impacts after it is deployed (e.g., performance monitoring, regular audit of sample predictions, human review of high-stakes decisions, reviewing downstream impacts of errors or low-confidence decisions, testing for concept drift)? | <ul><li>[✅ RobotsMali uses AI to create children's books in Mali's native languages, and incorporates human review to ensure that all AI-generated content is accurate and culturally sensitive.](https://restofworld.org/2024/mali-ai-translate-local-language-education/)</li><li>[⛔ Dutch Prime Minister and entire cabinet resign after investigations reveal that 26,000 innocent families were wrongly accused of social benefits fraud partially due to a discriminatory algorithm.](https://www.vice.com/en/article/jgq35d/how-a-discriminatory-algorithm-wrongly-accused-thousands-of-families-of-fraud)</li><li>[⛔ Sending police officers to areas of high predicted crime skews future training data collection as police are repeatedly sent back to the same neighborhoods regardless of the true crime rate.](https://www.smithsonianmag.com/innovation/artificial-intelligence-is-now-used-predict-crime-is-it-biased-180968337/)</li></ul>
-**E.2 Redress**: Have we discussed with our organization a plan for response if users are harmed by the results (e.g., how does the data science team evaluate these cases and update analysis and models to prevent future harm)? | <ul><li>[⛔ Software mistakes result in healthcare cuts for people with diabetes or cerebral palsy.](https://www.theverge.com/2018/3/21/17144260/healthcare-medicaid-algorithm-arkansas-cerebral-palsy)</li></ul>
+**E.2 Redress**: Have we discussed with our organization a plan for response if users are harmed by the results (e.g., how does the data science team evaluate these cases and update analysis and models to prevent future harm)? | <ul><li>[✅ Healing ARC uses a targeted, race-conscious algorithm to counteract documented inequities in access to heart failure care for Black and Latinx patients.](https://catalyst.nejm.org/doi/full/10.1056/CAT.22.0076)</li><li>[⛔ Software mistakes result in healthcare cuts for people with diabetes or cerebral palsy.](https://www.theverge.com/2018/3/21/17144260/healthcare-medicaid-algorithm-arkansas-cerebral-palsy)</li></ul>
 **E.3 Roll back**: Is there a way to turn off or roll back the model in production if necessary? | <ul><li>[⛔ Google "fixes" racist algorithm by removing gorillas from image-labeling technology.](https://www.theverge.com/2018/1/12/16882408/google-racist-gorillas-photo-recognition-algorithm-ai)</li><li>[⛔ Microsoft's Twitter chatbot Tay quickly becomes racist.](https://www.theguardian.com/technology/2016/mar/24/microsoft-scrambles-limit-pr-damage-over-abusive-ai-bot-tay)</li></ul>
 **E.4 Unintended use**: Have we taken steps to identify and prevent unintended uses and abuse of the model and do we have a plan to monitor these once the model is deployed? | <ul><li>[⛔ Generative AI can be exploited to create convincing scams like "virtual kidnapping".](https://www.trendmicro.com/vinfo/us/security/news/cybercrime-and-digital-threats/how-cybercriminals-can-perform-virtual-kidnapping-scams-using-ai-voice-cloning-tools-and-chatgpt)</li><li>[⛔ Deepfakes—realistic but fake videos generated with AI—span the gamut from celebrity porn to presidential statements.](http://theweek.com/articles/777592/rise-deepfakes)</li></ul>
diff --git a/docs/render_templates.py b/docs/render_templates.py
index 0c12f6a..d62c417 100644
--- a/docs/render_templates.py
+++ b/docs/render_templates.py
@@ -55,7 +55,7 @@ def make_table_of_links():
     for r in refs:
         refs_dict[r["line_id"]] = r["links"]
 
-    template = """<center>Checklist Question</center> | <center>Examples of Ethical Issues</center>
+    template = """<center>Checklist Question</center> | <center>Examples</center>
 --- | ---
 {lines}
 """

From 4362effcc04df25159e1a9ba56c00f0567bdc8ff Mon Sep 17 00:00:00 2001
From: Katie Wetstone <klwetstone@gmail.com>
Date: Thu, 13 Jun 2024 10:37:47 -0400
Subject: [PATCH 17/20] tweak readme

---
 README.md                          | 2 +-
 docs/docs/index.md                 | 2 +-
 docs/md_templates/_common_body.tpl | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/README.md b/README.md
index 3b309b1..c094c1c 100644
--- a/README.md
+++ b/README.md
@@ -45,7 +45,7 @@ For more configuration details, see the sections on [command line options](#comm
 
 # What is `deon` designed to do?
 
-We created `deon` with the goal of helping data scientists across the sector to be more intentional in their choices, and more aware of the ethical implications of their work. We use that perspective to make decisions about contributions, issues, PRs, and other maintenance and support activities.
+We created `deon` to help data scientists across the sector be more intentional in their choices, and more aware of the ethical implications of their work. We use that perspective to make decisions about contributions, issues, PRs, and other maintenance and support activities.
 
 
 1. 🔓 **Our goal is not to be arbitrators of what ethical concerns merit inclusion**. We have a [process for changing the default checklist](#changing-the-checklist), but we believe that many domain-specific concerns are not included and teams will benefit from developing [custom checklists](#custom-checklists). Not every checklist item will be relevant. We encourage teams to remove items, sections, or mark items as `N/A` as the concerns of their projects dictate.
diff --git a/docs/docs/index.md b/docs/docs/index.md
index 5648d58..a8f75db 100644
--- a/docs/docs/index.md
+++ b/docs/docs/index.md
@@ -38,7 +38,7 @@ For more configuration details, see the sections on [command line options](#comm
 
 # What is `deon` designed to do?
 
-We created `deon` with the goal of helping data scientists across the sector to be more intentional in their choices, and more aware of the ethical implications of their work. We use that perspective to make decisions about contributions, issues, PRs, and other maintenance and support activities.
+We created `deon` to help data scientists across the sector be more intentional in their choices, and more aware of the ethical implications of their work. We use that perspective to make decisions about contributions, issues, PRs, and other maintenance and support activities.
 
 
 1. 🔓 **Our goal is not to be arbitrators of what ethical concerns merit inclusion**. We have a [process for changing the default checklist](#changing-the-checklist), but we believe that many domain-specific concerns are not included and teams will benefit from developing [custom checklists](#custom-checklists). Not every checklist item will be relevant. We encourage teams to remove items, sections, or mark items as `N/A` as the concerns of their projects dictate.
diff --git a/docs/md_templates/_common_body.tpl b/docs/md_templates/_common_body.tpl
index 718574b..115924d 100644
--- a/docs/md_templates/_common_body.tpl
+++ b/docs/md_templates/_common_body.tpl
@@ -36,7 +36,7 @@ For more configuration details, see the sections on [command line options](#comm
 
 # What is `deon` designed to do?
 
-We created `deon` with the goal of helping data scientists across the sector to be more intentional in their choices, and more aware of the ethical implications of their work. We use that perspective to make decisions about contributions, issues, PRs, and other maintenance and support activities.
+We created `deon` to help data scientists across the sector be more intentional in their choices, and more aware of the ethical implications of their work. We use that perspective to make decisions about contributions, issues, PRs, and other maintenance and support activities.
 
 
 1. 🔓 **Our goal is not to be arbitrators of what ethical concerns merit inclusion**. We have a [process for changing the default checklist](#changing-the-checklist), but we believe that many domain-specific concerns are not included and teams will benefit from developing [custom checklists](#custom-checklists). Not every checklist item will be relevant. We encourage teams to remove items, sections, or mark items as `N/A` as the concerns of their projects dictate.

From 67d74702aaf0563a1998be33f6ba1d8e00fd03f3 Mon Sep 17 00:00:00 2001
From: Katie Wetstone <klwetstone@gmail.com>
Date: Thu, 13 Jun 2024 10:45:20 -0400
Subject: [PATCH 18/20] remove extra css

---
 docs/docs/extra_css/extra.css | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/docs/docs/extra_css/extra.css b/docs/docs/extra_css/extra.css
index 570b4aa..7ef5f78 100644
--- a/docs/docs/extra_css/extra.css
+++ b/docs/docs/extra_css/extra.css
@@ -59,9 +59,4 @@ hr.checklist-buffer {
 	 margin-top: 3em;
 	 border: none;
 	 border-top: medium double #888;
-}
-
-ul.bad {
-	list-style-type: upper-roman;
-	color: red;
 }
\ No newline at end of file

From 0fecd5224fabfa521b676d99c0ac292d85eb2178 Mon Sep 17 00:00:00 2001
From: Katie Wetstone <klwetstone@gmail.com>
Date: Thu, 13 Jun 2024 10:46:25 -0400
Subject: [PATCH 19/20] revert markdown files

---
 README.md             | 29 ++++++++++++--------------
 docs/docs/examples.md | 48 +++++++++++++++++++++----------------------
 docs/docs/index.md    | 29 ++++++++++++--------------
 3 files changed, 50 insertions(+), 56 deletions(-)

diff --git a/README.md b/README.md
index c094c1c..99ae008 100644
--- a/README.md
+++ b/README.md
@@ -43,30 +43,27 @@ Dig into the checklist questions to identify and navigate the ethical considerat
 
 For more configuration details, see the sections on [command line options](#command-line-options), [supported output file types](#supported-file-types), and [custom checklists](#custom-checklists).
 
-# What is `deon` designed to do?
+# Background and perspective
 
-We created `deon` to help data scientists across the sector be more intentional in their choices, and more aware of the ethical implications of their work. We use that perspective to make decisions about contributions, issues, PRs, and other maintenance and support activities.
+We have a particular perspective with this package that we will use to make decisions about contributions, issues, PRs, and other maintenance and support activities.
 
+First and foremost, our goal is not to be arbitrators of what ethical concerns merit inclusion. We have a [process for changing the default checklist](#changing-the-checklist), but we believe that many domain-specific concerns are not included and teams will benefit from developing [custom checklists](#custom-checklists). Not every checklist item will be relevant. We encourage teams to remove items, sections, or mark items as `N/A` as the concerns of their projects dictate.
 
-1. 🔓 **Our goal is not to be arbitrators of what ethical concerns merit inclusion**. We have a [process for changing the default checklist](#changing-the-checklist), but we believe that many domain-specific concerns are not included and teams will benefit from developing [custom checklists](#custom-checklists). Not every checklist item will be relevant. We encourage teams to remove items, sections, or mark items as `N/A` as the concerns of their projects dictate.
+Second, we built our initial list from a set of proposed items on [multiple checklists that we referenced](#checklist-citations). This checklist was heavily inspired by an article written by Mike Loukides, Hilary Mason, and DJ Patil and published by O'Reilly: ["Of Oaths and Checklists"](https://www.oreilly.com/ideas/of-oaths-and-checklists). We owe a great debt to the thinking that proceeded this, and we look forward to thoughtful engagement with the ongoing discussion about checklists for data science ethics.
 
-2. 📊 This checklist is designed to provoke conversations around **issues where data scientists have particular responsibility and perspective**. It's not up to data scientists alone to decide what the ethical course of action is. This has always been a responsibility of organizations that are part of civil society. Conversations should be part of a larger organizational commitment to doing what is right.
+Third, we believe in the power of examples to bring the principles of data ethics to bear on human experience. This repository includes a [list of real-world examples](http://deon.drivendata.org/examples/) connected with each item in the default checklist. We encourage you to contribute relevant use cases that you believe can benefit the community by their example. In addition, if you have a topic, idea, or comment that doesn't seem right for the documentation, please add it to the [wiki page](https://github.com/drivendataorg/deon/wiki) for this project!
 
-3. 💬 Items on the checklist are **meant to provoke discussion** among good-faith actors who take their ethical responsibilities seriously. We are working at a level of abstraction that cannot concretely recommend a specific action (e.g., "remove variable X from your model"). Because of this, most of the items are framed as prompts to discuss or consider. Teams will want to document these discussions and decisions for posterity.
+Fourth, it's not up to data scientists alone to decide what the ethical course of action is. This has always been a responsibility of organizations that are part of civil society. This checklist is designed to provoke conversations around issues where data scientists have particular responsibility and perspective. This conversation should be part of a larger organizational commitment to doing what is right.
 
-4. 🌎 We believe in the **power of examples** to bring the principles of data ethics to bear on human experience. This repository includes a [list of real-world examples](http://deon.drivendata.org/examples/) connected with each item in the default checklist. We encourage you to contribute relevant use cases that you believe can benefit the community by their example. In addition, if you have a topic, idea, or comment that doesn't seem right for the documentation, please add it to the [wiki page](https://github.com/drivendataorg/deon/wiki) for this project!
+Fifth, we believe the primary benefit of a checklist is ensuring that we don't overlook important work. Sometimes it is difficult with pressing deadlines and a demand to multitask to make sure we do the hard work to think about the big picture. This package is meant to help ensure that those discussions happen, even in fast-moving environments. Ethics is hard, and we expect some of the conversations that arise from this checklist may also be hard.
 
-5. 🔍 We believe the primary benefit of a checklist is **ensuring that we don't overlook important work**. Sometimes it is difficult with pressing deadlines and a demand to multitask to make sure we do the hard work to think about the big picture. This package is meant to help ensure that those discussions happen, even in fast-moving environments.
+Sixth, we are working at a level of abstraction that cannot concretely recommend a specific action (e.g., "remove variable X from your model"). Nearly all of the items on the checklist are meant to provoke discussion among good-faith actors who take their ethical responsibilities seriously. Because of this, most of the items are framed as prompts to discuss or consider. Teams will want to document these discussions and decisions for posterity.
 
-6. ❓ We can't define exhaustively every term that appears in the checklist. Some of these **terms are open to interpretation** or mean different things in different contexts. We recommend that when relevant, users create their own glossary for reference.
+Seventh, we can't define exhaustively every term that appears in the checklist. Some of these terms are open to interpretation or mean different things in different contexts. We recommend that when relevant, users create their own glossary for reference.
 
-7. ✨ We want to avoid any items that strictly fall into the realm of statistical best practices. Instead, we want to highlight the areas where we need to pay particular attention **above and beyond best practices**.
+Eighth, we want to avoid any items that strictly fall into the realm of statistical best practices. Instead, we want to highlight the areas where we need to pay particular attention above and beyond best practices.
 
-8. ✅ We want all the checklist items to be **as simple as possible** (but no simpler), and to be actionable.
-
-## Sources
-
-We built our initial list from a set of proposed items on [multiple checklists that we referenced](#checklist-citations). This checklist was heavily inspired by an article written by Mike Loukides, Hilary Mason, and DJ Patil and published by O'Reilly: ["Of Oaths and Checklists"](https://www.oreilly.com/ideas/of-oaths-and-checklists). We owe a great debt to the thinking that proceeded this, and we look forward to thoughtful engagement with the ongoing discussion about checklists for data science ethics.
+Ninth, we want all the checklist items to be as simple as possible (but no simpler), and to be actionable.
 
 # Using this tool
 
@@ -269,9 +266,9 @@ We're excited to see so many articles popping up on data ethics! The short list
 - [Technology is biased too. How do we fix it?](https://fivethirtyeight.com/features/technology-is-biased-too-how-do-we-fix-it/)
 - [The dark secret at the heart of AI](https://www.technologyreview.com/s/604087/the-dark-secret-at-the-heart-of-ai/)
 
-## Data ethics in the real world
+## Where things have gone wrong
 
-To make the ideas contained in the checklist more concrete, we've compiled [examples](http://deon.drivendata.org/examples/) of times when tradoffs were handled well, and times when things have gone wrong. They're paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction.
+To make the ideas contained in the checklist more concrete, we've compiled [examples](http://deon.drivendata.org/examples/) of times when things have gone wrong. They're paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction.
 
 We welcome contributions! Follow [these instructions](https://github.com/drivendataorg/deon/blob/main/CONTRIBUTING.md) to add an example.
 
diff --git a/docs/docs/examples.md b/docs/docs/examples.md
index c1ee2fc..deb11ac 100644
--- a/docs/docs/examples.md
+++ b/docs/docs/examples.md
@@ -1,34 +1,34 @@
 <div style="display: inline-block; line-height: 2.25em">&nbsp;</div>
 
-# Data ethics in the real world
+# Where things have gone wrong
 
-To make the ideas contained in the checklist more concrete, we've compiled **examples** of times when tradoffs were handled well, and times when things have gone wrong. Examples are paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction. Positive examples show how principles of `deon` can be followed in the real world.
+To make the ideas contained in the checklist more concrete, we've compiled examples of times when things have gone wrong. They're paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction.
 
-<center>Checklist Question</center> | <center>Examples</center>
+<center>Checklist Question</center> | <center>Examples of Ethical Issues</center>
 --- | ---
  | <center>**Data Collection**</center>
-**A.1 Informed consent**: If there are human subjects, have they given informed consent, where subjects affirmatively opt-in and have a clear understanding of the data uses to which they consent? | <ul><li>[✅ A voiceover studio is now required to get informed consent from a performer before using their likeness in AI-generated content.](https://variety.com/2024/biz/news/sag-aftra-ai-voiceover-studio-video-games-1235866313/)</li><li>[⛔ Facebook uses phone numbers provided for two-factor authentication to target users with ads.](https://techcrunch.com/2018/09/27/yes-facebook-is-using-your-2fa-phone-number-to-target-you-with-ads/)</li><li>[⛔ African-American men were enrolled in the Tuskegee Study on the progression of syphilis without being told the true purpose of the study or that treatment for syphilis was being withheld.](https://en.wikipedia.org/wiki/Tuskegee_syphilis_experiment)</li><li>[⛔ OpenAI's ChatGPT memorized and regurgitated entire poems without checking for copyright permissions.](https://news.cornell.edu/stories/2024/01/chatgpt-memorizes-and-spits-out-entire-poems)</li></ul>
-**A.2 Collection bias**: Have we considered sources of bias that could be introduced during data collection and survey design and taken steps to mitigate those? | <ul><li>[⛔ StreetBump, a smartphone app to passively detect potholes, may fail to direct public resources to areas where smartphone penetration is lower, such as lower income areas or areas with a larger elderly population.](https://hbr.org/2013/04/the-hidden-biases-in-big-data)</li><li>[⛔ Facial recognition cameras used for passport control register Asian's eyes as closed.](http://content.time.com/time/business/article/0,8599,1954643,00.html)</li></ul>
-**A.3 Limit PII exposure**: Have we considered ways to minimize exposure of personally identifiable information (PII) for example through anonymization or not collecting information that isn't relevant for analysis? | <ul><li>[✅ DuckDuckGo enables users to anonymously access ChatGPT by *not* collecting user IP addresses along with queries.](https://www.theverge.com/2024/6/6/24172719/duckduckgo-private-ai-chats-anonymous-gpt-3-5)</li><li>[⛔ Personal information on taxi drivers can be accessed in poorly anonymized taxi trips dataset released by New York City.](https://www.theguardian.com/technology/2014/jun/27/new-york-taxi-details-anonymised-data-researchers-warn)</li><li>[⛔ Netflix prize dataset of movie rankings by 500,000 customers is easily de-anonymized through cross referencing with other publicly available datasets.](https://www.wired.com/2007/12/why-anonymous-data-sometimes-isnt/)</li></ul>
-**A.4 Downstream bias mitigation**: Have we considered ways to enable testing downstream results for biased outcomes (e.g., collecting data on protected group status like race or gender)? | <ul><li>[⛔ In six major cities, Amazon's same day delivery service excludes many predominantly black neighborhoods.](https://www.bloomberg.com/graphics/2016-amazon-same-day/)</li><li>[⛔ Facial recognition software is significanty worse at identifying people with darker skin.](https://www.theregister.co.uk/2018/02/13/facial_recognition_software_is_better_at_white_men_than_black_women/)</li></ul>
+**A.1 Informed consent**: If there are human subjects, have they given informed consent, where subjects affirmatively opt-in and have a clear understanding of the data uses to which they consent? | <ul><li>[Facebook uses phone numbers provided for two-factor authentication to target users with ads.](https://techcrunch.com/2018/09/27/yes-facebook-is-using-your-2fa-phone-number-to-target-you-with-ads/)</li><li>[African-American men were enrolled in the Tuskegee Study on the progression of syphilis without being told the true purpose of the study or that treatment for syphilis was being withheld.](https://en.wikipedia.org/wiki/Tuskegee_syphilis_experiment)</li><li>[OpenAI's ChatGPT memorized and regurgitated entire poems without checking for copyright permissions.](https://news.cornell.edu/stories/2024/01/chatgpt-memorizes-and-spits-out-entire-poems)</li></ul>
+**A.2 Collection bias**: Have we considered sources of bias that could be introduced during data collection and survey design and taken steps to mitigate those? | <ul><li>[StreetBump, a smartphone app to passively detect potholes, may fail to direct public resources to areas where smartphone penetration is lower, such as lower income areas or areas with a larger elderly population.](https://hbr.org/2013/04/the-hidden-biases-in-big-data)</li><li>[Facial recognition cameras used for passport control register Asian's eyes as closed.](http://content.time.com/time/business/article/0,8599,1954643,00.html)</li></ul>
+**A.3 Limit PII exposure**: Have we considered ways to minimize exposure of personally identifiable information (PII) for example through anonymization or not collecting information that isn't relevant for analysis? | <ul><li>[Personal information on taxi drivers can be accessed in poorly anonymized taxi trips dataset released by New York City.](https://www.theguardian.com/technology/2014/jun/27/new-york-taxi-details-anonymised-data-researchers-warn)</li><li>[Netflix prize dataset of movie rankings by 500,000 customers is easily de-anonymized through cross referencing with other publicly available datasets.](https://www.wired.com/2007/12/why-anonymous-data-sometimes-isnt/)</li></ul>
+**A.4 Downstream bias mitigation**: Have we considered ways to enable testing downstream results for biased outcomes (e.g., collecting data on protected group status like race or gender)? | <ul><li>[In six major cities, Amazon's same day delivery service excludes many predominantly black neighborhoods.](https://www.bloomberg.com/graphics/2016-amazon-same-day/)</li><li>[Facial recognition software is significanty worse at identifying people with darker skin.](https://www.theregister.co.uk/2018/02/13/facial_recognition_software_is_better_at_white_men_than_black_women/)</li></ul>
  | <center>**Data Storage**</center>
-**B.1 Data security**: Do we have a plan to protect and secure data (e.g., encryption at rest and in transit, access controls on internal users and third parties, access logs, and up-to-date software)? | <ul><li>[✅ MediCapt, which documents forensic evidence in conflict regions, effectively protects sensitive information using encryption, limited access, and security audits.](https://phr.org/issues/sexual-violence/medicapt/)</li><li>[⛔ Personal and financial data for more than 146 million people was stolen in Equifax data breach.](https://www.nbcnews.com/news/us-news/equifax-breaks-down-just-how-bad-last-year-s-data-n872496)</li><li>[⛔ Cambridge Analytica harvested private information from over 50 million Facebook profiles without users' permission.](https://www.nytimes.com/2018/03/17/us/politics/cambridge-analytica-trump-campaign.html)</li><li>[⛔ AOL accidentally released 20 million search queries from 658,000 customers.](https://www.wired.com/2006/08/faq-aols-search-gaffe-and-you/)</li></ul>
-**B.2 Right to be forgotten**: Do we have a mechanism through which an individual can request their personal information be removed? | <ul><li>[✅ The EU's General Data Protection Regulation (GDPR) includes the "right to be forgotten."](https://www.eugdpr.org/the-regulation.html)</li></ul>
-**B.3 Data retention plan**: Is there a schedule or plan to delete the data after it is no longer needed? | <ul><li>[⛔ FedEx exposes private information of thousands of customers after a legacy s3 server was left open without a password.](https://www.zdnet.com/article/unsecured-server-exposes-fedex-customer-records/)</li></ul>
+**B.1 Data security**: Do we have a plan to protect and secure data (e.g., encryption at rest and in transit, access controls on internal users and third parties, access logs, and up-to-date software)? | <ul><li>[Personal and financial data for more than 146 million people was stolen in Equifax data breach.](https://www.nbcnews.com/news/us-news/equifax-breaks-down-just-how-bad-last-year-s-data-n872496)</li><li>[Cambridge Analytica harvested private information from over 50 million Facebook profiles without users' permission.](https://www.nytimes.com/2018/03/17/us/politics/cambridge-analytica-trump-campaign.html)</li><li>[AOL accidentally released 20 million search queries from 658,000 customers.](https://www.wired.com/2006/08/faq-aols-search-gaffe-and-you/)</li></ul>
+**B.2 Right to be forgotten**: Do we have a mechanism through which an individual can request their personal information be removed? | <ul><li>[The EU's General Data Protection Regulation (GDPR) includes the "right to be forgotten."](https://www.eugdpr.org/the-regulation.html)</li></ul>
+**B.3 Data retention plan**: Is there a schedule or plan to delete the data after it is no longer needed? | <ul><li>[FedEx exposes private information of thousands of customers after a legacy s3 server was left open without a password.](https://www.zdnet.com/article/unsecured-server-exposes-fedex-customer-records/)</li></ul>
  | <center>**Analysis**</center>
-**C.1 Missing perspectives**: Have we sought to address blindspots in the analysis through engagement with relevant stakeholders (e.g., checking assumptions and discussing implications with affected communities and subject matter experts)? | <ul><li>[✅ Code for America programmatically cleared >140,000 eligible criminal records by collaborating with multiple relevant stakeholders like policymakers, advocacy groups, and courts.](https://codeforamerica.org/programs/criminal-justice/automatic-record-clearance/)</li><li>[⛔ When Apple's HealthKit came out in 2014, women couldn't track menstruation.](https://www.theverge.com/2014/9/25/6844021/apple-promised-an-expansive-health-app-so-why-cant-i-track)</li></ul>
-**C.2 Dataset bias**: Have we examined the data for possible sources of bias and taken steps to mitigate or address these biases (e.g., stereotype perpetuation, confirmation bias, imbalanced classes, or omitted confounding variables)? | <ul><li>[✅ A study by Park et al shows how reweighting can mitigate racial bias when predicting risk of postpartum depression.](https://doi.org/10.1001/jamanetworkopen.2021.3909)</li><li>[⛔ word2vec, trained on Google News corpus, reinforces gender stereotypes.](https://www.technologyreview.com/s/602025/how-vector-space-mathematics-reveals-the-hidden-sexism-in-language/)</li><li>[⛔ Women are more likely to be shown lower-paying jobs than men in Google ads.](https://www.theguardian.com/technology/2015/jul/08/women-less-likely-ads-high-paid-jobs-google-study)</li></ul>
-**C.3 Honest representation**: Are our visualizations, summary statistics, and reports designed to honestly represent the underlying data? | <ul><li>[⛔ Misleading chart shown at Planned Parenthood hearing distorts actual trends of abortions vs. cancer screenings and preventative services.](https://www.politifact.com/truth-o-meter/statements/2015/oct/01/jason-chaffetz/chart-shown-planned-parenthood-hearing-misleading-/)</li><li>[⛔ Georgia Dept. of Health graph of COVID-19 cases falsely suggests a steeper decline when dates are ordered by total cases rather than chronologically.](https://www.vox.com/covid-19-coronavirus-us-response-trump/2020/5/18/21262265/georgia-covid-19-cases-declining-reopening)</li></ul>
-**C.4 Privacy in analysis**: Have we ensured that data with PII are not used or displayed unless necessary for the analysis? | <ul><li>[⛔ Strava heatmap of exercise routes reveals sensitive information on military bases and spy outposts.](https://www.theguardian.com/world/2018/jan/28/fitness-tracking-app-gives-away-location-of-secret-us-army-bases)</li></ul>
-**C.5 Auditability**: Is the process of generating the analysis well documented and reproducible if we discover issues in the future? | <ul><li>[✅ NASA's Transform to Open Science initiative is working to make research more reproducible and accessible.](https://nasa.github.io/Transform-to-Open-Science/)</li><li>[✅ Medic's Community Health Tooklit supports health workers in hard-to-reach areas. The toolkit is fully open source on Github for anyone to view or collaborate.](https://communityhealthtoolkit.org/)</li><li>[⛔ Excel error in well-known economics paper undermines justification of austerity measures.](https://www.bbc.com/news/magazine-22223190)</li></ul>
+**C.1 Missing perspectives**: Have we sought to address blindspots in the analysis through engagement with relevant stakeholders (e.g., checking assumptions and discussing implications with affected communities and subject matter experts)? | <ul><li>[When Apple's HealthKit came out in 2014, women couldn't track menstruation.](https://www.theverge.com/2014/9/25/6844021/apple-promised-an-expansive-health-app-so-why-cant-i-track)</li></ul>
+**C.2 Dataset bias**: Have we examined the data for possible sources of bias and taken steps to mitigate or address these biases (e.g., stereotype perpetuation, confirmation bias, imbalanced classes, or omitted confounding variables)? | <ul><li>[word2vec, trained on Google News corpus, reinforces gender stereotypes.](https://www.technologyreview.com/s/602025/how-vector-space-mathematics-reveals-the-hidden-sexism-in-language/)</li><li>[Women are more likely to be shown lower-paying jobs than men in Google ads.](https://www.theguardian.com/technology/2015/jul/08/women-less-likely-ads-high-paid-jobs-google-study)</li></ul>
+**C.3 Honest representation**: Are our visualizations, summary statistics, and reports designed to honestly represent the underlying data? | <ul><li>[Misleading chart shown at Planned Parenthood hearing distorts actual trends of abortions vs. cancer screenings and preventative services.](https://www.politifact.com/truth-o-meter/statements/2015/oct/01/jason-chaffetz/chart-shown-planned-parenthood-hearing-misleading-/)</li><li>[Georgia Dept. of Health graph of COVID-19 cases falsely suggests a steeper decline when dates are ordered by total cases rather than chronologically.](https://www.vox.com/covid-19-coronavirus-us-response-trump/2020/5/18/21262265/georgia-covid-19-cases-declining-reopening)</li></ul>
+**C.4 Privacy in analysis**: Have we ensured that data with PII are not used or displayed unless necessary for the analysis? | <ul><li>[Strava heatmap of exercise routes reveals sensitive information on military bases and spy outposts.](https://www.theguardian.com/world/2018/jan/28/fitness-tracking-app-gives-away-location-of-secret-us-army-bases)</li></ul>
+**C.5 Auditability**: Is the process of generating the analysis well documented and reproducible if we discover issues in the future? | <ul><li>[Excel error in well-known economics paper undermines justification of austerity measures.](https://www.bbc.com/news/magazine-22223190)</li></ul>
  | <center>**Modeling**</center>
-**D.1 Proxy discrimination**: Have we ensured that the model does not rely on variables or proxies for variables that are unfairly discriminatory? | <ul><li>[✅ Amazon developed an experimental AI recruiting tool, but did not deploy it because it learned to perpetuate bias against women.](https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G)</li><li>[⛔ In hypothetical trials, language models assign the death penalty more frequently to defendants who use African American dialects.](https://arxiv.org/abs/2403.00742)</li><li>[⛔ Variables used to predict child abuse and neglect are direct measurements of poverty, unfairly targeting low-income families for child welfare scrutiny.](https://www.wired.com/story/excerpt-from-automating-inequality/)</li><li>[⛔ Criminal sentencing risk asessments don't ask directly about race or income, but other demographic factors can end up being proxies.](https://www.themarshallproject.org/2015/08/04/the-new-science-of-sentencing)</li><li>[⛔ Creditworthiness algorithms based on nontraditional criteria such as grammatic habits, preferred grocery stores, and friends' credit scores can perpetuate systemic bias.](https://www.whitecase.com/publications/insight/algorithms-and-bias-what-lenders-need-know)</li></ul>
-**D.2 Fairness across groups**: Have we tested model results for fairness with respect to different affected groups (e.g., tested for disparate error rates)? | <ul><li>[✅ A study by Garriga et al uses ML best practices to test for and communicate fairness across racial groups for a model that predicts mental health crises.](https://www.nature.com/articles/s41591-022-01811-5)</li><li>[⛔ Apple credit card offers smaller lines of credit to women than men.](https://www.wired.com/story/the-apple-card-didnt-see-genderand-thats-the-problem/)</li><li>[⛔ With COMPAS, a risk-assessment algorithm used in criminal sentencing, black defendants are almost twice as likely as white defendants to be mislabeled as likely to reoffend.](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing)</li><li>[-- Northpointe's rebuttal to ProPublica article.](https://www.documentcloud.org/documents/2998391-ProPublica-Commentary-Final-070616.html)</li><li>[-- Related academic study.](https://www.liebertpub.com/doi/pdf/10.1089/big.2016.0047)</li><li>[⛔ Google's speech recognition software doesn't recognize women's voices as well as men's.](https://www.dailydot.com/debug/google-voice-recognition-gender-bias/)</li><li>[⛔ Google searches involving black-sounding names are more likely to serve up ads suggestive of a criminal record than white-sounding names.](https://www.technologyreview.com/s/510646/racism-is-poisoning-online-ad-delivery-says-harvard-professor/)</li><li>[⛔ OpenAI's GPT models show racial bias in ranking job applications based on candidate names.](https://www.bloomberg.com/graphics/2024-openai-gpt-hiring-racial-discrimination/)</li></ul>
-**D.3 Metric selection**: Have we considered the effects of optimizing for our defined metrics and considered additional metrics? | <ul><li>[✅ Facebook seeks to optimize "time well spent", prioritizing interaction over popularity.](https://www.wired.com/story/facebook-tweaks-newsfeed-to-favor-content-from-friends-family/)</li><li>[⛔ YouTube's search autofill suggests pedophiliac phrases due to high viewership of related videos.](https://gizmodo.com/youtubes-creepy-kid-problem-was-worse-than-we-thought-1820763240)</li><li>[⛔ A widely used commercial algorithm in the healthcare industry underestimates the care needs of black patients because it optimizes for spending as a proxy for need, introducing racial bias due to unequal access to care.](https://www.science.org/doi/10.1126/science.aax2342)</li></ul>
-**D.4 Explainability**: Can we explain in understandable terms a decision the model made in cases where a justification is needed? | <ul><li>[✅ GDPR includes a "right to explanation," i.e. meaningful information on the logic underlying automated decisions.](hhttps://academic.oup.com/idpl/article/7/4/233/4762325)</li><li>[⛔ Patients with pneumonia with a history of asthma are usually admitted to the intensive care unit as they have a high risk of dying from pneumonia. Given the success of the intensive care, neural networks predicted asthmatics had a low risk of dying and could therefore be sent home. Without explanatory models to identify this issue, patients may have been sent home to die.](http://people.dbmi.columbia.edu/noemie/papers/15kdd.pdf)</li></ul>
-**D.5 Communicate limitations**: Have we communicated the shortcomings, limitations, and biases of the model to relevant stakeholders in ways that can be generally understood? | <ul><li>[✅ OpenAI posted an explanation of how ChatGPT is trained to behave, its limitations, and future directions for improvement.](https://openai.com/index/how-should-ai-systems-behave/)</li><li>[⛔ Google Flu claims to accurately predict weekly influenza activity and then misses the 2009 swine flu pandemic.](https://www.forbes.com/sites/stevensalzberg/2014/03/23/why-google-flu-is-a-failure/#6fa6a1925535)</li></ul>
+**D.1 Proxy discrimination**: Have we ensured that the model does not rely on variables or proxies for variables that are unfairly discriminatory? | <ul><li>[In hypothetical trials, language models assign the death penalty more frequently to defendants who use African American dialects.](https://arxiv.org/abs/2403.00742)</li><li>[Variables used to predict child abuse and neglect are direct measurements of poverty, unfairly targeting low-income families for child welfare scrutiny.](https://www.wired.com/story/excerpt-from-automating-inequality/)</li><li>[Amazon scraps AI recruiting tool that showed bias against women.](https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G)</li><li>[Criminal sentencing risk asessments don't ask directly about race or income, but other demographic factors can end up being proxies.](https://www.themarshallproject.org/2015/08/04/the-new-science-of-sentencing)</li><li>[Creditworthiness algorithms based on nontraditional criteria such as grammatic habits, preferred grocery stores, and friends' credit scores can perpetuate systemic bias.](https://www.whitecase.com/publications/insight/algorithms-and-bias-what-lenders-need-know)</li></ul>
+**D.2 Fairness across groups**: Have we tested model results for fairness with respect to different affected groups (e.g., tested for disparate error rates)? | <ul><li>[Apple credit card offers smaller lines of credit to women than men.](https://www.wired.com/story/the-apple-card-didnt-see-genderand-thats-the-problem/)</li><li>[Google Photos tags two African-Americans as gorillas.](https://www.forbes.com/sites/mzhang/2015/07/01/google-photos-tags-two-african-americans-as-gorillas-through-facial-recognition-software/#12bdb1fd713d)</li><li>[With COMPAS, a risk-assessment algorithm used in criminal sentencing, black defendants are almost twice as likely as white defendants to be mislabeled as likely to reoffend.](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing)</li><li>[-- Northpointe's rebuttal to ProPublica article.](https://www.documentcloud.org/documents/2998391-ProPublica-Commentary-Final-070616.html)</li><li>[-- Related academic study.](https://www.liebertpub.com/doi/pdf/10.1089/big.2016.0047)</li><li>[Google's speech recognition software doesn't recognize women's voices as well as men's.](https://www.dailydot.com/debug/google-voice-recognition-gender-bias/)</li><li>[Google searches involving black-sounding names are more likely to serve up ads suggestive of a criminal record than white-sounding names.](https://www.technologyreview.com/s/510646/racism-is-poisoning-online-ad-delivery-says-harvard-professor/)</li><li>[-- Related academic study.](https://arxiv.org/abs/1301.6822)</li><li>[OpenAI's GPT models show racial bias in ranking job applications based on candidate names.](https://www.bloomberg.com/graphics/2024-openai-gpt-hiring-racial-discrimination/)</li></ul>
+**D.3 Metric selection**: Have we considered the effects of optimizing for our defined metrics and considered additional metrics? | <ul><li>[Facebook seeks to optimize "time well spent", prioritizing interaction over popularity.](https://www.wired.com/story/facebook-tweaks-newsfeed-to-favor-content-from-friends-family/)</li><li>[YouTube's search autofill suggests pedophiliac phrases due to high viewership of related videos.](https://gizmodo.com/youtubes-creepy-kid-problem-was-worse-than-we-thought-1820763240)</li><li>[A widely used commercial algorithm in the healthcare industry underestimates the care needs of black patients because it optimizes for spending as a proxy for need, introducing racial bias due to unequal access to care.](https://www.science.org/doi/10.1126/science.aax2342)</li></ul>
+**D.4 Explainability**: Can we explain in understandable terms a decision the model made in cases where a justification is needed? | <ul><li>[Patients with pneumonia with a history of asthma are usually admitted to the intensive care unit as they have a high risk of dying from pneumonia. Given the success of the intensive care, neural networks predicted asthmatics had a low risk of dying and could therefore be sent home. Without explanatory models to identify this issue, patients may have been sent home to die.](http://people.dbmi.columbia.edu/noemie/papers/15kdd.pdf)</li><li>[GDPR includes a "right to explanation," i.e. meaningful information on the logic underlying automated decisions.](hhttps://academic.oup.com/idpl/article/7/4/233/4762325)</li></ul>
+**D.5 Communicate limitations**: Have we communicated the shortcomings, limitations, and biases of the model to relevant stakeholders in ways that can be generally understood? | <ul><li>[Google Flu claims to accurately predict weekly influenza activity and then misses the 2009 swine flu pandemic.](https://www.forbes.com/sites/stevensalzberg/2014/03/23/why-google-flu-is-a-failure/#6fa6a1925535)</li></ul>
  | <center>**Deployment**</center>
-**E.1 Monitoring and evaluation**: Do we have a clear plan to monitor the model and its impacts after it is deployed (e.g., performance monitoring, regular audit of sample predictions, human review of high-stakes decisions, reviewing downstream impacts of errors or low-confidence decisions, testing for concept drift)? | <ul><li>[✅ RobotsMali uses AI to create children's books in Mali's native languages, and incorporates human review to ensure that all AI-generated content is accurate and culturally sensitive.](https://restofworld.org/2024/mali-ai-translate-local-language-education/)</li><li>[⛔ Dutch Prime Minister and entire cabinet resign after investigations reveal that 26,000 innocent families were wrongly accused of social benefits fraud partially due to a discriminatory algorithm.](https://www.vice.com/en/article/jgq35d/how-a-discriminatory-algorithm-wrongly-accused-thousands-of-families-of-fraud)</li><li>[⛔ Sending police officers to areas of high predicted crime skews future training data collection as police are repeatedly sent back to the same neighborhoods regardless of the true crime rate.](https://www.smithsonianmag.com/innovation/artificial-intelligence-is-now-used-predict-crime-is-it-biased-180968337/)</li></ul>
-**E.2 Redress**: Have we discussed with our organization a plan for response if users are harmed by the results (e.g., how does the data science team evaluate these cases and update analysis and models to prevent future harm)? | <ul><li>[✅ Healing ARC uses a targeted, race-conscious algorithm to counteract documented inequities in access to heart failure care for Black and Latinx patients.](https://catalyst.nejm.org/doi/full/10.1056/CAT.22.0076)</li><li>[⛔ Software mistakes result in healthcare cuts for people with diabetes or cerebral palsy.](https://www.theverge.com/2018/3/21/17144260/healthcare-medicaid-algorithm-arkansas-cerebral-palsy)</li></ul>
-**E.3 Roll back**: Is there a way to turn off or roll back the model in production if necessary? | <ul><li>[⛔ Google "fixes" racist algorithm by removing gorillas from image-labeling technology.](https://www.theverge.com/2018/1/12/16882408/google-racist-gorillas-photo-recognition-algorithm-ai)</li><li>[⛔ Microsoft's Twitter chatbot Tay quickly becomes racist.](https://www.theguardian.com/technology/2016/mar/24/microsoft-scrambles-limit-pr-damage-over-abusive-ai-bot-tay)</li></ul>
-**E.4 Unintended use**: Have we taken steps to identify and prevent unintended uses and abuse of the model and do we have a plan to monitor these once the model is deployed? | <ul><li>[⛔ Generative AI can be exploited to create convincing scams like "virtual kidnapping".](https://www.trendmicro.com/vinfo/us/security/news/cybercrime-and-digital-threats/how-cybercriminals-can-perform-virtual-kidnapping-scams-using-ai-voice-cloning-tools-and-chatgpt)</li><li>[⛔ Deepfakes—realistic but fake videos generated with AI—span the gamut from celebrity porn to presidential statements.](http://theweek.com/articles/777592/rise-deepfakes)</li></ul>
+**E.1 Monitoring and evaluation**: Do we have a clear plan to monitor the model and its impacts after it is deployed (e.g., performance monitoring, regular audit of sample predictions, human review of high-stakes decisions, reviewing downstream impacts of errors or low-confidence decisions, testing for concept drift)? | <ul><li>[Dutch Prime Minister and entire cabinet resign after investigations reveal that 26,000 innocent families were wrongly accused of social benefits fraud partially due to a discriminatory algorithm.](https://www.vice.com/en/article/jgq35d/how-a-discriminatory-algorithm-wrongly-accused-thousands-of-families-of-fraud)</li><li>[Sending police officers to areas of high predicted crime skews future training data collection as police are repeatedly sent back to the same neighborhoods regardless of the true crime rate.](https://www.smithsonianmag.com/innovation/artificial-intelligence-is-now-used-predict-crime-is-it-biased-180968337/)</li></ul>
+**E.2 Redress**: Have we discussed with our organization a plan for response if users are harmed by the results (e.g., how does the data science team evaluate these cases and update analysis and models to prevent future harm)? | <ul><li>[Software mistakes result in healthcare cuts for people with diabetes or cerebral palsy.](https://www.theverge.com/2018/3/21/17144260/healthcare-medicaid-algorithm-arkansas-cerebral-palsy)</li></ul>
+**E.3 Roll back**: Is there a way to turn off or roll back the model in production if necessary? | <ul><li>[Google "fixes" racist algorithm by removing gorillas from image-labeling technology.](https://www.theverge.com/2018/1/12/16882408/google-racist-gorillas-photo-recognition-algorithm-ai)</li><li>[Microsoft's Twitter chatbot Tay quickly becomes racist.](https://www.theguardian.com/technology/2016/mar/24/microsoft-scrambles-limit-pr-damage-over-abusive-ai-bot-tay)</li></ul>
+**E.4 Unintended use**: Have we taken steps to identify and prevent unintended uses and abuse of the model and do we have a plan to monitor these once the model is deployed? | <ul><li>[Generative AI can be exploited to create convincing scams like "virtual kidnapping".](https://www.trendmicro.com/vinfo/us/security/news/cybercrime-and-digital-threats/how-cybercriminals-can-perform-virtual-kidnapping-scams-using-ai-voice-cloning-tools-and-chatgpt)</li><li>[Deepfakes—realistic but fake videos generated with AI—span the gamut from celebrity porn to presidential statements.](http://theweek.com/articles/777592/rise-deepfakes)</li></ul>
\ No newline at end of file
diff --git a/docs/docs/index.md b/docs/docs/index.md
index a8f75db..c597afe 100644
--- a/docs/docs/index.md
+++ b/docs/docs/index.md
@@ -36,30 +36,27 @@ Dig into the checklist questions to identify and navigate the ethical considerat
 
 For more configuration details, see the sections on [command line options](#command-line-options), [supported output file types](#supported-file-types), and [custom checklists](#custom-checklists).
 
-# What is `deon` designed to do?
+# Background and perspective
 
-We created `deon` to help data scientists across the sector be more intentional in their choices, and more aware of the ethical implications of their work. We use that perspective to make decisions about contributions, issues, PRs, and other maintenance and support activities.
+We have a particular perspective with this package that we will use to make decisions about contributions, issues, PRs, and other maintenance and support activities.
 
+First and foremost, our goal is not to be arbitrators of what ethical concerns merit inclusion. We have a [process for changing the default checklist](#changing-the-checklist), but we believe that many domain-specific concerns are not included and teams will benefit from developing [custom checklists](#custom-checklists). Not every checklist item will be relevant. We encourage teams to remove items, sections, or mark items as `N/A` as the concerns of their projects dictate.
 
-1. 🔓 **Our goal is not to be arbitrators of what ethical concerns merit inclusion**. We have a [process for changing the default checklist](#changing-the-checklist), but we believe that many domain-specific concerns are not included and teams will benefit from developing [custom checklists](#custom-checklists). Not every checklist item will be relevant. We encourage teams to remove items, sections, or mark items as `N/A` as the concerns of their projects dictate.
+Second, we built our initial list from a set of proposed items on [multiple checklists that we referenced](#checklist-citations). This checklist was heavily inspired by an article written by Mike Loukides, Hilary Mason, and DJ Patil and published by O'Reilly: ["Of Oaths and Checklists"](https://www.oreilly.com/ideas/of-oaths-and-checklists). We owe a great debt to the thinking that proceeded this, and we look forward to thoughtful engagement with the ongoing discussion about checklists for data science ethics.
 
-2. 📊 This checklist is designed to provoke conversations around **issues where data scientists have particular responsibility and perspective**. It's not up to data scientists alone to decide what the ethical course of action is. This has always been a responsibility of organizations that are part of civil society. Conversations should be part of a larger organizational commitment to doing what is right.
+Third, we believe in the power of examples to bring the principles of data ethics to bear on human experience. This repository includes a [list of real-world examples](http://deon.drivendata.org/examples/) connected with each item in the default checklist. We encourage you to contribute relevant use cases that you believe can benefit the community by their example. In addition, if you have a topic, idea, or comment that doesn't seem right for the documentation, please add it to the [wiki page](https://github.com/drivendataorg/deon/wiki) for this project!
 
-3. 💬 Items on the checklist are **meant to provoke discussion** among good-faith actors who take their ethical responsibilities seriously. We are working at a level of abstraction that cannot concretely recommend a specific action (e.g., "remove variable X from your model"). Because of this, most of the items are framed as prompts to discuss or consider. Teams will want to document these discussions and decisions for posterity.
+Fourth, it's not up to data scientists alone to decide what the ethical course of action is. This has always been a responsibility of organizations that are part of civil society. This checklist is designed to provoke conversations around issues where data scientists have particular responsibility and perspective. This conversation should be part of a larger organizational commitment to doing what is right.
 
-4. 🌎 We believe in the **power of examples** to bring the principles of data ethics to bear on human experience. This repository includes a [list of real-world examples](http://deon.drivendata.org/examples/) connected with each item in the default checklist. We encourage you to contribute relevant use cases that you believe can benefit the community by their example. In addition, if you have a topic, idea, or comment that doesn't seem right for the documentation, please add it to the [wiki page](https://github.com/drivendataorg/deon/wiki) for this project!
+Fifth, we believe the primary benefit of a checklist is ensuring that we don't overlook important work. Sometimes it is difficult with pressing deadlines and a demand to multitask to make sure we do the hard work to think about the big picture. This package is meant to help ensure that those discussions happen, even in fast-moving environments. Ethics is hard, and we expect some of the conversations that arise from this checklist may also be hard.
 
-5. 🔍 We believe the primary benefit of a checklist is **ensuring that we don't overlook important work**. Sometimes it is difficult with pressing deadlines and a demand to multitask to make sure we do the hard work to think about the big picture. This package is meant to help ensure that those discussions happen, even in fast-moving environments.
+Sixth, we are working at a level of abstraction that cannot concretely recommend a specific action (e.g., "remove variable X from your model"). Nearly all of the items on the checklist are meant to provoke discussion among good-faith actors who take their ethical responsibilities seriously. Because of this, most of the items are framed as prompts to discuss or consider. Teams will want to document these discussions and decisions for posterity.
 
-6. ❓ We can't define exhaustively every term that appears in the checklist. Some of these **terms are open to interpretation** or mean different things in different contexts. We recommend that when relevant, users create their own glossary for reference.
+Seventh, we can't define exhaustively every term that appears in the checklist. Some of these terms are open to interpretation or mean different things in different contexts. We recommend that when relevant, users create their own glossary for reference.
 
-7. ✨ We want to avoid any items that strictly fall into the realm of statistical best practices. Instead, we want to highlight the areas where we need to pay particular attention **above and beyond best practices**.
+Eighth, we want to avoid any items that strictly fall into the realm of statistical best practices. Instead, we want to highlight the areas where we need to pay particular attention above and beyond best practices.
 
-8. ✅ We want all the checklist items to be **as simple as possible** (but no simpler), and to be actionable.
-
-## Sources
-
-We built our initial list from a set of proposed items on [multiple checklists that we referenced](#checklist-citations). This checklist was heavily inspired by an article written by Mike Loukides, Hilary Mason, and DJ Patil and published by O'Reilly: ["Of Oaths and Checklists"](https://www.oreilly.com/ideas/of-oaths-and-checklists). We owe a great debt to the thinking that proceeded this, and we look forward to thoughtful engagement with the ongoing discussion about checklists for data science ethics.
+Ninth, we want all the checklist items to be as simple as possible (but no simpler), and to be actionable.
 
 # Using this tool
 
@@ -262,9 +259,9 @@ We're excited to see so many articles popping up on data ethics! The short list
 - [Technology is biased too. How do we fix it?](https://fivethirtyeight.com/features/technology-is-biased-too-how-do-we-fix-it/)
 - [The dark secret at the heart of AI](https://www.technologyreview.com/s/604087/the-dark-secret-at-the-heart-of-ai/)
 
-## Data ethics in the real world
+## Where things have gone wrong
 
-To make the ideas contained in the checklist more concrete, we've compiled [examples](http://deon.drivendata.org/examples/) of times when tradoffs were handled well, and times when things have gone wrong. They're paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction.
+To make the ideas contained in the checklist more concrete, we've compiled [examples](http://deon.drivendata.org/examples/) of times when things have gone wrong. They're paired with the checklist questions to help illuminate where in the process ethics discussions may have helped provide a course correction.
 
 We welcome contributions! Follow [these instructions](https://github.com/drivendataorg/deon/blob/main/CONTRIBUTING.md) to add an example.
 

From f1859704c613a0b228264272ed4c36d8777340a4 Mon Sep 17 00:00:00 2001
From: Katie Wetstone <klwetstone@gmail.com>
Date: Thu, 13 Jun 2024 10:47:15 -0400
Subject: [PATCH 20/20] fully revert examples.md

---
 docs/docs/examples.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/docs/examples.md b/docs/docs/examples.md
index deb11ac..621f332 100644
--- a/docs/docs/examples.md
+++ b/docs/docs/examples.md
@@ -31,4 +31,4 @@ To make the ideas contained in the checklist more concrete, we've compiled examp
 **E.1 Monitoring and evaluation**: Do we have a clear plan to monitor the model and its impacts after it is deployed (e.g., performance monitoring, regular audit of sample predictions, human review of high-stakes decisions, reviewing downstream impacts of errors or low-confidence decisions, testing for concept drift)? | <ul><li>[Dutch Prime Minister and entire cabinet resign after investigations reveal that 26,000 innocent families were wrongly accused of social benefits fraud partially due to a discriminatory algorithm.](https://www.vice.com/en/article/jgq35d/how-a-discriminatory-algorithm-wrongly-accused-thousands-of-families-of-fraud)</li><li>[Sending police officers to areas of high predicted crime skews future training data collection as police are repeatedly sent back to the same neighborhoods regardless of the true crime rate.](https://www.smithsonianmag.com/innovation/artificial-intelligence-is-now-used-predict-crime-is-it-biased-180968337/)</li></ul>
 **E.2 Redress**: Have we discussed with our organization a plan for response if users are harmed by the results (e.g., how does the data science team evaluate these cases and update analysis and models to prevent future harm)? | <ul><li>[Software mistakes result in healthcare cuts for people with diabetes or cerebral palsy.](https://www.theverge.com/2018/3/21/17144260/healthcare-medicaid-algorithm-arkansas-cerebral-palsy)</li></ul>
 **E.3 Roll back**: Is there a way to turn off or roll back the model in production if necessary? | <ul><li>[Google "fixes" racist algorithm by removing gorillas from image-labeling technology.](https://www.theverge.com/2018/1/12/16882408/google-racist-gorillas-photo-recognition-algorithm-ai)</li><li>[Microsoft's Twitter chatbot Tay quickly becomes racist.](https://www.theguardian.com/technology/2016/mar/24/microsoft-scrambles-limit-pr-damage-over-abusive-ai-bot-tay)</li></ul>
-**E.4 Unintended use**: Have we taken steps to identify and prevent unintended uses and abuse of the model and do we have a plan to monitor these once the model is deployed? | <ul><li>[Generative AI can be exploited to create convincing scams like "virtual kidnapping".](https://www.trendmicro.com/vinfo/us/security/news/cybercrime-and-digital-threats/how-cybercriminals-can-perform-virtual-kidnapping-scams-using-ai-voice-cloning-tools-and-chatgpt)</li><li>[Deepfakes—realistic but fake videos generated with AI—span the gamut from celebrity porn to presidential statements.](http://theweek.com/articles/777592/rise-deepfakes)</li></ul>
\ No newline at end of file
+**E.4 Unintended use**: Have we taken steps to identify and prevent unintended uses and abuse of the model and do we have a plan to monitor these once the model is deployed? | <ul><li>[Generative AI can be exploited to create convincing scams like "virtual kidnapping".](https://www.trendmicro.com/vinfo/us/security/news/cybercrime-and-digital-threats/how-cybercriminals-can-perform-virtual-kidnapping-scams-using-ai-voice-cloning-tools-and-chatgpt)</li><li>[Deepfakes—realistic but fake videos generated with AI—span the gamut from celebrity porn to presidential statements.](http://theweek.com/articles/777592/rise-deepfakes)</li></ul>