Update 6-confidence-intervals.md

episodes/6-confidence-intervals.md

@@ -1,32 +1,33 @@
 ---
+
 title: "Estimating model uncertainty"
-teaching: 20
+teaching: 15
 exercises: 0
+
 ---
 
 :::::::::::::::::::::::::::::::::::::: questions 
 
 - What is model uncertainty, and how can it be categorized?  
-- How do uncertainty estimation methods differ from OOD detection methods?  
+- How do uncertainty estimation methods intersect with OOD detection methods?  
 - What are the computational challenges of estimating model uncertainty?  
 - When is uncertainty estimation useful, and what are its limitations?  
-- Why is OOD detection often preferred over uncertainty estimation in modern applications?  
+- Why is OOD detection often preferred over traditional uncertainty estimation techniques in modern applications?  
 
 ::::::::::::::::::::::::::::::::::::::::::::::::
 
 ::::::::::::::::::::::::::::::::::::: objectives
 
 - Define and distinguish between aleatoric and epistemic uncertainty in machine learning models.  
-- Understand the methods used to estimate model uncertainty, including Monte Carlo dropout, Bayesian neural networks, and model ensembles.  
+- Explore common techniques for estimating aleatoric and epistemic uncertainty.  
+- Understand why OOD detection has become a widely adopted approach in many real-world applications.  
 - Compare and contrast the goals and computational costs of uncertainty estimation and OOD detection.  
-- Explore the strengths and limitations of uncertainty estimation methods in real-world applications.  
-- Recognize scenarios where uncertainty estimation may still be valuable despite its computational cost.
-
+- Summarize when and where different uncertainty estimation methods are most useful.  
+
 ::::::::::::::::::::::::::::::::::::::::::::::::
 
 
 ### Estimating model uncertainty
-We currently provide a high-level overview of uncertainty estimation. Depending on demand, we may expand this section in the future to include specific techniques and their practical applications.
 
 Understanding how confident a model is in its predictions is a valuable tool for building trustworthy AI systems, especially in high-stakes settings like healthcare or autonomous vehicles. Model uncertainty estimation focuses on quantifying the model's confidence and is often used to identify predictions that require further review or caution.
 
@@ -35,26 +36,25 @@ Model uncertainty can be divided into two categories:
 - **Aleatoric uncertainty**: Inherent noise in the data (e.g., overlapping classes) that cannot be reduced, even with more data.
 - **Epistemic uncertainty**: Gaps in the model’s knowledge about the data distribution, which can be reduced by using more data or improved models.
 
-Common techniques for uncertainty estimation include **Monte Carlo dropout**, **Bayesian neural networks**, and **model ensembles**. While these methods provide valuable insights, they are often computationally expensive. For instance:
-
-- Monte Carlo dropout requires performing multiple forward passes through the model for each prediction.
-- Ensembles require training and running multiple models, effectively multiplying the computational cost by the size of the ensemble.
-- Bayesian approaches, while theoretically sound, are computationally prohibitive for large datasets or complex models, making them challenging to scale.
-
-#### How does this compare to OOD detection?
+#### Common techniques and their applications
 
-While uncertainty estimation highlights **low-confidence predictions**, **out-of-distribution (OOD) detection** focuses on identifying inputs that differ significantly from the training data. These inputs are more likely to produce unreliable predictions because the model has never encountered similar data. In practice, OOD methods are often more computationally efficient and scalable compared to uncertainty estimation, as they typically involve a single evaluation pass rather than multiple computations.
+| **Method**                | **Type of uncertainty**  | **Key strengths**                                             | **Key limitations**                                       | **Common use cases**                                                                                             |
+|---------------------------|-------------------------|-------------------------------------------------------------|----------------------------------------------------------|-------------------------------------------------------------------------------------------------------------|
+| **Predictive variance**    | Aleatoric              | Simple, intuitive for regression tasks                     | Limited to regression problems; doesn’t address epistemic uncertainty | Predicting confidence intervals in regression (e.g., house price predictions).                              |
+| **Heteroscedastic models** | Aleatoric              | Models variable noise across inputs                        | Requires specialized architectures or loss functions      | Tasks with varying noise levels across input types (e.g., object detection in noisy environments).           |
+| **Monte Carlo dropout**    | Epistemic             | Easy to implement in existing neural networks              | Computationally expensive due to multiple forward passes  | Flagging low-confidence predictions for medical diagnosis models.                                           |
+| **Bayesian neural nets**   | Epistemic             | Rigorous probabilistic foundation                          | Computationally prohibitive for large models/datasets     | Specialized research tasks requiring interpretable uncertainty measures.                                    |
+| **Ensemble models**        | Epistemic             | Effective and robust; captures diverse uncertainties       | Resource-intensive; requires training multiple models     | Robust predictions in financial risk assessment or autonomous systems.                                      |
+| **OOD detection**          | Epistemic             | Efficient, scalable, excels at rejecting anomalous inputs  | Limited to identifying OOD inputs, not fine-grained uncertainty | Flagging fraudulent transactions, detecting anomalies in vision or NLP pipelines.                           |
 
-As a result, OOD detection is becoming the preferred approach in many cases, particularly for tasks requiring robust detection of anomalous inputs.
+#### Why is OOD detection widely adopted?
 
-#### Weaknesses and use cases
+Among epistemic uncertainty methods, **OOD detection** has become a widely adopted approach in real-world applications due to its ability to efficiently identify inputs that fall outside the training data distribution, where predictions are inherently unreliable. Compared to methods like Monte Carlo dropout or Bayesian neural networks, which require multiple forward passes or computationally expensive probabilistic frameworks, many OOD detection techniques are lightweight and scalable. 
 
-Uncertainty estimation methods have several limitations:
+For example, in autonomous vehicles, OOD detection can help flag unexpected scenarios (e.g., unusual objects on the road) in near real-time, enabling safer decision-making. Similarly, in NLP, OOD methods are used to identify queries or statements that deviate from a model’s training corpus, such as out-of-context questions in a chatbot system. 
 
-- They require access to the model's internal structure (white-box methods), making them less applicable for black-box systems.
-- Their computational cost makes them impractical for applications requiring real-time predictions or working with large datasets.
-- They may struggle to distinguish between uncertainty due to **noisy data** and inputs that are truly **out of distribution**.
-- Despite their complexity, these methods often lack interpretability for stakeholders.
+While OOD detection excels at flagging anomalous inputs, it does not provide fine-grained uncertainty estimates for in-distribution data, making it best suited for tasks where the primary concern is identifying outliers or novel inputs.
 
-Uncertainty estimation is still valuable in scenarios where detailed confidence levels are needed, such as calibrating predictions, improving model robustness, or combining it with OOD detection for better decision-making. However, for many modern use cases, OOD methods provide a more efficient and scalable alternative.
+#### Summary
 
+While uncertainty estimation provides a broad framework for understanding model confidence, different methods are suited for specific types of uncertainty and use cases. OOD detection stands out as the most practical approach for handling epistemic uncertainty in modern applications, thanks to its efficiency and ability to reject anomalous inputs. Together, these methods form a complementary toolkit for building trustworthy AI systems.