-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathbasics.html
195 lines (180 loc) · 14.8 KB
/
basics.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
<meta charset="utf-8" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Basics — torchuq documentation</title>
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
<!--[if lt IE 9]>
<script src="_static/js/html5shiv.min.js"></script>
<![endif]-->
<script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>
<script src="_static/jquery.js"></script>
<script src="_static/underscore.js"></script>
<script src="_static/doctools.js"></script>
<script src="_static/js/theme.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Transformations" href="transformation.html" />
<link rel="prev" title="Quick Start" href="quick_start.html" />
</head>
<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
<nav data-toggle="wy-nav-shift" class="wy-nav-side">
<div class="wy-side-scroll">
<div class="wy-side-nav-search" >
<a href="index.html" class="icon icon-home"> torchuq
</a>
<div role="search">
<form id="rtd-search-form" class="wy-form" action="search.html" method="get">
<input type="text" name="q" placeholder="Search docs" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
<li class="toctree-l1"><a class="reference internal" href="quick_start.html">Quick Start</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">Basics</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#prediction-types">1. Prediction types</a></li>
<li class="toctree-l2"><a class="reference internal" href="#evaluation-metrics">2. Evaluation Metrics</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#computing-a-metric">2.1 Computing a metric</a></li>
<li class="toctree-l3"><a class="reference internal" href="#make-a-plot">2.2 Make a plot</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#transform">3. Transform</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#example">Example</a></li>
<li class="toctree-l3"><a class="reference internal" href="#online-prediction">Online Prediction</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="transformation.html">Transformations</a></li>
<li class="toctree-l1"><a class="reference internal" href="api_reference/index.html">API Reference</a></li>
</ul>
</div>
</div>
</nav>
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"><nav class="wy-nav-top" aria-label="Mobile navigation menu" >
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="index.html">torchuq</a>
</nav>
<div class="wy-nav-content">
<div class="rst-content">
<div role="navigation" aria-label="Page navigation">
<ul class="wy-breadcrumbs">
<li><a href="index.html" class="icon icon-home"></a> »</li>
<li>Basics</li>
<li class="wy-breadcrumbs-aside">
<a href="_sources/basics.md.txt" rel="nofollow"> View page source</a>
</li>
</ul>
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
<div itemprop="articleBody">
<section class="tex2jax_ignore mathjax_ignore" id="basics">
<h1>Basics<a class="headerlink" href="#basics" title="Permalink to this headline"></a></h1>
<p>This page contains the basic design philosophy and usage of TorchUQ.</p>
<p>The core of TorchUQ consists of the following three components: prediction types, evaluation, and transformation.</p>
<section id="prediction-types">
<h2>1. Prediction types<a class="headerlink" href="#prediction-types" title="Permalink to this headline"></a></h2>
<p>Before we start to work with any predictions we must think about how to represent our prediction. For example, when predicting the price of a house, we could represent it as a range, such as 120k-150k dollars, or we could represent it as a cumulative density function (CDF). TorchUQ supports around 10 different prediction types for regression and classification problems:</p>
<ul class="simple">
<li><p>For a list of types supported for regression problems, see [link]</p></li>
<li><p>For a list of types supported for classification problems, see [link]</p></li>
</ul>
<p>In the next major update, we intend to support additional prediction types, such as multi-variate predictions.</p>
</section>
<section id="evaluation-metrics">
<h2>2. Evaluation Metrics<a class="headerlink" href="#evaluation-metrics" title="Permalink to this headline"></a></h2>
<p>Given a prediction, we will want to evaluate how good it is at capturing the ground truth. In TorchUQ all evaluation functions are in the <code class="docutils literal notranslate"><span class="pre">torchuq.evaluate</span></code> sub-package, organized by prediction type. For example, functions that evaluate a categorical prediction are in the <code class="docutils literal notranslate"><span class="pre">torchuq.evaluate.categorical</span></code> sub-package, and functions that evaluate a distribution prediction are in the <code class="docutils literal notranslate"><span class="pre">torchuq.evaluate.distribution</span></code> sub-package.</p>
<p>There are two ways to evaluate predictions: by computing a metric (such as the L2 loss or the log-likelihood), or by a visualization plot (such as the reliability diagram).</p>
<section id="computing-a-metric">
<h3>2.1 Computing a metric<a class="headerlink" href="#computing-a-metric" title="Permalink to this headline"></a></h3>
<p>These functions compute a evaluation metric (such as L2 loss or ECE error) on a batch of predictions. All metric computation functions take the following format</p>
<p><code class="docutils literal notranslate"> <span class="pre">torchuq.evaluate.{prediction_type}.compute_{metirc_name}(predictions,</span> <span class="pre">labels,</span> <span class="pre">reduction='mean')</span></code></p>
<p>For example, to compute the ECE of a categorical prediction, use</p>
<p><code class="docutils literal notranslate"><span class="pre">torchuq.evaluate.categorical.compute_ece(predictions,</span> <span class="pre">labels)</span></code></p>
<p>Most metric evaluation functions take three arguments (but some may take more or less arguments)</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">predictions</span></code>: the prediction that we would like to evaluate. Must have the correct type. For example, if we use a function in the module <code class="docutils literal notranslate"><span class="pre">torchuq.evaluate.categorical</span></code> then the prediction must have <code class="docutils literal notranslate"><span class="pre">categorical</span></code> type.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">labels</span></code>: the true labels, not required for all functions. This should be an array of int for classification problems or an array of floats for regression problems.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">reduction</span></code>: the str that decides how the computed metric is aggregated across the batch. This argument works in the same way as in pytorch. For example, the mean reduction indicates that we want to average the evaluation metrics.</p></li>
</ul>
</section>
<section id="make-a-plot">
<h3>2.2 Make a plot<a class="headerlink" href="#make-a-plot" title="Permalink to this headline"></a></h3>
<p>These functions make a plot to visualize the quality of the batch of predictions.</p>
<p><code class="docutils literal notranslate"> <span class="pre">torchuq.evaluate.{predition_type}.plot_{visualization_name}(predictions,</span> <span class="pre">labels,</span> <span class="pre">ax=None)</span></code></p>
<p>For example, to compute the reliability diagram of a categorical prediction, use</p>
<p><code class="docutils literal notranslate"><span class="pre">torchuq.evaluate.categorical.plot_reliability_diagram(predictions,</span> <span class="pre">labels)</span></code></p>
<p>Most visualization functions take three arguments (but some may take more or less arguments)</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">predictions,</span> <span class="pre">labels</span></code>: same as metric evaluation functions</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">ax</span></code>: the matplotlib axes to make the figure on. If <code class="docutils literal notranslate"><span class="pre">ax</span> <span class="pre">is</span> <span class="pre">None</span></code> (recommended), then a new figure (of suitable size) will be created. If <code class="docutils literal notranslate"><span class="pre">ax</span> <span class="pre">is</span> <span class="pre">not</span> <span class="pre">None</span></code> then you should make sure the figure has the right size for visual appeal of the plot.</p></li>
</ul>
</section>
</section>
<section id="transform">
<h2>3. Transform<a class="headerlink" href="#transform" title="Permalink to this headline"></a></h2>
<p>The final and key component of TorchUQ is transformation. We will want to transform low quality predictions (such as uncalibrated predictions) into new high quality predictions, usually with the help of some data.</p>
<p>The main workhorse for transformation is the Calibrator abstract class, which defines a unified interface for an extremely large collection of calibration algorithms (including standard calibration algorithms, conformal prediction, decision calibration, multi-calibration, and many more).</p>
<p><img alt="Calibration Illustration" src="_images/calibrator.png" /></p>
<p>A calibrator class has three main functions:</p>
<ol class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">Calibrator.__init__</span></code>(input_type=’auto’)</p></li>
</ol>
<p>The initialization function can take optional arguments, but has only one required argument, which is the input type—it can be any of the prediction types in the previous section.</p>
<ol class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">Calibrator.train(predictions,</span> <span class="pre">labels,</span> <span class="pre">side_feature=None)</span></code></p></li>
</ol>
<p>The <code class="docutils literal notranslate"><span class="pre">Calibrator.train</span></code> function uses the training data to learn any parameters that is necessary to transform a low quality prediction into a high quality prediction. It takes as input a set of predictions and the corresponding labels. In addition, a few recalibration algorithms—such as group calibration or multicalibration—can take as input additional side features, and the transformation depends on the side feature.</p>
<ol class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">Calibrator.__call__(predictions,</span> <span class="pre">side_feature=None)</span></code></p></li>
</ol>
<p>To use the learned calibrator to transform new data, simply run <code class="docutils literal notranslate"><span class="pre">Calibrator(new_prediction)</span></code>. Note that several calibrator can transform a prediction into a different type. For example, the <code class="docutils literal notranslate"><span class="pre">ConformalCalibrator</span></code> class takes as input an original prediction of any type, and outputs new predictions that are distributions.</p>
<section id="example">
<h3>Example<a class="headerlink" href="#example" title="Permalink to this headline"></a></h3>
<p>For example, to use temperature scaling to recalibrate a categorical prediction use</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">calibrator</span> <span class="o">=</span> <span class="n">TemperatureScaling</span><span class="p">(</span><span class="n">verbose</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">calibrator</span><span class="o">.</span><span class="n">train</span><span class="p">(</span><span class="n">predictions</span><span class="p">,</span> <span class="n">labels</span><span class="p">)</span>
<span class="n">predictions_ts</span> <span class="o">=</span> <span class="n">calibrator</span><span class="p">(</span><span class="n">predictions</span><span class="p">)</span>
</pre></div>
</div>
</section>
<section id="online-prediction">
<h3>Online Prediction<a class="headerlink" href="#online-prediction" title="Permalink to this headline"></a></h3>
<p>Finally many algorithms can be used in the online prediction setup, where data becomes available in a sequential order and can be used to update the best predictor for future data. This is achieved by</p>
<p><code class="docutils literal notranslate"><span class="pre">Calibrator.update(predictions,</span> <span class="pre">labels,</span> <span class="pre">side_feature=None)</span> </code></p>
<p>This function works in the same way as <code class="docutils literal notranslate"><span class="pre">calibrator.train</span></code>—except that instead of training the calibrator from scratch, it updates the calibrator with the new data. For an online prediction example see [link].</p>
</section>
</section>
</section>
</div>
</div>
<footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
<a href="quick_start.html" class="btn btn-neutral float-left" title="Quick Start" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
<a href="transformation.html" class="btn btn-neutral float-right" title="Transformations" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
</div>
<hr/>
<div role="contentinfo">
<p>© Copyright 2021, torchuq team.</p>
</div>
Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
<a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
provided by <a href="https://readthedocs.org">Read the Docs</a>.
</footer>
</div>
</div>
</section>
</div>
<script>
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
</script>
</body>
</html>