-
Notifications
You must be signed in to change notification settings - Fork 102
/
Copy pathnotdata.html
289 lines (274 loc) · 37.9 KB
/
notdata.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<title>Chapter 1 Notations and data | Machine Learning for Factor Investing</title>
<meta name="author" content="Guillaume Coqueret and Tony Guida">
<meta name="generator" content="bookdown 0.24 with bs4_book()">
<meta property="og:title" content="Chapter 1 Notations and data | Machine Learning for Factor Investing">
<meta property="og:type" content="book">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Chapter 1 Notations and data | Machine Learning for Factor Investing">
<!-- JS --><script src="https://cdnjs.cloudflare.com/ajax/libs/clipboard.js/2.0.6/clipboard.min.js" integrity="sha256-inc5kl9MA1hkeYUt+EC3BhlIgyp/2jDIyBLS6k3UxPI=" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/fuse.js/6.4.6/fuse.js" integrity="sha512-zv6Ywkjyktsohkbp9bb45V6tEMoWhzFzXis+LrMehmJZZSys19Yxf1dopHx7WzIKxr5tK2dVcYmaCk2uqdjF4A==" crossorigin="anonymous"></script><script src="https://kit.fontawesome.com/6ecbd6c532.js" crossorigin="anonymous"></script><script src="libs/header-attrs-2.11/header-attrs.js"></script><script src="libs/jquery-3.6.0/jquery-3.6.0.min.js"></script><meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<link href="libs/bootstrap-4.6.0/bootstrap.min.css" rel="stylesheet">
<script src="libs/bootstrap-4.6.0/bootstrap.bundle.min.js"></script><script src="libs/bs3compat-0.3.1/transition.js"></script><script src="libs/bs3compat-0.3.1/tabs.js"></script><script src="libs/bs3compat-0.3.1/bs3compat.js"></script><link href="libs/bs4_book-1.0.0/bs4_book.css" rel="stylesheet">
<script src="libs/bs4_book-1.0.0/bs4_book.js"></script><script src="libs/kePrint-0.0.1/kePrint.js"></script><link href="libs/lightable-0.0.1/lightable.css" rel="stylesheet">
<script src="https://cdnjs.cloudflare.com/ajax/libs/autocomplete.js/0.38.0/autocomplete.jquery.min.js" integrity="sha512-GU9ayf+66Xx2TmpxqJpliWbT5PiGYxpaG8rfnBEk1LL8l1KGkRShhngwdXK1UgqhAzWpZHSiYPc09/NwDQIGyg==" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/mark.js/8.11.1/mark.min.js" integrity="sha512-5CYOlHXGh6QpOFA/TeTylKLWfB3ftPsde7AnmhuitiTX4K5SqCLBeKro6sPS8ilsz1Q4NRx3v8Ko2IBiszzdww==" crossorigin="anonymous"></script><!-- CSS --><meta name="description" content="1.1 Notations .container-fluid main { max-width: 60rem; } This section aims at providing the formal mathematical conventions that will be used throughout the book. Bold notations indicate vectors...">
<meta property="og:description" content="1.1 Notations .container-fluid main { max-width: 60rem; } This section aims at providing the formal mathematical conventions that will be used throughout the book. Bold notations indicate vectors...">
<meta name="twitter:description" content="1.1 Notations .container-fluid main { max-width: 60rem; } This section aims at providing the formal mathematical conventions that will be used throughout the book. Bold notations indicate vectors...">
</head>
<body data-spy="scroll" data-target="#toc">
<div class="container-fluid">
<div class="row">
<header class="col-sm-12 col-lg-3 sidebar sidebar-book"><a class="sr-only sr-only-focusable" href="#content">Skip to main content</a>
<div class="d-flex align-items-start justify-content-between">
<h1>
<a href="index.html" title="">Machine Learning for Factor Investing</a>
</h1>
<button class="btn btn-outline-primary d-lg-none ml-2 mt-1" type="button" data-toggle="collapse" data-target="#main-nav" aria-expanded="true" aria-controls="main-nav"><i class="fas fa-bars"></i><span class="sr-only">Show table of contents</span></button>
</div>
<div id="main-nav" class="collapse-lg">
<form role="search">
<input id="search" class="form-control" type="search" placeholder="Search" aria-label="Search">
</form>
<nav aria-label="Table of contents"><h2>Table of contents</h2>
<ul class="book-toc list-unstyled">
<li><a class="" href="index.html">Preface</a></li>
<li class="book-part">Introduction</li>
<li><a class="active" href="notdata.html"><span class="header-section-number">1</span> Notations and data</a></li>
<li><a class="" href="intro.html"><span class="header-section-number">2</span> Introduction</a></li>
<li><a class="" href="factor.html"><span class="header-section-number">3</span> Factor investing and asset pricing anomalies</a></li>
<li><a class="" href="Data.html"><span class="header-section-number">4</span> Data preprocessing</a></li>
<li class="book-part">Common supervised algorithms</li>
<li><a class="" href="lasso.html"><span class="header-section-number">5</span> Penalized regressions and sparse hedging for minimum variance portfolios</a></li>
<li><a class="" href="trees.html"><span class="header-section-number">6</span> Tree-based methods</a></li>
<li><a class="" href="NN.html"><span class="header-section-number">7</span> Neural networks</a></li>
<li><a class="" href="svm.html"><span class="header-section-number">8</span> Support vector machines</a></li>
<li><a class="" href="bayes.html"><span class="header-section-number">9</span> Bayesian methods</a></li>
<li class="book-part">From predictions to portfolios</li>
<li><a class="" href="valtune.html"><span class="header-section-number">10</span> Validating and tuning</a></li>
<li><a class="" href="ensemble.html"><span class="header-section-number">11</span> Ensemble models</a></li>
<li><a class="" href="backtest.html"><span class="header-section-number">12</span> Portfolio backtesting</a></li>
<li class="book-part">Further important topics</li>
<li><a class="" href="interp.html"><span class="header-section-number">13</span> Interpretability</a></li>
<li><a class="" href="causality.html"><span class="header-section-number">14</span> Two key concepts: causality and non-stationarity</a></li>
<li><a class="" href="unsup.html"><span class="header-section-number">15</span> Unsupervised learning</a></li>
<li><a class="" href="RL.html"><span class="header-section-number">16</span> Reinforcement learning</a></li>
<li class="book-part">Appendix</li>
<li><a class="" href="data-description.html"><span class="header-section-number">17</span> Data description</a></li>
<li><a class="" href="python.html"><span class="header-section-number">18</span> Python notebooks</a></li>
<li><a class="" href="solutions-to-exercises.html"><span class="header-section-number">19</span> Solutions to exercises</a></li>
</ul>
<div class="book-extra">
</div>
</nav>
</div>
</header><main class="col-sm-12 col-md-9 col-lg-7" id="content"><div id="notdata" class="section level1" number="1">
<h1>
<span class="header-section-number">1</span> Notations and data<a class="anchor" aria-label="anchor" href="#notdata"><i class="fas fa-link"></i></a>
</h1>
<div id="notations" class="section level2" number="1.1">
<h2>
<span class="header-section-number">1.1</span> Notations<a class="anchor" aria-label="anchor" href="#notations"><i class="fas fa-link"></i></a>
</h2>
<style>
.container-fluid main {
max-width: 60rem;
}
</style>
<p>This section aims at providing the formal mathematical conventions that will be used throughout the book.</p>
<p>Bold notations indicate vectors and matrices. We use capital letters for matrices and lower case letters for vectors.
<span class="math inline">\(\mathbf{v}'\)</span> and <span class="math inline">\(\mathbf{M}'\)</span> denote the transposes of <span class="math inline">\(\mathbf{v}\)</span> and <span class="math inline">\(\mathbf{M}\)</span>. <span class="math inline">\(\mathbf{M}=[m]_{i,j}\)</span>, where <span class="math inline">\(i\)</span> is the row index and <span class="math inline">\(j\)</span> the column index.</p>
<p>We will work with two notations in parallel. The first one is the pure machine learning notation in which the <strong>labels</strong> (also called <strong>output</strong>, <strong>dependent</strong> variables or <strong>predicted</strong> variables) <span class="math inline">\(\mathbf{y}=y_i\)</span> are approximated by functions of features <span class="math inline">\(\mathbf{X}_i=(x_{i,1},\dots,x_{i,K})\)</span>. The dimension of the feature matrix <span class="math inline">\(\mathbf{X}\)</span> is <span class="math inline">\(I\times K\)</span>: there are <span class="math inline">\(I\)</span> <strong>instances</strong>, <strong>records</strong>, or <strong>observations</strong> and each one of them has <span class="math inline">\(K\)</span> <strong>attributes</strong>, <strong>features</strong>, <strong>inputs</strong>, or <strong>predictors</strong> which will serve as <strong>independent</strong> and <strong>explanatory</strong> variables (all these terms will be used interchangeably). Sometimes, to ease notations, we will write <span class="math inline">\(\textbf{x}_i\)</span> for one instance (one row) of <span class="math inline">\(\textbf{X}\)</span> or <span class="math inline">\(\textbf{x}_k\)</span> for one (feature) column vector of <span class="math inline">\(\textbf{X}\)</span>.</p>
<p>The second notation type pertains to finance and will directly relate to the first. We will often work with discrete returns <span class="math inline">\(r_{t,n}=p_{t,n}/p_{t-1,n}-1\)</span> computed from price data. Here <span class="math inline">\(t\)</span> is the time index and <span class="math inline">\(n\)</span> the asset index. Unless specified otherwise, the return is always computed over one period, though this period can sometimes be one month or one year. Whenever confusion might occur, we will specify other notations for returns.</p>
<p>In line with our previous conventions, the number of return dates will be <span class="math inline">\(T\)</span> and the number of assets, <span class="math inline">\(N\)</span>. The features or characteristics of assets will be denoted with <span class="math inline">\(x_{t,n}^{(k)}\)</span>: it is the time-<span class="math inline">\(t\)</span> value of the <span class="math inline">\(k^{th}\)</span> attribute of firm or asset <span class="math inline">\(n\)</span>. In stacked notation, <span class="math inline">\(\mathbf{x}_{t,n}\)</span> will stand for the vector of characteristics of asset <span class="math inline">\(n\)</span> at time <span class="math inline">\(t\)</span>. Moreover, <span class="math inline">\(\mathbf{r}_t\)</span> stands for all returns at time <span class="math inline">\(t\)</span> while <span class="math inline">\(\mathbf{r}_n\)</span> stands for all returns of asset <span class="math inline">\(n\)</span>. Often, returns will play the role of the dependent variable, or label (in ML terms). For the riskless asset, we will use the notation <span class="math inline">\(r_{t,f}\)</span>.</p>
<p>The link between the two notations will most of the time be the following. One <strong>instance</strong> (or <strong>observation</strong>) <span class="math inline">\(i\)</span> will consist of one couple (<span class="math inline">\(t,n\)</span>) of one particular date and one particular firm (if the data is perfectly rectangular with no missing field, <span class="math inline">\(I=T\times N\)</span>). The label will usually be some performance measure of the firm computed over some future period, while the features will consist of the firm attributes at time <span class="math inline">\(t\)</span>. Hence, the purpose of the machine learning engine in factor investing will be to determine the model that maps the time-<span class="math inline">\(t\)</span> characteristics of firms to their future performance.</p>
<p>In terms of canonical matrices: <span class="math inline">\(\mathbf{I}_N\)</span> will denote the <span class="math inline">\((N\times N)\)</span> identity matrix.</p>
<p>From the probabilistic literature, we employ the expectation operator <span class="math inline">\(\mathbb{E}[\cdot]\)</span> and the conditional expectation <span class="math inline">\(\mathbb{E}_t[\cdot]\)</span>, where the corresponding filtration <span class="math inline">\(\mathcal{F}_t\)</span> corresponds to all information available at time <span class="math inline">\(t\)</span>. More precisely, <span class="math inline">\(\mathbb{E}_t[\cdot]=\mathbb{E}[\cdot | \mathcal{F}_t]\)</span>. <span class="math inline">\(\mathbb{V}[\cdot]\)</span> will denote the variance operator. Depending on the context, probabilities will be written simply <span class="math inline">\(P\)</span>, but sometimes we will use the heavier notation <span class="math inline">\(\mathbb{P}\)</span>. Probability density functions (pdfs) will be denoted with lowercase letters (<span class="math inline">\(f\)</span>) and cumulative distribution functions (cdfs) with uppercase letters (<span class="math inline">\(F\)</span>). We will write equality in distribution as <span class="math inline">\(X \overset{d}{=}Y\)</span>, which is equivalent to <span class="math inline">\(F_X(z)=F_Y(z)\)</span> for all <span class="math inline">\(z\)</span> on the support of the variables. For a random process <span class="math inline">\(X_t\)</span>, we say that it is <strong>stationary</strong> if the law of <span class="math inline">\(X_t\)</span> is constant through time, i.e., <span class="math inline">\(X_t\overset{d}{=}X_s\)</span>, where <span class="math inline">\(\overset{d}{=}\)</span> means equality in distribution.</p>
<p>Sometimes, asymptotic behaviors will be characterized with the usual Landau notation <span class="math inline">\(o(\cdot)\)</span> and <span class="math inline">\(O(\cdot)\)</span>. The symbol <span class="math inline">\(\propto\)</span> refers to proportionality: <span class="math inline">\(x\propto y\)</span> means that <span class="math inline">\(x\)</span> is proportional to <span class="math inline">\(y\)</span>. With respect to derivatives, we use the standard notation <span class="math inline">\(\frac{\partial}{\partial x}\)</span> when differentiating with respect to <span class="math inline">\(x\)</span>. We resort to the compact symbol <span class="math inline">\(\nabla\)</span> when all derivatives are computed (gradient vector).</p>
<p>In equations, the left-hand side and right-hand side can be written more compactly: l.h.s. and r.h.s., respectively.</p>
<p>Finally, we turn to functions. We list a few below:<br>
- <span class="math inline">\(1_{\{x \}}\)</span>: the indicator function of the condition <span class="math inline">\(x\)</span>, which is equal to one if <span class="math inline">\(x\)</span> is true and to zero otherwise.<br>
- <span class="math inline">\(\phi(\cdot)\)</span> and <span class="math inline">\(\Phi(\cdot)\)</span> are the standard Gaussian pdf and cdf.<br>
- card<span class="math inline">\((\cdot)=\#(\cdot)\)</span> are two notations for the cardinal function which evaluates the number of elements in a given set (provided as argument of the function).<br>
- <span class="math inline">\(\lfloor \cdot \rfloor\)</span> is the integer part function.<br>
- for a real number <span class="math inline">\(x\)</span>, <span class="math inline">\([x]^+\)</span> is the positive part of <span class="math inline">\(x\)</span>, that is <span class="math inline">\(\max(0,x)\)</span>.<br>
- tanh<span class="math inline">\((\cdot)\)</span> is the hyperbolic tangent: tanh<span class="math inline">\((x)=\frac{e^x-e^{-x}}{e^x+e^{-x}}\)</span>.<br>
- ReLu<span class="math inline">\((\cdot)\)</span> is the rectified linear unit: ReLu<span class="math inline">\((x)=\max(0,x)\)</span>.<br>
- s<span class="math inline">\((\cdot)\)</span> will be the softmax function: <span class="math inline">\(s(\textbf{x})_i=\frac{e^{x_i}}{\sum_{j=1}^Je^{x_j}}\)</span>, where the subscript <span class="math inline">\(i\)</span> refers to the <span class="math inline">\(i^{th}\)</span> element of the vector.</p>
</div>
<div id="dataset" class="section level2" number="1.2">
<h2>
<span class="header-section-number">1.2</span> Dataset<a class="anchor" aria-label="anchor" href="#dataset"><i class="fas fa-link"></i></a>
</h2>
<p>Throughout the book, and for the sake of <strong>reproducibility</strong>, we will illustrate the concepts we present with examples of implementation based on a single financial dataset available at <a href="https://github.com/shokru/mlfactor.github.io/tree/master/material" class="uri">https://github.com/shokru/mlfactor.github.io/tree/master/material</a>. This dataset comprises information on 1,207 stocks listed in the US (possibly originating from Canada or Mexico). The time range starts in November 1998 and ends in March 2019. For each point in time, 93 <strong>characteristics</strong> describe the firms in the sample. These attributes cover a wide range of topics:</p>
<ul>
<li>
<strong>valuation</strong> (earning yields, accounting ratios);<br>
</li>
<li>
<strong>profitability</strong> and quality (return on equity);<br>
</li>
<li>
<strong>momentum</strong> and technical analysis (past returns, relative strength index);<br>
</li>
<li>
<strong>risk</strong> (volatilities);<br>
</li>
<li>
<strong>estimates</strong> (earnings-per-share);<br>
</li>
<li>
<strong>volume</strong> and <strong>liquidity</strong> (share turnover).</li>
</ul>
<p>The sample is not perfectly rectangular: there are no missing points, but the number of firms and their attributes is not constant through time. This makes the computations in the backtest more tricky, but also more realistic.</p>
<div class="sourceCode" id="cb3"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va"><a href="https://tidyverse.tidyverse.org">tidyverse</a></span><span class="op">)</span> <span class="co"># Activate the data science package</span>
<span class="kw"><a href="https://rdrr.io/r/base/library.html">library</a></span><span class="op">(</span><span class="va"><a href="https://lubridate.tidyverse.org">lubridate</a></span><span class="op">)</span> <span class="co"># Activate the date management package</span>
<span class="fu"><a href="https://rdrr.io/r/base/load.html">load</a></span><span class="op">(</span><span class="st">"data_ml.RData"</span><span class="op">)</span> <span class="co"># Load the data</span>
<span class="va">data_ml</span> <span class="op"><-</span> <span class="va">data_ml</span> <span class="op"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%>%</a></span>
<span class="fu"><a href="https://dplyr.tidyverse.org/reference/filter.html">filter</a></span><span class="op">(</span><span class="va">date</span> <span class="op">></span> <span class="st">"1999-12-31"</span>, <span class="co"># Keep the date with sufficient data points</span>
<span class="va">date</span> <span class="op"><</span> <span class="st">"2019-01-01"</span><span class="op">)</span> <span class="op"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%>%</a></span>
<span class="fu"><a href="https://dplyr.tidyverse.org/reference/arrange.html">arrange</a></span><span class="op">(</span><span class="va">stock_id</span>, <span class="va">date</span><span class="op">)</span> <span class="co"># Order the data</span>
<span class="va">data_ml</span><span class="op">[</span><span class="fl">1</span><span class="op">:</span><span class="fl">6</span>, <span class="fl">1</span><span class="op">:</span><span class="fl">6</span><span class="op">]</span> <span class="co"># Sample values</span></code></pre></div>
<pre><code>## # A tibble: 6 × 6
## stock_id date Advt_12M_Usd Advt_3M_Usd Advt_6M_Usd Asset_Turnover
## <int> <date> <dbl> <dbl> <dbl> <dbl>
## 1 1 2000-01-31 0.41 0.39 0.42 0.19
## 2 1 2000-02-29 0.41 0.39 0.4 0.19
## 3 1 2000-03-31 0.4 0.37 0.37 0.2
## 4 1 2000-04-30 0.39 0.36 0.37 0.2
## 5 1 2000-05-31 0.4 0.42 0.4 0.2
## 6 1 2000-06-30 0.41 0.47 0.42 0.21</code></pre>
<p></p>
<p>The data has 99 columns and 268336 rows. The first two columns indicate the stock identifier and the date. The next 93 columns are the features (see Table <a href="data-description.html#tab:appendix1">17.1</a> in the Appendix for details). The last four columns are the labels. The points are sampled at the monthly frequency. As is always the case in practice, the number of assets changes with time, as is shown in Figure <a href="notdata.html#fig:nbassets">1.1</a>.</p>
<div class="sourceCode" id="cb5"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="va">data_ml</span> <span class="op"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%>%</a></span>
<span class="fu"><a href="https://dplyr.tidyverse.org/reference/group_by.html">group_by</a></span><span class="op">(</span><span class="va">date</span><span class="op">)</span> <span class="op"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%>%</a></span> <span class="co"># Group by date</span>
<span class="fu"><a href="https://dplyr.tidyverse.org/reference/summarise.html">summarize</a></span><span class="op">(</span>nb_assets <span class="op">=</span> <span class="va">stock_id</span> <span class="op"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%>%</a></span> <span class="co"># Count nb assets</span>
<span class="fu"><a href="https://rdrr.io/r/base/factor.html">as.factor</a></span><span class="op">(</span><span class="op">)</span> <span class="op"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%>%</a></span> <span class="fu"><a href="https://rdrr.io/r/base/nlevels.html">nlevels</a></span><span class="op">(</span><span class="op">)</span><span class="op">)</span> <span class="op"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%>%</a></span>
<span class="fu"><a href="https://ggplot2.tidyverse.org/reference/ggplot.html">ggplot</a></span><span class="op">(</span><span class="fu"><a href="https://ggplot2.tidyverse.org/reference/aes.html">aes</a></span><span class="op">(</span>x <span class="op">=</span> <span class="va">date</span>, y <span class="op">=</span> <span class="va">nb_assets</span><span class="op">)</span><span class="op">)</span> <span class="op">+</span> <span class="fu"><a href="https://ggplot2.tidyverse.org/reference/geom_bar.html">geom_col</a></span><span class="op">(</span><span class="op">)</span> <span class="op">+</span> <span class="co"># Plot</span>
<span class="fu"><a href="https://ggplot2.tidyverse.org/reference/ggtheme.html">theme_light</a></span><span class="op">(</span><span class="op">)</span> <span class="op">+</span> <span class="fu"><a href="https://ggplot2.tidyverse.org/reference/coord_fixed.html">coord_fixed</a></span><span class="op">(</span><span class="fl">3</span><span class="op">)</span></code></pre></div>
<div class="figure" style="text-align: center">
<span style="display:block;" id="fig:nbassets"></span>
<img src="ML_factor_files/figure-html/nbassets-1.png" alt="Number of assets through time." width="600"><p class="caption">
FIGURE 1.1: Number of assets through time.
</p>
</div>
<p></p>
<p>There are four immediate <strong>labels</strong> in the dataset: R1M_Usd, R3M_Usd, R6M_Usd and R12M_Usd, which correspond to the 1-month, 3-month, 6-month and 12-month future/forward returns of the stocks. The returns are <strong>total returns</strong>, that is, they incorporate potential <strong>dividend</strong> payments over the considered periods. This is a better proxy of financial gain compared to price returns only. We refer to the analysis of <span class="citation">Hartzmark and Solomon (<a href="solutions-to-exercises.html#ref-hartzmark2019dividend" role="doc-biblioref">2019</a>)</span> for a study on the impact of decoupling price returns and dividends. These labels are located in the last 4 columns of the dataset. We provide their descriptive statistics below.</p>
<pre><code>## # A tibble: 4 × 5
## Label mean sd min max
## <chr> <dbl> <dbl> <dbl> <dbl>
## 1 R12M_Usd 0.137 0.738 -0.991 96.0
## 2 R1M_Usd 0.0127 0.176 -0.922 30.2
## 3 R3M_Usd 0.0369 0.328 -0.929 39.4
## 4 R6M_Usd 0.0723 0.527 -0.98 107.</code></pre>
<p></p>
<p>In anticipation for future models, we keep the name of the predictors in memory. In addition, we also keep a much shorter list of predictors.</p>
<div class="sourceCode" id="cb7"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="va">features</span> <span class="op"><-</span> <span class="fu"><a href="https://rdrr.io/r/base/colnames.html">colnames</a></span><span class="op">(</span><span class="va">data_ml</span><span class="op">[</span><span class="fl">3</span><span class="op">:</span><span class="fl">95</span><span class="op">]</span><span class="op">)</span> <span class="co"># Keep the feature's column names (hard-coded, beware!)</span>
<span class="va">features_short</span> <span class="op"><-</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html">c</a></span><span class="op">(</span><span class="st">"Div_Yld"</span>, <span class="st">"Eps"</span>, <span class="st">"Mkt_Cap_12M_Usd"</span>, <span class="st">"Mom_11M_Usd"</span>,
<span class="st">"Ocf"</span>, <span class="st">"Pb"</span>, <span class="st">"Vol1Y_Usd"</span><span class="op">)</span></code></pre></div>
<p></p>
<p>The predictors have been uniformized, that is, for any given feature and time point, the distribution is uniform. Given 1,207 stocks, the graph below cannot display a perfect rectangle.</p>
<div class="sourceCode" id="cb8"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="va">data_ml</span> <span class="op"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%>%</a></span>
<span class="fu"><a href="https://dplyr.tidyverse.org/reference/filter.html">filter</a></span><span class="op">(</span><span class="va">date</span> <span class="op">==</span> <span class="st">"2000-02-29"</span><span class="op">)</span> <span class="op"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%>%</a></span>
<span class="fu"><a href="https://ggplot2.tidyverse.org/reference/ggplot.html">ggplot</a></span><span class="op">(</span><span class="fu"><a href="https://ggplot2.tidyverse.org/reference/aes.html">aes</a></span><span class="op">(</span>x <span class="op">=</span> <span class="va">Div_Yld</span><span class="op">)</span><span class="op">)</span> <span class="op">+</span> <span class="fu"><a href="https://ggplot2.tidyverse.org/reference/geom_histogram.html">geom_histogram</a></span><span class="op">(</span>bins <span class="op">=</span> <span class="fl">100</span><span class="op">)</span> <span class="op">+</span>
<span class="fu"><a href="https://ggplot2.tidyverse.org/reference/ggtheme.html">theme_light</a></span><span class="op">(</span><span class="op">)</span> <span class="op">+</span> <span class="fu"><a href="https://ggplot2.tidyverse.org/reference/coord_fixed.html">coord_fixed</a></span><span class="op">(</span><span class="fl">0.03</span><span class="op">)</span></code></pre></div>
<div class="figure" style="text-align: center">
<span style="display:block;" id="fig:datarectangle"></span>
<img src="ML_factor_files/figure-html/datarectangle-1.png" alt="Distribution of the dividend yield feature on date 2000-02-29." width="600"><p class="caption">
FIGURE 1.2: Distribution of the dividend yield feature on date 2000-02-29.
</p>
</div>
<p></p>
<p>The original labels (future returns) are numerical and will be used for regression exercises, that is, when the objective is to predict a scalar real number. Sometimes, the exercises can be different and the purpose may be to forecast <strong>categories</strong> (also called <strong>classes</strong>), like “buy”, “hold” or “sell”. In order to be able to perform this type of classification analysis, we create additional labels that are categorical.</p>
<div class="sourceCode" id="cb9"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="va">data_ml</span> <span class="op"><-</span> <span class="va">data_ml</span> <span class="op"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%>%</a></span>
<span class="fu"><a href="https://dplyr.tidyverse.org/reference/group_by.html">group_by</a></span><span class="op">(</span><span class="va">date</span><span class="op">)</span> <span class="op"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%>%</a></span> <span class="co"># Group by date</span>
<span class="fu"><a href="https://dplyr.tidyverse.org/reference/mutate.html">mutate</a></span><span class="op">(</span>R1M_Usd_C <span class="op">=</span> <span class="va">R1M_Usd</span> <span class="op">></span> <span class="fu"><a href="https://rdrr.io/r/stats/median.html">median</a></span><span class="op">(</span><span class="va">R1M_Usd</span><span class="op">)</span>, <span class="co"># Create the categorical labels</span>
R12M_Usd_C <span class="op">=</span> <span class="va">R12M_Usd</span> <span class="op">></span> <span class="fu"><a href="https://rdrr.io/r/stats/median.html">median</a></span><span class="op">(</span><span class="va">R12M_Usd</span><span class="op">)</span><span class="op">)</span> <span class="op"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%>%</a></span>
<span class="fu"><a href="https://dplyr.tidyverse.org/reference/group_by.html">ungroup</a></span><span class="op">(</span><span class="op">)</span> <span class="op"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%>%</a></span>
<span class="fu"><a href="https://dplyr.tidyverse.org/reference/mutate_all.html">mutate_if</a></span><span class="op">(</span><span class="va">is.logical</span>, <span class="va">as.factor</span><span class="op">)</span></code></pre></div>
<p></p>
<p>The new labels are binary: they are equal to 1 (true) if the original return is above that of the median return over the considered period and to 0 (false) if not. Hence, at each point in time, half of the sample has a label equal to zero and the other half to one: some stocks overperform and others underperform.</p>
<p>In machine learning, models are estimated on one portion of data (<strong>training set</strong>) and then tested on another portion of the data (<strong>testing set</strong>) to assess their quality. We split our sample accordingly.</p>
<div class="sourceCode" id="cb10"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="va">separation_date</span> <span class="op"><-</span> <span class="fu"><a href="https://rdrr.io/r/base/as.Date.html">as.Date</a></span><span class="op">(</span><span class="st">"2014-01-15"</span><span class="op">)</span>
<span class="va">training_sample</span> <span class="op"><-</span> <span class="fu"><a href="https://dplyr.tidyverse.org/reference/filter.html">filter</a></span><span class="op">(</span><span class="va">data_ml</span>, <span class="va">date</span> <span class="op"><</span> <span class="va">separation_date</span><span class="op">)</span>
<span class="va">testing_sample</span> <span class="op"><-</span> <span class="fu"><a href="https://dplyr.tidyverse.org/reference/filter.html">filter</a></span><span class="op">(</span><span class="va">data_ml</span>, <span class="va">date</span> <span class="op">>=</span> <span class="va">separation_date</span><span class="op">)</span></code></pre></div>
<p></p>
<p>We also keep in memory a few key variables, like the list of asset identifiers and a rectangular version of returns. For simplicity, in the computation of the latter, we shrink the investment universe to keep only the stocks for which we have the maximum number of points.</p>
<div class="sourceCode" id="cb11"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="va">stock_ids</span> <span class="op"><-</span> <span class="fu"><a href="https://rdrr.io/r/base/levels.html">levels</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/r/base/factor.html">as.factor</a></span><span class="op">(</span><span class="va">data_ml</span><span class="op">$</span><span class="va">stock_id</span><span class="op">)</span><span class="op">)</span> <span class="co"># A list of all stock_ids</span>
<span class="va">stock_days</span> <span class="op"><-</span> <span class="va">data_ml</span> <span class="op"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%>%</a></span> <span class="co"># Compute the number of data points per stock</span>
<span class="fu"><a href="https://dplyr.tidyverse.org/reference/group_by.html">group_by</a></span><span class="op">(</span><span class="va">stock_id</span><span class="op">)</span> <span class="op"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%>%</a></span> <span class="fu"><a href="https://dplyr.tidyverse.org/reference/summarise.html">summarize</a></span><span class="op">(</span>nb <span class="op">=</span> <span class="fu"><a href="https://dplyr.tidyverse.org/reference/context.html">n</a></span><span class="op">(</span><span class="op">)</span><span class="op">)</span>
<span class="va">stock_ids_short</span> <span class="op"><-</span> <span class="va">stock_ids</span><span class="op">[</span><span class="fu"><a href="https://rdrr.io/r/base/which.html">which</a></span><span class="op">(</span><span class="va">stock_days</span><span class="op">$</span><span class="va">nb</span> <span class="op">==</span> <span class="fu"><a href="https://rdrr.io/r/base/Extremes.html">max</a></span><span class="op">(</span><span class="va">stock_days</span><span class="op">$</span><span class="va">nb</span><span class="op">)</span><span class="op">)</span><span class="op">]</span> <span class="co"># Stocks with full data</span>
<span class="va">returns</span> <span class="op"><-</span> <span class="va">data_ml</span> <span class="op"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%>%</a></span> <span class="co"># Compute returns, in matrix format, in 3 steps:</span>
<span class="fu"><a href="https://dplyr.tidyverse.org/reference/filter.html">filter</a></span><span class="op">(</span><span class="va">stock_id</span> <span class="op"><a href="https://rdrr.io/r/base/match.html">%in%</a></span> <span class="va">stock_ids_short</span><span class="op">)</span> <span class="op"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%>%</a></span> <span class="co"># 1. Filtering the data</span>
<span class="fu">dplyr</span><span class="fu">::</span><span class="fu"><a href="https://dplyr.tidyverse.org/reference/select.html">select</a></span><span class="op">(</span><span class="va">date</span>, <span class="va">stock_id</span>, <span class="va">R1M_Usd</span><span class="op">)</span> <span class="op"><a href="https://magrittr.tidyverse.org/reference/pipe.html">%>%</a></span> <span class="co"># 2. Keep returns along with dates & firm names</span>
<span class="fu"><a href="https://tidyr.tidyverse.org/reference/pivot_wider.html">pivot_wider</a></span><span class="op">(</span>names_from <span class="op">=</span> <span class="st">"stock_id"</span>,
values_from <span class="op">=</span> <span class="st">"R1M_Usd"</span><span class="op">)</span> <span class="co"># 3. Put in matrix shape </span></code></pre></div>
<p></p>
</div>
</div>
<div class="chapter-nav">
<div class="prev"><a href="index.html">Preface</a></div>
<div class="next"><a href="intro.html"><span class="header-section-number">2</span> Introduction</a></div>
</div></main><div class="col-md-3 col-lg-2 d-none d-md-block sidebar sidebar-chapter">
<nav id="toc" data-toggle="toc" aria-label="On this page"><h2>On this page</h2>
<ul class="nav navbar-nav">
<li><a class="nav-link" href="#notdata"><span class="header-section-number">1</span> Notations and data</a></li>
<li><a class="nav-link" href="#notations"><span class="header-section-number">1.1</span> Notations</a></li>
<li><a class="nav-link" href="#dataset"><span class="header-section-number">1.2</span> Dataset</a></li>
</ul>
<div class="book-extra">
<ul class="list-unstyled">
</ul>
</div>
</nav>
</div>
</div>
</div> <!-- .container -->
<footer class="bg-primary text-light mt-5"><div class="container"><div class="row">
<div class="col-12 col-md-6 mt-3">
<p>"<strong>Machine Learning for Factor Investing</strong>" was written by Guillaume Coqueret and Tony Guida. It was last built on 2022-10-18.</p>
</div>
<div class="col-12 col-md-6 mt-3">
<p>This book was built by the <a class="text-light" href="https://bookdown.org">bookdown</a> R package.</p>
</div>
</div></div>
</footer><!-- dynamically load mathjax for compatibility with self-contained --><script>
(function () {
var script = document.createElement("script");
script.type = "text/javascript";
var src = "true";
if (src === "" || src === "true") src = "https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-MML-AM_CHTML";
if (location.protocol !== "file:")
if (/^https?:/.test(src))
src = src.replace(/^https?:/, '');
script.src = src;
document.getElementsByTagName("head")[0].appendChild(script);
})();
</script><script type="text/x-mathjax-config">const popovers = document.querySelectorAll('a.footnote-ref[data-toggle="popover"]');
for (let popover of popovers) {
const div = document.createElement('div');
div.setAttribute('style', 'position: absolute; top: 0, left:0; width:0, height:0, overflow: hidden; visibility: hidden;');
div.innerHTML = popover.getAttribute('data-content');
var has_math = div.querySelector("span.math");
if (has_math) {
document.body.appendChild(div);
MathJax.Hub.Queue(["Typeset", MathJax.Hub, div]);
MathJax.Hub.Queue(function() {
popover.setAttribute('data-content', div.innerHTML);
document.body.removeChild(div);
})
}
}
</script>
</body>
</html>