-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathevaluation.html
198 lines (186 loc) · 9.53 KB
/
evaluation.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="description" content="">
<meta name="author" content="">
<title> ICDAR 2019 cTDaR | Evaluation</title>
<link href="css/bootstrap.min.css" rel="stylesheet">
<link href="css/font-awesome.min.css" rel="stylesheet">
<link href="css/animate.min.css" rel="stylesheet">
<link href="css/lightbox.css" rel="stylesheet">
<link href="css/main.css" rel="stylesheet">
<link href="css/responsive.css" rel="stylesheet">
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.4/MathJax.js?config=TeX-MML-AM_CHTML">
</script>
<!--[if lt IE 9]>
<script src="js/html5shiv.js"></script>
<script src="js/respond.min.js"></script>
<![endif]-->
<link rel="shortcut icon" href="images/ico/favicon.ico">
<link rel="apple-touch-icon-precomposed" sizes="144x144" href="images/ico/apple-touch-icon-144-precomposed.png">
<link rel="apple-touch-icon-precomposed" sizes="114x114" href="images/ico/apple-touch-icon-114-precomposed.png">
<link rel="apple-touch-icon-precomposed" sizes="72x72" href="images/ico/apple-touch-icon-72-precomposed.png">
<link rel="apple-touch-icon-precomposed" href="images/ico/apple-touch-icon-57-precomposed.png">
</head><!--/head-->
<body>
<header id="header">
<div class="container">
<div class="row">
<div class="col-sm-12 overflow">
</div>
</div>
</div>
<div class="navbar navbar-inverse" role="banner">
<div class="container">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse">
<!-- <span class="sr-only">Toggle navigation</span> -->
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="index.html">
<h1><img src="images/logo.png" alt="logo" style="width: 45%; height: 40%; max-width: 500px; padding-left: 45px" ></h1>
</a>
</div>
<div class="collapse navbar-collapse">
<ul class="nav navbar-nav navbar-right">
<li><a href="index.html">Home</a></li>
<li class="dropdown"><a href="#">Tasks<i class="fa fa-angle-down"></i></a>
<ul role="menu" class="sub-menu">
<li><a href="tasks.html">Tasks</a></li>
<li><a href="results.html">Results</a></li>
</ul>
</li>
<li class="dropdown"><a href="#">Dataset<i class="fa fa-angle-down"></i></a>
<ul role="menu" class="sub-menu">
<li><a href="dataset-description.html">Description</a></li>
<li><a href="dataset-training.html">Training Dataset</a></li>
<li><a href="dataset-testing.html">Test Dataset</a></li>
</ul>
</li>
<li class="active"><a href="evaluation.html">Evaluation</a></li>
<li><a href="organizers.html">Organizers</a></li>
<li><a href="faq.html">FAQ</a></li>
</ul>
</div>
<!-- <div class="search">
<form role="form">
<i class="fa fa-search"></i>
<div class="field-toggle">
<input type="text" class="search-form" autocomplete="off" placeholder="Search">
</div>
</form>
</div> -->
</div>
</div>
</header>
<!--/#header-->
<section id="page-breadcrumb">
<div class="vertical-center sun">
<div class="container">
<div class="row">
<div class="action">
<div class="col-sm-12">
<h1 class="title">Evaluation</h1>
</div>
</div>
</div>
</div>
</div>
</section>
<!--/#page-breadcrumb-->
<section id="features">
<div class="container">
<div class="row">
<div class="single-features">
<div class="col-sm-10 col-sm-offset-1 wow fadeInRight" data-wow-duration="500ms" data-wow-delay="300ms">
<p style="font-weight: bold;">The evaluation tool can be downloaded <a href="https://github.com/cndplab-founder/ctdar_measurement_tool" target="_blank">here.</a></p>
<br>
<p>
The ICDAR 2019 cTDaR evaluates two aspects of table analysis: table detection and recognition. We choose the metric (i) to evaluate the performance of table region detection, and apply the metric (ii) to evaluate that of table recognition. Based on these measures, an overall performance of various algorithms can be compared with each other.
</p>
<br>
<ol type="i">
<li>
<p>
Metric for table region detection task
</p>
<p align="left">
IoU is calculated to tell if a table region is correctly detected. It's used to measure the overlapping of the detected polygons:
</p>
<p>$$IoU=\frac{area(GTP\bigcap DTP)}{area(GTP\bigcup DTP)}$$</p>
<p>
where <i>GTP</i> defines the Ground Truth Polygon of the table region and <i>DTP</i> defines the Detected Table Polygon. IoU has a range from 0 to 1, where 1 suggests the best possible segmentation. When evaluating, different threshold values of IoU will be used to determine if a region is considered as being detected correctly.
Then, the precision and recall values are computed from a method’s ranked output. Recall is defined as the proportion of all true positive examples ranked above a given rank. Precision is the proportion of all examples above that rank which are from the positive class. Furthermore, F1 score will be computed as the harmonic average of recall and precision value. Precision, recall and F1 scores are calculated with IoU threshold of 0.6, 0.7, 0.8 and 0.9 respectively.
</p>
</li>
<br>
<li>
<p>
Metric for table recognition task
</p>
<p>
This track is evaluated by comparing the structure of a table that is defined as a matrix of cells. For each cell, participants are required to return the coordinates of a polygon defining the cell (historical documents) or a polygon defining the convex hull of the cell's contents (modern documents). Additionally, participants must provide the start/end column/row information for each cell.We propose the following metric: Cell’s adjacency relation-based table structure evaluation (inspired by Gobel’s method [2]).
</p>
<ul>
<li>
<p>
For comparing two cell structures, we use the method: for each table region, we align each groundtruth cell to the predicted cell with IoU > σ, identify the valid predicted cells, and then generate a list of adjacency relations between each valid cell and its nearest neighbor in horizontal and vertical directions. Blank cells are not represented in the grid. No adjacency relations are generated between blank cells or a blank cell and a content cell. This 1-D list of adjacency relations can be compared to the groundtruth by using precision and recall measures. If both cells are identical and the direction matches, then it is marked as correctly retrieved; otherwise it is marked as incorrect.
</p>
<p>
The precision, recall and F1 score will be calculated under circumstances that IoU is equal to 0.6, 0.7, 0.8 and 0.9 as the evaluation for track A.
</p>
<!-- <div class="img-container" >
<img src="./images/data/3.jpg" style="width: 60%; height: 50%; max-width: 600px" />
<p>Figure 2: Comparison of an incorrectly detected cell structure with the groundtruth [2]</p>
</div> -->
</li>
</ul>
</li>
</ol>
<br>
<p>
the final ranks of teams are decided by the weighted average F1 (WAvg. F1) value of the whole dataset for each track. The WAvg. F1 value is defined as:
</p>
<!-- <p>$$IoU=\frac{area(GTP\bigcap DTP)}{area(GTP\bigcup DTP)}$$</p> -->
<p>$$ WAvg. F1 = \frac{\sum\limits_{i=1}^4 IoU_i \cdot F1@IoU_i}{\sum\limits_{i=1}^4 IoU_i}$$</p>
<p>
which shows that the weight of each F1 value is the corresponding IoU threshold. We think results with higher IoUs are more important than those with lower IoUs, so we use IoU threshold as the weight of each F1 value to get a definitive performance score for convenient comparison.
</p>
<br>
<p>
We will also release a number of tools to enable the participants to automatically compare their result to the groundtruth.
</p>
<br>
<h3>References</h3>
<p id="references2">[1] L. Gao, X. Yi, Z. Jiang, L. Hao and Z. Tang, “ICDAR 2017 POD Competition,” in ICDAR, 2017, pp. 1417-1422.</p>
<p id="references3">[2] M. C. Gobel, T. Hassan, E. Oro, G. Orsi, ”ICDAR2013 Table Competition,” in Proc. of the 12th ICDAR (IEEE, 2013), pp. 1449-1453.</p>
</div>
</div>
</div>
</section>
<!--/#features-->
<footer id="footer">
<div class="container">
<div class="row">
<div class="col-sm-12">
<div class="copyright-text text-center">
<p>© PKU Founder Group 2019. All Rights Reserved.</p>
<p>Designed by <a target="_blank" href="http://www.themeum.com">Themeum</a></p>
</div>
</div>
</div>
</div>
</footer>
<!--/#footer-->
<script type="text/javascript" src="js/jquery.js"></script>
<script type="text/javascript" src="js/bootstrap.min.js"></script>
<script type="text/javascript" src="js/lightbox.min.js"></script>
<script type="text/javascript" src="js/wow.min.js"></script>
<script type="text/javascript" src="js/main.js"></script>
</body>
</html>