-
Notifications
You must be signed in to change notification settings - Fork 59
/
index.html
269 lines (191 loc) · 11.5 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
<!DOCTYPE html>
<html lang="zh">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.17.1: http://docutils.sourceforge.net/" />
<title>VCED — VCED项目文档</title>
<script>
document.documentElement.dataset.mode = localStorage.getItem("mode") || "";
document.documentElement.dataset.theme = localStorage.getItem("theme") || "light"
</script>
<!-- Loaded before other Sphinx assets -->
<link href="_static/styles/theme.css?digest=92025949c220c2e29695" rel="stylesheet">
<link href="_static/styles/pydata-sphinx-theme.css?digest=92025949c220c2e29695" rel="stylesheet">
<link rel="stylesheet"
href="_static/vendor/fontawesome/5.13.0/css/all.min.css">
<link rel="preload" as="font" type="font/woff2" crossorigin
href="_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
<link rel="preload" as="font" type="font/woff2" crossorigin
href="_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
<link rel="stylesheet" type="text/css" href="_static/pygments.css" />
<link rel="stylesheet" type="text/css" href="_static/plot_directive.css" />
<!-- Pre-loaded scripts that we'll load fully later -->
<link rel="preload" as="script" href="_static/scripts/pydata-sphinx-theme.js?digest=92025949c220c2e29695">
<script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>
<script src="_static/jquery.js"></script>
<script src="_static/underscore.js"></script>
<script src="_static/_sphinx_javascript_frameworks_compat.js"></script>
<script src="_static/doctools.js"></script>
<link rel="index" title="索引" href="genindex.html" />
<link rel="search" title="搜索" href="search.html" />
<link rel="next" title="User Guide" href="user_guide/index.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="zh">
</head>
<body data-spy="scroll" data-target="#bd-toc-nav" data-offset="180" data-default-mode="">
<div class="bd-header-announcement container-fluid" id="banner">
</div>
<nav class="bd-header navbar navbar-light navbar-expand-lg bg-light fixed-top bd-navbar" id="navbar-main"><div class="bd-header__inner container-xl">
<div id="navbar-start">
<a class="navbar-brand logo" href="#">
<p class="title logo__title">VCED项目文档</p>
</a>
</div>
<button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbar-collapsible" aria-controls="navbar-collapsible" aria-expanded="false" aria-label="Toggle navigation">
<span class="fas fa-bars"></span>
</button>
<div id="navbar-collapsible" class="col-lg-9 collapse navbar-collapse">
<div id="navbar-center" class="mr-auto">
<div class="navbar-center-item">
<ul id="navbar-main-elements" class="navbar-nav">
<li class="toctree-l1 nav-item">
<a class="reference internal nav-link" href="user_guide/index.html">
User Guide
</a>
</li>
</ul>
</div>
</div>
<div id="navbar-end">
<div class="navbar-end-item">
<span id="theme-switch" class="btn btn-sm btn-outline-primary navbar-btn rounded-circle">
<a class="theme-switch" data-mode="light"><i class="fas fa-sun"></i></a>
<a class="theme-switch" data-mode="dark"><i class="far fa-moon"></i></a>
<a class="theme-switch" data-mode="auto"><i class="fas fa-adjust"></i></a>
</span>
</div>
<div class="navbar-end-item">
<ul id="navbar-icon-links" class="navbar-nav" aria-label="Icon Links">
<li class="nav-item">
<a class="nav-link" href="https://github.com/datawhalechina/vced" rel="noopener" target="_blank" title="GitHub"><span><i class="fab fa-github-square"></i></span>
<label class="sr-only">GitHub</label></a>
</li>
</ul>
</div>
</div>
</div>
</div>
</nav>
<div class="bd-container container-xl">
<div class="bd-container__inner row">
<!-- Only show if we have sidebars configured, else just a small margin -->
<div class="bd-sidebar-primary col-12 col-md-3 bd-sidebar">
<div class="sidebar-start-items"><form class="bd-search d-flex align-items-center" action="search.html" method="get">
<i class="icon fas fa-search"></i>
<input type="search" class="form-control" name="q" id="search-input" placeholder="Search the docs ..." aria-label="Search the docs ..." autocomplete="off" >
</form><nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
<div class="bd-toc-item active">
</div>
</nav>
</div>
<div class="sidebar-end-items">
</div>
</div>
<div class="bd-sidebar-secondary d-none d-xl-block col-xl-2 bd-toc">
<div class="toc-item">
<div class="tocsection onthispage mt-5 pt-1 pb-3">
<i class="fas fa-list"></i> On this page
</div>
<nav id="bd-toc-nav">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#id1">
VCED 是如何工作的
</a>
</li>
</ul>
</nav>
</div>
<div class="toc-item">
</div>
</div>
<div class="bd-content col-12 col-md-9 col-xl-7">
<article class="bd-article" role="main">
<section id="vced">
<h1>VCED<a class="headerlink" href="#vced" title="Permalink to this heading">#</a></h1>
<p>VCED: Video Clip Extraction by description</p>
<p>该项目可以通过文字描述来自动识别视频中相符合的片段进行视频剪辑，通过跨模态模型与向量检索，结合前后端分离的模式，帮助你快速接触新一代搜索技术。</p>
<section id="id1">
<h2>VCED 是如何工作的<a class="headerlink" href="#id1" title="Permalink to this heading">#</a></h2>
<p>VCED 项目的核心逻辑是通过多模态模型 CLIP 实现的，具体结构如下图所示：</p>
<p><img alt="CLIP" src="_images/CLIP.png" />
图片源自：《<a class="reference external" href="https://arxiv.org/pdf/2103.00020.pdf">Learning transferable visual models from natural language supervision</a>》</p>
<ul class="simple">
<li><p>首先将输入的文本通过 Text Encoder 进行 encoding，将文本转换为高维向量（embedding）。</p></li>
<li><p>其次将上传的视频通过 Image Encoder 进行 encoding，一般上传的视频会通过抽取关键帧的方式对内容进行提取，最终将上传的视频提取为多个图像，并对提起的关键帧转化为高维向量（embedding）。Text/Image Encoder 根据业务需要的不同可以选择不同的 AI 模型</p></li>
<li><p>跨模态模型的目的是将文本生成的向量数据与关键帧生成的向量数据映射到同一高维空间下，并使得文本描述与关键帧相符合的向量之间的距离尽可能的近</p></li>
<li><p>再将得到的高维向量进行存储，因为数据量可能较大，一般业务上会采用索引的方式来快速的对向量数据进行检索，常用的索引方法比如 PQ 算法，HNSW 算法等</p></li>
<li><p>最后输入一段文本后就可以用相同的方法将生成的向量与存储的向量进行比对，判断哪些向量之间的距离比较近，得到一个相似性分数，即可以找到离文本描述最相似的关键帧，再对原视频进行定位，最终找到相似的视频片段</p></li>
</ul>
<p>为了实现以上几个步骤，在本项目中引入了以下组件：</p>
<ul class="simple">
<li><p>为了解决如何对视频进行关键帧提取的问题，我们使用到了 ffmpeg，对应的实现方法在 <a class="reference external" href="https://github.com/datawhalechina/vced/blob/709de9a0a0ce6a0b534c243c5bb58e00a08c6379/code/service/videoLoader/video_loader.py">videoLoader</a> 中</p></li>
<li><p>为了解决如何将文本与图像转换为向量的问题，我们使用到了 jina 的 DocArray，对应处理文本的方法在 <a class="reference external" href="https://github.com/datawhalechina/vced/blob/709de9a0a0ce6a0b534c243c5bb58e00a08c6379/code/service/customClipText/clip_text.py">customClipText</a> 中，对应处理图像的方法在 <a class="reference external" href="https://github.com/datawhalechina/vced/blob/709de9a0a0ce6a0b534c243c5bb58e00a08c6379/code/service/customClipImage/clip_image.py">customClipImage</a> 中</p></li>
<li><p>为了解决如何将生成的向量数据存储以及检索的问题，我们使用到了 jina 自带的向量搜索方法，对应的实现方法在 <a class="reference external" href="https://github.com/datawhalechina/vced/blob/709de9a0a0ce6a0b534c243c5bb58e00a08c6379/code/service/customIndexer/executor.py">customIndexer</a> 中</p></li>
<li><p>为了解决前端显示的问题，我们使用到了 <a class="reference external" href="https://streamlit.io/">Streamlit</a> 框架实现前端效果，对应的实现方法在 <a class="reference external" href="https://github.com/datawhalechina/vced/blob/709de9a0a0ce6a0b534c243c5bb58e00a08c6379/code/web/app.py">web/app.py</a> 中</p></li>
</ul>
<p>在了解了本项目的逻辑后就让我们对这些组件进行进一步介绍，希望能够激发你更多的创造灵感😁</p>
<blockquote>
<div><p>本项目仅仅实现了一些基础的功能，还有许多可以完善的地方，下面就简单的列一些可以完善的点供你参考：<br/></p>
<ol class="arabic simple">
<li><p>目前向量检索使用的是最简单的暴力搜索，所以向量检索花费的时间很慢，这里可以优化</p></li>
<li><p>目前跨模态模型这里使用了比较大众的模型，文本与视频的匹配度有待提升</p></li>
<li><p>目前 VCED 项目仅能够处理对单个视频的检索，需要对项目改造来实现对多个视频片段的检索<br/></p></li>
</ol>
<p>还有许多地方可以完善，或者将这个项目扩充更多的功能，欢迎你发挥聪明才智作出更令人惊艳的作品🥰</p>
</div></blockquote>
<div class="toctree-wrapper compound">
<ul>
<li class="toctree-l1"><a class="reference internal" href="user_guide/index.html">User Guide</a><ul>
<li class="toctree-l2"><a class="reference internal" href="user_guide/jina.html">Jina</a></li>
<li class="toctree-l2"><a class="reference internal" href="user_guide/Frontend/index.html">项目前端介绍</a></li>
<li class="toctree-l2"><a class="reference internal" href="user_guide/Backend/index.html">项目后端简介</a></li>
</ul>
</li>
</ul>
</div>
</section>
</section>
</article>
<footer class="bd-footer-article">
<!-- Previous / next buttons -->
<div class='prev-next-area'>
<a class='right-next' id="next-link" href="user_guide/index.html" title="下一页 页">
<div class="prev-next-info">
<p class="prev-next-subtitle">下一页</p>
<p class="prev-next-title">User Guide</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
</div>
</footer>
</div>
</div>
</div>
<!-- Scripts loaded after <body> so the DOM is not blocked -->
<script src="_static/scripts/pydata-sphinx-theme.js?digest=92025949c220c2e29695"></script>
<footer class="bd-footer"><div class="bd-footer__inner container">
<div class="footer-item">
<p class="copyright">
© Copyright 2022, VCED.<br>
</p>
</div>
<div class="footer-item">
<p class="sphinx-version">
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 5.0.2.<br>
</p>
</div>
</div>
</footer>
</body>
</html>