Write external vocabulary directly to .vocabulary.external
#1253
Conversation
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #1253 +/- ##
==========================================
+ Coverage 87.52% 87.60% +0.08%
==========================================
Files 308 308
Lines 27909 27922 +13
Branches 3122 3124 +2
==========================================
+ Hits 24426 24461 +35
+ Misses 2346 2324 -22
Partials 1137 1137 ☔ View full report in Codecov by Sentry. |
| if (!_noIdMapsAndIgnoreExternalVocab) { | ||
| outfileExternal_ = | ||
| ad_utility::makeOfstream(basename + EXTERNAL_LITS_TEXT_FILE_NAME); | ||
| } | ||
|
|
||
| // Open and prepare all infiles and mmap output vectors. |
There was a problem hiding this comment.
Are the words "infiles" and "mmap output vectors" still appropriate?
src/index/VocabularyOnDisk.cpp
Outdated
| @@ -48,6 +48,34 @@ std::optional<string> VocabularyOnDisk::operator[](uint64_t idx) const { | |||
| return result; | |||
| } | |||
|
|
|||
| // _____________________________________________________________________________ | |||
| ad_utility::ConsumeratorImpl<std::string_view> | |||
| VocabularyOnDisk::getWordWriterImpl(std::string outFileName) { | |||
There was a problem hiding this comment.
The get is confusing, just wordWriterImpl would be better.
src/index/VocabularyOnDisk.cpp
Outdated
| while (co_await ad_utility::valueWasPushedTag) { | ||
| const auto& word = co_await ad_utility::nextValueTag; |
There was a problem hiding this comment.
Find better names for the tags. And do we really need two co_awaits here? Most natural would be something like for (const auto& word = co_await nextWordTag) { ... }.
src/index/VocabularyGenerator.h
Outdated
| // helper struct used in the priority queue for merging. | ||
| // represents tokens/words in a certain partial vocabulary |
There was a problem hiding this comment.
| // helper struct used in the priority queue for merging. | |
| // represents tokens/words in a certain partial vocabulary | |
| // Helper `struct` for a word from a partial vocabulary. |
hannahbast
changed the title
.vocabulary.external
|
So far, the code that merges the partial vocabularies (class
VocabularyMerger) first wrote the external vocabulary to a temporary file which was afterward copied to the final. vocabulary.externalfile. This wasted space during the index build and on HDD also cost considerable time. Now the result of the merge is written directly to. vocabulary.external.