Flekschas/faster beddb #135
Conversation
Genome wide this can lead to 20x faster queries but it increases the file size by a factor of 2.5
| tile_id = "{}.{}".format(curr_zoom, curr_tile) | ||
|
|
||
| tile_counts[tile_id] += 1 | ||
|
|
There was a problem hiding this comment.
Couldn't you put the tile_inserts.append() call from below here and save yourself a while loop?
There was a problem hiding this comment.
I don't think this works. The reason why is the following:
tile_counts only counts the first time an interval is inserted but not the subsequent times. Hence, this while loop is only called once, while the one below is called in higher zoom levels as well.
Here's a short example: say we set the max number of intervals per tile to 2. The first interval will be inserted at tile 0.0 and its counter will increase. However, we want to make sure that the most important tile is always shown in tile with a higher zoom level so we will insert tile-id<>interval-id pairs for higher zoom levels as well. However, we will not increase the counter for those tiles as otherwise we would never see more details as we zoom in in some cases. Does this somehow make sense? :)
This replicates the existing behavior of the beddb format.
Description
What was changed in this pull request?
I implemented a tile-based indexing strategy for beddb which can speed up queries by up to 20x at the expense of increasing the file size by a factor of ~2.5x
To avoid adding the burden of having to handle another format to the end-user I decided to mark this indexing using an appended
tto the version number. I.e., version3is the normal version while3tis the tile-index versionTo create a tile-indexed beddb file use
clodius aggregate bedfilewith--tile-index.Why is it necessary?
The range-based rtree indexing is getting slow with >5mio intervals (i.e.,
0.5s for a query) while the tile-based index remains fast with ~0.025s.Checklist
black .