Skip to content

QueryPlan Optimizer aims to run TPCH 100x faster #116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 392 commits into
base: main-23.10-20240617
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
392 commits
Select commit Hold shift + click to select a range
2d9bf54
Gen manifest
auxten Apr 8, 2023
de3b3c8
Upload pypi only on v tag
auxten Apr 8, 2023
0ecfbb8
Make mac-arm64 for apple silicon(M1, M2)
auxten Apr 9, 2023
10a1b46
Added Dataframe output format (#6)
nmreadelf Apr 16, 2023
31955b7
Add to pandas example (#5)
auxten Apr 16, 2023
ce86a4f
Add arch chart
auxten Apr 16, 2023
4e24ff5
Fix readme
auxten Apr 16, 2023
0bd7a38
AWS Lambda compatibility (#9)
lmangani Apr 16, 2023
09806ed
Fix build_mac_arm64.sh (#10)
auxten Apr 16, 2023
33b45fa
Fix version in __init__.py
auxten Apr 17, 2023
2a27324
Fix replace file content bug
auxten Apr 17, 2023
c20d63e
Try import pyarrow and pandas when necessary
auxten Apr 17, 2023
cc29615
Build on pr based on pybind and push tags
auxten Apr 17, 2023
6b51b49
Sleep 60 then kill clang if timeout
auxten Apr 17, 2023
26b1840
Update brew to avoid llvm@15 minor version unstable
auxten Apr 18, 2023
7017543
Add serverless demo and discord
auxten Apr 19, 2023
9e1b570
Add benchmark in readme
auxten Apr 20, 2023
f10cd77
Enable S3
lmangani Apr 19, 2023
d5aa561
Brew update before install
auxten Apr 21, 2023
aed2a8e
Remove timestamp for ccache files
auxten Apr 21, 2023
5ee37b1
Update build_wheels.yml
auxten Apr 21, 2023
096d460
Brew update before install
auxten Apr 21, 2023
ec4f64a
Update build_wheels.yml
auxten Apr 21, 2023
4f8d976
Fix ArrowTable output via '-m chdb sql arrowtable'
auxten Apr 21, 2023
bc2442e
Fix up tests
auxten Apr 21, 2023
fb778fa
Test 60 types of output and run tests in CI
auxten Apr 22, 2023
bcef843
Run tests on linux build
auxten Apr 22, 2023
d1c34c6
Fix dup run tests
auxten Apr 22, 2023
9cb6e5f
Use assertAlmostEqual for test output
auxten Apr 22, 2023
9a52198
Return non zero if any test fails
auxten Apr 22, 2023
b390552
Test ArrowTable, because Arrow binary is not stable cross plat
auxten Apr 22, 2023
54828ae
Ignore tz issue in test
auxten Apr 22, 2023
1f97ca8
Install pyarrow pandas before make test
auxten Apr 22, 2023
1ba70f2
Fix CIBW_TEST
auxten Apr 22, 2023
5660d01
Add branch name in ccache key to keep default branch a cache
auxten Apr 23, 2023
9b3731d
Trigger action when push to pybind
auxten Apr 23, 2023
f055673
Remove ref name in cache
auxten Apr 23, 2023
ad8dd28
Fix parallel test and test more in mac arm64 build (#21)
auxten Apr 25, 2023
6681a9d
Add function: chdb() to return version (#25)
lmangani Apr 28, 2023
ec2d84a
Add server example (#27)
lmangani Apr 28, 2023
54ebe8e
Add format example in readme code (#24)
Apr 28, 2023
e46cb05
Chinese readme (#29)
auxten Apr 28, 2023
e3eca8b
Test enable jemalloc in linux x86_64 build
auxten Apr 24, 2023
49e0c4e
Try macos-12-xl
auxten Apr 29, 2023
3b8e080
Use auxten/jemalloc for mallctl("arenas.lookup", NULL, NULL, &ptr, si…
auxten May 3, 2023
83a90c5
Add mallocAdapt with mem alloc wrapper and libc mem apis
auxten May 3, 2023
ac0c480
Add __real_free stub to make clickhouse-local compile succ
auxten May 3, 2023
26ddbf2
Move global inside_main to LocalChdb.cpp
auxten May 3, 2023
84f393c
Use je_xx funcs when USE_JEMALLOC defined
auxten May 3, 2023
221767c
Remove jemalloc_rename.h
auxten May 3, 2023
eacc3d1
Use je_ prefix for jemalloc
auxten May 3, 2023
3bbb390
Inject jemalloc symbols for necessary libs
auxten May 3, 2023
6c1eb81
Use ordinary macos-12
auxten May 3, 2023
9ae5d5a
Fix build.sh for linux to enable jemalloc
auxten May 3, 2023
ca18aca
Fix #if defined(USE_JEMALLOC) && (USE_JEMALLOC)
auxten May 3, 2023
758c46a
Fix typo
auxten May 3, 2023
d008c08
Update auxten/jemalloc fix if ptr present check alloc_ctx.edata == NULL
auxten May 3, 2023
6e3951d
Add timestamp for ccache to fix "another job may be creating this cache"
auxten May 4, 2023
881c67a
Fix output buffer flush caused memory realloc x2
auxten May 10, 2023
9c24a8a
Add test for issue #31
auxten May 10, 2023
907c3cd
Add test for issue #31
auxten May 11, 2023
e4400e1
Limit output of test
auxten May 11, 2023
65762f6
Make select result stable
auxten May 11, 2023
338366f
Add db api 2.0 driver
May 18, 2023
e42828e
Fix dbapi feature packaging
auxten May 18, 2023
3939927
Fix minor bug
auxten May 24, 2023
181e9b6
Remove arrow in dbapi
May 24, 2023
6777546
Readme
auxten May 25, 2023
6330f85
Action: ignore readme, examples, workflows
lmangani May 26, 2023
e6524cc
Use total downloads
auxten Jun 1, 2023
f264b3c
Add ArrowStream Arrow format
auxten May 31, 2023
31b7fae
Add Table and run chdb on Table
auxten Jun 1, 2023
6e285a5
Test to_pandas()
auxten Jun 1, 2023
bbaef18
Add hits_0.parquet download
auxten Jun 1, 2023
bf7d72a
Arrow format output is not deterministic, check the first 100 bytes
auxten Jun 1, 2023
11140d8
Ok, 10 bytes
auxten Jun 1, 2023
49d75ba
Not check arrow
auxten Jun 1, 2023
c83ce39
Drop Python 3.7 support
auxten Jun 1, 2023
58de865
Warn if pandas version lower then 2.0.0
auxten Jun 2, 2023
86b17bf
Use pandas_read_parquet
auxten Jun 2, 2023
b7a77e4
Add flush_to_disk for Table
auxten Jun 2, 2023
c1d9706
Fix fd leak in tests
auxten Jun 2, 2023
cd6453b
Update usage in readme
auxten Jun 4, 2023
521580e
Fix typo
auxten Jun 4, 2023
b6605bc
Improve format of data options
lmangani Jun 4, 2023
96fdaec
Improve data input style README-zh.md
lmangani Jun 4, 2023
cf33f3d
Update README.md
auxten Jun 5, 2023
4e45d64
Update README-zh.md
auxten Jun 5, 2023
22c199b
Use apache 2.0 license
auxten Jun 7, 2023
e007aa3
Let pandas choose backend itself
auxten Jun 8, 2023
ffda9b0
Check memoryview size before using
auxten Jun 12, 2023
a6fb676
Check format is supported before the engine runs too far
auxten Jun 15, 2023
80e590c
Fix logo blur
auxten Jun 17, 2023
01afdae
Add Colab Examples
lmangani Jun 17, 2023
b50b979
Restart Failed Actions
lmangani Jun 15, 2023
fd92cb3
Require secret.PAT_TOKEN
lmangani Jun 16, 2023
fb1a868
Update build_wheels.yml
lmangani Jun 16, 2023
fa95884
Update build_wheels.yml
lmangani Jun 16, 2023
e9c7a36
Update build_wheels.yml
lmangani Jun 16, 2023
e9b8fa2
Fix nullptr caused core dump
auxten Jun 26, 2023
fb290da
Change default branch to main
auxten Jun 28, 2023
7d00bfb
Fix discord link
lmangani Jun 27, 2023
2d88b15
Check self.use_memfd
auxten Jun 30, 2023
e9947e9
Fix missing endif()
auxten Jun 30, 2023
56bd2c1
Update format output for parquet Bool support
auxten Jun 30, 2023
f1ff5e1
Add contributors section
auxten Jun 30, 2023
950d07f
docs: update README.md [skip ci]
allcontributors[bot] Jun 30, 2023
95f12e2
docs: create .all-contributorsrc [skip ci]
allcontributors[bot] Jun 30, 2023
423e87c
docs: update README.md [skip ci]
allcontributors[bot] Jun 30, 2023
b3f8cad
docs: update .all-contributorsrc [skip ci]
allcontributors[bot] Jun 30, 2023
44412b5
docs: update README.md [skip ci]
allcontributors[bot] Jun 30, 2023
d616b00
docs: update .all-contributorsrc [skip ci]
allcontributors[bot] Jun 30, 2023
01d66dd
docs: update README.md [skip ci]
allcontributors[bot] Jun 30, 2023
4d23631
docs: update .all-contributorsrc [skip ci]
allcontributors[bot] Jun 30, 2023
106ca4d
Update README.md
auxten Jun 30, 2023
2f03945
Format README.md
lmangani Jun 30, 2023
76cbbd0
Update README.md
auxten Jul 3, 2023
455c7a8
Enable brotli
auxten Jul 3, 2023
1bee789
Add necessary alltypes_dictionary.parquet
auxten Jul 3, 2023
5da0365
Fix test data path
auxten Jul 3, 2023
35f8398
Fix data path in example
auxten Jul 3, 2023
be01771
Do nothing on StaticThreadPool re-initialize
auxten Jul 3, 2023
acb2f19
Use je_mallctl in zookeeper
auxten Jul 3, 2023
89aabe4
Cleanup unnecessary docs
auxten Jul 3, 2023
7e5aff2
Refactor LocalChdb:
auxten Jul 7, 2023
ff803dd
Use .byte() not .get_memview().tobytes()
auxten Jul 11, 2023
3d0e2d9
Make free_result not delete result pointer
auxten Jul 11, 2023
f137241
Fix tests
auxten Jul 11, 2023
562da7b
Add pandas dataframe example
auxten Jul 11, 2023
4ed535d
Update readme
auxten Jul 11, 2023
2e4d9c1
Fix all get_memview.tobytes() to just bytes(), and len(res) directly
auxten Jul 11, 2023
fbe30cc
Do not build on changes to the tests/ folder
lmangani Jul 12, 2023
1696416
Optimize test_parallel
auxten Jul 14, 2023
933fd72
Fix tls issue and use PIC and PIE
auxten Jul 14, 2023
2b3535a
Disable thin-lto for now
auxten Jul 14, 2023
099d2f3
Use -flto=thin in release build
auxten Jul 15, 2023
e903137
Max sleep time to 120s for easy debug
auxten Jul 15, 2023
6ccc182
Empty printHelp
auxten Jul 15, 2023
297e7b8
Init NamedCollections after setup users
auxten Jul 17, 2023
a97777d
Add test on issue 60
auxten Jul 17, 2023
cc860d4
Fix test issue 60 url
auxten Jul 17, 2023
db43d5a
Update CONTRIBUTING.md
auxten Jul 18, 2023
10aaaeb
delay before restarting failed actions
lmangani Jul 18, 2023
cb00d85
Add files via upload
reema93jain Jul 18, 2023
88cb561
docs: update README.md [skip ci]
allcontributors[bot] Jul 19, 2023
46798a2
docs: update .all-contributorsrc [skip ci]
allcontributors[bot] Jul 19, 2023
14e4787
Update README.md
auxten Jul 19, 2023
eefdbf1
Add query_stateful low level func
auxten Jul 17, 2023
db2b5ed
Add UUID generate_from_pid()
auxten Jul 20, 2023
744952c
Add SharedPtrContextHolder(todo: fix leak shared)
auxten Jul 20, 2023
d7fc3b6
Use pid as Session id
auxten Jul 20, 2023
2ab2829
Add ignore
auxten Jul 20, 2023
83def39
Use createSharedHolder to create shared_context in Client
auxten Jul 20, 2023
bc34f59
Free query_result_memory vector only in `free_result`
auxten Jul 20, 2023
955b8bb
Set background_schedule_pool_size 8
auxten Jul 21, 2023
bb789c2
No exception if table uuid exist for stateful query
auxten Jul 21, 2023
506c3bd
Init query_result_memory to nullptr on construct
auxten Jul 21, 2023
c85b1f7
Set max_thread_pool_free_size = 0 and use a leaky shared_context for now
auxten Jul 21, 2023
436d6be
Add chdb-todo: fix SharedPtrContextHolder.shared leak
auxten Jul 21, 2023
e7c25ea
Test for gc result on empty output
auxten Jul 21, 2023
8ff246d
Add test_final_join
auxten Jul 22, 2023
1597f98
Disable BackgroundCleanup
auxten Jul 22, 2023
db3db45
Set background_schedule_pool_size 0
auxten Jul 22, 2023
8272d54
Remove stateful tests for now
auxten Jul 22, 2023
793dfc6
Update readme
auxten Jul 23, 2023
0b7f93c
New arch chart
auxten Jul 23, 2023
a498a3f
Fix 0.11 issues2 (#67)
auxten Jul 23, 2023
f2f89e8
Created using Colaboratory
auxten Jul 24, 2023
42834e0
Created using Colaboratory
auxten Jul 24, 2023
10ec2da
Remove dup
auxten Jul 24, 2023
00b5d83
Update chDB_demos.ipynb
auxten Jul 24, 2023
b32da97
Fix arrow jemalloc (#68)
auxten Jul 24, 2023
fdbc30c
Enable jemalloc only Linux
auxten Jul 24, 2023
1d2502f
Add memory_pool_jemalloc.cc only Linux
auxten Jul 24, 2023
247fa54
Fix typo
auxten Jul 24, 2023
83288fd
Add Stateful Session Example
lmangani Jul 24, 2023
bbe7029
Update README.md
lmangani Jul 24, 2023
f55c357
Add session example in readme-ch
auxten Jul 25, 2023
01e326e
Try fix jemalloc on aarch64 (#69)
auxten Jul 26, 2023
02a7c1f
add chDB-GPT Example
lmangani Jul 26, 2023
7201343
Add some logo
auxten Jul 27, 2023
e8d008c
Only enable AVX and JIT on x86_64
auxten Jul 27, 2023
f997702
Add Demo chDB at [ClickHouse v23.7 livehouse!]
auxten Jul 28, 2023
ef16a69
Fix data dir name in ChDB_TPCH_SF10_Queries
auxten Jul 31, 2023
edad462
Broader arm64 compatibility (#77)
lmangani Jul 31, 2023
64e8190
Fix blog link
auxten Aug 3, 2023
ca09448
Plain ARM64 builder
lmangani Aug 3, 2023
08d1347
Delete build_wheels_arm64.yml
lmangani Aug 4, 2023
ed55a7b
Add arm64 linux build (#70)
nmreadelf Aug 6, 2023
8d5109e
Allow module publishing on TAG_NAME versions
lmangani Aug 6, 2023
2013531
Update README.md
lmangani Aug 6, 2023
3c66204
Fix twine upload path
lmangani Aug 6, 2023
8ba8780
Remove test action
lmangani Aug 7, 2023
d81480c
Populate global UDF settings ClickHouse PR #52643 (#81)
lmangani Aug 9, 2023
39a5e4c
Fix stateful mergetree (#83)
auxten Aug 9, 2023
cb19e31
Create FUNDING.yml
auxten Aug 13, 2023
9bd6129
Update FUNDING.yml
auxten Aug 13, 2023
c909e16
Update README
lmangani Aug 13, 2023
e503e5b
Update README.md
lmangani Aug 13, 2023
c330249
Query on multiple Pandas DataFrame (#89)
auxten Aug 17, 2023
7dec57c
Query stats (#91)
auxten Aug 17, 2023
6d51d42
Fix null
auxten Aug 17, 2023
f10a172
Add statistics funcs for Table
auxten Aug 18, 2023
2f49baa
Add CHDB_VERION for cmake (#96)
nmreadelf Aug 23, 2023
9a52781
enable output_format_arrow_string_as_string (#98)
nmreadelf Aug 27, 2023
fddb32d
Use new logo
auxten Aug 27, 2023
7054c59
Strip library (#99)
lmangani Aug 27, 2023
acdde20
Use snake logo
auxten Aug 27, 2023
d2929cc
Use white bg logo
auxten Aug 31, 2023
8b97106
Add dark theme logo
auxten Aug 31, 2023
f752d92
Fix dark theme
auxten Aug 31, 2023
5352706
All center
auxten Aug 31, 2023
4061f73
chdb + urleng session persistence example
lmangani Sep 1, 2023
c227ac8
Generate xml and py files for UDF
auxten Aug 27, 2023
2fad23d
Unified chdb.query with udf_path support
auxten Aug 27, 2023
4c24a1f
Use query instead of query_stateful
auxten Aug 27, 2023
88c98c1
Use output_format as cpp func 2nd arg
auxten Sep 3, 2023
3bbfb13
Fix up query default args value
auxten Sep 3, 2023
eeaa37c
Use temp dir for UDF config and executables
auxten Sep 3, 2023
580f87b
Support `with Session() as sess:` in chdb session
auxten Sep 3, 2023
0dfdd61
Add tests for UDF and with session
auxten Sep 3, 2023
afc3faa
Add Query with UDF
auxten Sep 3, 2023
409350e
Merge pull request #100 from chdb-io/binUdf
auxten Sep 4, 2023
1cdc4c1
Create udf.py
lmangani Sep 4, 2023
6bff252
Update README.md
lmangani Sep 4, 2023
0b0e412
Fix case of arrow
auxten Sep 4, 2023
1ceb146
Created using Colaboratory
auxten Sep 5, 2023
92024bd
No strip, which will casue wired things
auxten Sep 5, 2023
7905986
Add diff dir script
auxten Sep 9, 2023
fd431b8
Add ByConity original optimizer stuff
auxten Sep 10, 2023
571b943
Adapt optimizer patch no.1
auxten Sep 10, 2023
a0b8cfd
Fix src/Analyzers/QueryRewriter.cpp
auxten Sep 12, 2023
fdef58f
Diff dir in python
auxten Sep 14, 2023
974c377
Better utils/difftree.py
auxten Sep 15, 2023
3b5702b
Merging src/QueryPlan and src/Processors/QueryPlan
auxten Sep 15, 2023
55bfe40
Remove useless proto
auxten Sep 15, 2023
4c715c0
Rename src/QueryPlan
auxten Sep 17, 2023
5fc5c26
Ignore tmp bc stuff
auxten Sep 17, 2023
85be69d
Replace QueryPlan with Processors/QueryPlan
auxten Sep 17, 2023
d53b531
Update QueryPipeline to QueryPipelineBuilder
auxten Sep 17, 2023
35196b0
Implement getType, setInputStreams, copy of IQueryPlanStep for QueryP…
auxten Sep 21, 2023
07c5f07
Add elapsedMillisecondsAsDouble
auxten Sep 21, 2023
f7d92fd
Adapt optimizer patch no.2:
auxten Sep 27, 2023
62fba75
Adapt optimizer patch no.3:
auxten Oct 3, 2023
9d4dcd8
Ignore join_using_null_safe for now
auxten Oct 4, 2023
cbcce58
Replace all ASOF::Inequality with ASOFJoinInequality
auxten Oct 4, 2023
00040ad
Add unfinished NestedLoopJoin
auxten Oct 7, 2023
f2d75e2
Adapt optimizer patch no.4:
auxten Oct 7, 2023
558e748
Adapt optimizer patch no.5:
auxten Oct 23, 2023
10f38b9
Cleanup to rebase merge into previous commit
auxten Oct 23, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
The diff you're trying to view is too large. We only load the first 3000 changed files.
62 changes: 62 additions & 0 deletions .all-contributorsrc
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
{
"files": [
"README.md"
],
"imageSize": 100,
"commit": false,
"commitType": "docs",
"commitConvention": "angular",
"contributors": [
{
"login": "auxten",
"name": "auxten",
"avatar_url": "https://avatars.githubusercontent.com/u/240147?v=4",
"profile": "http://auxten.com",
"contributions": [
"code"
]
},
{
"login": "lmangani",
"name": "Lorenzo Mangani",
"avatar_url": "https://avatars.githubusercontent.com/u/1423657?v=4",
"profile": "https://metrico.in",
"contributions": [
"code"
]
},
{
"login": "laodouya",
"name": "laodouya",
"avatar_url": "https://avatars.githubusercontent.com/u/4847103?v=4",
"profile": "https://github.com/laodouya",
"contributions": [
"code"
]
},
{
"login": "nmreadelf",
"name": "nmreadelf",
"avatar_url": "https://avatars.githubusercontent.com/u/7260482?v=4",
"profile": "https://github.com/nmreadelf",
"contributions": [
"code"
]
},
{
"login": "reema93jain",
"name": "reema93jain",
"avatar_url": "https://avatars.githubusercontent.com/u/113460610?v=4",
"profile": "https://github.com/reema93jain",
"contributions": [
"test"
]
}
],
"contributorsPerLine": 7,
"skipCi": true,
"repoType": "github",
"repoHost": "https://github.com",
"projectName": "chdb",
"projectOwner": "chdb-io"
}
28 changes: 28 additions & 0 deletions .coveragerc
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# .coveragerc to control coverage.py
[run]
branch = True
source = chdb
# omit = bad_file.py

[paths]
source =
src/
*/site-packages/

[report]
# Regexes for lines to exclude from consideration
exclude_lines =
# Have to re-enable the standard pragma
pragma: no cover

# Don't complain about missing debug-only code:
def __repr__
if self\.debug

# Don't complain if tests don't hit defensive assertion code:
raise AssertionError
raise NotImplementedError

# Don't complain if non-runnable code isn't run:
if 0:
if __name__ == .__main__.:
1 change: 0 additions & 1 deletion .github/CODEOWNERS

This file was deleted.

13 changes: 13 additions & 0 deletions .github/FUNDING.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# These are supported funding model platforms

github: [chdb-io]
patreon: # Replace with a single Patreon username
open_collective: chdb
ko_fi: # Replace with a single Ko-fi username
tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
liberapay: # Replace with a single Liberapay username
issuehunt: # Replace with a single IssueHunt username
otechie: # Replace with a single Otechie username
lfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry
custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2']
9 changes: 0 additions & 9 deletions .github/actionlint.yml

This file was deleted.

Loading