From f69fb236cee909ddfdbbc7bdc531231660c4f269 Mon Sep 17 00:00:00 2001
From: Tom White
map_direct
reduction
¶Implemented using multiple rounds of calls to blockwise
and rechunk
.
reduction
¶Implemented using multiple rounds of a tree reduce operation followed by a final aggregation.
Cubed will optimize the graph before computing it - by fusing blockwise (map) operations.
+Cubed will automatically optimize the graph before computing it. For example by fusing blockwise (map) operations:
- - + +
-This is a simple case - still lots of optimizations left to do.
+In early 2024 we implemented more optimizations to give a 4.8x performance improvement on the "Quadratic Means" climate workload running on Lithops with AWS Lambda, with a 1.5 TB workload completing in around 100 seconds
+ +More details in Optimizing Cubed
groupby
operations
AkMWra_tyTTaA^F5(#gv&Y x??|Yw)3{Kks 65U!hQ?Vom)!#;!I-N|4o{vu>1}V1R8mw7+>5@escFBtxf$QTENo?EWe5xDMsK}6
z+oHimeuIgLDJ>;M;lul)!mjpmMrnURCnqOLa`OEHj8D#EH{i`jldq{~sIZ@u#s~f#
zx>k^v*W&ErVq$4IZKSDgYDz17^QQgRuV2@v8v@>jYTesg`k|q#n|yq768xko__cNL
z@gFJ<9-i++OfqLvwBjr*VKlU~mH##-KAoMNm2prqFfcgS*tFab7Vch?O^J=gDJUrD
zj3%Jjt;ng&;m=5*6T6|Rs!GYocrisME-9IXtj59C%;nScd?h6%WTi}ujmwjhNq7I5
zvWkkPUFYVm9Q))r(qPSR)!56<&OVyPXBJG2E*o_6qNu2-;_uJGkilglnj4XSS66kr
zb+NN$f?ncA$$QT>hkYf}P*TDa77@8;*OhoaM?g Gz7hc5ioVcy5xO^p8b3R}L8|tT31OXInhc|Dj
zdA=3=j$H8GI~za(N6D(pgwLak7O+O{JbM;FY8^fi6`9rt3H};0AB$4c&xT&6rdoL9
z$ZQ!TPDGa0&-f27QdDUio&oDCUv9l-{>3r`GsuCFphhW5+Sb%M`1OwCtR?AsJ5@}a){ap>BS08?`c3qH}Q5fXvlP-v~
zOWrxuI+o(P=AX5$7s*CJ$8h$kP3AXIR`FxChuKMaM_vgF3$vmwfCMDfgxsQ*VB@II
ze_#)_;xqt}DofS{YNFPtr3W1n=ggk{=nHLUMR-oq4J_Q#S+B#U{SYzM|A-hjP7H
wbMQA*-wAyVI2@erLNK--1y{KCsy1Lv3ws
zLN3dYqM{-MnVFd}Gc$iHe34uH9Y#%!D=#l^V}EyUzdPT^OGrqlIhjn!ZH~I=cOdR@
zuz2IMS1cEt_*`0}h#1^4xV)TebZiX#S!}*a`s(e?Rh47g-K`%Er2x+5<)v2tM?Tw}
zs@8%6CPgJB7%3?lG~FH~+0xodM;J*!K@s)qm%5&jzrR>aZ0!1Kb!DZLt7~Pyh;}o@
zf(FLtn%Y{|!(}Ol*&58**;yAauP_7xF)yI1s;V`coT4>lY+^zyAt7;p$3_3YDj{KE
zaAszVm6a9UHh*DgIa$XzxVc$Wzs4pwDT%nz)y>WASDw2_Ro
z7{uG2NFB<0o?CWbsK>l%NN}`EM8GD;7S!1hbi*?rvOAw^jgD8a7{1KJ5^H
zyh04s-&0sXz`8cTEMWQ&w?}yan@^uSd6t|U-j&3p0F(>4P*~rGc3$}bU-&aZu8JY)
zApgyq+JhGlBY+_;Kv6#Vn>y!db~e3;h{!;diy0)uy)1JkK0p@m4rG&r(wMbo5vr
zWKC4NA@B}q?(~6^wyv%`loW%FK>?599z>TVf4#C^eZ*_5qJjf*qFhAoDvQO}$6+Zc
zWWcR
t1}uw>t>${H_e8b&DPOT>Z^=W
zv;U5)
D8PWlwFm^TA5a5Gz@dbYrBz+qtrn0G~X8$&4KgdzO%C;I71=1A%_5&*1U0p
z6y|aedma(Jcqq8s=OFxGI1(MPE_HS)L;e5^zX}HnvL}q#cWKF8hbz#IQqE#3Z6zSu^
z@CyQGr~fu0;1L<3Zqo!{KLbM228#;QFR=TkWCvl8o+vh7gDnyadpgF(vEYU+-v`85
zfNnO-@z^^-9s*x~7(BJ$JDhB3Kr^t{s=cEJ_boE`2R99V2z;=