From 4f23a8b5dece97340feffd099e92a32109cd33a3 Mon Sep 17 00:00:00 2001 From: Niloofar Zarif Date: Tue, 11 Oct 2022 12:52:15 -0700 Subject: [PATCH] Multi-GPU training with DP and DDP documentation (#503) * Documentation for DP and DDP done * spell correction * cropped table image and changed it to tag * Update docs/source/multi_gpu_train.md Co-authored-by: Gabriel Moreira * Update docs/source/multi_gpu_train.md Co-authored-by: Gabriel Moreira * Update docs/source/multi_gpu_train.md Co-authored-by: Gabriel Moreira * Update docs/source/multi_gpu_train.md Co-authored-by: Gabriel Moreira * Update docs/source/multi_gpu_train.md Co-authored-by: Gabriel Moreira * Update docs/source/multi_gpu_train.md Co-authored-by: Gabriel Moreira * Update docs/source/multi_gpu_train.md Co-authored-by: Gabriel Moreira * Update docs/source/multi_gpu_train.md Co-authored-by: Gabriel Moreira * Update docs/source/multi_gpu_train.md Co-authored-by: Gabriel Moreira * Update docs/source/multi_gpu_train.md Co-authored-by: Gabriel Moreira * Update docs/source/multi_gpu_train.md Co-authored-by: Gabriel Moreira * Update docs/source/multi_gpu_train.md Co-authored-by: Gabriel Moreira * Update docs/source/multi_gpu_train.md Co-authored-by: Gabriel Moreira * added file to toc.yaml * revised the doc text * Update docs/source/multi_gpu_train.md Co-authored-by: nzarif Co-authored-by: Gabriel Moreira --- _images/DP_DDP_perf.png | Bin 0 -> 60300 bytes docs/source/multi_gpu_train.md | 54 +++++++++++++++++++++++++++++++++ docs/source/toc.yaml | 1 + 3 files changed, 55 insertions(+) create mode 100644 _images/DP_DDP_perf.png create mode 100644 docs/source/multi_gpu_train.md diff --git a/_images/DP_DDP_perf.png b/_images/DP_DDP_perf.png new file mode 100644 index 0000000000000000000000000000000000000000..75c402525149e752c343f31ac907c9f15a23b88f GIT binary patch literal 60300 zcmd43bySsY*Doqahjf=T5&{C!uqZ(!l}1_x1f)S4N$Hku=@vu<0ck{H5z-+dvOrO! zn=^Tycfav{|C}+--rqiB@CSpziu=0eHRrGHJ3>oCnGlZ_@6x49gtt``?_Roe`NO44 z==3<~@RvqeE#6C)0x#WGl+$@?`ZF8*>G&!0 zf$?cbh2Jgvluc{qP1nMLiGu};tw>4bZO&xv&d6Rp*+4iw-+QuP%_v(j^0=s(X2ZWC z=>dC?1$EfPPul)1L5aVAr4EZ=q(l;Q3S9hbqtymJ`p3TzT{S|JcKN1K80E|+hDNO% z?3yFUnU(&klO&nJ0;7-uiTC$wI>n@JrO?+0HcGwi`(E3F`{BDykNEH-jpc#ld9k2Z zboe%nL;@O@ZYm26dEIoSwJ!eILy|mifT;HibVjcg&u|{9PahK#dMFy3_p=8*W8-o( zS>WJEv}?{_R!{o1W@n6tqLfH`CiARI_OXw$w(Y+Q52;PZuyQrm1wJ~yw2WcyD!iO! zEmY$CsQ(GQ+;j}4@GLep=cUX-npLSx@fe|d9?LlWPk8gDWj=_ZP0<`b&+Z^H&LVqh5g15g@NRwuPU>;iL{&_I2&p16c?-i z-Vog|p4Ancs(d&W|5PYd?X}5nzQ?EwhPwODks*)Ii?m@yqz@x0xRPZBzBOzZ8AiLx zU{z>U`Q{9}sZz%ge=AyQF-hoaog=wy|EB z_Q#3yvzy;$pB>_TKh+d&7$*r4u^+i=7t4EQ?KxW=+3tJn*6}gw-jqj*TMznCA#Z{{ zi!W!Y{P0Y5ka~_Jqg|W5POnGG%O2d#go&5#g=ehCQ%8kDZodwOn)f#Bo4K)mNWaBy zTo9@oV#nVEll)$O3)ZH_&(QRm)c z&%~A&mCPI6PxT~)B!Fbu|%M*(p&u>ys{eQ12u}TJ=U~aSTsa6_b?j#a!3+DD% z9ucg)cw%H@fp%K|Zg`ha^Bn&i)L~$RNPgaFhIyt+`J)~~TXfkKP+h~5>Z6<8cs%lt@ty8V_ znqI0|Q_%yvs6Pc^_*Cvni_|L|x~6yC3fY);t7>U0;T>Jm%P!xBVk>8Vwq3~>n-u-ZH|1Dw}XKziY-gH@F+U1ebdjXq?N_~cR z&+=FWYaef-&bY=-Bm3y50u%hmnmD8BM1LXcg8L#&@q42mXZyH|xj4(>rL|an_Pp1d zC*|FNvrdRCv=j)%&i3l0B(#}C5sjcvSWZsyi{H`bDmoPl!xO%4MeI0T=Y^a{>^xP` zp}(9(&7g7aJ6CYVJ5l7Re6;o!u{3=3>yrIQ>$S!X?$hv7jJ-8h>g<>k3+h$_`Z#@3 z{-*muS7bfU?xT8~!{j>=*OuHAFfqujd0LIf+dsw%Y>2%&M6F!rYS?wqePiK3Kz2Kc zePqbccnsf0u}bGKF>M%wM0w@DPQBIzvOc7MzgQKrPeT(+C|X{bfR z`LZCVI=*Vd*Lb!Ot1~@Xf;hR=%u_z!V6DIgZ;U%x5GyYZQ4@1xo+jE9YsDd)^|QFg zvsX?yvd+6&@A@B-zsCO{*5JHIOF-?$hGbWt6<+XmQrJrVnEA#|o+;mGgL?|^2Gzr8 zQLeGgz;9j1m2clL+MU;Dv}YL%E755c1F0CoDybAE$>@w z&y9Q^XW_q`6!b!1<2}SG3Pm9(m=PnViQ9`Ami<|A?sT%tK925>aMFWL!m({5<`B{G zxZC=}LCJC@;osz}pFgf>PbZFK*|*=95Bo(zA&}acu4KH# z7D}nF^Ctu^O_#fTv_s?e0|iI9OC$XLX>)fUk|L3=TmgGtGFeq=m%ki1?AxRk`)&<% zh2Tkc+nT<2)7}a93)|~p8T{r;0d35~CH=-2&Z2Eb7lCcKEd1Rg@lZc*HCao#Nh943ChBiLKHLrj4A@ zy=gc5Qa$y>rmytNk^})?#a!24Em6{?00Umjt!n0#?it&64^fJq4 zsvGJE)=IZ$q^z`Mt_~_n?~mes-fNZ{B=J}HaywlDGZv$ed36N@$Ao|x4XJiR*}o2lWmvn)Jiw~r_!lgo#ZfTBgp<-Pa)WJ!t4?NLEk z)t1-_s_Tw)PB`Ax*LD{Ol25H9mLWw#56Yyf5$uDK>Sd} z>5~t>%~UzVC-QkddbT{lO*?&=p+za&BKdNL^>!9gWM4&JcKJNvPW6TcbXnps6O^D~X3P07)TRm9zyeus> zFZg|lZo+?J5<{5p3WFq06t$jqaolw2!^^&P!Ux?>BVT2OyOYJw9jB<=%ekA3#Z<7g z6Ai|{$yu@*B!?^X5Y$fL*ks{5U7vlgqMg#PaUEs>6TZaLy;QPDU30nCyy)sF%a*F= zu^}#`^{=jNb8xMv+eLU2371KfveVOr1rW4QDGX2j8bA9vX3gRExdqF;q&p3BXRkAp zcim)5^+782VHQnoi)hyRF4_|HL)q?dpd< zx)@ZewOX~n=aS&EZC$QuDs5#T@eaxziqM9iw&A5 zG;MQbf&>jJH25x7TKL`PoT>J;fSp9yTS=o7EUtn6)xtO^9oxH>dD)beZe=1C1wW2QPwDB{d#HdQn9yzrf;8Achd@F1X|vW+A6bC z{vlFbxI(7EmmHPM)!@I+-Cdr=D}gruQdfbU8-HR52ET1IMnk5tlu-y<_t(g(sIunC z;G`EAZUb>E>Sj`yNx%C(HhAYp-?J*=EV9)o9__2D<5b${aJE{Uc4PINkPHvySK>Kp`D2LB`7R4g!o!GxYLm#_Cq7Nrl~&2KYXN~U?9H2#qssG5 z5o4H+(l*l`C41KWoWpAC+Nn?OBccTHZTji`#2=xlXIzYVF4hUk3XZU*Zr)$#`LE>} z#7^~YVPEztaX%QGxn*kK^E_{nO{tu`BtRw>-J5wvQHT`Dv~zO5GHvT4D|Ij~TpbeQ zhh(K_>qc_fJ;cd}#aC9YQ89tVXxe6!860`N!_n{G-dp~}->2CYDzCoOHN0*3Bw6sh zeSoIIiP|lNY`O4ek!bKW-aNh$9Gii9iXRP6`_j-Nj|#uFZ~IW}QjWT|qdj9TSv%tn zp(hjR{$9t|^x!Vc$=Q}WkH4#3+Q1X;v5L+nc5wdM)gImG)y%OOmnoxXcIbz(Gn37O z{AlUgw{mFrC@Bd;cz4oD%XFr)D*!vO`7%s!ZNCL}eQLDct9;0*hVI zB0_HT=qEeB`tpH%w^dh6^rvB!0j97?$;+I%d^n)RatMVvE1B)dC@Tkt|gCj}l&}T%dc@zo4i0Ord2ZyYT1#33dej4T*d5KX} z1aj0R)|i_T7?mpQGgy)^tu~igEtnw`tSn><3)JekMY!{gPC4B9B+v+3ib@xA-W(A# zw>A`}o;dq2sW?fkaHo!7$=9a?E;tvBP9)4od!*&P7JNpSAit+~RDeGJ@u&t5Rg_7| zq8qQVLq<%Br69IG7DgRdhCt6?bVsaY?3P=V9MQ%9oR#r=M2jJOiV^3}ac6H}e7Bo7 z&(^iH8it{a3ZgK)L^)2tuiW$vt%v7zu@40@m(_8D73EW(SC;|kMSYEDO#Piz$S4UQ z9%WR&+JDn|Pb_LzS1A5rXa15bifqbQxU4^=r*Pdx#e#6NgVHy(1o=lT`1sqh4JR@U#M)E(Ss zRKM>;43URZAc@(PR4F-ZXQ7KGugO%Ei|`bxt#WeQeooVHqvt?G@;o4z1M7@`T}LU} z*z?IW;A5R)G9vT>fEp+g;x%R~tS?#NpB<0bz0Yat{_yh+T$BPSQml^IYs5@}YwTE% zcRht0gPcTB|4wSbi2BtBRwdDm4$4BM##yCo>=8i0H0GVx@QNZ+uV?T$YHWnFy(E-E zH8=16xUm&(VIviG+`ut-h?b3c#bu`8BXY5{a_~T~Ah}9M0(c>H$fg{s?Xp_BD4IKJ zdXI7=vp|hs6q)L<5l*vcu@PeEpZ3#l&#QI%Q%MpN9`2Z5>E}LOcKl&O!OnsO!)K-h zhN^L~a>-Vc2R%*#NI@-)(umf9f^x1Zd%J@nK{1QGj%&y~Luv01B!28S+X=|L@i7N& zR0FpwfOD0`EswWi*QblSo-GO}65qX?eyPr7#q>*R%(^1SHhX|k)EeboDl#@4+WzZN z-jmq<)Z6iqTu6H@GfLVp3mqH+@tBr8L|oC=N;#U0h;>Ak#c0&r2Mc;!Hx z<#@h6`V9)gcsCmitD9^_l_%#{uU$-e;p>))M4N`=u~hPS3d~1SFr=&~T6~mq#gDdP zSu?e71?4lSHANaoCY9lH6i7Wj!HGI*rte2F@~wwnN?uFROOvtqpoArF^m^AJ6gNew zg|?uj=ZXpYk%e$D~5f2X1UggvHa- zo$qf!+7#m-`z(0ZI-Nx9kJwYZ+8s;Q)y&?T^g+8X{za1SnNP8 zE3M<1uB5L;u0=^L$`qHS8Q5-7?L6A7!*-Wk&gL;jPdcv2FWSize}co_#_53U0@Kw- zlU>J5KV{;PI_Xo}9!t@`#I=De+)Y9BCcih0yR?%dvjau4dex@d+Gf|Z-X5&l+=RoP zwhKe!$?EJ3veJV_%%jRG&LfP$FDO@XFMGju>RR-Z`}MLy)e#`q-|jCBPA&5i2ndnChL!`& zL9f!t*uFfN#_YRR_Xnov6#l$W?{~3BGg;QIPRj!grJ{`gsnPwv)AnA`DQkIHflP$< zKPX4~{Zl%n#_lamw4y{d(q;`htEF8huk#lDSe3HU09`bkl}oY%2k$-xz74jhMUy}@L#Oy~ z8X1V7OE~?p0zv3h?dUV@>i?4;)0MS-ytk1l<9C!P>KM&q$;Qh&u()`BwELrnf?#{0 zD=R(v@V|dKc$Mu~u~I0jaxAreSKr(_*E-$(U-L0M)Zbeo#DyNBjrB3<%FD~|oGHgK z@)^GRHePlwWA$SaCq5-_nuPmfC#TJ3?;EW87>cj4gMd|!rdQ)E6-X;@ud9jO|Gfg8&xt$G$lNM}hh)arMfxH?f~Y&-fgKjX=h zCmFO!w1zCyjEtuBj|Sg5FMkab{LH-C6+xDroqdgkk&)4QI47n_`MVZK z_eja@)nUz)1Y8Xbjc+?!TOB*sRE-}gktk_>VIikP#yWJkSx`&f%H0k)-JQ%5ca`w? zDdxRn)1(&UAMmX&frv@Qck3H2-r3oiaxDGYXwj!q?ZUH@ZOxf=WWEBjf4<3`uFU1+ zz&SXu)oc6v=2EA&#p+P@OzU&AfK!Ugx<4kW!mltyIIe+H^wXy6 zIf-VjOpI2|VqOPqrFN^+5_-S!nJNV??*wj%}I{B0*e+c-=m8rc%xlsgn0 zuSuZ6*EF+oo__U^nVGrlZjoxXgos9uC*j8kUXqY3CB?089}7_!J2}!mGmQ>(^z_kJ z@Xj6$W+=^by)q7NQ;Z-z;z6QrF7fJ=#-HH{KkkX5;-Qa0-vU|Jk86xv#HpL&c5D*jE%oKUe7YMv!sdD^`ON zVLN|$J81`9LO?)3yFigzHjzUeZ}n?;)a&^c?_F-QR^RK_uNSGN6g<)<^QT9g9Q=9B zU;&lnC~7|y=rUHUu5Hl`s=flsPF-*>+)}yaPqw7zrn745^(wDo^~_?pKz%d6&~8|- z{JClq$>cZ&$zt6XD$K?>7eILv`K0dbFe|o>!gK!9o zkoH5_e1nih2Zk6Gufe6Fvx7g&p6Y3g78sb<#a%31H%Gl8y%pkUT(oTF+@nx?v2w0?9Y21i23{bXY^jp zgX0*}ogrd(g-K?vv-9%9qdKL>-y+T3(7jQ}{7qIaoJ2(n>??{%%uB%?zr&Yu+{gl=uYb1 z@5NjJ3;CU*I40R&Gp~7!s_z!{?`_Q9N#PE?F+po2#Cg%H|LyXqUC!WGir7trkXha8 z^1(sX*4D!ADzHdOO2P*mPnre9Jm=0EE76S8t+yU7&P3O*GAdd$8P1Vjo6dk1SYy)s zBFh`iq}A6eS#T3Z^HeQ!7Igu1ioICBogOqaG^lZnL^j;y7Yk4j-rTz7&f9Zv^NhyOH0tS9OmBsS^r!CeGPB-e&H=bD`QfYI~XI&%y{f~Ug(VbWNXK2F|nwn z3FH@?tt-_9yWK*V+`r%df{!kL!TnyHWw(0f%@`UXEWfKw#rid-F-)?pYUwwAOxA?k z+*(ZGHGDzmR8kqZKl;vGFk4O%Sr>DTO9VEqg}fu6$OOa7^XNWa!0S`>7TDv zhV@uOFp$3Yr$jxLcje*D+h9;STX?IQ!foVD7Ow@pAB&Ss!eZrsn1(A5KDfK`WWMQPU;MKt z!=?H)j7D@4Zd@lY0&G^7I>X1W^f3C>hjd%Ee;BLK=eL)1m=VUov2p$xg&{m6>+~h? z%#2zp;EX;786fcit#9kP?)O{^*lCWUc5N>h-njv1woYF|OyUvux^)9#!L6?y(%s4{+2tUn`o<&*Lwj_usPO?AxcL6{A}nF1!% zJ+Vt!)&sa;mOEiQRHl}D7G;jM@#XI+Aht^5+L3-u^1po%>LIV5ET>2PR{;HnkdVK{ zp+sJJ@)wv0`w#E_fBs`kaq;mb2QMN9H{E_tjfg!0tQ-Y?u23wNi%gcq0QNsFwDdW~ zzb}DY>^~C+`7EUWem&UY|GLshm4Dj|YQ}HW|6CnzmVaNjr}0+*T+#mzAM|76<;wG_ zZPtxN8zIWSN7&}o-hTpeEfU=OA%~bd;40q{y)O3DzvjER)05#dp=NVs{zJPV@ErQe z(*2Y6>m0&Go8C56PBj^9XPvHemBasUnB9p2w{+y^68`ayu?d0n1h> zj)p&JC*6A!29MO$9{vpyW&1F(v9ZfTLPAj9BLP=t^0C^X#Yb1j#slmHCgFm)2%RB0i`Z%{&TsRw1fBWyyjP{yV;gz@6e2X zb}*Q&<=;+YQKjN&e^Vz zqDXt!&B!P5KNeb!&yVqyZfD-y04&TX>tFdpBW4_ag+X?!CiZvhv4$P7pL!@B#r6Ad z%`0~n09A+GMPGr_SvXf~p#TfRXVicQedoGLt1pn4*T>(MBjpv8ZVChf2J`vGo5jax zO#TPVfcg&t0qAb;?{lvHnr{Xm7MF2)v^(}#HD;VdaqY1y&yg$Wv@2c}kFFd)aoYE91x2#gGJ!Nyi}65R)-QOI z%}sM^=#BfIQ6=pXrQ_nG*T^wwrFM%|e0+QKNJ96-614?a^EVt-wBP%(87M4?ksrVd z-+J&VUAGR;hWg44|Dc0a$DDD;hm1f@$&A_5(vY2}tx*K53`^-IALNWumc^d>-MI1g9)Q?#!Ge5sO`ERDFi=wQDBBS-i~Xh=xj z5FnM0O!a}S&dXR~)c_JkS@3P9N34rqhJ=QW1mYa+|JEE@?u{F}E$F_kruHAik_56aN#L}152HwO*)etuWg#`AfKI9}~uvu6i`AYri z0i5jJI)9kvy{cnWia=05i^NN&pMZ1T^*&lR9_j@Lvu|Y`ffA zTx}eA`#DFr6=Bgd<~D7GCM6AA&}hQZYU{pYA)kOUJjwcTQNbK7yqoHqLO*jq4uUp zlN*pyDgToLP!D+YD#hS8fFJFPuk9u`!S5)^_3t3SL#JASP1s%kY}XNjo0F46+&Wof z#)%f=(r7)_-hpy-h+g9C?~1lI0AExG+!EDmS4vfjqo^{24F%N(gz#5cHav8nbUYFvuKA zp@h+s^K+mD#C8BF{Q^71-jAH_`JQGVA7P51x_+NJN~>pSoq;O8gql&{Tc;N+XD~L= ztvv#;tARyEBoyB*>a&&Va(kia85j-C z&Bvg3$X5Y9bUZsbgf^%OEL6|J6BZsFlck1#=jSxpT^Z0}oR@q2K$ZEsCyi5M2*bu; z&(}UnXFif197dn9$H80G^A%mZ9gMB#~4;bBvUyx z)9;fZ-Fj7qpUo5a2uA>rjc`}7+E{Em2Mn!WNtMZ|o#*OyOXxTD;-j4~RXl^|CKpZ;Z>&wCvu>kAs zhAsihO{6!UREug)G!2ui{{Xs?$4>e+5iB%mzauwZAi`+ZV%R*Lo%5T~q4G0{jL~9Z z6MFdFo-{?Chd((qXmY#O56E1JMmnV~QR<(|yF(e zfQVi!P#(&8smA!t^?sn21UNW|e5fKP`KS2=arq(~jarK6v`5rNG%E#4QDMf)BnQ9e zo3XRYOVMe|X)!|*8+Z5i!pJX*waNWQaw)peIU7rwAnTUgb4F=zSa6IZV-j4{pXFXF z=Pw_gL9_19m2E4)vq2Fk`8lI;n~#RN0CMU4ocb*9y~~PfGRTT5aV>A2%+p>F#MW6s zR*J}zttbH;8#hmbE_>n?Kr3A9()JBN%BSJ4JA69=!XaBQVNe@c@Uq8%aYAs(v@GDD z7)V}>B|kq_e#HXWaZxU+NR=fvPN^Yo-x*zntm&5Q6=$#6ZKB>_tg(yb|b9&qf zDCP!%$KzZ&h*uxa=;JyI0AEg2o6vc&tiAQvpe0X9N+RRPN;?BM$`7aLjpO1%y>(pv zBLpI_PJCMHLF|`~P<%oX;82C1@Vtq|nhpV#vEKK446QKexnsV3D3a9zXZej&TKA*t z=`*!T!C#A_ZaPk4+)c^ile`(;*h=l)(|9(QB2@PnJjce1ffuOjeY}REbl3o>pU82i zAYiQ*!VJwjzaSYp%}@6^{mo+zDYO@zVU&#w3vXAyT_G}hIa>3<#Rgs3RdreJ1i>g%~v@zbD}4(h!_d-qw$b

b?tc%~J>e)&VsNm%tj z&(Q@@J#*(taba{zroOHy{wy%BOjR@JWO$ribz7A88!=J%K?eWUY5h7}nh z`&n!9ko0d?=q`~9_&on0`G3>cs_BS|@GXjgW{cmnv%2R~H-8$&+|)FRovOZZ(Y`Hy z{-3A5{}4R>fAB&29UUDSLZBx^6T-0*j->%_u|S9t8C}@^f7=}U-)i-3%ZQ(&Mp(_v z7D2wH)qjr-@c!{+^*o???S)#4P8bdT=ck@lMN{TLo@`Ebe@ymJd|&?X?6TN+e0==# zwqp(OUBX_U|IAFy-v6l9Gw|Q_q#g0L$Dhk5^iEDr;0!9Cx&N5RHv&zxA;TMJ23msW z@t(snUdqK$nY}o?S$xJflN~|&t7QAS@+sA359I6~ijjC;!yCe9t01}k>1CQJLCw7W z0dE(hTZrx)$6(8(rvhKS;bzsYK{Q9H-FE)Ba1qayul69=)`0y$vc*(J1A}64bZ?d ztU+1>!ueymeihoe8^z)uP~`zzd#sHx z56^otGU7iKJ!-!pIMP&6x05(P3YNhGlSMe|z4z{8uz*I~k3eo$ybx7r^h@Vje5kg_ zgH^k*@ob{HgZinnqInAB$dKTK$hCKOv%9a4N8;(!)HPm-wk~xAUbBqHfJ?@qm<V@%J%?iu+{>3KN+sA{T`5s`!YrO*_j8KMT*tZ<EN6TVoc9 zWFzVFP18`=1oE9&Z`S5-;O6PsYX-vxgr8gUT%=7Bh?i>Vr1$oLp#P{Jmg!`1Cr@L| zr)7Jq7cU)tQp}%G-kRBcbqzv1m?Ye6pw>ZzZhbUJ`_$>FxXU+0D1YmLCD=%t zU$p9afjihLQ|IHM#GJn(8DWY~4vG(H1xe;zZ1#&kpLM zQv;LIuh6T4vjCM>Bmo&N-wklU#NJc#8Qljc3EJ$pt9fZT%z1#vV+Vn9)wR zhh>i|U>da#4ic{IgR}Ry_1U$SX~LQVfkbz`3<>_E{V3rM76Km>Y`t1m~Ay z+i6&TZ!ANwE2tq(1wjMU_=0zl4ET@zQb#D4wrLdK zPQr?klaoUoQRrHBEWUHKPR6I;PQG@JgMg9xDQexmKl$2)Gld7r*520E1$oJWt)dsU zA;wGMP>-y|LjhnZR!zafY}owTjgO<3oZqO8mO;7#1}%e(Z<^8O=4K!|5O`XX)uAkL zAz|qT-fJ#qW@cj2($b*$6qEwh32Z33sr_WEIHsb4CziOAnzo&Isx2U(X)abFXn>H2 zX#Z}$Lbn<1x-c3cicB3y)Ige`Q-R147A$PY>y4S$g@k|JfKOQZ^V_!&32N1)d^7h# zGGrPZ*ScraaIEIem-fq)NRJQjRRQPHPCu9G3Q+|j8Uq4?1;tLo!nJEIO7Yjl?a+-b zf2H;xSS36IkLp8QEDBVKw8NyN>f2*6)K%W>nwjFcX{ws*V1wP^IUwMmY znblo%S|mG7lShF8rW@EZ@ErJ@9=ZE!yD2dxO0dwP#aK{d`d$}X!_O76 zzJhNf`Qd8b{_n-O`|`gw=9~2+xbqS|s#HP2+A)?T@B{&;1&IHl4ah+7c|}cYkSuhu zNIdUa0HpqhE2)vM$Ec;A3iN}JR&*rR3kI4|qdiR#h3kJ`NJ8|ChIvqEcmL8gwl+7l z#+xPc)L9EikXkVfklbElM1c%aDCPNUZl456?UP)J=*^Y(p$r8-cQlq@A7O{?{V|a} za+k{B`M70w4db6qY7>*HlWZsW}b6I%kbtiuG1B;uLULUi264btMe$c-7*H5hLNU-lQMj z=`So&maVaw=iE1Y1^?k8!?*#m1JoA)hBL+Pvo@THG!8Vvk?9i6TxbM`2M1koOiWuD z{$-Wgg;>(1`)|;jK!*bvCVWRNLwF5}KPV{ZONe}EbY64yZG^)2g6Nhn)E%9jBQlh; z4Yu_*BP{l3U)qB~_(Wa~0$_i2x*U?!wDtfIAlRdQK{$T5oyRJ4J3YVF9!(>Jk`TtV zY055?3&x_}`#zjoQ~G#ecsTBQ1LPm-JYl*5sQ8+x4FwFB97n~!&|o`Ot5=7If*zDV z2Z$m$ijpt#2KSeWz*glI7$wUBs0g(q-uV00Ls?#Lta=d%ZK78Q2oy;vxpi=I*#pi_ zxGp5!8KYTavC$GuqG1RHiNuZfk+uWtxGL->gf*n>Z95FiZ!ML52dhf=38bFff9NT- zIIzn5ow7@O^gZ0lfAoDIUAxt1{}%wUI*Kgt(R9sUJv{v%h! zmPeuM_kv41XyG17J{c032SwrT?%vHZzI|a@k00}(2&p04RAbf_Ylv9wOSsGjsgHNo z!;IyZ4RMKjGGXNyJ2EhK03)mgWfjPjxk45r`awe)P;CGQg+uMZ*lQg^Y&UEz)e&UK zMWAdG*RK-N+)&hBbW)C!PYi5zS;0%@p@a#t1rjAlc==g39gHN(foX#`gG?@1m?n75 zM~0&x#5)ff@W&d2^Gh#o}r^% z^49I}5|Dl}Ii2y$ITVu$+Q42Na}$YCGNuwaSg=}P`8NJVwbBCOw*ko-J2};!Nr0VD z^97fPX!3JpM6>5_`L%Uv=$OSNzX(FY!eB5c8~XzAGcZVd;s;rNkJGc7Q163HX7a}N z+c7rSfm<+Q3iU{WHitQl)uns{}s*z ze*Vwc=KskLQX2@KgBaRyuB_wEje$1Jf%K1QOczaspvR}bQfadhh^LLJOD8YY!LjX+I zj2citDPwpn9{`fHX@n8@zY?BzH-Yv+Q>5h4BgicZI6wPl4+xMn1(<0kXtlnBNrF~A zTo3|evA`hV#|tSbSp}vSQ_G`WN6xSYtB?1CkztNtHJ(htl4LU6(@HW|0lWhQK z_B5)+{q4TN4%+{%iKvdt{eot3=LN4g;AcM5cP^NnognNp%J@F-fTJ(3pa3k->tNHu zAqhSM5V^%*hR8E+h@>>4dqbY8)Cp+jU?B-HaV_99;pZ2L9Gqe3oP3m@V6@b+Zm58) zptS@X_KN=g^DUsY5xaH}JRK@luVfq)M53rL9Jlu)k&*GfErMU5Vf z&r+dk2f!is`_ZQt4kHChkiLYsxbDrVz@UhoTLCT@`86Fx$xF-z#4~gQNvAJ}@e}YP zwXTCi^k&iD_@xR7D2EF7zAInE6aGE-7fdg4bU|jGE`kvCR3I`B5}+U9_!nq3x!F28 zI+kBz2*iVtd^>>!)9X1vlXXmCMIkIAqZD`**iJUh_;23PuCA@&S_6d0XM=&GIRxpQ zcHD#*mtZVHtsGHD#(r4C3DNE;nn?RC9tjl{TXZ8U(C{|-@4qE}_*@{_?sNIWQdW^s zB`2W4V&7pN=U;H*WS`yjho$%GAE_UN)_V@5-gwZb1;S)+9FvtGtP;qp?;x&Im-7$( z{ohy&)3?0)%iU3)U%PM#HeeqK-jE<6a*IUM>fuV|*2SY}JzD2UhQ|lIwdlkmrxkqy z6YpV5AV-9hW!H0{mLUFB!swX!rI9P6zA*>+9dMCtd_TS5bEDq$1l|vY1VUVRv24JZ z6uynJLT>t9R&pd#H;o;lqN@!mtEu7P;YmsN;Qzp8tEu6=1{VkZwAl9>5BurGU0y;_ za!pnlPikS?QS82m20CH&MAIl8!Cz7;>KBpG@VlYS`A!QU#8zUGT~jwi*epW4R79v z3d;dO%ed))_weV+O3_B_OuL5dWGN*rE$tJy%PXb?qF`M7Mh(v~8w&G>)yhclP|yH6 z#ht$@8ol>Bz9~#eK6Mc-Bx|3amu6#AG&8&PGU^N91K=(trR2N@!~s9X%dAR;9A+By zs*Kdr1ek#%mkQX9vTa!bf*LEnlNw9OZ_=c=p;Kde8pYp=$k#5sWyJa_?{8c7U%bgb zzLN_Ko8J&Gvzc$vLj3pyoYvLV^-gIbNJ8TTaol5y+dq)5jC2oUsy^!ojL-ru*7@1`HW{1BrFMsT9a{2)i`)>Ho>qUS zsz!)h0AW9~2PE|Zq(JHLTW5Q&SW58=vYrL{1qs=!b!1uTNA})CM+svZa`L=M|t| zzASjH+j?!hjI4i#{|vxJ&^;gh8BpYo_BMQ0hZwT%r#=6g)dYe?bJ#b<9-JrBuQ2N~ zAXE>AJE=6$S9EL8L%_xl&$-Cd!B)Zkc7e_VXCazM>_H##46!sbrS?6$a)>1YV#7hf zl0ou(?K8a#^Tm|Q|5y8*xr*%`^^1{J)P$%$voom?4d6Z2oAMff}W0QJdJ zVALqk0CBmtP8M>2pOUvb6i*dF%B~uJRjxeK3*gr+3!qR->)DJ(UJ$!Oz6OX}W!Baj7f1AWP7#WI zZK`e=0!CVpN?ixH7AjcgVL!QW(Z+Qk>7(y}L{7a2>Nzwtq_sK4kBb6b6qq^whJu6E zDeQr~f%)?evGB`NEk181$gKjPaPI;sd88P6mXDQ4zXB%T4<1L@P!@s`Ty@rsOEj!c zf6a;SJcG_vl5i27qT3sEu7E$4nu4VR_2e7m|~F z7lFn9BayuSXk?BSr&S1~ zl`5Kt`gS~*5DoMHim5^(AD&pUl6{?!*RwTC*G|v0*5k@yTj{-&lLzWXA|b9jAtViH zG73%b_@rHWkx8~6qF?P!P2qt@s;HdJ@z;f1RM)g|D0J)`6Rp1ianN`9#tH3_|WNmPz&umZeAa_aKZm=LT@6<-9>AyQ3Bd0$KXlO1(J#h<^J8%N0pzk;L)^&7nC7Ov!8jkA!bNmg zpgwRYhllP8(dav;Kf~E3S=m`x9`w@$vXo1j3EZKk2qNlfDh*!BYlhSnTGe{ZksNs@ z_f5XsPT?q$LtxJns-;UPRbh$PkbI^4j7Pzv5(N)9($&#}r{8qDq>iT*#U!j%LX!FXS>|Xjsu9NZ*So<}0) z_dMA-^a>7o!}9WSupkFSGr923MSuDL^2)CbUN`{nU2*l6^a9)fwObKJK=YT8l2D=) zMQNb+f;Sze|Cx;Mfz|~uiv0rjg*ct0omdZMP+-pTV8R}$!N_ZJTf2auD8co=TsAzO z1la)pPCXaqi#w9Iim8x3nx`Qy*)@;z3A`SIApP9R51y7Xe|%^(&eQwk$shBX8Qy6j zJM}e(&YaJicBxs+c_zzE^6ft*Tgak{`ctm1Fw_V3aBMp6nOSSs#E!AxqTsBt?Wvx? zQnBuwQZS5)N0_A5aMa=!KX+ zgz|y8h{SL!O2^m%T%{d$HSo(Vzsip5wc`#UW@QA*_YwoR}FOUk= zt_G)Gf{7f4`(;kH*R93x|GH5nat+h1jl?W&lP;0hYxKdj#B6xx8ax&u#@G+@9RYqV zeWM};(nsT$lZSJd%jb+p4MZHREc#i@@mWnbbKW-905;~FE8-2@Qa{qSs)WxSz?tXz z{bRyai=f8Vs-ZctE1xwo;{zK@m80iEQ0>B1O51()KQ)&&Y3EVyi24y#~`~EMJ@h+}Vf{$r}%Wxaq`<51H zMg|afA+Ul|@hmn@1Mz#R?y`H{e1gst=1W%q5{Gq}3r|6Ny>PO&`#L+VfXKjZ#L3svwN172z2={CfPFpLOt1_di zD*J!Pd-HHC`}JQqBJ&h7msCiENMNbWl zmncPLN{07y)w7N5sQ`eDH@7MXX9Egys4@vT85pn18%3)_o@hSt)GQnR*GeMb^I+<7Shy zvk}k6UKjS)$6V@{p&O_8 zs#B&B<9SJ}V=xcYpVu(cz6IR{*9<`5$1H<~#V3ImS9n}{5IQ~HefeNrQ_1NM z2J_ysf`WlmvhOC~y~^I%j>G*a=w_!nV0pSCr1iQNm9Z_EnVAbLiiP0Or7l?@`Wi;c z#Q74`Mi6qD13KPnRU!J3MKyXis{L_w%{>A;)zsAZ)o#x#oE`}^)cJu;7On_~4hZ9o z-b|(fJ5E@RI!Q`O&Q;G1G1LkF^B8*mK)f&D*tL^44WfNIy0P3dM9y2_$`eNfp|%%` ztkZk?j%S~eu-Y|`)CGu-3gTI3ghk!kun#eN+f4Hhl>0qs>TiQ(IWfp9j${&+pXiYn*WIyT(Vy#KX~N5vwf9EwTAE%m zPTsvLuj4oEDe6EaV6?RKb0W4`x;Rwz0~2Kr+Nc;XTXP_B*yJPIIX3SX)N#YNX%i$t zqZVzn?)0&q9O!h|%qSo%9|BL#A!@i{_Yf{ooB&^^r|YkWM%r_Vp?1EBgbGI9G*zFD zQrAPHWAJzS`1pwBG+i)I2gli+<}@H*%(aGmH=}FMNf%Z!`G~x})UCYzx9Ar@?j5Z3 zlt}SGh25Z4Vw~0V)encHQPL$AEcp}kDz<1dp^1%vOmX<0O-ZYH%>qqI7vyb%8Aj^W zv!L~A0D1W>_|1$T0HRXd0qkVz(9E43ugcqyFRNNF#|%51>p!i& z1Bbz$1j!rg@5R|xj}Qb(u?9BlSQukpUthH1(Z=i2{tIWX#PKVvPJ7h4ep~agX2M@a z4{9oeJ-{fuv5TL5&o4#_BE$3Bf~lpW*HgX6kzaQcGIDKB^3)y;h&>z+&7p4`3~e=q zM6e=E)4nd8DSDor%4A#YUgQV0iHK`x+D|y{26H5A4)Rj-Nd{b4Uu-yei}lnc50&nH z;GiSS;X4+~M5zcfZ99AG^syH#P>Qb+kK}L5(ob4rtK#fNspF)raU^a4j9pYU$|%0u zUk&TPgKB$~0Fm=i0B|P3Ih`hOCGDP(s^_@H*|TS%pAMZDJy+^-DUqqltljLW22s)& z?v*6vcB0|)fqec_bvmTw{9Nq?I`4B`vwUtH<_E zcWcF%s%Mb1kA~oKpbkrUz)7UakQV1>#*W|y=jicr{xmq|ztp)II(>s{IsqGx$q>&x zW@*g@yyJ9B2(eM!t`X<QV&c2}+%8>QHex z6ZWDtSqF&wt+i$C|fL zcb*6ePq<_85ur6rC(p8Y{17`T(K)Nm!XvZdXtqU@rFd(Sv<+cd845b>%UDU8J)^#< zjj*rKb3BWyEgx7>QJC?X^rg?Je`P*T@4nzFhvrLRo<39Y)!J1#pI!Y}O5la!wu|ta zU4EdIL?&vZt!rYa?RViQcXG$o>Y>m35}4~HcH;#~YWkWVP$6o%Rlm6&w>x*pTZOwPfk6T_fYmOuq=;9l|H^nTzv0?(>lE4q#ZM}vsvuBhrt~7^nd}Z zk%3_JpjKAILL+MwkGKkxkkWqLNnip~>FYf`Fay1)7ar%LPi&@qe!yXPG$O+Jux-_`EZj=0}?!@t@Mi!!dUbSc``Z+)xkH`3sK zsM`Dg!1#+oIF3?&r6ltqn!%A@23f}14&o#p-IWUKUP_IG{E_qV|09-(@qbKbO~^E% z-5?ZK5JAWloMj0P3Awh_ig(SLW0sGlq`#?xkmC6(AvM4A1SC~5CdXc-`T^f>u6F6( zX0inW+mV))?)==`XHm2c(97O>TXln&iWOAqvOMefry`{6d={}|1=g^;gpd4?K8%51-a z*BAay{0_ZkwG=i#HYvsHOhTsyO)k9058cBO2o8mE>0=5uxUL!xD=WzxtO#fJ@JS^h zcA*qM>%&F~LJMnRj^esP-;#M8Pz0}>OBW6os5;M# zyx@rs!JTs8;aTaqK9U1BaE~s-4?$Fy)8J&pDo0vn?qy8NKF; zTfA2<$ZpsqD_es1oV{GnWqmQd{?c#=Bis*Tt+x_)hX*!f^RTf6eF_5VP4ZV)UqM*f z;Uf*Bdi_Ot@cCm3T)`acSXU{}tpYZXi<>YNX3~!v*OR`!CkcD%dp$XzF7r*uimC$sI>&y!@%3o3)f1WVEZS-i?eP+{6P-(3ob6#4SMkwO`0-A)U9uQH9r` z>2XeFQCnc$tpctuF$-I{otDn=F3{g7Q^P^~hVw1Qku{+C#JN&1>U=W|BR8pI7dqtL zsC|3YH5J<$Dds5G_y6(4+`q=z!@DBuV%b_3PTOftdU82dsmR<+(!h zFE=ew=JDEk@_E>w1o5~K5sxA@ztcdHL=j~pYF-tHg{wRcx;h0B0jH$I#q+?IO_E)0 zlEff>S+*^TijLkbN=nO-&OXv~kzxk7co-$G(Dh7_g|45FuoO#NnMI?fsi9$MWu^9W z{NDBJ+>utJJplcN1lRB5ycBpC{lgmP_G}6zB!!^VBR9*wjOCUVl9j!pIsyfzo0Qxc zooAqW242N)Vp#CVNTiazuo3Aq=wc@A&uJ@l(f6-VRZ>O^eZQO914>`BJPY@a?VC$v z>>C)H{1_{utr{7qWe=YA^xSHi(+(ziRE$~zjctuZa`cnb(8qRYE~)H$&Wq{En%2 zIfuANJ7wkNw(99E%&t)PohEIksDjE+ zyfer2`EAwCl^y#*W6R#^npkU_ke;-X$-V!!KHuoUrUP?hIqGUOyh})3>S_Oa3SBn5 zuzQo_BoHj}7lr{h@UzEZ66(utBn3jBJN1+LSV%h-$Qe2lWpaLd^L)xygzUlCIHFYeJ%` z?3PulxwyDSjj94m?`)gRh*J#~!K)__S_N)yZawE+>dtZ5BElqWe#W1b6OT4fSY;Uq z96t~R?cng?N9TAvaXvfAt+xOms;3wEZ4G75t5^Gq;|{al=H{hg;?>d(GcBE?E}q zsnZpoikhCoaD3iWl7vE{0$0a*>FfbWe{CP(ekXJhHBC)=Zw}Eq$}+bk!&~Thz~$6H z4TW|`rQ_AiRdkNLX9Pv-fLYnu3CgC86wi*1j>4Qz(MkpBaZo|nlqI0^&_d|00diVH zqS`LO+zQWnEJ;<&;b}Z|W9J=Xk@~G9s1KQZAw@PaGJ+;Lw71IFOLPpZzvx%$senhq zN`#-*01tN8z4S23p6AaQXm}9c(!Y{!RL-?W^kXJl(&Dj=@01?VpQ~}{z6eC!^SpCiTtJRL*KA~r;w}T0E992(kX^j>$&xXh?r}aLN zNP<)L@|9!m>Qj27nxp3TZ;644K$D@#qGO;~+46zly03N1d$$}-tJ6{oTmxHidbUcY zt@ViRZ?7(dX`XC|le6Gn6wC^U6?lTMn$M%3Unb%=6{VK7IL8kiBnHtL2A{khJRAp|_Fhq^FkcbunvKdhZxS zC0jMek~DCyEx!?{aIG3zt;8Ma_zDdjHQyP=#=@o_KPQyZK^})bgo9n%@h(cg;dbM3 zo~_WDOaiGWi_cVEq3fRl+9$J!8FUhg=-)*)oXxhEYunj(X}C z?jWMIKG~-%o9+ry`EYyCB>bwfo}p{p7<;m$7TVY&zX1#1nK-a}g>y08kFHauB!in ze6oBK8gkpXxvOq^kRkw|;OSstV5sXex(N)meTPGuu&S_xA~XDKiO|>RMM1@p0NnoQ z^Q%*rBbRQBEx*bA(VKQvDhCxY$HH5hTxn#-aStyVacmeq`E(g=omtwSNltA z$8h8rP#x4iV@~Zt+R-)8ubyyp6lL`Bwu6_^#u_*4wrzKzoLCmm;lKREKVlpI zU;m(gQ$My+Dh^uh+IqqG_XGKxkRL8Wa2@t{Swh-GV5IHeWC@-9eUSXMxo7_4x&MFj zV}!?&CqO|#MmN!4ebVyMMjVN7;{ZyLgQD?)@E8=0EdLU8^!t~?SD_6lNlJQyO6~lX z_;kK4`Ix5sw~^+bk7?E+S8Ncp_tyuJG3F@3SS6z~))b;) z-EhwFIDY?kH*f;qpW{cRG7-M_{6Dn1zZb>E6hcV+1_G(`eF4GLvfsaKOu4bT1qp)i zH=Ikx5pCm+MzaVm9#goaKFj|2R9Ky3UU;D%5;gVWomW)?mfm(e% z`GNNo`FRzVt zgvkB=Q8Nysf8-ASUL{is0eCv_>egLeuepj>iilb>SOzu@p-p(u`ZN@v+$*+0+=kxK zWDYcM(st*O9E}Z6P$I%p^Nirf;h~#Umsa+hH+dsqflq?Cg<>(hn66Tkscz^>;pR#N zr0M4AIrH@^_+Ex&-i&t|gD|-OkxMx81!Ap%LiBYH zUG3}xmKSGH{nV9#TonUw8n)m_Z*>y|1g|6g;96P7i;qtp{1vE02+4kLsg@7tAc*Si z?FG+Do$&D7k5gxNUl9$WM@17HZe}88mPZ@JkO6)!n+z*d7ZC>{A4oCii?*&R5XQVh zTj&b8>Y|d84&h5mdR6=vxKp8T+{sGZaWmJf{pkYGb0J+`fO+N61{7d&Nv>$Y-B3HM z;e52bw0MUIewzFabWo-chEtFc`6pqcZSBOFH#9L@K!6IB!|ekERQ52=)(qK--Hh-_j|dMHR9& z2&Tn}p`qZHyC+_sg*ytB0wIt)A-kS~j71V+T+rYAmv_5>)d5zHE-p9s6D$#S?6i)e zF7nfIPbeCGo)WuAZCFwKmD;1H#EDY?3~P%gqAItHGw)7^v}^XVP9Fq zZ+QZzCB&!je}t=s&%h7QTOjwKNMyy;AO*>4u(Z}>ja*z=xN}Y^a#V70OpDvF-dXSsnJ=&Z> z+Opn9n8O7>1`2`It0{>*Q)}q$TwN^g9-$MPLY$CU%6mH7lrCRoAV9><_4?d%2B(9^ zPle8{q;)|4Ach_xRwTa7=-R`;hm35Y(tr>z*FLC^JL+@R(5Fu^amGkY06kK>uX+;5 znlk!+)ZV(OaQHBDI?}WEK0rKfY})p*i(7(MQGl9qu6T$_lqqzZLcq_@4{)lY2iaj} z`U=_^t+0J+i|v!oIMM$Wr>6Lrt6qRNEG}iqA{#^Zz8w%a3C%Rem!fhzX zu=pEt9*`k8@`SMwmrmW}_wO;L)*d#eG*}rMg^_ZwcvN6GwBLL#;*|~ncJu%{J2tA1VBGByqi-zG#hQ7KQ zogR8BV5VX8(4@y#Ci9>fa~VL;II) z3@4z}w<EiP#F#vrp89yUfUByrg-K?Pwytfti@FFtYo#*6BI%~kwe zZj1<=CPZFAK|!(|{{rUbzg!mu$F*(Y>27Y;QHj+?iH%Nz2%U*ke zgQ1~C{pB0(hI9W&q9o7}G^l^nm4BDd_`AyZyL`q!>es&sg8Y{z;6M6~s(WP0DXB}PU2tW^pM)y|Hu zkWVvoD()xu)rl~MyAK}bUYNi8eqg>PC703Qil2zrD4$Drns>ts9-)a86|e7C9)Z5* z8t5_{G?_9IH@GfsR^5?y&v)=iU&h*oj-8Lc4Dc*=&~_Ta>IBpenhm z6b*MeyP?K|#9yj&?j$YR&lbYwI71wD4_vxiEuI>H!8RO3H7-+Nv+*(SHwLa#)_&8R z@%?VMQIxn~>*C;je2dfv887G z7t|)(fF-!(D!|Dv)&Fv~KcF!81QMlql1DkG4P<_)lx^iGVq442b@@%XmJN!WUl-E( zK7OW9#K7^4MQQZKTEF+QPHiLqs+56poCi|RUqeWKsULayYMXC~0;1mZ0htQ2A1dm)|g9b7jVexC(#AmLSXoXOHbBBoKA26cuxP53&$5E!GA09h8d9;$=Z(J(XK zTI1qp$kY#hdv#qpWWmtNVq*YmAQ(pVmmnOycn^670S|;E?rKfq{3kLoPM@x$*oVmD zWYEX0cJKwkH8nWL*UhRA$TBA|_2Lo4Abw+ovAE@9;lcaS$)E&&RC1oq`0f1|VJBy$ z$KC$+ZEElPa*4AWsNBEquW!0}JrK(Z;e$Qi9(z0}DL!9*{K2N{u-Jzasq~fpl&cc! zb39a?p00EMaK5vrJnnMAd!HyB)l}Yrd!!^3LM)ZgPLvSJ|FU=|)R$** z+(&rIp>6~8gRuhWD{wXW;A+g4IrXjyOnBeZmM z{C^xxW<3rjVj$V7kQA%Q&sRka@o_3Bd>!}PLTv9FmD6Iw*$a=>0M|yM7nSjPAbCN5G_{8lrKHOfVeGN zHScd5;eN3Q?TU?ztl7+%bfQ((rHeJ#v?YjEW9c}jx{VDqWFC1fuieLS1z*ZSeyP0CA#3gx;lZChnm7dcF~T?r5qfZI8SYy+mW-sRKsc1 z$+epGtHpM^p^Lws?631heLr_E`MR`yYi3lEa|t?0A{|uSI)ol|P8bu9{sTFu3p51- zw)X7yr(gJL?9x&X!E@j5;qo!=ITEvK&k~~rBFA8yT$rq5u{e}tvbo9GaR|9p8;)G8 zC31s_ut{ zA^z^l9Vgt5P%Ku9NK>e|*}Oes?mDlp@vDx65GK?Q?&|w|F)J50>TFF?yvF<36@n$~ zLQd?H;k$0r%Fz+m9+*3N&=SjAmFJwnVb%CnnWHJI?w=ZYdWVagdPjMTP2J~u-0szs z6&mkPm46%A_(qCG@pR{2d9k_6tmMUr*&xRP%Z%0g9cs@Pv-T$js;)i5%{kBXDa9>o z`Gk?ySnWeDOyB^wbQ;0*WW^$wdM>2WHUKmkw5;;kJ`Q$~ zFUh`-efJP$1zKW@%yK_q7diu@s4B6{%Xv#Fb!MmKNIqv-@!dB?!e&=3pF`cWCr(W&oV=;hY$CS@WQ^ny^r z!fr;(LmEMdbY)wYg>GfiW1M@|(in8KTU6vHl`a4-L8)f#xw`SrSka0|X)l)OM$4u_ zfsLTU#dQsMPR5>=pW8jhyMp}l0Nh{-Lcc6q^SH=7 zG7frO{&*c`D%Q>WD0wCF`gq+&#}DTl;O}M!4u>>0+=_{aP-0&3(VCr1W6)?75u2?1 zqNp*Tint^u$jwdo1EQcVP(Fk`gvQ0gVq+o-21=1;HDuZ*6w{5Uq-_);VOP_Yj}Hz~ zP#3|3s9ojYIZ9r?KNrhe&1XvVJVat`rQ46U8@9iZ+UP2+*ELRAf#acT5^vFjJ({2Q z0Zd{zYj3AfhcIE}n@!F=h(E+}B736szSvTfCbF8Hj9b*GJv7Zb0t{WzR{_T#M#PPs zdgo$_jDx{BAxm}e_4|uoxhFy;;gW3KlaQr@l{9>R)B0!=w!NYxkl#@Z-+m_+8k|ng zYCXI!wBW__=UZ?o_f#sSss$8iLry>B$eK$IUSEdU049y^{`tJG-5dVWaQ~|_r>BZR z$k4$LHWtbLn<#pVeSItLJlyEEv&29>Ca=rFWu#o$b&<0ufp1eqMT| zH+2v-kw^8mw-@+2qXcf_tsXjRi&N}d-@AWiPFv*=L`h8!Q}O=xq>KN1s^MzdQ4B5t z9+4!h&LW`5;lB_@_;PT6{ZjP)ii_}Y1MSsWXdSn66N)^DMKEaF-+%Wx3Vra&f^M8= z5(NulwmH1ai=>)@l!epOn}&*tEIGhRHlfL2n|XwTstdV%Lcys$v8+#qAGCb%RT<+=1-S?fZ-Me=h zSCVVa!`m;5P-2@5BH)R;R}3kzf&c*4Y;W<}>FWI0>XGs+Z_@0F70$Z5w04>p@DAR! zzB(ZGJ012ZmB*75nH%?x5rvl5afcEgYaZRMBJV}VO5k|$7dq)i9j8;scrN)gAFt+Y zT2J~Q9?MtP>2io|aUbQuSGbcC19ifFZ23U0yahVHrpbTKP(*UwuClg(2cqd7wtWoT z8}%*+LDWH1i|;TsMYW2NHX~-{gp!Bs)EkwhbeUFO+die;loa~vS(v^Q{^hoyr>u$k z)@!0h8qfzXus$~N#`U}LF6VS=NkoM1YCynB$a3BN_N4LJY`h=YaG}=bvUBNox~5`0 zAwISlmQPC>+w}w+);=aeY2(*U26{pVWQSFc{p%}tvD5lmOKze?LZF!cxS9Wyeg zBkxS(YMr2+KQyR7UcPr~@Nn1TqN<2GmQk*;@S{Hq|RpjpbO3U97 zLFTyNVnACcr{nQP2|4O z6P*ps3etyuvr7k7DKxbFdr*hm74qoC^5@kTd{|T6I+n{rt<#1T zhynbA`TCow6aNWR!|XleNDkcLw%o!L0Qq+i>6Yai&3=u`_-_jb)&7rBr;|6n1O1b$ z=u7nKzUpq#w@rqpt(NVI0Fewu3_SSAVSdzjh5fDNE!Z!&SHTi(V>#O|178IHaXmfS zaU#~te`@qnaXh_e#C?=I+X~!Pl1%($ zvH;)lo3$7l1hrjDOG`tKB5{KWXlrS4n~&<<*mkS)7?fX=Y3H{u@wsgd_AEWoOaC*3 zMK#dxQ>TRerUaDV_O8%$#%|$XfdN2sC~Fb;29C_r-F+uH0I6&i(=<0AkMcQtwx%C= z@J;{uC^6`0)bc5oQ_5|=Af27~ZqZ7WmMHLLJw6wG%|7X&+g6~LcJLdSTXK+6b5wUcbFD06estkOeGG9r547VZ?kT^ZIvy ztO(AND&4gHt`@7qO*`F6xdv?mATHr7>kh+RIoI;M{*6w@GZ=JePY{uGwz#I;5K>Z= z2%2>kHL~DcP;cn~4M@JnU930M5fUfwC z&or?7?()8Y2)B){OoCKDr$6b+5k!O7x$JES-9AgSMQpSU$o>#;<|0F^W8CgrN9dbQ z<>qd3FLWqry-qP4M?F!KYc+9m0F zDBCa%x01btj;{DtEIou|HkQNevwwz!nlN%P6C85~zUj;KdF zIf~?*_2bwe8t1p7&O{}S`aLKM{0@3x{X8*Pqb*6sSm0-oNE=GYOMeEy=f#oi@b%D> zuIERMYup8G4a|CPafZQ0#$ctSO}&I_FHQ9`flOUYB3HsdwTv!1+NX|X2J0y;3+2Jy zgZy=TGuPoTK7It$a&}vZlkw;h7*%f{A81bR*q@=CM@mqxaTfDOO_gEi$6moYMNxLu z)q0e?e4Y^2p)W_7+Ni7f99}u|@mb=x;}XMwHX>QyD@Kx?n2&NVuyhrkB0r1DIl)DY|#jY`Gc+^{C*X%@>wu4GlTq%T+xMIDo{4_xJQh6 zQY{5VY&~9Eo1F%Q{_Gjl_LROTl{F}se6eMs{pcf_kA?U*U)kIypMV{sUJt=76DR8X zy=@}D6r{1kPwYU{IDQ4x?wE!YMK>bUXHuQKrD8KbapIs!`#CkH>2(1jDgo!4D^t*f zXj4~2jA-d-YwsivCk3&uRs-WVO(`lQWCsM=W3UDBlU-U=Q|RJt29z2H2|P~j;4unU z3Z&N>l{gxKEkY@0$7Y_XXR~|b{34Zk+GHF`;Ds`d_G3fD#61rBQJleaRci`vxeI?> z_ zO01B!Zgxk`NoMHf%a;#d+=cmy=wS(#7pjrEAH81v!0(y1mO35$4pZ5b|Hrp4Q&n=k zp%Kt`sIsh<`0ItDuijtg9`Z%jJjaC2zlyL_d?OsD8tew6PKi2d)%H zK-yRA0`Ne*Ee^V^D+Nz~Wh|#Xr8CotzwokkP0*|nHIeQkGz{;PD`$h-c#4^yY=Km z4}@@H!o~y|SSD>h#fx?k?CgHb6%4Iy7hg}c8KbY5ejIz>nzPMU!LH*=OuLMtX-<@@ zQ^MaU1m<6NQRp3+AKcGQ#)^7@RsvTnNY?S8q5bJvba__bQaW$zT}lV&RJ^>AlqL_o z3)I{LdBBeH=1Zf|&20I2`SjFTAv(ta@eF@XupoQ7qgX99q6Xd1<>kcF{UkD23J4m0CbY zj-#~R4usIIeyGf6ucB0;-oJKjz3O7f$bKTc4u4T*Q{xyb$e!S3EwtaW!|E?(PjY0Ge%VdH!;>CT@?Repbw0Ko7q|-rZYM zjeWgh@yt^fADC>|a#bxOFEIE8cP?>RpR>)5+H=A+|AvtkF4j$bJ%JP}N2}OT{f28`rlauGC`uNLK#F?S6R_t%wZ94x?!cG( zXl{(i+s#6Y+JzoiGMiae3+O2@i8Ba^q9tSPT?&Ba95PiXjgKBVawKoQb`0W|oksys z+;~TQFpxXnys!_X^)>MW!2-nm1-I6$(Kin!a5y>*ZDvD`3DzTor5QU-YTRf=gTj3}VO^*Wg4s~1@YoFq{r1bIzeuTcrxbv&nQ&rIV0cFDO? zVk=#dv#f?9_cQJ@lPyHm#lKe72z9)LE4e3 z>Zf=sebqM#E^QbkYs1)s3j%!SarIQ5`4nz}c_@<-{Rt;d9f`7m+I{`r;*amPgnY;8 zB7d3)bVbErQuE_Pwz?m6!Qf*1OI_!6RhQF_wVq7b!9pCNL^#ra4_^F#(g^)WS^oct zGIW)~Nqg?^kN<6)#OHzk&aC_Yvg5vpsw&;7>mF|3<#5{ri~9_!I@Ed)B8~!m zZ1P6K35ix+lqoop=<2YkCeFtwT3Gnx#=(jwI-usd={Iu#boi8k3Q%KM-@P$x8&x^Z z(#@xYZX+7m5oa<+xtPhob%s)8bTwd>G==84f7bwCo)YBZ=O-rdWX#^Qp=f@ngx|iJ9ApNdZ%WA7UBwef$YFF#)d{mV5iOn*1wWJgk3FCeAGi=r-gb7+p z$bu3^XMEv7>~1u9j$1%5hokfp3?=~S@gTx!l_P*jDhRKD5eRZD{j1{LXlrAcZgBq7 zKluBxar6p$;r7h#!&PwKrTZIL(n%oFnRry7GKn-_#>g@JIwlCv2fdAEDI;I-nsZKx zX?K}>#!mH#HxCN&6Hx!v3`{NjIPK~ssI1H%!aCE1^1=mTT`qaT7#PcoAeq%Jtti7H zBWdLU1&7k(un4XSSoZJg_y3vyXmf+zs7xCr68BPH-ob@Fm~kn39^qpl!NE9S5bEG6 zkBQwjZ}%R9>Jx<#I#)9St`#9+hO9Jr3oz=a8tI8~?hXA-rMR=_T6FPBiIZ|lMxa7h zo>2R5vVsO|bOe7L|FRG5HLkJK@1MPY|DO7eD`r^#BP{g~S;0z}w=mM4KybGAVq*3nqbQn7 zl3Ao-qpKIjRzJD0|MuSQVtd_^v5}EgJQQll*<)Y6T*$Y1_v$7g8U>Z(zh z=zSO%usIcPCM?~Hm!RFz3{QM8t_x-dv@W(2a3SWnFg2alC=McZTUp ze&P*{Ne!^TToDzOcGCw_+^^FLgSf5+9x8>}{A__^uo1NI)c#f=o=*QW zzVc5;WfByU7!|1uXdZK`ND@JT4UQC9eFQ;2-X|7q#WVVc))QGut)4XH$>1sg=GwT` z_QQ1v9b)@2E;0me5xr&Z_Fcb6k86wln3m(0ceBChf=L8XMs;Q-1_D7&!- zSHFbWjyp+a(|>g~aIY#SdK5jNL}+@eKT8mgKNC2-Y61e~2_$)qwfcwVq}K?N$Y}E9 zJTE=pdj|4`3Z9SUXyRi`K`Ag+<^!{8-e!X7w~yyIjlaNecK)Et#d0(!v>!M64yy3#l84%!@N zRxvA#J^lnOMP2yA&QfY*AGu+M+so!ZNJ2=+8vj6(m`0KIaF!b~L_hBl(#%t$axO!s^kt+oeC!w4`=?LQLd>m}iQb!2QFz2)bkIxFP zCy{Io5=V8e(thc_M`P^|yNX*)nO%gdabas(>LAi-ZjF*c=ouN?D^x}8A*t6o4bcH! zFcdc-MXFFSlm>r)E?NEL;8ajHF;oR7dWQ3@1M7$(M*M+)-DrI8P?nkD2t=Lx_%b3R z=A(YBqn?5YoI?PyQ8;LDR0^9&u7Ox#5lxJ!RU%J3oB`;3<|n>%AIn%*xEI&$lXez0=}~P~B~GGyC?N zS2##X4hrN-Ip@xC5IQ~3b^cdR*6xL!%so4Vgn~zes+q7rmIp#uNbW#!m%1Hprrn2~ zxVydob_;X!=n5P#@H(7(DB)TP!(3z(jzIf2sE85!(}0y63|IPvjbdW-G4uLD8#c6D z>8rEg)I4@@MU!JP$A!x^oiT!#NKda~qfZLa*gL&8Y5omw)j8u=ghnQdCt3KnO^$QOk z+9Nbm@vkM!e@~s{|FU9Bz>Z6{1*>tL3eTXIm;pW#Z(&Q(EYJHNt&0Kk{Rh;qe^cdm zUIT%(GRsS3s`}bpcWWDs57mF~n5s3QR2+JF!!+qynxzj%P<_US5St143Z5CL-)yZf zC={B8>nA|5Y5M5wGXJ8Yga zX^=|%SblwpA46g5J5LbyPqtx9de`7XS^+O>H4SMg<<`YW!g3jejD2Volpmcj% zHeXGpa1meuP`0|h`_5d%4(Hp_4Xlw?3{7f1F<`uLEEY8mKiHb7XK~|fgu)a~y>E}g z*D|`ITSWPm;2!pS_SW7_q_ikCaWmS8sn5~f7I|;7<;HI?WYrp^2yY*6@^!fXjO$C- zW-;RtgW93un8!si*Th!&erS;>*=^rFU(edON~G(QTUF=4SZDL$vDCajhpyI%7?j=tkT=+nj z?%{R@I!@m4@}bgv@;Xdf^6RK+ZqA2{XKv}tnOG1*aN2i|c6@E#(eKCQFmjirBkT5!&*Z@kjUYN70^gKl6j+ zMp5x8IW+KPq6)(T9ks_yayKde4#@Ko|5vk8^DGk~rroEGYzlDni0mB5JTP2zyzx>Q zUteP2i8tlj^P6voWPS6*JeTkD?>6rj+6w8s%8B_=0@+FF&glPWElJGu^rBc~I~oiN z79!Z|htO-SZ?Qqnv%s%$%tYK)b->&A-Oy|f@T>VhOlOurg&q;4INxx?OP7ehgo$Aa z)R9J3Es<`W@3T0+z2u~gjg5&3hXk-(HqK{PC44asVE;<^s!(JA>c@So;Hphofno*l zsrC_DWY9;Og+)YU+%GDU%J(Y?VFIq_5NThrKH`3MMsJ0|LG_yor&swx_9iWHH}I-} z-?4)ssAz{ZZ|zRlUCb~FA<%y7*Xh~}+-Qroo!EdjjdkT^J6(1%hz5*kSjP3gS(Ywf z9Jvjt>3tlN#yb}+?yqZ0Y$ZyzchG%G;g&c%pnLBt?>6aZ(O=l!|^0 zQ+-Yki;W{s9$e6+h*dOv0da26Y2|`pl6FqfwBg%1-Pz~Ogq`%E(x@t?L_>#p8d?j2 zIkSACq@b`f^zjX3xl2kG5(NipS>(2q9f@E+{y`j z2p*z$7<)86zVPlTyWSggLt8wF;1t{jjxjdqpUH2t_r>4}-IStFNyuc|{QFd#a z_Gcu9MCJKxVV0Ys(qII0D1hVu@<;M$!u3G^uR~!oZ>Zxp%FD}NC1ix1A7ilxcSPpn zp_=C(G&Q20f{eobnW3&1WVzal@0-hRV1m(OaO;Sj5y)K<<#nF$bR4<9wju;DRW#OA1_x$nSKzeje9gTku#$6i%z0X4^4s@rlBQrvv8u4#e3Ma? zZbDu2(vjv=>RYz=e0t0PUyc{JwP+l<99Zx=vmiZs?n3VyO+|a%Wl@sTWhac%`aw!P ze8<={$IOVG3|oP{X`H_m`zuwGuHXjol;&xRd*! zo$1qCQ6nH#1}y{QZ)=ZKkzRRM1+D^p^LPkYTFX!TAx>>{^M*C(H5V}DaI}%4=`A`< z(Qy#$vtw-zU;;g|${`~u_9O%bCrvC$T@F~D%y?$4cU&Bmtq2Om$juF$tH~?&gQTnH zu8>O2N4PqWjf=zNuK+Rh_o}y@JD=`*)#XM~>522zQ0lx$k$kO2Kgw+NW<>IOq32@a+?$71ZFaY!k5ls2 zch{H4TIYsVE^!eWWiMSxQd-r}w{BH?alrrh_2!ZGG$;;AU15D<+6|ksvNA`tvn`pF ziBioEC8rclvXa;NO|h*Z5*Ty)!g&-<9Ra&KCifjr76%K1rW%b82uKvjX$mP2G(_u5 z(I8DCW@MSj)k1rS3wRczoR@*A0Ao*Z`r?~6&#+l`UgYLE4K?qGu&_)~zaiLb5Awk?Y*=04^YC&nIW@SZrZJ+;H z_9Ihj__iaD?PQyX1Hp$1N16?P9W_1_i{PtoVUo%b>Cg)i5j5Sb z`*6?RnlIZcqJ)!q@-n;1m18m!41O;^QBTFGEk#NVZVW5I&^&!q<#rSuF=d9LOwh6g zQdoeSVBSud&kFJzrdb})u&7X+Si!trK=^;A42ObB88L`+H5vRwdM7F2(#mYPV=u02 zgbk(MQU%^%N11px7j$BW2{Q5Re=wIUSll7N>x%e5U`K-s8hz=MF51nu=!!D*h>llL zSok(tg)%|RU(6~d5W%XJj~dY}5l= zw|qu;X8)u>mDtv~w@b^kqbx&IQ?qZ($v>b=CN%Vi&ACnu3KELUf}um3_mqks!C<`p zrvmbWYT}2vU%ogXGe%u zqo-3#Bu%yr?FU3!@9)dFn`;IX4y&qVgXl*?)Fj}d!bc$4C=(4Xf;Qy`2H#;g*OfWS zgGnp($(GEF=G6<|_Tbs3s)UV)Uy!sYBBj%Vpg37a3_JCF0NrdCc+<^dUZbqD2J0CQ z79W3v)C>+n3OjgZ^cai1da>b2en>8DA{6TY2CR?0v<%p?hbjxl9^Zp2D1Ra^(5@ft zt5TX)C>q4bEsAy-Ww0oO`0{j}Sf~rs*x4&gF><1W(B-T=06Z>UWZx3TUW%u`M8nT{ z#z%G>Fb-}k5A^w63#f)YRu17h3AdLL`Nh+s8uI(PRG#d8N>M@KGkte-eVppnn7W^* zW}f_#Uov(uj5?YwiA7a6oEUmhY$8EN@mZ~iX5&kgr5R@^c|CE`k1aN@qR4mXOm`x5 znEsl+bt^R`NnH6q)^JWuT$R)OB`0S972fti2guc6rc$D8xf+g-cH0!SfYcM-wPT+@ z3*b-gqxBs~rCB?NQJln#o@(zAtCkNBm$rLG`+T_mWE3DlQ;2A$ev%okLP8?dCk1$t z+G?3ZX%}&L3mrwQx{{L8r$c*02_5vV8>Eh)K!23vadC0!%%|`m8V}nL^DFkJ9#Lu5 zU4Xv5r4|g6t-bE9>ZmwriD+%#;ZyI2{P*3aX@^@TgyWXUD?rmJC$8?iN}K+3|4@i? zQqTS2!~A(7Xz-68LxFeb{>lEs)iVUyB_k@TH$lbPj6;y@yV1wQgiYHJDN0ChhR5pm zixe%%Af4{h=akl5qiSd#x?b*j(PRPb_Iq_Z1Sg%E>UYOrV51rr9EAV0& zZXdN5-)gBMIoP*DU~Xq|wdXusa3(Sf6E)p?j-}T``mmmGR2zMfdT)n8t<-`SCx74h zAI43nNzW+WEGg3?o6iw;K2)AuUwZPnNL7^V*9IV9LV9ZY%sulwv1^C~B{XGC8=Ad8 z_iU#Xp4+uwW`fJ_CF=KsF;@n|nxWRhbXw7dpQsn}ZVj$0=2P+%UodF$0SOL~{mz6Q($+rLAqtslP1mzc2^- zv6$E(01$#>t?C*M`s`^Z1_Q?CqZLVp#S1j!>PIyRmnd^*8X)*mS9M-C9EJ!Q*hQ-1 znVjjRYtiCtPgpaieYNs@Lb?Y4l`2%+L9X=pn+I`nS_0?oh^Q4BZM%-2o#FY}xtWQh zB$#XGMwZkPn{Qh%C)r13``C{d7ir}wNBA=e;NIX}X z9-Uks7XAt!BH#9tq~&S};myC(RsJ{oSO2Di|4#q~|CiWg|A2o+TvwH&$ox4!4|ckv zY)FEZkr64)-+;y1!f=TEK3jl$a%Mj3&aYEj8w+38ld{@#R8&+0hA{iME>ESredx{Z zj2tRk%fS^7^^a{k7QR7L>A&O+`CsJ28O@vkx8Ipj@*t*=ytBs6%E5)dEBB5F#`g$K zpfYXX6vOFKQwqE{C`Hm*wd+?V=<2VNgPO82R($3gHK5re&%tOQs555F0U16NaE95q z1?4(B*&v1z7!-P3rqPH#Ox^XWm}w5K%( zXgH8aKj!DRBE{{nt8ZATBJloQ{#pl4Gp|+VUTfL$3Nc`xP4W+(!A(ZY@qK+U>d$#P zzxUu&h$RlABH9~=;E?2oM*`eonkjJDGSAP6X2wJ&f}w)+t(Z$rVCX#Tfy99sD2|Ur zJ4!kQMCSaK)sGt+BdDgJhp;U}ZsQeVl=uu_h2+q11fj5g9DiHhR;HngyOU#jH6~!M zmMY?AW3$bh>MRQM=bHk>toIzINZ2;e{B}Nl`n0(@)<1xjnHl(LIU=uN1)KB+uyXY6 zelT=IJDyI;9`H#@MrCUOTinL!^>o{(knZQio(>MeO@sea3L(Hd4X`}gXN|-2t4zeL z$fSQ`&41-6uTW9?5ZMO!9P~=)oyd>SOayrnLzWhxy(I@ge=mBlMD#eCTJ~ovwx^X3 z>W1NxBs@P-Qiwzd#zOglfqX@-plh4Al3JJ>;?di*`IP~1HsOX4XB_zyHR0!L*0vx@ z?%5T(wb>nz5T)N=5W^G9T!|p7t)3Afj|<$h5jj2#%mvvd5Olm&TwqB6%n!0U zUI*owGBaL&-NoHV+YHjcdNEy_0I}|%ao6f!6jYQag;Y!1SU46nn8 z7dJ?r^ASO6PnEtayK_}l6_c}m%zuUQf2VJT3a^naAR<~;t>R9yT0rDZ1yC_=Wzq(G zokJCWCk>8LC^HcUDVFPt0DnsNGYK~pqY`jVbdy*80vYnPlD9v7>PgjX*52`mY zLsB5{AVRca2bKqj3jHvV?zFgyiMIws_e8`x|7>C$`E@0)0=spslN1=YN1JrQ95&5k z7fS{nBr0<|r5<+widY0&*nwmV0^ zUEAQiv=6}{Te*@_pviN6SdGeQ^G!|8aX0J7JRMH}`+5$Q>BTTXtGYI0?oXXke~9z1 za#lK)a8jp6MOsQ<<+uS)RM?VB}G==6?#KUwf z`|Iy_z%EaEQ^arzU9xE5-#V$;0U|vQwE~sX+CW8-kwa|NIc7PcUX6(0t!6>uCr$V2 zPsjT}Ws(e_8kaD4pL`=aO;rxzBKa|R%)2DFhikVeZBc!@joxZAFYR4KBp;4&{+)RH z@D9Lim6;wcn0j1jo$zz>a_V`bZy0lNT6o`h>n{`DoYRvMu<5ooDWkY@`#8Xx=gTmK z-VWNaZFJ9qtnxou_MtVZBhq{b;_;?xJTP--J5{KDGTuR?D-hYdQDRk++ELJGihtwl zLweJvE)5cX)2Ft!4ce&1h%VJxqZz1VMOnq8kz*Z_upw1|o9tGCDTJ&?@Sdd6ARUT;v&s zFtS2^#NdB_G1AK~7YM}tX2|Rz2f~YWU%YcO`QnZ>CHvcvrF*k8%WTik0;r#NT5Npl zqZC=VHM0g*z)OJMaJy?Za#1Qz zKxe9giG|o4j^^t_hQU7ltU0^WE2TsLc+e&5N=*%hAAGx z$n*#FT_c5!g2?YQZpj?j7Z<3rEtx%ni=Ll4#Vc>hQ}z3KzQa!Ts>c_{J}PuC42z<9 za{Jk!YTek?ov5JTGRx}Dq4^+yL#CYU<>aKF#w4Uvy`wgE?1pw}_$_mE7=6S(wgm14 zp|51KbN6u%0$wWoKx4!U3Ru?L4YPaQi(qJM@1_g()-^K@$u+pOvgB zLKC9GU3+GV-w9~<>+#ie3Rv=K==CD}w>>K1KDTn%<5R_TE-(i{FQ9b~b=!ExjVv_t zfGrZ*D_S}_3#+PzWHefx=b(I+v>VSvuVPs4avNKEZh67H@la=ZU<;H%g@*&x{8b`N2JA}8 z!fi!{B2E-7K3fSGTCbkq^Ug8`$~nT~wCV7)5?=GFLa7ToT>VhjR=+6!D3N9S5ncwA zcQ#!|)1mzXP8Tlw#*O!A`LVVQUDSUdeDnn^qm}=Lb49FYEFCuv1hkl+|HdMczVpD| z`?d@m6qapujzU${wh zE=o1jVvHtH zjf|%p+Mscnd5vmcysW}+{%r60|Ng;LYT3GP#%AmcA&-i2-m2JggtYUeSc@+!Z>4(J z?AC@uoffpe?aROeb%Xj-@7`FMo9)~z_EpLj|Cz5MOD}YjRReq86D3qHQr#PWv^m%z z=+`%BaQXA&oTKIZ%g_F-_UCt&_x|e_a7g^~(`;1#>s3KuG&Zb za*-_Fvj6#wMbiIrCuF|_{&`v14F7WTVy6E5!gcf#f4LF+zbvBf0xL84bx9AsSU*x` zVeXnRRj(C^bUeSlo4DS8`+_m|9d>&zFQ#V$_h{nWKMAtb{PDWovt|fpeGg~#yB$Hep2AYgi5lo-kR=J1L+R3on@y>IyI7HqZ^L<&p?Nxn z4s{%#VRH-%WUr-sLPf9=(n+jp|@-GPJhe&Guwq)?>wy@>Zd?N;= z1PJpYtEp06A|bIU-e>k)1x8k4|K(CBm<6*HXLLdCSs`F7EApEM3aM(x9vD)dzuEVkN zbic!?!TS~>TSNj$?AYOoNd(<4)mThFo~>;2X`q=Whh1qX)NZ{I`T#7E z_1jCTsj9kJpuXTCke8UJGN7b zK1Kl3jhw2EPCPM0RVy`3p#hiTB@m1PtH_IR%H(>FW)>$1waf`2KsF{POQpTT{ua?) z+!ul@*eELM+4RYawog5M92#xS?*24Q7INK-K&nh!8K!`^TLE5O1=|kG=2^Fm;@$Y{ zq4G`-!Kl=S3s+kVf&?`-!9lsMv?+yn8o4*_LcPIO);HoFM(8pfb4J0>=7iAga?U8qrnh&!PR`di( zf4)m~l3APe5*N`U$`p9{5ldb{hBqdtDqIdR;-DO^RC<-;@1@;!CI`-p7^m}fQ5W$T z^dSoWI(i}7pusnhHJeH^hd9-*(oGW!N}XA2a5j~RcajPHF-EZy$|qjfCik{>9DBIU zGF=#jWi?%1?jHraHKB$a_8O172SQx|Q8;X2rPz-QAfqX5X|#B(;`Ha$+8{4at?Q;) zt3Nt6X7&-964F5V*LbiW_ z<474{w+|Ct8Q-G56t8Ea9NP)SHlBXJ_=t!I%wI*DxF~Sga-R}>GMH9|7>U?o8i)y^ zZRFZRc(xx%I7W3U_q=X9Vi|nshhR`E!Yh9R&sV!93Gt&$!Ej-D5h---_TNiaZIl)T zwi5OAJ;y8Ip{HK?M_)SI8BRGjGk(^wMamUru%w3HAQh^QgB`!A6X!%g{U_R#px3H@ zoLts#2`UJV6FgUw5s}~7XccL5l9C`~UwCrWHBQwbt+(Xb9QYB?6h560n==csv9Yxt zC$fPgK^_x+(K1Vq;z;@-6g;A82?`2=qs9PZT5#oFZ;7%a{v|Z7@?zwXu(huJs3Ydu z64&H>{roDl-}ZAvz6P*H*h+t*$;7|vKw>l6e25MqCVXPMN!7k8SGAL}#e0at8i&!e z*Q?vPe{H*Mih=hH+44a83M8hVhEE1FS_EKI2;PS4Kml+|$@kh$yXhw8p^E2yQJ>xXkDD1g#r+o5(yB_qwujUvLT^>r zE=6*{!^(Sr$Fj7vbdMTqF@<$<=W{|-;LsM)HW{%KC@4W7bG(#1O^P3z#~ADukiD71m?iIiDMM?bea z`#%ngoX?9GSg^-O77wS{^|`%jVC1^9EoijE@Ya{N=i9{BdknVcK}FBS!LhmE0(3;> z0V^4^K(K-BKe6wEk)Hn7GSuhvLlEa!vVUK( zRYhj28n@eMcWF3HcKOwg%?y_*ya>&Iy_m#9{m$eVEN*mfUOJ7O%$)?QH#j)}*z3^4 z)}i&~__A&{8QubBaaCy^>3{{J!WEC>God2EDyDfNZt=Xe{VraJflSqacaB>fh+cX3 z=x=BDl=82Zf*9Fi5LrA9HCkBfz{Tl(7BB@0zmR)}gaIkAgL=DuM1AjdBL`Z}#mDYn z{z5c~|NKGM0W>1((T`W`E1bP9tvU(q?d_STc{t=*W{9%`};NiZj z=eN!!5k2bP`n3kjH*@$>b-Y8%OpppYz&WY6%}y@J3FEC&ECVS`oM1TJ1QiE=5csaQ(jyFi4X!`0VLxWySCL?*Das<@?}));hC8n;0(GjwUR;9j@vtyXBEHtKYQF!n5OX zq=k^6AZJ41B(2H4Rhz7RrjEV!W8cB)UFMu-s|!}pTE7{b9pxasMM1Aj-vhR2P;R%>ebIwdb;fnoN^p(W^)QH z`e5$4G!lK@!=2%1_x+l(D-7|vPkM4SZ+RA+KV4z8q9teeRb$fzoTT6q93T;cwxtzS z2OxLZMovy6NUOmz`}_G3ex6I4pzZ@7YKxsM#46=h?b`m=+06C=?&CKS2rcGQ5~u_odcJ_jb@H(cv-A*79ocQ(b|+u6S(62M3$3#`%v4%u{I;w8}j z^l^+9&Y6O7m~}lqVw^|npRJOGw&*5DtFc!zn~KUYv|EJvvm99@q9$KELu)gAoc&XZ zJ}ngKp*e3-TYg8Ohh~<@FbsMyYQ~4^4$yY!Vi78|zL}K*^f%L_K&HgA9Ui!B$A|0edMoHWJy$^5xnyPwQqN~RO!Ph?rk_H4xgkJJ za+f>@C9EPYHqIU^bwxJm$k_>t+V#E1iH|`3@zUvc2W^2)Z^5Id@^!K(13?^`Yrh7{ zu;D1k+L(qD-82DjliWRg9#7|B6@)McB(Bw$L`m1qxsd3xTI3{5aGSjaK_<}?jj8^L zA_EH6Of~uac^6Dc+ve?LN-`cMF=ZT`+G5pe6n|HHi#S=xCHHJ2^YgePr1*C&CwF-8 zC_DCEtuQJt^m*LszW*{C1(0;!GIGD@lot%yV%JGo81|LlMif|G;V+CqEOOx+|0v>t9hYk z3~xp6F!s;>+Dil@y&3jJbre!E*b`=%csCt9IAC`M_mfy*5IV*u>V|0zzM}9PT31D| zY{v0;ECYHxcWF){)rXc=>&ToTc{V+rLCzvGT8Vhm+M1gk@= zhW8w`eR#e~rrz$v{Nw{PtY_PL&SlVa?O>K}o`N?%H*77@9>%ezfgj@Lq6%eWe2cD` zCy;(9t0bscj>==}Fvd~rn64;tLNj`#={%9jD)Z{GCY|DerxYH)c)zStiNwvciX76* z*V~ttge=!=3^cuwSkz?P=GjbL^g4Y{@=%O@BY5 zm7!0Xf5Ow*qjX}m9O4Es)UM+(RO#bO^K_ev4;#B)|H*LaaI7*zEN%!hs&+4sl z1-?Dim%r3V*W2o(8|1%Z>YVoo$)fJQRH3(FFV&KscNl}&7+N^WMs+!4Eth`;KCD7G zs@xEBk`01Ka(i+_4;*k3!x&Lq^0YT?&=#iV;gCMb5e)fDcwnIT5i$vLaF@r-sf1Yy zg?r5U!=rOAK1eTO*!8WJ2AF-)x!3oF!1uEQ*^EPA8!fId7uo_Hgys(IKP;MTw|BtO zb%}S*D=wmqwGkbvUq1FPT?FX>9#F6jOk%jYj2xA8tO4Yis1R7wQUszghl4V2*?8!w zdiaGzHtOkCEYTo*b+dzs*9+-`6zJcCN9t_7QIvX|wwZq;7nQW&C46)u*ui)^+CG5N zi2Inr;QLv^(XL5tig|wj^vSfT;B9?p!yV`x!*AG`ZSm6L(nFk`##7S=8^|olJfT{n zH~cnjFF0_Bl=U&6dU5ECs?XTXr@Z;R zbKkS2&ZNKdRpTnU7G4kX`5pYD0=LB=C}XvVIA#lir-3tzFz0kU)DsXnZ3OG) z1}7YXYVjiB3?g<^r_v5>l)|1i#nxQ|P%qenv9#ysaaD$Y*Y5{c7&2uvwKerB&0tDw zcc!|B_YWbzTpAw(1cHd|p=@k&jzbijq2=&5Tw2+ThsEwvpY(hZo3WjNZW{PGvoi35j(i)FCkvZS47h+d$3laPFYYK# zu4Mh|oc{eoFuSOHN;zp3px@a$i_aL1Y@Uu;QSrOjo3sv}@NxJuJu@SDEG7NrJS2IY z%l-M2<7Ur48HJYGG5H@QHJPeyKiJyv(c!FKl2B&{XKu(oR!ZdgQ2+U=t=uUkCOf-z zj*_kox0c(AC?D4O$o0H=PvM>enQG-Ha?i6yAWb`NZU+g}+(5&lI${k=? z`Nj3~+{50C(_aR~uF~f5F&`*xa@+OZak0R^|4^jTTCS-Lb)wRnrTipL)mrxEHtnu# z3ogwDOU1%#e*rwOEctdX-%cf&b)r}>WPi3}KzChdG`tFDb3G%ODuPbiTw(z1#g5CUsejInx^{^()X}rq{w>T zq@D+gfw(@kb8Z#MIQkBQFF4?Eve4X7nW$pd7sP3o0l@fzO1JBG>2`ea zj~JFi(0NvT%sgdUQ8{I6Ign$KPCE@-VUbhlcjX{1X-2|{%{()(#tC*&<^r#-d>h^n zAV_ujla&=PYD&mGXS6Ext)xxidJFSVdFILOqw^XXRJ+bMa94T=xL;nRJ5N6ZLb6iq z*c0vqU=LnPpRBXWK>`NdGJkQ~aUSc0o*qtN^E);v$)H9%4Xy_#FaP&r4QO`CAK^q5 zUIE&^@%wV`-URz|C#|Q0dQ$0!fH|GXTc0cyeM2zV04zNSa$xJaLn? zM);>##SXWy;aMr^Pp{XW+;U*|#VZduC@PoH&SuaUXzt%*G9UX>;JL4YS-HE)%eoJz z8=_Q4Ed<#pXqx zuC!?kHy_>QeX{F{uG}A;&|$REsTE}*6@E-gtKrehDS z53uq*GaU{`EN%P7UvUxVCGX{m39X@n!c*=s!vkIy+FK>N4=-0RYO{|>^Xk|+DD4s ztx9x9T*;;oH33{GnW*NiwMa0-Z}rOsxSD8lF+hd@k~FO-H@HW;*8PNP0EX~2n%8YL zX)RJ~;#KT~xf}*+Z#*T3BTETh60G&C1jb@A=FDPinJ=(uQ&^+iHwzPHcL2v-Hkm)p zG2bC*JP|gkU9-VA&PhZmT|IH7Yw|ZKjhpHMmDzX4Yo9A@(DRkEGTf8VG4FHNm~Uc` zB34lLBfWH~cH2>#*=WsdtJ9aB-_>s9z26kGOlr6?rSoJZV*rgx&#mL1bxmeN_bn_m z4f9Mj(sMf;Z;)4>a~a#=d?;u6td5<5$JS#k_qwODq9yM_Dfu<6yROrO= ztC_CIg4nuxYm{;3eU|F<-_WTJ^T)k_uPbM?d%qmV8#j~WWW*K8G?|x&{Z`sX zMt?RQGxl+dYjQA-0H}^@8O{+adBKUG4ifGYoXwgnmU692izIZ4cJn=#FJZAlx67%b zG4erLxd$`!22b23`+K#NUtDO~bf~#efE`wC-tCueZerjEf?8Y^cqv^fT%HMqb;9Zr zBJ@9>G}9(XRUtXVzf0Rk=PruesX2Xj>7@z=K@ri{opB9+KwmtrL8Qt;=bc*#WdjB)Y=KmOl`w!XL|Bhf{ z^?yh@Z`$$&or@}f6-Z2+Go5LPeTG&PGEi}1ao~3Zy@1*sIq}sl8+j*+*H@N3nm9{p za7fywW8iaIdN?*fFl@z;Vl;8G^yte+&dL)k|KB)oV~1s+bi*5(z~&}*Rqa6%fgzZV zkP|I+D65B5VLTl@|UMEUi;Kv8qZI_q+82SIz2{1GMs=S-cOkj17XVyw>?Axe6NaX>Cnc;O{XllnddyA8a z9V|;65G&EFW*&cHLv@?(Lq%ofuH(i!ky}HbH-Z}bh7~?DPo`^+koGY8Ex($;C&O6= z8`1a)00Sa#k(SsT1By-ybk23dIIgUmG^HBjy21GTqkJR!UN{PAwqng5HF z+(5MpModV2UoMOmXX7YeyN6528=D^o?af3h(<#r+9b~2f`t-5rH*4_6M{)s7qVrbw zg~P)A$A{xbh-|$?;E^cHIZsUu*{NP(AVR`=Hgu6kiflDf>(8BPN9XVN3K1|^rU9g* zma5EQqHph$PApwk8h)P<`Qr?c?`D^-avu>c1fcPH&+;tILLIitb3wMS&(QCiuRBDp zYk8tIg8~A?4APh-UIHM&$G#~lA#n*41)_h;t`E|K?7Rn7DSF}38I0Zfd!(e8`-OiJ zF-JI+GL3UIE@^OCRuW|Ws%S*^W;v$?lOT9rrTqN|6A(59S)zFi^#xuQ`Z0hph-vqs!GuCh8efT4Ym>r z2tqjW7a3RFRGJrlbwHGL5ye)1)%!|L&#`U-;|e*7QK<=> zHa^HezPZ>b>+Sv6Py6PZZBE zTCMuc`o$i7a9E9T=7={kmShhHG}?O=-4<@FWiHF^7p7Mrc-6d63Ejoc-GU_zkX9Wb z{l4lA$$%8ElhP;UMeHoE+*NQhG*)yNniQA!Z7i>Z+n-w>uM{TJX*B{@j?cT&$s;Ot z*r;88BswW~VDvd?zH{9rDi`&Um362G^CRE-^?e_)Fa4j=f-VWNIy#D+7v-UB+sL4o z#ZoJP#T-uBy>{K%6pKP%8fu8J98&Dr4NU&{-1|axQ$d z4Ch?%v(SucgcG`kRDF2y(K`_9CP90_ak^20DwL-33=Fd7{Sn}c27&~nvg4s$q?S}#zG}@9_6JT3r>6?rq0#M=U;3SDC$F4=}pW_sYHpz z&+vqsZ`|_GZ(D@Pa{IlA;7s>+g17t$KbIKUhJ-jNQM$#L@bc^5pB9+c=N%apk5jUV z+vhPtk@Q}<=cT>kB}0GVl@uY@{DF0p^d?2S9j|51R?Dr_yyQRS|D4WDo8Zv@O@x#4 z`K7-X3+C~Zms=|Wu!PCN0)@{Qc}nvj%})i=TTPuv!0b7C<2uwL7Eou;>C-yG%8M7> z;L9~1w(k$3PX~pBD0Un*avHE(mnTk!ek1rbrm|#KH!Ruq%Xg)4sq2?H|M$3r8eNf(VY&91^+WUk#OIqswEj-B1=KB?(l| z!hP8(L8kKa=aN22($?GnWz!efMQc6*MXo)`M)~}m4m=k=sqZVObPE@9(cIde$(HM* zBX)P07g)Ew9lZM}TS&4$=~@!D2J`R3!#7-vj8wp@C5iO4jVC7~{J!^^*xk3(3P4}5RzyOgj?;5PtlI6~g2*KwR^ovU18O=O;1(I;7M z!vnD&>ou`$HfjQjmfosS#4Jd8AwPb!ItK{{cS^>Q)@HDGTRpgar6LR3~)E|E4z z5lfsy{pqGlOVgam%@b>p0%dr`$&hgf&YfOM;v`vpn2lmzsDDN9R?SpSG4dEJw?Q#g zl}~8cW@JrY*p}ONbD~k0D0LcUW?pjZN3SHvfQWy;rpYYNB_^e`R2_x0zP-yv_mg-J zMMA0*JW%Cmjb=6K|zu@xiO2w+D=xO3|E`g zi}1BA;Y9OUU0&RAEt@oR&?8;Uh0R~=a>LQ(PdN&*(_^7>cDJqK%(bV#J)3DX9h#;i0I34MLJJ)t32}Uv!@l=OuPG2gi&Un=Bx7{{HT? z{lZal%6E@1(Z@VC`Mx0UC-Ki0T}Q9*p!%|i7lWaALdqA@i4~nDS6zAr4=Vs=x4sz}op^*ePm0Ts|Wi3kf1yom=`2w?inG+~v)X%`hOlRAO6-ds*e zskMX4=1ki-5NNuc6}A4CCdIvKJk`oi)(+Oz>O?Pex_{o|SuU5#=CTuUL~Z8%>>zn7 zFH`NKTgdY3!$Gi)M=N($WpatE`#!H^3y)c_8Xp%we`xqNq9ykA_wP5U$e%r8rM1X? zpI9jlPAO)eOaaAsrQc68Tm5Q?p|XKK$zFJkTEb^(ssQ;jHYIBM*Bk@M315$yjw&=) z1{!cK<9ozkn1B?tXWxdi7o}Jl+S&GfojycaR~FmO+wH;rH;F6v|11sfe|aiQH!Xao z{!8f`;B#a1Ps2AzO40cBXB`bOGQ%$k^aM5Izx=Note: When using `DataParallel` the dataloader generates one batch on each train step then the batch will be divided between GPUs so the `per_device_train_batch_size` argument represents the total batch size in this mode, not size of the batch each GPU receives. + +### DistributedDataParallel + +This is the suggested and more efficient method. When a model is trained using the `DistributedDataParallel` mode: +- A separate process will be assigned to each GPU in the beginning and GPU-0 will replicate the model on each GPU +- On each step each GPU receives a different mini-batch produced by the dataloader +- On the backward pass the gradient from GPUs will be averaged for accumulation + +To learn more about `DistributedDataParallel` see the [PyTorch Documentation](https://pytorch.org/docs/master/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel). + +To train using the `DistributedDataParallel` mode user should use PyTorch distributed launcher to run the script: + +``` python -m torch.distributed.launch --nproc_per_node N_GPU your_script.py --your_arguments``` + +To have one process per GPU replace `N_GPU` with the number of GPUs you want to use and make sure `CUDA_VISIBLE_DEVICES` is set accordingly. + +Note: When using `DistributedDataParallel`, our data loader splits data between the GPUs based on dataset partitions. For that reason, the number of partitions of the dataset must be equal or larger than number of processes. If the parquet file has a small number of row groups (partitions), try repartitioning and saving it again using cudf or pandas before training. The dataloader checks `dataloader.dataset.npartitions` and will repartition if needed but we advise users to repartition the dataset and save it for better efficiency. Use pandas or cudf for repartitioning. Example of repartitioning a parquet file with cudf: + +```df.to_parquet("filename.parquet", row_group_size=10000)``` + +Choose `row_group_size` such that `nr_rows/row_group_size>=n_proc` because `n_rows=npartition*row_group_size`. + +### Performance Comparison + +We trained and evaluated a number of models using single GPU, `DataParallel` and `DistributedDataParallel` training modes and the results are shown in the table below. To reproduce, use the models included in [ci/test_integration.sh](https://github.com/NVIDIA-Merlin/Transformers4Rec/blob/main/ci/test_integration.sh). + +Performance comparison of diffrerent training modes
+ +These experiments used a machine with 2 Tesla V100-SXM2-32GB-LS GPUs. \ No newline at end of file diff --git a/docs/source/toc.yaml b/docs/source/toc.yaml index 2b691e0180..6a7e09b110 100644 --- a/docs/source/toc.yaml +++ b/docs/source/toc.yaml @@ -8,6 +8,7 @@ subtrees: - file: model_definition - file: training_eval - file: pipeline + - file: multi_gpu_train - file: examples/index title: Example Notebooks subtrees: