From 4b945a168fa98a8a7a9105bd37dd0c6f75efaef4 Mon Sep 17 00:00:00 2001 From: Philipp Schmid <32632186+philschmid@users.noreply.github.com> Date: Mon, 18 Mar 2024 22:54:10 +0100 Subject: [PATCH] Dgx (#1915) * v1 * cta * Update train-dgx-cloud.md Co-authored-by: Jeff Boudier <74218629+jeffboudier@users.noreply.github.com> * Update train-dgx-cloud.md Co-authored-by: Jeff Boudier <74218629+jeffboudier@users.noreply.github.com> * Update train-dgx-cloud.md Co-authored-by: Jeff Boudier <74218629+jeffboudier@users.noreply.github.com> * Update train-dgx-cloud.md Co-authored-by: Jeff Boudier <74218629+jeffboudier@users.noreply.github.com> * Update train-dgx-cloud.md Co-authored-by: Jeff Boudier <74218629+jeffboudier@users.noreply.github.com> --------- Co-authored-by: Jeff Boudier <74218629+jeffboudier@users.noreply.github.com> --- _blog.yml | 12 +++ assets/train-dgx-cloud/thumbnail.jpg | Bin 0 -> 32104 bytes train-dgx-cloud.md | 137 +++++++++++++++++++++++++++ 3 files changed, 149 insertions(+) create mode 100644 assets/train-dgx-cloud/thumbnail.jpg create mode 100644 train-dgx-cloud.md diff --git a/_blog.yml b/_blog.yml index 9f37bdce52..ad27645838 100644 --- a/_blog.yml +++ b/_blog.yml @@ -3651,3 +3651,15 @@ - quantization - transformers - diffusers + +- local: train-dgx-cloud + title: "Easily Train Models with H100 GPUs on NVIDIA DGX Cloud" + author: philschmid + thumbnail: /blog/assets/train-dgx-cloud/thumbnail.jpg + date: March 18, 2024 + tags: + - partnerships + - hardware + - nvidia + - llm + - training \ No newline at end of file diff --git a/assets/train-dgx-cloud/thumbnail.jpg b/assets/train-dgx-cloud/thumbnail.jpg new file mode 100644 index 0000000000000000000000000000000000000000..4c6f2dcbe76899b0ca10c7baca4ebc9e39f0be50 GIT binary patch literal 32104 zcmeFZby!?Wwm00kYjAgm1Of!t;NG}gtB> zj_zq`Q&9g~0YAF|SctGZ2>vioQ~+o!C>SiLpS=Jsh=I^BP`}yw#|It(77h^x`nR;- za{fq$++kp$;ozYVel7!0VIY~9Fqn|UgRj4E{G0qY0{=$f-w6C0fxjT|{PZ9s@~^@I zqcc`~^j{?$aC3bt;Qz0Zj`1Efh~!@-95B>B%r5e;l8!nmFoXE75)PQ?TKv5EuaXX9 zQMOD}^)HidTf9o-@V}D|y;K+?-}brbaBi33|8VW;=?pV)Py7G9!Z$klTe*Lq^xKo) z;NbH1N0Q|eg{6M<`7560#TIJt!YNG!p@;u}=KhB%DBZDboS4!G#L9nUt0k|=w0y?l z_x_Uzv~M3oq`Lm#`uz^odH5pJPHyj@`iL!7wWyWjtH0?{#e~o1MqQbFN45<|`ktm? z;=sEf^DC^;+9B>t>-Mfsm0EYJZo4FJo0zCK|0M4pKEkUNbiC@T@0gR2(7WQ(sNlY< z6sWn&{R!}#v4VfoQj~OJnj+BdS{>^DTX86$t}ETKrSjSNpM=q$A7n^n~GE-J?X=!{SHlp8#%Z-Gc_fnKxu)Llw_=19Gj2nqy}u z+{8zWXD)iCFD@!dYFOPW6V6Bv_t1W~| z`lM}k=BB(Z#vZKLw)d=qB2R~^I*|DZc&#()@#y>&!5<gbc9bG-Kx%ZfbSiSNqtS}z&zF|a=n+U22k50=OY z99R4C5LS*uq%MW}Y%m4Sh0341y~g_4I5lKuX<0Yv{d$W~v`Ar_`m6FV0026#;aFjI zGJ9q9PugTnrfV4fE)5~O{xs}q0=3O(!@%&(G1*UmE}A?9QLxBg*^J+{kI0XI(f@eC z%sAkho$vV*(AoUDaCYAR@5KD42ebXv&qZ}tI}rs>YPEZr27dgdh2`Z~3dSkIB3O*Ut)O&~USf;xgyDnsAu+*}x1@49HLb6@%&pbG%k`l=?MC`PwQ~hXG$M?|A0n z6Pg=@N(`3970h-YvRe5)<|XVu)h1>W+>`TVljOCC#UHZ_NU4~6sN8)BL;~!97_AFc zYZ5-xaLL-N21Pt9eMcAtNDV!W$=UlFV;X!>OHQ! z`__`w=fM`;*tK}A5ugWEkV3Rg_WZCQFx;mQa$8?=rj&dUq-_MC1e48Z-U^>FdudvAZuy+Ctug6f|I_++X?!#PfCee3tVP41vw zIl)M++&gswOL$wb_9=y4udLS6KOI#sY`xD;6w)XX-Tr4IR6e_wgWy-IknruU5fnXw zPTs#4l3O}${|OK_&Li`Hk!q8R<)P6rEWd6V*?E!CjNHJ*Z>MTR)Cz!_czCRR(>ANT z(J&_0TADIhKNs0=)05f;=?1>@#m1)d&g%^!&cn|`J-!vezv}u=G>oyA^J=65+UoVq zHskT(`iB_1g1j0(0dKjK;GlF{GReGsUTVv&?6c@+{r1n(8f0~sHJwaab7|$W3$lk6 zuWe)mgI0(qIwXrgCZ?9hwxg|!0ZV#F?NBsxIuAv89fe3TzqyzFf8bu|9EP${el-X| zO79gRm>6@zo+T4}n+nJR06eW5h8z)VAY4#+Yx%uYg8+b$#Wu~Cl(%NONu|N9tx+9` zBjO^zO8HMT6myg3xKdW&k*V3`9Ch7+*h%ASWh73En#Rs5t6Kp+rjL`lpNoobXRa-n zOsBuDeA(lAom$WC>Gkd92g!STr?I_+oZt_D(WK*l3)G|!Fnk4oijCho z_gdSJb@sC=q`7Tme3pwZ{YB6}BH`!PbA@i>Z$OpR9><>?Wpq%jv^J!SFAfe`doUE} z8<^UkX8Vjw1${bJGH{YVpS#2I5v`1z?An?OyAya8Xw9I#RVP{Zn-Y&qvdYHzFGjDl z-eidXFBZVmUYlMis1L@xBhwsh@%fqqX{L#fscRSu_s!`c5`G{Xm%#9NXU%d6`w&BZ zO(~V5JFbFP*Tva2oyE+xkxQY>RR91P+4uE>$~7zZ6+-i;%y|;eTXb3YUzGeq(#*6? zsbE%D2}7FqL835XBhT#n^rK;c!p+y&6K-MVklKdQXzIGQ8p{FT;QpB%(OGy|82fLM z{@Dclg2!z1!>_7f3QtRzvzg`e9(SIfQma<(H$5m;ysUf`N=N4Ot}nflzTSE2Xu97b z-6+s^9Uf)eFKW#Zb_@iM_8?SNFE+5@*$?x<9FT=X6$rOZ%)p&n^p{mP_4GUn(PDX4 zJT&Ztw(16TgpWS`s`DSya9dLAM2Ivn(f{GNnB?s8))bIQ+q?e01ne*|2W8eTX=I&( zdX{V(JkPv4-v}WJ4aIiTz;Fp<^ilIhixX~FB5^Z4rAZT|2 ze#jMWIcP$%pT*w%#ejcILa!FhR<$R|Y2u#dNc`8=nNII~Em~XF-wQ&D;xiKO`Xx=z zmkTQC3yp>0{n1rL=T>Gk%D`)s+vJ1mktK2UgZeo(@6-B6Sb`?vXfY*)f|PgH8z@|9 zSb{FM^{obvO4tWD8~a?hVqhO%`^wLE1q=9$x&zwvvjsMn*NZ}a^Ui?61rvX`@q~6dZjs(X4VaL z$4|ld)xXBkAGJ6eN~u~}n}_1)tAEFpJm+^2)h#H+uhNOkH~@J>FCPk)GG!<_Y6y4X zRIdEriN#(_DvS6f1zT&b*JW48RQFof@MZr>;0DN-TPtHH{-R*;O7H``{W7cM4EgQL z`9{~T1Nd*QZ^@G@onsPp(*w2F1O#VAcXlq+5)bxHn~yT@AM)=D);wz#&V(^uRkznB z`aD!Cn;Dz>J_={<4^WK7&v;u7WbQ5eX}MROJ_JXcyF+B>1$^GKv!r2Q{w|w$)B=cj+0j zETF$k(?p_*m(=Gk)N1jG?}=bvLe<)n1bG?M^{YDUqOz<=|G%Z+vvP6pbGKJzKkd(m zS9LvY@}bRWe>ziYBaNFmH4u{fjO!0Tl_wp~wmz}WUuk@ybJN_n~$=_7{nc}}YIXHAu(ske!e5^ot zyWR#f7gB&O++Afk)AM0|45Mc;1~kjkSS2L_6uM) zqh+Vo)MzfZ#clP|ylL74M=S4(I*-;m83&#MR>nw}cP`-it#i}sx`R*X*Cys^2a;|B zIOM<79srQp!3J{vVwpBLJ=rzi+gQ{yx}nmde00m*@4GiuNABwtB<_Jba!M_MuK<8G zw^#5_0KcHYTIQnq-1ps`$}>Snr=)-|0O0C&-kbg+*+556FSe7D?t zja{CX*LPR{r+$Era1nzb0B9>(i94!N7I>cY`R?MoHJ~5CyOLvsKz^gi_pO?decZUX z)*;l>!QPWFYqOaJBooaG-2vDR|KKQ6?Jm{F(KA)oP+WuHq zz@q3)l+pB-&x8T)2k(#PM+^+dZoZT5Un>kCeulr9`qilC&ig0#lbDv5{oe$QICY5z zzIhkyH&j=awXxEk#>z{QSv*@uxsek{b*z(jWE|`@EDyG{zvUmcTbd2@9VakHy-TK# zssEnY>*?D8FG+tMbN+t5`P3UDiaula-tz~C-}9NxPT@m}o%_RKUhHe~qFFN#sKakb z9o+S+8GoW7AXQ%>z}$7hlKlo!ugl?ILBhlJK6GEvuLfLg*%j_OAV4~Z8}X~(;ekvT zT}YvL2YsQIm+tRgymC}baTn%$$wKvXPnRvHq8_R0DfLYHFl{&3i0ghwN|c#u(VOG{ zH_d44iB49(JOco*^9TFv>x54j#-z48a^sHQL_^xXWxr$Id8PK5I-bsZKhx?Fen~w; z#A-NU6!_l5A^pA^Hb+7y=4{C!eOtd}V(l4)-JUQo>XO`(UzpSLzNNJOTV{P5`Y9HC{4f2FVu7Y7q1e~6%-V$n)|n9u^)JLN(vMxj*{0i<^V<)dON4WZ~H z-+m&*wnqaD4nSHLkVs9vVOof-W@&EX53sA*5tA!jK+@dR`b9KaM=l8TC%sTd?XgJv zwl*xbYCP|Zf6F9q_2o%F(CF@85c{yA{{5Ho|0B}oHuZ(x|ET1Dr;x>_1p&o>hw*>H z1OJXp!`k0aWR?YBd+LKKk}Z-kn5jTq#$ zpNH*Ue?H156EDU4)Re#ANK;K`O+Jrr_4uJ9`c!#?t~)Utm_w?8LLRWL0w;v zj#TPjI0}h_TVmy~Agp7>_#7uw34(pHX1q5QXqSHV=6XeAR~xp1bR(rRDixAK(sbRO z3X(Znm%%IFo=9VU0&GLPpyu_+XUyiUKm_)?N zjw_5YM@sRh<;G^x*&?_KCepz3`MTUkw9>}}g9r`h__$A_XBH_(5@}Zb$pvd-adl}p zU{pk*SAKK)_F)vB%->?4OYbgQ^|b{@vq43!&a2C4i{)|Zf13c0d&egX`h zpX?Sil$;reN(8m;SQ}Pdd%2f!>Wem$@=Rm|x={4d2KC*&Qav1PiF)5&#im#~+? zO-6-!gAoW^(J8L80)>w@I-{45htAE{cV}Cw&()EG)GEi{GPyYIgvtnW!GNMG+zw_- zYnJsXWHO{kuLL7;VFP6V=+0JilW8hAaAZ%%H`)VwuI6?u#ZqZ|Z{5|${4bZ-iAi!4 z5`yUy;`l{n)ypZCR6g?sD5T)_UOA|TQEW;gP|#qCQbfPi!FB8(e4C~b^W+rr*|RVX zZ@N3dvoBvfl8wK*K+1MsPZ8PH6jP#df6vKE_e?S(q#Z?PaehbG;wZ(ST=|QYg%@NL zvyVZ~K7!?<1M|Q#hOuHSGpw)tKj?~E5+%g@Z(_g5i{rEP-08UW^7?FUEdz95l%Y;w zvAXVWiv0;Nu37XXX-Lk9lY-|QQXxT?^0h4mDdH7u6tixIpN6V#+57^_>Han9c^oj48YHF9AHm|Lx*s&f2RHSbF1Ll?8 zIlhwCc$bQFvFq7jt)9QYTwJ#t{AmPYy z&LsC3#%OTG+F>((+Gd5pzTYkLLCzK(6^G>n*8*J6vOKG_!~h zTCDv>eszxvM^HPj26pzyHyGsAiz!VFEi|fH>K6YY zB<2OLxnaUD>v{)VG1Wx5jrOG+#nQUr&eLf|Q7vs6sjqX|c%et>93pG+m z4jg@sxr@|@gXj5HH7U7W7swnHdCD}(EvQ-WIZMUf>%s>61G?VNd2!cs^GS+GQv?MN zMHh34Qoxf7Bf|%6#6It?>1+n$m)wbqM$|OyfJcwZoNjJ6_SLo!6q@rRABk zl9st13wizH3Fq5P;UdLKR12d_hHZP+cLLj`0(4udN{kijgE+i5c}lNB3TO6aHdL`c zGicpn)sYlA+LkkmMW`i)Df3|3y4!ov?7uB4qCJ!L{^X}=`}~>x(+$NWXRI(%eC?Hd3hpv+@!Uc>-Bwi;$vmDgjpr!~;7uao~Ju9kGNgQ3Pp< z6_?axyESEDT!e2fIdX=M`UgwW)imud4+7P-%Xm2r06q=N2=760~DJr8P`yq#u7+9GuMkwdhx%+x&FD3|NtAVCae#M-v ze}x`GpJ!gIN;$ieOp}Xst;Q?OFIo=@Ws9d~pGuqVbUU1_fLm%ePa9N++nz62>{soS)!8t8 z@gJPWotmkiAZxC;ah`JOJTv1v>XuMl9!BnP+GcsmSrA`;59kaJ%_*KKoy8Xi`}T zP#Mf93q@ffGYP;9Tr5szl80}W7 zvzbxgSk|?w%SIxA%`<{m5J}8Jk#D<^Hp_SFURZB52yb%g2Y!Fd>fXDP-a6Q5dBwWC+TYh6hHoF#{`xmtbfoXaxk zSTNuu@S%G}Yg%}*-Yt|Wo}OK+9DHOomTP?Zm96}MeV9VQuqz#n$Il|{d#nEA_qB?$ zjD39r1;gUfrbd$>s;?S3s@hW>%=q~0+`O`N433-PvFNXjQEL{(uz3S*_0z-es4e;r ze$;R-IeIudr$Y90Ao6d!uRl5`4he7W=8j{T1vmM@bwjsFecWC9aP-)a7ICWagN)+K zPXMu@ndR*6YzfO-)h|^wj+&BbaM@y!(0tVWHk5S_L(3lcEtiBS`a)Dc;OG%Fgglof zIc6B0juXVO;=||j$z&yGKwR*+lx0MO7gz%>CxXQAI(=Sflx_OF7g)Vkd=rXaK;_%T z>Nw&R`>#i*Iq6w{U#~Cvi^4*qSGh!L+WA{EjGO5N<9ydSGD_O>9IZVb+-}8WS?`Bzbn5jz*UZk4Yl!M zAQvP-X?4{v!Wl|NarFF<&-+f>>O)Q<_!AA`SesO*;39s5iry<&kvRpseG zMEgRSvuK$gP13l4P(;p_f)a`33pIdMAKWpBN^_~ZutyTF+b3a>s)>3Cn+~v|o3Zyr zry7Pi_BH(!pC;%AdY(!y!<$X+pN_Ld_F+7;uco1#b>2^>&U1DviS`1^^Va0~A57)O zbSgJm)J0uv?HM;hfn5C(fs|PJXxE1ey@r+v>rSux>wbJsx-J@Ze!JYC=a(j+lylw$EpQmZzkp$&V;ki!&<|p7q=iy$HD=7!#3EvOg z&Tky+C61X=9yBI9!pO0b9caJz*sm`Yh`*U#70mnuke;e-M1pLfXaMYV3r>elw6pO$ z;t03;{r~_xz`SxnD^GGjERV(K#rxA``Oti!oeP%ozBDyO9acOA18Ljp&&w`N2OJVv zBGF1wVN>y+gCAzyG8X>mqH>Aa$Fk>i+iIfGl1<0lQl_{1`2y4fu6rOcnVO{hVmXyQ zS)N2ABnq^5gCb&LQg6_24BmjSNx})cKiU@MZ9Pr%PDHXW)?~@N{aBH_rc@LiL9#sy zDpj{c8x}#*r0sQ&BpR@6Zwwi>mI=r5kagt&UpK@)r((+{1Pzx0#of_@~1E+GQfLKr`>lHe_Ohh)pYbCERX@Q9%&3mWGb51U<# z?Q@UYM)u++mWw!*c`4;$)G!Nl68U&a7$s|YnX09pBLmWVXU5?u9)?V4&qXuCq)5y7 zCDh~0h0G4_kH%qksZ(bhAK-zz`mE6xslQS(Cd?sUl|t)`W@R6sSJ9s*HLii;E4~Ri zWVh(Esx?zg*bPU=ZK^H&2@q#R4`8bJ3BV8QRwA8wKr5Kc>JK2q+Tq4t_BA^bK)Yaiou`F*F#7E4z<3tp82%i_pY%7zw??eZKX9}p~m?F>?Qg$^d!ndFd= zWW#lDjm&5CGE-dsqY!3;HEG4%~ zOXxr^O_C}R=G$ogh|VydCv_}T2G(KYB#uhzmS4hR(E3fC;XaNKB?ZoS@UuiC?n^~$ zfZ5&-%s+)fCZFEKLwK)N9PcS2FU70o*Ez2O z@(1FC#~}FXI8WxC5fZx8T-!xkDr=SuDFcDoMg9_v<^@Qa`o%?xADvW>ou!sAWu+-L zMSF?$KQESaL-oBMG6QU`RBLMl^Mo|ig;2^cDhq5be^p75Nh|201z&#Fn2Kj&q?Dn- zbcEG8)tpg_BJvcGvl1}ghSk&M3y91&a5iK|O0riVrkM3?&={w-EZ zTV$PC?BFQb@GcZh`z$EOqtIT%pMWFtayej9JuX9<44=$kFl%2vq6{#+3o;wcqI`bw ziLM|`v99_jV7&{n5`fG|s4W3>3m?A2UvH~wM%1%DXGkRFvr5R)BskI>(X$Gb3VlAm z$GwU!o(H=`0$-*vdX~#W^RL~Fa>HUGQUn#n!2Pi#L07OYW^VG1?81I2el=cHtyJ#j zXpE>co|l)Yo|IG1D5VoF)Cv_2W9#1qRU10Zj@On%&-fZAkK$uYQF2P{Ik7~<(-2F) zG2wVq8Mg4!B?_>}!73iuO(Zri)yA)yy#N@rAw?Q*V!h?y4igrGrFxW6`QAp2Y>Frft0{R-CCT%(%tOx!Hlr zuQpVF0$NR84UpnByWkURp2;XAC8SYjo$@U#8!B3tk_k8g^*5|eNnG;YWlB^^2&h%e zy`$%kC+W#Um0Dslmw2%0&wu4osI?FD{8EQ>d5lGr&|NLoU*VGB;52t6uWro7--26S ze^*jBXy2W1>OKkFMu#rcH$qV;os4P<`B0VaVN{Nu|K{p%IK1GnHPL%j3X!gjsXLFHb# zYMMQ*K^})zeApb*MXt!DA78Db1@*h?V6#fLRxvUni)@e|xQZ_=b)|4{(&0R%K(?`I zKLY0-7wGcSdVRDbYY$!yZ%x_LihZnWNH8!;XJ!5gxPC{v@zHAS%yL3n!=xXX;F_fI zV#lqe#`}{CFK$In#nw4FCRL{Pr}4(PNg;4j93y9}g{gAU zMMil=CTPX8eERHmyx6bA7P5*Ga^SzjWUsW6<~-Joy;3%y=ReWh=2@?1iF3SO&dmIn9tsiaLF3oq8bkBNZI_-*x`6F(iIA5Ig1RM1&6dKtNOy%%Nzi=W zWIjz_X|n^SuN(9^;&5UKwC*v*mGb&Fbju^I9AD>x75hum>I@^b9d_q>k zpYKhMAW>p}mef(-EuMKLXLnG0BR9{Z>>na=kef2^+U8ZoOu1cltBx>5pgfWX&$#Ki@ zd3i=WFG?_r&+Kb5<*fD*v=8yhHh6*8p8`=(uPYBq{<0zVP~hAdk+Y?g)8@mD59iL) z3tby?%c+?b3KyugjN9%udrg?s5HQN~=JyWle9zVKTL;#gwA%$r`J~{U=vNYcC+Go< zOPk7t<~9L8u#&|mTwpzl1iIkGD9nFCGmMDc5NE#4JWR+qvNdxfH2OzA0wiSue8$dZCrRG4Op=+?8}&~hkM3t&`vw#|56Ag!0vu5` zg8J%}dGZP|W?Tm9a4-Yn1eg*nI`~KsdQR>pH_~O_;U{4D-~lg@>20B04Vh{r-)wFy zQrN|y19y*{qXiJi&Rw8qW&f_iJ3Glee7aTji1PvT+GIdhiSM@P!NdJOpuznMG;U&S z5-3(ij~Z=hAJdtJFjO;Q7P#+$*$PfnlMf>nT-GA6Fd$!pbf6+TO}0fkQ8FLUTV1kN zz99DADIvs}!ZFzw60zNq%jCks42>&9M+A03N!1TM54RICAdeMOJolNcg+cim9+y3X z^EK$EGdHL!Mfb9<)InTZK^h`xrQSa1uVohwSGr2{kI;Tt6LP@Ni1UB*jlxWvW?-R5+vrkdyi}jVscJMv`^s}vgtBGdZzV&0?A_G4 z=SP}lOAG0q=5|f_WhcNG`Vi;#80q(;H*VQwI(dl2N(hE6`rpdHjDHD0M$5`{ z#Y)$+J*t`PH?j*ITiX@F6)yxP0s-u_loL75VI8=#m(z?*p?M*#Hwu??p!}QS(Y~h6 zOYO%NJ)u-=wH(M2+%>_GZdO_0-ADLN*EgnR--?Bs1*vIab^vbNPIet_Cl{}KdxR#I z%vddTG(vESNJJr%oK(jJ2lki*?W{9jhE^T#$b~LuV+z#5iJK77SpZ}4{K!+xfjVQ>%0R>_qF&BS?L(v#uG#` z6764Id^8KkmmZ&0v?#;m-|VO`*DXK4bD+E*f9D>yP5-0?;8}2=($FuK=p<{3EY-}yo?qI~B8~=!{X*xZMYUON0)!TiTp^K?L z{j7Ca&Q%zv>5#h^N3;*|Vbk1EjkSAd;=@WcJVw|KfJDn#*}BTrFCNkPJQ&xO3wR~x zDXZr3Ga0sc+p5G}gT4zfUwZMvr3=J+R=+NbUR3oN)UI6ovF3HM)|b%lS(UZ(T`!V+ zAk0{-w`zNEZsDb68Y0;IdKt)pucKHWiJ+sAA2annD{hzZhzOCT@>a>F>6P@`p<|L9 zLaAJtGi0FZurl7p(nVhYGC?3(0h!1VFWWLAw%%nvHSOWYck6tk89z)rvMOsGL12RS zJtMB2m>Q8%9k&Hl=y)0Ao&@BV@9cj9W>h6GX%!p622S3&dh z7`Kahp!vm}httRqRe@&qfwHCciMglci1+z;@3y9A9+xmGXH(f-8CZ^k@%M^KM#V!( z0Hg0xuaw~+i_-i*8ecq7=eNe+ngh%M%MnZO$|$F%m+A`K4{NY;<;=9Sk;in@tCkjK zw~TP@kUHm;Wp#!c6m23+ozs!pjE%6w>t>%CO$ zi>6rWPct7Na;0UXlwelu<`{^00vpYk>ndikQpHn#K*_f{ z)~TBgzU8Iz)brGr>&)gT$vg_Wc2ETDRxS&F``ZS{8nOZEHcv5hZP2z=bblXWy4%xVD}Wz#Sp1C$;FGcqB| zOtb8kDMNEIoLo8(_~Zo_UA~u38TU7UFUBP(r$(6Mofh(gL;!~QcRqd)y(wBk6t_ea z69aNV*&>MI#Y@N0>~o~)sPr-IOJ6SYBgPC9N#@0BuO#HlA|d!)vl9$1%B9k*}i#UU?go_UEH$jXUtbfOQDF+YS5xI$B^)M@f*<( z1>P+>qBCi$erD+e07I0qZYI)_Nj6@2btyrVrJDlUYU?O4c zBvaa3ARC)nHFShb6spS(PS#+oI~lxC)yS6-fxwPSozpr>D-(7pGh@#fv{2CzwsdTR6y2S>qsdZX_e@3F(75VqJyr8*5W@nY)DXqW zH=mz?68`4~3A&til_Y^2xR0$9Bq$g8)Fzy%5&?Mh=ej%2Wk(v?%wzWGu9kztuUDn{ z>eCWlq)oT!NNNzR&xG%hdg5dSgGt38OHzRY|G$hfiXh0zWATqu)94Qm(a4wI&-hSv zf(e!P-eioN71z`E+8XAzzb%_6XdgD7j`QBVChuncYa|uH=@1%sr0^_3>{tVo8akPOtX8O-W5f&+K_SSTc;W%p#&NHC#)n2jG+a&ZWyONUn=OLt%Kn zXJ0pYsi`pY)3DiVOfb?a)ia{76Y^f0g_#JQh!vG|)w6S}q+>dt`-buxS=;*J^55)-_i<+=t7F}? zWhLPp@pIJDqhjw@m>P)!g5oV@`;qZH!Q5_zwsG~4!!$mdnQ%v*Sm|m&AbPF^rmnV3 z?MuAz+0z;|9O!^vViNQq3*Gg3)v>XJ*Gp{@q$C4%e4DmB=|0`PU~HLbCn;-of((_x zr>Ie2hUQViYtN!_zPSf8-=`<##J&T58z>b3s=}ujtUz$SVS%QBxF45I&3t|-E=h_n z%O^HQIjF8Q%*@%{wJ+h_<#N7|Suph6T`&&?i3NmoK>0|D-X zl3;BlSM!@5dgKyE1!1btE{6Vcg-A942*N=d<%{xI1sSPuzhy!zz)ZSse?>8;i;A>>U$zE9KLU zVv{(D9gX&0t+$)b_vI_=70(_dKdjyEN;?e@gg>vfA73m4OzbzBHrw8R<$anz{c+GZ z70sM2BP4=S*J3M%)UEesQTQHH?vE8cG>ImgEt8j+&6HYeoS?<-CdHUg>gta!+HlEa#$?_3o$dxbHtkdM_Gg zLRVg6S+!;w9Ac5DFeU36C*d~LY;!^-2p!;$tD<&{QG%>y;j#{LW!3GyQLZ=De^Y4c zCx8l3m?oC0Gg%#&kdwfTWr=C7_KqE$T4G>bOWYC`Ri?2$6qGU8PUGx__u=cIH1%chM9?k;rUH`Qk5^MEKDbalG!MDit%gyyho<5>cs>FNJ!{ z3#!4Ri!OaUJy}UF1!0K)u-e}HcL2C?K4Nc#GS;k|mSy+aC~r>L`R-HgVKot;c^_5}3ee0Nv=daM`sfIW9MO-0eL5A> zyyvB1s1P6K8;N*!qO8lN0I(G38q`Jd;)=_n3$b})N(R2-wv(Ol*CcqKN@=ClM^9ZX z%9AU)WI5Kf?&jw2#8@!v$N;+t31S*?bPPI}@QOd}8LwRud5Rmt9mquvd;|^l(DsB+ z{&wbxT4GgCA`8zAfbu6mA#vcb?jDNR60iZ(5X+guM#ad;IwDTL?@Ov@+0G5GwQ1VV z&b6wc=FZS0Sfr&Ek*-UkMSpoejnM zEiffPz8zt&X0b@}=A((YdE+PXFg|3BxGZ1zg-j<|I2H6(d1%+M3eQ&&givsx(RFGCYn*S5Q@2r#$B0&a*6^oJO5x#yob5wV5#lcP%){7T&qDsUwNSAt{}>&k3pMw&wh zz23xi+U#}w1TdGbxvoP4)v}TTWq6>h(qQ9b18%Sz?+!PoP-IO%|K}*(~S=DTJ`?>*WAWiNT zC<2?9rF}5FACN6v9v8y086TY*m2_lC@DmVKs-w)V(xttC0$E$3CP~UUU`kKdXqx<1 zLZ|4;uz13gFD>DC;Fck?8s~ikW4lVGkz49dfXk=0St&6hXlp!~bh zifb&}US{cyLF!ZEDtUXwYKm7>g6dv*nh2MrsXHr|8h&RZ^EoMkb~){6pqw|fom z(SHIYwPvzAsZ7z@a8Awh*>c-~9~=;9ilRot>0Tef73!*5UxuKv>J$h{my8#-kHlQX zPs8V~>em0Yjhydmj17^Q<0m!pg@_rKLk@YF&pC@(v#Dc}E@Kt5!OiH@QVbra;b@ha zoZqjSNl|8s&hV^4F%Ric5kQW-kw7<+=u+M~ay)G*e1}EIu@>5abJ4eyIE^J3{nYUB zts4|tM(HoqQ1NnNyt2?EDeI8HXFj6^E{URu2t|C@fOIX2ZNHALPlCUcL}lI;Gtx&#zxeU*&6A0u^ z1knoOlyLE-MaxTxVMr4>sl9sxoyp-`2)|R)i2F=UtdMlNlwi*1=y;&E9UE;y^3NNe z?buq?oz5#xR=tBO&(rPBW};m1d4F{7xQv1Eb5!#9Y)du>{zNk;=@@pPeSZHa?tg zIA*%Tjug*{V#nNlhX(89gLQz@%#j5>^rJ(9ETIznaSuyYZ4*xWzuVzUhj1 z-jN?Jg|-j09g7>cyegoq6f+nWKf<%fxk&JiZX%1a`$bWdlk%*(q6j^37On(Kg(_7r zBuYF!P4q|Jk~kbz)pJOa%uz`p?Sl0Pk3hz#*4=kYTtFRA}W0-0-B4qLmO4>xyFv*$i2c2U$ z>};jEbT<@i3UOZLJUkGOw8H7G;_Hbce_5JMSu#l{2yLSsl}2Jb z+mw-%W~-RdCz*+Ejexhy5vuwuPJ5Fbakg00jnsEKcA3eu9&dsy18XGZ$aXir!zE*0%cE>Nu~I)C6OEbYJLKPn7{S9($K+QR8osKT z*T~5&+s&`h3hc(MMBNqu)hp~u6ez`Qb#amQJtQb3IjZn0Ek+3U;j9-;RBozA|1g zhJRg`nO-?7^TI((a85|9GwT>dU<>i2Kb9N%vu|8&ja0sSNLn7r4JRa?G0?d>&)37X zUZ8DQh{POdUb%6)vU{!ff&rVcf*IeKx9{4{e2p&T5${$(-118}{Tk}c%chj~>T;!I zn?MlT^sz%>J!G0DA)mH|cJT@(sHJ9MEL@fgqc$m5^QU9)>*9-M;P=r0szKtrm{|@K zyS4|!ljzjOH8kAq7M$N7ZBkyVe&kB`CB#b2NZZLJ{r?L4>ZrJ$W!*`F1eXB8b#Qlg z7%a%(?ry;b2?PnjCAho0>p&nl!QCN1aCe8?$?x3v-dg9}yWW3$t-X8i-PP6ARn=d8 z&2uX$eAo0cK85n}=yr38*rqpfIt-CY>7yO;&WuBjzl^=^L_og^o(ot2lkGemTdp@w zNnVj()j*+&c3s7cF= zxvy#(8yIf&*>%`M?k9(W%EnOAepeHrT7(EN(gB=yEaB5y6{r7s8-`0Jc3J%v@1~SD z08|LFkJDfQU%6z1+kqU7{VlRcS%p+$C?D5Sz zdO6+7%Y-H}{&VE~v7PW*aYPriEHhCfNmO}v+nYHIZJ~4t$Wd~imdsDhY{$MX4&WU| zINLJl&Zy^k{ruVb*=dT^T&!??q}{Cg;hx{aJm(SWKdM;p`Uv;@Y3eadd=`8#&>|T7 z%S}={N7;=!88=c3gX3`|e&2qCU?oIsB6ob+MrBJ{6FnzjB)4Q+GD3!0N{8d|>Ong0 zQeh@{!p%L|NZ}JgqJ1s3RGc$yL(Pj7u738^k1y1(#dSASLL<0wfo)#Y>CFAsNy9S#1zDc6Zh(#Xn?hIR1X0AUSAXM zlDHUwEu%lfCFmxEYJLvTS$^ID7s<`G}>P3*& zK<4s;Vm7nECm%3J7fn49^?<%^w&L)AK-?y5Ro($wXveJ4H z$|xCaynWnoaCeWI+l8tz8t?ILLlQS`Iw=QNMAQ`ICq2`V#d|UPQqT*N;w}RmsR8t2 zCSkk$G*YH+^k6bswT}%&b3*d?y48xG64laQZ45+Tq>Bh~lWt5zhlgkhcfr<_#;#R; zj$oU_TQ@Yc4}SnkI?Y7RNY0IgvMTTla=n;dU3D=*IfXXJh$%}3uR@U2As?+wrqbn# zKSV{}{Tw}wFL1(cERM@#0=9FMil_QHQ1flQ)&aOYN{ZSW8R zd$dqfan<`xCI$W_fay|+V>8{Ent`xiirHF71Iv6^YfP30dwEyB@}$W3wya-I57&&% zmHa_5vY9^)KH;)#uKtHZGl`0VwQBSa_6A$Hpmif{dwW6+>l(^FGPt#@yM0<}%QpF3 zNp)yK%(SGIK$1I9_wm=_ZQsGE=TNT7f~f7L z30AL3C0mURj9eYRX;qwhaMRptQ0xqTk=CP6?2!_7f$_`2Dn-P(FV!(^zN7Cx`MZdTIpJF-`{0%Lcw;8 zP>aQ?;3d7at;8$VYzs{|5-MQ0IRT;OAo6K^&XV*Bgo-~T08zy-(5iG=d%!2UohKl) zscK@HRj6VvU3ENs!dgxoU7kWF^^I214*;L5F*`dxADhZng&kh9eDZs270qA1u}gXu zyGm#cjc%y+qv)kxAu~+NiVZPI6A8t}_~wZ$Kpyp{R_3}X83IQ- zAd2DVh0@8_3%T4U#T_-@(NHwXHRMMM%M_j-_kJ0m^C zCnVnsQ-k^~s>scuu9p%k?bD^Z^XV3%%r*Z%`${U0f8|EC(IJTcYhX-x* z?hxsA-mUdu=P8tk=1!8E2hYwJ=F@@C9a*efDBkD4J{CvZzg~OtSl-`!5o=spbv&1= zRx_JfGw$_TUryn9H$3cXP1y#q#lns>l;_>?X*z=CUu`WG(*oG;uZ_`$wCWZAKlC zWET(!ScCU)(i^6hkUyl}IpfRS9CJJxrn;`BmGu_zZPs=*BV!7Qq3}Y{uIvGfQ{RCM zKUY~E3w+#93*Xi?%XT8q8~V*eh<^TPpS?D3nH3`M;OZe)K0$-UK5r=j(S`<9Bz|lE z;sC0gvU37+WYe>fAHGS5ua!&(hT9Y1C)RK_6^qv=Ljnt%=d!T?i^EM1=&u#5ChQ%b zIt%Td77P~kPn%y9V?6}#4GR{baKMY1&1XKHGJAwp_SooY*d$;Bpf*Mvy3^%KI4D-0 z`B8TWX64%M9J^>`tl|Oe%*OY5%C*XIi<=K3ca5~zz!O;kH#`aLbXOBxWG;=c5Pd7_ z|9PHj+fp5Qo%R<0I5gBx70@OFhq>7=6R_NM5G|H1H5ginovtEt`MdBlVs-$u(dfou z@^e83Nm@;}pT?QvW=&gy;a`7+1O&qcN2l(t+*CS==&A^8@;P?&AdP#?@I*tHzJAkb z^m99`ly#7^)oL@po*)K=3A%mI_2Vb1ninUxB{dZJ14?S#JF$|{G zG@o5n?&<-V`9en8nl&Z1r%Ogj(5U=83O~K;l21s42mXUcw1qPAx}=I?Y90buhm~Lf zBjFcok1|(T5UPmB&8PR);SS}!H!eL+W(HhUlszUBOPs0-Vd4`?Jf*p>+dR!q0pc3V z=(3(b3T$-%^WZCiR}3fcF1NUz{>}qE6}L zHjRdWUZep3lp9t}w6iVxu}v467P(u;z@v1j7W7D?TX;Y#x{xfT(2{kgig@T589-y9 zNhLxp5aR$|HQed$<)&Y>TNq+iTkQt-s_wu2X!fmKWH#dZlO10eL`Kr;UeSk8SgsbB zp}cTqgfay~Y-#Sm6#xKk%k>J28;hN34vU#bXGgzAdB6MB4c$)Mm35 zwKVmma5=oAq3b($k6nha!9w$3aQ@7KC9YH~GDj&7_36=a z9A?_r@eO+awvp$hOGg1&La8OI~5640fK^XpYvQ+nfTM zttwTMMuzIrkT}nOh{=4>b&Ex+63tDz`xAd6KpK33?Q;wVp0tHMe7Sl2*{{e7XvT>7!il3!WN!m@kZ9Kn)^) zDRbVPU8bBS8jYzGMr!}UCo%H}kS-dH#Vx$$%Lke{jlJ6-yU$Yo9Y{8E;YUnx{{p)- z63svVfh~GJ<_BY1kii1He)ak_+*>3h#5eG;todK21ppj87B;)MGXgmUhYHvv7xNt+ zC6%h0Yb*{MC%17xT-PL*n1rOMOD(Q?jahtM_r^c;3a~?D{?aQ%L_gdJ(k02^hBD{x z%qM7O^&1{sgikLFuNs1Ny$QU_hZvR3SRr20Hb)jG)pV7@k>6*xaf@q{q3wAMHypQG z|7#PS%`<}$O4$|Su#xSX>aRw%eWxLJqJsX<#vR8{zL=aivDa13M?9zl(bJ9vmt}^y zk3>e_)6#?9@2Wd}Yne}2NLDurjt|l@%Z5{IV^?u1nyq^$w!g$-qPY;1IX^L0@u`t4 zU|;U!xbh{*+_2lr)(}&tnBCm3jY49}%;CZ`m^>?9=qhYDp5;M^D8gdcmoPriCE8*r zb3HsAu``6jgh^mJDU>`0B(vj0R`Gtd8MH}E)$E^g?4P96xq<5uYG}yg`a$*@LigBC zoB>WD53k*TKdwcgIG?=csco5!rnT`&?S65Z3yBSpH8j0RYOPAj$SQlWtg0Zo(EyDIthThq+vah0>+ z$$OxT(z(itxWK+NL(37jtA(8lM-sWRVg7mf%4_R-Z&Pz4^~;hkNN|`kWUn5&J}O32eT#ffEgqIA~T2I5z;@z+Xr!lk5CN^-?!0E6oF zU2n&9z`PmdSaCv1Twf-82CIwQQT>N?@&in%?r=^_)D~8kFNWWg91gRe;}XY#x+aHe zX2gdo*e_F}PhZxb`WP8YER5A7yKw!2x;&9~6lw={wKU^(hs+XP1kt+%`4%wdP)?Tf z3Suciu?q|(Bk|fzjMPtysJ#q6eNW?K2Nj3}_hNX>X!~N9MDt|T(wnf-S>wE|OA>|I zjw%+K)|j7+Nw&4MX*ULNTi`JVe~rQdyx7o9gI$|nS{`#ZVTFtrneyCeZ38Sjqo!AK zi(_|FGQvHXa9%`?V)5D^IJ8%|n(a1Y^BD+2-w1UTxW&4~_Mq3eo*5Z8-2xiKWN59p zPH~+-a<>=ptCu79ILp~*(2wTQCVpy@mS%DV)q%PagNqAoFCvK%j+KJ&3go5H7j#^MCfpGE-Q2~;vsRL644ZCo z;Ru{sS~DU6$I4t0-ln7z(AQNRm3Y7|67@8{;(nzmY_;hKcOdi=a>E<=s_nd~A3?-N zw@rQax(3>q#9Uz6*2l&mOGSw;rG;44uWUQ;5PwOEj^#V>zJqN2j;(Fl$^}Uz>YnVn zLXD8M5Ut!Cnx^!94V4zzt_XYR?N_n^G1&Q7{8+>Pk5=R3-#FA89#;e z5f9;S9VUfnpd4#VfcsijZ)b|IqBOeXj1!!XvSY-#^hDEVATy8mXa6&bSk15s)Wt#+VPea9g?EG$U~w%?b2Rg5~2uaUfSsg75*S z%?ig&Qkw_ynNJ7wh6LDB{2UowIk@To zfC?jafA_wmt*Zk_F%W_+2h8|6XFE6M-TGRp^*S@~v-U=fPOmy(q=(k5soDwOV)S@0 z>buc~XraFDxc$RBkoqmj8tT6!1Luz<5Trk1)sm)Gxzaok23JI|nSeSZQ1ciKL7REE zT-nWcF6~x-K+;y?R5d)nkz$UuJNTO_dsVydz{uz)m+_uCuImA@^oC-dZqHtUR|CLe zhLfZ*c$00D&2ym(jyMy&k-OV&;e-{NZjkXUdF&yHk0J7SLNn&OF_qsZdlg1#_xipZ z?E5jk6thw=3IIyRx4kd0+_T!A_RBj%yQ+RoqArA5CfrSC<{DL z3bdKEh>RL9tj->37#ytX4r@O6z8J!D$~4Y>^9?5@%p1*-GT89ojXqYU+@SnSsYn^1 zx8J=CrII9t_P-1mJH3!0nfo)C*Yh{iCGaER&<0sXP7M0_K(trh)VV6u1~pwOHs!Z5 zwzX*ugw5h)uU+fydxCMmYlHGbHAY5UV{t@EG?SF}q*iH`j~Jg$O00t2u%$Ke#iFD+9+jK%*1f$@&~ZRcd=mDgFQ;!9zg_<#QGV#Ie-(_B0HHGT?kGs zkI;+ILK6BA2{I!zYkPMWf>sLG}yS+j7lioIdUE<%4x}3z1oZ^ zJu3IblI3{UP1NYcaI@`ftw|(hM%k3AIp|_redVL@+zbap^&O9y3VCM(tU`j8!zF;1tY9H!~Mp-8w(qj>g$Z+ zra2*lWZ_37{K*JUTbI_dy6qwxIyK|uz)_t&BGb0@U;4BCXm6y zLqMtTn-2f^*}sO!Q2KWz3)Rvj8yUJN!bi9p5%O%2cS+X*?FiblUy=zj6GqBV*aubX zYDQ%#!;q0+d-UdMy(cd^pM`#>jmG@}SZ@(KqDvQt<$nqty>|5QgXm3%Oq2=ZGL!>An$dVOb*mMh z)UgEfT3zOs6;}WECKiERR(z>I7Z)n3WXV+}=Hs1|$ZiVTEH6hIFAMwCXsaEnIN1~- zM~&bKmy+|gUII3mH4fn!=fljHei3fxP=giJf`q+?{1~5pjeuXml}9$&`H5fHew94u z_fnqE@v`qAIK$2cm`4d2FvTlPA&nsmVO3Fo1XGhNWFS|o7uWNh|WZYWX3~-3c3FbVW z3&=6W3BudguY>yS{kxxS$~Uxxhvu?1^jV? z&YW7#n_=5dK(8&2G)L5pVsN4dH#&;R%hbovXL>%~psEJZd9_?R! zUxrTBAMi;Y#kR@~QC2!Yn;MZ248GZ{cV%%wKmM|SV^sD>`YjG~_K4L-QSf&s*Huo+$b43vyGHD&9 zMXV`&5f@yT(T045l@+q@Ov+_rDK)f*nb7pwF@aW`dUBSX1K*q0J zN%M{1irp2h2hDdceo%*RuTBPkU)d_gvcb@thlvWoS!AUIZ?Z9PSjtOrtdsz0Yj?j` zlE$DdPbC}U2>I`#$@&myZq8PChT{~tik-cjxzP#n;q%2XM7XB%&htaJ8qa?XbqHmt zvttaZ^s-_~BWMy}+F{i@U96O>IGV}#n$Yy53v71 zPGgLE|6({qJkJ#%5yv6Wll&0A6z=(U)b-`9NO5ye!AGjE($rDLXIMYpaHlExAQlRh zWmzQ(MJ{Eu`n^^HDW>FMOOg8z*%8%8YQ-^4hi(dSr>g6xd8O77AR}upAj<+q>$w0p z+Gw)46xe_gM@V%bEd$+QSsGlRlMU_q9{_YNi`tKBELA?8bcRfAxjRv3fgvK!9^UxO zE?CLYyT33A_Iu-ZlS4$|GUzOf*6Y3moFEa&HR5&_If+lx>}D=Dn5{L1(&8pc2Z6B8 z@uVWZO4&jx2>`X3Gm8c0?s;l73{KOlnSlpY?p4Sm!YxplY$kGRkGigf+ZioQc;Efz zyH9W1VzA)_?akk7gr|oS)zKuwSuzp@^Ld3wZ;UE`OYKl6*5krrt0ZixS~luP70O|t4#uokv1j;E1Mz0DKHU!U9b5AAcd%{5;W%& z8Atf4%K_u3E+|pjUdeGO*;gSQ)t8~v1L&PGYinYRYEe!aTL=eCUW8Bq_sQI6rBkE6 zhH~< zcMR^WD8q#dl$hxL3-U=gY&4iBVvQ+AS@i9M^a=?MOLn8rR4{P9mY8?6VS5s|^8R+B zWW=7HN*(9yP1^Q{ilSaOahO!eQDPW?SiNC)B{=;@%A8Y&9g*!_IDM;c6oG{u~sSLV;LXMO3l{Z6JANv z?AK1rH3ET_P!%9ipx!9kYE8>n2Wm!%*nlcLP(G_+iow=q+iF;RB9_zA1*Z&08$Hzk zP;a$8`FHL_Ql4&_7~O}*ED}xEfl&|`kO)baDY=^yh}=zhV(n~LZqSt($xxD*kLC?S zx9MX?RE)dVr=_C0zl17XpP*};Psiqm65~vN*o}hX5;Wk%l09@W_$Eopm>?)hjNfuyta%4GdF?lxOgwx_mGx)Q57!BB+S#R~0&15`S zJUl&6Aym|<*o3Yw^13>R_O}E9ze-OQvjQ?J$`v+3QJNc}l_F;*$3Ln{1#pNAnVIQS zY*TvcL7Xz9{WE_sb7OGIY(scmGnyA&Wb<>GVvbvs@TgGq&PneaERVHv={WNF`ntlV z3ES`C&Q&7hL&=Yox7>s&3(14J9hANj9ZY_S1k)pvy`(508w?^a@W9&%X2jkG+|q{K z^Koq+^b94TMAR1bywHS_FND9{mF5yBjL_r}{^`-eE9Ln?Jyxg`B?0U*HdM{Z3dBwhG18da$%)Qb7>^2NKwMKMw96Rl{@d|si#`00O+ z3Y$NF08D?UK1GkR=(L2-6)xT%K#$_f&0c4v>|?~IXyg7~?PUtd)Pd|GZX9 zwK(?dR&29(UdzxD3cci>olmW*Y9EC_cI?WlISmZF(hd$s{_Q4^0wfWD=aqyx&3t8Uh)&Is!zDQr z#*Gl?i7BO>;Rt;KN32q0aDRSEl?=O51j7pPzFgFS`ma~Xx|(!9=00>tspW!dk0eRP z&L`BrwRHHQ!(ywDP#=RT}yUW@F+Jkn^V%&RUNn7ip37iq;P?x zq0^t+gB#@YE?#F2Hy+TxHry<5IUWSZQt8oH`g=H9CIQ7gxU9fAGy&+%O;VaP>^K)W z6YR}5%yCET@M0|%<3lrh#M;wRhuxcTAVscirOCm9y&vwz-d~D?Oy7fU_QLD~#|2Vv z(^e3T0kUwDiI|Q#c{*cWX|njYbvS_6-No*4HY7n|X34BadgLWpVi5Yo5BSx?3XIi6 z;cMZFE?nIs(x;0(-%_#}ddU)HvL%S}9@HWPF+@geE0oS_a4e+o#HBOG3er?0{2>)` zn!O*iBz7NhT!fUWf=AFyoNG-1?^k^pm9cqI2(5{2GBH8rt(+>7W zrb@sjY9PHG!W#{p2AmqQmZ`j)K}|vHvT_-uHsT00#2am%EJ{WM!Nc9^lf0!As~S_4 z`zfd_OGSlHeLjxWG42`FWiFggluszSzb30K{5?6ZkZ$l?ADRT2?H_i_@% literal 0 HcmV?d00001 diff --git a/train-dgx-cloud.md b/train-dgx-cloud.md new file mode 100644 index 0000000000..ac131dc2d3 --- /dev/null +++ b/train-dgx-cloud.md @@ -0,0 +1,137 @@ +--- +title: "Easily Train Models with H100 GPUs on NVIDIA DGX Cloud" +thumbnail: /blog/assets/train-dgx-cloud/thumbnail.jpg +authors: +- user: philschmid +- user: jeffboudier +- user: rafaelpierrehf +- user: abhishek +--- + + +# Easily Train Models with H100 GPUs on NVIDIA DGX Cloud + +Today, we are thrilled to announce the launch of **Train on DGX Cloud**, a new service on the Hugging Face Hub, available to Enterprise Hub organizations. Train on DGX Cloud makes it easy to use open models with the accelerated compute infrastructure of NVIDIA DGX Cloud. Together, we built Train on DGX Cloud so that Enterprise Hub users can easily access the latest NVIDIA H100 Tensor Core GPUs, to fine-tune popular Generative AI models like Llama, Mistral, and Stable Diffusion, in just a few clicks within the [Hugging Face Hub](https://huggingface.co/models). + +
+ Thumbnail +
+ + + +## GPU Poor No More + +This new experience expands upon the [strategic partnership we announced last year](https://nvidianews.nvidia.com/news/nvidia-and-hugging-face-to-connect-millions-of-developers-to-generative-ai-supercomputing) to simplify the training and deployment of open Generative AI models on NVIDIA accelerated computing. One of the main problems developers and organizations face is the scarcity of GPU availability, and the time-consuming work of writing, testing, and debugging training scripts for AI models. Train with DGX Cloud offers an easy solution to these challenges, providing instant access to NVIDIA GPUs, starting with H100 on NVIDIA DGX Cloud. In addition, Train with DGX Cloud offers a simple no-code training job creation experience powered by Hugging Face AutoTrain and Hugging Face Spaces. + +[Enterprise Hub](https://huggingface.co/enterprise) organizations can give their teams instant access to powerful NVIDIA GPUs, only incurring charges per minute of compute instances used for their training jobs. + +_“Train on DGX Cloud is the easiest, fastest, most accessible way to train Generative AI models, combining instant access to powerful GPUs, pay-as-you-go, and no-code training,”_ says Abhishek Thakur, creator of Hugging Face AutoTrain. _“It will be a game changer for data scientists everywhere!”_ + +_"Today’s launch of Hugging Face Autotrain powered by DGX Cloud represents a noteworthy step toward simplifying AI model training,”_ said Alexis Bjorlin, vice president of DGX Cloud, NVIDIA. _“By integrating NVIDIA’s AI supercomputer in the cloud with Hugging Face’s user-friendly interface, we’re empowering organizations to accelerate their AI innovation."_ + + +## How it works + +Training Hugging Face models on NVIDIA DGX Cloud has never been easier. Below you will find a step-by-step tutorial to fine-tune Mistral 7B. + +_Note: You need access to an Organization with a [Hugging Face Enterprise](https://huggingface.co/enterprise) subscription to use Train on DGX Cloud_ + +You can find Train on DGX Cloud on the model page of supported Generative AI models. It currently supports the following model architectures: Llama, Falcon, Mistral, Mixtral, T5, Gemma, Stable Diffusion, and Stable Diffusion XL. + + +
+ Model Card +
+ + +Open the “Train” menu, and select “NVIDIA DGX Cloud” - this will open an interface where you can select your Enterprise Organization. + + +
+ Organization Selection +
+ + +Then, click on “Create new Space”. When using Train on DGX Cloud for the first time, the service will create a new Hugging Face Space within your Organization, so you can use AutoTrain to create training jobs that will be executed on NVIDIA DGX Cloud. When you want to create another training job later, you will automatically be redirected back to the existing AutoTrain Space. + +Once in the AutoTrain Space, you can create your training job by configuring the Hardware, Base Model, Task, and Training Parameters. + +
+ Create AutoTrain Job +
+ + + +For Hardware, you can select NVIDIA H100 GPUs, available in 1x, 2x, 4x and 8x instances, or L40S GPUs (coming soon). The training dataset must be directly uploaded in the “Upload Training File(s)” area. CSV and JSON files are currently supported. Make sure that the column mapping is correct following the example below. For Training Parameters, you can directly edit the JSON configuration on the right side, e.g., changing the number of epochs from 3 to 2. + +When everything is set up, you can start your training by clicking “Start Training”. AutoTrain will now validate your dataset, and ask you to confirm the training. + + +
+ Launched Training Job +
+ + + + +You can monitor your training by opening the “logs” of the Space. + + + +
+ Training Logs +
+ + + +After your training is complete, your fine-tuned model will be uploaded to a new private repository within your selected namespace on the Hugging Face Hub. + +Train on DGX Cloud is available today for all Enterprise Hub Organizations! Give the service a try, and let us know your feedback! + +## Pricing for Train on DGX Cloud + +Usage of Train on DGX Cloud is billed by the minute of the GPU instances used during your training jobs. Current prices for training jobs are $8.25 per GPU hour for H100 instances, and $2.75 per GPU hour for L40S instances. Usage fees accrue to your Enterprise Hub Organizations’ current monthly billing cycle, once a job is completed. You can check your current and past usage at any time within the billing settings of your Enterprise Hub Organization. + + + + + + + + + + + + + + + + + + +
NVIDIA GPU + GPU Memory + On-Demand Price/hr +
NVIDIA L40S + 48GB + $2.75 +
NVIDIA H100 + 80 GB + $8.25 +
+ + +For example, fine-tuning Mistral 7B on 1500 samples on a single NVIDIA L40S takes ~10 minutes and costs ~$0.45. + + +## We’re just getting started + +We are excited to collaborate with NVIDIA to democratize accelerated machine learning across open science, open source, and cloud services. + +Our collaboration on open science through [BigCode](https://huggingface.co/bigcode) enabled the training of [StarCoder 2 15B](https://huggingface.co/bigcode/starcoder2-15b), a fully open, state-of-the-art code LLM trained on more than 600 languages. + +Our collaboration on open source is fueling the new [optimum-nvidia library](https://github.com/huggingface/optimum-nvidia), accelerating the inference of LLMs on the latest NVIDIA GPUs and already achieving 1,200 tokens per second with Llama 2. + +Our collaboration on cloud services created Train on DGX Cloud today. We are also working with NVIDIA to optimize inference and make accelerated computing more accessible to the Hugging Face community, leveraging our collaboration on [NVIDIA TensorRT-LLM](https://developer.nvidia.com/blog/optimizing-inference-on-llms-with-tensorrt-llm-now-publicly-available/) and [optimum-nvidia](https://github.com/huggingface/optimum-nvidia). In addition, some of the most popular open models on Hugging Face will be on [NVIDIA NIM microservices](https://developer.nvidia.com/blog/nvidia-nim-offers-optimized-inference-microservices-for-deploying-ai-models-at-scale/), which was announced today at GTC. + +For those attending GTC this week, make sure to watch session [S63149](https://www.nvidia.com/gtc/session-catalog/?tab.allsessions=1700692987788001F1cG&search=S63149#/session/1704937870817001eXsB) on Wednesday 3/20, at 3pm PT where [Jeff](https://huggingface.co/jeffboudier) will guide you through Train on DGX Cloud and more. Also don't miss the next Hugging Cast where we will give a live demo of Train on DGX Cloud and you can ask questions directly to [Abhishek](https://huggingface.co/abhishek) and [Rafael](https://huggingface.co/rafaelpierrehf) on Thursday, 3/21, at 9am PT / 12pm ET / 17h CET - [Register here](https://streamyard.com/watch/YfEj26jJJg2w).