From 0b2a6f6cb60021d971e4b7acddc66fb8f225cb66 Mon Sep 17 00:00:00 2001 From: Sharon Lifshitz Date: Mon, 21 Dec 2020 01:24:30 +0200 Subject: [PATCH 01/15] [DOC REVIEW] welcome/README: Edit demo overviews Doc review for PR #293. NOTE: From README.md, the "View on GitHub" links open the notebooks in the current tab instead of in a new tab or window, despite the `target="_blank"` in the anchor tag (``). --- README.md | 56 ++++++++++++++++++++++++--------------------------- welcome.ipynb | 56 ++++++++++++++++++++++++--------------------------- 2 files changed, 52 insertions(+), 60 deletions(-) diff --git a/README.md b/README.md index cf7faa05..b8d8d0ed 100644 --- a/README.md +++ b/README.md @@ -80,77 +80,73 @@ You can get the latest demos from the GitHub repository by running the following Description - scikit-learn pipeline + scikit-learn Demo: Full AutoML Pipeline -
Open locally +
Open locally -
View on GitHub +
View on GitHub - - Builds a full end-to-end pipeline using scikit-learn and the UCI Iris data set + Demonstrates how to build a full end-to-end automated-ML (AutoML) pipeline using scikit-learn and the UCI Iris data set. - Image classification with distributed training + Image Classification with Distributed Training Demo -
Open locally +
Open locally -
View on GitHub +
View on GitHub - - Implements an end-to-end image-classification solution using TensorFlow (versions 1 or 2), Keras, Horovod, and Nuclio + Demonstrates an end-to-end image-classification solution using TensorFlow (versions 1 or 2), Keras, Horovod, and Nuclio. - Face recognition + Faces Demo: Real-Time Image Recognition with Deep Learning -
Open locally +
Open locally -
View on GitHub +
View on GitHub - - Real-time capture, recognition, and classification of face images over a video stream, as well as location tracking of identities using PyTorch, Image identification and tracking using OpenCV and labeling application for tagging unidentified faces using Streamlit + Demonstrates real-time capture, recognition, and classification of face images over a video stream, as well as location tracking of identities, using PyTorch, OpenCV, and Streamlit. - Customer churn prediction + Churn Demo: Real-Time Customer-Churn Prediction -
Open locally +
Open locally -
View on GitHub +
View on GitHub - - Analyses of customer-churn data using the Kaggle Telco Customer Churn data set, model training and validation using XGBoost, and model serving using real-time Nuclio serverless functions + Demonstrates analysis of customer-churn data using the Kaggle Telco Customer Churn data set, model training and validation using XGBoost, and model serving using real-time Nuclio serverless functions. - Stock trading + Stock-Analysis Demo -
Open locally +
Open locally -
View on GitHub +
View on GitHub - - Reads stock-exchange data from an internet service into a time-series database (TSDB) and performs real-time market-sentiment analysis on specific stocks; the data is saved to a platform NoSQL table for generating reports and analyzing and visualizing the data on a Grafana dashboard + Demonstrates how to tackle a common requirement of running a data-engineering pipeline as part of ML model serving by reading data from external data sources and generating insights using ML models. + The demo reads stock data from an external source, analyzes the related market news, and visualizes the analyzed data in a Grafana dashboard. - Network operations with drift detection + NetOps Demo: Predictive Network Operations/Telemetry -
Open locally +
Open locally -
View on GitHub +
View on GitHub - - Error prediction based on network device telematry. Training pipeline to create a new model, serving with streaming and concept drift to monitor the model performance + Demonstrates how to build an automated ML pipeline for predicting network outages based on network-device telemetry, also known as Network Operations (NetOps). + The demo implements both model training and inference, including model monitoring and concept-drift detection. diff --git a/welcome.ipynb b/welcome.ipynb index 6654b561..350ae149 100644 --- a/welcome.ipynb +++ b/welcome.ipynb @@ -150,77 +150,73 @@ " Description\n", " \n", " \n", - " scikit-learn pipeline\n", + " scikit-learn Demo: Full AutoML Pipeline\n", " \n", - "
Open locally\n", + "
Open locally\n", " \n", " \n", " \n", - "
View on GitHub\n", + "
View on GitHub\n", " \n", - " \n", - " Builds a full end-to-end pipeline using scikit-learn and the UCI Iris data set\n", + " Demonstrates how to build a full end-to-end automated-ML (AutoML) pipeline using scikit-learn and the UCI Iris data set.\n", " \n", " \n", " \n", - " Image classification with distributed training\n", + " Image Classification with Distributed Training Demo\n", " \n", - "
Open locally\n", + "
Open locally\n", " \n", " \n", " \n", - "
View on GitHub\n", + "
View on GitHub\n", " \n", - " \n", - " Implements an end-to-end image-classification solution using TensorFlow (versions 1 or 2), Keras, Horovod, and Nuclio\n", + " Demonstrates an end-to-end image-classification solution using TensorFlow (versions 1 or 2), Keras, Horovod, and Nuclio.\n", " \n", " \n", " \n", - " Face recognition\n", + " Faces Demo: Real-Time Image Recognition with Deep Learning\n", " \n", - "
Open locally\n", + "
Open locally\n", " \n", " \n", - "
View on GitHub\n", + "
View on GitHub\n", " \n", - " \n", - " Real-time capture, recognition, and classification of face images over a video stream, as well as location tracking of identities using PyTorch, Image identification and tracking using OpenCV and labeling application for tagging unidentified faces using Streamlit\n", + " Demonstrates real-time capture, recognition, and classification of face images over a video stream, as well as location tracking of identities, using PyTorch, OpenCV, and Streamlit.\n", " \n", " \n", " \n", - " Customer churn prediction\n", + " Churn Demo: Real-Time Customer-Churn Prediction\n", " \n", - "
Open locally\n", + "
Open locally\n", " \n", " \n", - "
View on GitHub\n", + "
View on GitHub\n", " \n", - " \n", - " Analyses of customer-churn data using the Kaggle Telco Customer Churn data set, model training and validation using XGBoost, and model serving using real-time Nuclio serverless functions\n", + " Demonstrates analysis of customer-churn data using the Kaggle Telco Customer Churn data set, model training and validation using XGBoost, and model serving using real-time Nuclio serverless functions.\n", " \n", " \n", " \n", - " Stock trading\n", + " Stock-Analysis Demo\n", " \n", - "
Open locally\n", + "
Open locally\n", " \n", " \n", - "
View on GitHub\n", + "
View on GitHub\n", " \n", - " \n", - " Reads stock-exchange data from an internet service into a time-series database (TSDB) and performs real-time market-sentiment analysis on specific stocks; the data is saved to a platform NoSQL table for generating reports and analyzing and visualizing the data on a Grafana dashboard\n", + " Demonstrates how to tackle a common requirement of running a data-engineering pipeline as part of ML model serving by reading data from external data sources and generating insights using ML models.\n", + " The demo reads stock data from an external source, analyzes the related market news, and visualizes the analyzed data in a Grafana dashboard.\n", " \n", " \n", " \n", - " Network operations with drift detection\n", + " NetOps Demo: Predictive Network Operations/Telemetry\n", " \n", - "
Open locally\n", + "
Open locally\n", " \n", " \n", - "
View on GitHub\n", + "
View on GitHub\n", " \n", - " \n", - " Error prediction based on network device telematry. Training pipeline to create a new model, serving with streaming and concept drift to monitor the model performance\n", + " Demonstrates how to build an automated ML pipeline for predicting network outages based on network-device telemetry, also known as Network Operations (NetOps).\n", + " The demo implements both model training and inference, including model monitoring and concept-drift detection.\n", " \n", " \n", " " From e6de6694cd6fd526567f66ce5c3ef0b983355794 Mon Sep 17 00:00:00 2001 From: Sharon Lifshitz Date: Mon, 21 Dec 2020 15:02:30 +0200 Subject: [PATCH 02/15] [DOC REVIEW] welcome NB/main README: edit demos, data ingestion and preparation, additional resources doc --- README.md | 25 +++++++++--------- ...-data-ingestion-and-preparation-button.png | Bin 2478 -> 0 bytes assets/images/open-mlrun-docs-button.png | Bin 1998 -> 0 bytes welcome.ipynb | 25 +++++++++--------- 4 files changed, 26 insertions(+), 24 deletions(-) delete mode 100644 assets/images/open-data-ingestion-and-preparation-button.png delete mode 100644 assets/images/open-mlrun-docs-button.png diff --git a/README.md b/README.md index b8d8d0ed..ce588263 100644 --- a/README.md +++ b/README.md @@ -138,7 +138,7 @@ You can get the latest demos from the GitHub repository by running the following - NetOps Demo: Predictive Network Operations/Telemetry + NetOps Demo: Predictive Network Operations / Telemetry
Open locally @@ -155,10 +155,11 @@ You can get the latest demos from the GitHub repository by running the following ## Data Ingestion and Preparation -The Iguazio Data Science Platform (**"the platform"**) allows storing data in any format. +The platform allows storing data in any format. The platform's multi-model data layer and related APIs provide enhanced support for working with NoSQL ("key-value"), time-series, and stream data. Various steps of the data science life cycle (pipeline) might require different tools and frameworks for working with data, especially when it comes to the different mechanisms required during the research and development phase versus the operational production phase. -The platform features a wide set of methods for manipulating and managing data, of different formats, in each step of the data life cycle, using a variety of frameworks, tools, and APIs — such as: +The platform features a wide array of methods for manipulating and managing data, of different formats, in each step of the data life cycle, using a variety of frameworks, tools, and APIs — such as as the following: + - Spark SQL and DataFrames - Spark Streaming - Presto SQL queries @@ -166,26 +167,26 @@ The platform features a wide set of methods for manipulating and managing data, - Dask - V3IO Frames Python library - V3IO SDK -- Web APIs. - -Refer to [data ingestion and preparation](./data-ingestion-and-preparation/README.ipynb) for an overview of various methods for collecting, storing, and manipulating data in the platform, and refers to sample tutorial notebooks that demonstrate how to use these methods. +- Web APIs -Open data ingestion and preparation +The data ingestion and preparation tutorial README (**data-ingestion-and-preparation/README.ipynb/.md**) provides an overview of various methods for collecting, storing, and manipulating data in the platform, and references to sample tutorial notebooks that demonstrate how to use these methods. +
+**▶ [Open the README notebook](./data-ingestion-and-preparation/README.ipynb) / [Markdown file](data-ingestion-and-preparation/README.md)** ## Additional Platform Resources -Refer to the [MLRun documentation](https://mlrun.readthedocs.io) for more information - -Open MLRun Documentation +You can find more information and resources in the MLRun documentation: +
+**▶ [Open the MLRun documentation](https://mlrun.readthedocs.io)** -Other resources: +You might also find the folloiwng resources useful: - [Introduction video](https://www.youtube.com/watch?v=8OmAN4wd7To) - [In-depth platform overview](platform-overview.ipynb) with a break down of the steps for developing a full data science workflow from development to production - [Platform components, services, and development ecosystem introduction](https://www.iguazio.com/docs/latest-release/intro/ecosystem/) -- [Iguazio platform References](https://iguazio.com/docs/latest-release/reference/) +- [Platform references](https://iguazio.com/docs/latest-release/reference/) - [nuclio-jupyter SDK](https://github.com/nuclio/nuclio-jupyter/blob/master/README.md) for creating and deploying Nuclio functions with Python and Jupyter Notebook diff --git a/assets/images/open-data-ingestion-and-preparation-button.png b/assets/images/open-data-ingestion-and-preparation-button.png deleted file mode 100644 index 731460fc8c70a3e87a67cbf8d057a509e4e6b1cd..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 2478 zcmV;f2~qZmP)001cn0{{R3-PqK60004oX+uL$Nkc;* zaB^>EX>4Tx04R}tkv&MmP!xqvQ>CI62Rn!;WT@iMf~bh2RIvyaN?V~-2a}in2u&K2 z6c<1NmzipIj037> z8L4Q(Y41AHR!EYl5(c%689 z)6zNb6NgwyQi#uqBL+1{{K$3L zby?xO#aS&^S+h_6!eCxoPIH}V2yrYTfh0u8sA2;jFNZGRuzcH;!_KLb}<%U`JjGoPf_ zT3YxB=-UP^u3MVC2VCv|15dhSNRH&E5()+2{fxdT2lU?p-K*~0n&&uu05UYIr5oVj z5Ev~|_L|4LyV~dW@15rSegNB@a?!e{or3@X010qNS#tmY5_A9n5_AFHW*>L}000Mc zNliru}P1P5oZp(R7C&)01w@-5#F#9-LM&U(iz^e9pAJe;I$;+wkP4Y zDdD&)fz~eJxiaFrHsicF-m^T5**%NdKHsxH(FxR&~@w4cdq1p?9+bj z)PL>NhUCPF@7Rjv#)|LQjOE9S@7a#z$dBa6kiF@U?8f)($M@{V z_wC8|?b7-1)#mrt`SIBL^4aj{+wkez`t#lT^xyI8;PLF?@$BO9?Beq5ht^T{rT_x`tkky^Y!)f{`>Xy_VxAm_4)nw_V@Sp_xSes{r>*{|NmA@AVdHF z00DGTPE!Ct=GbNc0004EOGiX+RS02$000I>NkltCBS6z4AZ zsZdgI;^st+7Z^B+V?sNHc^zeS2-^Jr-^@KHm$%)g;D?+aKY05lIeE@`aydy8Diufe zH(K+xnJZZ?LuP98t&ROksT{A(Hha5+AHQ0?v-r;9oz<@oX_kaC9u0PT&Dpi%C_5+J zYHvV~)^7qvZ>}F@wDjLJM)dh|(CL<+XO|_GQLnb`)6;DpP`Xu(jEdE* z?6K5x(CoVOv^rx-S4-xt)$Fj;GVZh-dRExGPHAA#$qq{{D}bObCt!I(M!4hZ zh|;~On-6>0U#VqOqSwt&-o)NxYYZU4jbaJXbkxyW@$v_ zQ3Z+usq&a-qV&8iOYrxqh!Vs-yaMC=K z4s2|U(>50cf#w4fG(b{jEs_}tG>M3u$RFS~Dy?1i_m)e=mlx083;4NZZL_G5d3O2X zkOF?xXO5J3{E)(Aw*2s1(yKZvR|}QNXIYucj|Y;t{AdVHnu`*^Mn;`T3;2<>7!!0c zSzW|h$q#Z&;zdJ#c3YKsHvQ1+9v7dT{ra}g&vlC$i|Qm+BAzibQME|fm5|Giawd`O z=EJ&9!Llixw;OHU-T*&>Yi=ge9AK?X7e~A{qN)`Lgq0pht4WH50McBP1~%atBu)*6 zSfJW`V1fc@dP!idNFl0WD75@l$j@NDQe)9Iz3P*3|Lphoy;oO(Pdu#BO@N>J>nzdd zhrJLbQN53}s}$mX9kCyupLYfAqHoIl2vMIOQ>n|32hwQL6&MI0%|&Ty5@{0Wd0TcG zmml3Cvg2oXwo?igs>J{ST8lu+N<3hBLVm_G72)+GLeSJR!-O3|lc)f>{D3(&&JXr9SwATVAk9HpGXEgz@ng+5Cg@@cO<=9VkD!>< zJZziD2FjJP{l8*UIgs8Jifg80M+Ez;L9Yoq;#k2Vit{YT6{)d)S`_}`D5?0OR!#7<*(`C(e&`bRcTTR$9j9DYKON)x3n z1dtY4|G50H-yl9PrTsz0#OXL{Esd>zB$u)EkJvxB>oD=!Z)f+z8R>3(|Dco~wKVqz za$**7VETlUh071iuZK2&|3EJa3$G2w_79*Y7XnBFC`-~teaAjmo zQW0Nd|3K`&J#ne{?&C;6_k?CiEo}cy>|X~HP$+ZRTgip*U(4@*?oH+AUUpY<5#Rra z?@wRWrqEM+ncbCIu<^E>kV+tIZN^jyo= z7?O)Q|EA85?&ZTf|10Tqz4Oc}m*D=7x<5ys`nMRCpJs2Rme~C{egBlc*jl=wxVf>k^&)#KwUpdHwfDD213*Lyvj6}907*qoM6N<$f_+sK4gdfE diff --git a/assets/images/open-mlrun-docs-button.png b/assets/images/open-mlrun-docs-button.png deleted file mode 100644 index 4e78d00f36e3b558364cffe369cab42485f075b0..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 1998 zcmV;<2Qm1GP)EX>4Tx04R}tkv&MmP!xqvQ>CI62Rn!;WT@iMf~bh2RIvyaN?V~-2a}in2u&K2 z6c<1NmzipIj037> z8L4Q(Y41AHR!EYl5(c%689 z)6zNb6NgwyQi#uqBL+1{{K$3L zby?xO#aS&^S+h_6!eCxoPIH}V2yrYTfh0u8sA2;jFNZGRuzcH;!_KLb}<%U`JjGoPf_ zT3YxB=-UP^u3MVC2VCv|15dhSNRH&E5()+2{fxdT2lU?p-K*~0n&&uu05UYIr5oVj z5Ev~|_L|4LyV~dW@15rSegNB@a?!e{or3@X010qNS#tmYE+YT{E+YYWr9XB6000Mc zNliru}P13BrCZ^!@u0aj2(FxR&~@w4cdq1p?9+bj z)PL>NhUCPF@7Rjv#)|LQjOE9S@7a#z$dBa6kiF@UesvW>AmaPzV_+A+495o>&5r%#`o;U z_w2{_?aBA;()sY!=J(h6@!0zE+3@Jw@afz7^WFON-|_3X_^6cdE z?dI|B=KS~R{P^ng`|SPs@BRAm{rmIv_4EGw_4W4k_4oDp{r2|v_xAVr_W1q&{{R2~ z;Vuq-00001bW%=J06^y0W&i*H0b)x>L~*YNT;c!#1Yb!+K~!ko-J01$(=ZsoE3{P@ zAfw)jj4{V(K}Atf;{l2XQIKsYcq_vxf|CFL#pFxcz}lQZ}0#IbVNZmuNJ;Ow@QZ>JU{O|msN zB#p_?+!k@KG?SpVV*)ORNt=AF8j7!eb&#>0gv&Z$E!rStDX`dfu;d^j*@$W?V5xFQ z$V`s^&bWwpSScxh)AU*;H$+ujL?qJ^VvQkGyF}rtlZchfP?sZWqKjP<&7zZr8i{bT z6Vf14?~zOEPKKM1CyiP{e@yfsoN&@k6R^|mpiRhX&k510i-hff&2DCy3a5G!PPwVX zSxPwABxLt5X0^L65;h0vY9gYQM+(c>^DztIBPW#@t$Fi_kqfUKygAV=CpHSc@0a&k zHTGF4Gze)Gq3Rb~tcvJ8EZXo6m^VSJ4;ByNy`fpFNIQx?v^i^m{8+R@yq)NVSzX-P zFK42_Ti_vzna+Eq75aTrg}Hn1_PjxOlBn-vSgDM(t`f30sf02AvJtX-8X@ax5W={n zouE04wGf5Zb;OmDH$vTz9MfoS?R_|-6CUem5W(OvDL`6UN2!RcFxXfn94*i~@(n@^ zcVSc%Le@?uMDWSSO_)OrFXjUm3zgH$SaZY?FQmDG^y@ead!H_7m-@5U#T5UH<;% zqIM~>d@RI6st#p~drT#j3ClNq+$z#f3nBIm9nyxeNr>PzV3J%6oV5^En!Zq)x%GJ< zxFvMn`2D#%A>z{Nekgp33a=18@nb3vvil*@{jl6B7aA7AI3t9oAem5!K%H_VPlsc# ze?9yw9j5NBSZT3Et?02#_>L(MTM0pAjgT3@&2EGs2GW4yG#V*71V0QcfDiPwe!d?P zVh}=DkqBF5hJs8h#at*Mc}57*8h(g$Kjg#GG0s?5BK1RvrNbJ{skaArbm-xZy~k=S zwGy&SB@yl;_R~Trf+K%k8XRd=1Z0U2r?C{Nu{5h=VrZ4cQpkte%Hs63HnZsh05c?23ketwX?>Ln_eJT%;$p5!|s7?!{dtka zJ(&DJLM zfJa?Cg6iy-iaqbu|81j9+Vd9Y7SDnlzTlbj=N97ZsUBGYBPaK}wX-L2en{plfOBpJ z8U6fFoB@+f{RFDI>0}\n", " \n", " \n", - " NetOps Demo: Predictive Network Operations/Telemetry\n", + " NetOps Demo: Predictive Network Operations / Telemetry\n", " \n", "
Open locally\n", " \n", @@ -235,10 +235,11 @@ "source": [ "## Data Ingestion and Preparation\n", "\n", - "The Iguazio Data Science Platform (**\"the platform\"**) allows storing data in any format.\n", + "The platform allows storing data in any format.\n", "The platform's multi-model data layer and related APIs provide enhanced support for working with NoSQL (\"key-value\"), time-series, and stream data.\n", "Various steps of the data science life cycle (pipeline) might require different tools and frameworks for working with data, especially when it comes to the different mechanisms required during the research and development phase versus the operational production phase.\n", - "The platform features a wide set of methods for manipulating and managing data, of different formats, in each step of the data life cycle, using a variety of frameworks, tools, and APIs — such as:\n", + "The platform features a wide array of methods for manipulating and managing data, of different formats, in each step of the data life cycle, using a variety of frameworks, tools, and APIs — such as as the following:\n", + "\n", "- Spark SQL and DataFrames\n", "- Spark Streaming\n", "- Presto SQL queries\n", @@ -246,11 +247,11 @@ "- Dask\n", "- V3IO Frames Python library\n", "- V3IO SDK\n", - "- Web APIs.\n", - "\n", - "Refer to [data ingestion and preparation](./data-ingestion-and-preparation/README.ipynb) for an overview of various methods for collecting, storing, and manipulating data in the platform, and refers to sample tutorial notebooks that demonstrate how to use these methods.\n", + "- Web APIs\n", "\n", - "\"Open" + "The data ingestion and preparation tutorial README (**data-ingestion-and-preparation/README.ipynb/.md**) provides an overview of various methods for collecting, storing, and manipulating data in the platform, and references to sample tutorial notebooks that demonstrate how to use these methods.\n", + "
\n", + "**▶ [Open the README notebook](./data-ingestion-and-preparation/README.ipynb) / [Markdown file](data-ingestion-and-preparation/README.md)**" ] }, { @@ -266,16 +267,16 @@ "source": [ "## Additional Platform Resources\n", "\n", - "Refer to the [MLRun documentation](https://mlrun.readthedocs.io) for more information\n", - "\n", - "\"Open\n", + "You can find more information and resources in the MLRun documentation:\n", + "
\n", + "**▶ [Open the MLRun documentation](https://mlrun.readthedocs.io)**\n", "\n", - "Other resources:\n", + "You might also find the folloiwng resources useful:\n", "\n", "- [Introduction video](https://www.youtube.com/watch?v=8OmAN4wd7To)\n", "- [In-depth platform overview](platform-overview.ipynb) with a break down of the steps for developing a full data science workflow from development to production\n", "- [Platform components, services, and development ecosystem introduction](https://www.iguazio.com/docs/latest-release/intro/ecosystem/)\n", - "- [Iguazio platform References](https://iguazio.com/docs/latest-release/reference/)\n", + "- [Platform references](https://iguazio.com/docs/latest-release/reference/)\n", "- [nuclio-jupyter SDK](https://github.com/nuclio/nuclio-jupyter/blob/master/README.md) for creating and deploying Nuclio functions with Python and Jupyter Notebook" ] }, From 816a11d03d2c41af28d6aa8dfae7bb5ae39c7e61 Mon Sep 17 00:00:00 2001 From: Sharon Lifshitz Date: Mon, 21 Dec 2020 17:55:50 +0200 Subject: [PATCH 03/15] [DOC[ welcome.ipynb.README.md: minor demo-doc cosmetics --- README.md | 10 +++++----- welcome.ipynb | 19 +++++++------------ 2 files changed, 12 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index ce588263..147c1a74 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ An initial introduction to the Iguazio Data Science Platform and the platform tu - [Data Science Workflow](#data-science-workflow) - [The Tutorial Notebooks](#the-tutorial-notebooks) - [Getting-Started Tutorial](#getting-started-tutorial) -- [End-to-End Use-Case Applications](#end-to-end-use-case-applications) +- [End-to-End Use-Case Applications (Demos)](#end-to-end-use-case-applications) - [Data Ingestion and Preparation](#data-ingestion-and-preparation) - [Additional Platform Resources](#platform-resources) - [Miscellaneous](#misc) @@ -60,7 +60,7 @@ Start out by running the getting-started tutorial to familiarize yourself with t -## End-to-End Use-Case Applications +## End-to-End Use-Case Applications (Demos) Iguazio provides full end-to-end use-case applications (demos) that demonstrate how to use the platform and related tools to address data science requirements for different industries and implementations. These demos are available in the [**MLRun demos repository**](https://github.com/mlrun/demos). @@ -73,10 +73,10 @@ You can get the latest demos from the GitHub repository by running the following ``` - + - - + diff --git a/welcome.ipynb b/welcome.ipynb index 57fee720..22b20dc6 100644 --- a/welcome.ipynb +++ b/welcome.ipynb @@ -17,7 +17,7 @@ "- [Data Science Workflow](#data-science-workflow)\n", "- [The Tutorial Notebooks](#the-tutorial-notebooks)\n", "- [Getting-Started Tutorial](#getting-started-tutorial)\n", - "- [End-to-End Use-Case Applications](#end-to-end-use-case-applications)\n", + "- [End-to-End Use-Case Applications (Demos)](#end-to-end-use-case-applications)\n", "- [Data Ingestion and Preparation](#data-ingestion-and-preparation)\n", "- [Additional Platform Resources](#platform-resources)\n", "- [Miscellaneous](#misc)" @@ -116,15 +116,10 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## End-to-End Use-Case Applications\n", + "## End-to-End Use-Case Applications (Demos)\n", + "\n", + "Iguazio provides full end-to-end use-case applications (demos) that demonstrate how to use the platform and related tools to address data science requirements for different industries and implementations. These demos are available in the [**MLRun demos repository**](https://github.com/mlrun/demos).\n", "\n", - "Iguazio provides full end-to-end use-case applications (demos) that demonstrate how to use the platform and related tools to address data science requirements for different industries and implementations. These demos are available in the [**MLRun demos repository**](https://github.com/mlrun/demos)." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ "You can get the latest demos from the GitHub repository by running the following command:" ] }, @@ -143,10 +138,10 @@ "metadata": {}, "source": [ "
Demo + Description
\n", - " \n", + " \n", " \n", - " \n", - " \n", + " \n", " \n", " \n", From 13dbf08508a4500930714fd9d7ced05847712295 Mon Sep 17 00:00:00 2001 From: Sharon Lifshitz Date: Mon, 21 Dec 2020 20:51:56 +0200 Subject: [PATCH 04/15] [DOC] New align_mlrun.sh script [IG-17473] --- README.md | 19 +++++++++++ .../getting-started-tutorial.ipynb | 11 ++++--- welcome.ipynb | 33 +++++++++++++++++++ 3 files changed, 58 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 147c1a74..489891f6 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,7 @@ An initial introduction to the Iguazio Data Science Platform and the platform tu - [The Tutorial Notebooks](#the-tutorial-notebooks) - [Getting-Started Tutorial](#getting-started-tutorial) - [End-to-End Use-Case Applications (Demos)](#end-to-end-use-case-applications) +- [Installing and Updating the MLRun Python Package](#mlrun-python-pkg-install-n-update) - [Data Ingestion and Preparation](#data-ingestion-and-preparation) - [Additional Platform Resources](#platform-resources) - [Miscellaneous](#misc) @@ -151,6 +152,24 @@ You can get the latest demos from the GitHub repository by running the following
Demo\n", + " \n", " Description
+ + +## Installing and Updating the MLRun Python Package + +The demo applications and many of the platform tutorials use [MLRun](https://github.com/mlrun/mlrun) — Iguazio's end-to-end open-source MLOps solution for managing and automating your entire analytics and machine-learning life cycle, from data ingestion through model development to full pipeline deployment in production. +MLRun is available in the platform via a default (pre-deployed) shared platform service (`mlrun`). +However, to use MLRun from Python code (such as in the demo and tutorial notebooks), you also need to install the [MLRun Python package](https://readthedocs.org/projects/mlrun/) (`mlrun`). +The version of the installed package must match the version of the platform's MLRun service and must be updated whenever the service's version is updated. + +The platform provides an [**align_mlrun.sh**](./align_mlrun.sh) script for simplifying the MLrun package installation and version synchronization with the MLRun service. +The script is available in the running-user directory (your Jupyter home directory), which is accessible via the **/User** data mount. +Use the following command to run this script for the initial package installation (after creating a new Jupyter Notebook service) and whenever the MLRun service is updated; (the command should be run for each Jupyter Notebook service): + + +```python +!/User/align_mlrun.sh +``` + ## Data Ingestion and Preparation diff --git a/getting-started-tutorial/getting-started-tutorial.ipynb b/getting-started-tutorial/getting-started-tutorial.ipynb index bb9a8747..32a3119a 100644 --- a/getting-started-tutorial/getting-started-tutorial.ipynb +++ b/getting-started-tutorial/getting-started-tutorial.ipynb @@ -70,13 +70,14 @@ "source": [ "The tutorial uses MLRun to create a project, implement and execute an ML pipeline, and track the execution.\n", "(For more information about MLRun, see Step 1.)\n", - "To use MLRun, you must first ensure that it's installed and running as a service on your platform cluster.\n", + "To use MLRun, you must first ensure that it's installed and running as a service on your platform clusters.\n", "Look for an `mlrun` service on the **Services** page of the platform dashboard.\n", "For more information and additional assistance, contact the Iguazio [support team](mailto:support@iguazio.com).\n", "\n", - "To use MLRun from Jupyter Notebook, you need to run the following code to install the `mlrun` Python package.\n", - "This needs to be done only once per Jupyter Notebook service.\n", - "> **Note:** You must **restart the Jupyter kernel** to complete the installation." + "To use MLRun from Jupyter Notebook, you also need to install the MLRun Python package (`mlrun`) and synchronize its version with the running MLRun service.\n", + "This needs to be done only once per Jupyter Notebook service for the initial installation and whenever the MLRun service is updated.\n", + "You can install and update the MLRun Python package by running the provided **align_mlrun.sh** script from Jupyter Notebook, as demonstrated in the following cell.\n", + "For more information, see the [**weclome.ipynb**](../welcome.ipynb#mlrun-python-pkg-install-n-update) notebook or [**REAMDE.md**](../README.md#mlrun-python-pkg-install-n-update) file." ] }, { @@ -85,7 +86,7 @@ "metadata": {}, "outputs": [], "source": [ - "%pip install mlrun" + "!/User/align_mlrun.sh" ] }, { diff --git a/welcome.ipynb b/welcome.ipynb index 22b20dc6..d8b30f82 100644 --- a/welcome.ipynb +++ b/welcome.ipynb @@ -18,6 +18,7 @@ "- [The Tutorial Notebooks](#the-tutorial-notebooks)\n", "- [Getting-Started Tutorial](#getting-started-tutorial)\n", "- [End-to-End Use-Case Applications (Demos)](#end-to-end-use-case-applications)\n", + "- [Installing and Updating the MLRun Python Package](#mlrun-python-pkg-install-n-update)\n", "- [Data Ingestion and Preparation](#data-ingestion-and-preparation)\n", "- [Additional Platform Resources](#platform-resources)\n", "- [Miscellaneous](#misc)" @@ -217,6 +218,38 @@ " " ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Installing and Updating the MLRun Python Package\n", + "\n", + "The demo applications and many of the platform tutorials use [MLRun](https://github.com/mlrun/mlrun) — Iguazio's end-to-end open-source MLOps solution for managing and automating your entire analytics and machine-learning life cycle, from data ingestion through model development to full pipeline deployment in production.\n", + "MLRun is available in the platform via a default (pre-deployed) shared platform service (`mlrun`).\n", + "However, to use MLRun from Python code (such as in the demo and tutorial notebooks), you also need to install the [MLRun Python package](https://readthedocs.org/projects/mlrun/) (`mlrun`).\n", + "The version of the installed package must match the version of the platform's MLRun service and must be updated whenever the service's version is updated.\n", + "\n", + "The platform provides an [**align_mlrun.sh**](./align_mlrun.sh) script for simplifying the MLrun package installation and version synchronization with the MLRun service.\n", + "The script is available in the running-user directory (your Jupyter home directory), which is accessible via the **/User** data mount.\n", + "Use the following command to run this script for the initial package installation (after creating a new Jupyter Notebook service) and whenever the MLRun service is updated; (the command should be run for each Jupyter Notebook service):" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!/User/align_mlrun.sh" + ] + }, { "cell_type": "markdown", "metadata": {}, From 052a654e15bf7b64ee9ad00cd7837936b165eb5c Mon Sep 17 00:00:00 2001 From: Sharon Lifshitz Date: Tue, 22 Dec 2020 00:42:58 +0200 Subject: [PATCH 05/15] [DOC] Edit the update-tutorials & get-demos script help texts & related NB doc --- README.md | 23 ++++++++++++----- get-additional-demos.sh | 56 +++++++++++++++++++++++------------------ igz-tutorials-get.sh | 30 +++++++++++----------- update-tutorials.ipynb | 26 +++++++++++-------- welcome.ipynb | 32 ++++++++++++++++++----- 5 files changed, 107 insertions(+), 60 deletions(-) diff --git a/README.md b/README.md index 489891f6..a6a28e6a 100644 --- a/README.md +++ b/README.md @@ -63,9 +63,12 @@ Start out by running the getting-started tutorial to familiarize yourself with t ## End-to-End Use-Case Applications (Demos) -Iguazio provides full end-to-end use-case applications (demos) that demonstrate how to use the platform and related tools to address data science requirements for different industries and implementations. These demos are available in the [**MLRun demos repository**](https://github.com/mlrun/demos). - -You can get the latest demos from the GitHub repository by running the following command: +Iguazio provides full end-to-end use-case applications (demos) that demonstrate how to use the platform and related tools to address data science requirements for different industries and implementations. +These demos are available in the [MLRun demos repository](https://github.com/mlrun/demos). +Use the provided [**get-additional-demos.sh**](./get-additional-demos.sh) script to get updated demos from this repository. +By default, the script retrieves the files from the latest release that matches the version of the platform's running MLRun service. +The files are copied to the **/v3io/users/<username>/demos** directory, where `` is the name of the running user (`$V3IO_USERNAME`) unless you set the `-u|--user` flag to another username. +> **Note:** Before running the script, close any open files in the **demos** directory. ```python @@ -73,6 +76,13 @@ You can get the latest demos from the GitHub repository by running the following !/User/get-additional-demos.sh ``` +For full usage instructions, run the script with the `-h` or `--help` flag: + + +```python +!/User/get-additional-demos.sh --help +``` + @@ -198,7 +208,7 @@ The data ingestion and preparation tutorial README (**data-ingestion-and-prepara You can find more information and resources in the MLRun documentation:
-**▶ [Open the MLRun documentation](https://mlrun.readthedocs.io)** +**▶ [View the MLRun documentation](https://mlrun.readthedocs.io)** You might also find the folloiwng resources useful: @@ -220,9 +230,10 @@ Virtual environments make it easy to cleanly separate projects and avoid problem See the [virtual-env](virtual-env.ipynb) tutorial notebook for step-by-step instructions for using conda to create your own Python virtual environments, which will appear as custom kernels in Jupyter Notebook. -### Updating the Tutorial Notebooks to the Latest Version +### Updating the Tutorial Notebooks -You can use the provided **igz-tutorials-get.sh** script to update the tutorial notebooks to the latest stable version available on [GitHub](https://github.com/v3io/tutorials/). +You can use the provided **igz-tutorials-get.sh** script to get updated platform tutorials from the [tutorials GitHub repository](https://github.com/v3io/tutorials/). +By default, the script retrieves the files from the latest release that matches the current platform version. For details, see the [**update-tutorials.ipynb**](update-tutorials.ipynb) notebook. diff --git a/get-additional-demos.sh b/get-additional-demos.sh index c93daecc..4c30ba20 100755 --- a/get-additional-demos.sh +++ b/get-additional-demos.sh @@ -14,16 +14,22 @@ user=${V3IO_USERNAME} USAGE="\ $SCRIPT: -Retrieves latest-release demos from the mlrun/demos GitHub repo. +Retrieves updated demos from the mlrun/demos GitHub repository. USAGE: ${SCRIPT} [OPTIONS] OPTIONS: -h|--help - Display this message and exit. - -b|--branch - Branch name (default is branch equivalent to the current MLRun version). - -u|--user - Username, which determines the directory to which to copy the files - /v3io/users/. - Default = \$V3IO_USERNAME (if set to a non-empty string). - --mlrun-ver - Determine the branch based on the MLRun version, unless -b/--branch option is specified. - --dry-run - Do not update the files, but rather show the pending changes. - --no-backup - Do not backup the current demos directory." + -b|--branch - Git branch name. Default: The latest release branch that + matches the version of the running MLRun service. + -u|--user - Username, which determines the directory to which to copy the + retrieved demo files (/v3io/users/). + Default: \$V3IO_USERNAME, if set to a non-empty string. + --mlrun-ver - The MLRun version for which to get demos; determines the Git + branch from which to get the demos, unless -b|--branch is set. + Default: The version of the running MLRun service. + --dry-run - Show files to update but don't execute the update. + --no-backup - Don't back up the existing demos directory before the update. + Default: Back up the existing demos directory to a + /v3io/users//demos.old./ directory." error_exit() { @@ -53,42 +59,42 @@ do branch=$2 shift else - error_usage "$1: missing branch name" + error_usage "$1: Missing branch name." fi ;; --branch=?*) branch=${1#*=} # Delete everything up to "=" and assign the remainder. ;; --branch=) # Handle the case of an empty --branch= - error_usage "$1: missing branch name" + error_usage "$1: Missing branch name." ;; -u|--user) if [ "$2" ]; then user=$2 shift else - error_usage "$1: missing user name" + error_usage "$1: Missing username." fi ;; --user=?*) user=${1#*=} # Delete everything up to "=" and assign the remainder. ;; --user=) # Handle the case of an empty --user= - error_usage "$1: missing user name" + error_usage "$1: Missing username." ;; --mlrun-ver) if [ "$2" ]; then mlrun_version=$2 shift else - error_usage "$1: missing MLRun version" + error_usage "$1: Missing MLRun version." fi ;; --mlrun-ver=?*) mlrun_version=${1#*=} # Delete everything up to "=" and assign the remainder. ;; --umlrun-ver=) # Handle the case of an empty --mlrun-ver= - error_usage "$1: missing MLRun version" + error_usage "$1: Missing MLRun version." ;; --dry-run) dry_run=1 @@ -96,7 +102,7 @@ do --no-backup) no_backup=1 ;; - -*) error_usage "$1: Unknown option" + -*) error_usage "$1: Unknown option." ;; *) break; esac @@ -104,15 +110,15 @@ do done if [ -z "${user}" ]; then - error_usage "Missing user name." + error_usage "Missing username." fi if [ ! -z "${dry_run}" ]; then - echo "Dry run, no files will be copied." + echo "Dry run; no files will be copied." fi if [ ! -z "${no_backup}" ]; then - echo "No backup of current demos directory will be created." + echo "The existing demos directory won't be backed up before the update." fi if [ -z "${branch}" ]; then @@ -121,21 +127,21 @@ if [ -z "${branch}" ]; then pip_mlrun=$(pip show mlrun | grep Version) || : if [ -z "${pip_mlrun}" ]; then - error_exit "MLRun version not found. Aborting." + error_exit "MLRun version not found. Aborting..." else - echo "Detected MLRun ${pip_mlrun}" + echo "Detected MLRun version: ${pip_mlrun}" mlrun_version="${pip_mlrun##Version: }" fi else - echo "Looking for demos based on specified MLRun version: ${mlrun_version}." + echo "Looking for demos for the specified MLRun version - ${mlrun_version}." fi tag_prefix=`echo ${mlrun_version} | cut -d . -f1-2` latest_tag=`git ls-remote --tags --refs --sort='v:refname' "${git_url}" "refs/tags/v${tag_prefix}.*" | tail -n1 | awk '{ print $2}'` if [ -z "${latest_tag}" ]; then - error_exit "No tag found with tag prefix 'v${tag_prefix}.*'. Aborting" + error_exit "Couldn't locate a Git tag with prefix 'v${tag_prefix}.*'. Aborting..." else - # Remote the prfix from the tag + # Remove the prefix from the Git tag branch=${latest_tag#refs/tags/} echo "Detected ${git_url} tag: ${branch}" fi @@ -147,7 +153,7 @@ echo "Updating demos from ${git_url} branch ${branch} to '${demos_dir}'..." temp_dir=$(mktemp -d /tmp/temp-get-demos.XXXXXXXXXX) trap "{ rm -rf $temp_dir; }" EXIT -echo "Copying to temporary directory '${temp_dir}'..." +echo "Copying files to a temporary directory '${temp_dir}'..." tar_url="${git_base_url}/archive/${branch}.tar.gz" wget -qO- "${tar_url}" | tar xz -C "${temp_dir}" --strip-components 1 @@ -156,6 +162,7 @@ if [ -z "${dry_run}" ]; then if [ -d "${demos_dir}" ]; then if [ -z "${no_backup}" ]; then + # Back up the existing demos directory dt=$(date '+%Y%m%d%H%M%S'); old_demos_dir="${dest_dir}/demos.old/${dt}" @@ -172,7 +179,8 @@ if [ -z "${dry_run}" ]; then mkdir -p "${demos_dir}" cp -RT "${temp_dir}" "${demos_dir}" else - echo "Files that will be copied to '${dest_dir}':" + # Dry run + echo "Identified the following files to copy to '${dest_dir}':" find "${temp_dir}/" -not -path '*/\.*' -type f -printf "%p\n" | sed -e "s|^${temp_dir}/|./demos/|" fi diff --git a/igz-tutorials-get.sh b/igz-tutorials-get.sh index 5f37bf4a..c41d43a1 100755 --- a/igz-tutorials-get.sh +++ b/igz-tutorials-get.sh @@ -13,15 +13,17 @@ user=${V3IO_USERNAME} USAGE="\ $SCRIPT: -Retrieves latest-release ${product} tutorial files from the v3io/tutorials GitHub repo. +Retrieves updated ${product} tutorial files from the v3io/tutorials GitHub repository. USAGE: ${SCRIPT} [OPTIONS] [] PARAMETERS: - - Username, which determines the directory to which to copy the files - /v3io/users/. - Default = \$V3IO_USERNAME (if set to a non-empty string) + - Username, which determines the directory to which to copy the + retrieved files (/v3io/users/). + Default: \$V3IO_USERNAME, if set to a non-empty string. OPTIONS: -h|--help - Display this message and exit. - -b|--branch - Branch name (default is latest tag of current platform version). - --dry-run - Do not update the files, but rather show the pending changes." + -b|--branch - Git branch name. Default: The latest release branch that + matches the current platform version. + --dry-run - Show files to update but don't execute the update." error_exit() { @@ -51,19 +53,19 @@ do branch=$2 shift else - error_usage "$1: missing branch name" + error_usage "$1: Missing branch name." fi ;; --branch=?*) branch=${1#*=} # Delete everything up to "=" and assign the remainder. ;; --branch=) # Handle the case of an empty --branch= - error_usage "$1: missing branch name" + error_usage "$1: Missing branch name." ;; --dry-run) dry_run=1 ;; - -*) error_usage "$1: Unknown option" + -*) error_usage "$1: Unknown option." ;; *) break; esac @@ -77,7 +79,7 @@ elif [ -z "${user}" ]; then fi if [ ! -z "${dry_run}" ]; then - echo "Dry run, no files will be copied." + echo "Dry run; no files will be copied." fi @@ -86,9 +88,9 @@ if [ -z "${branch}" ]; then echo "Detected platform version: ${platform_version}" latest_tag=`git ls-remote --tags --refs --sort='v:refname' "${git_url}" "refs/tags/v${platform_version}.*" | tail -n1 | awk '{ print $2}'` if [ -z "${latest_tag}" ]; then - error_exit "No tag found with tag prefix 'v${platform_version}.*'. Aborting" + error_exit "Couldn't locate a Git tag with prefix 'v${platform_version}.*'. Aborting..." else - # Remote the prfix from the tag + # Remove the prefix from the Git tag branch=${latest_tag#refs/tags/} echo "Detected ${git_url} tag: ${branch}" fi @@ -97,10 +99,9 @@ fi dest_dir="/v3io/users/${user}" echo "Updating ${product} tutorial files of branch ${branch} to '${dest_dir}'..." - temp_dir=$(mktemp -d /tmp/temp-igz-tutorials.XXXXXXXXXX) trap "{ rm -rf $temp_dir; }" EXIT -echo "Copying to temporary directory '${temp_dir}'..." +echo "Copying files to a temporary directory '${temp_dir}'..." # Get updated tutorials tar_url="${git_base_url}/archive/${branch}.tar.gz" @@ -111,7 +112,8 @@ if [ -z "${dry_run}" ]; then echo "Copying files to '${dest_dir}'..." cp -rf "${temp_dir}/"!(igz-tutorials-get.sh|update-tutorials.ipynb) "${dest_dir}" else - echo "Files that will be copied to '${dest_dir}':" + # Dry run + echo "Identified the following files to copy to '${dest_dir}':" find "${temp_dir}/"!(igz-tutorials-get.sh|update-tutorials.ipynb) -not -path '*/\.*' -type f -printf "%p\n" | sed -e "s|^${temp_dir}/|./|" fi diff --git a/update-tutorials.ipynb b/update-tutorials.ipynb index beae3106..8e6bac9f 100644 --- a/update-tutorials.ipynb +++ b/update-tutorials.ipynb @@ -11,13 +11,17 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "You can use the provided [**igz-tutorials-get.sh**](https://github.com/v3io/tutorials/blob/master/igz-tutorials-get.sh) script to update the tutorial notebooks to the latest stable version available on [GitHub](https://github.com/v3io/tutorials/). The script copies the file and directories of latest tag corresponding to the current platform version to the **/v3io/users/<username>** directory, where `` is the name of the running service user (`$V3IO_USERNAME`) unless you pass another username.\n", + "You can use the provided [**igz-tutorials-get.sh**](./igz-tutorials-get.sh) script to get updated platform tutorials from the [tutorials GitHub repository](https://github.com/v3io/tutorials/).\n", + "By default, the script retrieves the files from the latest release that matches the current platform version.\n", + "The files are copied to the **/v3io/users/<username>** directory, where `` is the name of the running user (`$V3IO_USERNAME`) unless you pass another username as an argument to the script.\n", "\n", "> **Note:**\n", - "> - Before running the script, close all GitHub tutorial files that might be updated, such as **welcome.ipynb**, **README.md**, or any of the **demo** or **data-ingestion-and-preparation** notebooks.
\n", + "> - Before running the script, close all GitHub tutorial files that might be updated, including all tutorial notebooks and other files.
\n", "> The script doesn't overwrite the current notebook (**update-tutorials.ipynb**) and the **igz-tutorials-get.sh** script nor any custom file or directory that you created.\n", - "> - You can use the ``--dry-run` parameter to list the files that are going to be updated without overwriting the existing files.\n", - "> - You can override the branch by spacifying the `-b` or `--branch` parameter follows by the branch name. Make sure that the updated notebooks match your version of the Iguazio Data Science Platform. If in doubt, consult the [customer-success team](mailto:support@iguazio.com)." + "> - You can use the `--dry-run` flag to list the files to update without copying any files to the platform.\n", + "> - You can override the default Git branch for the update by setting the `-b|--branch` flag to a specific branch name.\n", + "> When using this option, ensure that the files in the specified branch match your version of the platform.\n", + "> If in doubt, consult Iguazio's [customer-success team](mailto:support@iguazio.com)." ] }, { @@ -37,20 +41,22 @@ "output_type": "stream", "text": [ "igz-tutorials-get.sh:\n", - "Retrieves latest-release Iguazio Data Science Platform tutorial files from the v3io/tutorials GitHub repo.\n", + "Retrieves updated Iguazio Data Science Platform tutorial files from the v3io/tutorials GitHub repository.\n", "USAGE: igz-tutorials-get.sh [OPTIONS] []\n", "PARAMETERS:\n", - " - Username, which determines the directory to which to copy the files - /v3io/users/.\n", - " Default = $V3IO_USERNAME (if set to a non-empty string)\n", + " - Username, which determines the directory to which to copy the\n", + " retrieved files (/v3io/users/).\n", + " Default: $V3IO_USERNAME, if set to a non-empty string.\n", "OPTIONS:\n", " -h|--help - Display this message and exit.\n", - " -b|--branch - Branch name (default is latest tag of current platform version).\n", - " --dry-run - Do not update the files, but rather show the pending changes.\n" + " -b|--branch - Git branch name. Default: The latest release branch that\n", + " matches the current platform version.\n", + " --dry-run - Show files to update but don't execute the update.\n" ] } ], "source": [ - "!/User/igz-tutorials-get.sh -h" + "!/User/igz-tutorials-get.sh --help" ] }, { diff --git a/welcome.ipynb b/welcome.ipynb index d8b30f82..6d8a577b 100644 --- a/welcome.ipynb +++ b/welcome.ipynb @@ -119,9 +119,12 @@ "source": [ "## End-to-End Use-Case Applications (Demos)\n", "\n", - "Iguazio provides full end-to-end use-case applications (demos) that demonstrate how to use the platform and related tools to address data science requirements for different industries and implementations. These demos are available in the [**MLRun demos repository**](https://github.com/mlrun/demos).\n", - "\n", - "You can get the latest demos from the GitHub repository by running the following command:" + "Iguazio provides full end-to-end use-case applications (demos) that demonstrate how to use the platform and related tools to address data science requirements for different industries and implementations.\n", + "These demos are available in the [MLRun demos repository](https://github.com/mlrun/demos).\n", + "Use the provided [**get-additional-demos.sh**](./get-additional-demos.sh) script to get updated demos from this repository.\n", + "By default, the script retrieves the files from the latest release that matches the version of the platform's running MLRun service.\n", + "The files are copied to the **/v3io/users/<username>/demos** directory, where `` is the name of the running user (`$V3IO_USERNAME`) unless you set the `-u|--user` flag to another username.\n", + "> **Note:** Before running the script, close any open files in the **demos** directory." ] }, { @@ -134,6 +137,22 @@ "!/User/get-additional-demos.sh" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "For full usage instructions, run the script with the `-h` or `--help` flag:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!/User/get-additional-demos.sh --help" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -297,7 +316,7 @@ "\n", "You can find more information and resources in the MLRun documentation:\n", "
\n", - "**▶ [Open the MLRun documentation](https://mlrun.readthedocs.io)**\n", + "**▶ [View the MLRun documentation](https://mlrun.readthedocs.io)**\n", "\n", "You might also find the folloiwng resources useful:\n", "\n", @@ -339,9 +358,10 @@ "metadata": {}, "source": [ "\n", - "### Updating the Tutorial Notebooks to the Latest Version\n", + "### Updating the Tutorial Notebooks\n", "\n", - "You can use the provided **igz-tutorials-get.sh** script to update the tutorial notebooks to the latest stable version available on [GitHub](https://github.com/v3io/tutorials/).\n", + "You can use the provided **igz-tutorials-get.sh** script to get updated platform tutorials from the [tutorials GitHub repository](https://github.com/v3io/tutorials/).\n", + "By default, the script retrieves the files from the latest release that matches the current platform version.\n", "For details, see the [**update-tutorials.ipynb**](update-tutorials.ipynb) notebook." ] }, From 3289bdcbf69efb5eba826411e0547a426b5b8d42 Mon Sep 17 00:00:00 2001 From: Sharon Lifshitz Date: Tue, 22 Dec 2020 09:43:14 +0200 Subject: [PATCH 06/15] [DOC] get-additional-demos.sh: fix help text (PR #308 review( --- get-additional-demos.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/get-additional-demos.sh b/get-additional-demos.sh index 4c30ba20..ce35ca97 100755 --- a/get-additional-demos.sh +++ b/get-additional-demos.sh @@ -29,7 +29,7 @@ OPTIONS: --dry-run - Show files to update but don't execute the update. --no-backup - Don't back up the existing demos directory before the update. Default: Back up the existing demos directory to a - /v3io/users//demos.old./ directory." + /v3io/users//demos.old// directory." error_exit() { From 6f3279b921ce9c0b101481c8b33dc586ba3d345a Mon Sep 17 00:00:00 2001 From: Sharon Lifshitz Date: Tue, 22 Dec 2020 09:55:20 +0200 Subject: [PATCH 07/15] [DOC] get-additional-demos.sh updates to `mlrun` package version (PR #308 review) --- README.md | 2 +- get-additional-demos.sh | 4 ++-- welcome.ipynb | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index a6a28e6a..3db3f09d 100644 --- a/README.md +++ b/README.md @@ -66,7 +66,7 @@ Start out by running the getting-started tutorial to familiarize yourself with t Iguazio provides full end-to-end use-case applications (demos) that demonstrate how to use the platform and related tools to address data science requirements for different industries and implementations. These demos are available in the [MLRun demos repository](https://github.com/mlrun/demos). Use the provided [**get-additional-demos.sh**](./get-additional-demos.sh) script to get updated demos from this repository. -By default, the script retrieves the files from the latest release that matches the version of the platform's running MLRun service. +By default, the script retrieves the files from the latest release that matches the version of the installed `mlrun` package (see [Installing and Updating the MLRun Python Package](#mlrun-python-pkg-install-n-update)). The files are copied to the **/v3io/users/<username>/demos** directory, where `` is the name of the running user (`$V3IO_USERNAME`) unless you set the `-u|--user` flag to another username. > **Note:** Before running the script, close any open files in the **demos** directory. diff --git a/get-additional-demos.sh b/get-additional-demos.sh index ce35ca97..e1151f9f 100755 --- a/get-additional-demos.sh +++ b/get-additional-demos.sh @@ -19,13 +19,13 @@ USAGE: ${SCRIPT} [OPTIONS] OPTIONS: -h|--help - Display this message and exit. -b|--branch - Git branch name. Default: The latest release branch that - matches the version of the running MLRun service. + matches the version of the installed 'mlrun' package. -u|--user - Username, which determines the directory to which to copy the retrieved demo files (/v3io/users/). Default: \$V3IO_USERNAME, if set to a non-empty string. --mlrun-ver - The MLRun version for which to get demos; determines the Git branch from which to get the demos, unless -b|--branch is set. - Default: The version of the running MLRun service. + Default: The version of the installed 'mlrun' package. --dry-run - Show files to update but don't execute the update. --no-backup - Don't back up the existing demos directory before the update. Default: Back up the existing demos directory to a diff --git a/welcome.ipynb b/welcome.ipynb index 6d8a577b..dd5e08a3 100644 --- a/welcome.ipynb +++ b/welcome.ipynb @@ -122,7 +122,7 @@ "Iguazio provides full end-to-end use-case applications (demos) that demonstrate how to use the platform and related tools to address data science requirements for different industries and implementations.\n", "These demos are available in the [MLRun demos repository](https://github.com/mlrun/demos).\n", "Use the provided [**get-additional-demos.sh**](./get-additional-demos.sh) script to get updated demos from this repository.\n", - "By default, the script retrieves the files from the latest release that matches the version of the platform's running MLRun service.\n", + "By default, the script retrieves the files from the latest release that matches the version of the installed `mlrun` package (see [Installing and Updating the MLRun Python Package](#mlrun-python-pkg-install-n-update)).\n", "The files are copied to the **/v3io/users/<username>/demos** directory, where `` is the name of the running user (`$V3IO_USERNAME`) unless you set the `-u|--user` flag to another username.\n", "> **Note:** Before running the script, close any open files in the **demos** directory." ] From fcb5483a865f4dc65b9ca73fc32a511558a0b368 Mon Sep 17 00:00:00 2001 From: Sharon Lifshitz Date: Tue, 22 Dec 2020 12:17:25 +0200 Subject: [PATCH 08/15] Rename get-additional-demos.sh update-demos.sh --- README.md | 6 +++--- get-additional-demos.sh => update-demos.sh | 0 welcome.ipynb | 6 +++--- 3 files changed, 6 insertions(+), 6 deletions(-) rename get-additional-demos.sh => update-demos.sh (100%) diff --git a/README.md b/README.md index 3db3f09d..2e00d7ba 100644 --- a/README.md +++ b/README.md @@ -65,7 +65,7 @@ Start out by running the getting-started tutorial to familiarize yourself with t Iguazio provides full end-to-end use-case applications (demos) that demonstrate how to use the platform and related tools to address data science requirements for different industries and implementations. These demos are available in the [MLRun demos repository](https://github.com/mlrun/demos). -Use the provided [**get-additional-demos.sh**](./get-additional-demos.sh) script to get updated demos from this repository. +Use the provided [**update-demos.sh**](./update-demos.sh) script to get updated demos from this repository. By default, the script retrieves the files from the latest release that matches the version of the installed `mlrun` package (see [Installing and Updating the MLRun Python Package](#mlrun-python-pkg-install-n-update)). The files are copied to the **/v3io/users/<username>/demos** directory, where `` is the name of the running user (`$V3IO_USERNAME`) unless you set the `-u|--user` flag to another username. > **Note:** Before running the script, close any open files in the **demos** directory. @@ -73,14 +73,14 @@ The files are copied to the **/v3io/users/<username>/demos** directory, wh ```python # Get additional demos -!/User/get-additional-demos.sh +!/User/update-demos.sh ``` For full usage instructions, run the script with the `-h` or `--help` flag: ```python -!/User/get-additional-demos.sh --help +!/User/update-demos.sh --help ```
Demo
diff --git a/get-additional-demos.sh b/update-demos.sh similarity index 100% rename from get-additional-demos.sh rename to update-demos.sh diff --git a/welcome.ipynb b/welcome.ipynb index dd5e08a3..d6a509ec 100644 --- a/welcome.ipynb +++ b/welcome.ipynb @@ -121,7 +121,7 @@ "\n", "Iguazio provides full end-to-end use-case applications (demos) that demonstrate how to use the platform and related tools to address data science requirements for different industries and implementations.\n", "These demos are available in the [MLRun demos repository](https://github.com/mlrun/demos).\n", - "Use the provided [**get-additional-demos.sh**](./get-additional-demos.sh) script to get updated demos from this repository.\n", + "Use the provided [**update-demos.sh**](./update-demos.sh) script to get updated demos from this repository.\n", "By default, the script retrieves the files from the latest release that matches the version of the installed `mlrun` package (see [Installing and Updating the MLRun Python Package](#mlrun-python-pkg-install-n-update)).\n", "The files are copied to the **/v3io/users/<username>/demos** directory, where `` is the name of the running user (`$V3IO_USERNAME`) unless you set the `-u|--user` flag to another username.\n", "> **Note:** Before running the script, close any open files in the **demos** directory." @@ -134,7 +134,7 @@ "outputs": [], "source": [ "# Get additional demos\n", - "!/User/get-additional-demos.sh" + "!/User/update-demos.sh" ] }, { @@ -150,7 +150,7 @@ "metadata": {}, "outputs": [], "source": [ - "!/User/get-additional-demos.sh --help" + "!/User/update-demos.sh --help" ] }, { From 99b29522172f0c97f4ee35652f8d7c98625dc61d Mon Sep 17 00:00:00 2001 From: Sharon Lifshitz Date: Tue, 22 Dec 2020 17:33:26 +0200 Subject: [PATCH 09/15] [DOC] platform-overview.ipynb: minor doc edits --- platform-overview.ipynb | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/platform-overview.ipynb b/platform-overview.ipynb index 2070d363..963bf72b 100644 --- a/platform-overview.ipynb +++ b/platform-overview.ipynb @@ -117,9 +117,8 @@ "One of the most important and challenging areas of managing a data science environment is the ability to track experiments.\n", "Data scientists need a simple way to track and view current and historical experiments along with the metadata that is associated with each experiment. This capability is critical for comparing different runs, and eventually helps to determine the best model and configuration for production deployment.\n", "\n", - "The platform leverages the open-source [MLRun](https://github.com/mlrun/mlrun) library to help tackle these challenges. You can find examples of using MLRun in the MLRun demos.\n", - "\n", - "See the information about getting additional demos in [**welcome.ipynb**](welcome.ipynb#end-to-end-use-case-applications) (notebook) or [**README.md**](README.md#end-to-end-use-case-applications) (Markdown)." + "The platform leverages the open-source [MLRun](https://github.com/mlrun/mlrun) library to help tackle these challenges. You can find examples of using MLRun in the [MLRun demos](https://github.com/mlrun/demos/).\n", + "For information about retrieving and updating local copies of the MLRun demos, see [**welcome.ipynb**](welcome.ipynb#end-to-end-use-case-applications) (notebook) or [**README.md**](README.md#end-to-end-use-case-applications) (Markdown)." ] }, { From 64001ddaf49145e132c6004c260c633dda1a4a84 Mon Sep 17 00:00:00 2001 From: Sharon Lifshitz Date: Tue, 22 Dec 2020 21:50:16 +0200 Subject: [PATCH 10/15] [DOC] data-ingestion-and-preparation README edits & new grafwiz tutorial doc review + edit tutorial titles --- data-ingestion-and-preparation/README.ipynb | 76 +++-- data-ingestion-and-preparation/README.md | 60 ++-- .../grafana-grafwiz.ipynb | 279 +++++++++++++++--- data-ingestion-and-preparation/v3io-kv.ipynb | 2 +- .../v3io-streams.ipynb | 2 +- 5 files changed, 330 insertions(+), 89 deletions(-) diff --git a/data-ingestion-and-preparation/README.ipynb b/data-ingestion-and-preparation/README.ipynb index b9c76069..e6c84fc3 100644 --- a/data-ingestion-and-preparation/README.ipynb +++ b/data-ingestion-and-preparation/README.ipynb @@ -16,10 +16,10 @@ "- [Overview](#data-ingest-overview)\n", " - [Platform Data Containers](#platform-data-containers)\n", "- [Basic Flow](#data-ingest-basic-flow)\n", - "- [Iguazio's Platform Data Layer](#data-ingest-iguazio-platform-data-layer)\n", - " - [Iguazio's Data-Object API](#data-ingest-iguazio-data-object-api)\n", - " - [Iguazio's Key-Value API](#data-ingest-iguazio-kv-api)\n", - " - [Iguazio's Streaming API](#data-ingest-iguazio-streaming-api)\n", + "- [The Platform's Data Layer](#data-ingest-platform-data-layer)\n", + " - [The Data-Object Platform API](#data-ingest-platform-data-object-api)\n", + " - [The NoSQL (Key-Value) Platform API](#data-ingest-platform-nosql-api)\n", + " - [The Streaming Platform API](#data-ingest-platform-streaming-api)\n", "- [Reading from External Database](#data-ingest-external-dbs)\n", " - [Using Spark over JDBC](#data-ingest-spark-over-jdbs)\n", " - [Using SQLAlchemy](#data-ingest-sqlalchemy)\n", @@ -38,7 +38,7 @@ "- [Getting Data from AWS S3 Using curl](data-ingest-s3-curl)\n", "- [Running Distributed Python with Dask](#data-ingest-dask)\n", "- [Running DataFrames on GPUs using NVIDIA cuDF](#data-ingest-gpu)\n", - "- [Creating Dashboards with Grafana](#data-grafana)" + "- [Visualizing Data with Grafana](#data-ingest-grafana)" ] }, { @@ -55,7 +55,8 @@ "\n", "This tutorial provides an overview of various methods for collecting, storing, and manipulating data in the platform, and refers to sample tutorial notebooks that demonstrate how to use these methods.
\n", "For an in-depth overview of the platform and how it can be used to implement a full data science workflow, see the [**platform-overview**](../platform-overview.ipynb) tutorial notebook.\n", - "For full end-to-end platform use-case application demos, see the [**welcome notebook**](../welcome.ipynb#end-to-end-use-case-applications)\n", + "
\n", + "For information about the available full end-to-end platform use-case application demos, see the [**welcome**](../welcome.ipynb#end-to-end-use-case-applications) notebook or the matching [**README.md**](../README.md#end-to-end-use-case-applications) file.\n", "\n", "
\"pipeline-diagram\"
" ] @@ -118,79 +119,84 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "" + "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Iguazio's Platform Data Layer" + "## The Platform's Data Layer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "" + "\n", + "\n", + "The platform features an extremely fast and secure data layer that supports SQL, NoSQL, time-series databases, files (simple objects), and streaming, and exposes multiple APIs for working with the different data types — including [simple-object](#data-ingest-platform-data-object-api), [NoSQL (\"key-value\")](#data-ingest-platform-nosql-api), and [streaming](#data-ingest-platform-streaming-api) APIs." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "### Iguazio's Data-Object API" + "### The Data-Object Platform API" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "The platform’s Simple-Object API enables performing simple data-object operations that resembles Amazon’s Simple Storage Service (S3) API. In addition to the S3-like capabilities, the Simple-Object Web API enables appending data to existing objects. The [**v3io-objects notebook**](v3io-objects.ipynb) demonstrates the API usage." + "The platform’s Simple-Object API enables performing simple data-object and container operations that resemble the Amazon Web Services (AWS) Simple Storage Service (S3) API.\n", + "In addition to the S3-like capabilities, the Simple-Object Web API enables appending data to existing objects.\n", + "For more information and API usage examples, see the [**v3io-objects**](v3io-objects.ipynb) tutorial." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "" + "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "### Iguazio's Key-Value API" + "### The NoSQL (Key-Value) Platform API" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "The platform’s Key-Value (KV) API provides access to the NoSQL database service, which enables storing and consuming data in a tabular format. The [**v3io-kv notebook**](v3io-kv.ipynb) demonstrates the API usage." + "The platform’s NoSQL (a.k.a. Key-Value/KV) API provides access to the platform's NoSQL data store (database service), which enables storing and consuming data in a tabular format.\n", + "For more information and API usage examples, see the [**v3io-kv**](v3io-kv.ipynb) tutorial." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "" + "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "### Iguazio's Streaming API" + "### The Streaming Platform API" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "The platform’s Streaming API enables working with data in the platform as streams. The [**v3io-streams notebook**](v3io-streams.ipynb) demonstrates the API usage.\n", - "\n", - "For more general information see [working with Streams](#data-ingest-streams) section." + "The platform’s Streaming API enables working with data in the platform as streams.\n", + "For more information and API usage examples, see the [**v3io-streams**](v3io-streams.ipynb) tutorial.\n", + "In addition, see the [Working with Streams](#data-ingest-streams) section in the current tutorial for general information about different methods for working with data streams in the platform." ] }, { @@ -346,9 +352,11 @@ "The platform's streaming engine is currently available via the platform's [Streaming Web API](https://www.iguazio.com/docs/latest-release/reference/api-reference/web-apis/streaming-web-api/).
\n", "In addition, the platform's Spark-Streaming Integration API enables using the Spark Streaming API to work with platform streams, as explained in the next section ([Using Spark Streaming](#data-ingest-streams-spark)).\n", "\n", - "The [**v3io-streams notebook**](v3io-streams.ipynb) demonstrates basic usage of the streaming API.\n", + "The [**v3io-streams**](v3io-streams.ipynb) tutorial demonstrates basic usage of the streaming API.\n", "\n", - "The [**model meployment with streaming demo**](https://github.com/mlrun/demo-model-deployment-with-streaming) demo application includes an example of a Nuclio function that uses platform streams." + "" ] }, { @@ -556,20 +564,26 @@ ] }, { - "source": [ - "" - ], "cell_type": "markdown", - "metadata": {} + "metadata": {}, + "source": [ + "" + ] }, { + "cell_type": "markdown", + "metadata": {}, "source": [ - "## Creating Dashboards with Grafana\n", + "## Visualizing Data with Grafana\n", "\n", - "You can create a Grafana dashboard programmatically using the Iguazio API. This allows you to define a dashboard that reads from the Iguazio data layer and display tables and charts of this data. The [**grafana-grafwiz example**](grafana-grafwiz.ipynb) demonstrates how to do that." - ], - "cell_type": "markdown", - "metadata": {} + "The platform has a Grafana service with predefined dashboards that leverage the monitoring service to display monitoring data, such as performance statistics, for application services.\n", + "You can also define custom Grafana dashboards for monitoring, visualizing, and understanding data stored in the platform, such as time-series metrics and NoSQL data.\n", + "You can read and analyze data from the platform's data store and visualize it on Grafana dashboards in the desired formats, such as tables and graphs.\n", + "This can be done by using the custom `iguazio` data source, or by using a Prometheus data source for running Prometheus queries on platform TSDB tables.\n", + "You can also issue data alerts and create, explore, and share dashboards.\n", + "\n", + "You can use Iguazio's [grafwiz](https://github.com/v3io/grafwiz) Python library to create an deploy Grafana dashboards programmatically, as demonstrated in the [**grafana-grafwiz**](grafana-grafwiz.ipynb) tutorial." + ] } ], "metadata": { @@ -593,4 +607,4 @@ }, "nbformat": 4, "nbformat_minor": 4 -} \ No newline at end of file +} diff --git a/data-ingestion-and-preparation/README.md b/data-ingestion-and-preparation/README.md index 59157065..f51fde03 100644 --- a/data-ingestion-and-preparation/README.md +++ b/data-ingestion-and-preparation/README.md @@ -5,10 +5,10 @@ Learn about different methods for ingesting data into the Iguazio Data Science P - [Overview](#data-ingest-overview) - [Platform Data Containers](#platform-data-containers) - [Basic Flow](#data-ingest-basic-flow) -- [Iguazio's Platform Data Layer](#data-ingest-iguazio-platform-data-layer) - - [Iguazio's Data-Object API](#data-ingest-iguazio-data-object-api) - - [Iguazio's Key-Value API](#data-ingest-iguazio-kv-api) - - [Iguazio's Streaming API](#data-ingest-iguazio-streaming-api) +- [The Platform's Data Layer](#data-ingest-platform-data-layer) + - [The Data-Object Platform API](#data-ingest-platform-data-object-api) + - [The NoSQL (Key-Value) Platform API](#data-ingest-platform-nosql-api) + - [The Streaming Platform API](#data-ingest-platform-streaming-api) - [Reading from External Database](#data-ingest-external-dbs) - [Using Spark over JDBC](#data-ingest-spark-over-jdbs) - [Using SQLAlchemy](#data-ingest-sqlalchemy) @@ -27,7 +27,7 @@ Learn about different methods for ingesting data into the Iguazio Data Science P - [Getting Data from AWS S3 Using curl](data-ingest-s3-curl) - [Running Distributed Python with Dask](#data-ingest-dask) - [Running DataFrames on GPUs using NVIDIA cuDF](#data-ingest-gpu) -- [Creating Dashboards with Grafana](#data-grafana) +- [Visualizing Data with Grafana](#data-ingest-grafana) ## Overview @@ -39,7 +39,8 @@ The platform features a wide set of methods for manipulating and managing data, This tutorial provides an overview of various methods for collecting, storing, and manipulating data in the platform, and refers to sample tutorial notebooks that demonstrate how to use these methods.
For an in-depth overview of the platform and how it can be used to implement a full data science workflow, see the [**platform-overview**](../platform-overview.ipynb) tutorial notebook. -For full end-to-end platform use-case application demos, see the [**welcome notebook**](../welcome.ipynb#end-to-end-use-case-applications) +
+For information about the available full end-to-end platform use-case application demos, see the [**welcome**](../welcome.ipynb#end-to-end-use-case-applications) notebook or the matching [**README.md**](../README.md#end-to-end-use-case-applications) file.
pipeline-diagram
@@ -77,29 +78,34 @@ For detailed information and examples on how to set the data path for each inter The [**basic-data-ingestion-and-preparation**](basic-data-ingestion-and-preparation.ipynb) tutorial walks you through basic scenarios of ingesting data from external sources into the platform's data store and manipulating the data using different data formats. The tutorial includes an example of ingesting a CSV file from an AWS S3 bucket; converting it into a NoSQL table using Spark DataFrames; running SQL queries on the table; and converting the table into a Parquet file. - + -## Iguazio's Platform Data Layer +## The Platform's Data Layer - + -### Iguazio's Data-Object API +The platform features an extremely fast and secure data layer that supports SQL, NoSQL, time-series databases, files (simple objects), and streaming, and exposes multiple APIs for working with the different data types — including [simple-object](#data-ingest-platform-data-object-api), [NoSQL ("key-value")](#data-ingest-platform-nosql-api), and [streaming](#data-ingest-platform-streaming-api) APIs. -The platform’s Simple-Object API enables performing simple data-object operations that resembles Amazon’s Simple Storage Service (S3) API. In addition to the S3-like capabilities, the Simple-Object Web API enables appending data to existing objects. The [**v3io-objects notebook**](v3io-objects.ipynb) demonstrates the API usage. +### The Data-Object Platform API - +The platform’s Simple-Object API enables performing simple data-object and container operations that resemble the Amazon Web Services (AWS) Simple Storage Service (S3) API. +In addition to the S3-like capabilities, the Simple-Object Web API enables appending data to existing objects. +For more information and API usage examples, see the [**v3io-objects**](v3io-objects.ipynb) tutorial. -### Iguazio's Key-Value API + -The platform’s Key-Value (KV) API provides access to the NoSQL database service, which enables storing and consuming data in a tabular format. The [**v3io-kv notebook**](v3io-kv.ipynb) demonstrates the API usage. +### The NoSQL (Key-Value) Platform API - +The platform’s NoSQL (a.k.a. Key-Value/KV) API provides access to the platform's NoSQL data store (database service), which enables storing and consuming data in a tabular format. +For more information and API usage examples, see the [**v3io-kv**](v3io-kv.ipynb) tutorial. -### Iguazio's Streaming API + -The platform’s Streaming API enables working with data in the platform as streams. The [**v3io-streams notebook**](v3io-streams.ipynb) demonstrates the API usage. +### The Streaming Platform API -For more general information see [working with Streams](#data-ingest-streams) section. +The platform’s Streaming API enables working with data in the platform as streams. +For more information and API usage examples, see the [**v3io-streams**](v3io-streams.ipynb) tutorial. +In addition, see the [Working with Streams](#data-ingest-streams) section in the current tutorial for general information about different methods for working with data streams in the platform. ## Reading Data from External Databases @@ -180,9 +186,11 @@ You can use the platform's streaming engine to write data into a queue in a real The platform's streaming engine is currently available via the platform's [Streaming Web API](https://www.iguazio.com/docs/latest-release/reference/api-reference/web-apis/streaming-web-api/).
In addition, the platform's Spark-Streaming Integration API enables using the Spark Streaming API to work with platform streams, as explained in the next section ([Using Spark Streaming](#data-ingest-streams-spark)). -The [**v3io-streams notebook**](v3io-streams.ipynb) demonstrates basic usage of the streaming API. +The [**v3io-streams**](v3io-streams.ipynb) tutorial demonstrates basic usage of the streaming API. -The [**model meployment with streaming demo**](https://github.com/mlrun/demo-model-deployment-with-streaming) demo application includes an example of a Nuclio function that uses platform streams. + @@ -284,8 +292,14 @@ The [**gpu-cudf-vs-pd**](gpu-cudf-vs-pd.ipynb) tutorial demonstrates how to use > **Note:** To use the cuDF library, you need to create a RAPIDS Conda environment. > For more information, see the [**virtual-env**](../virtual-env.ipynb) tutorial. - + -## Creating Dashboards with Grafana +## Visualizing Data with Grafana -You can create a Grafana dashboard programmatically using the Iguazio API. This allows you to define a dashboard that reads from the Iguazio data layer and display tables and charts of this data. The [**grafana-grafwiz example**](grafana-grafwiz.ipynb) demonstrates how to do that. +The platform has a Grafana service with predefined dashboards that leverage the monitoring service to display monitoring data, such as performance statistics, for application services. +You can also define custom Grafana dashboards for monitoring, visualizing, and understanding data stored in the platform, such as time-series metrics and NoSQL data. +You can read and analyze data from the platform's data store and visualize it on Grafana dashboards in the desired formats, such as tables and graphs. +This can be done by using the custom `iguazio` data source, or by using a Prometheus data source for running Prometheus queries on platform TSDB tables. +You can also issue data alerts and create, explore, and share dashboards. + +You can use Iguazio's [grafwiz](https://github.com/v3io/grafwiz) Python library to create an deploy Grafana dashboards programmatically, as demonstrated in the [**grafana-grafwiz**](grafana-grafwiz.ipynb) tutorial. diff --git a/data-ingestion-and-preparation/grafana-grafwiz.ipynb b/data-ingestion-and-preparation/grafana-grafwiz.ipynb index 8cf4ebf4..8379b011 100644 --- a/data-ingestion-and-preparation/grafana-grafwiz.ipynb +++ b/data-ingestion-and-preparation/grafana-grafwiz.ipynb @@ -4,7 +4,55 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Make sure grafwiz is installed" + "# Generating a Grafana Dashboard with grafwiz\n", + "\n", + "This tutorial demonstrates how to use [grafwiz](https://github.com/v3io/grafwiz), Iguazio's open-source Python library for generating a Grafana dashboard programmatically." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "- [Setup](#grafwiz-setup)\n", + "- [Generating Data](#grafwiz-gen-data)\n", + "- [Creating a DataFrame with the Generated Data](#grafwiz-df-create)\n", + "- [Writing the Data to the Platform's Data Store](#grafwiz-write-to-data-store)\n", + "- [Adding a Platform Data Source to Grafana](#grafwiz-add-data-source)\n", + "- [Creating a Grafana Dashboard](#grafwiz-grafana-dashboard-create)\n", + "- [Adding Dashboard Visualization Elements](#grafwiz-add-dashboard-visualization-elements)\n", + "- [Deploying the Dashboard to Grafana](#grafwiz-grafana-dashboard-deploy)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup\n", + "\n", + "Initialize and configure your environment." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Installing grafwiz\n", + "\n", + "Ensure that the `grafwiz` Pyhton package is installed." ] }, { @@ -20,9 +68,34 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Add environment configurations and import the necessary libraries\n", - "* Please make sure you have a Grafana service, in order to create one, please go to Iguazio dashboard -> Service, Create new service\n", - "* To retrieve the url of Grafana service please use \"API\" tab, in the services dashboard" + "" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Creating a Grafana Service\n", + "\n", + "1. Ensure that you have a running platform Grafana service.\n", + " You can create such a service from the platform dashboard's **Services** page.\n", + "2. Copy the URL of your Grafana service from the **API** column in the services table on the **Services** dashboard page." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Defining Variables\n", + "\n", + "Define variables for your environment." ] }, { @@ -41,6 +114,22 @@ "rows = 3450" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Importing Libraries\n", + "\n", + "Import required libraries." + ] + }, { "cell_type": "code", "execution_count": null, @@ -56,14 +145,41 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Generate data for the dataframe" + "" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Creating a V3IO Frames Client\n", + "\n", + "Create a V3IO Frames client object." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "client = v3f.Client('framesd:8081',container=v3io_container)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "#### Date that will serve as an index for the time series and additional parameters" + "" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Generating Data\n", + "\n", + "Generate random data to visualize on the Grafana dashboard." ] }, { @@ -83,14 +199,23 @@ "\n", "time = sorted(generate_date(rows))\n", "volume = np.random.randint(low=100, high=10000, size=rows)\n", - "price = np.cumsum([0.0001] * rows + np.random.random(rows))\n" + "price = np.cumsum([0.0001] * rows + np.random.random(rows))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "#### Create a dataframe out of the generated index and add an index to the dataframe in order to make it accessible for tsdb" + "" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Creating a DataFrame with the Generated Data\n", + "\n", + "Store the generated data in a pandas DataFrame." ] }, { @@ -109,6 +234,13 @@ "stocks_df" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Define the `last_updated` column (attribute) as a DataFrame index column, which will be used to identify the ingestion times of the TSDB metric samples." + ] + }, { "cell_type": "code", "execution_count": null, @@ -124,7 +256,32 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "#### Create a client for v3io_frames create table and persist the dataframe in tsdb" + "" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Writing the Data to the Platform's Data Store\n", + "\n", + "Use the V3IO Frames API to write the data from the pandas DataFrame to TSDB and NoSQL tables in the platform's persistent data store." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Writing the Data to a TSDB Table\n", + "\n", + "Write the data from the DataFrame to a new platform TSDB table." ] }, { @@ -133,7 +290,6 @@ "metadata": {}, "outputs": [], "source": [ - "client = v3f.Client('framesd:8081',container=v3io_container)\n", "client.create(backend='tsdb', table=stocks_tsdb_table, rate='1/m', if_exists=1)\n", "client.write(backend='tsdb', table=stocks_tsdb_table, dfs=stocks_df_tsdb)" ] @@ -142,7 +298,16 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "#### Update the kv table in order of rows arrival to simulate real time changing data" + "" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Writing the Data to a NoSQL Table\n", + "\n", + "Write the data from the DataFrame to a new platform NoSQL table in order of rows arrival, to simulate real-time data consumption." ] }, { @@ -152,7 +317,7 @@ "outputs": [], "source": [ "expr_template = \"symbol='{symbol}';price='{price}';volume='{volume}';last_updated='{last_updated}'\"\n", - "# update NoSQL table with stock data\n", + "# Write the stock data to a NoSQL table\n", "for idx, record in stocks_df.iterrows():\n", " stock = {'symbol': sym, 'price': record['price'], 'volume': record['volume'], 'last_updated': record['last_updated']}\n", " expr = expr_template.format(**stock)\n", @@ -163,7 +328,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "#### Create a Datasource for grafana table and check that kv table exists and accessible" + "Infer the schema of the NoSQL table to verify that it can be accessed and displayed on the dashboard." ] }, { @@ -172,11 +337,7 @@ "metadata": {}, "outputs": [], "source": [ - "# Create datasources\n", - "DataSource(name='Iguazio').deploy(grafana_url, use_auth=True)\n", - "\n", - "# Verify the KV table can be shown\n", - "client = v3f.Client('framesd:8081', container=v3io_container)\n", + "# Infer the schema of the NoSQL table\n", "client.execute(backend='kv', table=stocks_kv_table, command='infer')" ] }, @@ -184,7 +345,42 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "#### Create a Dashboard define its datasource" + "" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Adding a Platform Data Source to Grafana\n", + "\n", + "Add an \"Iguazio\" data source for the platform's custom `iguazio` Grafana data source to your Grafana service." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Create a data source\n", + "DataSource(name='Iguazio').deploy(grafana_url, use_auth=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Creating a Grafana Dashboard\n", + "\n", + "Create a new Grafana dashboard that uses the platform's `iguazio` data source." ] }, { @@ -194,15 +390,23 @@ "outputs": [], "source": [ "# Create grafana dashboard\n", - "dash = Dashboard(\"stocks\", start='now-15m', dataSource='Iguazio', end='now')\n" + "dash = Dashboard(\"stocks\", start='now-15m', dataSource='Iguazio', end='now')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "#### Create a table and 2 graphs\n", - "Please note that you have to wait for couple of minutes to see the graphs being updated" + "" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Adding Dashboard Visualization Elements\n", + "\n", + "Create a table and two graphs for visualization the data from the NoSQL and TSDB tables on the Grafana dashboard." ] }, { @@ -211,25 +415,31 @@ "metadata": {}, "outputs": [], "source": [ - "# Create a table and log viewer in one row\n", + "# Create a table and log viewer for the NoSQL table in one row\n", "tbl = Table('Current Stocks Value', span=12).source(table=stocks_kv_table,fields=['symbol','volume', 'price', 'last_updated'],container=v3io_container)\n", "dash.row([tbl])\n", "\n", - "# Metrics row will\n", + "# Create a TSDB metrics graph\n", "metrics_row = [Graph(metric).series(table=stocks_tsdb_table, fields=[metric], container=v3io_container) for metric in ['price','volume']]\n", - "\n", - "dashboard = dash.row(metrics_row)\n", - "\n" + "dashboard = dash.row(metrics_row)" ] }, { - "cell_type": "code", - "execution_count": null, + "cell_type": "markdown", "metadata": {}, - "outputs": [], "source": [ - "# Deploy to Grafana\n", - "dash.deploy(grafana_url)" + "" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Deploying the Dashboard to Grafana\n", + "\n", + "Deploy the new Grafana dashboard to your Grafana service.\n", + "\n", + "> **Note:** Allow a few minutes for the dashboard loading to complete. " ] }, { @@ -237,7 +447,10 @@ "execution_count": null, "metadata": {}, "outputs": [], - "source": [] + "source": [ + "# Deploy to Grafana\n", + "dash.deploy(grafana_url)" + ] } ], "metadata": { diff --git a/data-ingestion-and-preparation/v3io-kv.ipynb b/data-ingestion-and-preparation/v3io-kv.ipynb index 8c426a4b..2df9bd35 100644 --- a/data-ingestion-and-preparation/v3io-kv.ipynb +++ b/data-ingestion-and-preparation/v3io-kv.ipynb @@ -4,7 +4,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Using Iguazio's Key-Value API" + "# Using the Platform's NoSQL (Key-Value) API" ] }, { diff --git a/data-ingestion-and-preparation/v3io-streams.ipynb b/data-ingestion-and-preparation/v3io-streams.ipynb index 99ec4ca8..6a1d6694 100644 --- a/data-ingestion-and-preparation/v3io-streams.ipynb +++ b/data-ingestion-and-preparation/v3io-streams.ipynb @@ -4,7 +4,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Using Iguazio's Streams API" + "# Using the Platform's Streaming API" ] }, { From 6bffdf2fd5d751cb5881b7f70e376f784cf8387e Mon Sep 17 00:00:00 2001 From: Sharon Lifshitz Date: Tue, 22 Dec 2020 22:15:47 +0200 Subject: [PATCH 11/15] [DOC] data-ingestion-and-preparation: v3io-objects.ipynb - edit title --- data-ingestion-and-preparation/v3io-objects.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-ingestion-and-preparation/v3io-objects.ipynb b/data-ingestion-and-preparation/v3io-objects.ipynb index 2d6bdfa9..006925cd 100644 --- a/data-ingestion-and-preparation/v3io-objects.ipynb +++ b/data-ingestion-and-preparation/v3io-objects.ipynb @@ -4,7 +4,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Using Iguazio's Data-Object API" + "# Using Paltform's Data-Object API" ] }, { From 4bad8908969dc689132e569a5f38c4fc3d83804f Mon Sep 17 00:00:00 2001 From: Sharon Lifshitz Date: Wed, 23 Dec 2020 00:04:50 +0200 Subject: [PATCH 12/15] [DOC] getting-started-tutorial: edit demo overviews & view function doc Doc review for PR #279 & PR #293. --- .../getting-started-tutorial.ipynb | 57 +++++++++---------- 1 file changed, 28 insertions(+), 29 deletions(-) diff --git a/getting-started-tutorial/getting-started-tutorial.ipynb b/getting-started-tutorial/getting-started-tutorial.ipynb index 32a3119a..12072eba 100644 --- a/getting-started-tutorial/getting-started-tutorial.ipynb +++ b/getting-started-tutorial/getting-started-tutorial.ipynb @@ -1556,13 +1556,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "#### View the Function Documentation" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ + "#### View the Function Documentation\n", + "\n", "Use the `doc` method to view the embedded documentation of the `describe` function." ] }, @@ -4030,28 +4025,32 @@ "source": [ "Congratulation. You've completed your first Iguazio Data Science Platform tutorial!.\n", "\n", - "- To learn more about deploying live endpoints and concept drift, refer to the **Network operations demo**:\n", - "
\n", - " \n", - " \n", - "
\n", - "
Open locally\n", - " 		\n", - "
View on GitHub\n", - "
\n", - "- For an example of distributed training of image classification pipeline using Tensorflow (versions 1 or 2), Keras, and Horovod, refer to the **Image classification with distributed training demo**:\n", - "\n", - " \n", - " \n", - "
\n", - "
Open locally\n", - " 		\n", - " \n", - "
View on GitHub\n", - "
\n", - "- To learn about bringing data into the system and data engineering using specific technologies, such as Spark, Dask, SQL queries and RAPIDS, refer to the [**data ingestion and preparation overview**](../data-ingestion-and-preparation/README.ipynb) .\n", - "\n", - "For more information about these demos and additional demos, see [**welcome.ipynb**](welcome.ipynb#end-to-end-use-case-applications) (notebook) or [**README.md**](README.md#end-to-end-use-case-applications) (Markdown)" + "- To learn more about deploying live endpoints and concept drift, refer to the **network-operations (NetOps) demo**:\n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
\n", + "
Open locally\n", + " 		\n", + "
View on GitHub\n", + "
\n", + "\n", + "- For an example of distributed training of an image-classification pipeline using TensorFlow (versions 1 or 2), Keras, and Horovod, refer to the **image-classification with distributed training demo**:\n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
\n", + "
Open locally\n", + " 		\n", + " \n", + "
View on GitHub\n", + "
\n", + "- For more information about these demos and additional demos, refer to the [**welcome.ipynb**](welcome.ipynb#end-to-end-use-case-applications) notebook or matching [**README.md**](README.md#end-to-end-use-case-applications) file.\n", + "- To learn about ingesting data into the platform and engineering the data using different technologies — such as Spark, Dask, SQL queries and RAPIDS — refer to the **data ingestion and preparation overview** — [notebook](../data-ingestion-and-preparation/README.ipynb) / [Markdown](../data-ingestion-and-preparation/README.md)." ] } ], From fae2e372e522a6dfba2b5f2a3f2f1d69f8af0535 Mon Sep 17 00:00:00 2001 From: Sharon Lifshitz Date: Wed, 23 Dec 2020 01:01:55 +0200 Subject: [PATCH 13/15] [DOC-REVIEW] getting-started-tutorial: edit the model-serving doc (Step 6) Doc review for PR #274 --- .../getting-started-tutorial.ipynb | 42 +++++++++---------- 1 file changed, 19 insertions(+), 23 deletions(-) diff --git a/getting-started-tutorial/getting-started-tutorial.ipynb b/getting-started-tutorial/getting-started-tutorial.ipynb index 12072eba..274949fb 100644 --- a/getting-started-tutorial/getting-started-tutorial.ipynb +++ b/getting-started-tutorial/getting-started-tutorial.ipynb @@ -2845,13 +2845,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Step 6: Serve Your Model" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ + "## Step 6: Serve Your Model\n", + "\n", "Add to your project a real-time model-server function (`serving`) via the [`model_server`](https://github.com/mlrun/functions/tree/master/model_server) MLRun marketplace function." ] }, @@ -2875,14 +2870,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Deploy the serving function" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Set the trained model and deploy the serving function" + "### Deploy the Serving Function\n", + "\n", + "Set the trained model and deploy the serving function." ] }, { @@ -2921,7 +2911,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "You can now run the model serving function. MLRun leverages Nuclio on Kubernetes, Nuclio scales up and down automatically and will scale to zero if remains unused. Therefore, unlike VMs that consume resources and require you to delete the endpoint when unused, here you do not have to worry about deleting this serving function from resource and cost point of view." + "You can now run the model-serving function.\n", + "MLRun leverages Nuclio on Kubernetes: Nuclio scales up and down automatically and scales to zero when unused.\n", + "Therefore, unlike virtual machines (VMs) that consume resources and require you to delete the endpoint when unused, here you don't need to worry about deleting this serving function because of resource costs." ] }, { @@ -2935,14 +2927,14 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Test the model from the notebook" + "### Test the Model from the Notebook" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Now define a test vector of inputs `test_data` and send them to the serving function `predict_url`" + "Now, define a test inputs vector (see `test_data`), and send the inputs to the serving function's `/predict` endpoint (see `predict_url`)." ] }, { @@ -3004,14 +2996,15 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Test via the a model tester function" + "### Test the Model Using A Model-Tester Function" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "A much more robust way to test your serving function is by creating a test function that sends a test dataset and compares the output to the expected output. Add to your project a REST model API tester function (`serving-tester`) which runs the model_server function with a specified table via the [`model_server_tester`](https://github.com/mlrun/functions/tree/master/model_server_tester) MLRun marketplace function." + "A much more robust way to test your serving function is by creating a test function that sends a test data set and compares the output of the model-serving function to the expected output.\n", + "To do this, add to your project a REST model-tester function (`serving-tester`) that runs the model-server function with a specific table via the [`model_server_tester` MLRun marketplace function](https://github.com/mlrun/functions/tree/master/model_server_tester), which issues HTTP calls to the model server." ] }, { @@ -3277,14 +3270,16 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Test the model via the UI" + "### Test the Model from the Dashboard" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "You can also run the model-serving function via the UI. The function name is the project name concatenated with `-sklearn server`. Use the following code to obtain the function name" + "You can also run the model-serving function from the MLRun dashboard.\n", + "The function name is a concatenation of the project name and `-model-server`.\n", + "Use the following code to obtain the function name:" ] }, { @@ -3315,7 +3310,8 @@ "```json\n", "{\"instances\": [[5.1, 3.5, 1.4, 0.2]]}\n", "```\n", - "You can perform these steps from the platform dashboard: select **Projects** from the side navigation menu, select your project (\"getting-started-iris-iguazio\"), and then select the `getting-stated-iris-iguazio-sklearn-server` function and execute the required steps, as demonstrated in the following dashboard flow:\n", + "You can perform these steps from the platform dashboard:\n", + "select **Projects** from the side navigation menu; select your project (\"getting-started-iris-iguazio\"); and then select the `getting-stated-iris-iguazio-sklearn-server` function and execute the required steps, as demonstrated in the following dashboard flow:\n", "

\n", "![getting_started-pipeline](./assets/getting_started-model_serving_test.gif)" ] From 6611b5c1df0249510308088a8f655613571e823a Mon Sep 17 00:00:00 2001 From: Sharon Lifshitz Date: Wed, 23 Dec 2020 10:27:23 +0200 Subject: [PATCH 14/15] [DOC] v3io_objects.ipynb: fix typo in title (PR #311) --- data-ingestion-and-preparation/v3io-objects.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data-ingestion-and-preparation/v3io-objects.ipynb b/data-ingestion-and-preparation/v3io-objects.ipynb index 006925cd..1a11d99b 100644 --- a/data-ingestion-and-preparation/v3io-objects.ipynb +++ b/data-ingestion-and-preparation/v3io-objects.ipynb @@ -4,7 +4,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Using Paltform's Data-Object API" + "# Using the Platform's Data-Object API" ] }, { From 6eb8345f6eff1b43548f44c8ae596b5bd1d194fc Mon Sep 17 00:00:00 2001 From: Sharon Lifshitz Date: Wed, 23 Dec 2020 21:02:30 +0200 Subject: [PATCH 15/15] [DOC] grafana-grafwiz.ipynb fixes & improvements (PR #311) --- .../grafana-grafwiz.ipynb | 22 ++++++++++--------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/data-ingestion-and-preparation/grafana-grafwiz.ipynb b/data-ingestion-and-preparation/grafana-grafwiz.ipynb index 8379b011..04d12e70 100644 --- a/data-ingestion-and-preparation/grafana-grafwiz.ipynb +++ b/data-ingestion-and-preparation/grafana-grafwiz.ipynb @@ -52,7 +52,7 @@ "source": [ "### Installing grafwiz\n", "\n", - "Ensure that the `grafwiz` Pyhton package is installed." + "Run the following code to ensure that the `grafwiz` Python package is installed, and the restart the Jupyter kernel." ] }, { @@ -79,7 +79,7 @@ "\n", "1. Ensure that you have a running platform Grafana service.\n", " You can create such a service from the platform dashboard's **Services** page.\n", - "2. Copy the URL of your Grafana service from the **API** column in the services table on the **Services** dashboard page." + "2. Copy the URL of your Grafana service from the service-name link on the **Services** dashboard page." ] }, { @@ -95,7 +95,9 @@ "source": [ "### Defining Variables\n", "\n", - "Define variables for your environment." + "Define variables for your environment.\n", + "\n", + "> **Note:** Replace the `` placeholder with the URL of your Grafana service, as copied in the previous step." ] }, { @@ -106,7 +108,7 @@ "source": [ "import os\n", "\n", - "grafana_url = 'http://grafana' \n", + "grafana_url = '' # TODO: Replace with the API URL of your Grafana API service.\n", "v3io_container = 'users'\n", "stocks_kv_table = os.path.join(os.getenv(\"V3IO_USERNAME\"),'stocks_kv_table')\n", "stocks_tsdb_table = os.path.join(os.getenv(\"V3IO_USERNAME\"),'stocks_tsdb_table')\n", @@ -406,7 +408,9 @@ "source": [ "## Adding Dashboard Visualization Elements\n", "\n", - "Create a table and two graphs for visualization the data from the NoSQL and TSDB tables on the Grafana dashboard." + "Create a table for the NoSQL table and graphs for each of the metrics in the TSDB table, to be used for visualizing the data on the Grafana dashboard.\n", + "\n", + "> **Note:** It might take a few minutes for the graphs to be updated with the data." ] }, { @@ -419,9 +423,9 @@ "tbl = Table('Current Stocks Value', span=12).source(table=stocks_kv_table,fields=['symbol','volume', 'price', 'last_updated'],container=v3io_container)\n", "dash.row([tbl])\n", "\n", - "# Create a TSDB metrics graph\n", + "# Create TSDB-metric graphs\n", "metrics_row = [Graph(metric).series(table=stocks_tsdb_table, fields=[metric], container=v3io_container) for metric in ['price','volume']]\n", - "dashboard = dash.row(metrics_row)" + "dash.row(metrics_row)" ] }, { @@ -437,9 +441,7 @@ "source": [ "## Deploying the Dashboard to Grafana\n", "\n", - "Deploy the new Grafana dashboard to your Grafana service.\n", - "\n", - "> **Note:** Allow a few minutes for the dashboard loading to complete. " + "Deploy the new Grafana dashboard to your Grafana service." ] }, {