In the past lesson, you learned about Machine Learning pipelines and how to implement a typical Machine Learning pipeline in Scikit-learn. The current lesson does not present any new content, it is only intended to put to work the knowledge you acquired in the previous lessons, while constructing a solution to a real world problem based on Machine Learning pipelines implemented with Scikit-learn.
We recommend you follow the process exemplified in the previous lesson "Machine Learning Pipelines", which consists of 8 clearly defined phases. If you get stuck at any point, you can always go back to the lesson "Machine Learning Pipelines" and re-read the material presented there on the particular phase where you got stuck. Everything you need to know to succeed in this Guided Project has already been presented on the previous lesson, so going back to review the material presented there is always a good idea.
In the previous lessons you "learned by being told", the current Guided Project is a great complement to that because now you "learn by doing." In this Guided Project, you are asked to choose a challenging Machine Learning problem that motivates you, to devise a Machine Learning solution following the phases presented in the previous lesson "Machine Learning Pipelines," and to implement the solution in Python using the Scikit-learn libraries. Next, we present the requirements your application must comply with.
- Input requirement: capacity to read a dataset stored on disk.
- Output requirement: capacity to save a Machine Learning pipeline to a file.
- Output requirement: capacity to output the estimations of a Machine Learning pipeline for a variety of problem instances.
- Reading external dataset from file.
- Creation of a Machine Learning pipeline containing a minimum of 2 steps.
- Training of Machine Learning pipeline.
- Evaluation of Machine Learning pipeline using cross validation.
- Saving the Machine Learning pipeline to a file on disk.
- Loading a Machine Learning pipeline from a file on disk.
- Classiying new problem instances with the Machine Learning pipeline read from file.
- Use Python as programming language.
- Use Pandas for reading the dataset into a dataframe.
- Use Scikit-learn for creating, training and evaluating the Machine Learning pipeline.
- Use a Scikit-learn pipeline for classifying new problem instances.
- Use the class sklearn.externals.joblib for saving and reading serialized pipelines.
- Software application implemented in one or several Python modules.
- Report explaining the problem, the dataset, the design of the solution, and the evaluation.
- Find an interesting dataset! Look in the Useful Resources section for sources of ideas.
- If you do not find a pre-existing dataset on the problem domain that you like, be creative: consider building the dataset yourself and donating the dataset to one of the public Machine Learning repositories.
- Break down the project into smaller tasks: the 8 phases presented in the previous lesson.
- Decide whether you will implement the software in a single Python module, or in several Python modules.
- If you feel completely lost and do not know where to start, start by executing the program presented in the previous lesson. Then follow the next suggestion in this list.
- Use an iterative approach: implement one phase per iteration.
- Reuse: adapt the code snippets of each phase, presented in the previous lesson, to your current problem.
Find some problem domain that motivates you, and ideally has a remarkable social and/or economic impact as well.These are a few ideas that you might want to consider:
- Malicious web site detection.
- Detection of spam.
- Diagnosis of Parkinson's disease.
- University of California at Irvine's Machine Learning Repository
- OpenML datasets
- Kaggle datasets
- Read dataset into Pandas dataframe and select the training set from it: 1 point.
- Pipeline formed, trained and evaluated: 3 points.
- Pipeline saved, read from disk and used for estimation of new instances: 2 point.
- Different experiments performed using different pipelines, to determine best choice: up to 2 points.