Home
Welcome to the Shifu wiki!
Shifu is an open-source, end-to-end machine learning and data mining framework built on top of Hadoop. Shifu is designed for data scientists, simplifying the life-cycle of building machine learning models. While originally built for fraud modeling, Shifu is generalized for many other modeling domains.
Shifu provides a simple command-line interface for each step of the model building process, including
- Statistic calculation & variable selection to determine the most predictive variables in your data
- Variable normalization
- Distributed variable selection based on sensitivity analysis
- Distributed neural network model training
- Distributed tree ensemble model training
- Post training analysis & model evaluation
Shifu’s fast Hadoop-based, distributed neural network / logistic regression / gradient boosted trees training can reduce model training time from days to hours on TB data sets. Shifu integrates with Pig/MapReduce workflows on Hadoop, and Shifu-trained models can be integrated into production code with standard PMML format or native format with a simple Java API. Shifu leverages Hadoop, Pig, Akka, Encog and other open source projects.
- Tutorial - Build Your First ML Model
- Machine Learning Pipeline in Shifu
- Distributed Neural Network Training in Shifu
- Distributed Tree Ensemble Model Training in Shifu
- Grid/Random Search Support in Shifu
- Variable Selection in Shifu
- Variable Binning in Shifu
- Variable Transform in Shifu
- Multiple Classification Support in Shifu
- Native Bagging Modeling Framework How to run Bagging in Shifu
- Assembling Models - Shifu combo
- CSV Format Support in Shifu
- Correlation Computing in Shifu
- Sampling & Filtering in Shifu Training Step
- Dropout Support in NN/GBDT
- Shifu Best Practices
- How to Deploy a Shifu Model to Production
- System Optimization to Accelerate Distributed Model Training
- [How to Tune a Good GBT Model]
- Distributed Logistic Regression in Shifu
- Segment Expansion for New Feature Generation
- Shifu How-To
- Shifu 0.2.X (old documents)
