Skip to content

Latest commit

 

History

History
114 lines (88 loc) · 5.41 KB

README.md

File metadata and controls

114 lines (88 loc) · 5.41 KB
Raw

PMML4S-Spark

Build Status pmml4s-spark Scala version support PMML4S-Spark is a PMML (Predictive Model Markup Language) scoring library for Spark as SparkML Transformer.

Features

PMML4S-Spark is the Spark wrapper of PMML4S, you can see PMML4S for details.

Prerequisites

  • Spark >= 2.0.0

Installation

SBT users
libraryDependencies += "org.pmml4s" %%  "pmml4s-spark" % pmml4sSparkVersion
Maven users
<dependency>
  <groupId>org.pmml4s</groupId>
  <artifactId>pmml4s-spark_${scala.version}</artifactId>
  <version>${pmml4s-spark.version}</version>
</dependency>

Use PMML for Spark in Scala

  1. Load model.

    import scala.io.Source
import org.pmml4s.model.Model
import org.pmml4s.spark.ScoreModel

// The main constructor accepts an object of org.pmml4s.model.Model
val model = ScoreModel(Model(Source.fromURL(new java.net.URL("http://dmg.org/pmml/pmml_examples/KNIME_PMML_4.1_Examples/single_iris_dectree.xml"))))

    or

    import org.pmml4s.spark.ScoreModel

// load model from those help methods, e.g. pathname, file object, a string, an array of bytes, or an input stream.
val model = ScoreModel.fromFile("single_iris_dectree.xml")

  2. Call transform(dataset) to run a batch score against an input dataset.

    // The data is from http://dmg.org/pmml/pmml_examples/Iris.csv
val df = spark.read.
  format("csv").
  options(Map("header" -> "true", "inferSchema" -> "true")).
  load("Iris.csv")

val scoreDf = model.transform(df)
scala> scoreDf.show(5)
+------------+-----------+------------+-----------+-----------+---------------+-----------+-----------------------+---------------------------+--------------------------+-------+
|sepal_length|sepal_width|petal_length|petal_width|      class|predicted_class|probability|probability_Iris-setosa|probability_Iris-versicolor|probability_Iris-virginica|node_id|
+------------+-----------+------------+-----------+-----------+---------------+-----------+-----------------------+---------------------------+--------------------------+-------+
|         5.1|        3.5|         1.4|        0.2|Iris-setosa|    Iris-setosa|        1.0|                    1.0|                        0.0|                       0.0|      1|
|         4.9|        3.0|         1.4|        0.2|Iris-setosa|    Iris-setosa|        1.0|                    1.0|                        0.0|                       0.0|      1|
|         4.7|        3.2|         1.3|        0.2|Iris-setosa|    Iris-setosa|        1.0|                    1.0|                        0.0|                       0.0|      1|
|         4.6|        3.1|         1.5|        0.2|Iris-setosa|    Iris-setosa|        1.0|                    1.0|                        0.0|                       0.0|      1|
|         5.0|        3.6|         1.4|        0.2|Iris-setosa|    Iris-setosa|        1.0|                    1.0|                        0.0|                       0.0|      1|
+------------+-----------+------------+-----------+-----------+---------------+-----------+-----------------------+---------------------------+--------------------------+-------+
only showing top 5 rows

Use PMML for Spark in Java

  1. Load model.

    import org.pmml4s.spark.ScoreModel;

// load model from those help methods, e.g. pathname, file object, a string, an array of bytes, or an input stream.
ScoreModel model = ScoreModel.fromFile("single_iris_dectree.xml");

  2. Call transform(dataset) to run a batch score against an input dataset.

    import org.apache.spark.sql.Dataset;

// The data is from http://dmg.org/pmml/pmml_examples/Iris.csv
Dataset<?> df = spark.read().
   format("csv").
   option("header", "true").
   option("inferSchema", "true").
   load("Iris.csv"); 

Dataset<?> scoreDf = model.transform(df);
scoreDf.show(5);

Use PMML in PySpark

See the PyPMML-Spark project. PyPMML-Spark is a Python PMML scoring library for PySpark as SparkML Transformer, it really is the Python API for PMML4s-Spark.

Use PMML in Scala or Java

See the PMML4S project. PMML4S is a PMML scoring library for Scala. It provides both Scala and Java Evaluator API for PMML.

Use PMML in Python

See the PyPMML project. PyPMML is a Python PMML scoring library, it really is the Python API for PMML4S.

Deploy PMML as REST API

See the AI-Serving project. AI-Serving is serving AI/ML models in the open standard formats PMML and ONNX with both HTTP (REST API) and gRPC endpoints.

Support

If you have any questions about the PMML4S-Spark library, please open issues on this repository.

Feedback and contributions to the project, no matter what kind, are always very welcome.

License

PMML4S-Spark is licensed under APL 2.0.