Introduce lineapy.get_function()

[mingjerli]

We are pleased to introduce lineapy.get_function(), a new LineaPy API for parameter refactoring. It extracts the code to create the list of targeted artifacts and packages the code as a python function. It also allows us to parameterize any literal assigned variables within the code. Furthermore, for any artifact calculation(not just targeted artifacts) in the code, we can swap the calculation part by loading the artifact value(any version) directly from the artifact store.

def lineapy.get_function(
    artifacts = list_of_lineapy_artifacts,
    input_parameters = list_of_variable_names,
    reuse_pre_computed_artifacts = list_of_lineapy_artifacts
) -> Callable

Example Usage

Let's say we have a Jupyter notebook that trains a machine learning model from training data and makes a prediction on inference data like following

training_data_path = '/data/train.csv'
training_data = load_data(training_data_path )
model = train_model(training_data)
lineapy.save(model, 'model')  # version 1

inference_data_path = '/data/inference.csv'
inference_data = load_data(inference_data_path )
prediction = run_prediction(model, inference_data )
lineapy.save(prediction, 'prediction')  # version 1

Now, if we want to retrain the model because we receive new training data. Instead of copy-paste the code to another notebook and manually changing the training_data_path variable, we can use

model_training_function = lineapy.get_function(
    [('model', 1)], 
    input_parameters=['training_data_path']
)
model = model_training_function(training_data_path='/data/new_train.csv')
lineapy.save(new_model, 'model') # version 2

After we train the model, now we want to make predictions on new data,

make_prediction = lineapy.get_function(
    [('prediction', 1)], 
    input_parameters=['inference_data_path'], 
    reuse_pre_computed_artifacts=[('model', 2)]
) # use latest version if not specified
prediction = make_prediction('inference_data_path'='/data/new_inference.csv')

If for any reason, we believe the old model performs better and we want to use the old model to make a prediction, we can achieve it with following

make_prediction = lineapy.get_function(
    [('prediction', 1)], 
    input_parameters=['inference_data_path'], 
    reuse_pre_computed_artifacts=[('model', 1)]
)
prediction = make_prediction('inference_data_path'='/data/new_inference.csv')