README.md

LLM-Blender API Reference

Table of Contents

• LLM-Blender API Reference
  • Table of Contents
  • llm_blender.llm_blender_ranker
    • LLMBlenderRanker
      • __init__
      • warm_up
      • run
  • llm_blender.llm_blender_evaluator
    • LLMBlenderEvaluator Objects
      • __init__
      • prepare_inputs
      • compute_mean_scores
      • compute_bleurt
      • compute_bartscore
      • compute_bertscore
      • compute_metrics

llm_blender.llm_blender_ranker

LLMBlenderRanker

@component
class LLMBlenderRanker()

Implements a LLM output ranking method with a pairwise reward model using the LLM Blender framework.

Usage Example:

llm_ranker = LLMBlenderRanker(model="llm-blender/PairRM")
answers = [
    GeneratedAnswer(data="Paris is the capital of France.", query="What makes Paris unique?", documents=[]),
    GeneratedAnswer(
        data="The Eiffel Tower is an iconic landmark in Paris.", query="What makes Paris unique?", documents=[]
    ),
    GeneratedAnswer(data="Berlin is a beautiful city.", query="What makes Paris unique?", documents=[]),
]
output = llm_ranker.run(answers=answers)
ranked_answers = output["answers"]
print(ranked_answers)

# [
#     GeneratedAnswer(
#         data="The Eiffel Tower is an iconic landmark in Paris.",
#         query="What makes Paris unique?",
#         documents=[],
#         meta={},
#     ),
#     GeneratedAnswer(
#         data="Paris is the capital of France.", query="What makes Paris unique?", documents=[], meta={}
#     ),
#     GeneratedAnswer(data="Berlin is a beautiful city.", query="What makes Paris unique?", documents=[], meta={}),
# ]

__init__

def __init__(model: str = "llm-blender/PairRM",
             device: str = "cpu",
             model_kwargs: Optional[Dict[str, Any]] = None)

Initialize a LLMBlenderRanker.

Arguments:

• model: Local path or name of the model in Hugging Face's model hub, such as 'llm-blender/PairRM'.
• device: The device on which the model is loaded. If None, the default device is automatically selected.
• model_kwargs: Keyword arguments to be passed to the LLM Blender model.

warm_up

def warm_up()

Warm up the pair ranking model used for scoring the answers.

run

@component.output_types(documents=List[GeneratedAnswer])
def run(answers: Variadic[List[GeneratedAnswer]])

Rank the output answers using the LLM Blender model.

Arguments:

• answers: A list of answers to be ranked.

Returns:

A list of ranked answers.

llm_blender.llm_blender_evaluator

LLMBlenderEvaluator Objects

class LLMBlenderEvaluator()

Implements an evaluator for assessing the performance of predictions against labels using BLEURT, BARTScore, and BERTScore.

__init__

def __init__(preds, labels)

Evaluates the performance of predictions against labels using BLEURT, BARTScore, and BERTScore.

Arguments:

• preds: A list of predicted outputs.
• labels: A list of reference or target outputs.

prepare_inputs

def prepare_inputs()

Ensures that predictions and labels are formatted correctly before computing scores.

compute_mean_scores

def compute_mean_scores(scores) -> float

Computes the mean of a list of scores.

Arguments:

• scores: A list of scores.

Returns:

The mean score.

compute_bleurt

def compute_bleurt() -> float

Computes the BLEURT score for the provided predictions and labels.

Returns:

The BLEURT score.

compute_bartscore

def compute_bartscore() -> float

Computes the BARTScore for the provided predictions and labels.

Returns:

The BARTScore.

compute_bertscore

def compute_bertscore() -> float

Computes the BERTScore for the provided predictions and labels.

Returns:

The BERTScore.

compute_metrics

def compute_metrics() -> Dict[str, float]

Computes BLEURT, BARTScore, and BERTScore for the provided predictions and labels.

Returns:

A dictionary containing the computed metrics.