You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, the entry barrier to running a retrieval evaluation example is not so low.
One needs to understand the concepts and initialize objects such as:
dataloader
metrics
pipeline
evaluator
and then run evaluator.compute() and pass the other 3 objects.
The same functionality could be achieved with a simpler interface, e.g. by initializing only a single object (let's say: EvaluationPipeline), which would handle the data loading and metrics under the hood, in addition to what the current pipeline implementation does.
Furthemore, some reasonable default values can be either set (without having to pass them via a hierarchy of YAML files) or automatically inferred (e.g. based on the existence of OPENAI_API_KEY or other environment variables).
Motivation
The entry barrier should be lower for new users only willing to run example evaluation scripts.
Feature description
Currently, the entry barrier to running a retrieval evaluation example is not so low.
One needs to understand the concepts and initialize objects such as:
and then run evaluator.compute() and pass the other 3 objects.
The same functionality could be achieved with a simpler interface, e.g. by initializing only a single object (let's say: EvaluationPipeline), which would handle the data loading and metrics under the hood, in addition to what the current pipeline implementation does.
Furthemore, some reasonable default values can be either set (without having to pass them via a hierarchy of YAML files) or automatically inferred (e.g. based on the existence of OPENAI_API_KEY or other environment variables).
Motivation
The entry barrier should be lower for new users only willing to run example evaluation scripts.
Additional context
initial idea from @mhordynski :
It should wrap lower-level concepts like
Evaluator
orOptimizer
in a higher-level API for simplicity.The text was updated successfully, but these errors were encountered: