Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/add support for chat and instruct model types #1131

Conversation

chakravarthik27
Copy link
Collaborator

This pull request brings several enhancements and bug fixes to the langtest module, focusing on model configuration and handling. The key updates include:

Model Configuration and Handling Improvements:

  • Configuration Types Added:

    • Introduced DatasetConfig and ModelConfig types to define configurations for datasets and models in langtest/types.py.
  • Class Initializer Changes:

    • Updated the Harness class initializer to accept ModelConfig and DatasetConfig types for the model and data parameters.
    • Modified the __init__ method in Harness to handle the model_type attribute.
  • Model Loading Enhancements:

    • Enhanced the load_model method in llm_modelhandler.py to support the model_type parameter, including chat models.
    • Updated transformers_modelhandler.py to manage model_type in the load_model and predict methods, with added support for chat models.

Utility and Miscellaneous Updates:

  • Utility Improvements:

    • Added a CHAT_MODEL_CLASSES mapping in utils.py for associating hub names with chat model classes.
    • Applied type annotations in config_utils.py for DEFAULTS_CONFIG.
  • Prompt Handling Updates:

    • Modified prompts.py to support prompt retrieval for the "transformers" hub.

These changes enhance the flexibility and robustness of the langtest module, making it easier to work with various model configurations and types.

@chakravarthik27 chakravarthik27 self-assigned this Oct 16, 2024
@chakravarthik27 chakravarthik27 linked an issue Oct 16, 2024 that may be closed by this pull request
@chakravarthik27 chakravarthik27 changed the base branch from main to release/2.5.0 October 17, 2024 06:05
@chakravarthik27 chakravarthik27 merged commit 0d7589d into release/2.5.0 Oct 21, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add Support for Chat and Completion Model Types
2 participants