Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

allow optionally fill attention mask in forward #457

Merged
merged 1 commit into from
Oct 20, 2023

Conversation

eaidova
Copy link
Collaborator

@eaidova eaidova commented Oct 19, 2023

What does this PR do?

allow passing inputs without attention mask for OVModelForCausalLM

For inference openvino expect that all inputs that exists in model filled before starting inference.
Input filling for text generation controlled when you run generate function and guarantee that all inputs provided.
But when user trying to use forward directly (building own custom generation pipeline for example), there is no knowledge which inputs available in model.
PyTorch models allow run model without passing some optional inputs, like attention_mask (filling it inside with default values). In our case, if user do not provide this input, it leads to error that do not have any context what went wrong.

These changes adding attention_mask filling inside forward for improving user experience

Before submitting

  • Did you write any new necessary tests?

@eaidova
Copy link
Collaborator Author

eaidova commented Oct 19, 2023

@echarlaix @AlexKoff88 @helena-intel could you please take a look?

@eaidova eaidova force-pushed the ea/default_attention_mask branch from 3a32269 to dbbe0f3 Compare October 19, 2023 07:25
@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Oct 19, 2023

The documentation is not available anymore as the PR was closed or merged.

@AlexKoff88 AlexKoff88 requested a review from echarlaix October 19, 2023 10:18
Copy link
Collaborator

@echarlaix echarlaix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the addition !

@echarlaix echarlaix merged commit b7703dc into huggingface:main Oct 20, 2023
6 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants