-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
undo forward-hook side-effect #19
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I proposed an alternate solution. However, feel free to merge this one if speed is of the essence.
onmt/trainer.py
Outdated
@@ -447,6 +447,11 @@ def validate(self, valid_iter, moving_average=None, task=None): | |||
# Set model back to training mode. | |||
valid_model.train() | |||
|
|||
# the forward hook `has_grad` was triggered, so we manually unset the flags to not fool the optim |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should work, as long as validation is never performed in the middle of an accumulation (a very reasonable assumption).
However, I think a better solution would be to modify the hook: have it check that module.training
is set. If unset, then return without doing anything.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the efficient thing would be to set param.has_grad = self.training
, but that's a case of clarity vs. efficiency.
Maybe a big fat explanatory comment would suffice
This reverts commit 5763591.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
closes #18