Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resuming from a previous run #18

Closed
tuero opened this issue Dec 21, 2021 · 2 comments
Closed

Resuming from a previous run #18

tuero opened this issue Dec 21, 2021 · 2 comments

Comments

@tuero
Copy link
Contributor

tuero commented Dec 21, 2021

I'm currently using this project as a way to track metrics during ML training. Since training can take long, I have a way which can save/store models/data as a checkpoint during an exit call, and will reload that to resume training. It would also be nice if metrics can continue to track rather than starting from scratch.

Would this be a useful feature to add? Currently, I've changed the constructor as the following (for my personal use)

// Header of TensorBoardLogger
// ofs_ = new std::ofstream(log_file, std::ios::out | std::ios::trunc | std::ios::binary);
ofs_ = new std::ofstream(log_file, std::ios::out | std::ios::app | std::ios::binary);

Adding this feature would be a quick change. Maybe have a resume flag as a default argument set false in the header for TensorBoardLogger? If this something that would be welcomed, I could make a PR.

@RustingSword
Copy link
Owner

Thanks and this is surely welcomed! I think it's a nice feature, and a resume flag is appropriate.

I have a small concern that, as discussed in #13, in the future the event file name may be auto generated. However when resuming from a previous file, the file name should be given by the user. How to make these two features compatible can be left to future.

@tuero
Copy link
Contributor Author

tuero commented Dec 22, 2021

Closing as #19 implements this.

@tuero tuero closed this as completed Dec 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants