Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SGD optimizer stub #139

Merged
merged 8 commits into from
Jun 22, 2023
Merged

Conversation

milancurcic
Copy link
Member

First attempt at defining the concrete optimizer procedure as a method of SGD optimizer type.

Currently defining the minimize subroutine as elemental to allow a scalar/array/rank-agnostic interface. It's possible that this won't work for all cases if we discover new requirements but let's try this for the time being.

@milancurcic milancurcic marked this pull request as ready for review June 21, 2023 20:55
@milancurcic milancurcic added the enhancement New feature or request label Jun 21, 2023
@milancurcic
Copy link
Member Author

This now works. There's an API change to the network % train() and network % update() methods which now require an argument of class(optimizer_base_type). (I wonder if it's possible to make this optional so we can default to sgd).

Once an optimizer is passed to network % update, it's passed to layer % update() for all layers. In layer % update, the weights and biases are accessed from the internal layer representation and passed to optimizer % minimize(). I borrowed the name minimize from Keras. optimizer % optimize() would be appropriate but sounds weird due to repetition. How about optimizer % update()?

The optimizer step for conv2d is currently not implemented but it may be easy to do so even in this PR (but convolutional training is broken anyway as explained in #142).

@Spnetic-5
Copy link
Collaborator

Spnetic-5 commented Jun 22, 2023

Thanks for bringing up the API change regarding the network methods. Making the optimizer argument optional and setting up SGD as default sounds like a good idea.

Regarding the naming, I think optimizer % minimize() is good, as it captures the essence of the operation. Furthermore, I'll study all the updations in the code.

@milancurcic milancurcic merged commit 31fc061 into modern-fortran:main Jun 22, 2023
@milancurcic milancurcic deleted the sgd-optimizer-stub branch June 22, 2023 15:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants