Embedding Layer

[OneAdder]

Input Embeddings Lookup Table (Trainable

Core

In Natural Language Processing input data is often encoded as indices of tokens in a vocabulary. Those indices are converted into vectors using weights which are trainable.
In theory, similar behaviour can be achieved by spreading input data by the desired size of vectors and then put through input2d and linear2d layers. However, it is very inefficient as we'll have to do matmul each time instead of simply getting an element by indices.
So, I created the layer that does it efficiently. It does not have a gradient as it is intended as an input layer. But it is trainable (get_params, get_gradients and set_params).

Positional

Apart from this core functionality, I also added positional encoding (output vectors + positions assigned to sin/cos waves). This will be needed for transformers.

Python Reference

Here: torch.nn.Embedding and a custom function for positional encoding.

[OneAdder]

@jvdp1 @milancurcic Ready for review

[jvdp1]

LGTM. Below are some minor suggestions.

[OneAdder]

@milancurcic Your opinion?

[milancurcic]

Thanks for the nudge. Will try to finish review and merge either today or Monday. Thank you for all the hard work!

[milancurcic]

@OneAdder Can you add the Embedding layer entry to the table of layers in the README? I assume that based on it being provided in nf that it's meant to be consumed by users directly, rather than only internally by other layers.

[OneAdder]

@milancurcic Readme updated. Thank you for resolving conflicts!

[milancurcic]

Thank you!