Wrong positional encoding

[varshiths]

position_enc = np.array([
      [pos / np.power(10000, 2.*i/num_units) for i in range(num_units)]
      for pos in range(T)])

In the paper, section 3.5, the embedding (before the application of sine or cosine) for even positions in [0..num_units] indexed by 2*i is the same as that for odd positions.
The correct code should be

position_enc = np.array([
      [pos / np.power(10000, (i-i%2)/num_units) for i in range(num_units)]
      for pos in range(T)])

[imhuay]

You are right!

[LauJames]

Great ! This issue makes me have a deeper understanding of position embedding.