Question on Hermite Polynomials Coefficients

[zhengyu-yang]

Thank you for this interesting paper. However, as I go through the paper and code, there is something that I am not very sure about. Based on my understanding, followings are the normalized Hermite polynomial from degree 0 to 3 (i.e. h_0 to h_3).

However, after reading your code, I am a little confused. In the following section of your code, the coefficient of the hermite polynomial doesn't match.

DeepHermites/Code/3-semisupervised_setting/aws_costestimates/epoch_measurements/mnist/4hermites_v2l_1epochs_w/lib/activations.py

Lines 48 to 54 in 03b2c8a

	def hermite(self, x, k, num_pol=5):
	return (
	(torch.full_like(x, k[0].item()) * torch.ones_like(x)) +
	(torch.full_like(x, k[1].item()) * x) +
	(torch.full_like(x, k[2].item()) * (x**2 - 1) / 1.4142135623730951)
	+ (torch.full_like(x, k[3].item()) *
	(x**3 - 3 * x) / 2.449489742783178))

[zhengyu-yang]

Also, when you use .item(), the gradient flow is cut off and the weights of hermite polynomials are no longer trainable.

[lokhande-vishnu]

Hi @elony314 , Thanks for your questions.

Coefficients of Hermite Polynomial Activations

The n^{th} coefficient of the hermite activation is computed by calculating the inner product between the ReLU function and the n^{th} normalized hermite polynomial. Our objective is to decompose ReLU function using Hermite Polynomial Basis. Please refer to the Section 4 of arXiv:1711.00501 for the definition of the inner product and a nice primer on Hermite functions. The foot-note on Page 5 of arXiv:1711.00501 provides the precise coefficient values for different 'n', which is what we use in the code.

.item(), the gradient flow is cut off

Thanks for pointing this out. In our model, we set the coefficients of hermite polynomials as trainable parameters .

The code that is linked in the question was not used for training but to compute the time and cost of running the script for a "single" epoch, hence, we did not require gradients to be updated.

I have added a new "activations.py" file that was used for training and renamed the existing file as "activations_timing.py" to resolve confusions. This new "activations.py" file was earlier made available in all the other directories such as 1-deep_autoencoder, 2-supervised_setting and 4-WhyHermitesProvideFasterConvergence.

https://github.com/lokhande-vishnu/DeepHermites/blob/master/Code/3-semisupervised_setting/aws_costestimates/epoch_measurements/mnist/4hermites_v2l_1epochs_w/lib/activations.py

[zhengyu-yang]

I am well aware of the weights (c_i) from arXiv:1711.00501, and I use the same value in my code. However, what I am referring to are the coefficients "inside" the hermite polynomials (h_i).

To be more specific, following is what I have derivated for normalized hermite polynomials from degree 0 to 3:

Following is what seems to be in your code:

[lokhande-vishnu]

We use probabilists' Hermite polynomials. You have computed physicists' Hermite polynomials.
Please see the section above "properties" in this link https://en.wikipedia.org/wiki/Hermite_polynomials

[zhengyu-yang]

Thanks a lot. I see. However, I think equation (1) in the paper is the physicists' Hermite polynomial, and I follow that one to derive the above equations.

[lokhande-vishnu]

Thanks for pointing this out. We will modify the equation into this in our next arxiv update.