forward pass projection

[Alexivia]

Hi,

Is there any way to define a layer that for forward-pass calculates its output by projecting the weight values into a different value range, but for the back-propagation uses the real weight values? i.e., as an example from the mnist.py file, if for the hidden layer 1 forward pass we could have hidden1 = tf.nn.relu(tf.matmul(images, projection_func(weights)) + biases), but for the gradient calculation during back-propagation the weights were updated without projection_func().

This would be useful to implement BNNs, to apply functions such as the ones mentioned by @Jony101K here.

Thanks.
Alexandre

[Alexivia]

@bhack, could you help me with this?

[gaohuazuo]

Maybe you can try https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/framework/function.py. It allows you to construct a new op with existing ones, and define its gradient.

from tensorflow.python.framework.function import Defun

def _projection_func_grad(op, grad):
    # compute the (fake) gradient, in this case just pass it through
    return grad

@Defun(tf.float32, python_grad_func=_projection_func_grad)
def projection_func(weights):
    # do something
    return weights

[aselle]

This question is better asked on StackOverflow since it is not a bug or feature request. There is also a larger community that reads questions there. Thanks!