softplus activation

[edgarriba]

@vdumoulin I'm a bit confused about why are you using softplus activation** for the discriminator since in the paper formulas there no mention or at least I cannot recognize it.

**https://github.com/IshmaelBelghazi/ALI/blob/master/ali/bricks.py#L69-L72

BTW, @vdumoulin we met at your talk in the CVC during the break time :D

[vdumoulin]

Nice to talk to you again! :) This is a numerically stable version of -log(sigmoid(x)):

-log(sigmoid(x)) = -log(1 / (1 + exp(-x))) = log(1 + exp(-x)) = softplus(-x)

Normally the substitution would happen automatically at the optimization stage of compiling the Theano function, but I encountered situations in the past where this was not triggered, so I use the numerically stable version directly.