Question about 'supervised_mnist' spike trains

[rafaelblevin821]

Hi, I have a quick question about the spike trains in supervised_mnist and eth_mnist using diehlandcook2015.

Suppose we have 28*28 images encoded with Poisson (time = 250, and dt = 1), so the input is 250, 1, 784...

Do each of these 'full' spike trains go to every neuron in the processing layer, and then one neuron is chosen as a winner while the rest are inhibited?

If so, would this not be problematic for larger images, time durations, and, therefore, larger spike trains?

EDIT: For reference, I am using images resized to 200*200, time=500, dt=1. Therefore, the input is 500, 1, 40,000.

[Hananel-Hazan]

Since the network has an inhibitory layer, the first neuron that spikes inhibits any other neuron in the layer and prevents the other neurons from spiking. There are many ways this can go wrong essentially when using many neurons. For example two or more neurons spike at the same time and send an inhibition signal to all neurons, that can cause massive inhibition. But roughly the architecture holds for scaling up.
Suggestion, if you are going to increase the input layer, you can also add some tweaks to the excitation/inhibition layer in such a way that you dont need to increase it with the same proportion to the input layer.

[rafaelblevin821]

Since the network has an inhibitory layer, the first neuron that spikes inhibits any other neuron in the layer and prevents the other neurons from spiking. There are many ways this can go wrong essentially when using many neurons. For example two or more neurons spike at the same time and send an inhibition signal to all neurons, that can cause massive inhibition. But roughly the architecture holds for scaling up. Suggestion, if you are going to increase the input layer, you can also add some tweaks to the excitation/inhibition layer in such a way that you dont need to increase it with the same proportion to the input layer.

Hi @Hananel-Hazan, I was wondering what would be the best way to reduce the size of the spike trains but still retain important information for training?

The average spike frequency for a 200*200 image (time/dt = 500/1) produces around 1,200,000 Poisson-encoded spikes. It is very hard for the network to learn anything from this amount of information. I have tried normalizing the image pixel values, reducing them from 0-255 to 0-63, which reduces the spike frequency but still produces around 45,000 spikes per spike train. I could lower even further but lose a lot of valuable information for training and testing.

This is why I was asking about splitting the spike trains into bins/batches (but this leads to a complete rewrite of the training and testing logic).

Would you have any suggestions on reducing spike trains but still retaining important information?

Or perhaps you know of any papers that explain how to do this?

[Hananel-Hazan]

Normalizing the image is a good step, what about lowering the intensity of the Poisson encoding (assuming you are using it, if not the encoding that you use to convert between integers to a spike train)?
Usually the problem is the opposite, not enough spikes to work with, aka vanishing spikes. If the above suggestion fails, I would take advantage of this vanishing spikes feature and add another layer that takes the input and converts it to yet another set of inputs, but with much more reduced spike activity. Another option is to add inhibitory neurons that will reduce the overall activity. The trick is how many inhibitory neurons to use, how many connections each inhibitory neuron has and what topology to use to connect between inhibitory and excitatory neurons.

[rafaelblevin821]

Another option is to add inhibitory neurons that will reduce the overall activity. The trick is how many inhibitory neurons to use, how many connections each inhibitory neuron has and what topology to use to connect between inhibitory and excitatory neurons.

I'm not sure I understand this... to my understanding, in the diehlandcook2015 model, all the excitatory neurons are connected one-to-one with all of the excitatory neurons, meaning that when the winning neuron fires, all other excitatory neurons are inhibited. How would adding more inhibitory neurons help? Would it not be easier to increase the strength of the inhibitory layer?

Also, I have an unrelated question... Consider that the neuron connection with the highest weight "wins", what is the mechanism that prevents it from continuing to win? Logically, the strongest connection would continually increase and continue to win after each iteration. I know that this is not the case, is there any way you could clarify this?

Again, I really appreciate your support @Hananel-Hazan.

[Hananel-Hazan]

I'm not sure I understand this... to my understanding, in the diehlandcook2015 model, all the excitatory neurons are connected one-to-one with all of the excitatory neurons, meaning that when the winning neuron fires, all other excitatory neurons are inhibited. How would adding more inhibitory neurons help? Would it not be easier to increase the strength of the inhibitory layer?

Yes, you are right, in the excitatory/inhibition architecture of diehlandcook2015, you only need to increase the inhibition weight to negate the excess activity.

Also, I have an unrelated question... Consider that the neuron connection with the highest weight "wins", what is the mechanism that prevents it from continuing to win? Logically, the strongest connection would continually increase and continue to win after each iteration. I know that this is not the case, is there any way you could clarify this?

Right, the winner neuron will continue to win and that will strength its connections. But there are two additional factor that prevent it to continue to win. 1. The winner neuron get specialized on one set of patter because of the STDP rule, other pattern have less chance to cause the neuron fire. 2. The sliding thresholds in the winner neuron that will increase on each spike thus make the neuron less sensitive to the same input thus give other neurons a chance.

[rafaelblevin821]

Thanks @Hananel-Hazan

2. The sliding thresholds in the winner neuron that will increase on each spike thus make the neuron less sensitive to the same input thus give other neurons a chance.

What is the mechanism or variable that increases the neuron threshold after spiking? Is it theta_plus or is there another mechanism?

[Hananel-Hazan]

Yes, theta_plus will increase the threshold after spiking and tc_theta_decay will do the opposite with lack of spiking

[rafaelblevin821]

@Hananel-Hazan, thanks for the clarification. Much appreciated.