If the weight is positive that feature helps the neuron fire and vice versa.
If the neural network is processing images, the weights form and image too. You can tell what a lot of the units are "looking" for by plotting the weights as an image.
You can clearly see various digit deformations in the weights. (white is positive weight, black is negative weight typically)
Lots of papers do this, this one was happened to be the first one I managed to google.
So the first layer is normally readable because weights are in the same space as the input feature space. The second layer is normally a jumble as its randomly initialised before convergence. But in the paper's example you can imagine the 2nd layer output unit representing a final classification of a 4, is probably summing up all the 4 deformation detectors in layer 1 and negatively summing up everything else.
Actually I just read that paper properly and I see they did visualise the higher level layer weights too:
"For visualizing what units do on the 2nd and 3rd layer, we used the activation maxi-
mization technique described by Erhan et al. (2009)"[1]
So there you have it! You can visualise all of a neural network! But you have to implement some specific numerics to get the data out. Reading the first layer is trivial, and subsequent layers need some analysis.
[1] Erhan 2009 Visualizing higher layer features of a deep network
(good find, thanks for asking the right questions!)
Just look at the one figure in the paper and you will see.
Because your final layer is positively encoded, that polarity trickles down through back propogation. I also think the positive weights thing is to do with the way features normally work. It makes sense to look for corners, not anti corners for object detection.
If the neural network is processing images, the weights form and image too. You can tell what a lot of the units are "looking" for by plotting the weights as an image.
page 14, Figure 3: jmlr.org/papers/volume11/erhan10a/erhan10a.pdf
You can clearly see various digit deformations in the weights. (white is positive weight, black is negative weight typically)
Lots of papers do this, this one was happened to be the first one I managed to google.
So the first layer is normally readable because weights are in the same space as the input feature space. The second layer is normally a jumble as its randomly initialised before convergence. But in the paper's example you can imagine the 2nd layer output unit representing a final classification of a 4, is probably summing up all the 4 deformation detectors in layer 1 and negatively summing up everything else.