How to add new nodes / neurons dynamically in tensorflow

Question

If I want to add new nodes to on of my tensorflow layers on the fly, how can I do that?

For example if I want to change the amount of hidden nodes from 10 to 11 after the model has been training for a while. Also, assume I know what value I want the weights coming in and out of this node/neuron to be.

I can create a whole new graph, but is there a different/better way?

Answer 1

Instead of creating a whole new graph you might be better off creating a graph which has initially more neurons than you need and mask it off by multiplying by a non-trainable variable which has ones and zeros. You can then change the value of this mask variable to allow effectively new neurons to act for the first time.