Understanding the parameters of a simple neural network in Pytorch

Question

I have defined a simple NN as follows in Pytorch:

my_model = nn.Sequential(
            nn.Linear(1, 5),
            nn.Tanh(),
            nn.Linear(5, 1)) 

I then iterate through the parameters and check their sizes:

[parameter.shape for parameter in my_model.parameters()]

I get:

[torch.Size([5, 1]), torch.Size([5]), torch.Size([1, 5]), torch.Size([1])]

I'm confused as to why the last Size is 1. Shouldn't there be 5 bias values as well, going out from the hidden layer into the output layer?

Answer 1

Um, so it looks like you are defining a two layers network.

The input size is [1], and the hidden layer size is [5], and next, the hidden layer is connected to the output layer, the output layer size is [1].

It should look like:

       *   <- input
   / / | \ \
   * * * * *  <- hidden layer
   \ \ | / /
       *   <- output

So for the hidden layer, there are 5 values as bias, for the output layer, there is only one value as bias. Sounds reasonable?