Understanding a residual block is quite easy. In traditional neural networks, each layer feeds into the next layer. In a network with residual blocks, each layer feeds into the next layer and directly into the layers about 2–3 hops away. That’s it. But understanding the intuition behind why it was required in the first place, why it is so important, and how similar it looks to some other state-of-the-art architectures is where we are going to focus on. There is more than one interpretation of why residual blocks are awesome and how & why they are one of the key ideas that can make a neural network show state-of-the-art performances on a wide range of tasks. Before diving into the details, here is a picture of how a residual block actually looks like.

We know neural networks are universal function approximators and that the accuracy increases with increasing the number of layers. But there is a limit to the number of layers added that results in an accuracy improvement. So, if neural networks were universal function approximators, then they should have been able to learn any simplex or complex function. But it turns out that, thanks to some problems like vanishing gradients and the curse of dimensionality, if we have sufficiently deep networks, it may not be able to learn simple functions like an identity function. Now, this is clearly undesirable.

Also, if we keep increasing the number of layers, we will see that the accuracy will saturate at one point and eventually degrade. And, this is usually not caused due to overfitting. So, it might seem that the shallower networks are learning better than their deeper counterparts, and this is quite counter-intuitive. But this is what is seen in practice and is popularly known as the degradation problem.

Without root causing into the degradation problem and the inability of a deep neural network to learn identity functions, let us start thinking about some possible solutions. In the degradation problem, we know that shallower networks perform better than the deeper counterparts with few more layers added to them. So, why not skip these extra layers and at least match the accuracy of the shallow sub-networks. But, how do you skip layers?

You can skip the training of few layers using skip connections or residual connections. This is what we see in the image above. In fact, if you look closely, we can directly learn an identity function by relying on skip connections only. This is the exact reason why skip connections are also called identity shortcut connections too. One solution for all the problems!

But why call it residual? Where is the residue? It’s time we let the mathematicians within us come to the surface. Let us consider a neural network block whose input is x, and we would like to learn the true distribution H(x). Let us denote the difference (or the residual) between this as

R(x) = Output — Input = H(x) — x

Rearranging it, we get,

H(x) = R(x) + x

Our residual block is overall trying to learn the true output, H(x). If you look closely at the image above, you will realize that since we have an identity connection coming from x, the layers are actually trying to learn the residual, R(x). So to summarize, the layers in a traditional network are learning the true output (H(x)), whereas the layers in a residual network are learning the residual (R(x)). Hence, the name: Residual Block.

It has also been observed that it is easier to learn residual of output and input, rather than only the input. As an added advantage, our network can now learn identity function by simply setting residual as zero. And if you truly understand backpropagation and how severe the problem of vanishing gradient becomes with increasing the number of layers, then you can clearly see that because of these skip connections, we can propagate larger gradients to initial layers, and these layers also could learn as fast as the final layers, giving us the ability to train deeper networks. The image below shows how to arrange the residual block and identity connections for the optimal gradient flow. It has been observed that pre-activations with batch normalizations generally give the best results (i.e., the right-most residual block in the image below gives the most promising results).

There are even more interpretations of residual blocks and ResNets, other than the ones already discussed above. While training ResNets, we either train the layers in residual blocks or skip the training for those layers using skip connections. So, different parts of networks will be trained at different rates for different training data points based on how the error flows backward in the network. This can be thought of as training an ensemble of different models on the dataset and getting the best possible accuracy.

Skipping training in some residual block layers can be looked at from an optimistic point of view too. In general, we do not know the optimal number of layers (or residual blocks) required for a neural network which might depend on the complexity of the dataset. Instead of treating the number of layers as an important hyperparameter to tune, by adding skip connections to our network, we are allowing the network to skip training for the layers that are not useful and do not add value in overall accuracy. In a way, skip connections make our neural networks dynamic to tune the number of layers during training optimally.

The image below shows multiple interpretations of a residual block.

Let us go a little into the history of skip connections. The idea of skipping connections between the layers was first introduced in Highway Networks. Highway networks had skip connections with gates that controlled how much information is passed through them, and these gates can be trained to open selectively. This idea is also seen in the LSTM networks that control how much information flows from the past data points seen by the network. These gates work similarly to control of memory flow from the previously seen data points. The same idea is shown in the image below.

Residual blocks are basically a special case of highway networks without any gates in their skip connections. Essentially, residual blocks allow memory (or information) to flow from initial to last layers. Despite the absence of gates in their skip connections, residual networks perform as well as any other highway network in practice. And before ending this article, below is an image of how the collection of all residual blocks completes into a ResNet.

If you are interested in knowing more about ResNet overall and its different variants, check out this article.

Residual blocks — Building blocks of ResNet was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Typically in convolutions, we use a 2D or a 3D kernel filter where we hope that each filter extracts some kind of a feature by convoluting in all the 2 or 3 dimensions, respectively. Specifically in 2D case, we try to extract simple features in initial layer and more complex features in the later layers. However if we want, we can factorize a 2D kernel into two 1D kernels as shown below.

Now, we can take these two 1D kernels and apply them one by one (in subsequent layers) on an image instead of applying the original 2D kernel. By doing so, we have actually reduced the number of parameters that we use for convolution and now have lesser parameters to train on. Also, the order in which we use these separable kernel filters does not really matter in general. To put things into perspective, a 5x5 kernel filter has 25 parameters whereas two kernels, a 1x5 kernel and a 5x1 kernel has only 10 parameters.

Of course, the reduction in parameters means we might have to compromise on the complexity of features that we learn. But, if you look at the image below, you can see that two 1D kernels can easily learn simple features that one 2D kernel was trying to learn. And, if you try to visualize these two 1D kernels (as shown below), you can see that they should be able to nearly learn any near-complex feature with decent accuracy. In the image, we are visualizing the weights of the 1D kernels “horz[c]” and “vert[r]” in the left and the image in the right shows a 3d plot of final weights learned as a result of applying the two 1D kernels one after the other.

These separable convolutional layers usually can learn simple to near-complex features in the image very efficiently and effortlessly. Therefore, intuitively it makes sense to use these separable layers more in initial layers that try to capture simple features compared to the final layers that try to capture much more complex features.

We could also see this as a way of regularizing our network, wherein we are trying to retain only truly independent parameters of the network. This way we are making our model computationally efficient with strong set of features learned at every layer. Also, by trying to convolve and learn features in each dimension, the separable convolution tries to learn more abstract features in each dimension. In a way, it is more focused on finding good features in independent individual dimensions and then kind of club them together in the end to extract a complex feature with minimal number of parameters.

Depthwise convolutions are a special case of separable convolution. In depthwise convolution, we first spatially convolve in X and Y dimension using 2D filters (having the size of third dimension = 1) and then we convolve channel wise in Z dimension using a1x1 filter. The image below shows how depthwise convolution is used in Xception networks.

You can read more about separable convolutions in an article by Chi-Feng Wang titled “A Basic Introduction to Separable Convolutions”. If you want more of a formal proof on separable convolutions, you can also refer here for more of a mathematical proof for 2D separable convolutions, supported with examples.

Separable convolutions — trading little accuracy for huge computational gains was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Usually, convolution filters are applied on an image layer by layer to get the final output feature maps. We increase the no of kernels per layer to learn more no of intermediate features, therefore increasing the no of channels in the next layer. But to learn more no. of features, we could also create two or more models that train and backpropagate in parallel. This process of using different set of convolution filter groups on same image is called as grouped convolution. In simple words, create a deep network with some number of layers and then replicate it so that there are more than 1 pathways for convolutions on a single image. The first use of this idea of grouped convolutions can be seen in Alexnet as shown below.

Grouped convolutions were used in Alexnet so that a deep neural network can be trained on less powerful GPUs with smaller RAM available at that time. To understand the importance of grouped convolutions, let us imagine a world without them and the problems we might face:

• We build deep networks with multiple kernels per layer resulting in multiple channel outputs per layer. This would result in wider networks based on the hope that our network learns a varied set of low level and high level features.
• Each kernel filter convolves on all the feature maps obtained on previous layer, resulting in lots of convolutions, some of which may be redundant.
• Training deeper models will be a nightmare because fitting these models on a single GPU wouldn’t be easy. Even if it were possible, we might have to use smaller batch size which would make the overall convergence difficult.

We can start to see how important grouped convolutions are for training deep networks. The following image shows how grouped convolutions in ResNext architecture be understood :

By using grouped convolution, let us address the above mentioned problems one-by-one:

• With grouped convolutions, we can build networks as wide as we want. Take one modular block of filter group and replicate them.
• Now, each filter convolves only on some of the feature maps obtained from kernel filters in its filter group, we are drastically reducing the no of computations to get output feature maps. Of course, you can argue that we are replicating the filter group, resulting in more number of computations. But to keep things in perspective, if we were to take all kernel filters in grouped convolutions and not use the concept of grouped convolutions, the number of computations will grow exponentially.
• We can parallelize the training in two ways:
• — Data parallelism: where we split the dataset into chunks and then we train on each chunk. Intuitively, each chunk can be understood as a mini batch used in mini batch gradient descent. Smaller the chunks, more data parallelism we can squeeze out of it. However, smaller chunks also mean that we are doing more like stochastic than batch gradient descent for training. This would result in slower and sometimes poorer convergence.
• — Model parallelism: here we try to parallelize the model such that we can take in as much as data as possible. Grouped convolutions enable efficient model parallelism, so much so that Alexnet was trained on GPUs with only 3GB RAM.

Apart from solving the above problems, grouped convolutions seem to be learning better representations of the data. Essentially, the correlation between kernel filters of different filter groups is usually quite less, which implies that, each filter group is learning a unique representation of the data. This is something which might not be easily possible in traditional neural networks as the kernels will usually correlate with each other. Infact, the image above shows the effect of increasing the no of groups on overall accuracy of the model. And the best part is that, all the accuracy comes with reduced computational cost.

This article was inspired from another fantastic article by Yani Ioannou titled A Tutorial on Filter Groups (Grouped Convolution) which provides a more detailed analysis.

Grouped Convolutions — convolutions in parallel was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Convolutional Neural Networks (CNNs) are neural networks that automatically extract useful features (without manual hand-tuning) from data-points like images to solve some given task like image classification or object detection. And now that you understand their use on your datasets, you start wondering: Apart from tuning various hyper-parameters of your network, how do I know what is the right kernel size for the network? Let’s dig further!

Let us set some common ground rules to stay on the same platform throughout the discussion:

• We will be looking primarily at 2D convolutions on images. These concepts also apply for 1D and 3D convolutions, but may not correlate directly.
• A 2D convolution filter like 3x3 will always have a third dimension in size. The third dimension is equal to the number of channels of the input image. For example, we apply a 3x3x1 convolution filter on gray-scale images (that has 1 black and white channel) whereas, we apply a 3x3x3 convolution filter on a colored image (with 3 channels, red, blue and green).
• We are assuming zero padding for rest of the discussion.

In a convolution, a convolution filter slides over all the pixels of the image taking their dot product. We do this hoping that the linear combination of the pixels weighted by the convolutional filter extracts some kind of feature from the image. It is done keeping these things in mind:

• Most of the useful features in an image are usually local and it makes sense to take few local pixels at a time to apply convolutions.
• Most of these useful features may be found in more than one place in an image. So, it makes sense to slide a single kernel all over the image in the hope of extracting that feature in different parts of the image using the same kernel.
• Also, an added benefit of using a small kernel instead of a fully connected network is to benefit from weight sharing and reduction in computational costs. To briefly explain this point, since we use the same kernel for different set of pixels in an image, the same weights are shared across these pixel sets as we convolve on them. And as the number of weights are less than a fully connected layer, we have lesser weights to back-propagate on.

Now that we have convolution filter size as one of the hyper-parameters to choose from, a choice needs to be made between smaller or larger filter size. Let us quickly compare both to choose the optimal filter size:

Now that we have some idea about the extraction using different sizes we will follow this up with a convolution example for small (3x3) and large filter sizes (5x5):

Based on the comparison above, we can conclude that smaller kernel sizes are and should be a popular choice over larger sizes.

Also, you might notice a preference for odd number as kernel size over a 2x2 or 4x4 kernel size. The explanation is as follows:

For an odd-sized filter, all the previous layer pixels would be symmetrically around the output pixel. Without this symmetry, we will have to account for distortions across the layers which happens when using an even sized kernel. Therefore, even sized kernel filters are mostly skipped to promote implementation simplicity. If you think of convolution as an interpolation from the given pixels to a center pixel, we cannot interpolate to a center pixel using an even-sized filter.

Therefore, in general, we would like to use smaller odd-sized kernel filters. But, 1x1 is eliminated from the list candidate optimal filter sizes as the features extracted would be fine grained and local, with no information from the neighboring pixels. Also, it is not really doing any useful feature extraction!

Hence, 3x3 convolution filters work in general, and is often the popular choice!

Originally published at icecreamlabs.com on August 19, 2018.

Deciding optimal filter size for CNNs was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.