Anyang Kayo Amorphous Technology Co.,Ltd. , https://www.kayoamotech.com
CNN model ideas, accelerated algorithm design and experimental examples
Since AlexNet won the ILSVRC 2012 ImageNet Image Classification Competition, the Convolutional Neural Network (CNN) has sparked a revolution in the field of computer vision. By replacing traditional hand-crafted features and classifiers, CNNs introduced an end-to-end processing approach that dramatically improved accuracy across image recognition tasks, even surpassing human performance on benchmarks like LFW. As CNN models have continued to evolve, they have become increasingly deep and large in size, pushing the limits of computational power and model efficiency.
This rapid growth brought about a new challenge: the models became too large to run efficiently on mobile or embedded devices. Their high bandwidth requirements also made them impractical for real-time applications over networks. Additionally, the energy consumption and processing speed of these massive models posed significant obstacles to practical deployment. Therefore, the need for smaller, faster, and more efficient models became urgent.
To address this, researchers began exploring various methods for model compression. Early techniques such as weight pruning and matrix singular value decomposition (SVD) were proposed, but they often failed to achieve satisfactory compression ratios or maintain high performance. In recent years, model miniaturization has been categorized into two main approaches: compression based on model weights and compression based on network architecture. Furthermore, from the perspective of computation speed, these methods can be further divided into those that only reduce model size and those that aim to reduce size while improving speed.
This paper discusses several representative works, including SqueezeNet, Deep Compression, XNOR-Net, Distillation, MobileNet, and ShuffleNet. These methods are classified according to their compression strategies and whether they consider speed improvements.
Table 2: Several classic compression methods and comparison
| Method | Compression Approach | Speed Consideration |
|---------------|----------------------|---------------------|
| SqueezeNet | Architecture | No |
| Deep Compression | Weights | No |
| XNOR-Net | Weights | Yes |
| Distillation | Architecture | No |
| MobileNet | Architecture | Yes |
| ShuffleNet | Architecture | Yes |
First, let's explore SqueezeNet.
1.1 Design Ideas
SqueezeNet, introduced by F. N. Iandola et al. in a 2016 paper titled *"SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and < 0.5MB model size,"* is a compact neural network designed to achieve high accuracy with minimal parameters. It compresses the original AlexNet model by a factor of around 510, reducing its size to less than 0.5MB while maintaining comparable performance.
The core principle behind SqueezeNet is to minimize the number of parameters without sacrificing accuracy. This is the ultimate goal of all model compression techniques. Based on this idea, SqueezeNet proposes three key design strategies:
Strategy 1: Replace 3×3 convolution kernels with 1×1 convolution kernels.
This strategy reduces the number of parameters significantly, as a 1×1 kernel uses only 1/9th of the parameters of a 3×3 kernel, theoretically reducing the model size by a factor of 9.
Strategy 2: Reduce the number of input channels fed into 3×3 convolutions.
For a convolution layer with a 3×3 kernel, the total number of parameters is given by N × C, where N is the number of output channels and C is the number of input channels. To reduce the overall parameter count, both N and C must be minimized.
Strategy 3: Delay downsampling as much as possible in the network.
Downsampling is typically controlled by the stride of convolution layers or pooling operations. Delayed downsampling allows higher-resolution feature maps to be preserved for longer, which can improve classification accuracy by retaining more spatial information.
Among these strategies, the first two focus on reducing the number of parameters, while the third aims to maximize model accuracy.