Neural Architecture Search and the Automation of Network Design
How evolutionary strategies search for network architectures rather than hand designing them
Introduction
Designing a neural network architecture has traditionally been a task for human experts. The choices involve layer types, connection patterns, layer widths and depths, and many smaller decisions that together determine how well a model performs on a given task. In 2017, Zoph and Le demonstrated that this process could be automated. A controller network trained by reinforcement learning proposed candidate architectures, each was evaluated by training it on a benchmark dataset, and the resulting performance signal improved the controller’s subsequent proposals. The architectures it found matched or exceeded the performance of architectures designed by hand (Zoph and Le, 2017). That result is the starting point for the field now called Neural Architecture Search (NAS), which also encompasses gradient-based and evolutionary methods for automating architectural decisions. NAS is one component of the broader field of Automated Machine Learning, or AutoML, which covers the automated search for model architectures, hyperparameters, and training configurations.
The architectures found by these methods power some of the most widely deployed computer vision systems in production, with search procedures that reduce the cost of finding good architectures from months of GPU time to days. Sejnowski raises the question of whether gradient descent is the only viable way to search algorithm space, or whether faster alternatives exist (Sejnowski, 2018). NAS is the current answer to that question. This post reviews the topic by organizing its content around a set of core questions, tracing how automated architecture search moved from a computational expense measured in months to a practical tool whose results run in production on every modern smartphone.
What problem does hand-designed architecture actually solve?
Neural network architecture design is a search problem over a combinatorially large space. A practitioner choosing an architecture decides how many layers to include, what type each layer should be, how wide it is, and how the layers connect to each other. Each of these decisions interacts with the others. A decision about layer type changes which connection patterns make sense, and a decision about width changes how much memory and computation the network requires.
The scale of this space is difficult to appreciate without a concrete example. A network of just 8 layers where each layer has 10 possible types and 5 possible widths already contains (10 × 5)⁸ = 50⁸ ≈ 4 × 10¹³ possible configurations before any connection patterns are considered. Adding a modest branching structure multiplies this count by a factor that grows rapidly with depth. Real search spaces in NAS research contain on the order of 10¹⁸ possible architectures, a number that exhaustive evaluation cannot reach.
Human intuition navigates this space by following established design principles. Convolutional layers work well for image data. Residual connections help gradients flow through deep networks. Depth improves representational power up to a point. These principles compress the search to a region that past successes have validated, but they also create a bias toward architectures that resemble what has already been built. The parts of the search space outside this region remain unexplored, and some of the highest-performing architectures found by automated search live there.
There is an information-theoretic way to frame what a good architecture actually does. Shannon’s source coding theorem states that the most efficient representation of a signal removes everything redundant or predictable. Applied to architecture design, a good architecture efficiently compresses task-relevant information from its input, passing forward what matters for the label and discarding what does not. This is the minimum description length principle applied to network design, which holds that the best model of a dataset minimizes the combined length of the model description and the data given the model. A network with unnecessary layers or redundant connections is a less efficient model than one that achieves the same task with fewer resources. Barlow’s efficient coding conjecture, which proposed that the visual system removes statistical redundancy from natural scenes, is an instance of the same principle applied to biological neural architecture (Sejnowski, 2018). NAS, understood from this angle, is a search for the most efficient model of the mapping the task requires.
How does an algorithm represent an architecture?
Automated search requires an encoding that captures architectural decisions in a form the algorithm can read, modify, and compare. This encoding problem is the same one that genetic programming faced in the 1990s. Koza’s genetic programming framework represented each program as a tree, where nodes were operations and leaves were inputs or constants (Koza, 1992). The tree was the genome, and search operators worked directly on its structure. Mutation replaced a node with a different operation. Crossover cut two parent trees and grafted subtrees together, producing a child that inherited structure from both parents. The resulting child was a new program that neither parent was, and selection kept the ones that performed well. Neural architecture search inherits this entire framework. The tree becomes a cell encoding, the operations become layer types, and the fitness function becomes validation accuracy on a benchmark task.
The most widely used encoding in modern NAS represents architectures as small repeated units called cells, where a cell is a compact graph of operations and the full network is built by stacking copies of that cell in a fixed arrangement. Two cell types are typical. A normal cell processes feature maps without changing their spatial resolution. A reduction cell halves the spatial resolution while doubling the number of feature channels. Because both cell types are applied identically at every stage of the network, the search only needs to find a good cell rather than designing the entire network from scratch. This reduces a search over 10¹⁸ possible architectures to a search over roughly 10⁶ cell structures, a reduction of twelve orders of magnitude.
A cell encoding behaves exactly like a genome in Koza’s sense. The cell is a directed acyclic graph, meaning a graph where edges point in one direction and no path loops back to its starting node, where edges carry operations and nodes represent intermediate feature representations. The genome is the list of operations assigned to each edge. Mutation changes one entry in that list, replacing a 3×3 convolution on a given edge with a 5×5 convolution, for example. Crossover takes two parent encodings and produces a child by inheriting the first half of edge assignments from one parent and the second half from the other. The child architecture is evaluated by training a small network built from that cell and recording its validation accuracy. That accuracy is the fitness score. High-fitness cells survive into the next generation. Low-fitness cells are discarded. The loop runs for thousands of generations. Holland’s genetic algorithm framework explains why this works. Useful substructures tend to propagate through the population over generations, so high-performing motifs accumulate rather than being lost (Holland, 1992).
The candidate set defines the vocabulary the search draws from. Standard convolutions of different kernel sizes detect local patterns at different scales. Depthwise separable convolutions apply a separate filter to each input channel before combining them, substantially reducing parameter count. Dilated convolutions sample inputs at regular intervals, capturing patterns over larger regions without adding parameters. Average and max pooling summarize local regions by their mean or maximum value respectively. Identity connections pass the feature map through unchanged, allowing the search to effectively skip an edge when no operation improves performance. A search over cell encodings is a search over the assignment of this vocabulary to a graph structure, which is the kind of structured discrete problem that evolutionary algorithms were designed to solve.
What did evolutionary search find that humans would not design?
Evolutionary NAS treats architecture design as a competition. A population of candidate architectures is maintained, improving through successive generations of selection and variation. This approach is applied to image classification using a method called regularized evolution (Real et al., 2019). The population begins with simple architectures. At each step, a small subset is sampled and the best-performing member is selected as a parent. A child is created by mutating the parent’s encoding, trained on a benchmark task, and added to the population. The weakest member of the sampled subset is then discarded. This replacement strategy is what the word regularized refers to. Discarding only from the current sample rather than from the entire population keeps older architectures alive longer, which prevents the search from converging too quickly on a local optimum and gives the population time to explore.
What this process finds becomes clear by contrasting it with a human-designed cell. A practitioner building a convolutional network before automated search became standard would typically stack layers in sequence, apply a 3×3 convolution, add a residual connection that bypasses one or two layers, and repeat. The design is clean, interpretable, and grounded in intuition about what each operation contributes. NASNet, an architecture found by automated search, looks strikingly different. Within each cell, multiple branches operate in parallel on different pairs of input states. One branch applies a 5×5 depthwise separable convolution while a parallel branch applies a 3×3 average pooling operation, and the two results are added together. Other branches combine an identity connection with a dilated convolution, or pair two separable convolutions with asymmetric kernels, meaning filters whose height and width differ, such as a 1×7 convolution rather than a square 3×3. The full cell contains several such parallel combinations, with outputs merged by addition at the end (Zoph and Le, 2017). No practitioner would have designed this. There is no intuitive principle that leads here. The search found it by evaluating thousands of combinations that human intuition would have dismissed before trying.
The performance advantage of these found architectures is measurable. NASNet-A-Large achieved a top-1 accuracy of 82.7 percent on ImageNet, the standard large-scale image classification benchmark, surpassing the best hand-designed architectures available at the time. The search finds the same primitive operations practitioners use, but in combinations and connection patterns that no practitioner would have reached systematically in the time available for manual design. A human expert evaluates perhaps dozens of architecture variants over months of iteration. An evolutionary search evaluates thousands over the same period, accumulating evidence across generations without fatigue or the cognitive bias that makes familiar patterns feel safer than unfamiliar ones.
Can architecture search be made differentiable?
Evolutionary search evaluates each candidate architecture by training it, and the cost of these training runs dominates the overall search budget. Evaluating thousands of architectures, even for a small number of training epochs each, requires substantial compute. The original search by Zoph and Le required approximately 800 GPUs running for 28 days. DARTS, short for Differentiable Architecture Search, takes a different approach to this problem (Liu et al., 2019).
The key idea in DARTS is to replace the single operation on each edge with a weighted mixture of all candidate operations simultaneously. In evolutionary search, each edge carries exactly one operation and the search commits to that choice. DARTS defers that commitment until after training is complete. The weights are called architecture parameters, denoted α. Gradient descent then optimizes both the network weights and the architecture parameters at the same time. The network learns which operations matter by adjusting α across a single training run.
The mixed operation ō(x), where the overbar denotes the weighted combination across all candidates, on an edge connecting nodes i and j takes the following form:
ō(x) = Σ_o (exp(α_o) / Σ_o′ exp(α_o′)) · o(x),
where the fraction involving the exponentials is the softmax function, which converts the raw architecture weights into a probability distribution that sums to one. Network weights w are updated on the training set. Architecture parameters α are updated on a separate validation set, so they reflect generalization quality rather than training fit. At the end of training, the architecture is discretized by keeping the operation with the highest α value on each edge.
The result is a search that completes in roughly 4 GPU-days on a single GPU, compared to 22,400 GPU-days for the original evolutionary approach, a reduction of more than 99.98 percent in compute cost. The efficiency comes with a tradeoff. During search, each operation benefits from gradient signal flowing through all the others. When the architecture is discretized and a single operation is selected per edge, that shared support disappears. Operations that performed well as part of a mixture do not always perform well alone. This is called the discretization gap and it is the main limitation of the original DARTS formulation. Later work addressed it through regularization terms that encourage α to concentrate on a single operation early in training, partial channel connections that reduce memory cost, and more careful discretization procedures that consider operation interactions rather than selecting each edge independently. The core contribution of making architecture parameters differentiable remains a standard component of modern NAS pipelines.
What made NAS cheap enough to use?
Get Eleventh Hour Enthusiast’s stories in your inbox
Join Medium for free to get updates from this writer.
Even after DARTS reduced the search cost dramatically, a fundamental expense remained. Every candidate architecture still needed at least some training to evaluate its fitness. Multiplied across thousands of candidates, this cost added up. One-shot NAS eliminates it entirely with a single conceptual shift. Instead of training thousands of separate architectures, train a single architecture that contains all candidates simultaneously.
This single architecture is called a supernetwork. It is structured so that every possible path through its layers corresponds to a valid candidate architecture. During training, one path is sampled at each step and only the weights on that path receive a gradient update. Because all paths share the same underlying weight parameters, a weight that appears in many architectures is updated by contributions from all of them over the course of training. When training is complete, individual architectures are evaluated by extracting their path from the supernetwork and measuring validation performance with the inherited weights, without any additional training. The shared weights produce rankings that correlate closely enough with independently trained rankings to identify the strongest candidates reliably.
EfficientNet, a family of image classification architectures found by automated search, illustrates what this regime produces in practice (Tan and Le, 2019). An NAS procedure first identified a baseline cell structure, then applied a compound scaling method that simultaneously increases network depth, width, and input resolution according to a fixed ratio. The insight behind compound scaling is that these three dimensions interact. A deeper network benefits from higher resolution inputs, and wider layers benefit from greater depth. Scaling one dimension alone wastes capacity. EfficientNet-B7, the largest variant in the family, achieved 84.3 percent top-1 accuracy on ImageNet while using 8.4 times fewer parameters than the best competing architecture at the time. EfficientNet became the dominant architecture for efficient image classification for several years, setting a new standard for the accuracy-to-parameter tradeoff on ImageNet.
The supernetwork approach introduced a technical challenge that took several years to resolve. Paths of very different sizes send conflicting gradient signals to the shared weights. The optimal weight value for a wide path differs from the optimal value for a narrow one, and training pulls the shared weights in both directions simultaneously. Later work addressed this through a range of techniques including separate normalization statistics for different path widths and progressive sampling strategies that gradually shift training toward the target architecture. These refinements made one-shot NAS the practical standard for most production search pipelines.
Does the target hardware change what a good architecture is?
Every search procedure discussed so far optimizes for accuracy on a benchmark dataset. That is a reasonable proxy for quality, but it is not the same as usefulness on a real device. An architecture that achieves top benchmark accuracy on ImageNet while running too slowly on a mobile processor is not a good architecture for the billions of devices it needs to run on. Hardware-aware NAS changes the fitness function to reflect this reality, adding a hardware efficiency term alongside validation accuracy so the search finds architectures that are both accurate and fast on the target device.
The reason hardware efficiency cannot be inferred from parameter count alone is that different operations behave differently on different hardware. A depthwise separable convolution uses far fewer multiply-add operations than a standard convolution, but whether it is actually faster depends on the device. A mobile processor and a server GPU differ in three properties that determine how fast an operation actually runs. Memory bandwidth is the rate at which data can be moved between memory and compute units. Parallelism is how many operations the hardware can perform simultaneously. Instruction support refers to whether the hardware has specialized circuitry for particular operations. An operation that is fast on one device can be slow on another for reasons that have nothing to do with its theoretical cost. An architecture that reports low latency on a server GPU may run two to five times slower on the embedded processor it will actually be deployed on. The only reliable way to know is to measure on the target hardware directly.
MnasNet, a mobile architecture found by hardware-aware search, did exactly this (Tan et al., 2019). Rather than using a proxy metric for efficiency, the search measured actual latency on a specific Pixel phone and included that measurement directly in the reward signal, meaning the scalar feedback the search algorithm receives after evaluating each candidate architecture. The reward function combined ImageNet accuracy and on-device latency, weighted by a target latency constraint. The architectures it found contain filter sizes, channel counts, and layer sequences that no accuracy-only search would have produced. EfficientDet extended the same approach to object detection, the task of identifying and localizing multiple objects within an image simultaneously, finding architectures that balance accuracy and latency on both server and mobile targets.
Hardware-aware NAS treats the deployment target as part of the problem specification rather than as a constraint applied after the architecture is chosen. This changes what the search finds. The same operations, connected in different patterns, produce very different behavior on different hardware. By making the device part of the fitness function, the search explores regions of the architecture space that accuracy-only methods never reach, and the architectures it finds there are the ones that actually run in production.
Did evolutionary computation ever really go away?
The tools behind modern NAS have older roots than their production deployment suggests. Koza described a framework for evolving programs represented as trees, where selection, crossover, and mutation operated on program structure rather than on numerical weights. Holland’s genetic algorithms provided the foundational theory of how useful substructures propagate through a population over generations. When deep learning took over after AlexNet in 2012, evolutionary computation receded from the mainstream AI discussion. The story that followed looked like a clean victory for gradient descent.
It was not a clean victory. The modern evolutionary NAS pipeline is recognizable to anyone familiar with the 1990s genetic algorithms literature. The cell encoding used in NASNet is a fixed-length genome. Mutation replaces one operation on an edge with another. Crossover produces a child architecture that inherits structure from two parents. Tournament selection, where a small subset of the population is sampled and the weakest member discarded after a child is produced, keeps the strongest candidates in the gene pool. Every component maps directly onto what Holland and Koza described three decades ago. The vocabulary and domain changed, but the algorithm running inside these production systems did not.
The reason evolutionary algorithms survived where other alternatives to gradient descent did not is that they were solving a different problem. Gradient descent is powerful at optimizing the weights of a fixed architecture, but it has no mechanism for searching over architectures. Architecture design is a discrete combinatorial problem, and evolutionary algorithms are built for exactly that. They explore multiple regions of the search space simultaneously, accumulate useful substructures across generations, and require no differentiability assumptions. The field did not choose evolutionary computation over gradient descent. It discovered that the two methods were solving different problems all along, and that both were needed. The architectures on every smartphone camera today are the result of that combination.
Where does automated architecture search appear in current systems?
The most immediate answer is in the device in most people’s pockets. EfficientNet and MobileNet variants found by NAS are the backbone of on-device computer vision in modern smartphones. Every time a phone camera segments a subject from a background, detects a face, or classifies a scene in real time, it is running an architecture that a search algorithm found. No human designed these architectures. They were discovered.
NAS has since moved into the transformer era. The same weight-sharing and hardware-aware search methods developed for convolutional architectures are now applied to find efficient transformer variants for language and vision tasks. NAS-BERT and its successors search over attention head configurations, layer depths, and hidden dimensions to find transformer architectures that meet specific latency and memory budgets on target hardware (Xu et al., 2021). The search problem is the same one that motivated the original NAS work. The architecture space is different, but the tools are the same.
The connection to the broader AutoML agenda has also deepened. Modern AutoML pipelines search not just over architectures but over the full model development process, including preprocessing steps, training schedules, and data augmentation strategies. NAS is the component of that pipeline that receives the most attention because architecture choices have the largest impact on both accuracy and deployment cost. The search space grew, the hardware targets multiplied, and the evolutionary and gradient-based methods that started with image classifiers scaled to meet both. The fitness function that once measured only ImageNet accuracy now measures a combination of accuracy, latency, memory footprint, and energy consumption across multiple target devices simultaneously.
Conclusion
Neural architecture search replaced human intuition with a search procedure over a space that human intuition cannot cover. The architectures it found are irregular, asymmetric, and unlike anything a practitioner would have designed, and they outperform hand-designed alternatives on every major benchmark. The tools that produced them, population-based selection, mutation, crossover, and fitness-driven elimination, are the same tools that Holland and Koza described three decades ago. The field that eclipsed evolutionary computation in 2012 built its best architectures with it.
References:
Koza, J.R. (1992). Genetic Programming. MIT Press. https://mitpress.mit.edu/9780262527910
Holland, J.H. (1992). Adaptation in Natural and Artificial Systems (2nd ed.). MIT Press. https://mitpress.mit.edu/9780262581110
Zoph, B. and Le, Q.V. (2017). Neural architecture search with reinforcement learning. ICLR 2017. https://arxiv.org/abs/1611.01578
Real, E., Aggarwal, A., Huang, Y., and Le, Q.V. (2019). Regularized evolution for image classifier architecture search. AAAI 2019. https://arxiv.org/abs/1802.01548
Liu, H., Simonyan, K., and Yang, Y. (2019). DARTS: Differentiable architecture search. ICLR 2019. https://arxiv.org/abs/1806.09055
Tan, M. and Le, Q.V. (2019). EfficientNet: Rethinking model scaling for convolutional neural networks. ICML 2019. https://arxiv.org/abs/1905.11946
Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., and Le, Q.V. (2019). MnasNet: Platform-aware neural architecture search for mobile. CVPR 2019. https://arxiv.org/abs/1807.11626
Xu, Y., Zhang, X., Zhang, M., Yang, Y., and Tian, Q. (2021). NAS-BERT: Task-agnostic and adaptive-size BERT compression with neural architecture search. KDD 2021. https://arxiv.org/abs/2105.14444
Sejnowski, T. (2018). The Deep Learning Revolution. MIT Press. https://mitpress.mit.edu/9780262038034/the-deep-learning-revolution/
