Cache Optimization: Hardware Techniques

Sohom Dasgupta

23 min read

·

4 days ago

--

Listen

Share

A processor can perform calculations extremely quickly, but it often has to wait for the data. Cache optimization is the art of reducing that waiting time without spending too much silicon area, power, or memory bandwidth. In this article, we will build the necessary background first, then explain the commonly discussed hardware techniques in plain language. This article is written for readers meeting cache optimization for the first time. You do not need to have designed a processor before. The goal is not just to memorize each technique, but to be able to explain what problem it solves, how it works, and what it gives up in return.

What is a cache and what is does?

A CPU can execute instructions much faster than main memory can deliver the required data. If every memory access had to wait for data from main memory, the CPU would spend a significant amount of time idle.

A cache is a small, fast memory that stores copies of data and instructions that are likely to be used again soon.

Think of it as having a desk in a library. The library contains thousands of books, but you don’t need all of them at the same time. Walking to the shelves every time you need to look up a fact can become tedious and time-consuming.

What’s the solution? Keep the books you use frequently on your desk so they’re readily accessible.

However, since the desk has limited space, you must decide which books deserve a place on it. A cache faces the same challenge: it has limited storage capacity, so it needs a way to decide which data to keep.

Understanding the Memory Hierarchy

A computer uses different types of memory to balance speed, storage capacity, and cost. These memory components are arranged in a structure known as the memory hierarchy. Let us use Fig 1 to understand the pyramid. At the top of this hierarchy are small, extremely fast registers located closest to the CPU which are the fastest but have very limited storage capacity. The CPU first looks for the required data in its registers and then into the caches. The cache itself is organized into multiple levels. L1 is typically the smallest and fastest cache, followed by the larger L2 cache and, in many processors, an even larger L3 cache. L1 and L2 caches usually reside inside the individual CPU cores while L3 cache is shared by multiple CPU cores. These caches are typically implemented using SRAM (Static RAM). If the required data is not found in the caches, the CPU must retrieve it from main memory (RAM). It is implemented using DRAM, which provides larger storage capacity but has higher access latency compared to cache memory. Further down, we have secondary storage, such as SSDs and HDDs, followed by external and offline storage, which offer much larger capacities but are comparatively slower. The fundamental principle of this hierarchy is that ‘as we move closer to the CPU, memory becomes faster, smaller, and more expensive per bit, whereas as we move away from the CPU, memory becomes slower, larger, and cheaper per bit’

This hierarchy allows the CPU to access frequently needed data quickly while still providing access to large amounts of information when required.

Before we move ahead, there are many terms that would come up which might seem confusing. So here are 5 terms to know related to a cache system

1. Cache hit: The requested block is already in the cache.
2. Cache miss: The requested block is not in the cache, so the system must fetch it from a lower level.
3. Hit time: How long it takes to access data when the cache contains it.
4. Miss penalty: The extra delay caused by a miss, including the work needed to obtain the data.
5. Bandwidth: How much data can be transferred per unit of time. A system can have high bandwidth but still have high latency for one request.

Locality: How caches work?

As humans, when we read about a topic from a book, we often refer to the same page again, read the content immediately before or after to understand and get better context about it. Programs behave in a similar way when accessing memory. They tend to reuse data they have accessed recently, or access data stored close to it in memory. This predictable behaviour is known as locality. It can be classified in 2 types:

Temporal Locality

This refers to the tendency of a program to access the same data or instructions repeatedly within a short period of time. Consider a loop that repeatedly uses a variable to calculate a running total. The variable is accessed again and again as the loop executes. Instead of fetching that variable from main memory every time, the cache can keep a copy readily available for subsequent accesses.

Spatial Locality

This refers to the tendency of a program to access memory locations that are close to one another. Imagine a program processing an array containing 1,000 numbers. It starts with the first element, then moves to the second, third, and so on. Since these elements are stored in nearby memory locations, accessing one element suggests that neighbouring elements may be needed soon as well. A cache takes advantage of this pattern by fetching a block of data containing the requested element and some of its neighbouring data, rather than fetching only a single element.

How Cache Placement Works

From the previous section, we understood the importance of data being efficiently fetched from the main memory and stored into the cache. But a cache does not usually fetch data one byte at a time. Instead, data is transferred between memory and the cache in fixed-size chunks called blocks or cache lines. For example, if a cache line is 64 bytes, requesting a single byte may cause the entire 64-byte block containing it to be brought into the cache.

So where should this block be stored?

The cache follows a placement rule to determine where each incoming block can be stored. Let us understand the three common approaches with the library examples:

• Direct-mapped cache: Each memory block is assigned exactly one possible location in the cache. Imagine that every book has an assigned spot on your desk. If two books are assigned the same spot, you must remove one before placing the other there. Even if other parts of the desk are empty, those books cannot use them.
• Set-associative cache: Each memory block is assigned to a particular set, but it can occupy any available line within that set. These possible locations are called ways. Imagine dividing your desk into several sections, with each section having two spaces for books. A book is assigned to a particular section, but you can place it in either of the two spaces. This gives you more flexibility and reduces the chances of books competing for the same spot.
• Fully associative cache: A memory block can be placed in any cache line. Imagine having no fixed sections or assigned spots on your desk. You can place any book in any available space. This gives you the greatest flexibility, but the system needs to search more broadly to find a particular book.

This brings up a question, ‘How Does the Cache Identify a Block?’. The cache needs a way to determine whether the data stored at a particular location matches the address requested by the CPU. To do this, it stores a small piece of address information called a tag alongside each cache line. When the CPU requests data, the cache uses part of the address to identify the relevant location or set. It then compares the address’s tag with the stored tag. The tags match, it is a cache hit. If not, it is a cache miss. This tag comparison is an essential part of determining whether the cache can immediately provide the requested data.

Now that we understand how caches store and locate data, let’s explore how hardware designers optimize these structures to make memory access faster, reduce latency, improve bandwidth, and use power more efficiently. These techniques aim to narrow the performance gap between the CPU and main memory.

Optimization Techniques

Before exploring cache optimization techniques, it is useful to understand how cache performance is measured.

Measuring Cache Performance: AMAT

A useful first-order model for evaluating memory access performance is Average Memory Access Time (AMAT):

AMAT = Hit Time + (Miss Rate × Miss Penalty)

This formula highlights three important factors: Hit time, Miss rate and Miss penalty which influence memory access time. We already discussed about these earlier under to know terms!

Cache optimization techniques target one or more of these factors. Broadly, they can be grouped into five categories:

1. Reducing hit time: Making it faster to access data already present in the cache.
2. Increasing cache bandwidth: Allowing the cache to handle more data transfers within a given period.
3. Reducing miss penalty: Minimizing the time spent waiting when requested data is not found in the cache.
4. Reducing miss rate: Improving the chances that requested data is already available in the cache.
5. Using parallelism to reduce miss penalty or miss rate: Fetching data in advance or overlapping memory operations to reduce the impact of cache misses.

There is no single best cache design for every workload. AMAT is a useful model for understanding cache performance, but since it is a first order model, it does not capture every aspect of modern processors. For example, out-of-order execution, overlapping memory requests, queueing delays, bandwidth limitations, and contention can all influence actual performance. Each optimization technique affects one or more of these factors, often introducing trade-offs in power consumption and hardware complexity.

The central design question is: Which performance bottleneck matters most for a particular processor, and which optimization offers the best trade-off? Let’s explore these techniques, understand how each works, and examine the problems they are designed to solve.

1. Small and simple L1 Cache

Bottleneck: L1 cache access time is often on the processor’s critical path. Since the L1 cache needs to respond quickly, a larger cache, higher associativity, or more complex selection logic can increase hit latency and energy consumption.

Mechanism: The cache designer chooses a size and organization that allow the cache to perform tag lookup, data access, tag comparison, and way selection within the processor’s timing and power budgets. A direct-mapped cache has only one possible location for a given memory block, making the lookup and selection process relatively simple. A set-associative cache offers multiple possible locations, but the cache must identify which way contains the requested block. Some designs overlap parts of the tag lookup and data access to reduce the overall hit time. If we try to correlate with the library example, direct-mapped cache arrangement would help finding the book easily but two frequently used books may compete for same spot. But if we go with a two-way set associative cache, each book is assigned to a particular section of the desk and that section has two available spaces. When you need a book, the cache checks the two candidate tags to identify the correct one. The second arrangement reduces the chances of books competing for the same location, but it requires more complex lookup and selection hardware.

Performance reasoning: A larger cache may reduce capacity misses, but it can also increase access time and energy consumption. Similarly, greater associativity may reduce conflict misses but require additional tag comparisons, multiplexing, and hardware activity. The designer must balance these factors. A reduction in miss rate is beneficial only if it compensates for any additional cost in hit time and power.

2. Way Prediction

Bottleneck: Associativity helps reduce conflict misses, but it can increase cache hit time. In a conventional set-associative cache, the cache may need to compare multiple tags and select the matching way before delivering the requested data. This additional work can extend the hit path.

Mechanism: Way prediction uses a small predictor to estimate which way is most likely to contain the requested memory block. The way predictor is a hardware component inside the cache system, typically implemented using a small amount of storage and associated control logic. A way-prediction mechanism can be implemented using four components: the Way Prediction Module (WPM), Way Record Buffer (WRB), Tag Record Buffer (TRB), and Fallback Multiplexer.

• Way Prediction Module (WPM): Acts as a lookup structure that uses the incoming address or set information to predict which cache way is likely to contain the requested data.
• Way Record Buffer (WRB): Stores information about previously predicted or accessed ways. This history can help the predictor make more accurate predictions for future requests.
• Tag Record Buffer (TRB): Holds tag information associated with the predicted way, allowing the cache to check whether the predicted way contains the requested memory block.
• Fallback Multiplexer: Handles the selection between the fast and fallback paths. When the predicted way is correct, it allows the cache to use the fast, single-way access path. When the prediction is incorrect, the cache uses the fallback path, checking the tags of the other ways to locate the requested block.

So, instead of waiting to identify the correct way through a conventional lookup, the cache uses this prediction to begin selecting the likely data early. Consider a 2-way set-associative cache with Way 0 and Way 1.

Suppose the predictor guesses that the requested block is in Way 1.

• If the block is actually in Way 1, the prediction is correct and the cache can provide the data through the fast path.
• If the block is in Way 0, the prediction is incorrect. The cache must check the alternative way, resulting in additional access time.

This allows the cache to retain the benefits of associativity while attempting to achieve a hit time closer to that of a simpler cache.

Performance Reasoning: Let us take a simplified model to understand the maths behind.

Expected Access Time = (Pc × Tc) + (Pw × Tw)

Pc = Probability of a correct prediction
Pw = Probability of a wrong prediction
Tc = Access time for a correct prediction
Tw = Total access time for an incorrect prediction

Suppose a way predictor has the following characteristics:

Prediction accuracy: 90%, Access time for a correct prediction: 1 cycle, Total access time for an incorrect prediction: 3 cycles. What is the expected cache access time?

Press enter or click to view image in full size

For comparison, if a conventional lookup took 2 cycles for every access, its average access time would be 2 cycles. Under these illustrative assumptions, way prediction reduces the average from 2 cycles to 1.2 cycles.

3. Pipelined access and multibanked caches

Bottleneck: As processor speeds increase, a cache may struggle to keep up with the rate at which the CPU requests data. A cache access path may also be too long to complete within a single clock cycle. This can limit the amount of data the cache can deliver over time. Two situations can contribute to this problem:

• A cache access may take too long to complete within a single clock cycle.
• A processor may issue multiple memory requests in the same clock cycle, while the cache may not be able to handle all of them simultaneously.

Mechanism: Hardware designers use two techniques to increase cache bandwidth:

A. Pipelined Cache Access

Instead of completing the entire cache access process in one clock cycle, the operation is divided into multiple pipeline stages. Each stage performs part of the access process. This allows the cache to begin processing a new request while an earlier request is still moving through the pipeline. As a result, the cache can operate at a higher clock frequency and improve its throughput. Pipelining is widely used in L1 caches to support higher clock frequencies and improve cache throughput.

However, pipelining increases access latency because each request must pass through multiple stages before completion.

For example, the following data from Computer Architecture, A Quantitative Approach describes how instruction-cache access latency changed across Intel processors:

• Intel Pentium: 1 clock cycle
• Pentium Pro to Pentium III: 2 clock cycles
• Pentium 4 and Intel Core i7: 4 clock cycles

The increased latency also affects processor performance:

• Instruction cache: More pipeline stages can increase the penalty for branch mispredictions because the processor may need more time to discard incorrectly fetched instructions and fetch instructions from the correct execution path.
• Data cache: More pipeline stages can increase the number of clock cycles between issuing a load instruction and using the data it retrieves.

Pipelining the instruction cache is generally easier than pipelining the data cache because effective branch prediction can help reduce the impact of instruction-fetch latency.

B. Multibanked Caches

Modern superscalar processors can issue multiple memory operations in a single clock cycle. To handle these requests, the cache can be divided into multiple independent banks, with each bank supporting an independent access. This allows multiple memory accesses to proceed simultaneously, provided they target different banks.

A common method of distributing memory blocks across banks is called sequential interleaving. In this method, consecutive memory blocks are assigned to consecutive banks.

For example, if a cache has four banks, the bank number can be determined using:

Bank Number = Block Address % 4

Press enter or click to view image in full size

However, if it requests blocks 0 and 4 together, both requests target Bank 0. This creates a bank conflict, which can prevent both accesses from being served independently in the same cycle.

Performance Impact:

Pipelining: Increases cache throughput by dividing the access process into stages, allowing the cache to operate at a higher clock frequency. The trade-off is increased access latency, which can affect branch recovery and data dependencies.

Multibanking: Increases cache bandwidth by allowing multiple independent accesses in the same clock cycle. Its effectiveness depends on how memory requests are distributed across the banks. Requests targeting the same bank can create conflicts.

Multiple banks are used in different cache levels for different purposes:

• L1 cache: Banking helps support multiple memory accesses per clock cycle. The book gives the Intel Core i7 as an example, with four L1 banks supporting up to two memory accesses per clock.
• L2 and L3 caches: Banking helps handle multiple outstanding L1 cache misses, provided the requests do not conflict by targeting the same bank. This is important for supporting nonblocking caches, which are discussed in the next optimization.

Multibanking can also help reduce energy consumption in both caches and DRAM.

4. Non Blocking Caches

Bottleneck: In a blocking cache, when a memory request results in a cache miss, the cache must wait for the requested data to arrive from the next memory level before processing further requests. This prevents the cache from servicing other requests, even if some of them could result in cache hits.

Mechanism: A nonblocking cache allows the processor to continue accessing the cache while an earlier cache miss is being serviced. It can also support multiple outstanding cache misses, allowing several memory requests to be processed concurrently.

Two important capabilities make this possible:

• Hit under miss: Allows the cache to service a cache hit while an earlier cache miss is still waiting for data.
• Miss under miss: Allows the cache to initiate another memory request even when one or more previous cache misses are still outstanding.

To manage these operations, nonblocking caches use hardware structures called Miss Status Holding Registers (MSHRs). These registers track outstanding cache misses and maintain the information required to process the returning data and associate it with the appropriate request. Suppose the processor issues three memory requests:

• Request A: Results in a cache miss. The cache sends a request to the next memory level and waits for the data.
• Request B: Arrives while A is still outstanding and results in a cache hit. A nonblocking cache can service B without waiting for A to complete. This is called hit under miss.
• Request C: Also results in a cache miss while A is still outstanding. If the cache has an available MSHR and the memory system can handle another request, it can initiate C without waiting for A to complete. This is called miss under miss.

Thus, multiple memory operations can remain in progress simultaneously instead of forcing the processor to wait for each miss to finish before proceeding.

Performance Impact: The primary benefit of nonblocking caches is overlapping memory operations. While one request waits for data from a slower memory level, the processor can continue servicing other cache requests and potentially make progress on independent instructions.This improves cache performance by reducing the time the processor spends waiting for memory operations to complete. However, the actual memory latency of an individual request may remain unchanged.The performance improvement depends on factors such as the number of available MSHRs, memory bandwidth, bank conflicts, data dependencies, and the processor’s ability to execute independent instructions.

5. Critical word first and early restart

Bottleneck: When a cache miss occurs, the memory system typically transfers an entire cache block into the cache. However, the processor may initially need only one word from that block to continue executing instructions. Waiting for the entire block to arrive before providing the requested word can introduce unnecessary delay.

Mechanism: Two techniques help reduce the waiting time for the requested data:

• Critical Word First: Instead of transferring the cache block in its normal order, the memory system prioritizes the word requested by the processor and sends it first. The remaining words are transferred afterward.
• Early Restart: The memory system follows the normal cache-block transfer order, but the processor is allowed to resume execution as soon as the requested word arrives, without waiting for the entire block to be transferred.

Both techniques aim to reduce the time required to deliver the critical data to the processor. Suppose a cache block contains 64 bytes, divided into eight words of 8 bytes each. A processor requests word 6 from this block, but the cache experiences a miss.

• Critical Word First: Word 6 is transferred first, followed by the remaining seven words.
• Early Restart: The words are transferred in their normal order. As soon as word 6 arrives, the processor can resume execution, while the remaining words continue to arrive.

In both cases, the processor may resume useful execution before the entire cache block has been transferred, provided the required data is available and the processor is ready to use it.

Performace Impact: The primary benefit is reducing the effective waiting time after a cache miss. Instead of waiting for the entire cache block to arrive, the processor can receive the requested word earlier and potentially continue executing dependent instructions. These techniques are particularly useful when cache blocks are large because transferring the entire block can take considerably longer than transferring an individual word. However, the remaining words still matter because spatial locality may cause the processor to access nearby data soon afterward.

6. Merging write buffers

Bottleneck: When the processor performs write operations, the data may need to be temporarily stored in a write buffer before being transferred to the next memory level. However, write buffers have a limited number of entries. If writes accumulate faster than they can be transferred, the buffer can become full, forcing the processor or cache to stall until some entries are released.

Mechanism: A merging write buffer combines multiple pending writes that target compatible locations within the same memory transfer region. Instead of allocating a separate buffer entry for every write, the buffer can combine multiple writes into a single entry. Valid bits are used to identify which bytes or words within that entry contain updated data. This allows the buffer to accommodate more pending writes without requiring additional entries. Let us understand with an example

Suppose a write buffer has an entry capable of holding four 64-bit words. Since each 64-bit word occupies 8 bytes, one entry can hold 4 × 8 = 32 bytes

Now suppose the processor performs four writes to four consecutive 8-byte words within the same 32-byte memory region.

Press enter or click to view image in full size

Without write merging, these writes could occupy four separate buffer entries (Fig 4.)

Press enter or click to view image in full size

With write merging, all four writes can be combined into a single buffer entry, with the valid bits identifying the updated words (Fig 5.)

As a result, the buffer uses one entry instead of four, leaving more entries available for other pending writes.

Performace Impact: The primary benefit of merging write buffers is reducing pressure on the limited buffer entries.

By combining compatible writes:

• The buffer can accommodate more pending writes using the same number of entries.
• Fewer separate write operations may need to be sent to the next memory level.
• The likelihood of the write buffer becoming full can be reduced, helping avoid processor stalls.

7. Hardware prefetching

Bottleneck: Normally, a memory request begins only when the processor needs data that is not already available in the cache. If this results in a cache miss, the processor must wait for the data to be retrieved from a lower memory level. This introduces memory latency that can delay instruction execution, particularly when the processor repeatedly accesses data from slower memory levels.

Mechanism: Hardware prefetching attempts to reduce this waiting time by predicting which instructions or data the processor is likely to need next and fetching them before they are explicitly requested. Instruction prefetch is frequently done in hardware outside of the cache. Typically, the processor fetches two blocks on a miss: the requested block and the next consec utive block. The requested block is placed in the instruction cache when it returns, and the prefetched block is placed in the instruction stream buffer. If the requested block is present in the instruction stream buffer, the original cache request is canceled, the block is read from the stream buffer, and the next prefetch request is issued.

Press enter or click to view image in full size

Hardware prefetchers monitor memory access patterns and use this information to predict future requests. This uses additional dedicated hardware logic inside the processor’s memory subsystem. It is not something that requires the compiler or software to explicitly request every prefetch. The exact placement and number of prefetchers vary between processors.

Common approaches include:

• Sequential prefetching: Detects sequential memory accesses and fetches the next cache line in advance.
• Stride prefetching: Detects regular access patterns with a fixed distance between memory addresses and prefetches the corresponding future cache lines.

The prefetched data is typically brought into a cache or an intermediate buffer, depending on the processor’s design. Consider a loop that processes elements of an array sequentially. As the processor accesses elements in the current cache line, the hardware prefetcher detects the sequential access pattern and requests the next cache line before the processor needs it. Ideally, the next cache line arrives while the processor is still processing the current one. When the processor eventually accesses that data, it may already be available in the cache, avoiding the delay of waiting for a demand fetch.

Performance Impact: The primary benefit of hardware prefetching is reducing the effective waiting time caused by cache misses by fetching data in advance.

Its effectiveness depends on several factors:

• Timeliness: The prefetched data must arrive before the processor needs it. If it arrives too late, it cannot hide the memory latency.
• Accuracy: The prefetcher should predict data that the processor will actually access. Incorrect predictions waste resources.
• Coverage: The prefetcher should successfully anticipate a significant portion of the cache misses that would otherwise occur.
• Resource cost: Prefetching consumes memory bandwidth, cache capacity, and power. Excessive prefetching can compete with demand requests and evict useful cache data.

8. HBM as an extension of memory hierarchy

Bottleneck: Modern processors, particularly GPUs, perform thousands of calculations in parallel. Workloads such as AI training, deep learning, scientific simulations, and large matrix multiplication require continuous access to enormous amounts of data.

The challenge is not just storing this data. The memory system must also deliver it quickly enough to keep the processor’s computational units occupied. Traditional memory technologies, such as DDR, provide large storage capacities but may not offer sufficient bandwidth for demanding GPU workloads. To address this challenge, modern GPUs increasingly use High Bandwidth Memory (HBM).

Mechanism: HBM is a type of DRAM designed to provide high memory bandwidth through a combination of 3D stacking and a very wide memory interface.

Unlike conventional memory modules, HBM consists of multiple DRAM dies stacked vertically. These dies communicate through tiny vertical electrical connections called Through-Silicon Vias (TSVs).

The HBM stack is placed close to the GPU, typically within the same package, allowing the processor to communicate with memory through a wide interface.

Its main architectural advantages include:

• 3D stacking: Multiple DRAM dies are stacked vertically, providing substantial memory capacity in a compact physical area.
• TSVs: These connections allow signals and data to travel between the stacked memory dies.
• Wide memory interface: Many bits can be transferred in parallel, providing high memory bandwidth.
• Close integration: Shorter physical connections help support high-speed communication and improve energy efficiency.

How HBM Is Used in Modern Systems?

HBM can serve different roles depending on the memory architecture.

1. HBM as the Main GPU Memory

In many modern GPUs, HBM serves as the primary high-bandwidth memory available to the GPU. It stores the data required for computation, including matrix values, model parameters, intermediate results, and other application data.

In this arrangement, the memory hierarchy may look like:

Registers → L1 Cache → L2 Cache → HBM

The GPU first checks its caches for the required data. If the data is not available, it is fetched from HBM.

Here, HBM is not an L4 cache. It is the GPU’s main memory, positioned below its on-chip caches.

2. HBM as an Additional Cache Level

Another architectural approach is to use HBM as a large cache between the processor’s last-level cache and system memory.

In this arrangement, HBM acts like an L4 cache:

Regsiters → L1 → L2 → L3 → HBM (L4) → Main Memory

The idea is to use HBM as a large, high-bandwidth storage layer that keeps frequently accessed data closer to the processor. When the processor requests data, the memory system checks whether the requested block is already present in HBM. If it is, the data can be retrieved from HBM instead of accessing the slower memory below it.

This approach can reduce accesses to the lower memory level, but it introduces an important challenge: the system must track which memory blocks are currently stored in HBM.

The Challenge: Managing Tags in a Large HBM Cache

A cache divides memory into smaller units called blocks or cache lines. Each block stored in the cache needs metadata, known as a tag, to identify the memory address it represents.

For a relatively small cache, storing this metadata is manageable. However, when HBM is used as a cache with a capacity of hundreds of megabytes or even gigabytes, the number of blocks becomes enormous.

Consider an HBM cache with a capacity of 1 GiB and a block size of 64 bytes.

The number of blocks would be:

(1 GiB = 2³⁰ bytes) / (64 bytes) = 16,777,216 blocks

This means the cache must track more than 16 million individual blocks.

Storing the tags for all these blocks can require a substantial amount of additional memory.

One way to reduce this overhead is to increase the cache block size.

For example, with a block size of 4 KiB:

(1 GiB) / (4 KiB = 4x2¹⁰ bytes) = 262,144 blocks

This represents a 98.44% reduction in the number of entries, significantly reducing the metadata storage requirement.

However, larger blocks introduce another trade-off. The processor may need only a small portion of a block, while the memory system transfers a much larger amount of data. This can waste bandwidth and reduce the number of distinct memory regions that fit in the cache.

Therefore, designing an HBM-based cache requires balancing tag-storage overhead, lookup latency, block size, and bandwidth utilization.

Different Approaches to HBM-Based Cache Design

Two approaches discussed in cache research are the Loh-Hill cache and the Alloy Cache.

i) Loh-Hill approach

This approach uses HBM to store both cache data and its associated tag information. When the processor requests a memory block, the system must determine whether that block is present in HBM before retrieving the required data. But this requires two accesses to DRAM: one for the tags and one for data. The separate tag lookup and data access can introduce additional latency, depending on how the memory is organized.

ii) Alloy Cache approach

Alloy Cache aims to reduce this lookup overhead by organizing tag information and data access so they can be handled together in a memory burst. It molds the tag and data together and uses a direct mapped cache structure. This allows the L4 access time to be reduced to a single HBM cycle by directly indexing the HBM cache and doing aburst transfer of both the tag and data

However, the approach can also transfer unnecessary data on a miss, consuming memory bandwidth that could otherwise be used for useful requests.

Performance Impact: HBM provides high bandwidth and allows substantial memory capacity to be placed close to the processor. When used as the main GPU memory, it helps supply data to highly parallel workloads at a high rate. When organized as an additional cache level, it can potentially reduce accesses to slower system memory and improve performance for workloads with large working sets.

However, the effectiveness of an HBM-based cache depends on more than its raw bandwidth. Tag lookup time, cache hit rate, block size, and unnecessary data transfers all influence overall performance.