A GPU Performance Number: What Exactly Is It Measuring?

British physicist Lord Kelvin once said: "When you can measure what you are speaking about, and express it in numbers, you know something about it." This quote comes from his lecture on electrical measurement.

But in a GPU performance test, the number did not bring an answer; instead, it brought a question.  The Pixel Fill Rate we measured was only half of the specification value, or even lower. Did the GPU fail to reach its rated capability, or did the test itself measure something else?

To answer this question, we began removing interference factors from the test one by one: format, blending, bandwidth, cache, instruction dependencies... In the end, we found that the difficulty of accurate measurement is not obtaining a number, but confirming what that number actually represents.

This also became the first step in building our GPU performance analysis framework.

What Is the Difference Between Specification Values and Actual Measurements?

Following this question, we further compared the measured Pixel Fill Rate results of multiple GPUs against their corresponding specification values. Under the current test configuration, some measured values differed significantly from the specification values.

Table 1: Comparison of measured and specification Pixel Fill Rate values for multiple GPUs under a specific test configuration.

Note: This table is intended to demonstrate the impact of test conditions on result interpretation, not for product performance ranking. See the test notes for the specific sources of specification values and measured values, frequency conventions, and calculation methods.

From the data, the measured values are only half or even a quarter of the specification values. Faced with this gap, we must first avoid a premature judgment: a measured value below the specification value does not necessarily mean the GPU has failed to reach its rated capability, nor does it directly indicate a problem with the testing tool itself. Specification values typically describe the upper limit of a hardware unit's capability under specific conditions, while general-purpose benchmarks often need to cover a more complete graphics pipeline and application behavior — the two have different testing goals and operating conditions.

Therefore, before comparing measured values with specification values, the more important question is: do these two numbers use the same definitions, test conditions, and calculation conventions?

Taking this Pixel Fill Rate test as an example, after further inspection, we found at least two variables worthy of attention:

1)     Rendering format. The test used the R16G16B16A16_FLOAT format. Different formats may correspond to different amounts of data per pixel and different processing paths, so the test results will also differ.

2)     Blending operation. The test enabled blending, and the GPU needed to first read back the SRC value from the framebuffer (16-bit × 4 channels), perform blending calculation with the new value, and then write back the result. Compared with not enabling blending, the processing path may be different.

How to Make the Test Closer to What It Intends to Measure

General-purpose benchmarks usually cover a relatively complete execution path. A graphics scene may simultaneously involve vertex processing, rasterization, texture sampling, depth testing, blending, and memory access, and ultimately produce a composite result such as frame rate or execution time.

We tried a different approach: instead of addition, do subtraction.

The core idea is to control key variables so that each test is closer to the object it intends to measure, and to verify the stability and repeatability of the results under clearly recorded software and hardware environments and test conditions.

•   Pixel Fill Rate: extreme subtraction

To test Pixel Fill Rate separately, the key is to control the influence of other factors:

Improve rasterization efficiency. Draw only the simplest full-screen triangle as the primitive, reducing vertex processing and primitive assembly overhead, and cover the render target in a relatively simple way.

1)     Disable blending. Reduce reading of existing framebuffer data and additional blending processing.

2)     Format alignment. Use the R8G8B8A8 format for the render target, and before comparison confirm that this format is comparable to the definition and calculation basis of the referenced specification value.

3)     Do not load textures. Keep the test focused on pure pixel fill capability.

The core idea of this "extreme subtraction" is: each non-target variable controlled reduces one possible source of interference, making the test result more focused on the target processing path.

•   Texel Fill Rate: cache isolation from bandwidth

The difficulty in testing Texel Fill Rate is that texture data is usually very large, and it is easy to turn the test into a bandwidth test rather than a sampling capability test. The solution is as follows:

1)     Use cache to isolate the impact of bandwidth. Choose a small 128×128 texture (R8G8B8A8 format) to increase the probability that texture data resides in cache.

2)     Each pixel simultaneously samples and combines 8 textures to increase the proportion of texture sampling in the overall load and attempt to improve the utilization of texture sampling units.

When the texture data is small enough and the probability of cache residency increases, the test is less affected by memory bandwidth limitations, making the result more focused on reflecting the achievable throughput of the texture sampling path under a specific configuration.

•   FP32 floating-point computation throughput: register residency, reducing memory access impact

When measuring the ALU computing throughput of shader units, interference is also faced. Factors such as memory access, instruction dependency, and operating frequency may all affect the results.

To make the test reflect more of the achievable throughput of the floating-point execution path, we adopt the following methods:

1)     By keeping the main operands resident in registers, reduce the impact of off-chip memory access latency on floating-point instruction execution.

2)     Use highly compact FMA (fused multiply-add) instructions. According to the calculation basis in this article, one FMA contains one multiplication and one addition and is counted as two floating-point operations.

3)     To prevent FMA instructions from being optimized away by the compiler, use two FMA instructions, a = a * b + b, b = b * a + a. Repeatedly execute these two FMA operations to increase the computational load, and verify through checking the actual compiled instructions that the target operations have not been eliminated or replaced.

4)     The test controls the register usage of each thread and configures a sufficient number of parallel threads to increase the scale of active threads in the execution units and reduce the impact of single dependency instruction latency on overall throughput. The actual concurrency, register usage, and instruction issue situation still need to be confirmed in combination with compilation results and performance metrics.

The formula for calculating the measured floating-point computation throughput is:

Floating-point operations per second =

Total number of threads * Number of effective FMA instructions executed per thread * 2 / Kernel execution time

The microbenchmarks in this article are used to observe the achievable throughput of a single GPU processing path under a specific configuration. The test results are not equivalent to real application performance, nor can they alone prove that a certain hardware unit has reached its absolute physical upper limit. The comparison between specification values and measured values needs to be established on the basis of consistency in metric definition, data format, operating frequency, operation counting, and timing basis.

Test Results and Discussion After Controlling Variables

Based on the above design ideas, we developed the RenderScope GPU performance testing tool and conducted tests on multiple graphics cards. The results are as follows.

•   Pixel Fill Rate test results

Table 2: Comparison of measured and specification Pixel Fill Rate values for multiple GPUs under a specific test configuration

It can be seen that after aligning some test conditions and reducing the impact of several non-target factors, the gap between the measured results and the specification values has narrowed. For some graphics cards, Pixel Fill Rate can reach 98.54% of the specification value.

•   Texel Fill Rate test results

Table 3: Comparison of measured and specification Texel Fill Rate values for multiple GPUs under a specific test configuration

In the current test, some graphics cards can reach 78.89% of the specification value in Texel Fill Rate. Although the test has excluded many interference factors, there is still a gap between Texel Fill Rate and the specification value. There may still be other interference factors involved here.

•   FP32 floating-point computation throughput test results

Table 4: Comparison of measured and specification FP32 FLOPS values for multiple GPUs under a specific test configuration

In the current test, the FP32 FLOPS test results are very close to the specification values, but for some graphics cards the floating-point test results exceed the official specification values, which still requires further research and analysis.

Overall, after adjusting the test conditions, some measured results are closer to the specification values, indicating that factors such as format, pipeline state, data residency, and instruction organization do indeed affect the comparison between specification values and measured values. However, the results of different tests are not completely consistent, and the current data are not sufficient to attribute the source of the difference to a single cause. What this article hopes to illustrate is that before discussing whether a GPU has reached its specified capability, it is necessary to first confirm whether the specification value and the measured value use comparable metric definitions, test conditions, and calculation bases. Controlling variables can improve the interpretability of results, but it cannot replace continuous verification of the remaining interference factors and abnormal results.

Starting from Numbers, But Not Ending at Numbers

Lord Kelvin emphasized that measurement and expressing something in numbers is an important starting point for understanding an object. But this exploration has also made us more convinced that having a number does not mean performance has already been understood. Only when it is clearly explained what this number measures, under what conditions it holds, and what factors affect it, does it truly have explanatory value.

Establishing a credible measurement baseline is the first step in GPU performance analysis. Next, we still need to enter real workloads, observe the execution process, locate performance bottlenecks, verify causal relationships, and ultimately translate the analysis results into a basis for software optimization and architectural design.

A number will not tell the truth on its own, but rigorous measurement can make it begin to speak.