Optimizing a CUDA Image-Processing Pipeline: From Bugs to Benchmarks

  • PublicPublic
  • 02 Sep, 2026
Optimizing a CUDA Image-Processing Pipeline: From Bugs to Benchmarks

This practical CUDA example processes a stream of red, green, and blue images: data is first copied from the CPU to the GPU, converted from RGB to grayscale, then each 32-by-32-pixel tile is sorted so its median can be computed, and finally the medians are copied back to the host. The starting implementation defines two kernels: computeRGBToGray for the color conversion and computeMedian<TILE_WIDTH, HISTO_SIZE> for the per-tile median, while main `main` allocates memory, launches the kernels, and frees resources inside an OpenMP parallel loop over three images.

The median kernel has a subtle but serious flaw. Its shared-memory store tile[index] = d_image_gray[index] incorrectly uses a global index to write into shared memory, which is scoped to a single thread block. Running the binary produces an "illegal memory access" error, and compute-sanitizer pinpoints the exact cause: an out-of-bounds __shared__ write of one byte at line 55, reported for thread (0,3,0) in block (20,0,0).

Figure 1. The first part of the image-processing pipeline. First, copy the RGB images from the host to the device, then convert them from RGB to grayscale
Figure 1. The first part of the image-processing pipeline. First, copy the RGB images from the host to the device, then convert them from RGB to grayscale

To prevent such indexing mistakes, NVIDIA introduced a new launch API in CCCL. Kernels are launched with cuda::make_config, cuda::block_dims, cuda::grid_dims and cuda::launch, then the kernel accepts the configuration as its first parameter and uses cuda::gpu_thread.index(cuda::grid, config) and cuda::gpu_thread.index(cuda::block, config), to retrieve global and block indices separately. Raw pointers can also be replaced with cuda::std::span, cuda::std::mdspan and cuda::shared_memory_mdspan, which trigger assertions in debug mode when an out-of-bounds access occurs.

Fig. 2. The second part of the image-processing pipeline: split each grayscale image into 32x32-pixel tiles, sort the pixels in each tile to select its median, then copy the median values from d
Fig. 2. The second part of the image-processing pipeline: split each grayscale image into 32x32-pixel tiles, sort the pixels in each tile to select its median, then copy the median values from d

Once the code is bug-free, it is benchmarked with Nsight Systems. Instrumenting the code with NVTX ranges, such as nvtx3::scoped_range and nvtxRangePushA/nvtxRangePop - makes the timeline easier to read and shows that the median-computing kernel dominates the profile: it takes about 2.1 seconds per image and accounts for 98.5% of GPU time, while memory operations account for only 1.5%.

Figure 3. The initial Nsight Systems timeline with NVTX annotations. The median-computing kernels dominate the profile, making up almost all GPU activity and most of the 6.8-second image processing
Figure 3. The initial Nsight Systems timeline with NVTX annotations. The median-computing kernels dominate the profile, making up almost all GPU activity and most of the 6.8-second image processing

From the CPU side, the whole image-computation phase takes 6.8 seconds, with nearly all of that time going to the median kernels. This timeline gives developers a clear target for the next optimization steps: the median computation is by far the most expensive part of the pipeline, and the subsequent stages of the walkthrough focus on improving it.