Jul 29, 2026 · research · edge-ai · gpu-profiling

Profiling Concurrent Vision Inference Workloads on NVIDIA Jetson

A deep-dive into edge AI resource sharing, GPU utilization, and performance bottlenecks.

The proliferation of IoT devices and advancements in network technologies have driven a massive demand for real-time data processing at the network edge. As applications like autonomous vehicles and smart cities scale up, relying entirely on cloud infrastructure introduces latency and bandwidth constraints. The solution? Edge computing via low-power AI accelerators, particularly edge GPUs like the NVIDIA Jetson lineup.

However, deploying concurrent vision workloads on edge devices isn't as simple as drag-and-drop. In our recent paper, Profiling Concurrent Vision Inference Workloads on NVIDIA Jetson — Extended, we conducted an in-depth analysis of resource sharing on edge GPUs to identify exactly where the bottlenecks hide.

Here is a comprehensive breakdown of our methodology and the key findings.

Methodology and Setup

To understand the intricate behavior of concurrent models, we evaluated two distinct devices from the NVIDIA Jetson family:

  • NVIDIA Jetson Orin Nano (Ampere Architecture)
  • NVIDIA Jetson Nano (Maxwell Architecture)

We profiled three distinct vision workloads representing different computer vision tasks:

  • ResNet50 (Image Classification)
  • YoloV8n (Object Detection)
  • FCN_ResNet50 (Image Segmentation)

Our profiling methodology was split into two phases to balance detailed metrics with runtime intrusion:

  1. Lightweight Profiling: Utilizing trtexec and jetson-stats to capture high-level SoC metrics (throughput, power consumption, and GPU memory) without heavily disrupting the inference loop.
  2. Deep-Dive Profiling: Utilizing NVIDIA Nsight Systems to track kernel-level metrics, CUDA API overheads, Streaming Multiprocessor (SM) utilization, and Tensor Core (TC) activity.

Key Takeaways and Results

Instead of relying solely on GPU utilization—which often paints a misleading picture of efficiency—we dug into the low-level metrics. Here is what we discovered:

Precision and Architecture Matter

  • Optimal Precision Varies by Device: For the cutting-edge Jetson Orin Nano, int8 precision models consistently outperformed others, delivering nearly 10x higher throughput for ResNet50. However, on the older Jetson Nano, fp16 models yielded the best throughput and memory consumption.
  • The Penalty of Unsupported Precision: The Jetson Nano lacks native hardware support for comprehensive int8 operations across all layers. When layers fall back to fp32, the resulting throughput drops and power consumption increases significantly.

The Illusion of 100% GPU Utilization

  • High SM Activity ≠ High Throughput: Our profiling showed that while SM active utilization often reached between 75% and 100% across workloads, SM issue slot utilization rarely exceeded 40% (and hovered around 25% for FCN_ResNet50). This indicates that while the Streaming Multiprocessors are "active", they are heavily stalled waiting for memory or data fetches.
  • Tensor Core Underutilization: Even on the Jetson Orin Nano, Tensor Core utilization rarely reached full capacity. For int8 models (which achieved the highest throughput), TC utilization remained surprisingly low (around 25% for ResNet50). This implies that throughput is constrained by memory bandwidth and non-TC instructions rather than raw computational power.

Scaling Concurrent Processes

  • Diminishing Returns: Increasing batch sizes improves the throughput-per-process (T/P). However, scaling up the number of concurrent processes steeply degrades T/P. For instance, running 8 concurrent processes of YoloV8n drastically drops individual throughput compared to a single process running a larger batch.
  • Unified Memory Limits: The unified RAM architecture of Jetson devices (where CPU and GPU share memory) creates a hard ceiling. Memory usage grows proportionally with both batch size and process count, often leading to system crashes if scaled recklessly.

CPU and Scheduling Bottlenecks

  • While the GPU is heavily scrutinized, CPU-side events such as thread scheduling and context switching frequently emerged as the ultimate bottleneck, particularly when managing multiple overlapping concurrent workloads.

Conclusion

Maximizing the efficiency of vision inference workloads on NVIDIA Jetson edge devices requires more than just pushing multiple models to the GPU. True optimization necessitates hardware-aware decision-making—matching the right weight precision to the target architecture, optimizing batch sizes to respect unified memory limits, and understanding that 100% GPU utilization often masks underlying instruction stalls.

For a more granular look at the data, the exact CDF graphs, and kernel-level trace analysis, I encourage you to read the full extended paper on arXiv.