Talk

Hands-on With CUDA-C on a Nvidia Jetson Nano GPU

Mohammed Billoo

Watch Now

42:14

EOC 2024

Login to Watch

Mohammed Billoo

Mohammed Billoo is Founder of MAB Labs, LLC. He has over 12 years of experience architecting, designing, implementing, and testing embedded software, with a core focus on embedded Linux. This includes custom board bring-up, writing custom device drivers, and writing application code. Mohammed also contributes to the Linux kernel and is an active participant in numerous open-source efforts. He is also an Adjunct Professor of Electrical Engineering at The Cooper Union for the Advancement of Science and Art, where he teaches courses in Digital Logic Design, Computer Architecture, and Advanced Computer Architecture. Mohammed received both his Bachelor's and Master's of Electrical Enginering from the same institution.

View full profile

Talk

Hands-on With CUDA-C on a Nvidia Jetson Nano GPU

Mohammed Billoo

42:14 EOC 2024

Mohammed Billoo

42:14

The explosion of silicon density in the past decade has resulted in increased massive compute capabilities of modern-day microprocessors in power-efficient packages. Processors supporting trillions of floating point operations (TFLOPS) are now becoming widely available in smaller form factors. Software paradigms have also been created to leverage the underlying hardware. In this talk, Mohammed Billoo will introduce Nvidia's CUDA framework, targeting a variant of the C programming language. Care needs to be taken when implementing applications using CUDA-C. The underlying hardware, specifically the memory architecture and layout, need to be considered to ensure that the final implementation is performant. Mohammed will cover the following topics in this talk:

Architectural differences between GPUs and CPUs
Relevant applications for GPUs
CUDA-C overview and build process
CUDA-C code structure

Mohammed will conclude the talk with a hands-on demonstration of an image processing algorithm implemented in CUDA-C on a Nvidia Jetson Nano development kit. He will compare the performance of the CUDA-C implementation on the GPU against a naive implementation on a CPU.

1 / 4

Please log in or create an account to test your knowledge and see the answers.

According to Mohammed Billoo, what is the primary reason GPUs deliver much higher floating‑point throughput than CPUs for suitable workloads?

A GPUs are optimized to perform the same operation on many data points simultaneously, prioritizing aggregate throughput over single‑thread latency.

B GPUs devote most silicon to control logic and branch prediction to reduce single‑thread latency.

C GPUs are designed primarily for integer‑heavy, sequential control tasks on a single core.

D GPUs include massive per‑thread caches to achieve the same low latency as CPUs.

E GPUs prioritize minimizing memory accesses by executing a single thread extremely fast.

When evaluating whether a GPU will meaningfully speed up your application, what key guideline from the talk should you use?

A The overall application speedup is limited by the fraction of the application that can be parallelized (Amdahl's law).

B Any small parallel portion accelerated on the GPU will always produce a near‑linear overall speedup regardless of the serial part.

C GPU speedup depends only on the GPU's TFLOPS rating and is independent of how much serial code exists.

D If you accelerate 30% of the code by 100x, the overall speedup will be about 30x.

E As long as your dataset is large, serial parts of the code can be ignored when estimating speedup.

What is the correct high‑level sequence of steps Mohammed describes for implementing a simple CUDA‑C kernel (vector addition) on the GPU?

A Allocate device memory with cudaMalloc, copy host data to device with cudaMemcpy (HostToDevice), launch the kernel with chosen grid/block sizes, copy results back with cudaMemcpy (DeviceToHost), then free device memory.

B Use malloc() for device memory and let nvcc automatically move host arrays to the GPU before kernel launch.

C Call the kernel directly on host pointers; CUDA automatically maps host buffers into device memory without explicit copies.

D Only call the kernel; cudaMalloc and cudaMemcpy are optional because the runtime auto‑transfers everything.

E Use cudaMallocHost for device allocations and standard memcpy() for transfers between CPU and GPU.

Why did Mohammed find that the GPU implementation of a 1D vector addition performed worse than the naive CPU implementation in his demo?

A Because the cost of allocating and transferring memory to/from the GPU dominates for small or 1D workloads, making the GPU slower than the CPU for that case.

B Because GPUs lack floating‑point units and cannot perform simple arithmetic efficiently.

C Because CUDA forces rewriting algorithms into assembly for simple tasks, which adds extra overhead.

D Because GPU threads are very expensive to create and schedule, so thread startup overhead kills performance.

E Because the nvcc JIT compilation for device code always dominates execution time for basic array operations.

Formatting help

italics	surround text with asterisks
bold	surround text with two asterisks
hyperlink	[hyperlink](https://example.com) or just a bare URL
inline code	surround text with single `backticks`
code block (multi-line)	wrap on its own lines with three backticks: ``` your code here ```
~~strikethrough~~	surround text with ~~two tilde characters~~
quote	prefix with >

Upvotes Newest Oldest

RaulPando

Score: 0 | 2 years ago | 1 reply

Thanks for a very informative presentation, as always. I particularly liked the "gotchas" one should keep in mind to leverage efficient parallelisation.
In your opinion, would you consider C++'s intrinsic nature of vector manipulation to generally more fitting than C when interfacing with CUDA or maybe when passing back the results to the host?
Do you know how available CUDA when it comes to integration with the Yocto project?
Looking forward to the follow-up article :)

mabSpeaker

Score: 0 | 2 years ago | no reply

Hi,
Great questions! Ultimately, vectors in the C++ STL are essentially C arrays under the hood with some extra features (e.g. automatically re-allocating memory as needed, optimally accessing elements). I would actually caution in using C++ vectors initially, since it's important to ensure that memory is aligned properly when passing data to the GPU. Otherwise, you risk losing the benefits of a GPU. CUDA supports exist in The Yocto Project. You can check out the meta-tegra layer here: https://github.com/OE4T/meta-tegra. I'll look to author another blog post on adding CUDA support, along with cross-compiling CUDA-C applications when using The Yocto Project.

Partners & Sponsors

STMicroelectronics

Memfault

NXP

Micro Digital, Inc.

MySQL

Become a Sponsor