# Democratizing Compute, Part 2: What exactly is “CUDA”? **URL:** \ **Category:** Community Showcase\ **Created:** [February 5, 2025, 7:41pm UTC](https://forum.modular.com/t/democratizing-compute-part-2-what-exactly-is-cuda/530 "2025-02-05T19:41:30Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![clattner](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/clattner/32/201_2.png) [@clattner](https://forum.modular.com/u/clattner)\ **Post date:** [February 5, 2025, 7:41pm UTC](https://forum.modular.com/t/democratizing-compute-part-2-what-exactly-is-cuda/530/1 "2025-02-05T19:41:30Z")
![CUDA](https://us1.discourse-cdn.com/flex001/uploads/modular/original/1X/21031b263d2ca6964ab8d22f024c4ec2f95633d8.jpeg) It seems like **everyone** has started talking about [CUDA](https://en.wikipedia.org/wiki/CUDA) in the last year: It’s the **backbone of deep learning,** the reason **novel hardware struggles to compete,** and the core of **NVIDIA’s moat** and **soaring market cap.** With DeepSeek, we got a startling revelation: its **breakthrough was made possible by “bypassing” CUDA** , [going directly to the PTX layer](https://www.tomshardware.com/tech-industry/artificial-intelligence/deepseeks-ai-breakthrough-bypasses-industry-standard-cuda-uses-assembly-like-ptx-programming-instead)… but what does this actually mean? It feels like everyone wants to break past the lock-in, but we have to **understand what we’re up against** before we can formulate a plan. This is Part 2 of Modular’s “ **Democratizing Compute** ” series, for more, see: - [Part 1: DeepSeek’s Impact on AI](https://forum.modular.com/blog/democratizing-compute-part-1-deepseeks-impact-on-ai) - Part 2: What exactly is “CUDA”? (this article) - Part 3: Why has CUDA been successful? (coming soon) CUDA’s dominance in AI is undeniable—but **most people don’t fully understand what CUDA actually is**. Some think it’s a programming language. Others call it a framework. Many assume it’s just **“that thing NVIDIA uses to make GPUs faster.”** None of these are entirely wrong—and many [brilliant people are trying to explain this](https://x.com/IanCutress/status/1884374138787357068)—but none capture the **full scope of “The CUDA Platform.”** CUDA is not just one thing. It’s a **huge, layered Platform** —a collection of technologies, software libraries, and low-level optimizations that together form a **massive parallel computing ecosystem**. It includes: - **A low-level parallel programming model** that allows developers to harness the raw power of GPUs with a C+±like syntax. - **A complex set of libraries and frameworks** —middleware that powers crucial vertical use cases like AI (e.g., cuDNN for PyTorch and TensorFlow). - **A suite of high-level solutions** like TensorRT-LLM and Triton, which enable AI workloads (e.g., LLM serving) without requiring deep CUDA expertise. …and that’s just scratching the surface. In this article, we’ll break down the **key layers of the CUDA Platform** , explore its **historical evolution** , and explain **why it’s so integral to AI computing today**. This sets the stage for the next part in our series, where we’ll dive into **why CUDA has been so successful.** Hint: it has a lot more to do with market incentives than it does the technology itself. Let’s dive in. 🚀 ## The Road to CUDA: From Graphics to General-Purpose Compute Before GPUs became the powerhouses of AI and scientific computing, they were **graphics processors—specialized processors for rendering images**. Early GPUs **hardwired** image rendering into silicon, meaning that every step of rendering (transformations, lighting, rasterization) was fixed. While efficient for graphics, these chips were **inflexible** —they couldn’t be repurposed for other types of computation. Everything changed in **2001** when NVIDIA introduced the **GeForce3** , the first GPU with **programmable shaders**. This was a **seismic shift** in computing: - 🎨 **Before:** Fixed-function GPUs could only apply pre-defined effects. - 🖥 **After:** Developers could **write their own shader programs** , unlocking **programmable graphics pipelines**. This advancement came with **Shader Model 1.0** , allowing developers to write **small, GPU-executed programs** for vertex and pixel processing. NVIDIA saw **where the future was heading:** instead of just improving graphics performance, GPUs could become **programmable parallel compute engines**. At the same time, it didn’t take long for researchers to ask: > “🤔 _If GPUs can run small programs for graphics, could we use them for non-graphics tasks?”_ One of the first serious attempts at this was the [**BrookGPU project**](http://graphics.stanford.edu/projects/brookgpu/) at Stanford. Brook introduced a programming model that let **CPUs offload compute tasks to the GPU** —a key idea that [set the stage for CUDA](https://www.nvidia.com/content/GTC/documents/1001_GTC09.pdf). This move was **strategic and transformative**. Instead of treating compute as a **side experiment** , NVIDIA **made it a first-class priority** , embedding CUDA deeply into **its hardware, software, and developer ecosystem**. ## The CUDA Parallel Programming Model In **2006** , NVIDIA launched **CUDA (”Compute Unified Device Architecture”)**—the first **general-purpose programming platform for GPUs**. The CUDA programming model is made up of two different things: the “CUDA programming language”, and the “NVIDIA Driver”. ![](https://us1.discourse-cdn.com/flex001/uploads/modular/original/1X/b21d05695bc303f390d592d22a40e060bb5df3a4.png) CUDA is a Layered Stack Requiring Deep Integration from Driver to Kernel The CUDA language is derived from C++, with enhancements to directly expose low-level features of the GPU—e.g. its ideas of “GPU threads” and memory. A programmer can use this language to define a “CUDA Kernel”—an independent calculation that runs on the GPU. A very simple example is: CUDA kernels allow programmers to define a custom computation that accesses local resources (like memory) and using the GPUs as very fast parallel compute units. This language is translated (”compiled”) down to “PTX”, which is an assembly language that is the lowest level supported interface to NVIDIA GPUs. But how does a program **actually execute code on a GPU?** That’s where the **NVIDIA Driver** comes in. It acts as the **bridge** between the CPU and the GPU, handling memory allocation, data transfers, and kernel execution. A simple example is: Note that all of this is very low level—full of fiddly details (e.g. pointers and “magic numbers”). If you get something wrong, you’re most often informed of this by a difficult to understand crash. Furthermore, CUDA exposes a lot of details that are specific to NVIDIA hardware—things like the “number of threads in a warp” (which we won’t explore here). Despite the challenges, these components enabled an entire generation of hardcore programmers to get access to the huge muscle that a GPU can apply to numeric problems. For example, the [**AlexNET**](https://en.wikipedia.org/wiki/AlexNet) **ignited modern deep learning in 2012**. It was made possible by custom CUDA kernels for AI operations like convolution, activations, pooling and normalization and the horsepower a GPU can provide. While the CUDA language and driver are **what most people typically think of** when they hear “CUDA,” this is far from the whole enchilada—it’s just the **filling inside**. Over time, **the CUDA Platform** grew to include much more, and as it did, the meaning of the original acronym fell away from being a useful way to describe CUDA. ## High-Level CUDA Libraries: Making GPU Programming More Accessible The CUDA programming model opened the door to **general-purpose GPU computing and is powerful** , but it brings two challenges: 1. CUDA is **difficult to use** , and even worse… 2. CUDA doesn’t help with **performance portability** Most kernels written for generation N will “keep working” on generation N+1, but often the performance is quite bad—far from the peak of what N+1 generation can deliver, even though GPUs are all about performance. This makes CUDA a **strong tool for expert engineers** , but a **steep learning curve for most developers.** But is also means that significant rewrites are required every time a new generation of GPU comes out (e.g. Blackwell is now emerging). As NVIDIA grew it wanted GPUs to be useful to people who were domain experts in their own problem spaces, but weren’t themselves GPU experts. NVIDIA’s solution to this problem was to start building rich and complicated **closed-source, high-level libraries** that abstract away low-level CUDA details. These include: - **cuDNN** (2014) – Accelerates deep learning (e.g., convolutions, activation functions). - **cuBLAS** – Optimized linear algebra routines. - **cuFFT** – Fast Fourier Transforms (FFT) on GPUs. - … and [many others](https://developer.nvidia.com/gpu-accelerated-libraries). With these libraries, developers could **tap into CUDA’s power without needing to write custom GPU code** , with NVIDIA taking on the burden of **rewriting these for every generation of hardware**. This was a big investment from NVIDIA, **but it worked**. The **cuDNN library** is especially important in this story—it paved the way for Google’s **TensorFlow** (2015) and Meta’s **PyTorch** (2016), enabling deep learning frameworks to take off. While there were earlier AI frameworks, these were the first frameworks to truly scale—modern AI frameworks now have _ **thousands** _ of these CUDA kernels and each is very difficult to write. As AI research exploded, NVIDIA aggressively pushed to expand these libraries to cover the important new use-cases. ![Image depicting a layered stack with AI Model Developers at the top, represented by a laptop icon with a sparkle. Below is a cloud labeled PyTorch Ecosystem, resting above a red block labeled PyTorch. Underneath are three more layers: a green block for CUDA Libraries, another green block for CUDA Language, and a blue block at the bottom labeled NVIDIA Driver. The structure highlights the deep dependency chain required to support PyTorch within the CUDA framework.](https://us1.discourse-cdn.com/flex001/uploads/modular/original/1X/977781a9b31f348afc9abf166dc663dbca516f44.png) PyTorch on CUDA is Builds on Multiple Layers of Dependencies NVIDIA’s investment into these **powerful GPU libraries** enabled the world to focus on building high-level AI frameworks like PyTorch and developer ecosystems like HuggingFace. Their next step was to make entire **solutions** that could be used **out of the box** —without needing to understand the CUDA programming model at all. ## Fully vertical solutions to ease the rapid growth of AI and GenAI The AI boom went far beyond research labs— **AI is now everywhere**. From **image generation** to **chatbots** , from **scientific discovery** to **code assistants** , **Generative AI (GenAI) has exploded across industries**, bringing a flood of new applications and developers into the field. At the same time, **a new wave of AI developers emerged, with very different needs.** In the early days, deep learning required **highly specialized engineers** who understood CUDA, HPC, and low-level GPU programming. Now, a new breed of developer—often called **AI engineers** —is building and deploying AI models without needing to touch low-level GPU code. To meet this demand, NVIDIA went beyond just providing libraries—it now offers **turnkey solutions** that abstract away **everything** under the hood. Instead of requiring **deep CUDA expertise** , these frameworks allow AI developers to **optimize and deploy models with minimal effort**. - **Triton Serving** – A high-performance serving system for AI models, allowing teams to efficiently run inference across multiple GPUs and CPUs. - **TensorRT** – A deep learning inference optimizer that **automatically tunes models** to run efficiently on NVIDIA hardware. - **TensorRT-LLM** – An even more specialized solution, built for **large language model (LLM) inference at scale**. - … plus many (many) other things. ![Image showing a vertical stack with AI Engineers at the top, represented by a laptop icon with a sparkle. Below are four layers: a green block labeled TensorRT-LLM, followed by CUDA Libraries, then CUDA Language, and finally a blue block at the bottom labeled NVIDIA Driver. The layered structure highlights the multiple dependencies required for AI development within the CUDA ecosystem.](https://us1.discourse-cdn.com/flex001/uploads/modular/original/1X/61e28a8c10fffd9d7bd5f17e915f5aa4afd1985e.png) Multiple Layers Exist Between NVIDIA Drivers to TensorRT-LLM These tools **completely shield AI engineers from CUDA’s low-level complexity** , letting them **focus on AI models and applications, not hardware details**. These systems provide significant leverage which has enabled the horizontal scale of AI applications. ## The “CUDA Platform” as a whole CUDA is often thought of as a **programming model** , a **set of libraries** , or even just **“that thing NVIDIA GPUs run AI on.”** But in reality, **CUDA is much more than that** —it is a **unifying brand, a truly vast collection of software, and a highly tuned ecosystem** , all deeply integrated with NVIDIA’s hardware. For this reason, the term “CUDA” is ambiguous—we prefer the term “The CUDA Platform” to clarify that we’re talking about something closer in spirit to the Java ecosystem, or even an operating system, than merely a programming language and runtime library. ![Image showing a layered stack of the CUDA ecosystem. At the top are icons for AI GPU Kernel Developers, AI Model Developers, and AI Engineers, with clouds for CUDA Kernels and PyTorch Ecosystem. Below are PyTorch, TensorRT-LLM, CUDA Libraries, CUDA Language, and the foundational NVIDIA Driver, highlighting CUDA’s complex dependencies.](https://us1.discourse-cdn.com/flex001/uploads/modular/original/1X/74de6471276b0521fd5c6348b73b9b096580c383.png) CUDA’s Expanding Complexity: A Multi-Layered Ecosystem Spanning Drivers, Languages, Libraries, and Frameworks At its core, the CUDA Platform consists of: - **A massive codebase** – Decades of optimized GPU software, spanning everything from matrix operations to AI inference. - **A vast ecosystem of tools & libraries** – From **cuDNN for deep learning** to **TensorRT for inference** , CUDA covers an **enormous range of workloads**. - **Hardware-tuned performance** – Every CUDA release is deeply optimized for **NVIDIA’s latest GPU architectures** , ensuring top-tier efficiency. - **Proprietary and opaque** – When developers interact with CUDA’s **library APIs** , much of what happens under the hood is **closed-source and deeply tied to NVIDIA’s ecosystem**. CUDA is powerful but sprawling set of technologies— **an entire software platform that sits at the foundation of modern GPU computing** , even going beyond AI specifically. Now that we know what “CUDA” is, we need to understand how it got to be so successful. Here’s a hint: CUDA’s success isn’t really about **performance** —it’s about **strategy, ecosystem, and momentum**. In the next post, we’ll explore what enabled NVIDIA’s CUDA software to shape and entrench the modern AI era. See you next time. 🚀 -Chris ### What’s next? Learn more about the MAX Platform and the Mojo programming language, and join us in building the next wave of AI innovation. - [**Get started with MAX**](https://docs.modular.com/max/get-started)–explore our latest platform to accelerate your AI work. - Check out the the [**Mojo manual**](https://docs.modular.com/mojo/manual/) and get started with our next-generation language for programming GPUs. - Continue the conversation in [**the Modular forum**](https://forum.modular.com/). ---
**Author:** ![melodyogonna](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/melodyogonna/32/33_2.png) [@melodyogonna](https://forum.modular.com/u/melodyogonna)\ **Post date:** [February 5, 2025, 8:18pm UTC](https://forum.modular.com/t/democratizing-compute-part-2-what-exactly-is-cuda/530/5 "2025-02-05T20:18:44Z")
Aha! This is why it has been difficult to understand exactly what CUDA is, it has gone from being just a framework to representing an ecosystem. Maybe they need to start saying “CUDA is not acronym” on their website - similar to LLVM, to indicate that it (CUDA) isn’t just one software. ---
**Author:** ![rcmpge](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/rcmpge/32/849_2.png) [@rcmpge](https://forum.modular.com/u/rcmpge)\ **Post date:** [February 5, 2025, 11:50pm UTC](https://forum.modular.com/t/democratizing-compute-part-2-what-exactly-is-cuda/530/6 "2025-02-05T23:50:20Z")
I think the development of Mojo’s ecosystem is much needed in this space. I only see NVIDIA accelerating their development processes. I would much welcome the CUDA community to Mojo to bridge gaps and pain points in this space. ---
**Author:** ![owenhilyard](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/owenhilyard/32/41_2.png) [@owenhilyard](https://forum.modular.com/u/owenhilyard)\ **Post date:** [February 6, 2025, 4:28pm UTC](https://forum.modular.com/t/democratizing-compute-part-2-what-exactly-is-cuda/530/7 "2025-02-06T16:28:13Z")
I think it’s also important to mention [Cuda-X](https://www.nvidia.com/en-us/technologies/cuda-x/) in this. Not only does this include Tensor-RT, but also pre-packaged toolkits for a lot of areas. There’s cuDF, which is essentially designed to be a GPU accelerated pandas backend. cuOpt does linear programming (the algorithmic kind). GPUDirect RDMA, Ethernet and Storage help with moving data around immensly. And there’s a gigantic group of domain specific libraries for HPC. A lot of it also goes beyond GPUs. Since the Mellanox acquisition, Nvidia has put a lot of work into integrating networking, since considering the network helps a lot of distributed computations. I think MAX can do all of that, but it’s going to be a long, long road. ---
**Author:** ![nlaanait](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/nlaanait/32/56_2.png) [@nlaanait](https://forum.modular.com/u/nlaanait)\ **Post date:** [February 7, 2025, 2:32am UTC](https://forum.modular.com/t/democratizing-compute-part-2-what-exactly-is-cuda/530/8 "2025-02-07T02:32:17Z")
Good point @owenhilyard ! `CUDA-X` is a massive ecosystem for pretty much any domain one can think of! If MAX+MOJO can achieve true code portability and performance across heterogeneous compute (something that’s been attempted in the past e.g. `openacc`), esp. in a critical and commercially-relevant area like GenAI inference, it would be a great first step towards unlocking all other domains from the vendor lock-in of “CUDA”. ---
**Author:** ![shancun](https://avatars.discourse-cdn.com/v4/letter/s/df705f/32.png) [@shancun](https://forum.modular.com/u/shancun)\ **Post date:** [February 7, 2025, 3:12am UTC](https://forum.modular.com/t/democratizing-compute-part-2-what-exactly-is-cuda/530/9 "2025-02-07T03:12:47Z")
CUDA is, after all, a platform driven by a profit-making company to quickly patch hardware to increase computing speed. It is essentially an efficient parallel computing platform. At the beginning, there was no pair of peripheral libraries, and there was no simple language to control and customize it, so the company provided a bunch of peripherals to reduce the difficulty. If you think about this problem from the bottom up and solve the programming complexity of parallel computing fundamentally, it will promote the rapid development of this industry. Because of the rapid development, the language to manipulate the parallel computing must be simple enough, and enough talents will participate in the development of the technology. ---
**Author:** ![rd4com](https://avatars.discourse-cdn.com/v4/letter/r/cdc98d/32.png) [@rd4com](https://forum.modular.com/u/rd4com)\ **Post date:** [February 7, 2025, 5:45pm UTC](https://forum.modular.com/t/democratizing-compute-part-2-what-exactly-is-cuda/530/10 "2025-02-07T17:45:21Z")
Great 🥳 , thanks ! 👍 ---
**Author:** ![owenhilyard](https://sea1.discourse-cdn.com/flex001/user_avatar/forum.modular.com/owenhilyard/32/41_2.png) [@owenhilyard](https://forum.modular.com/u/owenhilyard)\ **Post date:** [February 7, 2025, 8:45pm UTC](https://forum.modular.com/t/democratizing-compute-part-2-what-exactly-is-cuda/530/11 "2025-02-07T20:45:12Z")
I definitely agree. There is space for a compiler which can “see the whole problem”, and not just for AI training and HPC. As a distributed systems person, I’m convinced that a compiler which helps manage the complexity of a distributed system would be very helpful for a lot of developers. ---
**Author:** ![system](https://us1.discourse-cdn.com/flex001/uploads/modular/original/1X/2751e0fbdc595a99718b216730957e9db4448cfd.jpeg) [@system](https://forum.modular.com/u/system)\ **Post date:** [August 6, 2025, 8:45pm UTC](https://forum.modular.com/t/democratizing-compute-part-2-what-exactly-is-cuda/530/12 "2025-08-06T20:45:53Z")
Closed on August 6, 2025, 8:45pm UTC This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.