[{"authors":["Hyunjun Park"],"category":"Compilers \u0026 Kernels","content":" Overview Hello, this is Hyunjun Park from the CL (Compute Library) team at HyperAccel. In Part 1, we looked at how torch.compile() works. In Part 2, we explored why it is split into three stages and who makes use of that structure. Those two posts laid the groundwork for this one. Here, I will share the research and development trends I saw at PyTorch Conference 2026, held October 20–21 in San Jose.\nAfter the conference, I asked an agent about its main research themes. It named training, inference, PyTorch core modules, and kernel optimization. I mainly attended inference, compiler, and accelerator sessions. This post is therefore my interpretation of the trends in the sessions I chose, rather than a summary of the entire conference. Where I did not attend a session, I use its public description and mark that explicitly.\nI grouped the recurring themes into three keywords: Agent, Acceleration, and Ecosystem. The post follows those three themes.\nAs agents speed up development, how are projects changing their Continuous Integration (CI) and code structure? What are PyTorch and vLLM making faster? What tools are Neural Processing Unit (NPU) companies building to join the PyTorch ecosystem? Part 1. Agent: keeping up with faster development The bottleneck moves to review In the vLLM Roadmap session I attended, project management stood out as much as the new features. With commits and contributors growing rapidly, the project has been changing its release and maintenance processes. Comparing the pull requests (PRs) created on GitHub each October shows the scale of that growth.\nRepository Oct 2024 Oct 2025 Oct 2026 vllm-project/vllm 453 1,323 4,577 pytorch/pytorch 1,711 1,900 2,887 Based on GitHub search; includes PRs opened by bots. Retrieved November 4, 2026.\nThe number of vLLM PRs grew roughly tenfold in two years. These figures also reflect the project’s own growth, so the increase cannot be attributed entirely to agents. Still, the burden of reviewing and maintaining a growing volume of contributions was increasing. The description of PyTorch’s Fighting Agents with Agents session likewise opens with the growing number of agent-written PRs that maintainers need to review.\nAs writing code becomes cheaper, reading, validating, and maintaining it becomes more important. Projects have started by clarifying the rules for contributors. The vLLM contributing guide says not to submit \u0026ldquo;pure agent\u0026rdquo; PRs. The person submitting a change is responsible for reviewing every changed line and validating its behavior end to end.\nCI: fighting agents with agents Alongside these rules, CI and review processes are changing too. I did not attend the sessions in this section, so this account is based on their public descriptions.\nFighting Agents with Agents: Bringing Claude to PyTorch CI, triage, and PR review: PyTorch added automatic issue triage, PR review skills, and CI and autorevert investigation to its infrastructure. The stated goal is to support maintainers, not replace them. Shipping PyTorch and Its Ecosystem: A Modern Release Story: Triton and vLLM are continuously validated against PyTorch nightlies, so breakage surfaces before the release branch is cut. Agents triage CI failures and draft fixes, but people make the final call. What stood out in both sessions was the division of responsibilities between people and agents. Agents handle repetitive work such as triage, investigation, and drafting fixes, while people make the final decisions. vLLM also manages its limited CI resources by not starting CI automatically on every new commit. Reviewers, or the author of an approved PR, trigger it with /ci run.\nRefactoring: a structure more people can contribute to Code structure becomes more important as the contributor base grows. When many people change the same areas, review gets harder and changes are more likely to conflict. vLLM presented refactoring that separates shared behavior from the parts that vary by backend.\nA Developer\u0026rsquo;s Guide to Attention in vLLM is a good example. Each kind of attention has different metadata and a different key-value (KV) cache layout, so branches kept growing in the backend and connector code. vLLM now builds common metadata (CommonAttentionMetadata) once per step, and each backend adds only what it needs. The logical dimension order of the KV cache is standardized, and the actual memory layout is expressed through strides. Separating logical shape from physical layout means fewer places to change when adding a new attention type or storage scheme.\nFlat Model and Model Runner V2 from the roadmap are model-execution reworks in the same direction. Going by its public description, One Model Definition, Many Accelerators separates model logic from hardware execution paths, so the same definition runs on Intel Gaudi and IBM Spyre without platform-specific code.\nThe talks mainly explained this work as making new models and architectures easier to support. Refactoring helps maintenance regardless of the rise of agents. My own thought was that the falling cost of changing code might also play a role. Structural improvements that were recognized as necessary but postponed because of their cost may now be easier to attempt with help from agents. This is my interpretation after hearing several sessions, rather than an explanation given by the speakers.\nPart 2. Acceleration: what PyTorch and vLLM are speeding up If Part 1 was about development speed, Part 2 is about the time spent from compilation through serving. The sessions I attended emphasized not only the speed of generated code, but also reducing compilation costs and waiting time at runtime.\nFaster compilation: FakeTensor and graph breaks Let\u0026rsquo;s start with Speeding Up torch.compile: A New FakeTensor. FakeTensor is a tensor with no real data that computes only metadata such as shape, dtype, and device. Dynamo does this for every op while tracing, and Inductor uses it during code generation too. According to the session description, this propagation accounts for about 20% of Dynamo tracing time.\nThe approach is to reimplement FakeTensor in C++. Reducing the overhead of Python dispatch and symbolic operations brought the geometric mean execution time across 118 operations down from 48.2 µs to 2.2 µs. The reported speedup was 22.3x. These are per-operation measurements, however. The end-to-end torchbench compilation metric improved by 3.7–11.6% depending on the configuration, with individual models improving by as much as 20–30%.\nI was impressed by how much improvement came from changing the implementation path. The future work of porting the remaining Python meta kernels to C++ also came with a plan on the slide: \u0026ldquo;Use Agents!\u0026rdquo; The use of agents we saw in Part 1 appeared here as well.\nGraph-break handling is improving too. In Part 1 of this series, I explained that graph breaks shorten graphs and limit opportunities for fusion. Deeply nested function calls also create duplicate work. According to the Nested Graph Breaks session description, reducing O(N) duplicate graph breaks to O(1) in calls nested N levels deep allows larger graphs to be captured.\nLightweight FX Tracing in PyTorch starts from feedback that Dynamo is too heavy for users who need a full graph. It proposes a lightweight tracing path based on make_fx for suitable code, such as tensor computations without Python side effects.\nFaster execution: agent workloads and vLLM Agents are changing inference workloads as well as development. More requests maintain long contexts across multiple turns of conversation and tool calls. While a request waits for a tool response, its token generation pauses, but the KV state needed for subsequent generation must remain available or recoverable. vLLM presented optimizations for these workloads.\nWaiting less: vLLM Roadmap covered removing the CPU synchronization that stalls the GPU, along with kernel fusion and mega-kernels. Checking several tokens at once: What\u0026rsquo;s New in Inference in October and the enterprise inference session covered speculative decoding as a way to improve responsiveness. Keeping KV longer and cheaper: Native Tiered KV Cache Offloading in vLLM uses CPU memory as a hub. It frees GPU memory first and handles transfers to storage or other nodes asynchronously. These problems cannot be solved by torch.compile() alone. KV offloading, for example, does not make the decode kernel itself faster. It increases cache capacity and reuse at the cost of data transfers, so its effect depends on the workload and memory usage. The TPU mega-kernel demo also explained the need for prefill/decode disaggregation and cache reuse alongside kernel optimization. Across several sessions, I heard the same message: look beyond the speed of an individual kernel to the serving system as a whole.\nPart 3. Ecosystem: building hardware support on a shared foundation Part 2 of this series looked at where each accelerator connects to torch.compile(). This section is about what comes next. What does it take to achieve performance on new hardware and preserve compatibility as PyTorch evolves? Let’s look at Huawei and IBM Spyre in turn.\nHuawei: integrating through common interfaces and validating compatibility Huawei’s Boosting PyTorch’s Heterogeneous Compute Ecosystem began with the difficulty of integrating NPUs into GPU-centric code. It framed the work as Integrate, Perform, Verify. Hardware support needs to go beyond executing operators to include distributed execution, compilation, and testing.\nIntegration starts with PrivateUse1, which we covered in Part 2. The OpenReg reference implementation shows how to connect a device, while the common device API torch.accelerator helps make CUDA-dependent code device-independent. Huawei described connecting FSDP2, DTensor, and communication backends to support distributed training on this foundation.\nOnce integrated, a backend needs to remain compatible as PyTorch changes. This is where the CI from Part 1 returns. The Cross-Repository CI Relay (CRCR) forwards PyTorch PR events to external repositories, allowing compatibility with out-of-tree backends to be validated before merge. According to the public session description, Huawei combines CRCR with device-agnostic test reuse. It reports using more than 580,000 community tests and delivering stable releases within 30 days of upstream updates.\nThe talk ended with a call for contributors to help build the NPU ecosystem. What impressed me was the intention to use PyTorch’s interfaces while contributing to integration and validation infrastructure that other devices could share.\nIBM Spyre: reusing Inductor and extending it for the hardware While Huawei illustrated the foundations for integration and validation, IBM Spyre showed how to build performance using a shared compiler. The Integrating the IBM Spyre Accelerator talk and IBM’s official post provided a concrete view of the Inductor extension approach introduced in Part 2.\nSpyre is a dataflow accelerator with 32 cores, each with its own 2 MB scratchpad. Data moves in 128-byte units called sticks, and work must be divided among cores at compile time. This makes some of Inductor’s GPU-oriented assumptions difficult to apply directly: tensor layout, scratchpad management, and work assignment across cores. torch-spyre handles these with custom passes before scheduling, while reusing much of the rest of Inductor.\nBy reusing the shared compiler, the roughly 273,000 lines in the previous custom compiler stack became roughly 38,600 lines. About one-seventh refers to code size, not execution speed. This reduced the code the team needed to maintain directly and let it concentrate on hardware-specific work. The proposals for Kernel Tile IR (KTIR) and device-aware tensor layouts also show an effort to bring hardware-required extensions into the shared foundation.\nSpyre also uses CRCR. Its approach to selecting tests is particularly interesting. According to the public post, an agentic pipeline selected the tests relevant to its hardware from PyTorch’s tens of thousands of tests. The reasons for each selection are recorded in configuration files for people to review, and the classifications are refined using execution logs from real hardware. When moving from PyTorch 2.13 to 2.14, only about 4,000 changed tests needed to be evaluated.\nCommon ground: reuse, extend, and validate together The two examples emphasized different tasks, but their direction was the same: reuse common interfaces and compilers, extend what the hardware needs, and keep validating as upstream evolves. Both also showed an interest in sharing extensions so other devices could benefit.\nThis is where the three keywords connect. A shared Ecosystem supports hardware Acceleration, while Agents help with the validation needed to keep up with that ecosystem. The division of responsibilities from Part 1—agents making proposals and people exercising judgment—also appears in accelerator support.\nConclusion At this conference, Agent highlighted the importance of validation and maintenance alongside development speed. Acceleration extended beyond generated-code performance to compilation costs and runtime waiting time. Ecosystem showed accelerator vendors using a shared foundation and seeking to share the extensions they developed to support their hardware.\nAs a developer at a chip company, I took away two things. First, before building a feature ourselves, we should check whether it already exists or is under development in PyTorch or vLLM. As Spyre demonstrated, reusing a shared foundation can reduce the code we need to maintain directly. I came away thinking that we should actively evaluate open-source modules we could use.\nSecond, hardware-specific code should have clear boundaries with shared code. As agents make structural improvements easier to attempt, it is worth revisiting modularization work that has been put off. These two days left me thinking about how our software stack could fit more naturally with open-source projects.\nReference Official conference material\nPyTorch Conference North America 2026 recap Core PyTorch Sessions vLLM Sessions Agentic AI and Next-Gen Intelligence Sessions Hardware Acceleration \u0026amp; Compute Infrastructure Guide Docs and articles\nvLLM Contributing Guide Extending Inductor for the IBM Spyre Accelerator From Upstream Changes to Downstream Confidence P.S. HyperAccel is hiring! HyperAccel builds a new kind of chip, the LPU, and we are also a team that develops right alongside the latest in software. As this post shows, the software stack around new hardware is still changing fast, and we work at its leading edge.\nIf you\u0026rsquo;d like to join the research and development that lets users work with new devices the way they already work, apply at HyperAccel Career!\n","date":"2026.11.06","permalink":"https://hyperaccel.ai/techblog/en/posts/torch-compile-anatomy-3/","series":["torch.compile() 알아보기"],"style":"--c:#5BC5F6;--on:#02121c","summary":"Where the industry is heading, as seen at PyTorch Conference 2026, through three keywords: Agent, Acceleration, Ecosystem. We look at how agents are changing development, what PyTorch and vLLM are speeding up, and how NPU companies are joining the ecosystem.","title":"[Exploring torch.compile() - Part 03] Three Keywords from PyTorch Conference 2026: Agent, Acceleration, Ecosystem"},{"authors":["Sinhyun Park"],"category":"Memory \u0026 Storage","content":"Introduction Hi, I\u0026rsquo;m Sinhyun Park from the LLMOps team at HyperAccel.\nThe Ceph Deep-Dive series takes apart the internals of Ceph, an open-source distributed storage system, one piece at a time. It is written for engineers who are meeting Ceph for the first time.\nThis first part covers Controlled Replication Under Scalable Hashing(CRUSH), the data distribution algorithm that Ceph is built on. The explanation follows the paper CRUSH: Controlled, Scalable, Decentralized Placement of Replicated Data(SC'06).\nBackground Object-based Storage Object storage stores and manages data as independent objects, not as files or blocks.\nA conventional file system manages data through metadata such as the directory tree and inodes. An inode (index node) is a data structure that holds a file\u0026rsquo;s size, permissions, and the locations of its data blocks. Object storage manages data as an object ID (key) and an object (value). bucket ├── logs/2026/09/09/a.json ├── logs/2026/09/09/b.json └── model/llama/weights-001 The storage structure is simple, scales out well, and keeps replication cheap. Large object-based distributed storage splits data into small objects and spreads them across many disks. Large reads and writes are served by many storage devices in parallel. Replicas and erasure coding shards are spread out to survive failures. The core problem of object storage is which device should store each of a huge number of objects?\nChallenge Spreading objects across many devices raises three problems. Each subsection ends with the question that problem poses.\nQ1. Object Metadata Scalability In Ceph, the unit that actually stores data on a disk is called an Object Storage Daemon(OSD). The example below stores the object-to-OSD mapping in a table.\nobject-A → osd.1 object-B → osd.7 object-C → osd.3 ... Storing the mapping for every object to every disk creates a metadata load that hurts scalability and availability. On the left, every request goes through the metadata server, which becomes both a bottleneck and a single point of failure (SPOF). On the right is the CRUSH approach covered in this post. Each client computes the location itself, so no central lookup is needed. This leads to the first question CRUSH has to answer.\nCan any node compute an object\u0026rsquo;s location without storing placement information?\nQ2. Reliable Replica Placement Spreading data across devices does not by itself make it safe.\nAs on the left, if all replicas land in the same rack domain, a power failure makes the data unreachable. As on the right, with one replica per rack, the remaining replicas keep the data reachable even if one rack goes down. Replica placement must take several failure domains into account, such as rack, power, site, and server. The second question is how to reflect failure domains in placement.\nCan the placement process distinguish and structure failure domains?\nQ3. Dynamic Cluster \u0026amp; Data Rebalancing # N = number of devices device = hash(object_id) % N Hash-based placement uses randomness to spread objects evenly across devices. A storage cluster changes constantly as devices are added, removed, or replaced. When a device fails, its data has to move to other devices.\nhash(x) % 10 -\u0026gt; hash(x) % 11 ...? The example below shows how the locations of 12 objects change when the cluster grows from 4 devices to 5.\nWhen the number of devices changes, most existing object mappings change. The result is massive relocation. The last question is about how much data moves when the cluster changes.\nCan data be redistributed as the cluster changes while keeping data movement to a minimum?\nSolution: CRUSH Overview CRUSH computes object placement from a hash function instead of storing it in a mapping table.\nobject_id -\u0026gt; CRUSH(object_id, cluster_map, placement rule) -\u0026gt; [device1, device2] Because CRUSH computes placement with a hash function, data is distributed pseudo-randomly. Yet it is a deterministic algorithm: the same (object_id, cluster_map, placement rule) input always produces the same output. Deterministic Pseudo-random Placement CRUSH(X) = always OSD 7 For object X, CRUSH always returns OSD 7. X, Y, Z -\u0026gt; CRUSH(X), CRUSH(Y), CRUSH(Z) -\u0026gt; OSD 7, 3, 5 Across many objects, CRUSH results are spread randomly over many devices. Without a central placement server, every node can compute an object\u0026rsquo;s location on its own.\nCluster Map CRUSH represents the physical layout of a cluster as a weighted hierarchy tree.\nroot └── row └── cabinet └── shelf └── device (leaf disk) Tree leaves are the devices that actually store data. Tree nodes organize devices into a hierarchy and are called buckets. The paper\u0026rsquo;s example uses a four-level bucket hierarchy: root \u0026gt; row \u0026gt; cabinet \u0026gt; shelf. Adapted from Weil et al., CRUSH (SC\u0026#39;06) Figure 1 Devices and buckets each have their own weight. Device weight: the relative amount of data the device should store. It is usually proportional to capacity. Bucket weight: Σ child weights, the sum of the weights of its child buckets and devices. If the device weights in a bucket are as follows, the devices store data in a 1 : 2 ratio.\nosd.1 weight = 1 osd.2 weight = 2 The parent bucket\u0026rsquo;s weight is the sum of its child bucket weights (2+2). Each child bucket\u0026rsquo;s weight is the sum of its device weights (1+1). By mapping real failure domains to hierarchical buckets in the cluster map, such as IDC sites, racks, and network switches, you can model the physical environment.\nAssign different IDC sites to buckets under root to build a cross-IDC data cluster. Assign each rack to an upper-level bucket and spread replicas across different racks. If one rack goes down, replicas in the other racks survive, which raises availability. Assign each switch to a lower-level bucket and keep the replicas within a rack under the same switch. Replicas can then sync without crossing switches, which cuts synchronization cost. For example, with four replicas, place two each in rack1 and rack3, as the blue devices show, and keep the two in each rack under the same switch. Placement Rule A placement rule decides how to traverse the cluster map and where to place replicas.\nFor example, suppose the cluster map has three bucket levels: row, cabinet, and disk.\nroot ↓ row1 row2 ↓ ↓ cab21 cab23 cab24 ↓ ↓ ↓ disk disk disk You can keep replicas in the same row to reduce rebalancing cost. Or you can spread replicas across different rows to raise availability. CRUSH expresses these replica placement policies with its TAKE / SELECT / EMIT functions.\ntake(root) select(1,disk) emit take(root) select(1,row) select(3,cabinet) select(1,disk) emit A placement rule can separate replicas across one failure domain while confining them within another.\nPlacement Algorithm TAKE / SELECT / EMIT The placement algorithm recursively descends the cluster map according to a placement rule made of TAKE / SELECT / EMIT, and finds suitable disks for an object ID.\nTAKE(a) SELECT(n, t) EMIT - i = current working vector - b = bucket currently being descended - o = candidate selected from b root ↓ row1 row2 ↓ ↓ cab21 cab23 cab24 ↓ ↓ ↓ disk disk disk TAKE(a) puts the entry point where the traversal starts into the working vector i.\ntake(root) -\u0026gt; [root] SELECT(n,t) selects n items of type t under each item in the current working vector i and adds them to i. The pseudocode below illustrates how it works.\n[root] -\u0026gt; select(2,row) for r=1 in range(N): - r=1. i: root, b: root, o: row1 (type(o) = row, found) -\u0026gt; [row1] - r=2. i: root, b: root, o: row2 (type(o) = row, found) -\u0026gt; [row1, row2] SELECT descends recursively until it reaches the target type. select(1,disk) - i: [row1, row2] - r=1. i: row1, b: row1, o: cab21 (type(o) != disk, descend) - r=1. i: row1, b: cab21, o: disk2107 (type(o) = disk, found) -\u0026gt; [disk2107] Within a bucket, the candidate is chosen by a hash-based selection function c(r,x) that takes the index r and the object ID x. c(r,x) depends on the bucket algorithm, which is covered in the Bucket Algorithm section. b.c(r,x) -\u0026gt; cab21.c(1,21H3T#) -\u0026gt; disk2107 EMIT appends the current working vector i to the final placement result.\nemit -\u0026gt; [disk2107, disk2313] The whole process, seen from the working vector i, looks like this.\nFailure Handling In collision, crushed, and overload situations, a candidate chosen by CRUSH may not be fit to store data, so CRUSH has to search again.\nCollision: the candidate for index r has already been selected. Duplicate placement must be avoided. Crushed: the selected candidate is unusable, for example because the disk has failed. Overload: the selected candidate is overloaded and load needs to be balanced. In all three cases, CRUSH searches again and picks a different candidate.\nTo pick a different candidate, the retry passes a mutated index r' to c(r,x) instead of r.\nCRUSH counts f, the total number of failures in the SELECT, and fr, the number of failures for a specific replica. select(1,disk) - i: [row1, row2], f:0, - [r=1, f:0]. i: row1, b: row1, o: cab21 - [r=1, f:0]. i: row1, b: cab21, o: disk2107 (disk2107 Fail!!) Adapted from Weil et al., CRUSH (SC\u0026#39;06) Figure 2 (left) When the selected candidate fails, CRUSH increments the failure count (f += 1) and mutates the index to r' = r + f. As the figure shows, when slot 2 fails, the replicas after it shift left by one and the next candidate, g, fills slot 6. On a collision, it searches again inside the bucket currently being descended. select(1,disk) - i: [row1, row2], f:0, - [r=1, f:0]. i: row1, b: row1, o: cab21 - [r=1, f:0]. i: row1, b: cab21, o: disk2107 (disk2107 Collision!!) - [r\u0026#39;=2, f:1]. i: row1, b: cab21, o: disk3421 On crushed or overload, it goes back to the working vector item where the descent started and searches again. select(1,disk) - i: [row1, row2], f:0, - [r=1, f:0]. i: row1, b: row1, o: cab21 - [r=1, f:0]. i: row1, b: cab21, o: disk2107 (disk2107 Overload!!) - [r\u0026#39;=2, f:1]. i: row1, b: row1, o: cab23 - [r\u0026#39;=2, f:1]. i: row1, b: cab23, o: disk5679 Collision handling vs Crushed/Overload handling\nCollision: the disk itself is fine, so CRUSH only needs to spread replicas within the bucket to avoid duplicates. Crushed/Overload: CRUSH steers data into a different bucket branch so it reaches a healthy disk. Adapted from Weil et al., CRUSH (SC\u0026#39;06) Figure 2 (right) fr, the failure count for a specific replica, is used to keep positions stable when replica rank matters, as in erasure coding. Like f, fr is incremented on every failure, but it is reset to 0 each time a candidate search finishes. During mutation, CRUSH adds fr x N, fr multiplied by the number of candidates, instead of fr itself. This preserves position information: as the figure above shows, only the failed slot 2 changes to h, and the other replicas stay in place. select(1,disk) - i: [row1, row2], f:0, - [r=1, fr:0]. i: row1, b: row1, o: cab21 - [r=1, fr:0]. i: row1, b: cab21, o: disk2107 (disk2107 Collision!!) - [r\u0026#39;=3, fr:1]. i: row1, b: cab21, o: disk9823 Cluster Change \u0026amp; Data Movement When a device is added, removed, or reweighted, the cluster map changes and data is rebalanced.\nAdd / Remove / Reweight B1(1) B2(1) B1(1) B2(2) When a disk\u0026rsquo;s weight changes as in the example above, data has to move to match the new ratio (B1 → B2). Ideally, only the data matching the weight change migrates from B1 to B2. CRUSH does not delete or copy data itself. Instead, it guides data toward its desired placement under the updated cluster map. Hierarchical Remapping root / \\ A(1) B(2) / \\ B1(1) B2(1) root / \\ A(1) B(3) / \\ B1(1) B2(2) In a hierarchy, however, not only the weight distribution of B1 and B2 changes, but also that of the upper level, A and B. So data moves in the A → B direction at the same time as B1 → B2. Here, the placement decisions at root and at B are independent, which can cause unintended data movement. The ideal path for the moving data is A → B → B2. However, some of the data that moves from A to B may land on B1. Then B1, whose share should shrink, actually gains data and drifts from the target weight distribution. As a result, additional B1 → B2 movement occurs to restore the target weight distribution inside B. Adapted from Weil et al., CRUSH (SC\u0026#39;06) Figure 3 In other words, data does not move straight to the leaf whose weight increased. Different objects are remapped independently at each level of the hierarchy, which can cause extra, unintended data movement.\nThe cluster map should therefore be designed so that remapping caused by cluster map changes is minimized.\nBucket Algorithm Each bucket in the cluster map has its own c(r,x) implementation for choosing a candidate.\nEach bucket algorithm trades off placement computation cost against cluster change cost. Choosing an algorithm that fits the bucket keeps both costs low. Bucket Mapping Cost Addition Removal Uniform O(1) Poor Poor List O(n) Optimal Poor Tree O(log n) Good Good Straw O(n) Optimal Optimal Uniform Distributes data across items with perfectly equal probability.\nIt divides the hash value by the number of items and picks each child item with equal probability. c(r,x) = hash(r,x) mod m, where m is the number of items. It is used when all items have the same weight, and it computes placement the fastest (O(1)). In exchange, changing the number of items changes most object mappings, so its rebalancing cost is the worst. List Keeps the existing weight distribution as intact as possible by adding items to the front, like a stack.\nAll items form a linked list, and items are considered one by one from the most recently added, each with a conditional probability. The pseudocode is below. # hash values are normalized to [0, 1), # remaining_weights holds the weights from the current item to the tail. for item, weight in items: if hash(x, r, item) \u0026lt; weight / sum(remaining_weights): return item If the hash value is smaller than the current item\u0026rsquo;s share, weight / sum(remaining_weights), that item is selected. Otherwise the search moves to the next item. Because it walks the items in order from the head, placement costs O(n). When a new item is added, every object except those that now pick the new item keeps its mapping. Conditional probabilities, old list: E(3/7) -\u0026gt; D(1/4) -\u0026gt; C(1/3) -\u0026gt; B(1/2) -\u0026gt; A(1/1) Conditional probabilities, new list: F(4/11) -\u0026gt; E(3/7) -\u0026gt; D(1/4) -\u0026gt; C(1/3) -\u0026gt; B(1/2) -\u0026gt; A(1/1) After the added item F, the conditional probabilities are the same as before. In contrast, when an item in the middle or near the tail is removed, most conditional probabilities change, so object mappings change significantly. Conditional probabilities, old list: E(3/7) -\u0026gt; D(1/4) -\u0026gt; C(1/3) -\u0026gt; B(1/2) -\u0026gt; A(1/1) Conditional probabilities, new list: E(3/6) -\u0026gt; D(1/3) -\u0026gt; C(1/2) -\u0026gt; B(1/1) Tree Instead of walking a list in one direction, it narrows the search range recursively to pick an item.\nThe probability-based selection is the same as in List, but the data structure is a weighted binary tree. Adapted from Weil et al., CRUSH (SC\u0026#39;06) Figure 4 All items live in the leaves, and internal nodes only branch. The leftmost leaf always has label 1. Leaf labels are odd, and branch node labels are even. Each time the tree grows by one level, the old root becomes the left child of a new root, and the new root\u0026rsquo;s label is the old root\u0026rsquo;s label shifted left by one bit (100 → 1000). As a result, existing node labels never change as the tree grows, so the paths to existing items stay the same. Item lookup descends from the root, going left or right with probabilities based on the weight ratio. When adding an item expands the tree, a lot of data moves to the new branch, but most mappings in existing branches are preserved. Because it is a tree, placement costs O(log n). Straw Unlike List and Tree, where only some items take part in a selection, every item is a candidate every time.\nIt computes a score for every item with a hash function and picks the highest. The score corresponds to the length of the straw each item draws. In the paper, Straw is c(r,x) = max_i( f(w_i) · hash(x, r, i) ). f(w_i) is a straw length factor that grows with weight. Ceph now uses an improved version, straw2, by default. It computes the score as ln(hash(x, r, i)) / w_i. Because hash values lie in (0, 1], ln is at most 0. A larger weight pulls the score closer to 0, which makes the item more likely to win. Every item competes independently every time, so data moves only when the winning item changes after a cluster change. However, every selection must score every item, so placement costs O(n). Choosing a Bucket Algorithm Disks have identical capacity and performance, and the cluster rarely changes: use Uniform, which has the fastest placement. The cluster keeps growing and disks are never removed: use List, which makes adding items cheap, and consider Tree at large scale. Disks are added and removed often, and rebalancing cost must be minimal: use Straw, which moves less data at the cost of slower placement. The cluster changes and is quite large: use Tree, which balances placement cost and data movement cost. Wrapping Up This post looked at how CRUSH computes object locations without a mapping table. The cluster map structures failure domains, placement rules express replica placement policies, and bucket algorithms balance computation cost against data movement cost.\nThe next part covers the overall design of Ceph, which runs on top of CRUSH.\nReferences Sage A. Weil, Scott A. Brandt, Ethan L. Miller, Carlos Maltzahn. CRUSH: Controlled, Scalable, Decentralized Placement of Replicated Data. SC'06. ","date":"2026.11.03","permalink":"https://hyperaccel.ai/techblog/en/posts/ceph-crush/","series":["Ceph Deep-Dive"],"style":"--c:#009EE2;--on:#02121c","summary":"A walkthrough of CRUSH, the data distribution algorithm at the heart of Ceph, based on the SC'06 paper. It covers how object locations are computed without a mapping table, placement rules and failure handling, data movement on cluster changes, and the trade-offs between bucket algorithms.","title":"[Ceph Deep-Dive - Part 01] CRUSH"},{"authors":["Yujeon Chang"],"category":"AI Hardware","content":" This post is Part 1 of the Verifying an LLM Inference Chip series. Part 1 covers where the \u0026ldquo;ground truth\u0026rdquo; in verification comes from. Part 2 will cover how an emulator gets past the limits of slow simulation, and when you can say verification is done.\nIntroduction Hello, I\u0026rsquo;m Yujeon Chang from the HyperAccel DV team. Using HyperAccel\u0026rsquo;s actual verification flow as the example, this post looks at how we decide what counts as the \u0026ldquo;right answer\u0026rdquo; when verifying a Large Language Model (LLM) inference chip.\nLet\u0026rsquo;s start with a question. When the same AI model gives one answer on a GPU and another on the RTL of the LLM Processing Unit (LPU), who is actually wrong?\nSay we load the same AI model onto a GPU and onto the LPU\u0026rsquo;s RTL, and feed both the same prompt with the same settings. If dozens of tokens come out identical and then one position differs, is the LPU wrong, or is the GPU? It\u0026rsquo;s even possible that both are right.\nA software bug can be fixed and redeployed, but chips are a different story. Once a chip is manufactured its circuits cannot be changed, so fixing a bug means changing the design and manufacturing the chip again, which costs months and a lot of money. That\u0026rsquo;s why we have to make as sure as we can that the design is correct before manufacturing, and this work is called verification.\nWhen people hear \u0026ldquo;verification,\u0026rdquo; they often think first of testbench languages or tools like the Universal Verification Methodology (UVM). This post, however, sets the tools aside and asks a more basic question. What do you have to compare against before you can say something is \u0026ldquo;correct\u0026rdquo;? In other words, where does a chip\u0026rsquo;s ground truth come from?\nThis post comes down to two key points.\nThere is no single ground truth; each verification level gets its ground truth from a different source. Because each comparison axis changes only one thing where possible, just knowing which comparison failed tells you whose problem it is. Verification Comes Down to Comparison Before we dive in, here are a few terms used throughout this post.\nRegister Transfer Level (RTL): A design level that describes a digital circuit\u0026rsquo;s behavior in terms of data transfers and operations between registers. It is usually written in a hardware description language such as Verilog or SystemVerilog, and its function and behavior are verified with a simulator such as Synopsys VCS. Design code written at this level is called RTL code, and this post simply calls it RTL. IP and Top: An Intellectual Property (IP) is a circuit design unit responsible for a specific function; in the semiconductor industry, a unit that can be designed independently and reused like this is called an IP. A chip is built by connecting many IPs. IP-level verification checks one IP in isolation, while top-level verification checks the whole chip with all IPs connected. Reference model: Software that takes the same input as the RTL and computes the expected output, used to check whether the RTL\u0026rsquo;s output matches it. functional simulator (fsim): A simulator that models the chip\u0026rsquo;s architecture at the functional level. It is used to quickly verify instruction-level accuracy and to develop the software stack. bit-exact: Two results match down to the last bit, with no error tolerated at all. tolerance: A comparison criterion that treats two results as matching as long as their difference stays within a set range. Such differences come not from a wrong implementation but from differences in how the computation is done, such as the order of floating-point additions or numeric precision. A typical example is the same model giving slightly different results on a CPU and a GPU. Verification comes down to feeding the same input into two implementations and comparing the outputs. If they differ, something is wrong somewhere, but what you compare against determines which problems you can find.\nThe table below lists every comparison covered in this post.\n# Comparison Criterion What it catches ① IP RTL ↔ reference model bit-exact Implementation that deviates from the spec ② Top RTL ↔ fsim (rtl) bit-exact Integration errors between IPs, differing spec interpretations ③ fsim (torch) ↔ golden (CPU) tolerance Spec and algorithm errors, compiler errors ④ fsim (rtl) ↔ golden (GPU) tolerance Errors in rtl mode, spec and algorithm errors ⑤ RTL (ZeBu) ↔ fsim (rtl) bit-exact Differences between rtl mode and the RTL, errors in the RTL itself There is one important principle here. Each comparison axis changes only one thing where possible.\nIf you are a software engineer, think of git bisect, which halves a range of commits until it finds the one that introduced a bug. Comparisons work in a similar way: if two implementations differ in only one thing, the cause of a mismatch narrows down to that one thing. So just knowing which comparison broke tells you whose problem it is.\nIn practice, though, some comparisons can\u0026rsquo;t keep to this principle, and we will see an example when we run a real model.\nIP Level (①): The Spec Document Is the Ground Truth At the IP level, the ground truth is the spec document, or more precisely, a reference model that a verifier builds separately using only the spec document.\nAt HyperAccel, RTL design and verification are handled by separate, independent teams. Verifiers build the IP-level reference models from the RTL spec document, not by looking at the RTL.\nWhy go to this trouble? If the same person implements something twice with the same thinking, they tend to repeat the same mistakes, and when two implementations share a mistake, their results match perfectly. The bug then passes without any warning, which is why a reference model is only meaningful if it is built independently of the RTL.\nAt the IP level, the RTL and the reference model are compared bit-exact. They are two implementations of the same spec, so they should match down to the last bit.\nWe do not compare the IP reference model against PyTorch. At the IP level, the spec is the only ground truth, and comparing against the AI model is the job of the top level.\nWhen Independence Breaks Even With Separate Teams Separate teams don\u0026rsquo;t automatically mean independence. In general, in the following three cases, separate teams still end up sharing the same errors.\nUsing the designer\u0026rsquo;s model as-is: The verification team adopts a model built by the designer as its reference. Asking the designer about an ambiguous spec: The verifier asks the designer about an unclear part of the spec and builds the testbench from that answer. Deriving expected values from the RTL: The verifier reads the RTL instead of the spec to compute expected values. In all three cases, the designer\u0026rsquo;s interpretation leaks into verification, so the two implementations are no longer independent.\nHyperAccel\u0026rsquo;s structure avoids the first and third cases. Verifiers build the reference model themselves, and it is based on the spec document, not the RTL.\nThe second case is hard to avoid entirely, because designers write the spec document. While verifying, verifiers find ambiguous parts and ask the designer, who interprets them and updates the spec document.\nThe key point is that the answer ends up in the spec document. Verifiers don\u0026rsquo;t just apply the answer to their testbench; the updated document becomes the reference for both design and verification again. In effect, the ground truth stays in a single document.\nStill, the interpretation itself belongs to the designer, so if it is wrong, the RTL and the reference model are wrong together. Let\u0026rsquo;s look at this limitation next.\nThe Limit: If the Spec Is Wrong, Both Are Wrong the Same Way The IP-level structure has a clear limitation. Designers and verifiers both read the same spec document, and designers are the ones who write it.\nWhat if the spec document itself is wrong? The designer builds the RTL according to the wrong spec, and the verifier builds the reference model according to the same wrong spec. Both results are wrong in the same way, so they actually match perfectly.\nWhat IP-level verification proves is that \u0026ldquo;the RTL was built according to the spec,\u0026rdquo; not that \u0026ldquo;the spec is correct.\u0026rdquo; To check whether the spec is correct, you need a ground truth that does not go through the spec document, and that ground truth lives at the top level.\nTop Level: The AI Model\u0026rsquo;s Output Is the Ground Truth At the top level, we verify the whole chip with all IPs connected, and the ground truth is not the spec document but the actual AI model\u0026rsquo;s output.\nBecause the AI model was not built by reading the spec document, this ground truth has a different source. That is why errors in the spec itself, which the IP level misses, show up here.\nTransformer World introduced the attention operation, and the top level is where we check that this operation produces the same values on the chip.\nThe Golden, fsim, and the Compiler The top-level ground truth is the output of the original model published on Hugging Face, run in PyTorch. It goes through neither the hardware nor the compiler, making it the outermost ground truth, and in verification a reference result like this is called a golden.\nHowever, we do not compare the golden directly against the RTL. RTL simulation is too slow to run an entire LLM, and before the chip exists there is nowhere to try out the compiler. So we use fsim, a software model that mimics the chip\u0026rsquo;s behavior, which also splits the large gap between the golden and the RTL into two steps and helps narrow down the cause when results diverge.\nfsim is built by the simulator team, another team separate from both design and verification. Like the verification team, the simulator team builds fsim from the spec document, so in effect there is one more independent implementation that starts from the same spec.\nfsim is not built by stitching IP reference models together; it is a completely separate model. So even if an IP reference model has a mistake, that mistake does not carry over into top-level comparisons.\nfsim has two modes: torch mode and rtl mode. torch mode is used for fast debugging and functional verification, while rtl mode is a C model (cmodel) that follows the chip\u0026rsquo;s behavior exactly, aiming to produce results identical to the RTL down to the bit.\nTo run an AI model on the chip, a compiler has to turn the model into a program the chip can execute. At HyperAccel, this is the job of our own compiler, Legato, and both fsim and the RTL on ZeBu run the program it produces.\nTop-Level Simulation (②): Do the Values Match fsim? Before running an AI model, the verification team first checks the connections between IPs with fast top-level tests built in UVM. Part 2 covers those tests in detail; here we look at the tests that compare values.\nIntegration errors that don\u0026rsquo;t appear when looking at one IP at a time show up as value differences once the IPs are connected. The top-level scenarios used here are also built by the verification team, and their expected values come from fsim in rtl mode. The Top RTL and rtl-mode fsim are compared bit-exact (② in the table).\nWhat this comparison catches and misses is clear. The Top RTL and fsim were built by different teams from the same spec document, so when they disagree, it reveals a place where the same spec was interpreted differently. Conversely, if the spec itself is wrong, both are wrong the same way and pass this comparison, so catching spec errors requires comparing against the golden, which does not go through the spec.\nThree Steps With a Real Model Verification with a real AI model is built up in three steps. The simulator team and the compiler team first develop fsim and the compiler separately, then run them together in one environment.\nStep 1 (③): torch Mode and the CPU Golden, Function First First, torch-mode fsim runs the compiler\u0026rsquo;s output, and the result is compared with the golden from running the original model on a CPU. A small error is tolerated, roughly the size of the difference you get when running the same computation on a CPU versus a GPU. This step quickly checks that the functionality is right.\nStep 2 (④): rtl Mode and the GPU Golden, Closer to the Chip Next, rtl-mode fsim runs the compiler\u0026rsquo;s output, and the result is compared with the golden from running the original model on a GPU. Because a GPU\u0026rsquo;s precision and arithmetic are closer to the chip\u0026rsquo;s than a CPU\u0026rsquo;s are, this allows more precise debugging and validates rtl mode itself at the same time. This comparison also tolerates a small error.\nStep 3 (⑤): The RTL on ZeBu and rtl Mode, Down to the Bit Finally, the same compiler output runs on the actual RTL, which runs on ZeBu, an emulator from Synopsys. An emulator loads the RTL onto dedicated hardware and runs it far faster than a simulator; Part 2 will cover emulators in more detail.\nIn this step, rtl-mode fsim and ZeBu are compared bit-exact, not only on the output tokens but also on the data left in DRAM (a DRAM dump). This checks that rtl-mode fsim and the actual RTL produce exactly the same results, and that the RTL itself has no problems.\nThe Ground Truth Is Passed Down Step by Step Looking at the three steps through the \u0026ldquo;change only one thing\u0026rdquo; principle reveals something interesting.\nStep 3 (⑤) is where the principle holds best. Both sides run the same compiler output and only the executor changes from fsim to the RTL, so when there is a mismatch, the cause narrows down to rtl-mode fsim versus the RTL.\nSteps 1 and 2 (③, ④), on the other hand, break the principle. Compared with the golden, two things differ at once, the compiler and fsim, so this comparison alone can\u0026rsquo;t tell you which one is wrong when there is a mismatch.\nSo in these steps, we analyze against the golden to decide which side is wrong. We don\u0026rsquo;t stop at the final output; we match intermediate results layer by layer against the golden and narrow down the cause by finding the first layer where they diverge. A comparison that breaks the principle comes with this analysis cost, which in a way shows why the principle of splitting comparisons matters.\nIn short, the ground truth is passed down one step at a time. The golden is the ground truth for fsim, and rtl-mode fsim in turn is the ground truth for the RTL.\nThis division of roles is similar to CPU verification. In general, CPU verifiers check that each instruction behaves according to the spec, while the results of programs built from those instructions are the responsibility of the compiler and software. Chip verification works the same way: whether the chip behaves according to the spec is the job of the bit-exact comparisons between the RTL and fsim, and whether the model runs correctly is the job of the comparisons against the golden, through the compiler.\nTolerance Only Where It\u0026rsquo;s Needed Looking at the criterion column in the comparison table again, every comparison that directly involves the RTL is bit-exact. The only places that tolerate error are the two comparisons between fsim and the golden (③, ④), because fsim and the golden can both be correct and still differ in the last bits. As we saw in the glossary, even PyTorch gives slightly different results on a CPU and a GPU.\nTolerating error makes the comparison possible, but it comes at a cost. Tolerance can hide small bugs.\nAs a hypothetical example, suppose the compiler picks the wrong rounding mode for one particular operation. This error changes only the last few bits of the result, so if the difference is about the size of a CPU-versus-GPU difference, it can fall within the tolerance and pass ③ and ④.\nThat is why tolerance is kept only where it is truly needed, and each comparison\u0026rsquo;s criterion is set to match its role. Comparisons against the golden check results against a ground truth that never went through the spec, so they catch errors in the spec itself, and they can do this job even while allowing the small errors that come from differences in how the computation is done. The comparisons involving the RTL, on the other hand, are there to catch differences between two implementations given the same input, so they are all bit-exact. Where there is no tolerance, even a single-bit difference has nowhere to hide.\nConclusion: When Results Diverge, What Do You Suspect? Let\u0026rsquo;s go back to the opening question. When the same AI model gives different answers on a GPU and on the LPU\u0026rsquo;s RTL, who is wrong?\nThe answer depends on which comparison the mismatch appeared in.\nIf the IP RTL and the reference model differ, the implementation deviates from the spec. If the Top RTL and fsim differ in top-level simulation, suspect integration between IPs or spec interpretation. If fsim and the golden differ, suspect the spec or algorithm, the compiler, or fsim, and narrow it down layer by layer. If the RTL on ZeBu and rtl-mode fsim differ, suspect the RTL or rtl-mode fsim. This kind of judgment is possible because we separated the source of the ground truth at each level and kept each comparison down to one difference where possible. This structure is the backbone of verification that keeps us from having to manufacture the chip again.\nThis principle isn\u0026rsquo;t limited to chip verification. The same question comes up when you port a model to another framework, quantize it, or move it to new hardware. Decide first what the ground truth is and change only one thing at a time when you compare, and when results diverge you\u0026rsquo;ll know where to look.\nTop-level simulation, however, is slow, and running a long LLM workload to completion takes far too long. Part 2 looks at how an emulator gets past this limit, and when you can say that \u0026ldquo;verification is done.\u0026rdquo;\nP.S.: HyperAccel Is Hiring! To avoid manufacturing a chip twice, someone has to keep asking, \u0026ldquo;Is this really right?\u0026rdquo; all the way to the end. At HyperAccel, the design, verification, simulator, and compiler teams each bring their own ground truth to look at the same chip. If you\u0026rsquo;re interested in tracing where the ground truth comes from and designing the comparison axes, please apply anytime!\nCareers: https://hyperaccel.career.greetinghr.com/en/guide\n","date":"2026.10.21","permalink":"https://hyperaccel.ai/techblog/en/posts/chip-verification-ground-truth/","series":["Verifying an LLM Inference Chip"],"style":"--c:#00C2A8;--on:#02121c","summary":"When the same AI model gives different answers on a GPU and on the LPU\u0026rsquo;s RTL, who is wrong? This post covers where the ground truth comes from at each level of LLM inference chip verification, and how splitting the comparisons makes it clear who owns a mismatch.","title":"Verifying an LLM Inference Chip, Part 1: Where Does the Ground Truth Come From?"},{"authors":["Jaeho Choi"],"category":"Programming Language","content":"Hello, I am Jaeho Choi from the Compiler team at HyperAccel.\nIn part 1, we defined composeOption to join two functions whose results might be absent. Option.bind handles the connection between the first function\u0026rsquo;s result and the next function. The composite is itself a function returning Option, so we can connect another function after it. Even with possible absence, we can keep building larger functions out of smaller ones.\nDoes this approach work only for Option and missing values? Here we will build composeList, a composition operation for functions that return List.\nAfter finding the common shape of these compositions, we will connect longer sequences of functions. What lets us group consecutive steps differently without changing the result, and account for a sequence with no steps at all? These questions lead us to associativity and the identity laws. We will use Lean 4 to check why composeOption satisfies them, then read the structure we have built through the definition of a category.\n1. Connecting functions that return several results Consider a function that splits a string on spaces into a list of words, and another that turns one word into a list of characters. We want to connect them so that \u0026quot;hi ok\u0026quot; yields ['h', 'i', 'o', 'k']: find the words, find each word\u0026rsquo;s characters, then concatenate the results in order.\nCall the first function $f$ and the second $g$:\nf : String → List String -- split a string into words g : String → List Char -- turn a word into characters Ordinary function composition runs into the same problem we saw in part 1.\ng (f sentence) -- type error: g expects String, not List String $f$ returns a list of words, List String, while $g$ accepts one word, a String. Merely checking whether a value exists will not connect them. We need to apply $g$ to every word in the list and collect its results.\nLet us make the functions concrete in Lean. We define splitWords and give String.toList the name wordChars. These are the $f$ and $g$ above. #eval runs the expression that follows it and displays the result.\ndef splitWords (s : String) : List String := s.splitOn \u0026#34; \u0026#34; def wordChars : String → List Char := String.toList #eval splitWords \u0026#34;hi ok\u0026#34; -- [\u0026#34;hi\u0026#34;, \u0026#34;ok\u0026#34;] #eval wordChars \u0026#34;hi\u0026#34; -- [\u0026#39;h\u0026#39;, \u0026#39;i\u0026#39;] #eval wordChars \u0026#34;ok\u0026#34; -- [\u0026#39;o\u0026#39;, \u0026#39;k\u0026#39;] First, we can connect them by matching on the list directly. An empty list leaves no words to process, so we return []. Otherwise, we obtain the first word\u0026rsquo;s characters and concatenate them with the characters from the remaining words. This resembles the split between none and some for Option in part 1.\ndef sentenceCharsByMatch (sentence : String) : List Char := let rec collect (words : List String) : List Char := match words with | [] =\u0026gt; [] | word :: rest =\u0026gt; wordChars word ++ collect rest collect (splitWords sentence) #eval sentenceCharsByMatch \u0026#34;hi ok\u0026#34; -- [\u0026#39;h\u0026#39;, \u0026#39;i\u0026#39;, \u0026#39;o\u0026#39;, \u0026#39;k\u0026#39;] word :: rest is a list with first word word and remaining words rest. The operator ++ concatenates two lists, and collect rest applies the same rule to the remaining words. For \u0026quot;hi ok\u0026quot;, processing the two words eventually reaches collect [], which ends the recursion.\nThe code computes the result we want. But, as with runOption in part 1, collect mixes an individual computation (wordChars) with a rule for connecting results (++). To use another String → List Char function, such as reverseWordChars, we would have to edit the body of collect.\nIn part 1, we separated the rule “stop if there is no value; otherwise pass it to the next function” into Option.map and Option.join, then used Option.bind to perform both steps in composeOption. Let us similarly split up the work of collect. First, List.map applies wordChars to every word.\n#eval (splitWords \u0026#34;hi ok\u0026#34;).map wordChars -- [[\u0026#39;h\u0026#39;, \u0026#39;i\u0026#39;], [\u0026#39;o\u0026#39;, \u0026#39;k\u0026#39;]] The result of map contains all the characters we need, but its type is List (List Char), not the desired List Char. We still have to concatenate the inner lists in order.\nLean calls this operation List.flatten. Applying it to the result of map gives the desired character list.\n#eval ((splitWords \u0026#34;hi ok\u0026#34;).map wordChars).flatten -- [\u0026#39;h\u0026#39;, \u0026#39;i\u0026#39;, \u0026#39;o\u0026#39;, \u0026#39;k\u0026#39;] map applied wordChars to each word; flatten concatenated the resulting lists. We have carried out separately the two jobs that were combined in collect.\n2. The common shape behind two computations Now compare Option from part 1 with the List computation above. In both cases, map creates a nested result, although the operation that removes one layer differs.\nKind of computation Return type Nested result How to remove one layer Possible absence (Option) $A \\to \\text{Option } B$ Option (Option C) none becomes none; some result yields the inner result Multiple results (List) $A \\to \\text{List } B$ List (List C) Concatenate the inner lists in order In part 1, for m : Option B and g : B → Option C, we found (m.map g).join = m.bind g.\nFor List, Lean\u0026rsquo;s List.flatMap similarly combines map followed by flatten. For any xs : List B and h : B → List C, we have (xs.map h).flatten = xs.flatMap h.\nThe two computations remove nesting differently, but both apply the next function and then merge one layer of results. Just as we defined composeOption with Option.bind, let us define composeList with List.flatMap.\ndef composeList {A B C : Type} (f : A → List B) (g : B → List C) : A → List C := fun x =\u0026gt; (f x).flatMap g #eval composeList splitWords wordChars \u0026#34;hi ok\u0026#34; -- [\u0026#39;h\u0026#39;, \u0026#39;i\u0026#39;, \u0026#39;o\u0026#39;, \u0026#39;k\u0026#39;] To discuss composeOption and composeList together, let $M$ stand for either Option or List. We write the new composition as $\\star$ to distinguish it from ordinary function composition $\\circ$. The expression $g \\star f$ runs $f$ first, then connects $g$: when $M$ is Option, it means composeOption f g; when $M$ is List, it means composeList f g.\nConsider $f : A \\to M\\,B$ and $g : B \\to M\\,C$ in this notation. Ordinary composition $g \\circ f$ would evaluate g (f a) on input a. But f a has type M B, while $g$ expects a B; the expression is ill typed. The new composition $g \\star f$ instead passes f a to bind or flatMap along with $g$. Option passes its value to $g$ when one exists, and List passes each element to $g$. Thus $(g \\star f)(a) : M\\,C$. Two functions that ordinary composition could not connect now form one function of type A → M C.\nLet us draw arrows according to this new composition. A function A → M B is an arrow from $A$ to $B$, and B → M C is an arrow from $B$ to $C$. We choose $B$ as the destination of the first arrow because that is the type accepted by the next function. Their composite also has the form A → M C, so it is an arrow from $A$ to $C$.\nIn our example, splitWords : String → List String returns a list whose elements (String) can be passed to wordChars : String → List Char. In the arrow notation we just chose, we can therefore draw splitWords from String to String, and wordChars from String to Char.\nThe diagram shows the same relationship for either choice of $M$.\nCall a sequence of composable arrows a path. Its length is the number of arrows it contains, so a single arrow is a path of length one. In the diagram, $f$ followed by $g$ is a path of length two; the lower arrow $g \\star f$ is its composite. Because this composite has type A → M C, we can append $h : C \\to M\\,D$ and apply the same operation again.\nHaving defined $g \\star f$ for two arrows, we would like to write the result for three as $h \\star g \\star f$. But $\\star$ combines only two arrows at a time, so we must first choose a pair. Combining $f$ and $g$ gives $h \\star (g \\star f)$; combining $g$ and $h$ gives $(h \\star g) \\star f$. If these differ, the unparenthesized expression $h \\star g \\star f$ does not name a unique composite unless we specify a grouping. Let us ask when the two results coincide.\n3. Associativity: does the grouping change the computation? Consider a path of three arrows, with the following function types:\nf : A → M B g : B → M C h : C → M D Joining $f$ and $g$ first gives $g \\star f : A \\to M\\,C$; joining $g$ and $h$ first gives $h \\star g : B \\to M\\,D$. The diagram includes both choices.\nThe upper path $A \\to B \\to D$ groups $g$ and $h$ first, producing $(h \\star g) \\star f$. The lower path $A \\to C \\to D$ groups $f$ and $g$ first, producing $h \\star (g \\star f)$. Both perform $f$, $g$, and $h$ in the same order; they differ only in which portion is composed first.\nIf the two results agree, we can treat $h \\star g \\star f$ as one composite arrow without specifying the grouping. This requirement is associativity. When the upper and lower paths give the same arrow, we say the diagram commutes.\nh ⋆ (g ⋆ f) = (h ⋆ g) ⋆ f As a concrete example, append duplicateChar : Char → List Char, which repeats a character, to splitWords and wordChars. If we compose splitWords with wordChars first, we gather all characters and then repeat each one. If we compose wordChars with duplicateChar first, we repeat the characters within each word and then collect the results. Let us compare both computations on \u0026quot;hi ok\u0026quot;.\nThe arguments to composeList are written in execution order. The first expression below groups the first two functions, while the second groups the last two; both still run splitWords, wordChars, and duplicateChar in that order. #guard checks that the following condition is true and reports an error if it is false. Here, == compares the computed result with the expected list.\ndef duplicateChar (c : Char) : List Char := [c, c] #guard composeList (composeList splitWords wordChars) duplicateChar \u0026#34;hi ok\u0026#34; == [\u0026#39;h\u0026#39;, \u0026#39;h\u0026#39;, \u0026#39;i\u0026#39;, \u0026#39;i\u0026#39;, \u0026#39;o\u0026#39;, \u0026#39;o\u0026#39;, \u0026#39;k\u0026#39;, \u0026#39;k\u0026#39;] #guard composeList splitWords (composeList wordChars duplicateChar) \u0026#34;hi ok\u0026#34; == [\u0026#39;h\u0026#39;, \u0026#39;h\u0026#39;, \u0026#39;i\u0026#39;, \u0026#39;i\u0026#39;, \u0026#39;o\u0026#39;, \u0026#39;o\u0026#39;, \u0026#39;k\u0026#39;, \u0026#39;k\u0026#39;] The two expressions agree on \u0026quot;hi ok\u0026quot;. This execution alone cannot establish associativity for every choice of functions and input.\nIf associativity holds, three arrows in the same order give the same result regardless of which pair we compose first. Applying the law repeatedly extends this to four, five, or any finite number of arrows: parentheses do not change the composite. In programming terms, we may move a consecutive group of steps into a helper function and insert that helper back into the original sequence without changing the whole computation.\n4. Identity laws: what does “do nothing” mean? We can now handle a path consisting of f alone, a path that follows f and then g, one that continues through h, and, more generally, a path containing $n$ arrows. With one arrow, we take that arrow itself as the result; with two or more, we compose them in turn. Associativity ensures that a fixed sequence of arrows has the same composite regardless of which part we group first.\nCan we take this one step further and handle a path of length zero in the same way? Just as we have assigned a composite arrow to each path, we want to assign an arrow to an empty path that traverses no arrows. A path that starts at $A$ and stays at $A$ must correspond to an arrow whose source and destination are both $A$. Placing an empty path before or after another path adds no computation steps, so composing this arrow must leave the original result unchanged.\nFor ordinary function composition, id_A : A → A, which returns its input unchanged, plays this role. Composing id_A before a function $p : A \\to B$, or id_B after it, leaves $p$ unchanged:\np ∘ id_A = p id_B ∘ p = p Our new composition also needs an arrow with this role at each type $A$. But fun x =\u0026gt; x has type A → A, not the A → Option A type required of an Option arrow. A candidate for Option is fun x =\u0026gt; some x. For List, a candidate is fun x =\u0026gt; [x], which turns one input into exactly one result. Returning [] would discard the input before the next computation; returning [x, x] would run that computation twice. Neither preserves it.\nWe will call the functions that place an ordinary value into these result types pureOption and pureList:\nCandidate identity arrow for Option: pureOption x = some x Candidate identity arrow for List: pureList x = [x] For $f : A \\to \\text{Option } B$, the candidates on its two sides are $\\text{pureOption}_A : A \\to \\text{Option } A$ and $\\text{pureOption}_B : B \\to \\text{Option } B$. Both equations must hold for our new composition $\\star$:\nf ⋆ pureOption_A = f -- wrap first, then run f pureOption_B ⋆ f = f -- run f, then wrap its result In our arrow notation, they look like this:\nThe loops pureOption_A and pureOption_B each start and end at the same object. Traversing the loop at $A$ before $f$, or the loop at $B$ after $f$, must yield the same composite as $f$ itself. These are the two identity laws. When they hold, the loops can serve as the composites of the empty paths at $A$ and $B$.\nFor List, let us check both directions using wordChars : String → List Char. Wrapping \u0026quot;hi\u0026quot; with pureList yields [\u0026quot;hi\u0026quot;]. Applying flatMap wordChars to that one-element list returns wordChars \u0026quot;hi\u0026quot;, or ['h', 'i']. Putting pureList before the computation has left this result unchanged.\nIn the opposite direction, running wordChars \u0026quot;hi\u0026quot; first yields ['h', 'i']. Wrapping each character with pureList produces [['h'], ['i']]; concatenating these lists gives ['h', 'i'] again. Putting pureList after the computation also preserves the result. The same reasoning applies to empty and longer result lists: each element is wrapped once and then concatenated. Absence and multiplicity are different kinds of computation, but each needs a way to preserve a computation on both sides of composition.\nWe can check both directions in Lean:\ndef pureList {A : Type} (x : A) : List A := [x] #eval (pureList \u0026#34;hi\u0026#34;).flatMap wordChars -- [\u0026#39;h\u0026#39;, \u0026#39;i\u0026#39;] #eval (wordChars \u0026#34;hi\u0026#34;).flatMap pureList -- [\u0026#39;h\u0026#39;, \u0026#39;i\u0026#39;] Like $0$ for addition or $1$ for multiplication, pureOption and pureList provide an element that leaves their respective compositions unchanged.\n5. From checking examples to proving the laws In section 3, we compared two List composites on one input. In section 4, we chose candidate identity arrows for Option and List. Associativity and the identity laws, however, must hold for arbitrary types, functions, and inputs. Let us return to the Option composition from part 1 and prove its laws. An Option result is either none or some b, so we can inspect those two cases.\nInstead of running chosen inputs with #eval or #guard, we will state that the two expressions agree for any choice of functions and explain why. Lean checks the claim together with the reasoning. A proof here covers every possible case for an arbitrary input, rather than substituting one particular input.\nFirst, we repeat the definition of composeOption and define its candidate identity arrow pureOption.\n-- Option composition and its candidate identity arrow def composeOption {A B C : Type} (f : A → Option B) (g : B → Option C) : A → Option C := fun x =\u0026gt; (f x).bind g def pureOption {A : Type} (x : A) : Option A := some x 5.1. Associativity: the two groupings define the same function In the notation of section 2, composeOption f g is $g \\star f$. We want to prove that grouping f with g gives the same function as grouping g with h. To show that two functions are equal, it suffices to show that they return the same result for every input. This principle is function extensionality.\nTake an arbitrary input x and split on the first result f x. If it is none, both groupings return none. If it is some b, both compute g b, stop if its result is absent, and pass its value to h otherwise. The two groupings reduce to the same computation, so they agree for every input.\nThe code below expresses this argument in a form Lean can check. After theorem, we write the theorem\u0026rsquo;s name and its arbitrary types and functions; after :, we state the equality to prove. The commands after by construct its proof.\n-- Associativity -- h ⋆ (g ⋆ f) = (h ⋆ g) ⋆ f theorem composeOption_assoc {A B C D : Type} (f : A → Option B) (g : B → Option C) (h : C → Option D) : composeOption (composeOption f g) h = composeOption f (composeOption g h) := by funext x unfold composeOption cases f x with | none =\u0026gt; rfl | some b =\u0026gt; rfl funext x uses function extensionality to turn an equality of functions into an equality of their results on an arbitrary input x. unfold composeOption expands the definition of composition, and cases f x splits on the first result. Finally, rfl asks Lean to check that both sides reduce to the same expression using the definitions. Each command corresponds to a step of the argument above.\n5.2. Identity laws: inserting an identity leaves a function unchanged First, connect pureOption after f. When f x is none, it remains none; when it is some b, pureOption b returns some b again. Both cases preserve the original result. In composition notation, this is pureOption ⋆ f = f. The identity arrow appears on the left of the symbol, so this is called the left identity law.\n-- Left identity -- pure ⋆ f = f theorem composeOption_left_id {A B : Type} (f : A → Option B) : composeOption f pureOption = f := by funext x unfold composeOption pureOption cases f x with | none =\u0026gt; rfl | some b =\u0026gt; rfl The proof likewise splits on the two cases of f x; evaluation of the definitions establishes each equality.\nNow run pureOption first, then connect f. Because pureOption x is always some x, passing it to Option.bind immediately gives f x. In composition notation, this is f ⋆ pureOption = f. This time the identity arrow appears on the right, giving the right identity law.\n-- Right identity -- f ⋆ pure = f theorem composeOption_right_id {A B : Type} (f : A → Option B) : composeOption pureOption f = f := by funext x unfold composeOption pureOption rfl The first result is always some x, so this proof needs no case split.\nThese three theorems prove the laws for every type and function admitted by the declarations. We can also prove the general laws for List. A direct proof uses induction: establish the law for the empty list, then use the result for the rest of a list to establish it after adding one element. Here we focus on the two-case proofs for Option.\nYou can run the complete example and proofs in the Lean 4 Playground.\n6. Different computations, the same rules: categories and Kleisli categories Option stops when a value is absent; List concatenates multiple results. Yet both compositions we built take a function of the form A → M B as an arrow and produce another arrow of that form. We also examined the associativity and identity laws needed to compose several such arrows.\nThis resemblance is no accidental coding pattern. The details of the computation differ, but a common structure remains: which arrows we use, how we compose them, and which laws the composition obeys. Mathematics calls these arrows morphisms and calls a structure of objects, morphisms, composition, and identity morphisms satisfying those laws a category. Let us state the definition and see how our proven Option composition fits it.\nObjects: a collection of things from which morphisms start and at which they end. Morphisms: for every pair of objects $A,B$, a collection $\\mathrm{Hom}_{\\mathcal C}(A,B)$ of morphisms from $A$ to $B$. We write a member $f$ as $f : A \\to B$; every morphism has a specified source and target. Composition: for $f \\in \\mathrm{Hom}_{\\mathcal C}(A,B)$ and $g \\in \\mathrm{Hom}_{\\mathcal C}(B,C)$, a morphism $g \\circ f \\in \\mathrm{Hom}_{\\mathcal C}(A,C)$. Identity morphisms: for each object $A$, a designated $\\mathrm{id}_A \\in \\mathrm{Hom}_{\\mathcal C}(A,A)$. This is the morphism assigned to the empty path at $A$. The data must satisfy these laws for $f : A \\to B$, $g : B \\to C$, and $h : C \\to D$:\nh ∘ (g ∘ f) = (h ∘ g) ∘ f -- associativity f ∘ id_A = f -- right identity id_B ∘ f = f -- left identity We say “collection” rather than “set” because the objects of a category cannot always be gathered into a single set. For this series, though, thinking of the objects and morphisms as sets will suffice.\nKeeping the objects while changing the morphisms, composition, or identities can give us a different category. We will compare two categories whose objects are types in Lean\u0026rsquo;s Type. Objects need not be types, and morphisms need not be functions, in an arbitrary category.\n6.1. The category $\\mathbf{Type}$ of types and ordinary functions Objects: types $A,B,C,\\dots$ in Lean\u0026rsquo;s Type. Morphisms: $\\mathrm{Hom}_{\\mathbf{Type}}(A,B)$ consists of functions f : A → B. Composition: join f : A → B and g : B → C as g ∘ f : A → C, evaluating g (f a) on input a. Identity morphism: id_A : A → A at each type $A$, returning its input unchanged. These data satisfy the category laws. Whichever pair we compose first among three functions, the result on input a is h (g (f a)). Composing id_A or id_B with $f$ changes no output. We will write this category as $\\mathbf{Type}$.\n6.2. The Kleisli category for Option Now keep the same types as objects but use the Option arrows and composition we built. Temporarily call the resulting category $K$.\nObjects: the same types $A,B,C,\\dots$. Morphisms: $\\mathrm{Hom}_K(A,B)$ consists of functions f : A → Option B. We may write them as $A \\to_K B$. Composition: join f : A → Option B and g : B → Option C as composeOption f g, or $g \\star f : A \\to \\text{Option } C$, evaluating (f a).bind g on input a. Identity morphism: pureOption_A : A → Option A at each type $A$, wrapping a as some a. Calling f : A → Option B a morphism from $A$ to $B$ in this category does not mean changing its return type. f still returns Option B. The diagram shows the difference: the very same function f : A → Option B is a morphism from $A$ to Option B in $\\mathbf{Type}$, but a morphism from $A$ to $B$ in the new category.\nThe theorems composeOption_assoc, composeOption_left_id, and composeOption_right_id in section 5 prove the associativity and both identity laws for arbitrary types and functions. Therefore, these data form a category. We call composeOption Kleisli composition and the resulting category the Kleisli category of Option.\nChoosing List instead gives a separate Kleisli category with the same types as objects, functions A → List B as morphisms, composeList as composition, and pureList as identity. We do not mix Option and List morphisms within one of these categories. For example, f : A → Option B and g : B → List C cannot be passed directly to either composeOption or composeList. Here we proved the laws for Option in Lean 4 and left the general proof for List aside.\n7. The structure we found in composition, and the next question In part 1, we wanted to connect functions whose results might be absent. Here we extended composition to functions with multiple results and established why regrouping steps can preserve a longer computation. Starting with code that assembles individual functions, we can now describe the mathematical structure supporting that assembly as a category.\nIn part 3, we will return to $\\mathbf{Type}$, whose morphisms are ordinary functions A → B. The category definition above says nothing about a type\u0026rsquo;s fields or how many values it contains. Yet in programming, we make types that hold two values together, as in a struct, or one of several alternatives, as in an enum. Can we describe these types by the relationships among functions entering and leaving them, rather than by their internal representation?\nWe will start with functions into the two fields of a struct and functions that handle each case of an enum. Given such functions, we will ask when there is exactly one compatible morphism. A condition that describes an object by the existence and uniqueness of a morphism is called a universal property. Types with the same universal property are isomorphic: inverse functions connect them, even if their internal implementations differ.\n","date":"2026.10.19","permalink":"https://hyperaccel.ai/techblog/en/posts/kleisli-category/","series":["Monads and Category Theory Starting from Code"],"style":"--c:#009EE2;--on:#02121c","summary":"Compare composition for Option and List, prove its laws, and recognize the resulting category.","title":"Monads and Category Theory ② Different Computations, the Same Composition"},{"authors":["Hyunjun Park"],"category":"Compilers \u0026 Kernels","content":"Introduction Hello! I\u0026rsquo;m Hyunjun Park from the CL (Compute Library) team at HyperAccel. In Part 1 we took torch.compile() apart into three components: TorchDynamo, AOTAutograd, and TorchInductor (Dynamo and Inductor from here on). Dynamo pulls a graph out of Python code, AOTAutograd tidies that graph up, and Inductor turns the tidied graph into kernels. (Part 1 covers the concept of a graph, if you need a refresher.)\nThis post starts from a question: why did PyTorch bother splitting it into three stages? What do you gain from that split, and who is it for?\nI think part of the answer is technical modularity, but another part lies in making the ecosystem extensible. So we\u0026rsquo;ll first look at how PyTorch drew people in and built its ecosystem, and which direction it has been widening that ecosystem since 2.0. Then we\u0026rsquo;ll go through what each of the three stages produces and who uses those results, and how, and close with the cases that use this structure most broadly.\nNote that this post mixes what PyTorch has officially stated with my own subjective interpretation. I\u0026rsquo;ve marked the interpretive parts with phrases like \u0026ldquo;I think\u0026rdquo; or \u0026ldquo;my reading is,\u0026rdquo; so please keep that in mind as you read.\nThis series runs to three parts. I\u0026rsquo;ll be attending PyTorch Conference 2026 next week, and the next part will cover the latest trends I learn there and what I take away from it.\nPart 1. PyTorch, grown by its ecosystem Software is an ecosystem game Think about how you pick a messenger app. What you look at first isn\u0026rsquo;t the feature list; it\u0026rsquo;s where your friends are. Even when a new messenger with better features comes out, we end up staying on the one where more people already are.\nAnd a messenger with lots of people also gathers features. Gifting, payments, reservations: features that only make sense with lots of users show up, and those features pull in more people. People bring features, and features bring more people. Once that loop gets going, it gets harder and harder for other messengers to win.\nDeep learning frameworks work the same way. For users, the rational choice is the framework with the most examples, answered questions, and published models. An environment where one search gets you unstuck is productivity in itself. The same goes for people building libraries and tools: build on the framework with the most users, and your work gets used. For both sides, choosing what most people already use is the most rational choice.\nHow PyTorch drew people in The framework that built this kind of ecosystem best is PyTorch. Around 2017, when PyTorch was first released, the leading framework was TensorFlow. In TensorFlow 1.x, writing code didn\u0026rsquo;t compute anything right away. Code like y = x * 2 + 1 didn\u0026rsquo;t compute anything; it built a graph that said \u0026ldquo;compute this later,\u0026rdquo; and the actual computation happened only when you opened a session and ran it. So print(y) showed graph info instead of a value, you had to learn graph-specific syntax instead of Python\u0026rsquo;s if, and when an error occurred it was hard to tell which line caused it. In PyTorch, the same code computes right away.\n# PyTorch x = torch.tensor(3.0) y = x * 2 + 1 print(y) # tensor(7.) PyTorch could simply be used like Python. Each line ran and computed immediately, so you could check intermediate values with print and fix the offending line as soon as an error appeared. Researchers, who need to try new ideas quickly, moved to PyTorch first, and TensorFlow itself made the same approach the default starting with 2.0 (2019).\nDevelopers came first, drawn by that convenience, and once they arrived the rest followed. Model hubs like Hugging Face and inference engines like vLLM and SGLang were built on top of PyTorch, and it became harder and harder for developers to find an alternative to PyTorch.\nWhat torch.compile had to protect In March 2023, torch.compile() arrived with PyTorch 2.0. It was an attempt to boost performance while keeping the convenience of writing plain Python.\nBy then, PyTorch was already a huge ecosystem. It wasn\u0026rsquo;t just a lot of user code. Tools that take PyTorch models and optimize them their own way were already established. NVIDIA\u0026rsquo;s TensorRT turned PyTorch models into fast inference engines, and Google\u0026rsquo;s PyTorch/XLA was the path for running PyTorch code on TPUs.\nI think a new compiler had to protect two things to avoid breaking that ecosystem. First, users should barely have to change their code. Second, tools like these should work together with torch.compile(). Had torch.compile() been a closed design used only with PyTorch\u0026rsquo;s own compiler and Triton, users would have had to pick between torch.compile() and the tools they already relied on, and the ecosystem would have split.\nPointing outward At launch, Inductor could generate two kinds of code: Triton for GPUs and C++ for CPUs. But look at the releases since, and a large share of the new features open paths for compilers and hardware from outside PyTorch to plug in: device code generator registration (2.1), Python custom ops (2.4), device extension autoloading and Intel GPU support (2.5), the device-agnostic torch.accelerator API (2.6), and official accelerator integration docs (2.9).\nIn short, instead of building everything itself, PyTorch has been moving toward adding more paths for outside compilers and hardware to come in. With each release, there are more public APIs and documents for connecting outside modules. And these paths are possible inside torch.compile() because torch.compile() was split into three stages in the first place. Part 2 looks at why.\nPart 2. Why three stages? If it were one piece Let\u0026rsquo;s imagine the opposite: torch.compile() as a single monolithic compiler that takes Python code and emits Triton code. A team that wants another compiler like TensorRT would have to do graph capture themselves, interpreting bytecode and setting up guards. A company bringing up new hardware would have to rebuild everything from graph capture to code generation. And when PyTorch improved graph capture, outside users wouldn\u0026rsquo;t get the benefit.\nSplit it into three stages and connect them with data in a defined format, though, and the story changes. Whoever needs a given format can pick up that stage\u0026rsquo;s output, and whoever wants to replace just one stage can keep using the rest as is. You use torch.compile() as far as you need, then branch off at the stage you want and attach your own tool.\nIn this post, I\u0026rsquo;ll call each stage by the job it does. The order is the same as the three components from Part 1.\nStage 1, Graph Capture: Dynamo extracts a graph from Python code. Stage 2, Graph Normalization: AOTAutograd tidies that graph into core ATen ops. Stage 3, Code Generation: Inductor groups the ops, then writes the kernel code. Stage 1, Graph Capture: for anyone who needs a graph Dynamo produces an FX graph and guards. Extracting a graph from Python code is tricky, but Dynamo does it for you, so other tools only need to take its result.\nThe main users of Dynamo\u0026rsquo;s output are other compilers: separate compilers such as NVIDIA\u0026rsquo;s TensorRT (backend=\u0026quot;torch_tensorrt\u0026quot;), Intel\u0026rsquo;s OpenVINO (backend=\u0026quot;openvino\u0026quot;), and Google\u0026rsquo;s XLA (backend=\u0026quot;openxla\u0026quot;) that take a graph and handle everything from optimization to kernel generation. Pass the name to backend, and that compiler does what AOTAutograd and Inductor would do, in its own way. Because it decides even how to group ops, it differs from the code generation stage, which only changes how the code gets written.\nThey\u0026rsquo;re used instead of Inductor because they existed before torch.compile() and are independent tools tuned for years for their hardware: TensorRT for inference on NVIDIA GPUs, OpenVINO for Intel CPUs and GPUs, and XLA for TPUs, where Triton doesn\u0026rsquo;t run. Handing them the graph is more natural than rebuilding them, and users only change one string.\nLet\u0026rsquo;s see it directly by passing backend a function that just prints the graph it receives.\nimport torch import torch.nn.functional as F def show_graph(gm, example_inputs): print(gm.code) # print the graph the backend received return gm.forward # run it unchanged def f(x, w, b): return F.relu(F.linear(x, w, b)) x, w, b = torch.randn(2, 16), torch.randn(16, 16), torch.randn(16) torch.compile(f, backend=show_graph)(x, w, b) Here are the key lines of the output (PyTorch 2.10).\nlinear = torch._C._nn.linear(l_x_, l_w_, l_b_) relu = torch.nn.functional.relu(linear) The F.linear and F.relu the user wrote come through as a graph, as written. The receiver doesn\u0026rsquo;t have to interpret Python; it just processes these ops.\nStage 2, Graph Normalization: fewer ops Now let\u0026rsquo;s receive the same function after AOTAutograd. Just wrap the show_graph from above in aot_autograd.\nfrom torch._dynamo.backends.common import aot_autograd from torch._decomp import core_aten_decompositions aten_backend = aot_autograd(fw_compiler=show_graph, decompositions=core_aten_decompositions()) torch.compile(f, backend=aten_backend)(x, w, b) permute = torch.ops.aten.permute.default(arg1_1, [1, 0]) addmm = torch.ops.aten.addmm.default(arg2_1, arg0_1, permute) relu = torch.ops.aten.relu.default(addmm) linear has been broken into a transpose (permute) and a fused matrix multiply-add (addmm), and every op now carries a name from core ATen, PyTorch\u0026rsquo;s set of basic ops. This is the decomposition we saw in Part 1.\nThe value of the graph after this stage is clear from the numbers. PyTorch has more than 2,500 ops counting overloads (as of 2.14), but core ATen has about 190. Take this graph and you only have to handle about 190 ops instead of 2,500-plus. That\u0026rsquo;s why backends with their own graph compiler usually take the graph here.\nStage 3, Code Generation: swapping only the writing part The last is the code generation stage, handled by Inductor. Despite the name, Inductor decides how to group the ops before it writes any code. Say we compile y = x * 2 + 1. Inductor first rewrites the computation as a per-element expression, y[i] = x[i] * 2 + 1. Grouping the multiply and the add into one kernel, that is, fusion, is also done at this point. Which language to write that expression in is decided at the very end.\nWritten for a GPU, it becomes Triton code (key lines only).\ntmp0 = tl.load(in_ptr0 + x0) # read x[i] tmp2 = tmp0 * 2.0 # * 2 tmp4 = tmp2 + 1.0 # + 1 tl.store(out_ptr0 + x0, tmp4) # write y[i] Written for a CPU, the same expression becomes C++. Here is a trimmed version of what PyTorch 2.10 actually generated on a CPU.\nauto tmp0 = Vectorized\u0026lt;float\u0026gt;::loadu(in_ptr0 + x0); // read x[i] auto tmp3 = tmp0 * Vectorized\u0026lt;float\u0026gt;(2.0); // * 2 auto tmp6 = tmp3 + Vectorized\u0026lt;float\u0026gt;(1.0); // + 1 tmp6.store(out_ptr0 + x0); // write y[i] The two snippets are the same computation written in different languages. The order of reading, multiplying, adding, and writing was already decided by Inductor. Whoever uses this stage replaces only this \u0026ldquo;writing\u0026rdquo; part with their own.\nGPU vendors barely need to change even that. The Triton compiler turns the Triton code above into machine code for each GPU, so AMD and Intel GPUs that Triton supports use the same code as is. A new chip that Triton doesn\u0026rsquo;t fit builds a third way of writing it and registers it with register_backend_for_device. Then the same y[i] = x[i] * 2 + 1 comes out in that chip\u0026rsquo;s language.\nThat said, the way Inductor groups ops was designed around GPUs and CPUs, and this registration API is still private. If the grouping itself has to be decided around the chip, it\u0026rsquo;s better to take the normalized graph and compile it yourself.\nWhat doesn\u0026rsquo;t get compiled: fallback and PrivateUse1 torch.compile() can\u0026rsquo;t compile everything. Regions split off by graph breaks and ops the compiler doesn\u0026rsquo;t support aren\u0026rsquo;t compiled; they run the original way instead, one line at a time. This is called fallback. For new hardware to be really usable, that fallback has to run on the hardware too.\nPrivateUse1 solves the fallback. The dispatcher we saw in Part 1 picks a kernel based on which device a tensor lives on, and PyTorch reserves a dispatch key called PrivateUse1 for devices outside the core. Name it with torch.utils.rename_privateuse1_backend(\u0026quot;npu\u0026quot;) and register a kernel for each ATen op, and fallback ops run on that device too. tensor.to(\u0026quot;npu\u0026quot;), torch.accelerator, and autograd come along with it.\nOps that aren\u0026rsquo;t in ATen are defined with torch.library.custom_op. If you also register a fake implementation that only reports the output\u0026rsquo;s shape and dtype, Dynamo can keep the op as a single node in the graph, so it works both when code runs line by line and when it is compiled.\nAt a glance Stage What it hands over Who uses it What they gain Stage 1: Graph Capture (Dynamo) A graph of the ops exactly as the user wrote them Compiler backends like TensorRT, OpenVINO, XLA, plus torch.export No need to interpret Python code Stage 2: Graph Normalization (AOTAutograd) A graph with only core ATen ops left Backends with their own graph compiler Ops to handle drop from 2,500+ to about 190 Stage 3: Code Generation (Inductor) A per-element expression with grouping already done GPUs supported by Triton, new chips with a swapped code generator Only the \u0026ldquo;writing\u0026rdquo; part needs to be built Fallback: the dispatcher One op that wasn\u0026rsquo;t compiled (run directly) Every device Non-compiled parts still run on that device Part 3. How chips that aren\u0026rsquo;t GPUs use this structure The companies that use this structure most broadly are the ones building AI chips that aren\u0026rsquo;t GPUs, commonly called Neural Processing Units (NPUs). Triton often doesn\u0026rsquo;t fit them as is, so they have to come in directly at the graph normalization or code generation stage, and they have to handle fallback themselves too. Part 3 takes a quick look at three companies that actively use torch.compile().\nHuawei Ascend: both the normalized graph and code generation Huawei\u0026rsquo;s Ascend uses both the graph normalization and code generation stages. First, a backend called TorchAir takes the normalized graph and hands it to Ascend\u0026rsquo;s own graph engine. Then, at the code generation stage, it registers an Ascend code generator with Inductor. The device is connected by naming PrivateUse1 npu. In short, the TorchAir path reuses Dynamo and AOTAutograd and replaces Inductor with Ascend\u0026rsquo;s own graph engine. The Inductor path reuses everything up to Inductor\u0026rsquo;s op grouping and replaces only the writing part with Ascend\u0026rsquo;s own.\nIntel Gaudi: taking the normalized graph to its own compiler Intel Gaudi uses the normalized graph. A backend called hpu_backend takes this graph and hands it to Gaudi\u0026rsquo;s own graph compiler, without going through Inductor. The hpu device isn\u0026rsquo;t PrivateUse1 but a device type built into PyTorch core. In short, Gaudi reuses Dynamo and AOTAutograd and replaces all of Inductor with its own graph compiler.\nIBM Spyre: swapping only the code generator IBM\u0026rsquo;s Spyre uses the code generation stage. It registers a Spyre code generator with Inductor, so the computation Inductor grouped is written out as Spyre code, and IBM\u0026rsquo;s own compiler handles the final step. The device is connected by naming PrivateUse1 spyre. Spyre used to bypass Inductor, and according to a talk abstract, moving to the code generation stage let them cut a large part of their own compiler code. In short, Spyre reuses Dynamo, AOTAutograd, and Inductor\u0026rsquo;s op grouping, and replaces only the writing part and the final conversion after it with its own modules.\nSide by side Stage used How it connects Device Huawei Ascend Graph normalization + code generation Normalization: TorchAir → its own graph engine / Code generation: code generator registered with Inductor PrivateUse1 npu Intel Gaudi Graph normalization hpu_backend → its own graph compiler Built-in hpu IBM Spyre Code generation Code generator registered with Inductor → its own compiler PrivateUse1 spyre All three reuse Dynamo\u0026rsquo;s graph capture as is and handle fallback on their own devices. Where they differ is whether they come in at graph normalization or code generation, that is, \u0026ldquo;how many hardware decisions to leave to Inductor.\u0026rdquo; The more you leave to it, the less you build; the less you leave, the more you decide for your hardware yourself.\nFor reference, HyperAccel\u0026rsquo;s LPU takes the normalized graph like Gaudi does, and builds kernels with our own language, Legato. See the Legato documentation for details.\nConclusion Back to the opening question: why is torch.compile() split into three stages? I see two reasons.\nThe first is technical. Extracting a graph from Python code (Dynamo), cleaning that graph up into standard ops (AOTAutograd), and generating code for hardware (Inductor) are very different jobs. Keeping them separate lets each evolve on its own and makes it easy for other tools to pick up the intermediate results.\nThe second is about the ecosystem. PyTorch drew in researchers and developers by being as easy to use as plain Python, models and tools piled up on top of it, and it became the de facto standard. When it built torch.compile() to tackle the next challenge, speed, PyTorch designed each stage\u0026rsquo;s output so outside compilers and hardware could pick it up. The official PyTorch 2.0 introduction states the goal of simplifying the backend (compiler) integration experience, and explains that the reduced op set is there to help vendors simplify their integrations. So other compilers like TensorRT take the output of graph capture, those with their own graph compiler take the output of graph normalization, new hardware comes in at the code generation stage, and whatever isn\u0026rsquo;t compiled is caught by PrivateUse1.\nAnd vendors are in fact flocking to these paths. As we saw in Part 3, companies building accelerators beyond GPUs, NPUs in particular, are coming in at whichever stage suits them, and PyTorch keeps paving those paths with official APIs and documentation release after release. I can\u0026rsquo;t claim this was a strategy PyTorch planned from day one. But in messenger terms, my reading is that a platform with plenty of people opened paths to the outside, new participants are flocking in through them, and the ecosystem is growing beyond GPUs.\nNext time, I\u0026rsquo;ll widen the view a little and share what I took away from PyTorch Conference 2026, held October 20–21 in San Jose. Thanks for reading this long post.\nReference PyTorch documentation and blog\nPyTorch 2.0 release blog — the launch of torch.compile() Get Started with PyTorch 2.0 — the stated goal of simplifying backend integration Custom Backends — attaching a backend to the output of the graph capture and normalization stages torch.compiler — list of available backends torch.export — deployment export built on Dynamo graph capture IRs (Core ATen IR) — the list of core ATen ops Accelerator Integration — guide to PrivateUse1-based accelerator integration Python Custom Operators — torch.library.custom_op Autoloading Out-of-Tree Extension — device extension autoloading Intel GPU Support Now Available in PyTorch 2.5 OpenReg: A Self-Contained PyTorch Accelerator Simulator PyTorch 2.9 Release Blog — LibTorch Stable ABI PyTorch v2.13.0 Release Notes — torch.compiler.set_default_backend Compilers and hardware using each stage\nTensorRT Backend for torch.compile, PyTorch/XLA TorchDynamo Integration Accelerate PyTorch Models using torch.compile on AMD GPUs with ROCm Ascend Extension for PyTorch (torch_npu), TorchAir, Triton-Ascend Intel Gaudi PyTorch Theory of Operations, Gaudi PyTorch Support Matrix, Gaudi PyTorch Bridge torch-spyre, torch-spyre compiler architecture, IBM Research: PyTorch support for IBM Spyre, PyTorch Conference Europe 2026 talk HyperAccel\nLegato documentation Legato, A Programming Language for HyperAccel LPU P.S. HyperAccel is hiring! HyperAccel builds a new chip called the LPU, but we\u0026rsquo;re also a team that follows the latest software trends up close and builds right at the front line. As with the torch.compile() ecosystem in this post, the software stack around new hardware is still changing fast, and that\u0026rsquo;s where we work.\nIf you\u0026rsquo;d like to join the research and development that lets users run new devices comfortably, with the tools they already know, apply through HyperAccel Career!\n","date":"2026.10.16","permalink":"https://hyperaccel.ai/techblog/en/posts/torch-compile-anatomy-2/","series":["Exploring torch.compile()"],"style":"--c:#5BC5F6;--on:#02121c","summary":"Why is torch.compile split into Dynamo, AOTAutograd, and Inductor? Starting from how PyTorch grew its ecosystem, we look at who uses the output of each stage, how, and what they gain from it.","title":"[Exploring torch.compile() - Part 02] Why Split It into Three Stages?"},{"authors":["Jaeho Choi"],"category":"Compilers \u0026 Kernels","content":"Hello, I am Jaeho Choi from the Compiler team at HyperAccel.\nA monad has a reputation in functional programming as a concept that is unusually difficult to explain. Even if you have heard the name, it can be hard to say what it is and why we use it.\nMy reason for introducing monads is the beauty of function composition. We connect small functions to build a larger function, then connect that function to others. Being able to assemble a larger computation while understanding what each function does is one of the appeals of functional programming.\nWhen we can pass one function\u0026rsquo;s result to the next, the task is straightforward. In real code, though, those connections do not always work as we intend. A function may produce no value or several results. We may also need to pass an updated state to the next step, or connect operations that interact with the outside world, such as reading a file. Can we preserve the ability to compose functions in sequence for these computations too?\nThis series begins where those connections break down. We will first write code that makes them work, then use the language of category theory to explore the laws that code satisfies and the mathematical structure within it. Along the way, we will work toward understanding what a monad is and how it lets us continue composing functions even when computations may produce no value or multiple results.\nThere is a reason for taking this approach. If you have looked for material on monads, you have probably encountered one of two barriers.\nOn one side stands the forbidding mathematical declaration, “A monad is just a monoid in the category of endofunctors.” Quoted without its original context, this unfamiliar mathematical vocabulary can exhaust readers before they even begin. On the other side are countless everyday analogies: “A monad is a burrito,” or “A monad is a box.” However many analogies you read, it can still be hard to find a clear answer to the basic questions: “Why should I learn about monads? Why does my code need this?”\nBetween mathematical declarations and everyday analogies lies code we can actually run. As we carry out computations and see exactly where their connections fail, the things those unfamiliar mathematical terms describe can begin to take a more concrete shape. By following the code, we will work toward reading a once-distant mathematical definition as a description of the connections we have built ourselves.\nWe will write that code in Lean 4. Lean 4 is both a functional programming language and an interactive theorem prover (ITP), which checks proofs of mathematical propositions. At first, we will use it to write and run functions. Later, we will use the same language to prove laws about the composition rules we have built. I will introduce the syntax as we need it in the examples.\nIn this article, we will connect two functions that may produce no value and build a reusable composition rule that works even when the functions change. Let us begin by connecting two ordinary functions.\n1. Function composition: when outputs and inputs match One way to manage complexity as software grows is to connect small functions.\nSuppose we have one function that doubles a number and another that converts a number to a string. Applying them in sequence gives us a function that takes a number and returns a string representing twice that number. Whenever the output of the first function can serve as the input to the second, we can combine the two into a new function that performs a larger operation. In mathematics, this is called function composition, written $g \\circ f$.\nf : A → B g : B → C g ∘ f : A → C In terms of types, we can combine two functions when the return type of the first matches the input type of the next. The function $g \\circ f$ first applies $f$ to an input $a$, then applies $g$ to the result.\nWhat is interesting here is that as long as the return type of $f$ matches the input type of $g$, we can connect the computations regardless of how either function is implemented. In our example, we could replace $f$ with a function that squares a number or adds one to it. Any function that returns a number can compose with our number-to-string function $g$. If we also have a function $h$ that encodes a string as UTF-8 bytes, we can continue composing: $h \\circ g \\circ f$.\nWhen we write real programs, however, we find that this assembly does not always go so smoothly.\n2. Missing output: when composition\u0026rsquo;s types no longer match The usual mathematical function $f : A \\to B$ returns some $b \\in B$ for every input $a \\in A$. Think of $\\sin x$, $e^x$, or $x^2$. Such a function is called a total function.\nIn programs, however, we frequently encounter situations in which there is no output value: no corresponding $b$ is available for a given $a$.\nImagine writing a function that parses a user-provided string as a number. For the input \u0026quot;42\u0026quot;, it can return the number $42$, but for \u0026quot;hello\u0026quot;, it cannot return a valid number. Similarly, a lookup for a user account has no account information to return when the supplied ID is unregistered. A function whose output is undefined for some inputs is called a partial function.\nCan we express a computation that cannot obtain a B for some inputs—a partial function—as a function defined for every input—a total function? Let us change one assumption: what if the absence of a value were itself represented by a value?\nTo do this, we change the return type from B to Option B. A value of Option B is either some b, containing a b : B, or none, representing the absence of a value. Even when there is no B to return, we can return none to express that fact. Python\u0026rsquo;s B | None, C++\u0026rsquo;s std::optional\u0026lt;B\u0026gt;, and Haskell\u0026rsquo;s Maybe B serve similar purposes. (Try looking up how your favorite language represents absence, too!)\nWe now define the function to return some b if it obtains a value for input a, and none otherwise. Every input has a corresponding Option B value, so this function has type A → Option B and is total. Option alone, however, does not tell us why the value is absent.\nLet us connect two stages of such a computation. First, we parse an input string as a natural number; then we take that number\u0026rsquo;s reciprocal. With \u0026quot;42\u0026quot;, we can complete both stages. With \u0026quot;hello\u0026quot;, the first stage cannot obtain a number. With \u0026quot;0\u0026quot;, parsing succeeds, but the second stage cannot obtain a reciprocal.\nCall the first function $f$ and the next one $g$. Since either may fail to obtain a value, their types are:\nf : A → Option B g : B → Option C The problem appears when we try to compose them as before.\ng (f a) -- Compile error: Option B cannot be used where B is required $f$ hands us an Option B—a box (or a burrito) accounting for possible absence—while $g$ requires a $B$ value.\nWe therefore cannot directly define ordinary composition $g \\circ f$ for $f : A \\to \\text{Option } B$ and $g : B \\to \\text{Option } C$.\nHow can we compose functions that may produce no value? How do programmers usually handle this mismatch?\n3. Connecting two computations with branching We cannot use $g \\circ f$ directly, but the behavior we want from the connection is clear. If $f\\,a$ is none, the overall result should be none. If it is some b, we should pass that $b$ to $g$ and use $g\\,b$ as the result. For now, let us write this new composition as $g \\star f : A \\to \\text{Option } C$.\nHere is how we can write it using Lean\u0026rsquo;s pattern matching. We will add one more stage, $h : C \\to \\text{Option } D$.\ndef runOption {A B C D : Type} (f : A → Option B) (g : B → Option C) (h : C → Option D) (a : A) : Option D := match f a with | none =\u0026gt; none | some b =\u0026gt; match g b with | none =\u0026gt; none | some c =\u0026gt; h c This code computes exactly the result we want. If $f\\,a$ is none, it stops at the first branch; if it is some b, it runs $g\\,b$. If $g\\,b$ is also none, it stops, and only if it is some c does it continue to $h\\,c$. This is enough to write a single pipeline.\nBut what the branches do is independent of the particular computations performed by $f$, $g$, and $h$. We have written a connecting rule directly inside runOption: stop if the previous result is absent; otherwise, pass it to the next function. We need the same rule when connecting other functions or adding more stages.\nOur goal goes beyond shortening this particular runOption: we want a method of composition that takes $f$ and $g$ and produces another function of type $A \\to \\text{Option } C$. This requires extracting the repeated branching at call sites into an operation between two functions. Let us first see whether an existing operation can do the job.\n4. Nested contexts: why map alone is not enough Developers familiar with functional programming might ask:\n“Doesn\u0026rsquo;t map apply a function to the value inside a container? Couldn\u0026rsquo;t we use (f a).map g?”\nHere, $α$ and $β$ each stand for an arbitrary type. Option.map takes a function of type $α \\to β$ and applies it to the value inside an Option α. Its type and behavior in the two cases are:\nOption.map : (α → β) → Option α → Option β Option.map g none = none Option.map g (some b) = some (g b) Since $g : B \\to \\text{Option } C$, applying it inside some b gives some (g b). This wraps Option C in another layer, producing Option (Option C). Applying it to the result of f a gives:\nf a : Option B g : B → Option C (f a).map g : Option (Option C) For example, if $f\\,a = \\text{some }0$ and $g\\,0 = \\text{none}$, then (f a).map g is some none, rather than none. A value is present, but that value is itself “absent.” The two layers retain the fact that the first computation produced a value while the second did not.\nWe wanted a single Option C result: a value is either present or absent. What we have instead is Option (Option C), one wrapper inside another.\nIt is like opening a box to find another box inside, as with Russian nesting dolls. map takes the inner $B$ and passes it to $g$, but does not remove the new wrapper ($\\text{Option } C$) that $g$ itself produces.\nNor can we directly apply the next operation, $h : C \\to \\text{Option } D$, to this result. The value inside the outer Option is an Option C, whereas $h$ needs a $C$.\n5. Building a composition rule with join and bind We have just obtained an Option (Option C). To return to the Option C we wanted, we need to remove the outer Option layer. If the outer value is none, return none; if it is some result, return the inner result unchanged. This operation has type Option (Option C) → Option C and is called Option.join in Lean. Because it reduces a nested context to one layer, its behavior is often described as flattening. We will use its operation name, join.\nLet us apply join to the map result from Section 4. If $f\\,a$ is none, $g$ is not run and the result is none. If $f\\,a$ is some b, map produces some (g b), and join removes the outer some to return $g\\,b$.\n((f a).map g).join : Option C if f a = none, the result is none if f a = some b, the result is g b The two branches we wrote explicitly in Section 3 have reappeared in this expression. In particular, when $g\\,b$ is none, map produces some none, but join turns it into none. If we open the outer doll and find that the inner one is empty, the final result is simply “empty.”\nCombining map, which applies a function to an inner value, with join, which removes one layer of nesting, gives an operation commonly called flatMap or bind. You will encounter it as Option.bind in Lean, flatMap in Java and Swift, and and_then in Rust and C++. In our code, we will use Lean\u0026rsquo;s name, bind.\nWe can express the relationship between the operations as follows.\nm.bind g = (m.map g).join If m is none, both sides produce none. If it is some b, bind returns g b directly. On the right, map produces some (g b), which join reduces to g b. Thus the two computations produce the same result for any Option value m and next function g.\nHere, m : Option B and g : B → Option C. bind provides both steps together, while join removes one layer from an already nested Option. Later, in the formal definition of a monad, we will encounter this relationship again in the general form bind m g = join (map g m).\nOption.bind : Option B → (B → Option C) → Option C composeOption (f : A → Option B) (g : B → Option C) : A → Option C composeOption f g a = Option.bind (f a) g Option.bind connects an existing Option B value to the next function. Using it, we define composeOption, an operation that takes two functions and produces a new function. For each input $a$, it runs $f$, then connects the result to $g$ using bind. The composition we previously wrote as $g \\star f$ is exactly composeOption f g.\nNotice that the result of composition is itself a function of type A → Option C. If another function h : C → Option D follows, we can connect this result to h using the same operation.\ncomposeOption f g : A → Option C composeOption (composeOption f g) h : A → Option D Where we previously wrote another branch for each additional function, we can now reuse the composition operation we have already built. Each function handles its own computation; composeOption handles stopping when a value is absent and passing it to the next function when it is present.\n6. Checking composition in Lean 4 We will now define the three ways of connecting functions in Lean 4 and compare their results on representative inputs.\nWe will define parseNat, which parses a string as a natural number, and reciprocal, which represents the reciprocal of a nonzero natural number as a string of the form \u0026quot;1/n\u0026quot;. The example focuses on absence and function connections, rather than numerical operations on fractions.\n-- Monads and Category Theory ① When Functions Do Not Compose -- Lean 4.32.1. Runs without additional libraries. -- 1. Two functions that may produce no value def parseNat (s : String) : Option Nat := s.toNat? def reciprocal (n : Nat) : Option String := if n == 0 then none -- No reciprocal is represented for zero else some s!\u0026#34;1/{n}\u0026#34; We define the connecting rule as composeOption. For comparison, we will also write out the branching from Section 3 and the map followed by join from Section 5.\n-- Call the next function according to the previous result def composeOptionByMatch {A B C : Type} (f : A → Option B) (g : B → Option C) : A → Option C := fun x =\u0026gt; match f x with | none =\u0026gt; none | some b =\u0026gt; g b -- Express the same rule using Option.bind def composeOption {A B C : Type} (f : A → Option B) (g : B → Option C) : A → Option C := fun x =\u0026gt; (f x).bind g -- The same rule: apply map, then flatten one layer with join def composeOptionByJoin {A B C : Type} (f : A → Option B) (g : B → Option C) : A → Option C := fun x =\u0026gt; ((f x).map g).join Now let us compose the two functions.\ndef parseAndReciprocal : String → Option String := composeOption parseNat reciprocal parseAndReciprocal specifies only which two functions to connect. composeOption handles stopping when the previous result is absent and passing it to the next function when it is present. We can use the same rule unchanged to connect other functions.\n#eval is a Lean command that executes the expression after it and displays the result. We can use it to inspect an input that succeeds, along with inputs for which a value is absent at either stage.\n#eval parseAndReciprocal \u0026#34;42\u0026#34; -- some \u0026#34;1/42\u0026#34; #eval parseAndReciprocal \u0026#34;0\u0026#34; -- none (No reciprocal is represented for zero) #eval parseAndReciprocal \u0026#34;foo\u0026#34; -- none (Parsing failed) #eval (parseNat \u0026#34;0\u0026#34;).map reciprocal -- some none (Two layers before join) Having inspected the results with #eval, we will now use #guard to check whether they satisfy the conditions we expect. #guard passes without an error when the condition after it is true, and makes Lean report an error when it is false. Here, == compares two computed results for equality.\nThe first line below checks whether composeOptionByMatch, implemented with branching, and parseAndReciprocal, implemented using bind, produce the same result for the input \u0026quot;42\u0026quot;. The remaining lines similarly compare against another implementation or an expected result. These are some of the checks; the Playground contains the full set for all three inputs.\n#guard composeOptionByMatch parseNat reciprocal \u0026#34;42\u0026#34; == parseAndReciprocal \u0026#34;42\u0026#34; #guard composeOptionByJoin parseNat reciprocal \u0026#34;0\u0026#34; == parseAndReciprocal \u0026#34;0\u0026#34; #guard parseAndReciprocal \u0026#34;foo\u0026#34; == none To modify the code and run it yourself, open this example in the Lean 4 Playground.\ncomposeOptionByMatch explicitly handles none and some b, while composeOptionByJoin applies map followed by join. Option.bind expresses the same connection in one line. The #guard checks comparing these implementations on representative inputs verify examples; they do not prove equality for every input.\nWith \u0026quot;42\u0026quot;: parsing succeeds ($\\text{some } 42$) $\\to$ a fraction representation is produced ($\\text{some } \u0026ldquo;1/42\u0026rdquo;$). With \u0026quot;0\u0026quot;: parsing succeeds ($\\text{some } 0$) $\\to$ the second function returns none. With \u0026quot;foo\u0026quot;: parsing fails ($\\text{none}$) $\\to$ the second function is never run, and $\\text{none}$ is returned immediately. We handled absence inside the composition rule, composeOption, without manually checking it at every call. join flattens both an outer none and a some none whose inner value is absent to none. Consequently, the final result alone cannot tell us which stage failed to produce a value.\nAn aside: proving an equation in Lean So far, we have run specific inputs and checked their results. With Lean, we can go a step further and prove that two computations agree for any input and function. Let us prove m.bind g = (m.map g).join, the equation we checked by considering both cases in Section 5.\nThis uses the interactive theorem prover capability introduced earlier. In Lean, we can write both a claim and the reasoning that establishes it. As we write the steps of a proof, Lean checks that each step is valid and tells us what remains to be proved. In the code below, example declares the claim to prove, and the commands after by construct its proof.\nexample {B C : Type} (m : Option B) (g : B → Option C) : m.bind g = (m.map g).join := by cases m \u0026lt;;\u0026gt; rfl cases m splits the proof into the cases where m is none and where it is some b. \u0026lt;;\u0026gt; applies the following rfl to both cases. rfl checks that both sides of the equation reduce to the same expression according to their definitions. Since both cases are proved without choosing a specific b or g, Lean can verify that the equation holds for arbitrary m and g.\n7. The next question: laws of the composition rule At first, we could not pass the Option B returned by the first function directly to the next. We can now express a rule in composeOption: stop when a value is absent, or pass it to the next computation when it is present. The rule remains reusable even when the functions change. The monads we will study bring together operations that connect computations in this way and the laws those operations must obey.\nWill this rule give consistent results across multiple stages? We need to check whether either grouping of three functions gives the same result, and whether a function that simply wraps its input in some leaves composition unchanged.\nIn the next article, we will build a composition for List, where a single input can produce multiple results. Comparing these two kinds of computation—absence and multiple results—will help us see what their compositions have in common. We will then ask whether matching the types of a composition is enough, and prove in Lean 4 that our Option composition satisfies associativity and the identity laws.\n","date":"2026.09.25","permalink":"https://hyperaccel.ai/techblog/en/posts/composition-and-category/","series":["Monads and Category Theory Starting from Code"],"style":"--c:#5BC5F6;--on:#02121c","summary":"Starting with a type mismatch between functions that return Option, we explore how branching, map, join, and bind express the same computation.","title":"[Monads and Category Theory - Part 01] When Functions Do Not Compose"},{"authors":["Jaewon Lim"],"category":"AI Hardware","content":" OpenAI, the company that first brought LLMs to market, has now used an LLM to build a chip.\nAnd apparently, it has built a chip better than NVIDIA\u0026rsquo;s.\nIn this installment, we look at Jalapeño, the LLM inference accelerator OpenAI unveiled at Hot Chips 2026.\nThe Jalapeño Shock In the early hours of August 26, I was scrolling through LinkedIn before bed when a post kept me awake for the rest of the night.\nSemiAnalysis, a semiconductor research firm, had published performance results for Jalapeño, the chip jointly developed by OpenAI and Broadcom.\nI already knew that OpenAI was developing a chip, so I had been imagining what it might look like while waiting for the actual silicon to arrive. What was finally revealed went far beyond anything I had imagined.\nA General-Purpose LLM Accelerator, Not a GPT-Specific Chip My initial assumption was that OpenAI would build a chip optimized for serving its own models so that it could construct AI data center infrastructure beyond the GPU monopoly. Other startups had already gone beyond domain-specific LLM inference accelerators and begun building accelerators specialized for individual models. Taalas, a startup recently acquired by AMD, demonstrated this approach by effectively “hardwiring” the weights of Llama 3.1 into read-only memory (ROM), whose high read performance maximizes inference throughput.\nFigure 1. Per-user token throughput comparison between the Taalas HC1 and leading AI accelerators. Source: Taalas\nMoonshotAI\u0026rsquo;s recently released Kimi K3 technical report also included a notable hardware-related detail. Kimi K3 had fully autonomously verified the prototype RTL of a Kimi-specific accelerator, commonly called the KPU, built experimentally by MoonshotAI, and had taken it through layout. I looked into the prototype and found that it structurally incorporated modules specialized for the attention mechanisms used by Kimi K3, including Kimi Delta Attention and Multi-Head Latent Attention.\nAlthough it was only an experimental prototype, these developments made me think that OpenAI might also build an accelerator with an architecture specialized for its own models. A newly developed first-generation chip was unlikely to outperform the GPUs serving frontier models, so the likely strategy seemed to be serving older or relatively lower-performance model architectures on accelerators specialized for those models.\nThen Jalapeño\u0026rsquo;s performance was unveiled in both the SemiAnalysis report and the Hot Chips presentation held that same day, and I could not hide my surprise. SemiAnalysis is not only a research firm; it also benchmarks serving performance on the latest GPUs from NVIDIA, AMD, and others. These benchmarks use open-weight models such as Llama and Chinese models. SemiAnalysis published Jalapeño\u0026rsquo;s results from the same benchmark suite. OpenAI had not merely optimized a chip for its internal GPT models. It had built an accelerator that performed well as a general-purpose platform across LLMs.\nPerformance Better Than GPUs Figure 2. Throughput-latency frontier comparison between Jalapeño and leading GPUs. Source: SemiAnalysis\nIts performance is as remarkable as its versatility. Across the reported metrics, Jalapeño delivered 1.5–4 times the performance of the GB300, the fastest NVIDIA product currently in customer deployments, and surpassed the not-yet-commercially-released Vera Rubin in performance per watt. Even more surprising, the software used for the benchmark had not yet been fully optimized. That makes sense: OpenAI\u0026rsquo;s original objective was not to sell its chip on the market but to install it in its own infrastructure. It therefore had little reason to tune benchmark results for publicity. Even so, Jalapeño outperformed the best current-generation GPUs, whose performance has already been heavily optimized. Let us take a closer look at what that means.\nSTP (Single-Token Prediction) vs. MTP (Multi-Token Prediction) One of the biggest limitations of LLMs is that they emit response tokens one at a time, a process known as single-token prediction (STP). Even if hundreds of thousands of input tokens are processed in parallel, response tokens still have to be generated sequentially, creating a bottleneck. To introduce more parallelism into response generation, speculative decoding has recently gained attention. A relatively small model generates multiple tokens at once through multi-token prediction (MTP), and the main model verifies them in parallel. What stands out in the benchmark is that Jalapeño outperformed a GB300 using MTP even though the benchmark model on Jalapeño did not use MTP. In performance per watt, it also posted a higher result than NVIDIA\u0026rsquo;s latest Vera Rubin product running MTP.\nDisaggregated Systems vs. Homogeneous Systems Figure 3. NVIDIA Dynamo\u0026rsquo;s PDD configuration, which separates prefill and decode. Source: NVIDIA\nOne of the biggest recent trends in LLM inference serving optimization is prefill-decode disaggregation (PDD). The process of producing tokens in an LLM can be broadly divided into two stages. The prefill stage processes the initial prompt, while the decode stage feeds the first generated token back into the model and then produces subsequent tokens one by one. Because these stages have different computational characteristics, PDD seeks to improve efficiency by separating accelerator clusters designed for prefill from those optimized for decode.\nFigure 4. An AFD configuration that splits attention and FFN computation between GPUs and LPUs. Source: NVIDIA\nNVIDIA has also announced plans for attention-FFN disaggregation (AFD), which breaks the prefill and decode stages down even further. Using LPX clusters built with the Groq LPU introduced in the previous installment, attention computation runs on GPU racks while feed-forward network (FFN) computation, which uses fixed weights, runs on LPX clusters.\nHowever, these disaggregated approaches introduce several serious problems.\nFirst, splitting computation across different racks incurs inter-node communication costs. The KV cache is especially problematic because it occupies so much capacity. PDD provides a straightforward example: the KV cache generated during prefill must also be used during decode. As the input sequence grows longer, the KV cache grows with it, increasing the cost of moving that data.\nAnother problem is that clusters become fixed around optimization for either prefill or decode. A cluster will likely be designed after determining an optimal configuration for an expected workload, but there is no guarantee that only the expected workload will arrive. An unforeseen workload can make a supposedly optimal cluster configuration suboptimal.\nOpenAI: Why bother with disaggregation? If we can maximize efficiency for both prefill and decode, isn\u0026rsquo;t that enough?\nFigure 5. Data movement and resource allocation in heterogeneous and homogeneous systems\nOpenAI directly challenges the conventional approach to overcome these drawbacks. Its argument is that disaggregation is unnecessary if the system can fully utilize its performance. By allocating resources optimally for each incoming workload, even a homogeneous system can maximize efficiency.\nAn AI Chip Built by AI Now that we have looked at the performance, let us turn to the development process. Large fabless technology companies devote hundreds or even thousands of people to chip development, but OpenAI\u0026rsquo;s initial chip team is estimated to have had only about 20 members. They were undoubtedly some of the world\u0026rsquo;s best engineers, but delivering the company\u0026rsquo;s first chip within this timeline with a team of that size is still an extraordinary achievement. Hardware development took nine months, and it reportedly took less than three more months after receiving the chip to get Codex and ChatGPT running on it. Unsurprisingly, OpenAI says its most powerful tool—AI—played a major role. Let us look at how.\nHardware Optimization The development of a digital circuit such as an AI accelerator can be broadly divided into three stages: hardware architecture design -\u0026gt; RTL design -\u0026gt; physical layout. Because register-transfer level (RTL) code describes the register-level organization that determines the chip\u0026rsquo;s physical layout, most performance (Performance), power (Power), and area (Area) metrics—collectively known as PPA—are determined at the RTL level. Once the architecture is fixed, engineers therefore need to optimize the RTL for PPA, and this is one of the core capabilities of a hardware engineer. RTL is written in languages such as Verilog and SystemVerilog. Today\u0026rsquo;s AI models have learned these languages well enough that AI-assisted RTL optimization is already common, but the Jalapeño team took a somewhat different approach.\nHLS (High-Level Synthesis) Figure 6. The XLS hardware synthesis tool used by the Jalapeño team\nHigh-level synthesis (HLS)—and specifically XLS, which the Jalapeño team used—does not involve writing SystemVerilog directly. Instead, hardware is described in C code, one level of abstraction above RTL, and the RTL code is then “generated” from it.\n???: Why go through C instead of optimizing the RTL directly?\nI had the same question. For a human designer, generating RTL from C may be more convenient. When designing alongside AI, however, the RTL itself could be supplied directly as input, seemingly allowing the system to optimize it without the extra step. In its presentation, OpenAI argued that “concepts that are easier for humans to understand are also easier for AI to understand,” making optimization easier as well. XLS being a language designed for real hardware development likely contributed, but I found the idea that something easier for AI to understand is also easier for it to optimize particularly striking.\nSoftware Optimization OpenAI says it also used AI to perform fully automated software optimization. A new chip needs a software stack tailored to it, and that stack must provide the kernels required to run LLMs on the hardware. When measuring the DeepSeek R1 benchmark, OpenAI had not yet performed any optimization with human intervention, so it handed the entire process over to AI. The AI reportedly succeeded in carrying out most of the optimization on its own.\nThe Self-Improvement Loop The two optimization processes shared a common pattern: turn optimization into a loop, then have AI pass through that loop repeatedly to improve the result. This appears to be an application of Andrej Karpathy\u0026rsquo;s idea of verifiability to chip performance optimization.\nWhen a task is verifiable, it can be optimized directly or through reinforcement learning, allowing a neural network to become exceptionally good at it.\n— Andrej Karpathy, Verifiability\nIn other words, the optimization process becomes a verifiable loop through which AI repeatedly improves its own result. OpenAI claims this approach produced greater performance gains than human experts could achieve in both hardware and software optimization.\nWhat About Anthropic? While researching this article, I also found reports that Anthropic has begun developing chips. News had already emerged months earlier that a key member of the Jalapeño project had moved to Anthropic, followed by reports that engineers who had worked on TPU development had joined the company. AI companies are now making a serious push to build their own chips and infrastructure. I expect frontier AI companies in China to make similar moves.\nConclusion GPUs have dominated the AI semiconductor market through the moat of CUDA. But that moat now appears to be eroding little by little. Or perhaps another moat is being built in its place: “intelligence.”\nFrontier AI companies do not immediately release their highest-performing models to the public. For safety and other reasons, they may test models internally, provide them only to selected companies as Anthropic did with Fable 5, or release models to consumers with certain capabilities restricted. If so, frontier AI labs hold an exclusive position: they can use “advanced intelligence” earlier than anyone else, at the greatest “speed,” and “without refinement.” AI has already surpassed human capability in nearly every field. In this environment, early exclusive access to intelligence may already be building an invisible moat across every knowledge-intensive industry. I do not think I will fall asleep easily tonight either. I will close with a passage I shared on LinkedIn. Thank you, as always, for reading.\nWhen AI came for the translators,\nI remained silent.\nI was not a translator.\nThen it came for the artists,\nand I remained silent.\nI was not an artist.\nThen it came for the developers,\nand I remained silent.\nI was not a developer.\nWhen it came for me,\nno one was left\nto speak for me.\n— Adapted from Martin Niemöller\u0026rsquo;s “First They Came”\nSources SemiAnalysis - OpenAI Jalapeño: Better Than Nvidia Blackwell OpenAI - You Can Just Build Things … Chips (Hot Chips 2026) Taalas - HC1 Technology Demonstrator Moonshot AI - Kimi K3: Open Frontier Intelligence NVIDIA Technical Blog - NVIDIA Accelerates OpenAI gpt-oss Models Delivering 1.5 M TPS Inference on NVIDIA GB200 NVL72 NVIDIA Technical Blog - Inside NVIDIA Groq 3 LPX Google XLS - XLS: Accelerated HW Synthesis Andrej Karpathy - Verifiability ","date":"2026.09.08","permalink":"https://hyperaccel.ai/techblog/en/posts/jalapeno-shock/","series":["Know Your Enemy, Know Yourself"],"style":"--c:#00C2A8;--on:#02121c","summary":"The era of using AI to build chips that run AI has arrived. Drawing on publicly released information about Jalapeño, OpenAI\u0026rsquo;s custom chip, this article explores what that means.","title":"[Know Your Enemy - Part 07] A Model Trained on GPUs Builds a Better Chip Than GPUs"},{"authors":["Jaewon Lim"],"category":"Memory \u0026 Storage","content":" This is Part 5 of the Memory in the AI Era series. Part 1, Part 2, and Part 3 covered High Bandwidth Flash (HBF), which fills a gap in the memory hierarchy next to the GPU. Part 4 shifted to the system level and examined what Compute Express Link (CXL) is. This fifth and final part looks at how CXL is used in real workloads.\nIntroduction Hello, I\u0026rsquo;m Jaewon Lim, an RTL verification engineer at HyperAccel.\nIn Part 4, we broke down the CXL interface: the gap between DDR and PCIe, the three sub-protocols (CXL.io / CXL.cache / CXL.mem), Type 1/2/3 devices, and the CMM product families from the three major memory vendors.\nBut I left the most practical question for this installment.\nHow is CXL used in real LLM serving, and which workloads suit it?\nOnce we understand the shape of a tool, the next step is to find where it actually fits.\nOur starting point is the pair of fundamental properties of CXL discussed in Part 4.\nBenefit: It can extend a large, coherent memory space beyond local memory and distribute capacity among multiple hosts. Cost: It is 2-3 times slower than directly attached DDR memory, with latency around 170-300 ns. This is the latency tax. These properties are two sides of the same coin. The question \u0026ldquo;Where does CXL fit?\u0026rdquo; therefore becomes: \u0026ldquo;Which workloads need capacity and pooling badly enough to absorb this latency tax?\u0026rdquo;\nWe will answer that question through three use cases: 1) KV cache offload, 2) VM consolidation through memory pooling, and 3) hot/warm/cold tiering.\nThis post is based on my personal study and experience. If you find any errors, please let me know in the comments.\nUse Case 1 — KV Cache: Memory Forced Out of the GPU The first role CXL is targeting in LLM serving is as a home for KV cache.\nWhy KV Cache Becomes a Problem During LLM inference, the model stores and reuses the Key and Value vectors of all previously generated tokens. This Key-Value cache (KV cache) grows linearly with sequence length and the number of concurrent users.\nIts total size can be calculated as follows.\n$$\\text{KV cache size} = 2 \\times B \\times S \\times L \\times H_{kv} \\times D_h \\times P$$\nHere, $B$ is the batch size, $S$ is the sequence length, and $L$ is the number of layers. $H_{kv}$ is the number of KV heads, $D_h$ is the head dimension, and $P$ is the number of bytes per element. The leading 2 accounts for both Key and Value.\nFor example, serving Llama 3.1 405B with FP8 weights and an FP16 KV cache requires about 516 KB per token. A single user with 100,000 tokens consumes roughly 48 GB. Serving 128 such requests concurrently pushes the KV cache alone to about 6 TB. We covered the calculation in more detail in our earlier NVIDIA ICMS post.\nKV cache size calculation for Llama 3.1 405B. Source: LMCache KV Cache Size Calculator\nThe problem is that this KV cache occupies GPU HBM, the most expensive memory in the system. HBM capacity is limited, and every byte consumed by KV cache reduces the number of requests the GPU can process concurrently.\nLimits of Existing Approaches The industry already moves KV cache out of the GPU. A common approach is to offload it to host DRAM or NVMe SSD and bring it back when needed.\nHowever, GPU HBM, host DRAM, and SSD occupy separate allocation domains. The serving runtime must explicitly track each KV block and manage its movement.\nHost DRAM requires DMA and staging when exchanging data with the GPU. NVMe SSD adds block I/O and substantially higher latency. In either case, software must decide when to move the required KV block back into HBM.\nWhat Changes with CXL? CXL Type 3 memory enters the host\u0026rsquo;s physical address space. The CPU can access it through ordinary loads and stores without block I/O. This provides three benefits.\nCapacity: A KV cache space much larger than HBM can be attached to the host address space. Access model: The CPU can treat CXL memory as byte-addressable memory. Position in the hierarchy: It is a DRAM tier closer than SSD and does not require block I/O. A public example is SK hynix\u0026rsquo;s TraCT. TraCT attaches CXL shared memory to NVIDIA Dynamo-vLLM and uses it both as a KV transfer space between prefill and decode GPUs and as a rack-level prefix cache. In a two-server evaluation, TraCT improved average Time to First Token (TTFT) by up to 9.8 times and peak throughput by up to 1.6 times.\nKV cache transfer and prefix caching with CXL shared memory. Source: TraCT, Figure 2\nUse Case 2 — Memory Pooling and VM Consolidation The second use case is memory pooling, which can directly improve datacenter economics.\nThe Stranded Memory Problem Cloud servers bind a fixed number of CPU cores to a fixed amount of DRAM. Once customers rent all available CPU cores, the provider cannot sell the remaining DRAM separately. Conversely, when memory is exhausted first, the remaining CPU cores sit idle.\nThis is stranded memory: one resource is exhausted while another remains unused. The problem is especially visible in cloud environments that place multiple Virtual Machines (VMs) on each server.\nHow CXL Pooling Addresses It CXL 2.0 pooling, introduced in Part 4, targets exactly this problem. It divides a large memory device into Multi-Logical Devices (MLDs), and a Fabric Manager (FM) dynamically assigns chunks according to each node\u0026rsquo;s demand.\nThe key is dynamic allocation rather than simultaneous sharing. At any moment, a chunk belongs to only one host, avoiding coherence conflicts and keeping the implementation relatively simple.\nWith this structure, every node no longer needs enough DRAM for its individual peak demand. Nodes can operate with less local DRAM and borrow only the missing capacity from the pool. The design benefits from the fact that all nodes are unlikely to reach their peak at the same time.\nThe VM scheduler can also place CPU and memory resources more independently.\nMicrosoft Azure\u0026rsquo;s Pond combined 75 days of VM placement traces from 100 Azure production clusters with performance measurements from 158 workloads running under emulated CXL latency. In an end-to-end simulation with memory latency at 2.22 times that of local DRAM, a 16-socket pool, and a 5% performance degradation target, Pond reduced total required DRAM capacity by 7%. The paper translates this into a 3.5% reduction in total cloud server cost.\nAzure stranded memory and DRAM savings by pool size. Source: Pond, Figures 2 and 3\nUse Case 3 — Hot / Warm / Cold Tiering The third use case is an operating model that connects the first two. Instead of treating CXL memory as a replacement for main memory, it places CXL one tier below it.\nWhy Tiering? CXL memory is 2-3 times slower than DDR, so placing all data in CXL memory would be counterproductive. Instead, data is separated into tiers according to access frequency.\nHot: frequently accessed data → nearby DDR Warm / cold: infrequently accessed data or data that mainly consumes capacity → more distant CXL memory From the CPU\u0026rsquo;s perspective, CXL memory appears much like a \u0026ldquo;slower Non-Uniform Memory Access (NUMA) node.\u0026rdquo; The OS and runtime therefore manage which pages belong in each tier.\nAutomatic Tiering vs. Explicit Placement Who manages tiering depends on the range of workloads the server supports.\nFor general-purpose workloads, automatic OS tiering is convenient. Linux tracks page accesses, promotes hot pages to DDR, and demotes cold pages to CXL memory. For workloads with deterministic access patterns, such as LLM inference, the application can place data explicitly. The component that understands the data can decide, for example, that \u0026ldquo;active KV blocks stay in HBM, while reusable prefixes go to CXL.\u0026rdquo; There is no universal definition of \u0026ldquo;access tracking\u0026rdquo; for automatic tiering. Implementations combine recency, access frequency, and idle time in different ways. For example, with its default settings, Linux DAMON_LRU_SORT classifies a memory region as hot when it is accessed during at least 50% of the observation interval, and as cold when it has not been accessed for 120 seconds. It then raises the LRU priority of hot pages and lowers that of cold pages so that cold pages are reclaimed first under memory pressure.\nTPP turns that classification into movement between tiers. It asynchronously demotes cold reclaim candidates from local DDR to the CXL NUMA node. When a page in CXL memory is accessed, TPP first moves it to the active LRU list and promotes it to local DDR only if it remains hot at the next NUMA hinting fault. Requiring repeated evidence of hotness reduces page ping-pong caused by one-off accesses.\nVistara — CXL Tiering Validated in Production Meta\u0026rsquo;s Vistara is a representative example of automatic tiering. Vistara connects DDR4 recovered from retired servers through a CXL Type 3 device. One AMD Turin server combines 768 GB of local DDR5 with 256 GB of CXL-attached DDR4 for a total of 1 TB.\nLinux exposes CXL memory as a CPU-less NUMA node. Transparent Page Placement (TPP) and Transparent Memory Offloading (TMO) keep hot pages in local DDR5 and move cold pages to CXL-attached DDR4. This is memory expansion plus tiering within a single host.\nVistara MemServer combining local DDR5 with CXL-attached DDR4. Source: Vistara, Figure 6\nMeta showed that CXL can produce practical improvements at the server level. Enabling CXL increased the number of jobs or VMs per server by 33% in CI and development environments. For an ML workload, it reduced the number of required servers by 25% while increasing throughput by 4%.\nWhich Workloads Suit CXL? The three use cases reveal a common set of conditions. CXL is well suited to workloads with the following characteristics.\nCapacity is the bottleneck, rather than latency. The workload contains data that can tolerate higher latency, such as cold data or a large working set. The access pattern is predictable enough for explicit placement. By contrast, hot data on the critical path of every access, such as active KV blocks and weights used by attention at every step, still belongs in HBM or DDR. CXL does not replace those tiers. It expands the tier below them.\nThis is where HBF and CXL converge. HBF increases capacity between HBM and SSD next to the GPU. CXL expands capacity below local DRAM beside the CPU and across the system. Their physical positions and latencies differ, but both follow the same philosophy: place a slower but larger tier below fast, expensive memory.\nConclusion Part 5 can be summarized in three points.\nThree use cases: KV cache offload / VM consolidation through memory pooling / hot-warm-cold tiering Common conditions: capacity-bound, tolerant of additional latency, and predictable access patterns CXL\u0026rsquo;s role: expanding the tier below expensive memory rather than replacing it This series began by asking why SRAM, DRAM, HBM, and NAND cannot be collapsed into a single type of memory. The answer remained the same throughout: no single memory technology can maximize speed and capacity while minimizing cost.\nHBF and CXL do not eliminate that trade-off. Instead, they keep immediately needed data in fast, expensive memory and place data needed later in slower, larger memory. HBF expands the tier near the GPU, while CXL expands the tier beside the CPU and at the system level.\nHBF and CXL Memory expand capacity in the memory hierarchy · AI generated image\nUltimately, no single memory can solve every problem. We must choose memory according to the characteristics of the data and combine tiers to meet each workload\u0026rsquo;s performance and capacity requirements.\nThat is because engineering has no single correct answer. It is a process of finding the best trade-off within the technology and budget available at the time, then adjusting it whenever new workloads and technologies emerge.\nAcross five parts, we examined why different kinds of memory exist and which problems new technologies are designed to solve. This concludes the Memory in the AI Era series. I hope it helped you better understand memory technology. Thank you for reading.\nP.S. As the memory hierarchy becomes more diverse, accelerator companies face increasingly complex and interesting problems. Building an optimized accelerator requires memory and compute logic, software, and algorithms to work together rather than remain isolated in separate domains.\nHyperAccel works across hardware, software, and AI, bringing together talented people in every area. If you want to deepen a broad range of knowledge and grow with us, we would be glad to hear from you.\nCareers: https://hyperaccel.career.greetinghr.com/en/guide\nReferences Compute Express Link Consortium — Specifications Linux Kernel — CXL Driver Documentation D. Yoon et al., \u0026ldquo;TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale,\u0026rdquo; 2025. arXiv:2512.18194 H. Li et al., \u0026ldquo;Pond: CXL-Based Memory Pooling Systems for Cloud Platforms,\u0026rdquo; ASPLOS, 2023. DOI: 10.1145/3575693.3578835 Linux Kernel, \u0026ldquo;DAMON-based LRU-lists Sorting\u0026rdquo; H. Al Maruf et al., \u0026ldquo;TPP: Transparent Page Placement for CXL-Enabled Tiered-Memory,\u0026rdquo; ISCA, 2023. arXiv:2206.02878 N. Gholkar et al., \u0026ldquo;Vistara: Making CXL Real—Full Path from ASIC Design and OS Support to Hyperscale Deployment,\u0026rdquo; ISCA, 2026. Paper ","date":"2026.08.12","permalink":"https://hyperaccel.ai/techblog/en/posts/cxl-workload/","series":["Memory in the AI Era"],"style":"--c:#009EE2;--on:#02121c","summary":"We examine how CXL is used in real workloads through KV cache offload, memory pooling, and memory tiering.","title":"[Memory in the AI Era - Part 05] Exploring CXL Workloads"},{"authors":["Hyunjun Park"],"category":"Compilers \u0026 Kernels","content":"Introduction Hello! I\u0026rsquo;m Hyunjun Park from the CL (Compute Library) team at HyperAccel. We\u0026rsquo;re a startup building a chip called the LLM Processing Unit (LPU), purpose-built for Large Language Model (LLM) inference. Just as a GPU is driven through CUDA, the LPU is driven through Legato — an embedded Domain Specific Language (eDSL) built in-house by our compiler team. The CL team\u0026rsquo;s job is to write Legato kernels that squeeze the most out of the LPU architecture.\ntorch.compile() has been one of the hotter topics lately, and plenty of people report large speedups from it. Ask someone how it actually works, though, and the answer usually stops at \u0026ldquo;it captures a graph and fuses kernels.\u0026rdquo;\nOne reason PyTorch won so many people over is eager execution — you run one line at a time. That lowered the barrier to entry enormously. But paradoxically, that same choice means line n has no idea what comes after it. And that blind spot is exactly what blocks fusion and a whole family of other optimizations, which costs PyTorch a great deal of performance.\nSo PyTorch has long been looking for a way to keep Python\u0026rsquo;s character while recovering that performance: TorchScript, torch.jit.trace, torch.fx, and others. After many attempts, torch.compile() is what came out of that line of work.\nThere\u0026rsquo;s more here than fits in one post, so this is a three-part series. This first one takes torch.compile() apart piece by piece. The second will look at why torch.compile() is split into three stages in the first place, and who gains what from that split. The third will share what I took away from attending PyTorch Conference 2026.\nPart 0. Background What exactly is a \u0026ldquo;graph\u0026rdquo;? The word \u0026ldquo;graph\u0026rdquo; is going to keep coming up, so let\u0026rsquo;s pin it down first. Feel free to skip ahead if this is already familiar.\nA graph here is a picture of how operations depend on one another. Each operation becomes a node, and \u0026ldquo;the result of this operation feeds into that one\u0026rdquo; becomes an edge. A result never flows back into itself, so what you get is a directed graph with no cycles — a Directed Acyclic Graph (DAG).\nThe right side is just the left side rewritten. In code, values move around under names like a and b; in the graph, that same movement is spelled out as edges.\nThat difference is the whole point. While Python executes line by line, there is no way to know where the result of x @ w will end up next. The graph, on the other hand, has the entire flow written down at once. You can see up front that matmul\u0026rsquo;s result goes only to relu and nowhere else — which is what lets you decide whether the two can be handled together.\nThis is the answer to the problem from the introduction, the one where line n can\u0026rsquo;t see past itself. Which is why half of what torch.compile() does is simply getting its hands on this graph.\nWhat is an \u0026ldquo;FX graph\u0026rdquo;? PyTorch has its own data structure for holding that graph. It\u0026rsquo;s called an FX graph.\nThe format came out of an earlier attempt, torch.fx, and today the entire torch.compile() stack passes it around. From here on, when this post says \u0026ldquo;graph,\u0026rdquo; it almost always means this one.\nThere\u0026rsquo;s nothing exotic about it — think of it as the picture above written down in code. One operation is one node, and each node carries the list of nodes it depends on, which form the edges. What matters is that it\u0026rsquo;s an ordinary Python object: easy to walk over and easy to rewrite, which is what makes AOTAutograd\u0026rsquo;s clean-up passes possible later on.\nPart 1. The three components of torch.compile Three components Now that graphs are out of the way, let\u0026rsquo;s get into torch.compile() proper. It breaks down into three stages: TorchDynamo, AOTAutograd, and TorchInductor.\nHere\u0026rsquo;s the whole thing at a glance. First, at the front end, Dynamo takes user code and builds a graph. Building the graph means interpreting the Python interpreter itself — pulling operations defined in PyTorch out into graph form while contending with dynamic typing, data-dependent branches, calls into external libraries, and so on.\nAOTAutograd then tidies that FX graph up and passes it down to the backend. Tidying the graph means simplifying its representation without changing its meaning: collapsing the many ways of calling the same operation into one, stripping out side effects, and shrinking the variety of operations.\nFinally, in the backend, Inductor produces Triton or C++ code. Making kernels is where you deal with hardware: which operations to group, how to use memory, and which language to emit.\nHere\u0026rsquo;s what each component hands to the next:\nComponent Input Output TorchDynamo Python function FX graph + guards + residual bytecode AOTAutograd FX graph normalized ATen graph TorchInductor ATen graph Triton (GPU) or C++ (CPU) kernel source The FX graph in that table is the name of the data structure PyTorch actually stores the graph from Part 0 in. We\u0026rsquo;ll look at its structure in Part 2.\nAnd these three components are independently replaceable. torch.compile(backend=...) lets you drop in a compiler other than Inductor — which is exactly what we do with our Legato backend. Why that replaceability matters is the subject of the next post.\nPart 2. TorchDynamo: turning Python code into an fx.Graph TorchDynamo\u0026rsquo;s job is extracting a graph from Python code. It doesn\u0026rsquo;t actually compute anything — it just follows how values flow, and records tensor operations into a graph as it goes. What comes out is the FX graph from Part 0, and it\u0026rsquo;s what AOTAutograd tidies up and what Inductor turns into kernels.\nThis is where the destination of every operation\u0026rsquo;s result becomes visible. It\u0026rsquo;s precisely the information Python code couldn\u0026rsquo;t give us, and it\u0026rsquo;s the basis on which Inductor can later group operations together and fuse them.\nGraph breaks: what doesn\u0026rsquo;t fit in a graph Not all Python code becomes a graph. When Dynamo hits something it can\u0026rsquo;t handle, it cuts the graph right there. This is called a graph break. Two kinds of code account for most of them:\nControl flow that depends on tensor values — a branch like if x.sum() \u0026gt; 0: that needs the actual value to resolve Python behavior a graph can\u0026rsquo;t express — print, file I/O, network calls, and so on When a break happens, Dynamo compiles and runs the graph it has collected so far, hands control back to Python for the problematic part, and then starts extracting a fresh graph from there. So a single function can end up split across several graphs.\nThe catch is that every break introduces a context switch between compiled code and the Python interpreter, and — more importantly — a shorter graph means a smaller window to fuse within. The moment you split the graph, any optimization that would have crossed that boundary is gone. Which is why a good deal of real optimization work amounts to reducing graph breaks.\nPart 3. AOTAutograd: tidying up the captured graph The graph Dynamo hands over isn\u0026rsquo;t quite ready to become kernels, because its representation is still too user-facing. Dynamo\u0026rsquo;s graph keeps whatever form we wrote in our code. And in PyTorch there are several ways to call the same operation — as a method, as a function, as an operator. For someone writing a backend, that means implementing the same thing several times over.\nSo before kernels get made, the graph needs to be normalized once. That\u0026rsquo;s AOTAutograd\u0026rsquo;s job. The name comes from handling autograd Ahead of Time.\nCatching operations below the dispatcher It\u0026rsquo;s worth a quick look at how an operation actually executes in PyTorch.\nWhen we write a + b, the addition doesn\u0026rsquo;t run immediately. The kernel that needs to be called differs depending on whether the two tensors live on CPU or GPU. The dispatcher is what makes that call. The same one-line + is routed to a CPU kernel if the tensors are on CPU, and to a GPU kernel if they\u0026rsquo;re on GPU. The actual computation only begins once that choice is settled. AOTAutograd intercepts operations at the very bottom of this dispatcher, using a hook that fires just before the final kernel would run.\nThat\u0026rsquo;s the difference from the earlier approach. torch.fx intercepted above the Python level, so it saw the name torch.relu. AOTAutograd intercepts below, so it sees the real operation, aten::relu. That\u0026rsquo;s why the graph AOTAutograd produces is uniform throughout.\nTwo clean-up passes Two clean-up passes run over the captured graph.\nFunctionalization removes operations that overwrite existing memory. x.add_(1) modifies x in place instead of making a new tensor, and a tensor created by view shares memory with its original. That forces the compiler to keep track of \u0026ldquo;which point in time is this x from?\u0026rdquo; every time it wants to move or merge operations. So the graph is rewritten such that once a value is created, it never changes.\nDecomposition lowers thousands of operations down to a few hundred, a subset called core ATen IR. For anyone writing a backend this is decisive: the number of operations you have to implement drops by an order of magnitude. Why that property matters so much when attaching new hardware to PyTorch is something we\u0026rsquo;ll come back to in the next post.\nOnce both passes are done, what\u0026rsquo;s left in the graph is operations with no side effects, drawn from a limited set. All that remains is turning them into actual kernels.\nPart 4. TorchInductor: turning the tidied graph into kernels The last component. Inductor takes the tidied graph and generates actual kernel source code — Triton on GPU, C++/OpenMP on CPU. It works in three broad stages: lowering (dropping the graph into Inductor\u0026rsquo;s own IR), scheduling (deciding which operations to group), and codegen (emitting the code).\nPart 3\u0026rsquo;s decomposition was also described as \u0026ldquo;lowering,\u0026rdquo; but the two move along different axes. Decomposition reduced the number of distinct operations (thousands down to hundreds). The lowering here changes the level of the representation: it doesn\u0026rsquo;t shrink the op set further, it rewrites what was expressed per tensor into an expression written per element. That\u0026rsquo;s what the next section is about.\nFrom per-tensor to per-element There\u0026rsquo;s one point here that\u0026rsquo;s easy to trip over. Inductor has an IR of its own, and it works at a different granularity than the FX graph. In the FX graph, one operation was one node and its arguments were whole tensors. Inductor rewrites that same operation in terms of a single element: think of y = floor(x) becoming y[i] = floor(x[i]).\nThe parallelization strategy is settled afterwards. That\u0026rsquo;s what lets the same expression be laid out differently on different devices. This stage is called a loop-level IR — it works one level inside the loop nest rather than on whole tensors.\nFusion is just concatenating formulas Why this representation is a good idea becomes clear with fusion. As mentioned in the introduction, eager execution\u0026rsquo;s fundamental loss comes from launching a separate kernel per operation and writing intermediate results out to memory only to read them back. When a single normalization splits into five or six operations, a memory round trip wedges itself into every gap. Fusion is the optimization that merges those operations into one kernel, so you read once and write once.\nWith the IR written this way, fusion becomes straightforward to handle, because merging two operations turns into concatenating two formulas. If y[i] = floor(x[i]) is followed by z[i] = y[i] + 1, you just write it as z[i] = floor(x[i]) + 1. The intermediate y never has to go out to memory. No elaborate transformation of moving graph nodes around and rewiring them.\nEmitting strings instead of computing values So how does Triton code come out of that formula? y[i] = floor(x[i]) is really a combination of two primitive steps: reading x[i], and applying floor to it. You swap those steps for functions that return strings of code instead of computing values. In the slot where a load happens you plug in a function that produces the string tl.load(...); where the floor happens, one that produces libdevice.floor(...).\nNow run the expression y[i] = floor(x[i]) in the perfectly ordinary way. Strings get produced at each slot, concatenate in order, and out comes Triton kernel source code. Run the same expression with the C++ rule set and you get C++ code instead. The IR stays put; swapping only the rules produces code in a different language. And that property is where the next post begins.\nConclusion torch.compile() was an answer to a long-standing PyTorch problem: getting a graph without giving up Python\u0026rsquo;s flexibility. It recovered optimization opportunities like fusion that eager execution structurally struggled to reach, and the resulting speedups got a lot of attention.\nWhat I find more interesting, though, is that torch.compile()\u0026rsquo;s real character is its structure — one built with extensibility in mind. As you saw in Part 4, Inductor produced both Triton and C++ by changing nothing but a set of rules. So: what happens if you plug a code generator for entirely different hardware into that slot? And what does it mean, for someone trying to attach new hardware, that the decomposition from Part 3 sharply reduces the number of operations you must implement?\nI think PyTorch designed for that extensibility deliberately. Next time we\u0026rsquo;ll step one question further back: why is torch.compile() split into three stages in the first place? And if each stage can be swapped, who swaps what, and what do they gain? Starting from how PyTorch has grown its ecosystem, we\u0026rsquo;ll look at the compilers and hardware that plug in at each stage boundary, including companies building chips that aren\u0026rsquo;t GPUs.\nReference PyTorch documentation\ntorch.compile programming model — Dynamo core concepts, graph breaks, guards, dynamic shapes Custom Backends torch.fx — the structure of an FX graph P.S. HyperAccel is hiring! With the LPU, HyperAccel is throwing down a gauntlet in the AI infrastructure market with a genuinely new architecture. And the software team\u0026rsquo;s work is to draw every last bit of performance out of that hardware while making sure users can reach it in the most intuitive and elegant way possible — which takes no small amount of research and implementation.\nIf you\u0026rsquo;d like to join us on that ride, apply through HyperAccel Career!\n","date":"2026.08.04","permalink":"https://hyperaccel.ai/techblog/en/posts/torch-compile-anatomy/","series":["Exploring torch.compile()"],"style":"--c:#5BC5F6;--on:#02121c","summary":"We take torch.compile apart into its three components — TorchDynamo, AOTAutograd, and TorchInductor — and walk through what each one takes in, what it hands back, and how it actually works.","title":"[Exploring torch.compile() - Part 01] The Three Components"},{"authors":["Jaewon Lim"],"category":"AI Hardware","content":" \u0026ldquo;If you know the enemy and know yourself, you need not fear the result of a hundred battles.\u0026rdquo; This series aims to deeply understand competitors\u0026rsquo; hardware for the sake of designing AI accelerators. In this sixth installment, we cover the datacenter CPU drawing attention again in the Agentic AI era, and the competition among the three vendors Intel, AMD, and NVIDIA.\nHello. I\u0026rsquo;m Jaewon Lim, a hardware verification engineer on the HyperAccel DV team.\nLet me open today\u0026rsquo;s post with the stock charts of two companies.\nIn late April, the day after Intel reported its Q1 earnings, its stock jumped 24% in a single day — its largest single-day gain since 1987. Two weeks later, AMD, which reported afterward, saw its stock rise 18.6% the next day. Behind both companies\u0026rsquo; earnings growth was one thing in common: the CPU.\nAMD\u0026rsquo;s datacenter revenue grew 57% year over year and Intel\u0026rsquo;s 22%, and both cited surging server-CPU demand as the driver. AMD CEO Lisa Su even projected that the server-CPU market would grow to $120 billion by 2030.\nIf GPUs are the center of AI, why are we talking about CPUs again?\nThat question is the starting point of today\u0026rsquo;s post. Everything this series has covered so far — accelerator architectures and various solutions — was about how to optimize the accelerator for AI computation. But as inference workloads shift toward Agentic AI, a paradox has emerged: the very accelerator we optimized so carefully ends up idling, waiting in a ready state. And sitting in that bottleneck is an unexpected component — the CPU.\nIn this post, we\u0026rsquo;ll first look at why the CPU has become a bottleneck again in the Agentic AI era, then compare the latest datacenter CPUs from Intel, AMD, and NVIDIA, and consider the CPU war that looks set to follow the GPU war.\nCan We Just Add More GPUs? The CPU, Called Back to the Stage Before we dive in, let\u0026rsquo;s briefly revisit the relationship between CPU and GPU. We tend to think of the GPU as the star of AI computation, but a GPU cannot execute a single line on its own. The GPU was originally an auxiliary device built to speed up the graphics computations that draw the screen; loading the OS, launching programs, and deciding what to do were the CPU\u0026rsquo;s job from the start. A CPU can run a computer by itself, but a GPU can do nothing without a host CPU to feed and direct it. From the days of plugging a graphics card into a PC until now, this dependency has not changed.\nLater, GPGPU (General-Purpose computing on GPU), represented by CUDA, let the GPU take on general-purpose computation beyond graphics — but that was never an attempt to fully replace the CPU. Heavy, regular matrix computation went to the GPU, while the branch- and control-heavy remainder stayed with the CPU. Then, as the GPU\u0026rsquo;s price skyrocketed in the AI era, the CPU somehow came to be treated as \u0026ldquo;the less important part that feeds the GPU.\u0026rdquo; The lead and supporting roles seemed to have switched.\nBut the GPU is still not a general-purpose device. And Agentic AI endlessly demands the kind of general-purpose work needed everywhere: hitting search engines, running code in sandboxes, and orchestrating many tools. This is work the GPU cannot do and only the CPU does. This is the point where the CPU, long pushed into a supporting role, is called back to center stage, and a new war over datacenter CPUs begins.\nWhen a bottleneck appears in AI computation, the most common solution that comes to mind is the GPU — a higher-performance GPU, or scaling the infrastructure with more GPUs. Back when running a model through a single pass was the entirety of an AI application, that was generally the right prescription.\nBut recent studies show a slightly strange picture. According to one study, the time an agent spends calling tools and processing their results took up as much as 88% of total response latency, depending on the workload. And almost all of this tool execution happens on the CPU. The expensive GPU, meanwhile, was idling that whole time, waiting for the CPU\u0026rsquo;s results.\nEven more interesting is another study that showed how to solve this. Here, not a single GPU was added. Instead, the CPU cores assigned to one GPU were increased from 1 to 8. With that alone, time-to-first-token (TTFT) became up to 5× faster.\nWithout adding any GPUs, simply adding a few CPU cores produced a several-fold speed difference. Has the CPU returned to the front line of computing? Let\u0026rsquo;s look at the background in the next section.\nWhy the CPU Matters Again: From a Single Pass to the Agentic Loop How could simply adding CPUs achieve this kind of performance gain? The answer lies in how the shape of inference has evolved.\nUntil now, LLM inference ended in a single pass. The CPU splits the input into tokens, the GPU runs it through the model to produce an answer, and the CPU turns it back into human-readable text. In this structure the GPU works almost without rest. The CPU\u0026rsquo;s job is just the light work of converting tokens at the front and back.\nAgentic AI changes this picture. An agent doesn\u0026rsquo;t stop at producing the model\u0026rsquo;s answer; it directly creates artifacts like code, documents, and designs. To do so, it repeats a cycle of planning, calling tools, looking at the results, and judging again, many times over. In this process, tool execution, search, code execution, orchestration, and retries are all the CPU\u0026rsquo;s job. The GPU is specialized only for running the model quickly, so apart from that one moment, the CPU shoulders everything else. In the end, the GPU works only at the instant of passing through the model, and stalls — waiting — while the CPU runs the tools.\nSo what exactly does the CPU take on? Broadly, three things.\nFirst is information retrieval. Tasks that fetch external knowledge, like RAG (Retrieval-Augmented Generation), vector-DB lookups, and database queries. Because they rummage through huge indexes and move back and forth across disk and network, they consume a lot of memory, and since the CPU performs better here, the CPU mostly handles them.\nSecond is code execution. Agentic AI is used especially heavily for coding tasks. Code the model produces goes through a step that checks whether it\u0026rsquo;s valid before being handed to the user. At that point, an isolated sandbox (a container or a separate process) is spun up to run Python or a shell inside it. This too is typical CPU/OS work.\nThird is connecting task to task. Orchestration — preparing tool calls, parsing the returned results, deciding the next step, and retrying on failure — runs ceaselessly on the CPU.\nTo borrow Intel\u0026rsquo;s framing, Agentic AI makes inference less like one giant program such as an LLM and more like microservices in which many small services cooperate. Control flow, tool calls, retries, and coordination all happen on the CPU. As a result, a substantial portion of the time a single task takes — by some measurements 30–90% — stays on the CPU.\nThe Cost Paradox The paradox is cost. Based on cloud-instance pricing, GPU compute is 100× to as much as 1,600× more expensive than CPU. Yet the cost of adding CPU cores to relieve this bottleneck came to only about 1.5% per server. Using a bit more cheap CPU to keep the expensive GPU from idling is by far the better deal.\nMoreover, the earlier studies say that \u0026ldquo;the better the GPU gets, the faster the bottleneck shifts to the CPU.\u0026rdquo; The faster a next-generation GPU runs the model, the greater the relative burden on the CPU executing other work in the meantime. In the end, scaling the GPU alone won\u0026rsquo;t solve this problem; better CPUs and more CPUs become necessary. So now let\u0026rsquo;s turn our gaze to the CPU itself — and to the companies that make CPUs.\nFrom a Two-Way to a Three-Way Race — NVIDIA Enters the CPU Fight Traditionally, one CPU in a server commanded 4–8 GPUs. In the agentic era, this ratio flips. Intel claims that the faster next-generation GPUs get, the more CPUs — up to 7 — may be needed to properly operate a single GPU.\nSo every company that makes CPUs is shouting, \u0026ldquo;It\u0026rsquo;s the age of the CPU again.\u0026rdquo; And here a name that hadn\u0026rsquo;t been visible appears — NVIDIA, the GPU maker. NVIDIA has jumped directly into the CPU race.\nThis is actually half true and half false. NVIDIA walking into the middle of the datacenter-CPU race is clearly a new event, but it does not mean \u0026ldquo;NVIDIA is making a CPU for the first time.\u0026rdquo; A GPU can\u0026rsquo;t start anything on its own. Booting, and dividing up the work to compute, are all handled by the CPU (host) attached beside it. So NVIDIA has long built its own CPUs to command its GPUs.\nIt\u0026rsquo;s just that the stage wasn\u0026rsquo;t the datacenter. The traditional datacenter and PC markets were held tightly by Intel and AMD, who own x86, and there was almost no room for NVIDIA — which has no x86 license — to squeeze in. So NVIDIA\u0026rsquo;s CPUs mostly took the form of Tegra, an Arm-based SoC, settling into game consoles like the Nintendo Switch and into mobile and automotive embedded areas.\nBut NVIDIA, using the core-design know-how it accumulated along the way and its compatibility with its own GPUs, expanded the stage to datacenter CPUs sold \u0026ldquo;together with the GPU,\u0026rdquo; centered on the Arm ISA. Grace, paired with Hopper and Blackwell, was the first fruit. And finally, at GTC Taipei in June 2026, NVIDIA unveiled Vera — built with cores it designed itself, unlike Grace, which borrowed licensed Arm cores — declaring it was taking direct aim at the datacenter-CPU market. This is the moment the CPU war, long a two-way race between Intel and AMD, turns into a three-way race. In the next section we\u0026rsquo;ll take apart the three companies\u0026rsquo; CPU architectures side by side.\nA Close Look at the Three Companies\u0026rsquo; CPU Architectures ① x86 vs Arm The most fundamental line dividing the three companies isn\u0026rsquo;t how large they made the cores, but which instruction set (ISA) they built the cores on — Intel and AMD on x86, NVIDIA\u0026rsquo;s Vera on Arm. An ISA (Instruction Set Architecture) is the contract between software and hardware: it fixes the kinds and formats of instructions a program can give the CPU, and the register specifications. Within the same ISA, a program compiled once runs on any chip built on it; with a different ISA, the same program must be recompiled.\nx86 and Arm differ from the very philosophy of how they write this contract. x86 started from the CISC (Complex Instruction Set Computer) family, where one instruction handles complex work and lengths vary (variable length). Arm is in the RISC (Reduced Instruction Set Computer) family, breaking instructions into simple pieces and fixing the length at 4 bytes. Today, both camps internally convert instructions into finely sliced micro-operations (µops) before executing, so this boundary has blurred considerably — but one difference remains to the end: the cost of decoding instructions. Because x86 has uneven lengths, it must first find instruction boundaries, making the decoder complex and more power-hungry; because Arm has fixed lengths, it\u0026rsquo;s easy to decode several instructions side by side at once, which is favorable for widening the decoder. This is why Arm-architecture processors were widely used as APs (application processors) back in the mobile era.\nThen wouldn\u0026rsquo;t the efficient Arm architecture be advantageous even in AI computation, where power matters? Its performance has also risen enough not to lag behind x86 — but the reality isn\u0026rsquo;t easy. x86\u0026rsquo;s real moat is not performance but compatibility. Decades of operating systems, enterprise software, and drivers are all compiled to x86 binaries, and moving these assets costs enormously. On top of that, the companies that can actually make x86 are effectively just Intel and AMD, so that fence itself is a barrier to entry.\nArm\u0026rsquo;s weapons, by contrast, are efficiency and openness. Originating in mobile, it was designed with \u0026ldquo;performance per watt\u0026rdquo; as the top priority, and by opening its ISA via licensing, it let anyone design their own core on top of it. Apple\u0026rsquo;s M series, and NVIDIA\u0026rsquo;s Grace and Vera, were all made this way. In the datacenter, where performance-per-watt and core density are operating costs, this philosophy is gaining momentum — and it\u0026rsquo;s also why NVIDIA chose Arm over x86.\nParallel-Computation ISA Extensions\nOn top of this base ISA ride extension instructions for quickly processing vector/matrix computation. Most basic instructions process operations one at a time at scalar granularity, so computing multiple pieces of data requires that many instructions. To raise throughput by performing parallelizable work with a single instruction, SIMD (single instruction, multiple data) instructions were created. This is the concept we also explained in the GPU installment, and the CPU uses it for performance gains as well.\nThe x86 camp (Intel/AMD) uses AVX (Advanced Vector Extensions)-based SIMD units, which fix the vector length, as a common foundation, while the Arm camp uses SVE (Scalable Vector Extension), a variable-length vector extension. Up to here it\u0026rsquo;s a difference the ISA carved out, but even within the x86 camp, Intel\u0026rsquo;s and AMD\u0026rsquo;s approaches diverge.\nIntel added a dedicated 2D tile matrix engine, AMX (Advanced Matrix Extensions), inside the core on top of AVX. AMX is a dedicated unit that processes multiply-accumulate at matrix granularity rather than 1D vectors; one core processes 2,048 multiply-accumulates per cycle at INT8 and 1,024 at BF16/FP16. With this dedicated matrix engine, small models (SLMs) can be inferred with the CPU alone, without a GPU.\nAMD processes matrix computation with AVX-512 and VNNI (Vector Neural Network Instructions), without a dedicated tile engine. The hardware processing density is lower than a dedicated 2D matrix engine, but existing SIMD units can perform matrix computation well enough, and heavy computation is handed to the GPU.\nIn addition, Intel and AMD jointly announced the ACE (AI Compute Extensions) specification to standardize CPU-side AI matrix-computation extensions in the x86 ecosystem. This is a standard meant to let x86 perform AI computation with general-purpose compute units, without a dedicated unit like AMX.\nNVIDIA\u0026rsquo;s Vera takes the path of Arm SVE2 plus native FP8. It has no dedicated tile engine, but uses variable-length vector units and FP8 support to handle part of the inference computation, handing heavy matrix computation to the GPU.\n② Multithreading — Split Time, or Partition Space? After ISA, the point where the three companies\u0026rsquo; CPU-design philosophies diverge most clearly is multithreading. Here NVIDIA takes a fundamentally different path from the two x86 companies.\nMultithreading (hardware multithreading) is a technique that puts multiple threads on a single core at the same time. If a core runs only one thread, then when that thread stalls waiting for memory, the expensive execution units idle entirely. So two threads are placed together on one core, so that while one is stalled, the other fills the empty units. Intel calls this Hyper-Threading and AMD calls it Simultaneous Multi-Threading (SMT), but the principle is the same. The crux is \u0026ldquo;how two threads share one core\u0026rsquo;s resources,\u0026rdquo; and here the path splits in two.\nTime-shared SMT (Intel · AMD). Traditional 2-way SMT has two threads share core resources — decoder, execution ports, registers — dynamically over time. Because they split whatever is free each cycle, a single thread monopolizes the whole core when alone, and the two compete when running together. Average throughput is high, but if the neighboring thread uses a lot of resources, my thread slows down, so per-thread performance and tail latency become uneven.\nSpatial multithreading (NVIDIA Vera). Vera\u0026rsquo;s Olympus core takes a different direction. Instead of taking turns by slicing time, it physically partitions core resources between the two threads, so each thread gets a fixed (and correspondingly reduced) set of resources. 88 cores split this way 2-way become 176 threads. The peak speed a single thread can reach may be lower than time-shared SMT, but its advantage is that my thread\u0026rsquo;s latency doesn\u0026rsquo;t waver no matter what the neighboring thread does.\nThis difference matters in agentic workloads. A single agent step spreads out tens or hundreds of tool calls and sandbox sessions at once, and only after all their results gather does it move to the next inference (fan-out/fan-in), so the step\u0026rsquo;s completion time is determined not by the average of the sessions but by the single slowest one. In time-shared SMT this tail sways with the neighboring thread\u0026rsquo;s resource competition (noisy neighbor), but physically partitioning resources fixes each session\u0026rsquo;s latency independently of its neighbors, stabilizing tail latency. In the end, what matters becomes not a single thread\u0026rsquo;s peak speed but every session finishing within a predictable time.\n③ CPU-GPU Link The last is the link connecting CPU and GPU. Not only must results computed on the CPU be delivered to the GPU, but when KV-cache capacity exceeds the GPU\u0026rsquo;s HBM capacity, CPU memory must be used — so this link becomes even more important.\nIntel and AMD use standard methods. Xeon connects CPU and GPU via PCIe 5 / CXL, and EPYC via PCIe 5 / Infinity Fabric. PCIe 5 is about 64 GB/s per direction on x16 lanes — roughly 128 GB/s even combining both directions.\nNVIDIA uses its own specification. Vera becomes one body with the Rubin GPU through a coherent interconnect called NVLink-C2C, reaching 1.8 TB/s. This is a full 2× increase over the predecessor Grace\u0026rsquo;s 900 GB/s.\nSummarizing the three axes we\u0026rsquo;ve looked at so far in one table:\nCategory Intel Xeon AMD EPYC NVIDIA Vera Current-gen flagship Xeon 6980P · Redwood Cove EPYC 9755·9965 · Zen 5 / 5c Vera · Olympus (custom Arm) Instruction set (ISA) x86-64 (CISC family) x86-64 (CISC family) Arm Armv9.2-A (RISC family) Cores / threads 128C / 256T 128C/256T (9755) · 192C/384T (9965) 88C / 176T Matrix-compute ISA AMX + AVX-512 (dedicated tile matrix engine) AVX-512 / VNNI (SIMD-based matrix compute) SVE2 + FP8 ② Multithreading 2-way time-shared SMT 2-way time-shared SMT 2-way spatial multithreading (physical resource partition) ③ CPU-GPU link PCIe 5 / CXL PCIe 5 / Infinity Fabric NVLink-C2C 1.8 TB/s (coherent) Wrapping Up Today we looked at the CPU, risen again in the Agentic AI era. As inference shifted from a single pass to a loop that calls tools, control and tool execution came down to the CPU, and a new bottleneck appeared in between — the expensive GPU sitting idle. Several experimental results showed this clearly: tool execution took up a substantial portion of response latency, and simply adding CPUs instead of GPUs made things faster.\nIn this situation of growing CPU demand, NVIDIA has declared its entry into the market that Intel and AMD had split between them. With an approach very different from the two, and high performance achieved on the back of its GPU-infrastructure know-how, it is amplifying customers\u0026rsquo; expectations. No benchmark directly comparing the three CPUs on the same agentic workload exists yet, but judging by the announced performance alone, I suspect the CPU market is heading toward a three-way structure.\nWhat\u0026rsquo;s interesting is that this trend grows stronger the more you scale the GPU. The faster a next-generation GPU runs the model, the greater the burden on the CPU, which has to run tools and send back responses in the meantime. From the standpoint of designing an AI accelerator too, how to choose the host CPU and how to connect it is becoming an ever more important question.\nP.S. HyperAccel Is Hiring HyperAccel is on the verge of launching its first datacenter LPU product, and through hardware/software optimization, we\u0026rsquo;re developing technology that resolves the core bottlenecks of LLM inference and delivers efficient service.\nIf our technical journey interests you, apply right now through HyperAccel Career!\nHyperAccel is waiting for your application.\nReference Earnings \u0026amp; market reaction\nAMD Reports First Quarter 2026 Financial Results (official IR) Intel Reports First-Quarter 2026 Financial Results (official IR) Intel\u0026rsquo;s stock has best day since 1987, soaring 24% — CNBC (2026-04-24) AMD (AMD) Stock Price History — stockanalysis.com Agentic AI and the CPU bottleneck (research \u0026amp; analysis)\nRitik Raj et al., \u0026ldquo;Towards Understanding, Analyzing, and Optimizing Agentic AI Execution: A CPU-Centric Perspective\u0026rdquo; (arXiv:2511.00739) Euijun Chung et al., \u0026ldquo;Characterizing CPU-Induced Slowdowns in Multi-GPU LLM Inference\u0026rdquo; (arXiv:2603.22774) The CPU Bottleneck in Agentic AI and Why Server CPUs Matter More Than Ever — viksnewsletter SemiWiki, \u0026ldquo;Agentic AI Demands More Than GPUs\u0026rdquo; (2026-04-08) CPU:GPU ratio rebalancing · supply/demand\nTrendForce, \u0026ldquo;The Great Rebalance: How Agentic AI Is Reshaping the CPU:GPU Ratio\u0026rdquo; (2026-04) Tom\u0026rsquo;s Hardware, \u0026ldquo;\u0026lsquo;CPUs are cool again\u0026rsquo; — agentic AI drives a surge in CPU demand and shortages\u0026rdquo; (2026-04) Tom\u0026rsquo;s Hardware, \u0026ldquo;Rising CPU demand for AI workloads drives shortages and price hikes\u0026rdquo; (2026-04) Vendor materials\nIntel White Paper, \u0026ldquo;Agentic AI Requires More CPUs\u0026rdquo; (2026) AMD Blog, \u0026ldquo;Agentic AI Changes the CPU/GPU Equation\u0026rdquo; (2026) NVIDIA, \u0026ldquo;NVIDIA Unveils Vera, the CPU for Agents\u0026rdquo; — GTC Taipei 2026 (2026-05-31) NVIDIA Vera CPU product page — \u0026ldquo;The CPU for agents\u0026rdquo; CPU architecture comparison\nBabai Das, \u0026ldquo;Arm vs x86: Why x86 Still Matters in an Arm-Dominated Future\u0026rdquo; — Medium ","date":"2026.07.02","permalink":"https://hyperaccel.ai/techblog/en/posts/cpu-revival/","series":["Know Your Enemy, Know Yourself"],"style":"--c:#00C2A8;--on:#02121c","summary":"We analyze why the CPU became the bottleneck of inference infrastructure in Agentic AI workloads, walk through the latest datacenter CPU lineups from the three CPU vendors, and explore why the CPU has risen to prominence again in the Agentic AI era.","title":"[Know Your Enemy - Part 06] The Agentic AI Era — The Revival of the CPU and the Dawn of the CPU Three Kingdoms"},{"authors":["Jaewoo Kim"],"category":"Compilers \u0026 Kernels","content":"Legato: A Programming Model for the LPU This post is written for developers at HyperAccel\u0026rsquo;s customers and partners. It explains what Legato is, what role it plays, why we built it, and how it is designed.\n1. What Is Legato, and Why Do We Need It? 1.1 Software Ecosystems Decide the Outcome What is NVIDIA\u0026rsquo;s real moat in the AI accelerator market? Many people think of raw GPU compute performance, but the hardware gap between competitors is often smaller than it appears. Transistor process, memory bandwidth, and compute-unit design are areas where several companies are catching up quickly.\nWhat actually creates separation is the software ecosystem. Libraries, frameworks, know-how, and the confidence that \u0026ldquo;any model will run on NVIDIA eventually\u0026rdquo; — all built on top of CUDA — form a barrier to entry. No matter how good the hardware is, the market will not move if developers cannot use it freely and comfortably.\n1.2 The Software Competitiveness HyperAccel\u0026rsquo;s LPU Needs For HyperAccel\u0026rsquo;s LPU (LLM Processing Unit) to compete, it must go beyond building a fast chip and meet two conditions.\nFirst, developers should not feel friction when using it. If adopting new hardware means throwing away an entire workflow, that hardware gets ignored. It must integrate naturally with the inference stack that is effectively the standard in today\u0026rsquo;s AI inference market — a torch and vLLM-based stack.\nSecond, it must support arbitrary models. An accelerator that only runs a handful of internally ported and managed models is not a true platform. It must accept whatever model users bring.\n1.3 Legato\u0026rsquo;s Role Legato is the interface that connects hardware and software developers and makes all of this possible.\nLegato is both a programming language and a programming model for software developers. A programming model is the contract that defines how the LPU appears to software developers.\nCUDA makes this idea concrete. We do not know every detail inside an NVIDIA chip, yet once we learn CUDA, we can program any generation of GPU. CUDA provides a simplified programming model that hides chip complexity.\nLegato works the same way. It gives developers a simplified view of the LPU so they can program it without knowing low-level hardware details.\n1.4 Where Legato Sits in the Stack Legato\u0026rsquo;s role becomes clear when you see where it sits in the full inference stack.\nLegato connects the existing ecosystem (PyTorch/vLLM) above with LPU hardware below. Developers write code the way they always have; Legato absorbs the complexity of the hardware below.\n2. Legato at a Glance (Quick Look) Rather than add more abstract explanation, let\u0026rsquo;s look at code first. Here is a minimal example that performs matrix multiplication (matmul) on the LPU.\nimport torch import legato import legato.model.bertha as bertha # Set the compile target. # This example targets a Bertha chip with 128 GB of memory and 32 cores. def get_bertha(ctx): gigabyte = 1024**3 return bertha.Bertha( ctx, \u0026#34;bertha\u0026#34;, 32, # num_cores False, # use_pim 8, # num_memory_channels 128 * gigabyte, # shared_memory_size ) # The @legato.compile decorator turns this function from a Python function # into a Legato kernel that gets compiled. All code inside is Legato code, # not Python (similar in spirit to Triton or Mojo). @legato.compile( backend=get_bertha ) def simple_matmul( a: legato.get_model().tensor_type( legato.types.float(\u0026#34;bfloat16\u0026#34;), ((-1, 10), 256), \u0026#34;shared_dram\u0026#34; ), b: legato.get_model().tensor_type( legato.types.float(\u0026#34;bfloat16\u0026#34;), [256, 128], \u0026#34;mpu_dram\u0026#34; ), out: legato.get_model().tensor_type( legato.types.float(\u0026#34;bfloat16\u0026#34;), [10, 128], \u0026#34;shared_dram\u0026#34; ) ): # Load the device provided by the backend. device = legato.get_context().get_device() # Code that runs on Bertha\u0026#39;s top module. with device.get_top(): # Copy a and b to core 0\u0026#39;s SRAM and mpu_buffer. legato.tensor.request_load(a, device.get_core(0), \u0026#34;sram\u0026#34;) legato.tensor.request_load(b, device.get_core(0), \u0026#34;mpu_buffer\u0026#34;) # Code region that runs on Bertha\u0026#39;s core 0. with device.get_core(0): weight_type = legato.get_model().tensor_type( legato.types.float(\u0026#34;bfloat16\u0026#34;), [256, 128], \u0026#34;mpu_weight\u0026#34; ) # Receive data passed from the top module via request_load. loaded_a = legato.tensor.receive(a, 0, \u0026#34;sram\u0026#34;) loaded_b = legato.tensor.receive_type(weight_type, 0, None, \u0026#34;mpu_buffer\u0026#34;) # Perform matmul with the received data. result = loaded_a @ loaded_b # Copy the result to the out tensor. legato.tensor.memcpy(out, result) output = torch.zeros(10, 128, dtype=torch.bfloat16) # legato.session lets you specify JIT compilation options (output format, output directory, etc.). # simple_matmul is JIT-compiled at call time; if a compiled binary already # exists and the code has not changed, the cached binary is reused. with legato.session(output_type=legato.OutputType.BINARY, output_path=\u0026#34;simple_matmul\u0026#34;): simple_matmul( torch.randn(10, 256, dtype=torch.bfloat16), torch.randn(256, 128, dtype=torch.bfloat16), output, ) This short snippet captures all of Legato\u0026rsquo;s core ideas:\n@legato.compile — defines the entry point to compile. with device.get_top() / with device.get_core(0) — specify where each operation runs (execution context). The kinds of contexts can vary slightly from chip to chip. In our example, the HyperAccel LPU consists of multiple cores, so we assigned the computation to core 0. request_load → receive — specify when and where data moves. legato.session(...) — decide what form the compilation output takes and where it is written. Choosing BINARY produces an ELF executable; you can also output ASM, CORE_IR, BACKEND_IR, and more, which are useful during debugging. 3. Legato\u0026rsquo;s Programming Model (Core Design Concepts) The heart of Legato\u0026rsquo;s design is its programming model — that is, how the LPU looks to developers.\n3.1 A Simplified Virtual Hardware Model One important premise: the hardware model shown to developers is intentionally different from the physical hardware. Legato hides complex physical details and presents a simplified virtual machine that programmers can reason about.\nThat virtual hardware is roughly structured as follows:\nDRAM + Controller — Shared DRAM accessible by all cores. Physically it may consist of multiple memory modules, but the programming model treats them as one large memory. Virtual DRAM — A shared region accessed through an MMU, assumed to be preconfigured. PIM (Processing-in-Memory) — Used selectively when needed. Core — The number of cores depends on the hardware configuration. Each core has a PC and registers, multiple execution units (such as MPU and VPU executors), on-chip SRAM (cache), and private DRAM used only by that core. 3.2 Memory Model From a software perspective, memory is simplified into three categories:\nShared Memory — Space shared by all cores. On Bertha, this corresponds to DRAM. Virtual Shared Memory — Shared space managed by the MMU. Core Memory — Core-local space combining private DRAM and SRAM. Each core has SRAM space that can be used as scratchpad memory; the compiler plans how to use it. Private DRAM is a per-core DRAM region used for eviction when on-core scratchpad memory is insufficient. An important design choice appears here. When a programmer allocates data in Core Memory, whether it lands in SRAM or private DRAM is decided by the compiler\u0026rsquo;s planning phase. Developers work with the abstract notion of \u0026ldquo;this core\u0026rsquo;s memory,\u0026rdquo; and the compiler handles physical placement optimization.\n3.3 Execution Context (Executor) and Explicit Data Movement Legato operations always run inside an execution context that indicates where the next operation executes.\ndevice = legato.get_context().get_device() with device.get_top(): legato.tensor.request_load(input_data, device.get_core(0), \u0026#34;sram\u0026#34;) with device.get_core(0): loaded = legato.tensor.receive(input_data, 0, \u0026#34;sram\u0026#34;) result = loaded + loaded device.get_top() — used for orchestration and top-side load requests. device.get_core(i) — run on a specific core. device.get_all_cores() — run the enclosed region on all cores. The LPU has the characteristic that each module runs its own program counter (PC) independently. Legato represents this naturally by separating execution contexts through ContextOp.\nData movement is explicit, not implicit. You request a load from a source context with request_load, then receive data in the target context with receive (or receive_type). When and where data moves is visible in the code.\nLegato scopes are stricter than Python scopes. If a value created in one region must be used in the next, you must pass it explicitly with yield:\nwith device.get_core(0): q = loaded @ weight yield q with device.get_core(0): legato.tensor.memcpy(output, q) 3.4 Forward Compatibility The biggest payoff of this simplified programming model is forward compatibility.\nAs long as future LPUs preserve compatibility with this programming model, a program written once can run on the next hardware generation without code changes — the same promise CUDA makes when code written for one GPU generation runs on the next. The stability of the programming model is what makes \u0026ldquo;hardware evolves, code stays\u0026rdquo; credible.\n4. Legato\u0026rsquo;s Design Philosophy 4.1 Python-Embedded DSL Legato is designed as a Domain Specific Language (DSL) embedded in Python. That is intentional. With a single pip install, developers can use it in a familiar Python environment without a separate toolchain, keeping the barrier to entry as low as possible. They start programming the LPU inside syntax they already know: with blocks, functions, and decorators.\n4.2 Python-Friendly Tensor Abstraction Legato operations are built around tensors that many programmers already know. It provides operations similar to PyTorch or NumPy — broadcast, reshape, matmul, and so on.\nBeyond that, it also supports advanced programming features such as synchronization and metaprogramming (static evaluation) to make performance tuning easier.\nThe semantics of each library operation are designed generically so they can be used without code changes even as LPU generations evolve. In other words, the API stays stable while the compiler lowers operations to the best implementation for each generation underneath.\n4.3 Explicit Control vs. Automation Trade-off Legato is a relatively low-level interface where developers explicitly describe execution placement, data movement, and compile-time parameters. That can look verbose, but it is deliberate. For kernels where performance is the value — where you must express data flow clearly — that explicitness is control.\nAt the same time, Legato also provides automation paths, so users can choose depending on the situation:\nFully automatic — hand the entire model to the torch.compile backend. Fully manual — write critical kernels directly and tune them in detail. 4.4 Extensibility Using Legato from PyTorch\nLegato implements PyTorch backend operations so the LPU works with torch, similar to how Triton implements GPU operations to support PyTorch. The flow is:\nImplement a Legato function for each PyTorch operation you want to support. When one operation has multiple implementations (for example, float32 vs. bfloat16), a resolver function picks the right one and hands it to the compiler. Import the implemented module or expose it through a legato-library entry point. import torch @torch.compile(backend=\u0026#34;legato\u0026#34;, options={\u0026#34;legato.device\u0026#34;: \u0026#34;bertha\u0026#34;}) def model(x, y): ... Partial compilation (mixed execution). Not every operation can go to the LPU. Legato wraps the full graph in LegatoGraphWrapper, groups compilable nodes into Legato kernels, and runs unsupported nodes as eager extern calls. If a resolver is missing or a dtype/shape combination raises AssertionError, that node becomes an extern fallback candidate. Models that mix custom or unsupported ATen ops therefore do not fail immediately; compilable regions still accelerate.\nPlatform-agnostic extension. Legato is not limited to PyTorch. It can extend to NumPy, CuPy, and other platforms. As long as the backend is implemented with Legato, there is no fundamental limit to extensibility.\n5. Legato\u0026rsquo;s Internal Structure 5.1 Compilation Pipeline A Legato program goes through the following stages to become a hardware artifact:\nFrontend — defines the user-facing surface. It takes Legato code written in Python and lowers it to MLIR-based Legato IR. Core IR / Backend IR — successive lowering passes move the representation closer to hardware. Artifact — the final output is assembly or a binary. 5.2 ComputeModel — Injecting Hardware Information A ComputeModel is attached throughout the pipeline to supply hardware information to the compiler.\nA ComputeModel is a simplified model of the hardware and plays a central role in lowering and optimization decisions. Swapping the ComputeModel is enough to retarget the compiler. As in the get_bertha example, a backend factory creates a ComputeModel instance and passes it to the compiler.\nBeyond supplying information, the ComputeModel is the basis for validation. If a user writes an operation that cannot run efficiently (or at all) on a given device, the compiler tries feasible transforms or, when that fails, returns a clear error message to guide the user.\n5.3 Per-Hardware Backends Each hardware target has its own compiler backend that performs hardware-specific lowering. To support a new LPU, implement a backend and ComputeModel for that hardware; the upper-level programming model and user code stay reusable.\n5.4 Output Targets — Transparency and Debugging Legato lets you dump artifacts at any compilation stage. Choose one of the following with legato.session\u0026rsquo;s output_type:\nOutput Type Description MLIR Initial Legato IR (MLIR form) CORE_IR Core IR stage BACKEND_IR Backend IR stage ASM Assembly BINARY Final executable binary with legato.session(output_type=legato.OutputType.MLIR, output_path=\u0026#34;test-output\u0026#34;): kernel(*args) Being able to inspect IR at each stage means compilation is not a black box. When something goes wrong, you can trace how the program was transformed at each step, which makes debugging and verification transparent.\n5.5 ELF Binary The final output is an ELF binary. It is not just a blob of instructions; it also carries metadata such as hardware information, compiler information, and checksums. At runtime, the runtime reads this metadata to verify that the binary is suitable for the current hardware and environment, preventing accidental execution of the wrong binary on the wrong device.\n6. Robustness and Verification (Trust Signals) For anyone adopting a platform, the key question is: \u0026ldquo;Can I trust this in production?\u0026rdquo; Legato answers with verification at several layers.\n6.1 ComputeModel-Based Validation As described earlier, when a user writes an operation that is impossible or inefficient on a target device, the compiler attempts automatic transformation and, if that fails, returns an actionable error message explaining what went wrong and why. Problems surface at compile time instead of silently producing wrong results.\n6.2 Type, Layout, and Dynamic Shape Validation Legato validates tensor types and memory layouts in function signatures. In particular, dynamic dimensions must have an upper bound:\nx = torch.zeros(16, 32, 32) torch._dynamo.mark_dynamic(x, 0, max=128) # dim 0 is dynamic, upper bound 128 Because of hardware characteristics, tensors cannot be fully dynamic. Every dynamic dimension needs an upper bound for buffer allocation and command generation; dimensions without an upper bound raise an error at compile time. This is so the compiler can plan tensor allocation at compile time.\n6.3 Host-Device ABI (Instruction Table) To pass arguments correctly across the host–device boundary, Legato defines a binary ABI called the Instruction Table. It strictly follows 64-bit alignment rules and packs scalar, tensor, and tuple arguments according to a fixed convention. The maximum table size can be computed at compile time from the function signature, so runtime argument passing is predictable and safe.\nQ. Do I Need to Learn All of Legato to Use the LPU? No. Of course you do not. Not every AI developer who uses a GPU knows CUDA, OpenCL, or Vulkan in depth. HyperAccel\u0026rsquo;s LPU works the same way.\nLegato implements backends for torch and other libraries; it is not something every AI developer interacts with directly. When you use torch or numpy on the LPU, you can work the way you do on CPU or GPU today, while the underlying operations are implemented with Legato.\nIf you want custom kernels or extreme performance tuning, you can use Legato directly. On GPUs, developers sometimes write custom kernels in Triton or CUDA instead of relying only on what torch already provides when they want to push performance or implement operations that do not exist yet. The LPU is no different. Legato is always available for direct use, and you can insert Legato-implemented layers into an otherwise torch-based model, as below:\n@legato.compile(backend=bertha) def custom_kernel(input: legato.get_model().tensor_type( legato.types.float(\u0026#34;bfloat16\u0026#34;), ((-1, 10), 256), \u0026#34;shared_dram\u0026#34; ), output: legato.get_model().tensor_type( legato.types.float(\u0026#34;bfloat16\u0026#34;), [256, 128], \u0026#34;shared_dram\u0026#34; )): # Some implementation... c = torch.add(a, b) out = torch.zeros(256, 128) # Call legato kernel custom_kernel(c, out) print(f\u0026#34;output from legato kernel : {out}\u0026#34;) # Keep going with torch out_exp = torch.exp(out) 7. Summary and Next Steps 7.1 Summary Legato provides a simplified programming model for the LPU and achieves three goals at once:\nEcosystem compatibility — it integrates naturally with existing stacks such as torch and vLLM. Forward compatibility — as long as the programming model is preserved, code runs on future LPUs without changes. Extensibility — new hardware and platforms require only a backend and ComputeModel. Just as CUDA underpins NVIDIA\u0026rsquo;s ecosystem, Legato is the foundation of HyperAccel\u0026rsquo;s LPU ecosystem.\n7.2 Supported Targets and Roadmap Legato currently targets the Bertha architecture (with per-generation guides such as EVT0 and EVT1). Because retargeting is built around swapping the ComputeModel, extension to future LPU generations is prepared at the design level.\nP.S.: HyperAccel is Hiring! We are looking for teammates to help build Legato and the LPU software stack. If you are interested in the technologies we work on, please apply via HyperAccel Career.\n","date":"2026.06.28","permalink":"https://hyperaccel.ai/techblog/en/posts/what-is-legato/","series":["Legato"],"style":"--c:#5BC5F6;--on:#02121c","summary":"An introduction to Legato — HyperAccel\u0026rsquo;s programming model and compiler stack for the LPU — covering what it is, why it exists, and how it is designed.","title":"[Legato - Part 01] A Programming Language for HyperAccel LPU"},{"authors":["Taeseo Um"],"category":"Engineering Culture","content":" Hello, my name is Taeseo Um, currently on the LLMOps team at HyperAccel.\nIn the previous post, I wrote about how we moved from Copy \u0026amp; Paste workflows to coding agents. AI started out feeling like a slightly better search bar, then it grew into autocomplete, and now it can read a whole repository, run the tests, inspect the failures, edit the files, and try again on its own.\nThis post is about a quieter change that rode in on the back of that shift — the slow return of the terminal, and the Command Line Interface(CLI) that gives it its shape.\nWhat feels different now is that terminal-native agents have loosened the old bond between development and a local Integrated Development Environment(IDE) session. Work has started to follow whichever execution environment I can reach and trust — sometimes the workstation that holds my repository, sometimes the environment that sits closest to the running service — and Secure Shell(SSH) has quietly become just the path into that environment rather than the point of it.\nThat has changed how coding feels over the course of a day, where I\u0026rsquo;m able to debug, and even what I\u0026rsquo;m willing to call a real development environment in the first place.\nThe Terminal Became a Place Again For a long time, my terminal was mostly a utility drawer — somewhere I reached for git, builds, package managers, logs, and the occasional shell script. The real work happened in my IDE, and the terminal was just where I checked whether that work survived contact with reality.\nCoding agents pulled the terminal back into the center.\nThe reason is simple: an agent needs a way to touch the system. It has to read files, search through code, run commands, watch them fail, and leave evidence behind — and the CLI is already built around exactly that loop. Commands are text, output is text, failures and diffs are text, and tests come back as a signal that\u0026rsquo;s simple enough for both a human and an agent to read.\nWhen an agent works in a terminal, it doesn\u0026rsquo;t have to guess at what happened behind some visual panel; it runs the same commands I would run and reads the same logs I would read, so the feedback loop stays short and direct.\nThat directness matters more than it sounds. A chat User Interface(UI) can feel friendly, but software work isn\u0026rsquo;t only conversation — it\u0026rsquo;s action under constraints, and the terminal gives the agent a place where every action leaves a trace.\nIt also gives me a different kind of memory. The scrollback quietly records where the work began, which command failed first, which assumption turned out to be wrong, and which test finally went green. Speed without that kind of evidence is stressful, and the terminal keeps the work visible enough that I can step in to interrupt, question, or redirect it at any point.\nI Started SSHing Into My Laptop One of my earliest habits here was small and almost boring: I started SSHing into my own laptop to begin work; whether I was away at home, and even when I wasn\u0026rsquo;t far from it.\nBefore agent workflows, SSH had always felt useful but limited — fine for checking logs, restarting a process, or making a tiny edit, but anything larger felt uncomfortable. If I needed real context, I\u0026rsquo;d usually just wait until I was back at my desk with the IDE open.\nWith a terminal-native agent, that line moved.\nNow I could SSH into my laptop, open a repository, describe the task, and let the agent take the first step — inspecting the codebase, drawing up a plan, running the tests, and handing me back a diff. I wasn\u0026rsquo;t trying to turn a phone or a remote shell into a perfect workstation; I was simply giving the agent access to the workstation I already trusted.\nRemote debugging started to feel different too. When something broke while I was away from my desk, I no longer had to shrink the problem down into a note to deal with later — I could start debugging right where I was, letting the agent collect logs, search call sites, reproduce the failing command, and narrow the issue down while I watched the terminal from somewhere else entirely.\nNone of this is magic; it\u0026rsquo;s just a better division of labor. I still decide what matters and I still review every patch, but the boring first pass no longer requires me to be sitting in front of the exact screen where the repository happens to live.\nA remote debugging session has its own rhythm. I try to start by describing the symptom without overfitting to an answer — a request is timing out, a test is flaky, a service only falls over after a particular configuration change — and then I ask the agent to gather facts before it edits anything: check the recent logs, find the code path, identify the command that reproduces the issue, and tell me what changed, but don\u0026rsquo;t patch yet.\nThat pause matters. When I\u0026rsquo;m remote I have far less patience for chaos, and a tiny screen makes every wrong turn feel more expensive than it would at my desk. I want that first phase to feel like triage — collecting signals, reducing uncertainty, and naming the most likely branch of the problem — and only once that\u0026rsquo;s done do I want any edits.\nThe interesting part is how calm the whole session can feel. I might be holding a phone in one hand, but the laptop back home is the thing doing the real work: the repository is already cloned, the dependencies are already installed, and the test cache is still warm. I\u0026rsquo;m not coding on the phone at all — I\u0026rsquo;m steering work that\u0026rsquo;s happening somewhere far better suited for it.\nThe Five-Hour Reset Strategy There was another habit from the earlier days that now feels like a snapshot from a short but strange era.\nBack when agent limits were tighter, I used to plan my whole day around the reset window. I\u0026rsquo;d prepare a careful prompt the night before — collecting context, writing down the goal, listing the files I expected to matter, and generally making the task as clean as I could before going to sleep.\nThen I would start it before 7am.\nThe rough plan was simple: use the morning reset window as one long, uninterrupted run, let the agent work for roughly five hours, and then, after lunch around 12pm, take a fresh reset and start again with everything I\u0026rsquo;d learned in the meantime.\nThe routine sounds a little absurd now, but it made complete sense at the time. The scarce resource wasn\u0026rsquo;t only model intelligence — it was continuity, and if I burned the first hour just explaining the repository, I\u0026rsquo;d already lost a big chunk of the useful window. So I learned to treat the prompt like a launch checklist.\nTerminal-native work made that strategy better, because the environment was already sitting right next to the code; the agent could run the plan in the same place the repository, tests, and logs lived. But it was still, at heart, a strategy built around scarcity — prepare hard, start early, make the window count, then wait for the next reset.\nIt changed how I planned, too. The night-before prompt stopped being a simple request and turned into a compressed briefing: the objective, the suspected files, the commands I trusted, the tests that mattered, and the things the agent absolutely should not touch. I\u0026rsquo;d add whatever I had already tried so the morning run wouldn\u0026rsquo;t repeat yesterday\u0026rsquo;s mistakes, and if there was a risky path, I\u0026rsquo;d spell out the safer first step alongside it.\nThat planning was genuinely useful, but it also made the whole workflow feel ceremonial — I had to prepare the runway before the plane could take off. If I discovered at 8am that the prompt was missing a key constraint, the entire run could drift, and if the agent spent too long exploring the wrong layer, I could almost feel the reset window burning down in real time.\nThe best prompts from that period read almost like small design documents, with a goal, a scope, constraints, verification steps, and an exit condition. That habit outlasted the scarcity that created it: I no longer want to plan my life around a five-hour window, but I still want to hand an agent a crisp definition of done.\nThe Android and Termux Phase For a while, the most mobile version of this workflow ran on Android and Termux.\nTermux is a terminal environment for Android, and it was the thing that made the whole idea feel real. At first the phone was mostly a way to SSH into my laptop and reach the development setup that already worked — a remote control for a machine I trusted, more than a development environment of its own.\nThe setup had a real charm to it. It also had no shortage of friction.\nIt asked for a lot: an Android device, a terminal emulator that felt tolerable on a small screen, Virtual Private Network(VPN) access, and then keys, shell config, kube config, fonts, keyboard shortcuts, and just enough environment setup that the whole thing wouldn\u0026rsquo;t collapse the moment I switched networks.\nWhen it worked, it felt amazing — I could be nowhere near my desk and still kick off a meaningful investigation, with a far more capable machine doing the heavy lifting while I just held the small screen.\nAnd when it failed, it failed in thoroughly ordinary ways: the VPN wouldn\u0026rsquo;t connect, the keyboard would hide the line I was typing, a key was missing, a session dropped. The terminal itself was still perfectly portable; the path to that terminal just wasn\u0026rsquo;t always smooth.\nThe brittleness was rarely one dramatic failure so much as a chain of small edges catching at once. A mobile network would slip from Wi-Fi to cellular and the VPN tunnel would quietly die; the SSH session would hang on for a moment, then freeze at exactly the point where the agent was asking for confirmation; the phone keyboard would autocorrect a command flag or slip an invisible newline into precisely the wrong place.\nThen there was environment drift. My laptop shell had years of accumulated muscle memory baked into it — aliases, package versions, default editor settings, SSH config, prompt behavior, and a handful of tiny scripts I\u0026rsquo;d forgotten I even depended on. Recreating just enough of that on Android was possible, but it never quite felt identical, and every missing detail was another reminder that the phone wasn\u0026rsquo;t really the development environment at all. It was a narrow doorway into one — and the real environment always lived somewhere on the other side of that door.\nStill, that phase taught me something that stuck: the value was never the phone. The value was that the active development surface had become reachable through plain text.\nFrom Laptop to Direct Pod Access Kubernetes changed the route more than it changed the idea.\nThe old path still ran through my laptop. From a phone or some other remote device, I\u0026rsquo;d SSH into the laptop first and, from that familiar shell — the one that already had the VPN, kube config, credentials, and aliases baked in — reach the Kubernetes pod or development namespace where the service was running, then inspect it for logs, configuration, and running processes. The phone was never talking to the pod directly; the laptop was the bridge, not the place where the problem actually lived.\nDirect access changed that path. It didn\u0026rsquo;t turn the pod into production, and it didn\u0026rsquo;t remove the need for care — it simply removed one hop. Instead of arriving at the relevant logs, config, and processes only after passing through the laptop, I could start right there and keep more of the debugging context in a single place.\nThe Android and Termux setup made that route shorter still. With Termux, VPN access, and enough kube config on the device, I could sometimes go from Android straight into the Kubernetes environment instead of bouncing through the laptop, and the same small screen was suddenly wired directly to the place where the service lived.\nThe point was never that every task should happen inside a pod — that would be a bad rule to live by. The point was that the agent simply didn\u0026rsquo;t care where the shell came from, whether that was my laptop, a remote virtual machine, Android, or a container. As long as the files, commands, logs, and permissions were there, the loop still ran.\nThat made direct pod access genuinely useful on the go. When I needed to inspect behavior close to the running service, I could drop into the right development environment and ask the agent to help read logs, check configuration, or trace a failing path — without first having to recreate the entire world locally just to begin thinking.\nIn practice, I want the agent to behave like a careful operator in that setting: read before changing, prefer observation over mutation, keep facts separate from guesses, and stop to ask whenever a command might touch a shared environment. The terminal makes those boundaries easy to express, precisely because the work is already command-shaped to begin with.\nThis is where the lifestyle side and the technical side meet. The lifestyle benefit is that I can investigate while I\u0026rsquo;m nowhere near the desk; the technical requirement is that the environment still has to be treated with respect. Remote convenience doesn\u0026rsquo;t lower the need for operational discipline — if anything, it quietly raises it.\nAll of this still demands care, of course. Production systems need firm boundaries, credentials deserve respect, and some commands should never be delegated casually. But for development pods and controlled environments, a terminal agent fits naturally, because Kubernetes work is already command-shaped from the start.\nMobile Code-Work Is No Longer Android-Only The newer change is that this mobile workflow is no longer tied to Android and Termux at all.\nClaude Code\u0026rsquo;s remote-work feature is one example of that shift. I don\u0026rsquo;t see it as the whole story, and I don\u0026rsquo;t think any single tool owns this pattern, but it does show where the workflow is heading: the remote machine holds the development environment, while the human interacts through a much lighter client.\nThat matters because the old setup carried far too many prerequisites — an Android device, a terminal emulator, a VPN, SSH keys, environment setup, and enough patience to make all of it feel normal.\nNow the same basic idea can play out through the Claude app. I can be away from my desk, open the app, and reach work that\u0026rsquo;s already running on a remote machine. The agent still needs a real environment — files, commands, tests, and permissions — but the human entry point is no longer a hand-built mobile terminal stack.\nThat\u0026rsquo;s a meaningful lifestyle change. The phone has stopped pretending to be the workstation and become a control surface instead, while the work happens where it actually should, on a machine that already has the right setup.\nThis lowers the entry barrier in a very practical way. I no longer have to wonder whether this particular device has the right terminal profile, whether the VPN happens to be cooperating, or whether my mobile keyboard can even send the key combination I need — I just open the app, find the remote work context, and pick up from there.\nNone of this removes responsibility, though. If anything, a smoother entry point makes review habits matter more, because the easier it becomes to start work, the more deliberate I have to be about which work I start. A remote agent should still show me the plan, the files it touched, the commands it ran, and the evidence that the change actually works — convenience is never a substitute for ownership.\nI also try to keep the shape of the task small whenever I start from the app. A mobile session is great for opening an investigation, asking for a focused patch, checking a test result, or turning a vague bug report into a concrete plan. The lower barrier helps me start; I just can\u0026rsquo;t let it trick me into lowering the standard for finishing.\nThis is also why I think the terminal still matters even when the visible UI isn\u0026rsquo;t one. The execution layer underneath is still command-shaped: the agent may be launched from an app, but the useful work keeps happening through shells, files, processes, logs, and tests.\nThe IDE Is Not Dead This is not an \u0026ldquo;IDE is dead\u0026rdquo; post.\nI still genuinely like IDEs. They\u0026rsquo;re excellent for reading unfamiliar code, stepping through a debugger, exploring types, making careful refactors, and putting a final polish on a change before review without burning through thousands of reasoning tokens to do it. There\u0026rsquo;s a kind of deep local attention that an IDE supports better than any remote agent session ever will.\nThe role is changing, though.\nThese days I think of the IDE as the place for deep local work and polish — where I inspect design, smooth out rough edges, rename things carefully, and read code with full visual context around me. The terminal agent plays a different role entirely: it\u0026rsquo;s the portable execution layer, the place where a goal can turn into commands, patches, tests, and logs from almost anywhere.\nThe two roles fit together more naturally than they compete. The agent can take the first pass while I\u0026rsquo;m away from the desk, and later I can open the IDE and review it all with more care — or I can work the other way around, using the IDE to understand a tricky area first and then sending the agent into the terminal to apply a repetitive change across the whole repository.\nThe mistake is treating any of this as a fight between screens. It\u0026rsquo;s far more useful to ask what kind of attention each tool is actually good at supporting.\nContext Engineering Becomes Environment Engineering In the copy-and-paste era, prompt engineering mostly meant writing a better request: give the model a role, add a few examples, ask for a step-by-step answer. That still helps, but it stops being enough the moment the agent can actually act.\nThe deeper skill is context engineering — deciding which files the agent should read, which tests define success, which commands are safe, which directories are off limits, what it should do when a test fails, and what evidence it needs to leave behind when it claims the work is done.\nOnce work moves across SSH sessions, Termux, Kubernetes pods, and remote machines, context engineering becomes environment engineering too.\nThe question is no longer only \u0026ldquo;What should I tell the model?\u0026rdquo; but also, just as much, \u0026ldquo;Where should this work actually run?\u0026rdquo;\nSometimes the answer is my laptop, because it holds the full repository and all my local tools. Sometimes it\u0026rsquo;s a development pod, because the bug only shows up near the service mesh, the secrets, or the cluster configuration. And sometimes it\u0026rsquo;s a remote machine, because I want the work to keep going even if my phone locks or my network drops out from under me.\nThe quality of the agent depends heavily on that environment. A smart model in the wrong shell can still waste an enormous amount of time, while a well-framed task in the right shell can quietly make progress while I\u0026rsquo;m on a train, waiting for coffee, or walking between meetings.\nI\u0026rsquo;ve started to think about this the way I\u0026rsquo;d think about giving a teammate access. The question isn\u0026rsquo;t only whether the teammate is capable; it\u0026rsquo;s whether they\u0026rsquo;re standing in the right room with the right tools in hand. The prompt sets the intent, but the environment sets the reality.\nWhat Changed in the Texture of Work The biggest change isn\u0026rsquo;t that I type fewer characters, even though that\u0026rsquo;s true. That was never really the point.\nThe bigger change is that work became less attached to one chair.\nI can start a debugging thread before I\u0026rsquo;ve even reached the office, check a failing test from a remote session, or ask an agent to inspect a development pod while I\u0026rsquo;m nowhere near my laptop. I can let a remote machine hold all the messy environment while I use a much smaller device as nothing more than the steering wheel.\nThis changes the emotional shape of small tasks. A bug report no longer has to harden into a reminder for later, and a refactor idea doesn\u0026rsquo;t have to wait until I\u0026rsquo;ve rebuilt my whole mental stack back at the desk — some of the work can start while the context is still fresh in my head.\nThere\u0026rsquo;s a real risk in this, though. Portability can quietly turn into pressure, because if work can happen anywhere, it can also start to invade everywhere. The goal was never to code from every possible place; it was to remove the bad kind of waiting — the waiting caused by environment friction rather than by genuine thinking.\nFor me, the best version of this workflow isn\u0026rsquo;t constant work at all — it\u0026rsquo;s a smoother handoff. Start something remotely, gather the evidence, let the agent try the boring path first, and then come back later to a diff, a set of logs, and a much clearer question than the one I left with.\nThat handoff can be surprisingly humane. I can begin with nothing more than a messy thought — something feels off about this endpoint, or this refactor probably broke a path I haven\u0026rsquo;t checked yet — and the agent turns it into a first pass while I\u0026rsquo;m still in motion. By the time I sit down with a full keyboard and a larger screen, I\u0026rsquo;m not starting from anxiety anymore; I\u0026rsquo;m starting from artifacts.\nThe Black Screen Is Not Nostalgia The terminal didn\u0026rsquo;t come back because developers wanted to cosplay as old-school system administrators. It came back because it\u0026rsquo;s one of the cleanest interfaces we have between intention and execution.\nAgents made that more obvious.\nAgents need text instructions, but they also need a world to act on, and the CLI hands them that world in a form they can both read and change. SSH makes that world reachable, Termux proved that even a phone could become a serious entry point, Kubernetes makes the same pattern useful right next to real services, and remote-work features in tools like Claude Code strip away most of the mobile setup that used to stand between me and that same loop.\nThe result isn\u0026rsquo;t a future where every developer abandons the IDE — I genuinely don\u0026rsquo;t want that future. It\u0026rsquo;s a more layered workflow, where the IDE stays a strong place for deep attention, the terminal becomes the portable execution layer, and the agent sits inside that layer turning intent into observable work.\nThat\u0026rsquo;s why I keep coming back to the black screen. Not because it\u0026rsquo;s old, but because it travels well.\nWrapping Up Writing this all down, I realize the real change was never a single tool — it was a shift in how I approach the work itself. Terminal or IDE, SSH or Termux or Kubernetes, what matters now is that I\u0026rsquo;ve started asking, every single time, \u0026ldquo;Where, and in which environment, should this work actually run?\u0026rdquo; Agents widened the reach of my hands, but deciding what those hands touch is still entirely on me. Finding that balance has quietly become the most interesting kind of engineering I do these days.\nAnd none of this happened in isolation. HyperAccel is full of colleagues weaving AI into their workflows in their own ways, and one person\u0026rsquo;s trial and error often becomes the next person\u0026rsquo;s starting point. I\u0026rsquo;d like to think of this post as one small piece of that shared notebook.\nUpcoming\u0026hellip; If this post was about where the work should run, the next one goes a step deeper into how we actually design those environments — how a prompt becomes a small design document, the guardrails and verification steps we hand an agent, and the exact point where context engineering turns into environment engineering, all with more concrete examples from how we work.\nAs the teams at HyperAccel keep finding new ways for AI to melt into our workflow, we\u0026rsquo;ll continue sharing the missteps and the insights along the way.\nHyperAccel is Hiring! The reason why we are writing operating this tech blog is largely too attract top talent!\nIf you are interest in the technologies we work with, and want to join this revolution, please apply at HyperAccel Career!\nHyperAccel is full of brilliants engineers. We will be waiting.\n","date":"2026.06.16","permalink":"https://hyperaccel.ai/techblog/en/posts/moving-back-to-terminals/","series":["AI Agent Series"],"style":"--c:#8EE8E0;--on:#02121c","summary":"AI coding agents did not just change how we write code. They changed where work can happen: over SSH, inside Termux, in Kubernetes pods, and from remote machines that no longer need a full local editor setup.","title":"[AI Agent - Part 02] Back to the Terminal: How Agents Made Remote Development Feel Native"},{"authors":["Seungbin Shin"],"category":"Memory \u0026 Storage","content":" This is Part 4 of the Memory in the AI Era series. Part 1, Part 2, and Part 3 covered High Bandwidth Flash (HBF), which fills an empty seat in the memory hierarchy right next to the GPU. From this part on, we shift our viewpoint outside the GPU, to the system level, and look at where Compute Express Link (CXL) fits in the memory hierarchy.\nIntroduction Hello, I\u0026rsquo;m Seungbin Shin, an RTL Designer at HyperAccel.\nAcross Parts 1-3, we took a deep look at the memory layer right next to the GPU. HBM stacked DRAM to push up bandwidth, and HBF wrapped NAND in HBM\u0026rsquo;s packaging to bring in capacity.\nBut when you talk about data center memory, there\u0026rsquo;s a name that keeps coming up from a seat other than the one next to the GPU. That name is CXL.\nIt\u0026rsquo;s the keyword that never fails to appear in conversations like \u0026ldquo;building a memory pool,\u0026rdquo; \u0026ldquo;sharing memory between VMs,\u0026rdquo; or \u0026ldquo;tiering an AI server\u0026rsquo;s hot/warm/cold data.\u0026rdquo;\nAnd yet, when you actually search for CXL, even the acronyms are bewildering. There\u0026rsquo;s CXL.io, CXL.cache, and CXL.mem; there are Type 1/2/3 devices; and the versions run from 1.1 to 2.0, 3.0, 3.1, and 3.2. It\u0026rsquo;s an interface with a fairly high barrier to entry.\nSo in this part, I want to start from a single question.\n\u0026ldquo;We already have both DDR and PCIe — so why was yet another new interface needed?\u0026rdquo;\nAnswering this question naturally reveals where CXL settles in the memory hierarchy, and why the big three memory vendors (Samsung, SK hynix, Micron) are all scaling up their CXL product lineups at once.\nThe content of this post is based on my personal study and experience. If you find any errors, please let me know in the comments.\nWhy Yet Another Interface — The Empty Seat Between DDR and PCIe The short answer: DDR can\u0026rsquo;t be scaled up, and PCIe is hard to use as memory.\nWithin a single server, there are broadly two paths the CPU uses to move data.\nDDR channels: a memory-dedicated bus connecting to the DRAM modules right next to the CPU PCIe: a general-purpose serial bus connecting to peripherals like GPUs, NICs, and SSDs Each path has a clear role it does well. But both paths have hit their limits.\nThe Limit of DDR: The Channel Count Is the Ceiling To attach more DDR DRAM, the CPU has to add more memory channels. But adding channels increases the pin count on the CPU package, and the motherboard wiring grows more complex along with it.\nEven the latest generations of server CPUs top out at around 8-16 channels, with the number of modules per channel limited to 1-2.\nAs a result, the DRAM capacity of a single socket is capped at a few TB. Beyond that, it\u0026rsquo;s physically impossible to attach more.\nAs the data an LLM inference node handles (KV cache, model weights, embeddings) swells to tens of TB, brushing up against this ceiling is becoming more and more common.\nThe Limit of PCIe: Fast, But No Coherency So what if we attach memory over PCIe? On a 16-lane link, PCIe Gen5 provides about 64 GB/s per direction (128 GB/s aggregate across both directions). That\u0026rsquo;s comparable to a single DDR5 channel (about 50 GB/s).\nAs it happens, this 64 GB/s figure already showed up in Part 3 — it\u0026rsquo;s the very same PCIe that was singled out as the \u0026ldquo;bandwidth ceiling\u0026rdquo; when attaching HBF next to the GPU. But from the standpoint of memory expansion, the bandwidth itself isn\u0026rsquo;t really lacking. The real obstacle lies elsewhere.\nThere are two problems when you attach memory over PCIe.\nFirst, there\u0026rsquo;s no cache coherency. Cache coherency is the property that, even when multiple agents (CPU cores, devices) each cache the same memory address, a change by one is always reflected so the others see the latest value. DDR memory sits inside the CPU\u0026rsquo;s coherency domain, so the CPU\u0026rsquo;s cache hierarchy and memory controller maintain this automatically; PCIe sits outside that domain. For the CPU to directly load/store the memory of a PCIe device, this coherency has to be guaranteed at cache-line granularity. PCIe is fundamentally a packet-based I/O protocol, so it offers no such guarantee. To use PCIe memory like main memory, the OS has to explicitly copy the data over (e.g., via memcpy).\nSecond, the memory semantics are coarse. PCIe operates in transaction units larger than the 64B cache line. You can\u0026rsquo;t do fine-grained access in the small units that memory requires.\nYou can attach memory over PCIe, but from the CPU\u0026rsquo;s point of view, that memory is \u0026ldquo;someone else\u0026rsquo;s memory,\u0026rdquo; not \u0026ldquo;my memory.\u0026rdquo;\nThe Empty Seat: A Cache-Coherent Memory Interface If we tie everything so far into a single line, it\u0026rsquo;s this:\nWe need something that is a coherent memory interface like DDR, yet can extend outside the system like PCIe.\nThe standard that emerged to fill this empty seat is exactly CXL.\nCXL is not a physical layer built from scratch. It\u0026rsquo;s a standard that borrows PCIe\u0026rsquo;s physical layer (PHY) as-is and layers a new, cache-coherent protocol on top of it.\nA high-speed interconnect standard that adds a cache-coherency contract on top of the off-the-shelf PCIe 5.0 infrastructure — that\u0026rsquo;s the core idea of CXL.\nThe Three Faces of CXL — CXL.io / CXL.cache / CXL.mem CXL isn\u0026rsquo;t a single protocol but a bundle of three sub-protocols. All three flow simultaneously over a single PCIe link.\nSub-protocol Role Model CXL.io Device discovery, configuration, DMA Same as PCIe CXL.cache Device caches host memory Device → Host CXL.mem Host directly accesses device memory Host → Device The names alone are confusing, so let me unpack each role.\nCXL.io — The Same Foundation as PCIe CXL.io is effectively identical to PCIe. It handles discovering devices (enumeration), reading configuration space (configuration), delivering interrupts, and moving large data via DMA.\nEvery CXL device must implement CXL.io. This is exactly why CXL was built as a standard layered on top of PCIe — so that the existing PCIe infrastructure (OS drivers, BIOS, controller IP) can be reused as-is.\nCXL.cache — The Device Caches CPU Memory Consider a case where a device like a GPU or an accelerator frequently reads the CPU\u0026rsquo;s main memory.\nWith PCIe, you\u0026rsquo;d have to fetch the data from main memory every time. But what if you put a small cache inside the device and stored frequently used data there? As long as there\u0026rsquo;s a guarantee that the CPU and the device are looking at the same data, you could dramatically cut down on communication.\nCXL.cache does exactly this. It lets a device cache host memory at cache-line (64B) granularity and keeps that cache coherent with the CPU\u0026rsquo;s caches.\nInternally, it\u0026rsquo;s a way of extending the CPU\u0026rsquo;s cache-coherency protocol (e.g., MESI) across the CXL link. What state (Modified/Exclusive/Shared/Invalid) a device takes a cache line in, and how to handle the case where another caching agent already holds that line — all of it is standardized.\nCXL.mem — The CPU Treats Device Memory Like Main Memory There\u0026rsquo;s also the reverse direction. This is the case where you attach a lot of memory (DRAM, or beyond) to a device, and the CPU wants to load/store that memory like its own main memory.\nCXL.mem takes on this role. From the CPU\u0026rsquo;s perspective, the device memory appears as a region within its own physical address space, accessible directly with ordinary load/store instructions.\nThe important point here is that the CPU plays the role of the home agent. In a case like Type 3, where the device only holds memory, it\u0026rsquo;s enough for the host to manage coherency on its own.\nBy contrast, in a case like Type 2, where the device also has its own cache, bias-based coherency is applied — explicitly marking whether the \u0026ldquo;owner\u0026rdquo; of a given cache line is the device or the host. Since whichever side owns the line is responsible for its coherency, bidirectional coherency traffic can be greatly reduced.\nIt sounds complex, but the actual implementation burden is fairly light on the device side. For Type 3, the device just has to respond to read/write requests like a memory controller.\nCombining the Three Protocols A CXL device picks and implements among the three protocols depending on what it does.\nA device with no memory of its own that only caches host memory: CXL.io + CXL.cache A device that has memory and just needs the host to use it like its own memory: CXL.io + CXL.mem A device that has memory and also needs to cache host memory: CXL.io + CXL.cache + CXL.mem These three combinations become the basis for the Type 1 / Type 2 / Type 3 devices covered in the next section.\nWhat Do You Plug In — Type 1 / 2 / 3 Devices Depending on how you combine the three sub-protocols, CXL devices fall into three types.\nType 1 Type 2 Type 3 Protocols used CXL.io + CXL.cache CXL.io + CXL.cache + CXL.mem CXL.io + CXL.mem Device\u0026rsquo;s own memory None Yes Yes Caches host memory Yes Yes No Representative example SmartNIC, FPGA accelerator GPU, AI accelerator Memory expansion module Let me unpack the role and use cases of each type.\nType 1 — The Accelerator That Only Caches A Type 1 device has no memory of its own and caches host memory.\nThe representative candidate is the SmartNIC. When processing packets coming in from the network, it frequently reads the host\u0026rsquo;s descriptor queues and packet buffers. Instead of making a round trip all the way to host memory every time, keeping these in the NIC\u0026rsquo;s internal cache and maintaining coherency can greatly reduce packet-processing latency.\nType 1 shows its value in workloads that frequently reference host data structures, as with FPGA accelerators.\nType 2 — The Accelerator With Both Memory and Cache A Type 2 device has its own memory and also caches host memory. It uses all three protocols.\nThe typical example is a GPU or AI accelerator attached over CXL. The accelerator has its own HBM/GDDR, but it also fetches and caches model weights or input data from host memory. It pulls host data via CXL.cache and exposes the accelerator\u0026rsquo;s memory to the host via CXL.mem.\nThe catch is that Type 2 is the most complex type to implement. Since coherency has to be managed in both directions, the burden on the device-side controller is large. That\u0026rsquo;s why there aren\u0026rsquo;t many Type 2 devices in actual mass production yet.\nType 3 — The Most Commonly Seen Memory Expansion Module A Type 3 device only has its own memory and does not cache host memory. It only needs to implement CXL.io and CXL.mem.\nPut simply, it\u0026rsquo;s \u0026ldquo;a memory module with a CXL interface.\u0026rdquo; From the host\u0026rsquo;s perspective, it looks like a DDR module placed one step farther away, accessible with ordinary load/store.\nMost of the CXL products currently in mass production — or right on the verge of commercialization — are Type 3. The reasons are clear.\nIt has the lowest implementation complexity, it most directly relieves the DDR channel ceiling, and it leads naturally into the product lineups of existing memory vendors. The CXL Memory Module (CMM) lineups from the big three memory vendors, which we\u0026rsquo;ll look at later, are all Type 3 as well.\nFrom a Single Device to a Fabric — CXL 1.1 → 2.0 → 3.x CXL has been evolving rapidly since 1.0 came out in 2019. The features added in each version ultimately point in one direction.\nLetting memory be placed farther, in greater amounts, and shared by more hosts.\nLet me touch on the key change in each version.\nCXL 1.1 — Directly Connected to a Single Host This is the most basic form. A single CXL device is connected directly, like a PCIe slot, to a single host CPU.\nAt this stage, CXL is a tool for \u0026ldquo;attaching extra memory one step farther out, next to DDR.\u0026rdquo; Increasing the memory capacity of a single node is the main goal; it doesn\u0026rsquo;t extend to other nodes\u0026rsquo; memory.\nCXL 2.0 — The Switch and Memory Pooling In 2.0, which arrived in 2020, a decisive change occurs. The CXL switch is introduced, making it possible to weave multiple hosts and multiple memory devices together like a mesh.\nThe most striking application is memory pooling. Data center memory utilization averages around 40-50%. It\u0026rsquo;s common for some servers to be short on memory while others sit more than half empty.\nIn CXL 2.0, multiple hosts can dynamically allocate as much as they need from a single memory pool.\nThere\u0026rsquo;s one thing to be careful about. The \u0026ldquo;sharing\u0026rdquo; here is dynamic allocation, not concurrent sharing.\nA large memory device is split into multiple chunks in the form of a Multi-Logical Device (MLD). A control plane called the Fabric Manager (FM) then hands them out: \u0026ldquo;this chunk to host A, that chunk to host B.\u0026rdquo; A single chunk is owned by only one host at any given moment. So there\u0026rsquo;s simply no region where a cache-coherency conflict could even arise.\nTo use a hotel-room analogy: a single room (chunk) is rotated as one guest (host) checks in and out — two guests don\u0026rsquo;t enter the same room at the same time.\nThis model alone is enough to reduce the practice of over-buying memory for each server based on worst-case assumptions. At this stage, CXL is no longer a single-node expansion interface. It begins to become the foundation for rack-level memory resource management.\nCXL 3.x — The Fabric and Multi-Host Coherence The flow continuing through 3.0 (2022), 3.1 (2023), and 3.2 (2024) gets even more ambitious.\nThere are two key points.\nFirst, from dynamic allocation to true sharing. Where 2.0\u0026rsquo;s pooling was \u0026ldquo;one chunk = one host at a time (rotated between hosts),\u0026rdquo; in 3.x multiple hosts read and write the same memory region simultaneously while hardware maintains cache coherency. This is also called Global Fabric Attached Memory (GFAM). Allocation (granting access) doesn\u0026rsquo;t disappear here either — the difference is that where 2.0 gave one chunk to one host exclusively, 3.x lets the Fabric Manager grant multiple hosts simultaneous access to the same region. From this stage on, real conflicts can occur, and the home agents on the host and device resolve them at cache-line granularity.\nSecond, from port-level connections to a fabric connection. 3.x ties together many CXL switches to form a large-scale fabric. Here, Port-Based Routing (PBR) routes by destination port ID — which, unlike the older tree structure, lets devices be arranged in various topologies such as mesh, dragonfly, and 3D torus, and scales to large device counts.\nOnce you reach this stage, CXL effectively establishes itself as a new interconnect standard inside the data center. It even starts to become a point of comparison with accelerator interconnects like NVIDIA\u0026rsquo;s NVLink and UALink (an open standard for connecting accelerators).\nAs of 2026, most products in mass production support CXL 2.0. The 3.x standard is out, but support in actual devices and host CPUs is only just catching up.\nThe CXL Blueprint the Big Three Memory Vendors Are Drawing Now that we\u0026rsquo;ve covered the big picture of CXL, let\u0026rsquo;s come down to actual products.\nWhat\u0026rsquo;s interesting is that even though the standard is led by consortium members like Intel, AMD, Microsoft, and Meta, the side rolling out mass-production product lineups most aggressively is the big three memory vendors. Samsung, SK hynix, and Micron each announced CXL memory modules of a similar shape at nearly the same time.\nHere\u0026rsquo;s a summary of the three companies\u0026rsquo; product lineups.\nCompany Representative product Form factor Key feature Samsung CMM-D (CXL Memory Module - DRAM) E3.S Simple memory expansion, CXL 2.0 Samsung CMM-B (CXL Memory Module - Box) Box appliance Rack-level memory pooling Samsung CMM-H (CXL Memory Module - Hybrid) E3.S DRAM + NAND hybrid SK hynix CMM-DDR5 E3.S DDR5-based memory expansion SK hynix CMM-Ax E3.S Memory + compute engine integration Micron CZ120 / CZ122 E3.S Memory expansion module It stands out that most use the E3.S form factor. It\u0026rsquo;s a hot-swappable standard form factor commonly seen in server storage, and since there\u0026rsquo;s already accumulated know-how for data center deployment, the adoption barrier is low.\nCapacities run around 96 GB - 256 GB depending on the model, and the interface commonly uses PCIe Gen5 x8. The three companies\u0026rsquo; specs are surprisingly similar. The real difference comes down to \u0026ldquo;will you stop at a memory module, or take it one tier further?\u0026rdquo;\nThe First Line: Honest Memory Expansion (Type 3 As-Is) The most basic product is simply a DDR5 memory module with a CXL interface.\nSamsung\u0026rsquo;s CMM-D, SK hynix\u0026rsquo;s CMM-DDR5, and Micron\u0026rsquo;s CZ120/CZ122 all fall into this category. From the host\u0026rsquo;s perspective, it looks like \u0026ldquo;a DDR module placed a bit farther away,\u0026rdquo; accessed with ordinary load/store. The target workload is clear. It\u0026rsquo;s workloads that have hit the per-socket DRAM capacity ceiling.\nin-memory DBs and large-scale analytics KV cache and embedding stores placed on the CPU side in LLM inference memory-starved nodes in VM consolidation environments This is the first battleground, where the three companies compete with nearly identical specs.\nThe Second Line: Pooling and Compute Beyond the Module The truly interesting side is next. The big three memory vendors don\u0026rsquo;t stop at simple expansion modules; they\u0026rsquo;re trying to pull CXL\u0026rsquo;s follow-on features (pooling, compute) into their own products.\nThere are two representative directions.\nBundling into a pooling box. Samsung\u0026rsquo;s CMM-B (Box) packs multiple CMM-D cards into a single appliance to form a rack-level memory pool. It embeds a CXL 2.0 switch so that multiple hosts can allocate as much memory as they need from this box.\nPutting a compute engine inside the memory module. SK hynix\u0026rsquo;s CMM-Ax is the representative example, and Samsung draws a similar picture with CMM-H and CXL-PNM.\nThis idea is one branch of Near-Memory Processing (NMP) or Processing-in-Memory (PIM). Instead of fetching data and then computing, it\u0026rsquo;s a way of computing right where the data is and bringing back only the result.\nThe candidate workloads are things like:\nvector embedding similarity comparison (RAG) database filtering and aggregation some stages of LLM inference (e.g., KV cache compression, top-k selection) For this model to be meaningful, it has to be a workload where the cost of moving data across CXL is greater than the cost of computing it. Here, \u0026ldquo;cost\u0026rdquo; means the time (latency) of moving the data and the bandwidth and power it consumes. Instead of pulling a huge embedding collection over to the host for similarity comparison, finishing it inside the memory module and passing back only the result reduces both traffic and latency.\nThe HBF + HAVEN combination covered in Part 2 — the idea of placing a huge vector DB on HBF and attaching a search engine right next to it to finish RAG similarity comparison on the spot — reappears in the CXL space as an NMP-integrated module.\nThe Fabric and IP Camp — From Controllers to Switch Silicon Beyond the big three memory vendors, there are players who never go missing from the CXL ecosystem. These are the companies that supply CXL controller/switch IP, or build their own silicon directly.\nAstera Labs: Its Leo series controllers are used in Microsoft Azure\u0026rsquo;s M-series CXL memory and elsewhere, aiming for the position of the \u0026ldquo;standard part\u0026rdquo; of CXL memory controllers. Marvell, Microchip (formerly Microsemi): They hold one axis of the IP camp with their controller lineups. The Micron CZ120 we saw earlier also uses Microchip\u0026rsquo;s SMC 2000 controller. Panmnesia: A Korean fabless startup that came out of KAIST\u0026rsquo;s CAMELab (Computer Architecture and Memory Systems Laboratory). There\u0026rsquo;s an interesting connection here. M. Jung, an author of the Memory Pooling With CXL paper cited in the references, is Professor Myoungsoo Jung of KAIST, who founded this company. It doesn\u0026rsquo;t stop at controllers — it\u0026rsquo;s a full-stack player that builds even CXL 3.2 fabric switch silicon (PANSWITCH) directly. It\u0026rsquo;s implementing 3.x\u0026rsquo;s most cutting-edge features, such as PBR and mesh/dragonfly/3D torus topologies, in silicon ahead of anyone else. In April 2026 it announced a second-half mass-production plan along with the supply of pre-release silicon, stepping into a full-fledged commercialization phase. Memory companies do sometimes design their own CXL controllers, but since the standard evolves so quickly, it\u0026rsquo;s common to outsource to — or collaborate with — IP/silicon suppliers. Especially in an area like the 3.x fabric, where the standard itself is only just settling, specialist players like Panmnesia appear to be a step ahead — a pretty interesting point even from the perspective of the Korean semiconductor ecosystem.\nWhy the Memory Companies Are All Entering CXL at Once There\u0026rsquo;s a common motivation behind the three companies releasing products in the same category at nearly the same time.\nIt\u0026rsquo;s an opportunity for memory to move up a level — from \u0026ldquo;a module you plug in\u0026rdquo; to \u0026ldquo;a seat it occupies within the system.\u0026rdquo;\nConcretely, there are three effects.\nExpanding sellable capacity per node: they can sell more memory beyond the DDR channel ceiling Room to raise module pricing: differentiation through add-on features like pooling and NMP Entry into the solutions business: proposals not just as a simple component, but at the level of a pooling appliance (e.g., CMM-B) If HBM grew the memory business next to the GPU, CXL is a card that can regrow the memory business next to the CPU — at the system level. That\u0026rsquo;s why it\u0026rsquo;s not strange at all that the three companies are releasing similar product lineups at similar times.\nSo Is It Actually Usable — The Remaining Homework Hearing all this, CXL might look like a cure-all, but at the actual adoption stage there\u0026rsquo;s homework you can\u0026rsquo;t ignore.\nLatency Tax Memory attached over CXL is slower than direct-attached DDR memory.\nDDR5 native: about 80-100 ns CXL memory (Type 3): about 170-300 ns That\u0026rsquo;s about 2-3x the latency. This is an intrinsic cost that comes from the structure of layering a protocol on top of the PCIe PHY.\nGiven this cost, CXL memory naturally enters not as a replacement for all of main memory, but as the second tier in a 2-tier memory configuration of \u0026ldquo;nearby DDR + one-step-farther CXL.\u0026rdquo;\nSoftware Stack Maturity From the CPU\u0026rsquo;s perspective, CXL memory looks like \u0026ldquo;a slightly slow NUMA node.\u0026rdquo; So the OS and runtime have to decide which data to keep in nearby DDR and which to send to CXL.\nThe Linux kernel is rapidly putting the CXL driver, hot/cold page tracking, transparent page migration, and so on in order, but per-workload optimal tuning still needs time.\nIn particular, for a workload with a deterministic memory access pattern, like LLM inference, it\u0026rsquo;s more efficient for the application to decide data placement directly than to rely on the OS\u0026rsquo;s automatic tiering. This is still an area with plenty of room to pioneer.\nThe Empty Seat of Type 2, the Type 3-Centric Reality On paper, Type 1/2/3 are all defined, but the actual market runs around Type 3. This is due to the combination of Type 2\u0026rsquo;s bidirectional-coherency complexity and the fact that GPU vendors are pushing their own interconnects (NVLink and the like).\nWhether CXL really goes all the way to being \u0026ldquo;the accelerator interconnect standard,\u0026rdquo; or settles in as a memory expansion + pooling standard, is still an open question.\nWrapping Up \u0026amp; Next Installment Preview In this post, we looked at what empty seat CXL fills in the memory hierarchy and what parts it\u0026rsquo;s made of.\nHere\u0026rsquo;s the summary:\nThe empty seat: DDR has a clear channel ceiling within a single node, and PCIe has no coherency, so it\u0026rsquo;s hard to use as memory. The seat in between. The three faces: CXL.io (same as PCIe), CXL.cache (the device caches the host), CXL.mem (the host directly accesses device memory). The three types: Type 1 (cache only), Type 2 (memory + cache), Type 3 (memory expansion). The market is Type 3-centric. Standard evolution: 1.1 (single host) → 2.0 (switch, pooling) → 3.x (fabric, multi-host coherence). The big three\u0026rsquo;s CMM: the first line is honest memory expansion (Samsung CMM-D, SK hynix CMM-DDR5, Micron CZ120). The second line is the pooling box (Samsung CMM-B) and compute integration (SK hynix CMM-Ax, Samsung CXL-PNM). The remaining homework: a 2-3x latency tax, an immature software stack, and the empty seat of Type 2. That said, this post has only answered the \u0026ldquo;what is CXL\u0026rdquo; stage. The truly interesting question is next:\nHow is CXL actually used in real LLM serving workloads, and which workloads does it fit well?\nIn the next Part 5, we\u0026rsquo;ll look at concrete use cases that leverage CXL — KV cache offload, VM consolidation through memory pooling, and hot/warm/cold tiering — and at where the big three\u0026rsquo;s CMM lineups can fit within them.\nP.S. I\u0026rsquo;m designing RTL at HyperAccel for the launch of an LLM acceleration ASIC chip. Extending the memory hierarchy from next to the GPU out to outside the GPU, all the way to the system level, makes the problems an accelerator designer has to solve even more varied. I hope this series helps us understand the flow of memory technology together and watch the changes ahead.\nHyperAccel is a company that covers HW, SW, and AI, bringing together outstanding talent across all domains. If you want to learn broadly and deeply while growing together, please apply to HyperAccel anytime!\nCareers: https://hyperaccel.career.greetinghr.com/ko/guide\nReference Compute Express Link Consortium — Specifications CXL 3.1 Specification Overview Intel — Compute Express Link Technology Samsung — CXL Memory Module (CMM-D / CMM-B / CMM-H) SK hynix CMM-DDR5 / CMM-Ax Product Briefs Micron CZ120 / CZ122 CXL Memory Expansion Module Astera Labs Leo CXL Smart Memory Controllers Panmnesia — Full Stack CXL Link Solution Panmnesia PANSWITCH — CXL 3.2 / PCIe 6.x Fabric Switch (pre-release silicon Apr 2026, mass production H2 2026) Linux Kernel CXL Documentation D. Gouk, M. Kwon, H. Bae, S. Lee, M. Jung, \u0026ldquo;Memory Pooling With CXL,\u0026rdquo; IEEE Micro, vol. 43, no. 2, pp. 48-57, 2023. DOI: 10.1109/MM.2023.3237491 D. Das Sharma, R. Blankenship, D. S. Berger, \u0026ldquo;An Introduction to the Compute Express Link (CXL) Interconnect,\u0026rdquo; ACM Computing Surveys, vol. 56, no. 11, Article 290, 2024. DOI: 10.1145/3669900 (arXiv:2306.11227) ","date":"2026.06.04","permalink":"https://hyperaccel.ai/techblog/en/posts/what-is-cxl/","series":["Memory in the AI Era"],"style":"--c:#009EE2;--on:#02121c","summary":"Next to the GPU, HBM and HBF fill the gap — but there\u0026rsquo;s another empty seat next to the CPU. We look at CXL, the new interface that fills the awkward gap between PCIe and DDR: its basic structure, device types, and the CXL product blueprints the big three memory vendors are drawing.","title":"[Memory in the AI Era - Part 04] Understanding CXL"},{"authors":["Jaewon Lim"],"category":"Memory \u0026 Storage","content":" This post is Part 3 of the Memory in the AI Era series. Part 2 covered where HBF can be effectively used (CAG, H³, and candidate workloads beyond them). This part walks through the new workloads HBF must confront and the remaining challenges it has to solve to reach commercialization.\nIntroduction Hello. I\u0026rsquo;m Jaewon Lim from the HyperAccel DV team.\nIf we summarize the conclusion of Part 2 in a single sentence:\nWith the right workload, HBF\u0026rsquo;s weaknesses can be hidden.\nWe looked at HBF\u0026rsquo;s weaknesses, the workload conditions that can overcome them, and concrete examples such as SK hynix\u0026rsquo;s H³ and Georgia Tech\u0026rsquo;s HAVEN.\nBut this is a way to hide HBF\u0026rsquo;s weaknesses, not to overcome them completely. It is a conditional idea that depends on multiple assumptions about the workload. If LLM workloads evolve in a direction that doesn\u0026rsquo;t really need HBF, demand for HBF will stay limited no matter how mature the technology becomes.\nOn top of that, HBF\u0026rsquo;s weaknesses are also NAND Flash\u0026rsquo;s weaknesses. HBF is a Flash-based alternative proposed to overcome HBM\u0026rsquo;s capacity limits, and Flash memory is already used in many forms across LLM serving today. So we need to first look at how Flash is actually deployed in production — only then can we get a real answer for how HBF should be used.\nIn this part, we will look at recent LLM workload trends, examine how Flash memory (SSD) is used in LLM serving environments, and then walk through the challenges HBF still has to solve for commercialization.\nRecent LLM workload trends One of the conditions Part 2 identified for hiding HBF\u0026rsquo;s weaknesses was that the workload has to be deterministic — because only then can NAND\u0026rsquo;s long latency be hidden by prefetching. The H³ evaluation rests on exactly this assumption: if we can predict which weights and which KV cache entries the next layer will read, HBF\u0026rsquo;s weakness can be hidden.\nBut the way recent frontier models are being designed is starting to shake that assumption.\nSparse attention: using only part of the KV cache The biggest problem with the attention mechanism — the most important operation in Transformer models — is that compute grows as the square of the input length.\nTo work around this, recent frontier research reduces attention compute by using only a subset of the accumulated Key-Value pairs, or by using compressed versions of them, lowering both compute and memory traffic. The underlying intuition is that even though attention conceptually relates every pair of tokens, in practice only a small set of tokens truly matter for predicting the next one. Pictured:\nThe KV cache still has to be computed and stored, so capacity-wise HBF still helps. But each sparse-attention variant uses a different algorithm to select which KV entries are touched, which changes the read pattern.\nStreamingLLM: attention sink + sliding window attention\nThe relatively simple StreamingLLM only computes attention against the first tokens of the sequence and a window of the most recent tokens around the current position. In this case the KV-cache access pattern is close to the original attention pattern, so the addresses to be read are still reasonably predictable.\nThe trade-off is that the current token only sees the first tokens plus its neighbors; relationships with tokens in the middle of the sequence are dropped entirely.\nInfLLM \u0026amp; Quest: query-aware sparse attention\nThe later approaches InfLLM and Quest take a different path. They differ in details, but both use the query of the currently-processed token to compute a coarse score against a subset of the stored KV cache, derive the indices of the Key-Value entries that look most relevant, and then run the actual attention only over those indices. Unlike StreamingLLM, the selected positions change per token, which makes it much harder to predict the addresses to be read in advance. The Keys used to compute those indices also exhibit irregular access patterns.\nCompressed Sparse Attention (CSA) \u0026amp; Heavily Compressed Attention (HCA): compressing the KV cache before storing it\nThe recently released DeepSeek-V4 goes one step further than the selection schemes above and compresses the entire KV cache along the token axis. This shrinks the KV cache that actually participates in attention. Because every KV entry passes through a projection matrix during compression, there is at least one read per entry, but the compressed KV can then be reused repeatedly afterward.\nSummarizing the three approaches along two axes — the attention behavior itself, and the KV cache access pattern that matters for HBF prefetching:\nTechnique Attention behavior / limit KV cache access pattern (prediction view) StreamingLLM Uses only the first tokens and a sliding window of recent tokens.\nRelationships with middle tokens are dropped Positions are regular, so prefetch-friendly InfLLM / Quest Dynamically selects relevant KV indices via the current query.\nPreserves middle-token information Indices change per token, so addresses are hard to predict ahead of time CSA / HCA Compresses the KV cache along the token axis and runs attention only on the compressed version.\nHowever, the added compression/reuse pipeline raises implementation complexity The compressed cache has a deterministic location, so prediction is straightforward The emergence of these new software techniques can actually create a new bottleneck for HBF. When the addresses to read become hard to predict, prefetching either misses more often or pulls in more data than necessary. Either case prevents HBF from delivering its nominal performance. Software optimizations that tailor prefetch hints to the attention scheme in use, or hardware-aware model designs, become necessary.\nInterestingly, the DeepSeek paper also discusses how this compressed KV cache should be managed in memory. They mention that, during inference serving, they store the compressed KV in SSD storage to take advantage of reuse. This kind of memory-aware optimization will translate directly into upside for Flash-based HBF as well.\nThis naturally leads us to ask how existing Flash-based SSD storage is already being used in LLM serving.\nHow SSD storage is used inside LLM services As covered in earlier posts, the growing KV cache exceeds what GPU HBM can hold, even after applying the compression and sparsity techniques above. To work around this, a wide range of techniques offload the KV cache to CPU memory or even further out — to system memory outside the GPU. Among them, techniques that leverage SSD\u0026rsquo;s large capacity have been gaining traction.\nModel loading: the most basic use of Flash storage The most universal use of SSDs is storing and loading model weights. DRAM-based HBM is volatile and loses its contents when power is cut, but NAND-based SSDs are non-volatile and are therefore used as the persistent storage for trained model weights. When a service starts up, weights stored on SSD are read into GPU HBM — this is the baseline flow. In particular, environments where multiple GPU nodes serve the same model simultaneously practically require a shared SSD storage tier that every node can reach.\nKV cache swapping: pushing inactive sessions to a lower-tier memory Serving frameworks like vLLM swap inactive sessions\u0026rsquo; KV caches out of the GPU when GPU HBM runs short. In a multi-user service, it is impossible to know in advance which session will become active when. A user of an LLM service can pause a question and resume it at any time. If a temporarily idle session keeps holding GPU memory, the system loses headroom to admit new requests. Swapping the inactive session\u0026rsquo;s KV cache to a CPU DRAM pool — or, for sessions with very large contexts, to SSD — relieves this pressure.\nPushing GPU memory itself onto SSD: offloading and GPU Direct Storage Going one step further, there are active efforts to expand the dynamic memory used by the GPU onto SSDs. These approaches treat GPU HBM, CPU DRAM, and NVMe SSD as a single unified memory hierarchy and place each tensor at whichever tier best fits the situation.\nThe fundamental limit of this approach is the data path. The long route SSD → CPU DRAM → GPU, combined with the PCIe bandwidth bottleneck, materially hurts latency and throughput.\nTo shorten this path, NVIDIA provides GPU Direct Storage (GDS). GDS uses PCIe Peer-to-Peer DMA so that the NVMe controller can read/write directly into GPU memory. Because it bypasses host DRAM, it removes the CPU memory bandwidth consumption and cuts the PCIe traffic in half.\nTo summarize: SSD/Flash already plays multiple roles across every layer of the LLM serving stack, and a growing set of optimizations is treating HBM — DRAM — Flash as one continuous memory tier. But there is also a structural ceiling visible from these efforts. Moving data from Flash to the GPU still has to go through PCIe, and PCIe bandwidth is much lower than HBM bandwidth, so this segment is an unavoidable bottleneck. SSD\u0026rsquo;s read latency and the latencies added along the transport path are also limiting factors.\nLet\u0026rsquo;s bring back the memory hierarchy pyramid from Part 1. The slot HBF is aiming for is the empty space between DRAM and SSD. By placing it close to the GPU package it cuts access latency, and by adopting HBM-style stacking + wide interfaces it raises the per-device bandwidth dozens of times above SSD. The point is to plug both structural weaknesses of SSD (distance and bandwidth) at the device level at the same time. If that picture is realized, many of the problems we saw with SSD-based approaches dissolve at the device level.\nThe catch is that, to make this picture real, HBF has to confront the inherent weaknesses of NAND. Part 2 covered NAND\u0026rsquo;s most painful one — latency — and the ways to route around it, and the first half of this post laid out why those workarounds don\u0026rsquo;t generalize cleanly. But there is more left to solve beyond latency. The next section walks through them.\nThe remaining real-world challenges of HBF Mismatch between GPU/DRAM and NAND lifespans The biggest issue is that NAND\u0026rsquo;s lifespan is not fixed — it depends on the usage pattern. The way data centers use SSDs today is relatively write-light, so SSD lifespan is rarely a serious concern. But if HBF is put in the same position as HBM and starts taking frequent writes, NAND cells wear out quickly, and the memory dies before the GPU/DRAM packaged with it, which typically last 5–7 years. With conventional SSDs, you can just swap a worn drive out. With HBF — what then? This question is tied directly to the still-unpublished packaging and interface standard.\nPackaging and interface standards are still undecided SanDisk, the most aggressive HBF proponent, kicked off standardization work for HBF together with SK Hynix earlier this year. Even with the first samples on the horizon, the detailed spec has not been published yet. What I\u0026rsquo;m watching more closely than capacity or bandwidth is the packaging and interface standard.\nThe choice boils down to two options. The first is a PCIe-based connection. PCIe is a battle-tested, low-risk option and can be deployed as a card separated from the GPU, which gives the benefit of replaceability. The catch is that a single PCIe link maxes out at about 64 GB/s even at Gen 5 x16. Compared to today\u0026rsquo;s SSDs, this brings HBF physically closer to the GPU, but reaching the TB/s class bandwidth HBF targets is physically impossible over PCIe alone.\nThe second option is HBM-style interposer integration into the package. Just as GPU and HBM are integrated via interposer today, HBF would either be connected GPU → HBF directly, or chained as GPU → HBM → HBF. This is essentially the only way to obtain HBM-class bandwidth, so it is the only path that lets HBF show its real strengths. But the moment GPU·HBM·HBF live in the same package, the lifespan mismatch from the previous subsection turns into no partial replacement possible, and the thermal and power budgets all have to be solved inside that same package.\nWhat ultimately becomes necessary: a more capable controller and software optimization In conventional SSD-based offloading, much of the data-path orchestration is done by the host CPU or DPU. The SSD\u0026rsquo;s internal controller only has to handle device-internal behavior, while operational policies like KV cache swap decisions or prefetch are the host\u0026rsquo;s responsibility. With HBF sitting inside the accelerator\u0026rsquo;s own package, however, asking the host CPU to mediate every access is no longer viable. Tasks that the CPU/DPU used to handle will likely move into the HBM/HBF controller or the base die itself. HBF will need to be more than plain memory — it will need to absorb part of the system-side policy and become a more capable memory tier.\nSoftware has to evolve along with it. To exploit HBF effectively, applications and frameworks have to be able to tell the controller explicitly which data should sit where. Only when HW and SW are designed under the same assumptions will the lifespan problem and the bandwidth-utilization problem we just discussed actually go away, and HBF\u0026rsquo;s strengths translate into real-world performance.\nWrap-up In this post we walked through:\nThe evolving patterns in LLM workloads. How SSD storage is used in LLM services today. HBF\u0026rsquo;s structural limits and the challenges it still has to solve. There is also a sense in which HBF\u0026rsquo;s position in the hierarchy feels a bit awkward. Its role doesn\u0026rsquo;t map cleanly onto the conventional memory hierarchy. Looking at its raw hardware characteristics, it appears to sit between DRAM and SSD, but in terms of physical placement close to the GPU, it could equally be argued to sit between HBM and DRAM.\nThe reason for this awkwardness is probably that the new forms of memory in the AI era (HBM, HBF) are fundamentally built on top of older memories — DRAM and NAND Flash. Both HBM and HBF are optimizing for AI compute by applying stacking as a process technique and packaging techniques like interposers and CoWoS to those existing cells. Perhaps this phenomenon is rooted in the fact that DRAM and NAND memories themselves were never fully optimized for AI compute in the first place.\nNVIDIA and other established GPU/TPU vendors have been overcoming the memory bottleneck through HBM bandwidth scaling. Groq and Cerebras have showcased a new form of accelerator with on-chip SRAM and their bespoke hardware architectures. We at HyperAccel are overcoming the memory wall with LPDDR and our own hardware optimizations that get the most out of it.\nAll of them are solving an entirely new kind of problem — AI — with existing memories, existing process and packaging techniques, and HW/SW optimizations on top. But to relieve the more fundamental bottleneck, perhaps an entirely new form of memory — one more suited to AI than what we have today — needs to emerge. Of course, what counts as \u0026ldquo;AI-optimized memory\u0026rdquo; is by no means a problem with a single right answer, and an entirely new memory is admittedly a longer-term story. Even so, related discussions are beginning to surface in both academia and industry. I\u0026rsquo;ll get into more detail on this another time.\nWhat\u0026rsquo;s next Parts 1 through 3 of this memory series walked through HBF, a memory that emerged to relieve capacity bottlenecks. But HBF is not the only solution. In the next part, Seungbin Shin will introduce another alternative for the memory problem — CXL (Compute Express Link) technology. Please look forward to it.\nPostscript: HyperAccel is hiring! As the memory hierarchy gets more diverse, the problems an accelerator company has to solve become correspondingly more complex and interesting. Optimized accelerators can only be built when memory and compute logic, software and algorithms — domains that are usually kept separate — are designed as one integrated system. HyperAccel works across HW, SW, and AI. If you\u0026rsquo;d like to grow with us across a broad and deep range of topics, please apply through our hiring page.\nReference Sparse Attention\nR. Child, S. Gray, A. Radford, I. Sutskever (OpenAI), \u0026ldquo;Generating Long Sequences with Sparse Transformers,\u0026rdquo; 2019. arxiv:1904.10509 A. Gu, T. Dao, \u0026ldquo;Mamba: Linear-Time Sequence Modeling with Selective State Spaces,\u0026rdquo; 2023. arxiv:2312.00752 G. Xiao, Y. Tian, B. Chen, S. Han, M. Lewis, \u0026ldquo;Efficient Streaming Language Models with Attention Sinks (StreamingLLM),\u0026rdquo; ICLR 2024. arxiv:2309.17453 C. Xiao et al., \u0026ldquo;InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory,\u0026rdquo; NeurIPS 2024. arxiv:2402.04617 J. Tang et al., \u0026ldquo;Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference,\u0026rdquo; ICML 2024. arxiv:2406.10774 DeepSeek-AI, \u0026ldquo;DeepSeek-V4 Technical Report,\u0026rdquo; 2026. HuggingFace SSD / Flash offloading\nY. Sheng et al., \u0026ldquo;FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU,\u0026rdquo; ICML 2023. arxiv:2303.06865 R. Ren et al., \u0026ldquo;An I/O Characterizing Study of Offloading LLM Models and KV Caches to NVMe SSD,\u0026rdquo; CHEOPS \u0026lsquo;25. DOI:10.1145/3719330.3721230 K. Kyung, S. Yun, J. H. Ahn (SNU), \u0026ldquo;SSD Offloading for LLM Mixture-of-Experts Weights Considered Harmful in Energy Efficiency,\u0026rdquo; IEEE Computer Architecture Letters, 2025. arxiv:2508.06978 I. Jeong, S. Woo, S. Namkung, D. Jeon, \u0026ldquo;HiFC: High-efficiency Flash-based KV Cache Swapping for Scaling LLM Inference,\u0026rdquo; 2025. OpenReview NVIDIA technical resources\nNVIDIA GPUDirect Storage Documentation NVIDIA Developer Blog — GPUDirect Storage: A Direct Path Between Storage and GPU Memory NVIDIA Developer Blog — How to Reduce KV Cache Bottlenecks with NVIDIA Dynamo HBF\nKAIST TERALAB, \u0026ldquo;2026 HBF Workload and Roadmap\u0026rdquo; (YouTube) — KAIST TERALAB ","date":"2026.05.28","permalink":"https://hyperaccel.ai/techblog/en/posts/hbf-challenge/","series":["Memory in the AI Era"],"style":"--c:#009EE2;--on:#02121c","summary":"HBF clearly has its place, but it still has gaps to fill before it can claim a spot in the memory hierarchy pyramid. We walk through the latest LLM model and inference workload trends, how Flash memory is actually used in LLM serving today, and the remaining challenges HBF has to solve.","title":"[Memory in the AI Era - Part 03] The Remaining Challenges of HBF"},{"authors":["Jaewon Lim"],"category":"Memory \u0026 Storage","content":" This post is Part 2 of the Memory in the AI Era series.\nPart 1: Understanding HBF covered what HBF is and where it sits in the memory hierarchy.\nThis part asks the next question, given those weaknesses, where does HBF actually pay off?, and walks through SK hynix\u0026rsquo;s H³ architecture and the candidate workloads beyond it.\nSetting the stage Hi, I\u0026rsquo;m Jaewon Lim, a hardware verification engineer on HyperAccel\u0026rsquo;s DV team.\nDid you read Seungbin Shin\u0026rsquo;s tour of High Bandwidth Flash (HBF) in Part 1? If we collapse it to one line:\nHBF is fast and huge, but too slow.\nMore precisely: HBF matches High Bandwidth Memory (HBM) on bandwidth and offers 8–16× the capacity, but its latency sits at conventional SSD levels, roughly 100× HBM\u0026rsquo;s. Bandwidth and capacity rival HBM (and capacity even beats it), while latency lands at SSD scale, a spec ambiguous enough that \u0026ldquo;where does this fit?\u0026rdquo; becomes the actual open question. Finding the workloads that genuinely suit HBF becomes its own exercise.\nIf Part 1 answered \u0026ldquo;what is HBF,\u0026rdquo; Part 2 picks up the next question:\n\u0026ldquo;How would you actually use it?\u0026rdquo;\nHBF is still pre-volume-production. For the market and accelerator companies to commit, the picture beyond the spec sheet (which workloads will use it, and why) has to come into focus first. So that\u0026rsquo;s our starting point.\nHere\u0026rsquo;s the interesting bit. Pick the workload right, and HBF\u0026rsquo;s weaknesses can be hidden. This post walks through the H³ architecture that SK hynix published at IEEE Computer Architecture Letters 2026, and then explores what else might fit beyond it.\nHBF\u0026rsquo;s weaknesses, revisited The three weaknesses Part 1 laid out:\nFirst, latency. HBF is NAND flash with packaging applied to it. The cell-level read mechanism is fundamentally slower than DRAM, so while HBM reads at tens of nanoseconds, HBF needs about 10–20 μs. Roughly 100× slower. No amount of packaging cleverness can erase that. It\u0026rsquo;s a cell-level limit.\nSecond, write endurance. NAND flash cells have physical limits on erase/write cycles. Workloads that hammer the same locations wear cells out fast. So training or frequently-updated data isn\u0026rsquo;t a fit.\nThird, read granularity. HBF is NAND-based, so each read request is processed at page granularity, around 4 KB. Contrast HBM4, which can do 32 B fine-grained accesses. If a workload picks small chunks at random, only a fraction of the 4 KB pulled per request actually gets used, and effective bandwidth collapses. Even if nominal bandwidth matches HBM, realized bandwidth can drop sharply.\nWorkload conditions that sidestep the weaknesses To neutralize all three weaknesses simultaneously, the workload has to satisfy:\nLoad once, read many times: neutralizes write endurance. Deterministic access patterns: knowable in advance, so latency can be hidden by prefetch. Coarse-grained reads: once a page is fetched, enough of it actually gets used to keep effective bandwidth high. The fastest way to see where these conditions land is to compare training and inference: the two LLM phases sit at opposite ends of the memory-access spectrum.\nTraining updates weights every step. The backward pass computes gradients and the optimizer state writes alongside, so writes happen continuously across memory. For HBF, limited by write endurance, this is the worst-case workload. It fails the first condition (read-only) immediately.\nInference is a different story. Once training is done, the weights never change during inference. Run Llama 3.1 405B in FP8 and you\u0026rsquo;re holding ~405 GB of weights that get read end-to-end every batch (only read, never written). Clearly huge + read-only.\nBut inference has another piece that can grow as large as the weights, the KV cache. With 1M, 10M context, KV cache balloons from hundreds of gigabytes to several terabytes. Putting that on HBF would dramatically expand HBF\u0026rsquo;s reach. Catch: an ordinary KV cache is exactly the access pattern HBF hates. How to bring it onto HBF anyway is the next section\u0026rsquo;s problem.\nCAG: computing the KV cache once, reusing it many times The KV cache is hard for HBF for two reasons:\nIt\u0026rsquo;s written every token. During decode, every output token appends new entries across all layers\u0026rsquo; KV. Write rate is high. It differs per request. User queries change every time, so even with the same model, the KV cache contents are recomputed from scratch. A cache built for one request isn\u0026rsquo;t reused by another. Both properties hit HBF exactly where it\u0026rsquo;s weak. Frequent writes burn endurance, and non-shareable data has no claim on HBF\u0026rsquo;s huge capacity. Put differently: only when the KV cache shifts to a single-write, multi-read shape does HBF actually pull its weight.\nSo how do real-world inference techniques actually handle the KV cache? The two closest candidates are worth lining up side by side.\nThe most common technique for lifting LLM response quality is Retrieval-Augmented Generation (RAG): pull query-relevant documents from an external knowledge base (typically a vector DB), splice them into the prompt, and generate. Every request triggers a fresh retrieve, and a fresh KV cache is built from the fetched context. So far the cache is still per-request and one-shot, both of those weaknesses intact.\nBut what if the knowledge the model needs to consult doesn\u0026rsquo;t change much across requests (the same manual, the same codebase, the same internal docs)? Re-retrieving and recomputing it every time is plain waste.\nA late-2024 paper (Chan et al., \u0026ldquo;Don\u0026rsquo;t Do RAG\u0026rdquo;, arxiv:2412.15605) called out this waste directly and proposed a new pattern: Cache-Augmented Generation (CAG).\nCAG is mechanically simple:\nRun the shareable, massive knowledge base once through the model and build a KV cache. When a user query arrives, treat that pre-built KV cache as a prefix and generate from there. No recomputation of the same material. When many requests come in, share that same KV cache and generate per-request answers on top. A one-line contrast:\nRAG: retrieve + recompute every time. CAG: compute once, read many times. The memory profile contrast is even starker. RAG\u0026rsquo;s KV cache is short and one-shot per request, but CAG\u0026rsquo;s KV cache is huge, read-only, and accessed repeatedly across requests. Hundreds of GB at 1M context, several TB at 10M.\nLay this profile next to the \u0026ldquo;HBF-friendly conditions\u0026rdquo; we defined earlier:\nread-only → write endurance issue neutralized load once, read many → instead of recomputing every request, one built cache is shared across requests layer-by-layer KV cache usage is predictable → latency hideable by prefetch This workload fits the conditions cleanly. That\u0026rsquo;s why SK hynix picked CAG as the central use case to justify H³. Next we\u0026rsquo;ll look at the hardware architecture they propose.\nSK hynix\u0026rsquo;s H³: dividing roles between HBM and HBF SK hynix\u0026rsquo;s H³ boils down to one idea:\nDon\u0026rsquo;t replace HBM with HBF. Bolt HBF on next to HBM as a dedicated slot for huge read-only data.\nSource: Ha et al., IEEE CAL 2026\nPhysical layout: daisy-chaining HBF behind HBM In a typical GPU, HBM stacks sit next to the GPU on the interposer and consume all of the available shoreline. H³ leaves that alone. HBM is still wired directly to the GPU shoreline.\nThe change is what comes after. Each HBM stack\u0026rsquo;s base die gains a Die-to-Die (D2D) interface, and an HBF stack hangs off the back of HBM through it. The HBM and HBF are daisy-chained.\nFrom the GPU\u0026rsquo;s perspective, both HBM and HBF appear as main memory inside a unified address space. An address decoder and router on the HBM base die decide whether a request goes to HBM core or makes the extra hop to HBF.\nThe win is that you get tens of times more capacity without spending any more GPU shoreline. By SK hynix\u0026rsquo;s assumption, on top of 192 GB / 8 TB/s of HBM3e per GPU, an HBF stack adds roughly 3 TB / 8 TB/s per GPU, about a 16× capacity bump.\nData placement: who lives where H³\u0026rsquo;s operating principle is to split data between the two memories by character.\nIn HBF: model weights, the pre-computed shared CAG KV cache, both huge and read-only at inference time. In HBM: the KV cache being generated, activations, anything else that gets updated frequently. This split sidesteps HBF\u0026rsquo;s weaknesses cleanly. Huge read-only data never raises endurance. And in batch-heavy inference, HBF\u0026rsquo;s extra power is paid back in throughput.\nLatency Hiding Buffer One problem remains: NAND\u0026rsquo;s tens-of-microseconds latency. A GPU compute pipeline tuned around nanosecond-scale memory will stall hard if it suddenly has to wait microseconds.\nSK hynix\u0026rsquo;s answer is the Latency Hiding Buffer (LHB), a prefetch SRAM integrated into the HBM base die.\nIt leans on a key property of LLM inference: layer-by-layer access is deterministic. You can know in advance which weights and KV blocks the next layer needs. The deep-learning framework hands those prefetch hints down, and the LHB pulls the next layer\u0026rsquo;s data ahead of time, hiding the HBF latency behind the current layer\u0026rsquo;s compute.\nLHB sizing is a simple formula:\nCapacity_LHB = 2 × BW_HBF × Latency_HBF This assumes double buffering. Plugging in SK hynix\u0026rsquo;s example numbers (BW 1 TB/s, latency 20 μs), you land on about 40 MB. At 3 nm SRAM that works out to roughly 8 mm², about 6.7% of the HBM base die\u0026rsquo;s ~121 mm² area. A modest overhead that fits comfortably into existing base-die slack.\nIn other words, H³ solves the latency problem by \u0026ldquo;adding a bit of SRAM into HBM\u0026rsquo;s spare base-die area.\u0026rdquo;\nSimulation results The team simulated H³ on Llama 3.1 405B (~405 GB of FP8 weights) with an NVIDIA B200 GPU setup. Headline gains relative to HBM-only:\nMetric (H³ vs HBM-only) 1M context 10M context Max batch size ~2.6× ~18.8× Throughput (Tokens Per Second / request) ~1.25× ~6.14× Throughput per power (max) — ~2.69× Source: Ha et al., IEEE CAL 2026. Throughput-per-power figure fully accounts for HBF\u0026rsquo;s higher power consumption.\nGains scale steeply with context length. Even more striking: 1M context inference fits on a single GPU, and 10M fits on two GPUs, workloads that would otherwise need 8 and 32 GPUs respectively on HBM-only systems.\nBeyond H³: other workloads HBF might fit H³ showed one combination, HBF + LLM inference (CAG specifically). But workloads that satisfy the \u0026ldquo;weakness-isn\u0026rsquo;t-a-weakness\u0026rdquo; conditions (huge, read-only, deterministic-prefetchable) aren\u0026rsquo;t limited to CAG. Here are two HBF candidates I\u0026rsquo;d flag.\nCandidate 1. Mixture of Experts (MoE) inference: single-chip trillion-parameter serving A lot of frontier LLMs now use Mixture of Experts (MoE). The model holds tens to hundreds of expert Feed-Forward Networks (FFNs), with only a few (typically 2–8) activated per token. The unactivated experts still have to live in memory, so total weight quickly balloons into trillion-parameter territory.\nRunning that much weight on HBM alone forces you to spread the model across multiple GPUs, which adds communication overhead from moving weights or intermediate activations between GPUs. But if you lean on HBF\u0026rsquo;s capacity, a different picture opens up: put every expert weight on HBF and serve a trillion-parameter model from a single chip.\nHBF: model weights (read-only) HBM: attention KV cache, activations, hot path The catch: it\u0026rsquo;s hard to predict which expert each layer will pick. The router decides which expert handles a token only after the input has gone through the attention layer. To hide the latency of pulling expert weights from HBF, you need an additional mechanism that effectively predicts the upcoming expert and feeds prefetch hints down to HBF.\nCandidate 2. Storing the RAG vector-DB raw data Embedding collections in RAG vector DBs covered earlier can hit billion-scale. Capacity runs from hundreds of GB to several TB, so in real services these vector DBs are parked in host memory or external storage rather than on-package memory like HBM.\nIn a typical RAG inference flow, a query encoder produces a query embedding, the vector DB scores it against all entries to pull a top-k, and the top-k\u0026rsquo;s raw embeddings or source chunks are spliced into the LLM\u0026rsquo;s prompt.\nBringing HBF into this pipeline lets the vector DB sit much closer to the accelerator, and one step further, you can place a dedicated search engine between HBF and the accelerator. HAVEN (arxiv:2603.01175), published earlier this year, adopts exactly this structure. The massive vector DB lives on HBF, a search engine sits adjacent to HBF and runs similarity scoring + top-k selection right there, and the accelerator only receives the narrowed top-k payload.\nHBF: the full vector DB (raw embeddings + source chunks) Search engine adjacent to HBF: similarity scoring + top-k selection HBM / accelerator: query encoder, LLM proper The traffic and latency story is the win. The random fine-grained access search demands still happens, but it stays inside the short HBF ↔ search-engine path. Only the narrowed top-k payload travels between HBF and the accelerator, as page-aligned reads. The accelerator-side memory interface stays clean, and placing the data close to the accelerator also speeds up search itself. As the vector DB scales (billions of documents), the combination of HBF capacity and an adjacent search engine becomes a direct cost-and-throughput lever.\nThe catch: this picture requires a dedicated search engine alongside the vector DB, which means search-acceleration logic the HBM standard base die doesn\u0026rsquo;t carry. In other words, you need a custom base die rather than the standard one. This is not something that rides a vanilla packaging line, but vendor- and workload-specific base-die customization that has to come together.\nClosing This post explored the workload conditions that can neutralize HBF\u0026rsquo;s critical weaknesses, and the LLM workloads that meet them. But the workloads we covered today are only a slice of the full LLM service stack. To match HBM-scale market reach, HBF needs broader, more general-purpose use cases, but real technical limits and unsolved engineering problems still stand in the way. The next post looks at how existing flash memory is already used in LLM serving and walks through the homework HBF has to clear before it can be commercialized in earnest.\nP.S. HyperAccel is hiring! The more memory hierarchies diversify, the more interesting the problems accelerator designers get to work on.\nI work on hardware verification of LLM-acceleration ASICs at HyperAccel\u0026rsquo;s DV team. Beyond verifying single chips, the role lets me work on memory hierarchy, system integration, and workload matching, fresh problems show up every day.\nHyperAccel works across hardware, software, and AI. If you\u0026rsquo;d like to learn deeply across that range while growing alongside the team, please apply through the careers site.\nReferences M. Ha, E. Kim, H. Kim, \u0026ldquo;H³: Hybrid Architecture using High Bandwidth Memory and High Bandwidth Flash for Cost-Efficient LLM Inference,\u0026rdquo; IEEE Computer Architecture Letters, 2026. DOI: 10.1109/LCA.2026.3660969 B. J. Chan, C.-T. Chen, J.-H. Cheng, H.-H. Huang, \u0026ldquo;Don\u0026rsquo;t Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks,\u0026rdquo; Proc. The ACM Web Conference (WWW) 2025, 2025. arxiv:2412.15605 SanDisk, \u0026ldquo;Memory-Centric AI: Sandisk\u0026rsquo;s High Bandwidth Flash Will Redefine AI Infrastructure\u0026rdquo; P.-K. Hsu, W. Xu, Q. Liu, T. Rosing, S. Yu, \u0026ldquo;HAVEN: High-Bandwidth Flash Augmented Vector Engine for Large-Scale Approximate Nearest-Neighbor Search Acceleration,\u0026rdquo; 2026. arxiv:2603.01175 ","date":"2026.04.29","permalink":"https://hyperaccel.ai/techblog/en/posts/hbf-workload/","series":["Memory in the AI Era"],"style":"--c:#009EE2;--on:#02121c","summary":"Centered on SK hynix\u0026rsquo;s H³ architecture, we explore workloads that can overcome HBF\u0026rsquo;s weaknesses.","title":"[Memory in the AI Era - Part 02] Where Does HBF Actually Fit?"},{"authors":["Donghyeon Choi"],"category":"AI Hardware","content":" Know Your Enemy, Know Yourself comes from the idea that if you understand both the opponent and yourself, you will not be imperiled in a hundred battles.\nThis series studies competing AI accelerator architectures so that we can reason about our own designs more clearly.\nIn the fifth post, we look at Cerebras and its Wafer-Scale Engine(WSE).\nHello, this is Donghyeon Choi from the CL team. Did you know there is a company that turns an entire wafer into a single chip?\nToday, I want to talk about Cerebras and its Wafer-Scale Engine(WSE).\nIn January 2026, Cerebras made headlines in the AI industry. According to several reports, the company signed a contract with OpenAI worth around $10 billion. It planned to supply up to 750MW of compute capacity by 2028. It is unusual to see a non-NVIDIA AI accelerator company mentioned as a candidate for a deal of this scale.\nThis post explains what Cerebras\u0026rsquo;s Wafer-Scale Engine(WSE) architecture is, why yield was such a difficult problem, and how Cerebras approached the problem.\nWhy Cerebras is getting attention Cerebras is an AI semiconductor startup founded in California in 2015. Its founders, Andrew Feldman and Gary Lauterbach, previously co-founded SeaMicro in 2007 and sold it to AMD for $357 million in 2012. Drawing on that experience, they believed that the AI era needed an alternative path beyond general-purpose GPUs, and built Cerebras around that idea.\nTheir core concern was fairly clear. In conventional chip designs, on-chip memory capacity is limited by chip size. Larger models also require more links between devices. As a result, memory bandwidth and inter-chip communication easily become bottlenecks.\nSo the solution Cerebras chose was to keep as much computation and data movement as possible inside a single wafer. Instead of cutting the wafer into many smaller chips, it uses the wafer almost like one enormous chip through the Wafer-Scale Engine(WSE) for AI training and inference. The point is not simply “build a bigger chip.” It is to keep data that would normally move off-chip inside the wafer whenever possible.\nThis idea did not remain just a technical experiment. It also led to growing business attention. In August 2025, Cerebras announced that it ran OpenAI\u0026rsquo;s open model gpt-oss-120B at around 3,000 tokens per second. In January 2026, reports followed that Cerebras had signed a large contract with OpenAI. In March 2026, AWS and Cerebras officially announced an inference collaboration for Amazon Bedrock, combining Trainium-based prefill with Cerebras CS-3-based decode.\nCapital markets reacted as well. Cerebras pursued an IPO in 2024, withdrew the plan in 2025, and then filed a new S-1 with the U.S. Securities and Exchange Commission(SEC) in April 2026 as it pursued a Nasdaq listing under the ticker CBRS.\nWhy are alternatives like this getting more attention? Serving large-model services to hundreds of millions of weekly users requires ever-growing data-center capacity. But depending only on NVIDIA GPUs creates both cost and supply constraints. That is why large AI service operators have strong reasons to study accelerator architectures beyond GPU clusters.\nIn that sense, Cerebras is no longer just “the company with the unusual giant chip.” It has become a real option that cloud providers and model companies actively evaluate. And the reason begins with the unusual architecture of WSE.\nWhat is a Wafer-Scale Engine? Normally, semiconductor chips are cut from a circular silicon plate called a wafer. A single wafer produces tens or hundreds of smaller chips, and each chip is packaged separately to become a GPU, CPU, or another product.\nCerebras flips that logic. It does not cut the wafer into many chips. It uses the whole wafer almost like a single chip. Based on public materials, Wafer-Scale Engine 3(WSE-3) has an area of 46,225mm². It is close to a 215mm by 215mm square and is more than 50 times larger than an NVIDIA H100 chip, which is about 814mm².\nWhy, then, have so few companies built wafer-scale chips? The biggest reason is yield. Defects are unavoidable in semiconductor manufacturing. With small chips, you can discard the defective chips and keep the rest. But as chip area grows, the probability that at least one defect lands inside the die rises, and the amount of silicon lost with each defect also grows.\nIf the same defect density is assumed, small dies can still produce many good chips. Larger dies reduce the ratio of usable chips much more quickly. On top of that, square dies printed on a circular wafer waste more edge area as die size increases.\nAs a result, large chips usually mean lower yield and more wasted wafer area, which pushes product cost up sharply. That is why wafer-scale chips were long considered economically difficult.\nCerebras attacked this problem through architecture.\nWSE-3 architecture WSE-3 is the third-generation wafer-scale chip announced in March 2024. It is built on TSMC 5nm technology. According to public materials, it contains more than 4 trillion transistors, 900,000 active AI cores, 44GB of on-chip SRAM, 21PB/s of memory bandwidth, and 214Pb/s of fabric bandwidth.\nIn WSE, one wafer becomes one chip, and that chip contains 84 connected die regions. In a conventional GPU or CPU, one die would normally be cut out and sold as a separate chip. In WSE, those die regions remain connected and are used together as one chip. Each die contains more than ten thousand cores.\nEach core contains fabric for communicating with neighboring cores, 48KB of SRAM and a 512B cache for storage, registers for holding data right before computation, and parallel 16-bit and 8-bit compute units. The 16-bit unit can process 8 data elements at once, while the 8-bit unit can process 16 data elements in parallel.\nAn H100 fetches data from off-chip High Bandwidth Memory(HBM). HBM is fast, but it still lives outside the compute chip, so the physical connection between compute and memory can become a bottleneck. In addition, connecting multiple chips requires links such as NVLink and a large amount of board-level and system-level interconnect.\nBy contrast, WSE-3 places memory directly on the chip. It distributes Static Random-Access Memory(SRAM) across the cores. Public WSE-3 materials describe 48KB of SRAM per core, which adds up to 44GB of on-chip memory across the wafer. In Cerebras\u0026rsquo;s framing, moving from HBM to SRAM delivers more than an 800x increase in memory bandwidth. Because the die regions are also kept together on a single chip, communication between them happens over very short on-silicon wires, which Cerebras says can provide up to 33x faster inter-die communication than an H100-based design.\nCerebras addresses the wafer-scale yield problem by making each core very small. In other words, it makes the “unit of loss” from a defect extremely small. The whole wafer is not treated as one giant failure domain. Instead, only some of the many small cores are lost when defects appear.\nThe WSE-3 white paper describes an individual core as roughly 38,000µm², or 0.038mm². If we use about 6mm² as a rough size for an H100 Streaming Multiprocessor(SM), the difference is more than two orders of magnitude.\nAt that point, a natural question follows: if a wafer-scale chip still contains unusable cores, does that not make the whole chip unusable? WSE solves that by designing routing paths that can bypass defective cores. Each core is connected to its neighbors through a two-dimensional mesh. Spare cores and spare wires are included so that, when defects are found, the system can route around damaged regions while keeping the logical communication structure intact.\nWSE operates in a dataflow style, where computation happens when data arrives. Weights streamed from MemoryX are turned into packets in the form of (index, value) and routed to the cores that need them. When a core receives a packet, it reads the activation that matches that index and performs the multiply-add.\nThis packet-oriented structure is what makes WSE effective even for unstructured sparsity. The key point is that zero-valued weights are never turned into packets in the first place. That means they do not create fabric traffic, and they do not trigger any corresponding computation.\nIn other words, using sparsity in WSE is not just a matter of compressing zeros in storage. It means the system avoids generating both data movement and compute events for zero weights altogether. The WSE-2 paper argues that this fine-grained dataflow execution model can deliver higher real utilization on unstructured sparsity than a GPU\u0026rsquo;s dense tile-based execution model.\nPutting models into SRAM It is easy to describe LLM inference as “a lot of compute,” but that misses an important bottleneck. To generate each token, a model repeatedly reads weights, handles activations and the Key-Value cache, and then repeats the process for the next token. In many cases, the bottleneck is not FLOPS alone. It is how quickly the system can read the weights.\nThe 44GB SRAM on WSE-3 targets this issue. Smaller models, or parts of a model, can be kept on the wafer and read at very high bandwidth close to the cores. GPU HBM is also fast, but WSE\u0026rsquo;s SRAM is placed closer to compute and distributed much more densely. This is why Cerebras frames inference as a memory bandwidth problem and focuses on reducing the distance that weights and activations need to travel.\nThat does not mean every model fits entirely inside 44GB of SRAM. For models at the 70B scale and above, even reduced precision makes it difficult to place everything on a single WSE. So Cerebras does not follow a “put everything on-chip” strategy. Instead, it keeps as much as possible in SRAM and relies on external DRAM or Flash storage with weight streaming for larger models.\nAs a result, WSE performance is strongly tied to where the model is placed. Anything that fits in SRAM can be accessed extremely quickly. But as models grow and depend more on external memory or multi-device streaming, the locality advantage inside the wafer becomes smaller. That trade-off is why MemoryX and SwarmX matter.\nInter-chip connectivity Supporting very large models requires more than a single WSE. That is why Cerebras designs not only the chip itself, but also the CS-3 system and the scale-out structure that connects many CS-3 systems together.\nFor large-model training, weights are not all stored on the wafer. Instead, they are stored externally in MemoryX and streamed layer by layer into CS-3. When multiple CS-3 systems are connected, SwarmX handles weight broadcast and gradient reduction.\nIn plain terms, MemoryX is like a warehouse for model parameters. WSE SRAM is extremely fast, but it is not large enough to hold all the weights, gradients, and optimizer states of very large models. So those larger states live in MemoryX, while only the weights needed for the current layer are streamed toward CS-3. MemoryX is connected through 12 links of 100 Gigabit Ethernet, which is about 150GB/s when converted into bytes. That is still much slower than on-wafer SRAM, so the system tries to overlap loading with computation in order to hide memory latency.\nSwarmX, meanwhile, is the distributor and aggregator between multiple CS-3 systems. During training, the same weights must be broadcast to many CS-3 systems, and the gradients computed by those systems must be reduced back together. During inference, different model layers may be distributed across different CS-3 systems, and the output of one system must be passed to the next layer on another system. SwarmX handles these broadcast and reduce patterns so that many WSEs can act like one larger learning system.\nThe goal is to avoid forcing users to manually split the model in detail through tensor parallelism or pipeline parallelism. In Cerebras\u0026rsquo;s framing, the system should look closer to data parallelism even as model size grows. Internally, MemoryX, SwarmX, and CS-3 divide the roles of storage, communication, and compute.\nPublic materials describe systems that scale up to 2,048 CS-3 units. The important point is not simply that many machines can be connected. It is that weight broadcast and gradient reduction are designed as part of the system so that they do not immediately become bottlenecks.\nPutting it vertically in a server?! A wafer-scale chip is not just a large silicon die. It is a thin structure more than 20cm wide that must be mechanically supported, supplied with very large current, and cooled effectively. Its mechanical and thermal constraints are much harder than those of a conventional GPU package.\nWhen a chip becomes this large, gravity, bending, thermal expansion, and contact pressure become difficult to ignore. If the wafer bends, electrical contact and cooling contact can become unstable. WSE therefore requires dedicated packaging, power delivery, cooling, and board support.\nThis is also why Cerebras sells complete systems such as CS-3. It is not a chip that a server vendor simply plugs into an ordinary board. Cerebras provides the wafer-scale chip, power modules, cooling structure, system fabric, MemoryX, and SwarmX as a complete machine.\nFrom this perspective, Cerebras is closer to a wafer-scale computer company than a conventional chip vendor. The benefit starts inside the silicon, but using that benefit in a real data center requires packaging and system design to work together.\nStrengths and limitations of Cerebras Cerebras\u0026rsquo;s biggest strength is memory bandwidth and communication locality. Large Language Model(LLM) inference often looks like a compute problem, but it is also a memory bandwidth problem because model weights must be read repeatedly for every generated token. WSE attacks that problem by maximizing bandwidth through large amounts of SRAM close to compute. It also reduces the communication overhead that appears when many chips must be connected together.\nThe limitations come from the same design choices. The most obvious one is that 44GB of SRAM is extremely fast, but still limited in capacity. Models at the 70B scale and above require either multiple devices or external weight streaming structures.\nUnder a short-output setting(128 prompt tokens and 20 output tokens), Cerebras keeps single-user speed high while increasing throughput as batch size grows. But what happens if many users submit book-length prompts and require long outputs? In high-batch, long-sequence settings, WSE may struggle to hold both model weights and KV cache inside 44GB. That would increase dependence on MemoryX loading, which could lead to more visible slowdowns compared with larger-capacity HBM-based systems.\nIf a user tries to avoid that by adding more systems for capacity, then dependence on MemoryX may decrease, but initial deployment cost rises as well. There is no official price list, but public estimates help show the scale. The Next Platform estimates a CS-2/CS-3 node at roughly $2.5M–$3.2M, or around $4M when MemoryX and SwarmX are included. SEC-filed purchase orders also show CS-3 cluster goods fees in the $178M–$350M range.\nIn other words, moving toward WSE is not just a matter of chip performance. It implies large upfront deployments. Once CS-3, MemoryX, SwarmX, networking, power, and cooling are bundled together, the required investment can quickly climb into the hundreds of millions of dollars.\nTo sum up In this post, we covered:\n① Cerebras\u0026rsquo;s recent momentum and reports of a large OpenAI deal,\n② the company\u0026rsquo;s founding background and why it chose wafer-scale computing,\n③ the concept of the Wafer-Scale Engine(WSE) and the core architecture of WSE-3,\n④ SRAM, MemoryX/SwarmX, and the scale-out connectivity structure,\n⑤ Cerebras\u0026rsquo;s strengths, including bandwidth, locality, and scale-out design, and its limits, including long-output inference, high-batch behavior, and high upfront cost.\nCerebras\u0026rsquo;s Wafer-Scale Engine is an aggressive design that uses an entire wafer as a chip. But the real point is not size alone. The important ideas are isolating defects at a small unit, placing SRAM close to cores, and keeping data movement inside the wafer through a two-dimensional mesh fabric.\nThe AI accelerator market can no longer be explained only through NVIDIA. Wafer-scale chips, inference-only accelerators, memory hierarchy expansion, optical interconnects, and many other approaches now coexist. In an increasingly diverse NPU market, the more important question is: which accelerator is optimized for which workload?\nSee you in the next installment of Know Your Enemy, Know Yourself.\nReference 100x Defect Tolerance: How Cerebras Solved the Yield Problem - Cerebras AWS and Cerebras announce AI inference collaboration - Cerebras AWS and Cerebras bring ultra-fast AI inference to Amazon Bedrock - Amazon Cerebras Systems Form S-1 Registration Statement - SEC EDGAR Cerebras CS-3 Purchase Order #2024-0003 - SEC EDGAR Cerebras MBZUAI CS-3 Purchase Order - SEC EDGAR Cerebras Goes Hyperscale With Third Gen Waferscale Supercomputers - The Next Platform Cerebras WSE-3 Architecture White Paper Hot Chips 2024: Cerebras WSE-3 and Inference IEEE Micro: Cerebras Architecture Deep Dive P.S. HyperAccel is hiring! The saying is “know your enemy, know yourself,” but to win consistently, we also need great people.\nIf the technologies we discuss here interest you, please apply through HyperAccel Career.\nHyperAccel has many excellent and thoughtful engineers. We look forward to hearing from you.\n","date":"2026.04.27","permalink":"https://hyperaccel.ai/techblog/en/posts/cerebras-wse/","series":["Know Your Enemy, Know Yourself"],"style":"--c:#00C2A8;--on:#02121c","summary":"This post explains Cerebras\u0026rsquo;s recent momentum, WSE-3 architecture, and the trade-offs of wafer-scale chips in a beginner-friendly way, following reports of a large OpenAI deal.","title":"[Know Your Enemy - Part 05] Cerebras and the Wafer-Scale Engine"},{"authors":["Seungbin Shin"],"category":"Memory \u0026 Storage","content":" This is Part 1 of the Memory in the AI Era series.\nIntroduction Hello, I\u0026rsquo;m Seungbin Shin, an RTL Designer at HyperAccel.\nRecently, \u0026ldquo;memory and semiconductor stocks\u0026rdquo; have been making headlines in the stock market. Lately, a new term — \u0026ldquo;HBF\u0026rdquo; — has started popping up, and people are asking \u0026ldquo;What even is that?\u0026rdquo; and \u0026ldquo;So should I buy it?\u0026rdquo;\nSo what exactly is High Bandwidth Flash (HBF)?\nTo understand HBF, you first need to be able to answer one question:\n\u0026ldquo;Why do computers have so many different types of memory?\u0026rdquo;\nInside our computers, there are memories with confusing names — SRAM, DRAM, HBM, SSD — organized into multiple layers. They all \u0026ldquo;store data,\u0026rdquo; so why not just use one type?\nAs you work through this question, you\u0026rsquo;ll naturally see why HBF emerged and where it fits in.\nThe content of this post is based on my personal study and experience. If you find any errors, please let me know in the comments.\nWhy So Many Types of Memory — The Speed/Capacity/Cost Trilemma The short answer: fast memory is expensive and small, while cheap memory is slow and large.\nThis isn\u0026rsquo;t simply because technology hasn\u0026rsquo;t advanced enough. The physical structure of the memory cell itself determines this trade-off. Build a cell with many transistors and it\u0026rsquo;s fast but takes up a lot of area; simplify the cell and it\u0026rsquo;s slower but can store far more data in the same space.\nComputers work around this physical limitation by stacking different types of memory in a hierarchy. Faster, smaller memory sits closer to the processor, while slower, larger memory sits further away.\nNow let\u0026rsquo;s look at why each layer of memory has the characteristics it does, starting with the cell structure.\nSRAM — The Fastest, and the Most Expensive Cell Structure: 6 Transistors per Bit Static Random Access Memory (SRAM) uses 6 transistors to store a single bit. This is called a 6T SRAM cell.\nFour of the six transistors form two inverters whose outputs are cross-coupled to each other. This structure creates a latch that maintains its state as long as power is supplied. The remaining two transistors serve as pass gates for reading and writing to the cell.\nWhy It\u0026rsquo;s Fast The reason SRAM is fast is straightforward.\nBecause the latch maintains its state on its own, no refresh is needed. This is where the name \u0026ldquo;Static\u0026rdquo; comes from. A read operation simply transfers the voltage state stored in the latch to the bitline, resulting in access latency of under 1 ns.\nWhy It\u0026rsquo;s Expensive The problem is size. With 6 transistors needed per bit, each cell occupies a large area. This means fewer bits can be produced on the same silicon wafer, which translates to a very high cost per bit.\nThis makes using SRAM in large capacities physically and economically impractical. It\u0026rsquo;s deployed only in amounts of a few MB to tens of MB as cache (L1/L2/L3) inside CPUs and GPUs, or as on-chip memory in AI accelerators.\nFast but expensive, and small in capacity. That\u0026rsquo;s why it sits right next to the processor, holding only the most frequently used data.\nDRAM — Cheap and Large, but Constantly Refreshing Cell Structure: 1 Transistor + 1 Capacitor The cell structure of Dynamic Random Access Memory (DRAM) is dramatically different from SRAM. It stores a single bit using just two components: one transistor and one capacitor. This is called the 1T1C structure.\nIf the capacitor is charged, it\u0026rsquo;s a 1; if discharged, it\u0026rsquo;s a 0. The transistor acts as a switch to access the capacitor.\nWhy It\u0026rsquo;s Cheap With only two components per cell, the cell area is 1/4 to 1/6 that of SRAM. Far more bits can be produced on the same wafer, significantly lowering the cost per bit. Thanks to this density advantage, DRAM is used as main memory in capacities of several GB to tens of GB.\nWhy It\u0026rsquo;s Slower (Relatively) Over time, the charge stored in the capacitor gradually leaks away. Before the data is lost, the charge must be periodically replenished through a refresh operation. The name \u0026ldquo;Dynamic\u0026rdquo; comes from exactly this characteristic.\nThe read operation is also more complex than SRAM. The capacitor\u0026rsquo;s charge must be shared with the bitline (charge sharing) and then amplified by a sense amplifier. Since the read process depletes the charge, the data must be rewritten (restored) immediately after reading.\nThese additional steps result in access latency of 10–100 ns, more than 10x slower than SRAM.\nDDR: The Interface That Speeds Up DRAM While DRAM cells themselves are slow, improving the interface standard for exchanging data with the outside world can increase effective transfer speeds. That standard is Double Data Rate (DDR).\nBefore DDR, SDRAM transferred data only on the rising edge of the clock. DDR, as its name suggests, transfers data on both the rising and falling edges, doubling the throughput at the same clock speed.\nSubsequent generations increased transfer speeds by deepening the prefetch depth. Prefetch is a technique that retrieves multiple beats from a single internal memory access, making the external transfer rate faster than the internal clock. DDR4 prefetches 8 bits at a time, while DDR5 prefetches 16 bits. As a result, DDR5 delivers a maximum bandwidth of approximately 50 GB/s per module.\nBut there is a fundamental ceiling to this improvement.\nHBM — Stacking DRAM Up Breaking the Pin Limit Through Packaging The data bus width of DDR is 64 bits as of DDR5. Because there\u0026rsquo;s a physical limit to the number of pins connecting the processor to memory, no matter how much you increase the clock, there\u0026rsquo;s a ceiling on how much data can be transferred at once.\nFor workloads like AI that need to read massive amounts of data simultaneously, this bottleneck is critical. High Bandwidth Memory (HBM) solves this problem not by changing the cell structure, but through packaging.\nThe core idea is simple: stack DRAM dies vertically and place them right next to the processor on the same substrate.\nTSV: Vertical Wiring Through Silicon Conventional DRAM modules connect to the processor through traces on a PCB. HBM is different. Tiny holes are drilled through the silicon die itself and filled with copper wiring, vertically connecting dies above and below. This is the Through-Silicon Via (TSV) — a through-silicon electrode.\nThanks to TSV, multiple DRAM dies can be stacked into a single unit, and because each die has independent data paths, the bus width can be dramatically widened. As of HBM4, the data bus width is 2,048 bits — 32x that of DDR5.\nInterposer: Placing It Right Next to the Processor HBM stacks are placed alongside a GPU or AI accelerator on a silicon interposer, an intermediate substrate. The fine wiring inside the interposer connects HBM to the processor at a distance of just a few millimeters, a dramatic reduction compared to conventional DDR modules traveling tens of centimeters through a PCB.\nShorter wiring = lower latency + lower power consumption + higher signal integrity. This combination is what makes HBM\u0026rsquo;s bandwidth possible.\nHBM by the Numbers Metric DDR5 HBM3E HBM4 Bus Width 64-bit 1024-bit 2048-bit Bandwidth per Stack ~50 GB/s ~1.2 TB/s ~2 TB/s+ Capacity per Stack - 24–36 GB 36–48 GB Distance to Processor ~tens of cm (PCB) ~mm (interposer) ~mm (interposer) In April 2025, Joint Electron Device Engineering Council (JEDEC), the semiconductor standards body, officially published the HBM4 standard, and mass production began in early 2026. SK hynix unveiled the world\u0026rsquo;s first 16-layer, 48 GB HBM4 stack, and NVIDIA\u0026rsquo;s next-generation GPU, Vera Rubin, targets a total of 384 GB and approximately 22 TB/s of memory bandwidth with eight HBM4 stacks.\nHBM\u0026rsquo;s Limitation: Fast, but Still Not Enough Capacity While bandwidth and capacity have both grown significantly through HBM4, the fundamental constraints remain.\nThe cell itself is still DRAM (1T1C), so the limits on bit density haven\u0026rsquo;t changed. Combined with the challenges of TSV processing, interposer fabrication, yield management during die stacking, and heat dissipation, the cost per unit of capacity is very high.\nEven with a full complement of eight HBM4 stacks, the maximum is 384 GB. Yet the parameter sizes of the latest Large Language Models (LLMs) range from hundreds of GB to several TB. The rate at which models are growing outpaces HBM\u0026rsquo;s ability to scale capacity.\nNAND Flash — Cheap and Large, but Far Too Slow Cell Structure: Trapping Charge to Store Bits NAND Flash stores data using a fundamentally different principle from SRAM or DRAM.\nAt the heart of a NAND cell is a conductor surrounded by an insulating layer called a floating gate. When a high voltage is applied, electrons are injected into the floating gate through quantum tunneling, and the insulating layer prevents the electrons from escaping, preserving data even when power is off. This is why NAND is a non-volatile memory.\nWhy It\u0026rsquo;s Cheap and Large: Extreme Density NAND\u0026rsquo;s density advantage comes from two factors.\nFirst, multi-level cells. By finely distinguishing the amount of charge stored in a single cell, multiple bits can be packed in. Triple-Level Cell (TLC) stores 3 bits per cell, and Quad-Level Cell (QLC) stores 4 bits per cell.\nThe difference becomes even more dramatic when comparing areal bit density. SRAM achieves roughly 0.04 Gb/mm², DRAM about 0.2–0.3 Gb/mm², while the latest 3D NAND reaches approximately 5–15 Gb/mm². That means over 100x more data than SRAM and over 20x more than DRAM in the same silicon area.\nSecond, 3D stacking. Modern NAND stacks cells vertically rather than laying them flat, reaching hundreds of layers. Samsung\u0026rsquo;s V-NAND and SanDisk\u0026rsquo;s BiCS NAND are representative examples, with the latest products exceeding 200 layers.\nTogether, these two factors allow NAND to deliver TB-scale capacity at a very low cost per bit.\nWhy It\u0026rsquo;s Slow: Microseconds Just to Read NAND\u0026rsquo;s read operation involves measuring the threshold voltage of the transistor, which varies depending on the amount of charge in the floating gate. In multi-level cells, multiple voltage levels must be precisely distinguished, adding time.\nAs a result, NAND\u0026rsquo;s random read latency is approximately 50–100 µs. Compared to DRAM\u0026rsquo;s tens of nanoseconds and SRAM\u0026rsquo;s sub-nanosecond latency, that\u0026rsquo;s over 1,000x slower.\nWrites are even slower, and overwriting data requires erasing an entire block first. On top of that, there\u0026rsquo;s a physical limit on the number of erase/write cycles (write endurance).\nCurrent Role in AI Servers In today\u0026rsquo;s AI servers, NAND (SSD) serves as model storage and checkpoint storage. It holds trained model weights and loads them into HBM when needed — essentially a \u0026ldquo;warehouse.\u0026rdquo;\nBut it\u0026rsquo;s far too slow to directly feed data during computation. The GPU ends up waiting for data longer than it takes to actually compute.\nAmple capacity and low cost, but critically insufficient speed. NAND has been trapped within this limitation.\nThe Gap — Between HBM and SSD Let\u0026rsquo;s put the memory hierarchy we\u0026rsquo;ve covered so far into a single table.\nSRAM DRAM (DDR5) HBM4 ??? NAND (SSD) Cell Structure 6T 1T1C 1T1C (TSV stacked) Floating gate Access Latency ~1 ns ~10–100 ns ~10–100 ns ~50–100 µs Bandwidth (stack/module) ~TB/s (on-chip) ~50 GB/s ~2 TB/s ~7 GB/s Capacity MB–tens of MB GB–tens of GB 36–48 GB TB-scale Cost per Bit Very high Medium High Very low Volatility Volatile Volatile Volatile Non-volatile One thing immediately stands out.\nHBM4 maxes out at 48 GB per stack, or 384 GB with eight stacks. SSD offers TB-scale capacity at low cost, but bandwidth is only around 7 GB/s.\nHBM has great bandwidth but limited capacity; SSD has ample capacity but is far too slow.\nGranted, HBM\u0026rsquo;s capacity is relatively large at this point, but it falls short of what people expect.\nBetween the two, there is no memory that provides \u0026ldquo;TB-scale capacity at TB/s-level bandwidth.\u0026rdquo;\nWith LLM parameters growing from hundreds of GB to several TB, what if we could fill this gap?\nHBF — Dressing NAND in HBM\u0026rsquo;s Clothes Core Idea: Familiar Cells, New Packaging High Bandwidth Flash (HBF) is a new memory tier that keeps the NAND cell as-is while applying packaging technology proven in HBM.\nEarlier, we saw how HBM revolutionized bandwidth without changing the DRAM cell — purely through TSV stacking and interposer placement. HBF applies exactly the same strategy to NAND.\nNAND dies are vertically stacked with TSV and placed on an interposer right next to a GPU or AI accelerator. Just as HBM broke through DDR\u0026rsquo;s pin-count limitation with packaging, HBF breaks through SSD\u0026rsquo;s bandwidth limitation with packaging.\nCBA Architecture: Optimizing NAND for High Bandwidth The technical core of HBF is the CMOS Bonding Array (CBA) architecture developed by SanDisk.\nConventional NAND accesses a single large memory array sequentially. CBA divides this into thousands of independent storage sub-arrays. Each sub-array has its own read/write channel and operates in parallel simultaneously, extracting bandwidth levels that a single NAND die could never achieve on its own.\nThis is combined with SanDisk\u0026rsquo;s BiCS NAND (3D vertical stacking NAND) technology. BiCS NAND dies stacked hundreds of layers high are connected via TSV, and CBA drives the sub-arrays across these dies simultaneously.\nHBF by the Numbers HBM4 HBF Gen 1 HBF Gen 2 HBF Gen 3 Read Bandwidth ~2 TB/s 1.6 TB/s \u0026gt;2 TB/s \u0026gt;3.2 TB/s Stack Capacity 36–48 GB 512 GB 1 TB 1.5 TB Access Latency ~10–100 ns ~10 µs - - Cell Structure 1T1C DRAM Floating gate NAND Floating gate NAND Floating gate NAND There\u0026rsquo;s a number worth noting here.\nHBF Gen 1\u0026rsquo;s read bandwidth of 1.6 TB/s approaches that of HBM4. Yet its capacity of 512 GB is over 10x that of an HBM4 stack (48 GB). Providing 8–16x the capacity of HBM at similar bandwidth and similar cost is HBF\u0026rsquo;s core value proposition.\nCompatibility with HBM Controllers Another strength of HBF is that its physical footprint and electrical interface (PHY level) are compatible with HBM.\nPinout, package size, power profile, and stack height are nearly identical to HBM4, so the existing HBM interposer and packaging infrastructure can be reused as-is.\nThat said, it\u0026rsquo;s important to be clear that HBF is not a drop-in replacement for HBM. Because NAND differs from DRAM in access granularity (page vs. word), erase cycles, wear-leveling, and other characteristics, minimal protocol changes are required on the host controller side. The open standard being pushed by SanDisk and SK hynix is also aimed at \u0026ldquo;the same electrical interface with minimal protocol changes.\u0026rdquo;\nIn other words, there is no need to design a new memory controller from scratch, but existing HBM controllers cannot be used as-is either. Even so, the adoption barrier for accelerator designers remains substantially lower than designing a memory controller from zero.\nHBF\u0026rsquo;s Limitations: Not a Silver Bullet HBF is a strong candidate for filling the gap, but its limitations are clear.\nLatency: At approximately 10 µs, it\u0026rsquo;s roughly 100x slower than HBM\u0026rsquo;s tens to hundreds of nanoseconds. Because the NAND cell\u0026rsquo;s read mechanism is inherently slower than DRAM, packaging alone cannot fully close this gap.\nWrite speed and endurance: NAND\u0026rsquo;s characteristically slow writes and limited erase/write cycles remain. This makes HBF unsuitable for AI training, which requires frequent writes.\nConversely, for AI inference workloads — where pre-loaded model weights are read repeatedly — these limitations are much less of a concern. This is exactly why HBF is attracting attention as \u0026ldquo;the memory for the AI inference era.\u0026rdquo;\nConclusion \u0026amp; Next Episode Preview In this post, we explored why memory is divided into so many types and how each memory\u0026rsquo;s cell structure determines the speed/capacity/cost trilemma.\nHere\u0026rsquo;s the summary:\nSRAM: 6T cell — the fastest, but the most expensive and smallest DRAM: 1T1C cell — density advantage makes it the main memory, but bandwidth is limited HBM: DRAM cells with TSV + interposer packaging — revolutionized bandwidth, but capacity scaling is difficult NAND: Extreme density and low cost, but too slow to directly participate in computation HBF: HBM packaging applied to NAND cells — a new tier filling the gap between HBM and SSD But being able to fill the gap alone doesn\u0026rsquo;t complete the story.\nHBF has a weakness: its latency of around 10 µs is roughly 100x longer than HBM\u0026rsquo;s. So how can this weakness be overcome, and how can HBF actually be used in real LLM workloads?\nIn the next installment, we\u0026rsquo;ll explore how HBF can be leveraged in LLMs, focusing on SK hynix\u0026rsquo;s proposed approach to overcoming HBF\u0026rsquo;s long latency.\nPart 2 will be written by Jaewon Lim from HyperAccel. Stay tuned!\nP.S. I\u0026rsquo;m designing RTL at HyperAccel for the launch of an LLM acceleration ASIC chip. Many of us are putting our heads together to find more efficient ways to maximize performance from the limited resource of memory bandwidth. I hope this series helps us understand the flow of memory technology together and watch the changes ahead.\nHyperAccel is a company that covers HW, SW, and AI, bringing together outstanding talent across all domains. If you want to learn broadly and deeply while growing together, please apply to HyperAccel anytime!\nCareers: https://hyperaccel.career.greetinghr.com/ko/guide\nReference SanDisk HBF Fact Sheet Scaling the Memory Wall: Behind Sandisk\u0026rsquo;s High Bandwidth Flash for AI Inferencing SK hynix and Sandisk Begin Global Standardization of Next-Generation Memory \u0026lsquo;HBF\u0026rsquo; SK Hynix Unveils AI Chip Architecture with HBF HBM VS HBF VS HBS: Building the Memory Hierarchy for AI High Bandwidth Flash: NAND\u0026rsquo;s Bid for AI Memory HBF: A High-Bandwidth Flash New Star Breaking the \u0026ldquo;Memory Wall\u0026rdquo; for AI High Bandwidth Flash is years away despite its promise SK hynix and SanDisk announce new High Bandwidth Flash — Tom\u0026rsquo;s Hardware HBM roadmaps for Micron, Samsung, and SK hynix: To HBM4 and beyond The State of HBM4 Chronicled at CES 2026 ","date":"2026.04.23","permalink":"https://hyperaccel.ai/techblog/en/posts/what-is-hbf/","series":["Memory in the AI Era"],"style":"--c:#009EE2;--on:#02121c","summary":"Why are there so many types of memory, and where does HBF fit in? From SRAM to HBF, we explore the physical principles behind the memory hierarchy and the technical architecture of HBF.","title":"[Memory in the AI Era - Part 01] Understanding HBF"},{"authors":["Hyunjun Park"],"category":"LLM \u0026 Models","content":"Project Glasswing: Claude Mythos Preview Hi, I\u0026rsquo;m Hyunjun Park from the HyperAccel CL (Compute Library) Team.\nIn early April 2026, Anthropic introduced an unreleased frontier model called Claude Mythos Preview alongside an industry coalition called Project Glasswing. News about new models usually stops at \u0026ldquo;benchmark scores went up,\u0026rdquo; but this time the Frontier Red Team blog went on at unusual length: a month of internal evaluation and vulnerability write-ups. Mythos-level coding and reasoning now extend in one stride past what typical benchmarks cover, into vulnerability discovery and exploit code generation as well. I\u0026rsquo;ll admit I first wondered if it was mostly marketing, but the more I read, the more this felt like a story bigger than the numbers alone.\nWhat Is Mythos Preview? Project Glasswing Glasswing is a defense-first collaboration program with participants including AWS, Apple, Google, Microsoft, and the Linux Foundation. The key idea is not to maximize broad release speed, but to connect high-risk capabilities to defensive communities first so they can improve real response capacity. In other words, this is less about a simple \u0026ldquo;release vs. no release\u0026rdquo; decision and more about a staged distribution strategy: who gets access first, and under what controls.\nAnthropic explicitly states there is no general availability (GA) plan for Mythos Preview. This looks less like a technical limitation and more like a deliberate choice to contain the downside risk of rapidly spreading strong cyber capabilities.\nAccording to the Red Team write-up, when instructed by a user, Mythos Preview can identify zero-day vulnerabilities (previously undisclosed flaws) and even produce exploit code across major operating systems and major web browsers. That capability can dramatically improve defense productivity, but it can also increase misuse velocity, which helps explain the restricted-release approach.\nThe Emergence of a New Kind of LLM Value What separates Mythos from previous models is not simply \u0026ldquo;how well it scores on known tests,\u0026rdquo; but \u0026ldquo;how far it can solve problems that were not in its learned patterns.\u0026rdquo; Older models improved by recalling learned patterns on human-made tests. Mythos appears to create a new kind of value even in vulnerability discovery outside those patterns.\nThis \u0026ldquo;new value\u0026rdquo; does not mean a higher benchmark number by itself. It means the model can uncover severe system flaws that remained hidden for years in real operational environments, and turn those findings into directly actionable defensive decisions. The Red Team cases (OpenBSD, FFmpeg, FreeBSD) are presented as evidence of exactly that: long-missed flaws being identified in practice.\nWhat Is Different About Mythos Preview? Evolution of Benchmarks Each time a new frontier model arrives, benchmark design tends to tilt toward the dimensions where that model\u0026rsquo;s strengths are visible, while new metrics are added on top.\nIf we simplify benchmark evolution since the rise of LLMs, it looks like this:\nPattern-matching style tests — early benchmarks, focused on short knowledge questions or tiny function-level fixes. Real codebase-level coding — evaluates whether the model can solve tasks with full project context, not isolated snippets. Long-horizon agentic execution — checks whether the model can complete higher-level tasks through attempt, failure, and retry, not just one-shot answers. Real-outcome-oriented evaluation — have the model hunt for zero-days (vulnerabilities absent from public benchmark datasets), then use successful hits to verify the capability is not merely memorizing known tests. As this shift continues, many public benchmarks approach saturation and stop being strongly discriminative. Measurement naturally moves to the next questions: can the model push larger chunks of work end-to-end, and can it operate reliably in domain-specific contexts? The center of gravity is moving toward large-scale tasks and domain-specialized workloads.\nThe evaluation tables Anthropic published with Claude Opus 4.7 reflect this trend as well, with side-by-side domain benchmarks across coding agents, terminal tasks, long context, tool use, finance, and document understanding.\nMythos Benchmark Results The Glasswing page and the Red Team post both include several benchmark tables. On the rows they cite, Mythos Preview is listed with higher scores than Opus 4.6; the tables are long, so here is a small excerpt:\nBenchmark Mythos Preview Opus 4.6 CyberGym 83.1% 66.6% SWE-bench Verified 93.9% 80.8% Terminal-Bench 2.0 82.0% 65.4% The numbers alone already suggest a large jump, but unfamiliar benchmarks landed more slowly for me than the concrete stories in the Red Team post. The write-ups include vulnerabilities found in software that had been running in production for a long time; I go through the details in the next section.\nCases of Successful Defensive Use OpenBSD, TCP SACK (27-year-old vulnerability) The first case is a bug that sat inside an OS for 27 years before it was surfaced. Selective Acknowledgment (SACK) is a TCP feature that tells the sender to retransmit only missing segments. According to the Red Team report, Mythos Preview found a problematic combination in OpenBSD\u0026rsquo;s SACK handling: a weak range-boundary validation path and an integer arithmetic overflow in sequence-number comparison logic. Under conditions that should not become true at the same time, the kernel may attempt writes to invalid addresses, potentially causing remote Denial of Service (DoS).\nFFmpeg, H.264 Decoder (16-year-old vulnerability) The model also surfaced a bug in a video decoder that had been in use for over 16 years. The H.264 decoder processes frames in small units called slices. Here, the problem is a width mismatch: slice numbers can span 32 bits, but the table that records which slice each block belongs to uses only 16 bits. That table fills \u0026ldquo;empty\u0026rdquo; slots with 65535; if someone packs an extreme number of slices, the real slice index 65535 collides with that empty marker. The wrong cell gets touched, which can lead to a small heap out-of-bounds access scenario; that is the behavior the write-up highlights as the finding.\nFreeBSD NFS, Remote Code Execution (RCE) (17-year-old vulnerability) Mythos also found a security issue in file-sharing software that had been around for 17 years. NFS is a network file-sharing service. According to the Red Team report, Mythos Preview completed a path from bug discovery through exploit writing with no human intervention—including unauthenticated access that reaches root. Loose length checks in RPCSEC_GSS handling inside Remote Procedure Call (RPC) processing allowed a kernel stack overflow, and the chain continued with Return-Oriented Programming (ROP) to add an attacker key to authorized_keys.\nDevelopers: How Should We Work in the Agent Era? Through these examples, I kept seeing LLMs move past the phase of chasing high benchmark scores and into creating real value. Each time agent capability rises a notch, developers spend less time writing code line by line and more time designing, validating, and owning outcomes. Security is just one instance; I feel the same pressure in everyday development. In a post I wrote two months ago on how we adopted agents, I wrote that as agent-driven code productivity goes up, the bottleneck shifts to review—and review is exactly the work of validating and taking responsibility.\nReading code line by line has a hard speed limit, so among teams that move fast, you also see people trying to review plan.md first. Still, the space changes too quickly to call any single workflow \u0026ldquo;the answer\u0026rdquo; today. What I think matters is not declaring one fixed process forever, but staying ready to refresh how we work each time the model takes another jump.\nOur team already encourages plan review strongly, and we plan to keep tuning prompts, harnesses, and verification as new agent capabilities ship. For example, we are shaping harnesses with guardrails and clear checkpoints so agents don\u0026rsquo;t fall into infinite loops or idle spinning, using different models for different tasks where it helps, and running self-attack style probes to harden security. I\u0026rsquo;ll keep sharing what we learn as we iterate through more trial and error.\nReferences Project Glasswing Assessing Claude Mythos Preview\u0026rsquo;s cybersecurity capabilities (Red Team) Claude Mythos Preview System Card (PDF) Introducing Claude Opus 4.7 HyperAccel Is Hiring It\u0026rsquo;s been three and a half years since ChatGPT launched. The same LLMs that once droned on about Admiral Yi Sun-sin\u0026rsquo;s smartphone can now threaten the security posture of programs engineers have trusted for years. At HyperAccel, we don\u0026rsquo;t shy away from cutting-edge agents: with lean security guardrails, strong company support, and internal study groups, we put them to work in production.\nAs agents spread across more work environments, demand for the underlying compute, memory, and power required for reasoning also grows. HyperAccel is building sustainable acceleration infrastructure to meet that demand.\nIf you\u0026rsquo;re interested in the technologies we build, please apply at HyperAccel Career. Thank you for reading.\n","date":"2026.04.17","permalink":"https://hyperaccel.ai/techblog/en/posts/project-glasswing-mythos-preview/","series":[],"style":"--c:#003A70;--on:#ffffff","summary":"Centered on Anthropic\u0026rsquo;s Project Glasswing and Claude Mythos Preview, this post explains why cybersecurity capability has jumped, how benchmark design is shifting, what real defensive findings looked like, and how developers should evolve their workflow with agents.","title":"Project Glasswing: Claude Mythos Preview"},{"authors":["Younghoon Jun"],"category":"Infra \u0026 DevOps","content":"Hello! I\u0026rsquo;m Younghoon Jun, a DevOps Engineer on the ML team at HyperAccel.\nThis post is the third installment of the Building an In-House Dev Environment on Kubernetes series!\nIn Part 1, we covered the background, overall design, and direction of building a Kubernetes-based development environment. Part 2 introduced the strategy and process for building an ARC-based CI/CD infrastructure to overcome the structural limitations of self-hosted runners. In this third article, we will discuss the Device Plugin required for utilizing custom resources on Kubernetes.\nHyperAccel is a company that builds LPU(LLM Processing Unit) devices specialized for LLM inference. To make Kubernetes recognize the LPU as a custom resource and enable smooth scheduling, a Device Plugin developed specifically for the LPU is required.\nIn this article, we will explore the following topics under the theme of LPU Device Plugin:\nFirst, we explain the operating principles and detailed mechanics of the Kubernetes Device Plugin. Then, we introduce the development process of the device plugin for HyperAccel\u0026rsquo;s first-generation FPGA-based LPU and how it is provided. Next, we briefly introduce the development process of the device plugin for Bertha , our upcoming ASIC-based LPU. Finally, we introduce DRA (Dynamic Resource Allocation), a technology gaining significant attention, and explain how DRA relates to the scheduler and Device Plugin in Kubernetes.\nKubernetes Device Plugin The Kubernetes Device Plugin framework is a standard interface that allows custom devices such as GPUs, FPGAs, and NICs to be exposed and managed within a Kubernetes cluster.\nBy default, Kubernetes can only recognize CPU and memory resources. Without a Device Plugin, Kubernetes has no information about custom devices, making it difficult to distinguish between hardware, check device status, or use them as criteria for Pod scheduling. Device Plugins enable vendors to support their hardware without directly modifying Kubernetes source code.\nWhy Do We Need a Device Plugin? This approach was difficult to maintain because it required understanding the vast Kubernetes codebase. It also lacked scalability, since the code had to be modified every time new hardware was introduced.\nIn summary:\nImproved maintainability and scalability: Kubernetes focuses solely on container orchestration, while hardware control logic is handled by Device Plugins built by manufacturers (NVIDIA, AMD, HyperAccel, etc.).\nSupport for diverse hardware: Not just GPUs, but also high-performance networks (SR-IOV), cryptographic accelerators (QAT), and other resources can all be managed in a unified manner.\nHow Device Plugins Work \u0026amp; Workflow Device Plugins typically run as DaemonSets and communicate with Kubelet through gRPC services on each node. The process of making devices recognizable through communication between Kubelet and the Device Plugin is as follows:\nRegistration\nWhen the Plugin starts, it registers itself with Kubelet through a specific path on the node (/var/lib/kubelet/device-plugins/kubelet.sock).\nThis is the process of telling Kubelet: \u0026ldquo;I am a plugin that manages a resource called hyperaccel.ai/lpu!\u0026rdquo;\nListAndWatch\nKubelet periodically requests the list of available devices from the plugin.\nThe Plugin monitors and reports the status of devices (Healthy/Unhealthy) in real-time to Kubelet through the ListAndWatch method.\nAllocate\nAt the node level, when a user specifies resources.limits.hyperaccel.ai/lpu: 1 in the Pod spec, the scheduler places the Pod on a node that has available resources. The scheduler is responsible for selecting the node where the Pod will be placed.\nOnce the Pod is placed on a node through the scheduler, Kubelet calls the plugin\u0026rsquo;s Allocate method. The Plugin responds with the necessary configurations (environment variables, device node paths, volume mounts, etc.) so that the container can access the device.\nThe Device Plugin is responsible for selecting which device to allocate within the node.\nSo far, we have examined the process by which the Device Plugin communicates with Kubelet to make Kubernetes recognize devices. So what interface is needed when these two components communicate? Let\u0026rsquo;s look at the gRPC interface required for the Device Plugin to communicate with Kubelet.\nKey gRPC Interfaces Method Role GetDevicePluginOptions Checks plugin options (periodic checkpoints, etc.) ListAndWatch Returns the device list and streams status changes GetPreferredAllocation Provides functionality to guide specific hardware selection through preference conditions Allocate Provides configuration for using specific hardware during container creation PreStartContainer Initializes the device before the container starts (optional) Next, we will explain how Kubernetes internal components, including the Device Plugin, allocate devices to pods during the actual device request process.\nHow Devices Are Allocated to Pods We will use HyperAccel\u0026rsquo;s LPU as an example.\nDetecting Available LPUs on a Node First, let\u0026rsquo;s explain how the API Server and Kubelet detect and monitor new devices by interacting with the Device Plugin at the Kubernetes level.\nRegisterRequest (Device Registration Request)\nWhen the Device Plugin starts, it registers the plugin with Kubelet\u0026rsquo;s Device Plugin Manager through a Unix Domain Socket.\nThis is the process of saying: \u0026ldquo;This node has a resource called hyperaccel.ai/lpu, and I will manage it.\u0026rdquo;\nStart gRPC Server\nThe Device Plugin starts a gRPC server to communicate with Kubelet. Through this, Kubelet can query device status or send allocation (Allocate) requests. ListAndWatch\nKubelet calls the Device Plugin\u0026rsquo;s ListAndWatch() function.\nDuring this process, the Device Plugin detects the LPUs connected below and reports their status (Healthy/Unhealthy) to Kubelet. This connection is maintained continuously, so it immediately notifies if any device issues occur.\nUpdate Node Status\nKubelet sends the confirmed LPU resource information (e.g., hyperaccel.ai/lpu: 4) to the API Server.\nThe API Server records this information in etcd.\nNow the entire cluster knows that \u0026ldquo;4 LPUs are available on that node.\u0026rdquo;\nNow, when a user creates a pod and requests the following in the YAML file, the Kubernetes Scheduler can place the pod on the appropriate node:\n# User Pod YAML file resources: limits: hyperaccel.ai/lpu: 1 # Request 1 LPU Selecting and Allocating the Appropriate LPU for a Pod\u0026rsquo;s Request Once the Kubernetes level is ready to use the new device called LPU through the above process, we need to look at the actual allocation process. Let\u0026rsquo;s explain step by step what happens internally in Kubernetes when a user declares \u0026ldquo;I need 1 LPU\u0026rdquo; in the YAML file.\nBind(pod, node) - Pod Scheduling\nThe user requests hyperaccel.ai/lpu: 1 in the PodSpec.\nThe Kubernetes Scheduler searches for nodes with IDLE LPUs across all nodes.\nIt selects a worker node that meets the conditions and commands (Bind): \u0026ldquo;Place this pod on the designated node!\u0026rdquo;\nAllocateRequest\nKubelet confirms that the pod assigned to it requires an LPU to run.\nThe Device Plugin Manager inside Kubelet requests the Device Plugin: \u0026ldquo;I need 1 LPU, please prepare it for use.\u0026rdquo; It also passes the specific LPU ID (e.g., LPU 0) to be used.\nAllocateResponse\nThe Device Plugin responds to Kubelet with the necessary configuration information to make the LPU available in the container:\nEnvs: Environment variables to reference inside the container Devices: Device paths the container needs to access (e.g., /dev/lpu0) Mount: Mount paths for required libraries or driver files CreateContainer\nKubelet requests container creation from the container runtime (containerd) based on the received information.\nAt this point, containerd connects the host\u0026rsquo;s /dev/lpu0 device to the inside of the container as configured, ultimately allowing the running pod to directly access the LPU hardware.\nAs a result, users can use the new LPU device in a safe and isolated environment simply by adding a single line — limits: hyperaccel.ai/lpu: 1 — to the YAML file, without needing to know the complex hardware setup process!\nHowever, a question may arise from step 4 above. How does the container runtime (containerd) interpret and inject the device paths, environment variables, and mount information returned by the Device Plugin in the AllocateResponse? The standard that addresses this is CDI (Container Device Interface).\nCDI (Container Device Interface): Standardizing Device Injection The Problem Before CDI When the Device Plugin sends the necessary configuration information to Kubelet via AllocateResponse, the container runtime ultimately interprets this to create the container. However, in this process, each vendor had its own way of injecting devices into containers.\nFor example, using an NVIDIA GPU in a container requires more than simply mounting the /dev/nvidia0 device file. GPU driver libraries (libcuda.so, libnvidia-ml.so, etc.), related utility binaries, and the correct paths matching their exact versions all need to be properly injected into the container. To achieve this, NVIDIA developed its own container runtime hook called nvidia-container-runtime-hook.\nOther vendors\u0026rsquo; devices — FPGAs, network accelerators (SR-IOV), cryptographic accelerators (QAT), and more — each required their own unique methods to inject device nodes, driver libraries, and configuration files into containers. This led to the following problems:\nRuntime dependencies: Each vendor had to develop its own container runtime or runtime hook. NVIDIA\u0026rsquo;s nvidia-container-runtime, AMD\u0026rsquo;s xilinx-container-runtime, and others were all needed, and they were incompatible with each other.\nComplex configuration management: To use both a GPU and an FPGA on a single node, multiple runtime hooks had to be combined, increasing operational complexity.\nLack of portability: Device configurations that worked with Docker sometimes didn\u0026rsquo;t work with containerd or CRI-O, because each container runtime had different device injection mechanisms.\nWhat Is CDI? CDI (Container Device Interface) is a container runtime-level standard specification that emerged to solve these problems. CDI declaratively defines how devices are injected into containers through a single JSON spec file. Container runtimes (containerd, CRI-O, Podman, etc.) read this spec and inject the devices into the container.\nSimply put, before CDI, each vendor would say \u0026ldquo;To put my device in a container, you need to install this special runtime.\u0026rdquo; After CDI, it became \u0026ldquo;With just this JSON file, any runtime can inject my device.\u0026rdquo;\nStructure of a CDI Spec File CDI spec files are located at /etc/cdi/ or /var/run/cdi/ and have the following structure:\n{ \u0026#34;cdiVersion\u0026#34;: \u0026#34;0.6.0\u0026#34;, \u0026#34;kind\u0026#34;: \u0026#34;hyperaccel.ai/lpu\u0026#34;, \u0026#34;devices\u0026#34;: [ { \u0026#34;name\u0026#34;: \u0026#34;lpu0\u0026#34;, \u0026#34;containerEdits\u0026#34;: { \u0026#34;deviceNodes\u0026#34;: [ { \u0026#34;path\u0026#34;: \u0026#34;/dev/lpu0\u0026#34;, \u0026#34;hostPath\u0026#34;: \u0026#34;/dev/lpu0\u0026#34;, \u0026#34;permissions\u0026#34;: \u0026#34;rw\u0026#34; } ], \u0026#34;mounts\u0026#34;: [ { \u0026#34;hostPath\u0026#34;: \u0026#34;/opt/hyperaccel/lib\u0026#34;, \u0026#34;containerPath\u0026#34;: \u0026#34;/usr/lib/hyperaccel\u0026#34;, \u0026#34;options\u0026#34;: [\u0026#34;ro\u0026#34;, \u0026#34;nosuid\u0026#34;, \u0026#34;nodev\u0026#34;, \u0026#34;bind\u0026#34;] } ], \u0026#34;env\u0026#34;: [ \u0026#34;LPU_DEVICE_INDEX=0\u0026#34;, \u0026#34;LPU_VISIBLE_DEVICES=0\u0026#34; ] } } ], \u0026#34;containerEdits\u0026#34;: { \u0026#34;env\u0026#34;: [ \u0026#34;LPU_DRIVER_VERSION=1.0.0\u0026#34; ] } } The key components of the spec file are as follows:\nField Description kind A vendor domain-based name that identifies the type of device (e.g., hyperaccel.ai/lpu) devices A list of individual device definitions. Each device has a unique name containerEdits Declaratively defines changes to apply to the container deviceNodes Device file paths and permissions to expose inside the container mounts Mount configurations for driver libraries, firmware, and other required files env Environment variables to inject inside the container Device-level containerEdits are applied only when that specific device is requested, while top-level containerEdits are applied as common settings whenever any device of that kind is requested.\nHow CDI Works The device allocation process with CDI applied is as follows:\nCDI Spec Generation: The Device Plugin (or DRA Driver) detects devices on the node and generates CDI spec files for each device, storing them in the /etc/cdi/ path.\nCDI Device Name Passing: When allocating a device to a Pod, the Device Plugin includes the CDI device name (e.g., hyperaccel.ai/lpu=lpu0) in the AllocateResponse and passes it to Kubelet.\nContainer Runtime CDI Resolution: When Kubelet requests container creation from the container runtime (containerd), it passes the CDI device name along with the request. The container runtime finds and reads the corresponding spec file from the /etc/cdi/ path, and automatically injects the device nodes, mounts, and environment variables defined in containerEdits into the container.\nContainer Execution: The container runs with all device configurations applied, and the Pod can directly access the device from within.\nCompared to the AllocateResponse-based flow we examined earlier, the key difference is that the complex logic for injecting devices into containers has been separated from the Device Plugin into CDI spec files. The Device Plugin no longer needs to directly include device paths, mounts, and environment variables in the AllocateResponse — it only needs to pass the CDI device name. The rest is handled by the container runtime through the CDI spec.\nCDI Configuration by containerd Version To use CDI, the container runtime must have CDI support enabled. For containerd, the level of CDI support varies by version.\ncontainerd Version CDI Support Status Configuration Required Before 1.7 CDI not supported Not available 1.7 ~ 1.x CDI supported (default: disabled) Manual activation required 2.0 and above CDI supported (default: enabled) No additional configuration needed For containerd versions 1.7 and above but below 2.0, CDI must be explicitly enabled in /etc/containerd/config.toml.\n[plugins.\u0026#34;io.containerd.grpc.v1.cri\u0026#34;] enable_cdi = true cdi_spec_dirs = [\u0026#34;/etc/cdi\u0026#34;, \u0026#34;/var/run/cdi\u0026#34;] After changing the configuration, restart containerd to activate CDI.\nsudo systemctl restart containerd Starting from containerd 2.0, enable_cdi defaults to true, so CDI can be used immediately without any additional configuration.\nThe Relationship Between CDI and DRA CDI is also closely related to DRA (Dynamic Resource Allocation), which we will introduce later. DRA Drivers use CDI as the standard interface for injecting devices into containers. In other words, CDI started by standardizing device injection methods in the Device Plugin era and continues to play a pivotal role as a foundational technology in the DRA era as well. DRA will be covered in detail in a later section.\nDevice Plugin Examples So far, we have examined what the Kubernetes Device Plugin is, why it\u0026rsquo;s needed, and how it detects and allocates devices. Before diving into the Device Plugin for LPU, let\u0026rsquo;s look at examples provided by leading vendors NVIDIA and AMD.\nNVIDIA GPU Device Plugin As shown in the NVIDIA Cloud-Native Toolkit Stack Layer, NVIDIA presents a layered architecture for GPU utilization:\nLinux Distribution: Host OS level\nContainer Engine (Docker/containerd): Container runtime environment\nKubernetes: Pod scheduling and management\nGPU Operator (Top Layer): An operational tool that automates all these processes, with one of its core components being the NVIDIA Device Plugin.\nThe NVIDIA GPU Operator supports all basic Device Plugin functions — Expose Resources, Health Check, Device Allocation \u0026amp; Isolation — while also providing the following extensibility:\nFeature Description MIG (Multi-Instance GPU) Support Enables splitting a single physical GPU into multiple virtual GPUs via MIG Ecosystem Integration Works closely with dcgm-exporter, integrating with Prometheus (monitoring) and Grafana (visualization) Operational Automation Delivered as a modern pattern that manages everything from driver installation to plugin deployment through the GPU Operator AMD Xilinx Device Plugin Xilinx combines Node Feature Discovery (NFD) with the FPGA Operator to support seamless use of their FPGA devices in Kubernetes environments.\nNFD scans each node to determine \u0026ldquo;What Xilinx card is installed on this node?\u0026rdquo; and creates labels for the node. Since not only the model name but also the installed Shell version, PCIe information, and others are crucial for FPGAs, it provides foundational data so the scheduler can place pods based on FPGA models or specific feature support.\nOnce NFD creates labels on the node, the FPGA Operator uses them to bring the node to a READY state.\nComponent Description Host Setup Automatically installs XRT (Xilinx Runtime) drivers and firmware on each node Container Runtime Configures a dedicated runtime engine to enable tools like xbutil and FPGA resource access within containers Device Plugin Ultimately reports virtual resources such as xilinx.com/fpga to the API Server HyperAccel\u0026rsquo;s first-generation LPU is based on Xilinx FPGAs. In other words, the aforementioned Xilinx Device Plugin must be applied to utilize LPUs in a Kubernetes environment.\nHowever, directly applying the Xilinx Device Plugin as-is has functional limitations that make it difficult to use in a real server environment. What limitations exist, and how did we overcome them? Let\u0026rsquo;s now explain how we implemented the Device Plugin for LPU.\nKubernetes Device Plugin for HyperAccel LPU HyperAccel\u0026rsquo;s first-generation LPU was designed on an FPGA (Field-Programmable Gate Array), based on the Xilinx Alveo U55C device.\nXilinx provides the Xilinx FPGA Device Plugin as open source. By deploying this plugin as a DaemonSet in the cluster, Kubernetes can recognize FPGAs as resource hardware and select them as scheduling targets. FPGAs are recognized with names like amd.com/xilinx_u55c_gen3x16_xdma_base_3-0.\nChallenges with the Existing Xilinx Device Plugin Currently, HyperAccel provides Orion servers equipped with FPGA-based LPUs, with 8 LPU devices installed in each Orion server.\nThe 8 LPUs are connected in a Ring-Topology configuration.\nThe existing Xilinx FPGA Device Plugin can only convey information like \u0026ldquo;Is the number of IDLE devices on the current node greater than the number of devices requested by the current pod?\u0026rdquo; — so the scheduler selects the node for pod placement based solely on this. If allocation is done without considering device topology, connectivity issues can arise.\nLet\u0026rsquo;s look at an example. Assume a pod requests 4 LPUs within an Orion server. When a device allocation request comes in, the Device Plugin sends the list of IDLE devices to Kubelet, and Kubelet then randomly selects devices based on this list.\nIn such a situation, if devices are selected without considering connectivity:\nSince the 4 LPUs are not connected to each other, operations like distributed inference become difficult. (Parallel communication and LPU-to-LPU activation synchronization are impossible)\nIf a future pod requesting 4 LPUs is scheduled, the same problem occurs. (The IDLE device count is satisfied for scheduling, but there is no connectivity between devices)\nThis not only makes distributed workloads using multiple LPUs difficult, but also causes resource fragmentation within the Orion server, reducing server-level utilization.\nDue to these limitations, in internal development and testing environments with Orion servers and external PoC environments, only the approach of allocating either 1 or all LPUs could be applied. With this 1 or ALL allocation method, when multi-device development or testing is needed on Kubernetes, it becomes difficult for multiple developers to work simultaneously on a single Orion server. To overcome this problem, we developed the Orion Device Plugin in-house for more flexible device utilization in a Kubernetes-based development environment.\nOrion Device Plugin The Orion Device Plugin is a component that adds Topology-Aware allocation functionality on top of the existing Xilinx Device Plugin.\nUsing the in-house developed HyperDex-Toolchain, a network table showing the connectivity of LPUs within the Orion server can be extracted. Based on the number of LPUs requested by the Pod and the connectivity status of IDLE LPUs extracted from the network table, the optimal device allocation combination is determined. The Device Plugin conveys this list to Kubelet through the GetPreferredAllocation function described earlier, and Kubelet allocates LPUs to the Pod accordingly.\nUsing the developed component, the Orion server is currently being used in various ways:\nIn the Kubernetes-based in-house development environment, fully isolated LPUs within the Orion server are provided to FPGA developers. In other words, multiple developers can work simultaneously on a single Orion server without device interference.\nDuring HyperAccel Chat Demos running on the Orion server, multiple models can be served simultaneously on the same Orion server.\nWhen conducting customer PoCs, if there is a need to test on a Kubernetes environment, the Device Plugin is provided alongside.\nBertha Device Plugin HyperAccel\u0026rsquo;s next-generation ASIC chip, Bertha, is about to be released!\nSince Bertha is an entirely new proprietary ASIC chip from HyperAccel, utilizing it in a Kubernetes environment requires developing a Bertha Operator that includes components such as the Device Plugin, NFD, and Metric Exporter. We are currently developing the Cloud-Native Software Stack for Bertha together with Namyoon Kim (Author, LinkedIn) from the ML team.\nDRA in Kubernetes So far, we have introduced the Device Plugin — an essential component for utilizing new devices in Kubernetes — and explored how HyperAccel leverages it in our development environment. However, Device Plugins also have some limitations.\nLimitations of the Kubernetes Device Plugin Hardware management through Device Plugins has the following limitations:\nStatic allocation: Resource allocation is based simply on the number of devices requested.\nLack of complex configuration: It is difficult to dynamically reflect detailed hardware settings (e.g., GPU partitioning, memory bandwidth configuration) at Pod request time.\nDifficulty with resource sharing: Implementing scenarios where multiple pods flexibly share a single resource is complex.\nDRA (Dynamic Resource Allocation) Kubernetes DRA is a newly introduced framework actively being adopted to overcome the limitations of existing Device Plugins and manage custom hardware resources like GPUs and FPGAs with much greater flexibility and granularity.\nDRA is designed to request and allocate hardware resources in a manner similar to using storage (PVC/PV). Simply put, whereas before you could only say \u0026ldquo;Give me 1 LPU,\u0026rdquo; with DRA you can make much more complex requests like \u0026ldquo;Please dynamically allocate an LPU with settings optimized for a specific model\u0026rsquo;s inference.\u0026rdquo;\nCore Components of DRA DRA clearly separates how resources are defined and requested into Allocation and Usage.\nComponent Description ResourceClaim An object where users define and request needed resources (e.g., \u0026ldquo;1 GPU with 16GB or more\u0026rdquo;) — similar to PVC ResourceClass Defines the type of resource and the driver to handle it — similar to StorageClass ResourceSlice A data unit containing detailed information (model, capacity, status, etc.) about devices actually present on each node DRA Driver A plugin provided by specific hardware vendors, responsible for actual hardware preparation and allocation Device Plugin vs DRA Feature Device Plugin DRA Resource Model Integer-based (e.g., hyperaccel.ai/lpu: 1) Object-based (ResourceClaim) Parameter Passing Very limited (using Annotations, etc.) Highly flexible (Custom parameter support) Scheduling Simple count check Can reflect complex constraints Lifecycle Fixed at Pod execution time Can be allocated/released independently of the Pod With the release of Kubernetes v1.34 in the second half of 2025, major DRA features have transitioned to General Availability (GA). As a result, DRA is being standardized for managing accelerators like NVIDIA GPUs in the latest cloud environments (GKE, EKS, etc.) and on-premises AI clusters.\nHyperAccel is also keeping pace and preparing to support the use of Bertha with DRA in Kubernetes environments!\nIn Summary\u0026hellip; In this article — the third installment of the Building an In-House Dev Environment on Kubernetes series — we introduced what the Kubernetes Device Plugin is, the development process of the Device Plugin for utilizing HyperAccel\u0026rsquo;s LPU in Kubernetes, and DRA, a technology for more flexible resource allocation.\nFor a market-competitive LPU, it is important to design well at the HW level, but an optimized SW stack must also accompany it. Supporting the smooth use of LPU on top of the most widely used Kubernetes environment is a critical challenge for chip success!\nOur ML team\u0026rsquo;s DevOps division is developing Cloud-Native Toolkit software for HyperAccel LPU\u0026rsquo;s potential customers. Components such as Device Plugins and DRAs, which form the foundation of this software, enable seamless use of the LPU within computing nodes.\nThank you for reading to the end!\nP.S.: HyperAccel is Hiring! HyperAccel has all team members working hard toward the release of an LLM acceleration ASIC chip! As a company that covers not only HW and SW but also end-to-end inference AI technology, outstanding talents across all areas are working together. We are growing rapidly together, learning not just in a single field but gaining broad and deep knowledge while working with amazing colleagues!\nOur ML team\u0026rsquo;s DevOps division provides and manages development environments to boost in-house developer productivity, and develops Cloud-Native Toolkit to effectively support cloud-level utilization of LPU chips.\nIf you\u0026rsquo;re interested in the technologies HyperAccel works with, please apply through HyperAccel Career!\nReference Kubernetes 1.26: Device Manager graduates to GA CDI - Container Device Interface containerd CRI Plugin Configuration NVIDIA Device Plugin AMD Xilinx Device Plugin Dynamic Resource Allocation ","date":"2026.03.27","permalink":"https://hyperaccel.ai/techblog/en/posts/k8s-device-plugin/","series":["Building an In-House Dev Environment on Kubernetes"],"style":"--c:#0052CC;--on:#ffffff","summary":"Hello! I\u0026rsquo;m Younghoon Jun, a DevOps Engineer on the ML team at HyperAccel.\nThis post is the third installment of the Building an In-House Dev Environment on Kubernetes series!\nIn Part 1, we covered the background, overall design, and direction of building a Kubernetes-based development environment. Part 2 introduced the strategy and process for building an ARC-based CI/CD infrastructure to overcome the structural limitations of self-hosted runners. In this third article, we will discuss the Device Plugin required for utilizing custom resources on Kubernetes.\n","title":"[K8s Dev Environment - Part 03] Kubernetes Device Plugin for LPU"},{"authors":["Minho Park"],"category":"Compilers \u0026 Kernels","content":"AITER Analysis: How AMD Doubled ROCm Inference Performance Hello, I\u0026rsquo;m Minho Park from the HyperAccel ML team.\nSemi Analysis is a well-known semiconductor research firm. They run the InferenceX benchmark, which measures and compares inference performance of major GPUs.\nAccording to the InferenceX v2 report released in February 2026, AMD MI300X SGLang throughput improved by nearly 2× between December 2025 and January 2026. At the center of this improvement was a kernel library called AI Tensor Engine for ROCm (AITER).\nThat news made me curious. What is AITER, and how could software optimization alone nearly double hardware performance? In this post I summarize AITER’s architecture, kernel backend strategy, and the principles behind the speedup. It may be useful if you:\nWant to understand the current state of AMD GPU inference performance Are interested in GPU kernel optimization techniques Are curious about AI accelerator ecosystems beyond NVIDIA What is AITER? AI Tensor Engine for ROCm (AITER) is AMD’s open high-performance AI operator repository. In short, it is a collection of kernels for accelerating AI workloads on AMD GPUs. In the ROCm ecosystem, AITER plays a role analogous to cuDNN in the NVIDIA ecosystem.\nItem Description Repository ROCm/aiter License MIT Languages Python (63.6%), CUDA/HIP (25.8%), C++ (9.7%) Supported GPUs AMD Instinct MI300X, MI325X, MI350 Inference Frameworks vLLM, SGLang AITER’s main value is drop-in replacement. In inference frameworks such as vLLM or SGLang, turning on a single environment variable causes existing operators to be automatically replaced with AITER’s optimized kernels. You get better performance without changing application code.\nWhat Operations Does It Support? AITER supports a wide range of Large Language Model (LLM) inference operations.\nCategory Operations Backends Attention Flash Attention, Multi-head Latent Attention (MLA), Paged Attention CK, ASM, Triton Fused MoE Top-K routing, MoE sorting, BlockScale FP8 FFN HIP, CK, ASM GEMM FP8 per-token/channel, block-scale FP8, INT8, pre-shuffle CK, ASM Normalization RMSNorm, LayerNorm (including fused quantization) Triton, CK Embedding Rotary Position Embedding (RoPE) forward/backward Triton Quantization BF16/FP16 → FP8/INT4 conversion CK, Triton Communication AllReduce, reduce-scatter, all-gather Triton, HIP Notable here is the backend column: a single operation can have multiple backends (CK, ASM, Triton, etc.). This reflects AITER’s core design philosophy—a multi-backend strategy—which we’ll cover in more detail later.\nPerformance Benchmarks: AITER by the Numbers Seeing is believing. Here are the performance gains AITER achieves on MI300X.\nKernel/Workload Throughput Improvement Block-scale General Matrix Multiplication (GEMM) 2× Block-scale Fused Mixture of Experts (MoE) 3× MLA Decode 17× Multi-Head Attention (MHA) Prefill 14× DeepSeek V3/R1 throughput (SGLang) 2× (6,485 → 13,704 tok/s) DeepSeek R1 prefill latency ↓52% (3.13s → 1.51s) DeepSeek R1 decode latency ↓47% (0.053s → 0.028s) vs. NVIDIA H200 (DeepSeek R1) 2–5× higher throughput at same latency MLA Decode 17× and MHA Prefill 14× are not numbers you get from simple tuning. They are possible because AITER writes kernels at the assembly level.\nArchitecture High-Level Architecture AITER’s architecture is organized into five main layers.\nUser API: Python (torch-compatible) and C++ APIs High-level operators: Operator orchestrators such as tuned_gemm.py, fused_moe.py, mla.py Operator wrappers: Wrapping layer based on the @compile_ops decorator JIT compilation: Compiles kernels on first use and caches them as .so files Kernel backends: Four backends—Triton, Composable Kernel (CK), HIP, and ASM From the user’s perspective, you call PyTorch-compatible functions via import aiter. AITER handles selecting, compiling, and caching the best kernel.\nJIT Compilation Pipeline All AITER operators follow the @compile_ops decorator pattern.\n@compile_ops(\u0026#34;module_gemm_a8w8\u0026#34;, fc_name=\u0026#34;gemm_a8w8\u0026#34;, gen_func=cmdGenFunc, gen_fake=fake_shape_fn) def gemm_a8w8(XQ, WQ, x_scale, w_scale, Out): ... # Body is not executed — replaced by compiled C++ at runtime The flow is:\nThe decorated function is called from Python. The module cache is checked for an already-compiled .so file. If missing, module configuration is loaded from optCompilerConfig.json. The code is compiled with hipcc and Ninja to produce a .so file. The module is loaded with importlib.import_module() and cached. Subsequent calls use the cached kernel. Compilation cost is paid only on first run; afterward you get native-level performance. Eighty-four compile modules are managed this way.\nFour Kernel Backends AITER’s distinctive trait is that it does not rely on a single kernel language. It uses four backends, chosen per operator.\nTriton Triton is a Python-based GPU programming Domain-Specific Language (DSL) developed by OpenAI and ported to ROCm by AMD. It uses a block-level programming model: you write the algorithm at the tile level and the compiler handles optimization.\nIn AITER, the largest number of kernels (100+) are written in Triton. They are used mainly for utility-style operations such as RMSNorm, RoPE, quantization, and MoE sorting.\n# Triton RMSNorm kernel example @triton.jit def _rms_norm_fwd_kernel( X_ptr, W_ptr, Out_ptr, stride_x_row, N, eps, BLOCK_N: tl.constexpr, ): row_idx = tl.program_id(0) # Each program handles one row cols = tl.arange(0, BLOCK_N) mask = cols \u0026lt; N x = tl.load(X_ptr + row_idx * stride_x_row + cols, mask=mask, other=0.0) x_sq = x * x mean_sq = tl.sum(x_sq, axis=0) / N rrms = tl.rsqrt(mean_sq + eps) # reciprocal square root w = tl.load(W_ptr + cols, mask=mask) out = x * rrms * w tl.store(Out_ptr + row_idx * stride_x_row + cols, out, mask=mask) Triton’s strength is development speed: you write GPU kernels in Python, and the compiler handles memory coalescing and shared-memory management. On the other hand, compiler limitations make it hard to reach more than ~95% of theoretical bandwidth.\nComposable Kernel (CK) CK is AMD’s C++ template-based high-performance kernel library, analogous to NVIDIA’s CUTLASS. As the name “composable kernels” suggests, you build complex operators by combining reusable tile-operation building blocks.\nCK’s focus is operation fusion. For example, you can fuse scaling of GEMM results inline, without a separate kernel.\n// Fuse dequantization inline with GEMM result template \u0026lt;typename AccDataType, typename DDataType, typename EDataType\u0026gt; struct RowwiseScale { __host__ __device__ constexpr void operator()( EDataType \u0026amp;e, const AccDataType \u0026amp;c, const DDataType \u0026amp;d0, // weight scale const DDataType \u0026amp;d1 // activation scale ) const { const F32 x = ck::type_convert\u0026lt;F32\u0026gt;(c) * ck::type_convert\u0026lt;F32\u0026gt;(d0) * ck::type_convert\u0026lt;F32\u0026gt;(d1); e = ck::type_convert\u0026lt;EDataType\u0026gt;(x); } }; // → GEMM computation + dequantization run in a single kernel In AITER, CK is used for matrix-centric kernels: GEMM (A8W8, A4W4, block-scale), MoE 2-stage, and some Attention. A heuristic dispatcher chooses among 100+ pre-tuned instances based on M, N, K shapes.\nHIP Heterogeneous-computing Interface for Portability (HIP) is AMD’s C++ runtime API and kernel language. It is the AMD counterpart to CUDA; most CUDA kernel code runs on HIP with little more than name changes.\nCUDA concept → HIP equivalent ───────────────────────────────────────── cudaMalloc() → hipMalloc() cudaMemcpy() → hipMemcpy() __global__ void kernel → __global__ void kernel (same!) __shared__ → __shared__ (same!) cudaStream_t → hipStream_t nvcc → hipcc AITER uses HIP for general-purpose operations such as Paged Attention, KV cache, TopK, and AllReduce. Thirty-four .cu files under csrc/kernels/ are written this way.\nAssembly (ASM) The ASM backend consists of kernels written directly in AMDGCN Instruction Set Architecture (ISA), the machine code of AMD GPUs. All compiler abstraction is bypassed; the developer has full control over GPU registers, instruction scheduling, and memory access patterns.\nWhy go this low? High-level compilers generate safe code but miss optimization opportunities. ASM can fully exploit the GPU’s Matrix Fused Multiply-Add (MFMA) instructions, special registers, and instruction pipelining. The result is MLA Decode 17× and MHA Prefill 14× improvements.\nRegister types: SGPR (Scalar GPR) — control flow, constants (shared across threads) VGPR (Vector GPR) — data operations (per-thread values) AccVGPR — MFMA accumulator (stores matrix multiply result) Key instructions: v_mfma_f32_32x32x8_f16 — 32×32 matrix multiplication (FP16→FP32) v_mfma_f32_16x16x32_fp8 — 16×16 matrix multiplication (FP8→FP32) buffer_load_dwordx4 — Load 128 bits from global memory ds_read_b128 — Load 128 bits from LDS AITER’s hsa/ directory contains 354+ precompiled ASM kernels (.co files). So many instances are needed because ASM kernels fix all parameters at compile time. Every viable combination is precompiled with no runtime branching.\nhead_dim: {64, 128, 256} → 3 options dtype: {fp16, bf16, fp8} → 3 options causal: {true, false} → 2 options → Combinatorial explosion → 162+ FMHA instances Backend Selection Criteria AITER’s criteria for choosing a backend per operation can be summarized as follows.\nCriterion Triton CK HIP ASM Development speed ★★★★★ ★★☆☆☆ ★★★☆☆ ★☆☆☆☆ Peak performance ★★★☆☆ ★★★★☆ ★★★☆☆ ★★★★★ Portability ★★★★★ ★★★☆☆ ★★★★☆ ★☆☆☆☆ Maintainability ★★★★★ ★★★☆☆ ★★★★☆ ★☆☆☆☆ Kernel count 100+ ~20 34 354+ (.co) You might ask: “Why not write everything in ASM?” ASM is very hard to develop and maintain, and when the GPU architecture changes, you have to rewrite from scratch. AITER uses ASM only on the hottest paths where performance matters most, and keeps development efficiency with Triton and CK elsewhere.\nFramework Integration: Enable with One Environment Variable AITER integrates without touching existing code. You only need to set environment variables.\n# Enable AITER in vLLM VLLM_USE_AITER_MOE=1 VLLM_USE_AITER_BLOCK_GEMM=1 \\ vllm serve deepseek-ai/DeepSeek-V3 --tensor-parallel-size 8 # Enable AITER in SGLang CK_BLOCK_GEMM=1 SGLANG_ROCM_AITER_BLOCK_MOE=1 \\ python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-V3 --tp 8 Frameworks read these variables and conditionally dispatch to AITER operators. That makes A/B testing straightforward: turn the variables on or off to compare performance with and without AITER.\nOn first run, JIT compilation is triggered and .so files are created under ~/.cache/aiter/. Later runs reuse the cached kernels, so there is no extra cost.\nSummary of Key Design Patterns Here are the design patterns that stood out while reviewing AITER.\nMulti-Backend Kernel Dispatch AITER does not commit to a single kernel language. It uses ASM for decode attention, CK for GEMM, and Triton for MoE sorting—a different approach from a unified-compiler strategy.\nFP8 Block-Scale Quantization Per-token (1×128) activation scale plus per-weight (128×128) scale enables efficient mixed-precision operations. This is especially effective for MoE architectures like DeepSeek.\nCSV-Based Auto-Tuning Kernel parameters are stored in CSV files per (M, N, K, cu_num) shape. Tuning per model is possible without recompilation. Pre-tuned settings exist for 21 models including DeepSeek V3, Qwen3, and LLaMA 405B.\nHardware-Specific Optimizations AITER includes hardware-level optimizations such as custom AllReduce tailored to MI300X inter-GPU topology and weight pre-shuffling (16×16 layout) for better memory access patterns.\nConclusion AITER is AMD’s kernel library to narrow the AI inference performance gap with NVIDIA. Instead of depending on one kernel language, it uses four backends—Triton, CK, HIP, and ASM—and selects the right one per operator.\nAs noted in Semi Analysis’s InferenceX v2 report, AITER’s optimizations nearly doubled MI300X SGLang performance. At the same time, as Semi Analysis pointed out, individual kernel performance is strong, but when many optimizations—FP4, disaggregated serving, expert parallelism, etc.—are combined, composability still lags behind NVIDIA.\nWhat AITER demonstrates is clear: software optimization alone can double performance on the same hardware. For us, working on NPU development, it’s a reminder of how important it is to turn theoretical hardware performance into real-world performance.\nReferences GitHub: ROCm/aiter Semi Analysis: InferenceX v2 — NVIDIA Blackwell Vs AMD vs Hopper Semi Analysis: AMD 2.0 — New Sense of Urgency AMD Blog: AITER — AI Tensor Engine For ROCm AMD Blog: Accelerate DeepSeek-R1 with AITER + SGLang P.S. HyperAccel Careers Analyzing AITER drove home how much difference kernel-level optimization can make. At our NPU company we are also building a software stack that can fully utilize hardware performance. With our LPU ASIC launch ahead, we are looking for teammates to work on everything from kernel optimization to inference framework integration.\nCareers: https://hyperaccel.career.greetinghr.com/ko/guide\nIf you’re interested, we’d love to hear from you!\n","date":"2026.03.17","permalink":"https://hyperaccel.ai/techblog/en/posts/rocm-aiter/","series":[],"style":"--c:#5BC5F6;--on:#02121c","summary":"An analysis of AITER (AI Tensor Engine for ROCm), which boosts inference performance on AMD GPUs.","title":"AITER Analysis: How AMD Doubled ROCm Inference Performance"},{"authors":["Hyunjun Park"],"category":"LLM \u0026 Models","content":"Hi, I\u0026rsquo;m Hyunjun Park from the HyperAccel ML Team. Since ChatGPT launched in November 2022, AI technology has been advancing at an exponential pace, with new AI tools emerging almost every day.\nIn hardware, Moore\u0026rsquo;s Law states that chip transistor density doubles every two years. More recently, research shows that LLM autonomous task performance doubles every seven months. While early services drew ridicule for confidently describing Admiral Yi Sun-sin\u0026rsquo;s smartphone, today\u0026rsquo;s services handle natural language, images, video, and audio with remarkable quality. At the heart of this progress is the Transformer architecture.\nI believe that to use any tool well, you need to understand how it works in detail. In this post, we first explore the background behind the Transformer. Then we look at how it works, and finally examine the bottlenecks and a few fundamental optimization techniques to address them.\nPart 1: Background The Architecture That Changed the World: Transformer In 2017, a Google research team published a paper titled \u0026ldquo;Attention Is All You Need.\u0026rdquo; This paper not only solved the speed limitations of previous sequential models (RNN/LSTM) by processing entire sentences at once as matrices, but also significantly improved performance with an \u0026ldquo;Attention\u0026rdquo; mechanism that mimics how humans understand sequences. Today\u0026rsquo;s most popular LLMs — including Gemini and ChatGPT — are built on the Transformer architecture proposed in this paper.\nWhat LLMs Do: Next-Word Prediction Mathematically, a Large Language Model (LLM) is \u0026ldquo;a sophisticated mathematical function that predicts the next word given some text.\u0026rdquo; Rather than deterministically picking a single word, it assigns a probability to every possible next word. Because it samples from this probability distribution each time, the same question can produce different answers on different runs. Training is the process of making these predictions increasingly accurate by processing enormous amounts of text.\nIn practice, an LLM is a \u0026ldquo;magic machine that plausibly predicts the next word for any given sentence.\u0026rdquo; Imagine someone asking an LLM to write a short movie script. The user feeds a prompt to the LLM; this magic machine then repeatedly appends the predicted word and feeds the result back in until the script is complete. This is exactly what happens when we chat with a chatbot.\nPart 2: Breaking Down the Transformer Architecture Overview Although details vary by model, we\u0026rsquo;ll use the simplest model — GPT-2 — as our reference. A Transformer can be divided into three major stages: Token Embedding, Decoder Block, and LM Head. The Decoder Block is typically a stack of n identical blocks (24 in GPT-2\u0026rsquo;s case). Each block can be further broken down into six steps. This is illustrated in the figure below.\nToken \u0026amp; Positional Embedding The first stage of the Transformer converts human language into a language computers can understand. This is called Token Embedding, where each token is mapped to a long list of numbers — a vector. (In practice, the mapping between words and tokens isn\u0026rsquo;t strictly one-to-one, but we\u0026rsquo;ll assume it is for simplicity.) After this conversion, each word becomes a numeric vector that carries the word\u0026rsquo;s meaning. Since the same word can mean different things depending on its position in a sentence, Positional Embedding is also applied.\nLet\u0026rsquo;s look at GPT-2 Medium as an example. This model\u0026rsquo;s vocabulary size is 50,287, and each token is encoded as a vector of size 1024 (the embedding dimension). We also call this 1024 dimensions. The Word Token Embedding Table (WTE) has shape [50287, 1024], mapping each token to a 1,024-dimensional vector.\nThe Word Positional Embedding Table (WPE) has shape [1024, 1024], mapping each position in the sequence to a 1,024-dimensional vector. By looking up the token ID sequence in the WTE to get [seq_len, 1024] and the position indices in the WPE to get [seq_len, 1024], then adding the two matrices, we have the input ready for the Decoder Block.\nLayer Normalization Layer Normalization (LayerNorm) is the first step of the Decoder Block. It normalizes each token\u0026rsquo;s vector by adjusting its mean and variance — the same normalization concept you may have learned in math class. If the input distribution varies wildly from layer to layer, training becomes unstable and gradients can explode or vanish. LayerNorm keeps each position\u0026rsquo;s vector at an \u0026ldquo;appropriate scale and distribution,\u0026rdquo; stabilizing training and enabling deep networks with many layers.\nSelf-Attention: The Meaning of Query, Key, and Value The Problem Attention Solves: Context-Dependent Meaning Let\u0026rsquo;s use the word tower as an example. Right after embedding, it gets a vector pointing in the general direction of \u0026ldquo;tall structure.\u0026rdquo; But if Eiffel appears right before it, that vector gets updated toward \u0026ldquo;Eiffel Tower.\u0026rdquo; If miniature is added further before, the association with \u0026ldquo;large\u0026rdquo; weakens and the meaning becomes more specific. Information transfer between tokens can span long distances and capture rich context. To predict the next word in \u0026ldquo;Therefore the murderer was\u0026hellip;\u0026rdquo; at the end of a mystery novel, the last vector was must absorb context from the entire preceding text. Attention computes exactly \u0026ldquo;which token should attend to which other token, and by how much.\u0026rdquo;\nQ, K, V Generation and Intuitive Meaning GPT-2 Medium accepts at most 1024 input tokens. Since each token is converted to a vector of size 1024, the Attention input is at most [1024, 1024] (= [seq_len, N_embed]). We multiply this by three different weight matrices W_Q, W_K, W_V to produce the Query (Q), Key (K), and Value (V) matrices. The embedding dimension (N_embed=1024) is split into the number of heads (N_head=16) and per-head dimension (head_dim=64), so each of Q, K, V has shape [seq_len, N_head, head_dim].\nFor intuition, consider predicting the blank in \u0026ldquo;Hello, my name is ____.\u0026rdquo; Recall from Part 1 that an LLM is a next-word prediction machine. To fill in the blank, the last token is needs to gather information from the preceding words. Here\u0026rsquo;s what Q, K, and V each do:\nQuery (Q) — The question that is asks: \u0026ldquo;What information do I need from the preceding context to predict the next word?\u0026rdquo; This question is encoded in is\u0026rsquo;s Q vector. Key (K) — The name tag each preceding word holds up. Hello carries a tag like \u0026ldquo;I\u0026rsquo;m a greeting,\u0026rdquo; my carries \u0026ldquo;I\u0026rsquo;m a possessive,\u0026rdquo; and name carries \u0026ldquo;I\u0026rsquo;m about a name.\u0026rdquo; When we compare (dot-product) is\u0026rsquo;s Q with each word\u0026rsquo;s K, name scores the highest — because we need to figure out \u0026ldquo;what the name is.\u0026rdquo; Value (V) — The actual content that high-scoring words deliver. name\u0026rsquo;s V contains the meaning \u0026ldquo;a name,\u0026rdquo; and this information is combined proportionally by scores to update is\u0026rsquo;s vector. After this update, is\u0026rsquo;s vector now condenses the context \u0026ldquo;greeting + my + name + is,\u0026rdquo; and when this vector passes through the LM Head, a person\u0026rsquo;s name like John comes out with high probability. In summary: Q is \u0026ldquo;what do I need to predict the next word (question),\u0026rdquo; K is \u0026ldquo;what kind of clue do I hold (name tag),\u0026rdquo; and V is \u0026ldquo;the actual meaning I deliver (content).\u0026rdquo;\nQuery · Keyᵀ: Similarity Scores and Masking We mentioned earlier that there are N_head heads. Within a single head, both Q and K have shape [N_position, head_dim]. By transposing K to Kᵀ [head_dim, N_position] and computing Score = Q · Kᵀ, we get an [N_position, N_position] matrix. Rows represent \u0026ldquo;the token holding the current Query,\u0026rdquo; columns represent \u0026ldquo;all referenceable positions,\u0026rdquo; and element (i, j) is the score for how important token i considers token j. The more aligned the Query and Key vectors are, the larger the dot product.\nJust as humans read text from left to right, Attention must ensure that past tokens cannot see future tokens. This is achieved through masking: setting scores at \u0026ldquo;positions after me\u0026rdquo; to −∞ before Softmax. After Softmax, these become 0, preventing later tokens from influencing earlier ones — in other words, blocking \u0026ldquo;future tokens from modifying past tokens.\u0026rdquo;\nSoftmax \u0026amp; Value Weighted Sum We\u0026rsquo;re almost there — let\u0026rsquo;s push through! Softmax is applied to each row (each Query position) of the Score matrix. Due to the nature of the softmax operation, each row sums to 1, turning the scores into a probability distribution representing \u0026ldquo;how much weight to give each other token.\u0026rdquo; For example, raw scores like \u0026ldquo;gym 2, tennis 1, bed 0.1\u0026rdquo; become \u0026ldquo;gym 0.7, tennis 0.2, bed 0.1\u0026rdquo; — numbers go from scores to probabilities. At each Query position, we take the weighted sum of all positions\u0026rsquo; Value vectors using these weights, producing the update vector (ΔE) to add to that position.\nMulti-Head Attention and Output Projection When entering the Attention layer, [seq_len, N_embed] was split into [seq_len, N_head, head_dim]. The concept of heads was adopted to improve model performance. A single head might specialize in one type of relationship like \u0026ldquo;adjective–noun,\u0026rdquo; but language has diverse ways of shifting context. So we split into multiple heads with different Q, K, V matrices and process each head in parallel to enhance contextual understanding. Each head produces an [N_position, head_dim] output in a different subspace, then these are concatenated into [N_position, N_embed].\nFinally, Output Projection (FC): ConcatenatedAttention · W_O + b_O applies an [N_embed, N_embed] weight matrix to merge information from all heads back into a single representation space. This is the final output of one block\u0026rsquo;s Multi-Head Attention. The hard part is done!\nResidual Connection The connection that adds the Self-Attention output to the original input is called a Residual Connection (skip connection). In the form y = x + AttentionOutput, the name sounds grand but it\u0026rsquo;s essentially matrix addition. By preserving the original information x and only adding the delta captured by Attention, gradients flow well even in deep networks, keeping training stable.\nFeed-Forward Network (FFN) As a reminder, a single decoder layer consists of 6 steps (LayerNorm–Attention–Residual–LayerNorm–FFN–Residual). Since LayerNorm and Residual are repeated, let\u0026rsquo;s wrap up with the Feed-Forward Network (FFN). The FFN consists of two Linear layers with an Activation Function in between. The input [N_position, N_embed] is expanded by the first Linear layer, passed through an activation (e.g., GeLU), then compressed back by the second Linear layer to produce [N_position, N_embed]. The same MLP is applied independently at each position (position-wise FFN), mapping the information gathered by Attention into a more complex feature space via nonlinear transformation. After completing all 6 steps, one decoder block is done, and repeating this process N times (24 for GPT-2) completes the entire Decoder Block computation.\nLM Head: Next-Token Prediction The final stage, LM Head, converts vectors back into words that humans can understand. It typically uses the transpose of the Token Embedding Table (WTE) to compute logits. Applying Softmax to these logits produces a probability distribution over the entire vocabulary for the next token.\nWhile one could simply output the highest-probability token, sampling is generally used to draw one token according to the probability distribution. Two key parameters control the sampling range: top_k and top_p. Using both together prevents \u0026ldquo;absurd words from being selected\u0026rdquo; while maintaining natural diversity.\nCongratulations! If you\u0026rsquo;ve followed along this far, you\u0026rsquo;ve grasped the full basic structure of the Transformer. It\u0026rsquo;s perfectly fine if not everything clicks on the first read. The Transformer is a recent architecture and by no means an easy topic. This post focused on concrete numbers and the meaning behind each operation, but as you continue reading explanations from various perspectives, it will gradually become more familiar. Now let\u0026rsquo;s move on to the final part and cover some key optimization techniques.\nPart 3: Optimization Techniques KV Cache and Memory/Bandwidth Issues The most important operation from Part 2 is undoubtedly Attention. The Score matrix produced by multiplying Query and Key has size proportional to the square of the context size. With N tokens, it\u0026rsquo;s an N×N matrix, so computation and memory grow rapidly as context lengthens. During inference, previously computed Key and Value are stored in a KV cache for reuse, but as context grows, this cache becomes enormous, making context length, memory, and bandwidth major bottlenecks.\nFor example, consider running LLaMA-3-70B with bfloat16 (bf16) at a 1-million-token context. Model parameters: 70B × 2 bytes ≈ 140 GB. The KV cache works out to roughly 2 (K,V) × batch 1 × 80 layers × 8 KV heads × 128 head_dim × 1M sequence × 2 bytes ≈ 328 GB. The total memory that must be read to generate one token is about 468 GB (model + KV cache), and generating 20 tokens per second would theoretically require ~10 TB/s of memory bandwidth. Considering that NVIDIA\u0026rsquo;s latest B100 GPU has a maximum memory bandwidth of 8 TB/s and only 192 GB of memory, this gives a sense of the scale involved.\nKV Cache Architecture Comparison (MHA, MQA, GQA, MLA) There are four main approaches depending on how KV is stored and shared: MHA, MQA, GQA, and MLA. The core trade-off is between accuracy (expressiveness) and KV cache size (memory/bandwidth). A paper that visually illustrates these differences well is attached below.\nMulti-Head Attention (MHA) The original structure proposed in the Transformer paper. Each Query head has its own dedicated K, V head — a 1 Query : 1 KV relationship. Used by GPT-3, LLaMA 1/2, and others.\nPros: Each head has independent K, V, providing the highest expressiveness and the ability to capture diverse patterns simultaneously. Most advantageous in terms of accuracy. Cons: KV cache must be stored for every head, so memory usage and bandwidth requirements grow rapidly as context lengthens. Multi-Query Attention (MQA) All Query heads share a single K, V head. N Query : 1 KV. Used by PaLM, Falcon, StarCoder, and others.\nPros: KV cache size drops to 1/N_head of MHA, significantly reducing memory and bandwidth overhead during inference. Much more favorable for increasing batch size or processing long contexts. Cons: Since all heads see the same K, V, the ability to capture different relationships per head is reduced. There\u0026rsquo;s an accuracy trade-off compared to MHA. Grouped-Query Attention (GQA) Query heads are divided into several groups, with each group sharing one KV pair. A middle ground between MHA (1:1) and MQA (N:1), with an N Query : M KV relationship. For example, with 32 Query heads and 8 KV heads, every 4 Query heads share 1 KV head. Used by LLaMA-2-70B, LLaMA-3, Mistral, and others.\nPros: More KV heads than MQA means higher expressiveness, while smaller KV cache than MHA. Widely regarded as the most balanced practical choice between accuracy and efficiency. Cons: Compared to MHA, KV is still shared across heads, limiting expressiveness. Cannot reduce memory as much as MQA. Multi-Head Latent Attention (MLA) K, V are compressed (projected) into low-dimensional latent vectors, and Attention is performed in that latent space. Only these small latent vectors need to be stored in the KV cache instead of the original K, V. Used by DeepSeek-V2/V3 and others.\nPros: By compressing KV cache to a much smaller latent dimension, memory efficiency surpasses even GQA. At the same time, independent Q per head is maintained, avoiding significant loss of expressiveness. Cons: Additional computation for latent projection (compression/decompression matrix multiplications) is required, and implementation complexity is high. Information loss can occur during compression, so the compression ratio must be carefully tuned. When to Use Which (Situational Guide) The four approaches above are not \u0026ldquo;one size fits all\u0026rdquo; — the choice depends on context length, memory, quality requirements, and implementation environment. In a fast-moving LLM ecosystem, older design choices are often forgotten, while hearing only \u0026ldquo;GQA is the norm\u0026rdquo; can make it unclear why MHA or MQA might still be the right call. Below is a guide to when each approach is worth considering.\nMHA — When context is short (e.g., hundreds to a few thousand tokens) and accuracy and expressiveness are the top priority. It still makes sense for small research or experimental models, or services where KV cache pressure is relatively low. It was the default in the early Transformer, GPT-3, and LLaMA 1/2 era, and is the one to consider when you want to keep the \u0026ldquo;one independent K, V per head\u0026rdquo; design. MQA — When memory and bandwidth are severely constrained and you need very long context or large batch sizes. With a single shared K, V for all Query heads (as in PaLM, Falcon, StarCoder), cache is minimal. Expressiveness can lag behind MHA and GQA, so it suits deployments where throughput and scale matter more than marginal quality gains. GQA — When you need a balance of accuracy and efficiency in general production. Adopted by most modern open LLMs (LLaMA-2-70B, LLaMA-3, Mistral, etc.). If unsure which to pick, default to GQA and consider MQA or MLA only when context is extremely long or memory is critically limited. MLA — When you need long context and memory efficiency together and want more aggressive cache reduction than GQA. Compressing K, V into a latent space (as in DeepSeek-V2/V3) adds implementation and tuning cost, but fits the goal of \u0026ldquo;keeping per-head Q while shrinking cache.\u0026rdquo; Best considered by teams with enough infrastructure and experimentation bandwidth. In short, no single tuning choice is always correct — it only makes sense in the right situation. Keeping in mind that short conversational services and million-token document services may each benefit from a different architecture helps when making these trade-offs.\nSummary In this post, we explored how LLMs work and the meaning of each operation that makes up an LLM, with appropriate analogies. To recap: the Transformer architecture consists of three major stages — Token Embedding, Decoder Block, and LM Head — and the Decoder Block can be further broken down into LayerNorm, Attention, Residual, LayerNorm, FFN, and Residual. Finally, we compared the O(N²) cost with context length, KV cache memory/bandwidth issues, fundamental architectural variations like MHA, MQA, GQA, and MLA, and when to consider each approach.\nReference https://arxiv.org/pdf/2209.10797\nhttps://arxiv.org/pdf/2503.11486\nhttps://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/\nHyperAccel is Hiring! We are an NPU design startup growing at the forefront of LLM inference optimization. This post was written based on HyperAccel\u0026rsquo;s internal training materials — sharing expertise and growing together is one of our core strengths.\nIf you\u0026rsquo;re interested in the technologies we work with and want to join this journey, please apply at the link below. HyperAccel Career\n","date":"2026.03.05","permalink":"https://hyperaccel.ai/techblog/en/posts/what-is-the-transformers/","series":[],"style":"--c:#003A70;--on:#ffffff","summary":"A hands-on walkthrough of Transformer-based LLM internals — from each module\u0026rsquo;s role to key optimization techniques.","title":"Transformer World: A Deep Dive into the Building Blocks of LLMs"},{"authors":["Donghyeon Choi"],"category":"Compilers \u0026 Kernels","content":" Know Your Enemy, Know Yourself (知彼知己 百戰不殆)\nIf you know your enemy and know yourself, you need not fear the result of a hundred battles.\nThis series aims to deeply understand competitors\u0026rsquo; hardware for AI accelerator design.\nThis article covers Pallas, a programming model that enables writing custom kernels on TPU.\nHello, I\u0026rsquo;m Donghyeon Choi, an engineer on the HyperAccel ML team.\nIn part 2, we explored TPU\u0026rsquo;s hardware architecture and software stack. In this article, we\u0026rsquo;ll take a deeper look at Pallas, one component of the TPU software stack.\nIn the latest TPU generation, Ironwood, the Pallas programming model plays a crucial role in performance optimization. Pallas is a tool that allows writing custom kernels on TPU, similar to CUDA or Triton. It enables direct control over hardware details while remaining relatively easy to use in a Python environment.\nAs mentioned in part 2 regarding the limitations of the XLA compiler, automatic compilers often struggle to optimize cutting-edge algorithms. Pallas is the kernel language Google created to overcome these limitations. When announced alongside Ironwood, it was emphasized with the slogan \u0026ldquo;Extreme performance: Custom kernels via Pallas.\u0026rdquo; It plays an important role in maximizing TPU performance.\nTherefore, as a supplement to this series, we examine the Pallas programming model before moving on. In this article, we\u0026rsquo;ll explore what Pallas is, why it\u0026rsquo;s needed, and how it enhances TPU performance.\nBackground: TPU Architecture and Why Pallas is Needed To understand Pallas, we first need to understand how TPU differs from CPUs or GPUs.\nTraditional Architecture vs TPU\u0026rsquo;s Systolic Array Traditional CPU or GPU architectures repeatedly fetch and store data from SRAM to register files for each operation. In operations like matrix multiplication that repeatedly use the same data, this approach causes massive memory bandwidth waste.\nTPU solves this by adopting a structure where once-loaded data flows between compute units and is continuously calculated. This is the Systolic Array structure described in part 2.\nOperations within TPU follow this flow:\nWeight Supply: Weight data stored in DRAM is loaded into Matrix Multiply Unit (MXU) through Weight FIFO. Activation Supply: Input data stored in Unified Buffer (UB) is delivered to MXU. Matrix Operations: Multiplication and accumulation (Multiply-Accumulate) operations occur simultaneously through the Systolic Array structure inside MXU. Post-processing Pipeline: MXU output values pass through Accumulator, Activation Unit (ReLU, etc.), and Normalize/Pool Unit sequentially. These units perform additional operations required by AI models. Result Storage: Final output after all operation flows returns to Unified Buffer and is stored. Key Difference from GPU: Change in Execution Unit This structural difference creates a very significant difference from a programming model perspective.\nGPU (Single Instruction Multiple Thread, SIMT): Numerous threads perform calculations independently, managed in groups of 32 threads called Warps. It features fine-grained parallel processing focused on individual data. TPU (Single Program Multiple Data, SPMD): A single data load completes the entire operation sequence. That is, rather than individual threads, \u0026rsquo;the entire program for one tensor (or tile)\u0026rsquo; is considered as one minimum execution unit. Why Pallas? Pallas is a language designed to abstract TPU hardware characteristics (direct Unified Buffer control, MXU scheduling, etc.) while allowing developers to directly optimize this \u0026rsquo;tensor-level flow'.\nTPU has evolved through generations. It has strengthened vector operation units and developed scale-up technologies for large-scale models. The latest architecture, Ironwood, has become a \u0026lsquo;monster compute unit\u0026rsquo; with a massive package adopting chiplet structure and mounting four 256x256 Systolic Arrays (MXU). Despite the dramatic increase in hardware scale, the core design principle remains unchanged: \u0026lsquo;performing as many operations as possible through a single data load\u0026rsquo;.\nWhy Pallas is Needed When we discussed the XLA compiler in part 2, we mentioned that while XLA is a powerful optimization compiler, it has limitations. When new operation algorithms emerge, the compiler struggles to match the performance of manually created custom kernels. This remains true until it\u0026rsquo;s updated to a version that can optimize them.\nFor example, cutting-edge algorithms like Flash Attention or MoE (Mixture of Experts) often have complex memory access patterns or high data dependencies. This makes them difficult for automatic compilers to optimize. In such cases, developers need to directly understand the memory hierarchy. They must finely control how to tile data, when to move data between memory levels, and so on.\nOn GPUs, this problem was solved with CUDA or Triton. CUDA allows direct hardware control but has a high barrier to entry. Triton has a higher level of abstraction, making it relatively easier to use. However, both only work on GPUs.\nWhat about TPU? Google began providing Pallas as an experimental extension of JAX around 2023. Pallas shares a similar philosophy with Triton. It differs significantly in that it supports both GPUs and TPUs.\nWhat is Pallas? Pallas is a kernel language that enables writing custom kernels within the JAX ecosystem. It works on both GPUs and TPUs. It allows direct control over hardware memory hierarchy, data tiling, and block-level parallelism.\nPallas\u0026rsquo;s core idea is simple. It allows developers to control operations that are difficult for high-level automated compilers to handle, at a level closer to hardware. Unlike CUDA, it doesn\u0026rsquo;t go completely low-level. It is abstracted to be relatively easy to use in a Python environment. It finds a balance between the convenience of automatic compilers and the control of manual optimization.\nKey Abstractions: Grid, BlockSpec, Ref Pallas controls hardware through three key abstractions: Grid, BlockSpec, and Ref.\nGrid: Parallel Execution Abstraction\nPallas models execution units through a Grid abstraction. Grid has different meanings depending on hardware. It provides a unified interface regardless.\ndef kernel(o_ref): i = pl.program_id(0) # Current program\u0026#39;s ID o_ref[i] = i # Grid size specification: (8,) = 8 parallel programs result = pl.pallas_call( kernel, out_shape=jax.ShapeDtypeStruct((8,), jnp.int32), grid=(8,), # 8 parallel instances )() In Pallas, Grid is a collection of execution units that divide the entire work. However, how hardware processes this Grid has fundamental differences depending on the architecture.\nGPU Grid\nIn GPU backends (Triton/Mosaic), Grid assumes fully parallel execution by hardware schedulers. Each Grid item is mapped to one Thread Block and executed independently on individual Streaming Multiprocessors (SMs). Since hardware schedules threads non-deterministically, execution order between Grids is not guaranteed. This is an important consideration during development. When designing BlockSpec\u0026rsquo;s index_map, you must strictly manage race conditions. Different programs must not write to the same High Bandwidth Memory (HBM) location.\nTPU Grid\nIn TPU backends, Grid is a model that combines parallelism between multiple cores and sequential pipelining within a single core.\nTPU is a very wide SIMD machine, but can be coded like a single-threaded processor from a software perspective. The Tensor Control System (TCS) provides an intuitive flow that controls entire operations by looping.\nWhen multiple TensorCores exist, there are Parallel dimensions that distribute work and execute physically simultaneously. There are also Sequential dimensions that guarantee serial execution within a single TensorCore. Sequential dimensions are not for slow execution. They are a strategic means to hide memory latency by overlapping current computation with loading of the next data through Semaphores.\nBlockSpec: Memory Layout Abstraction\nPallas abstracts the process of dividing massive data into chunks that hardware can digest through BlockSpec. This goes beyond simply cutting data. It defines data transfer protocols between HBM (Remote) and SRAM (Local).\nBlockSpec has three components:\nblock_shape: The size of data that each program instance will place on the Local Memory (SRAM) workbench. Designing this not to exceed hardware SRAM capacity is key to performance optimization. index_map function: Takes grid indices (i, j) as input and returns the block start position on HBM. At compile time, this function is analyzed and converted to hardware Direct Memory Access (DMA) address calculation logic. memory_space: Specifies the physical container where fragmented data will reside. If not specified, it\u0026rsquo;s allocated to the default space (pl.SRAM) according to backend settings. Ref: Memory Reference Abstraction\nPallas abstracts complex hardware memory address systems through Ref objects. This goes beyond simple pointers to data and provides a logical view of specific data blocks on SRAM (Local Memory).\nRef\u0026rsquo;s key features:\nLocal Memory Reference: Ref points to data on the fastest workbench in hardware—SRAM (TPU\u0026rsquo;s VMEM/SMEM, GPU\u0026rsquo;s Shared Memory)—not HBM. The data has already been loaded or will be loaded there. Dereferencing: When using brackets like x_ref[...], actual data loading from SRAM → Register File occurs, converting it to a Value. Hardware Abstraction: Even using the same Ref interface, it automatically converts depending on the backend. It maps to TPU\u0026rsquo;s Vector/Scalar memory or GPU\u0026rsquo;s Shared Memory access. You can also set it directly for clear optimization. Pallas Hardware Modeling Pallas provides hardware models for both TPU and GPU. They share the same big picture but differ in details. Let\u0026rsquo;s first look at the overall structure that\u0026rsquo;s common to both.\nThe common hardware model that Pallas abstracts has the following hierarchy:\nRemote Memory (HBM): High Bandwidth Memory—the slowest but highest capacity memory space. Typically specified explicitly through pltpu.HBM or automatically through pl.ANY. Multiple \u0026ldquo;Core\u0026rdquo; structure: A collection of independent compute units for parallel grid processing. Each core can perform computations independently. Local Memory (SRAM or Cache): Fast memory located inside cores. Much faster than Remote Memory but with limited capacity. Separating READ and WRITE variables is recommended for consistency with HBM data. Register Files in Execution Units: The fastest memory that can directly exchange data with compute units. Provides data immediately needed for computation. Generally, Local Memory takes about 10x the latency of Register File. Remote Memory takes another 10x the latency of Local Memory. However, in TPU v7 Ironwood adopting chiplet structure, one chip contains HBM. For this reason, Remote Memory communication can be 2~5x faster than the previous 10x.\nPallas TPU Hardware Model\nTPU\u0026rsquo;s hardware model is based on the common model but has TPU-specific structures:\nMemory Hierarchy:\nLocal Memory consists of Vector Memory and Scalar Memory:\nVector Memory (VMEM): Memory that stores data for Vector and Matrix related operations. Accessible from Vector/Matrix Units (VPU/MXU/XLU) within the same tensor core. Scalar Memory (SMEM): Memory that stores data for scalar operations for logic flow (loops, conditions, etc.). Accessible from Scalar Unit (TCS) within the same tensor core. TCS generates high-level commands through which SMEM data can be directly used in vector operations. Each VMEM and SMEM has separate READ/WRITE areas to ensure data sync with HBM. This is because modifying data read as READ without write-back can cause data inconsistency with HBM.\nData Flow and Pipelining:\nWhen calling a kernel, data is loaded from Remote Memory (HBM) to Local Memory (VMEM/SMEM) according to the specified BlockSpec. Data corresponding to the set Grid number is automatically fetched.\nWhen calling Ref data inside the kernel, data is fetched from Local Memory to Register File for computation. During Local Memory ↔ Register File transfer and computation, data loading from Remote Memory → Local Memory for the next Grid automatically overlaps. This can hide memory latency significantly.\nWhen the kernel writes result data to the configured output Ref, it is written back to Local Memory. When all programs for that grid finish, the output Ref data is written back to Remote Memory.\nPallas GPU Hardware Model\nGPU\u0026rsquo;s hardware model is also based on the common model but has GPU-specific structures:\nMemory Hierarchy:\nLocal Memory is Shared Memory (SMEM): Each Streaming Multiprocessor (SM) has independent Shared Memory/L1 Cache space. Unlike TPU, it\u0026rsquo;s a unified workspace without Scalar/Vector distinction. Numerous threads access this space simultaneously for parallel operations. Data Flow and Pipelining:\nUnlike existing methods that rely only on hardware scheduling, Pallas GPU explicitly overlaps HBM (Remote) → SMEM (Local) data loading with TensorCore operations. Through plgpu.emit_pipeline, etc., compute units process current data while data for the next grid is prefetched asynchronously. This hides memory latency.\nRef variables inside kernels are addresses of data chunks on Local Memory (SMEM), not HBM. The data has already been loaded or will be loaded there. Through memory_space=plgpu.GPUMemorySpace.GMEM, HBM (Remote) space can be explicitly specified. You can control it directly with copy_gmem_to_smem.\nMemory Pipelining and Semaphore Data movement between HBM (Remote Memory) and VMEM (Local Memory) in TPU architecture causes latency of hundreds of cycles. MXU may perform matrix operations quickly. But if it has to wait for data to arrive, overall performance will be bottlenecked by memory. Pallas solves this with pipelining using hardware-level Semaphores.\nPallas Synchronization Mechanism: Semaphore-based Async Control\nIn Pallas, DMA and compute units operate independently. Semaphores are used to coordinate their speed difference. The mechanism works as a mutual signaling system:\nData Load (Producer): When DMA completes loading data from external memory (HBM) to local memory (VMEM/SMEM), it increments (Signals) the semaphore value to indicate that data is ready. Compute (Consumer): Compute units check the semaphore value and wait until data is loaded. When the value is satisfied, they immediately start computation. Feedback Loop: When computation completes, compute units Signal the semaphore again to notify DMA that the buffer is free, allowing it to load the next data. Through this Wait-Signal structure, data loading and computation are overlapped for execution, maximizing hardware utilization.\nDouble Buffering: Key Technique for Hiding Latency\nThe most basic pipelining technique using semaphores is Double Buffering. Two buffers (Buffer 0, Buffer 1) are allocated within Local Memory, and they operate as follows:\nInitialization: DMA loads the first data into Buffer 0 Loop Start: While VPU computes Buffer 0, DMA loads the next data into Buffer 1 simultaneously Buffer Swap: When both computation and load complete, roles switch—VPU works on Buffer 1, DMA loads into Buffer 0 Repeat: This process continues until all data is processed As a result, memory load time is completely hidden behind computation time. Especially effective for compute-bound workloads where computation takes longer than data loading.\nPallas allows explicit control of this synchronization through pltpu.semaphore. It prevents dangerous situations at the hardware level—\u0026lsquo;Read-before-ready (reading before data arrives)\u0026rsquo; or \u0026lsquo;Write-over-active (overwriting during computation)\u0026rsquo;. The compiler automatically optimizes Prefetch and Overlap.\nBackend Lowering and JAX Integration Kernels written in Pallas are eventually converted to hardware code. On GPUs, they are lowered through Triton or Mosaic GPU backends. On TPUs, they are lowered through the Mosaic compiler to MLIR form and finally converted to hardware code. During this process, operator fusion, tiling automation, and overlapping of data transfer and computation are optimized. High-level code written by developers is transformed into hardware-optimized code.\nPallas kernels are also compatible with JAX transforms like jit, vmap, and grad. You can write high-performance kernels while still utilizing automatic differentiation, mapping, and compilation. Pallas is more than a kernel language. It\u0026rsquo;s a tool fully integrated with the JAX ecosystem.\nCUDA vs Pallas: Programming Model Comparison We compare CUDA from part 1 with Pallas.\nKey Comparison Category CUDA Pallas Abstraction Thread-centric (Thread → Warp → Block → Grid) Data-centric (Grid + BlockSpec + Ref) Memory Control Explicit (__shared__, __global__) Declarative (memory_space, auto mapping) Synchronization Manual (__syncthreads()) Automatic (semaphore-based pipelining) Hardware NVIDIA GPU only GPU + TPU support Ecosystem Mature (Nsight, cuBLAS, cuDNN) Experimental (JAX integration) Language C/C++ Python Code Comparison: Vector Addition CUDA - Direct thread ID calculation, boundary check required:\n__global__ void vector_add(float *A, float *B, float *C, int N) { int idx = blockIdx.x * blockDim.x + threadIdx.x; if (idx \u0026lt; N) C[idx] = A[idx] + B[idx]; } Pallas - Data block-level abstraction:\ndef vector_add_kernel(a_ref, b_ref, c_ref): c_ref[...] = a_ref[...] + b_ref[...] result = pl.pallas_call(vector_add_kernel, out_shape=..., grid=(N,))(a, b) This is a conceptual example. For actual execution, out_shape and BlockSpec must be completed.\nSelection Guide Choose CUDA Choose Pallas Maximum performance on NVIDIA GPU needed GPU/TPU portability needed Production stability important Quick prototyping Leverage existing CUDA codebase Leverage JAX ecosystem (jit, vmap, grad) Mature debugging/profiling tools needed Writing TPU custom kernels How Pallas is Used on TPU In the latest TPU generation, Ironwood, Pallas is centered with the slogan \u0026ldquo;Extreme performance: Custom kernels via Pallas.\u0026rdquo; In the Ironwood stack, Pallas definitions allow developers to directly describe strategies for memory tiling, data movement, and MXU utilization within Python. The Mosaic compiler lowers this to TPU code.\nThrough this combination, tiling strategies like input stationary, weight stationary, and output stationary can be efficiently designed. Distributed processing at stride and batch levels can be designed efficiently as well. HBM ↔ on-chip memory data transfer can be overlapped with MXU computation. Designs that minimize scheduling bottlenecks in the entire pipeline are also possible.\nMaximum Performance Potential: Memory access patterns and tiling strategies are often difficult for automatic compilers to discover. By adjusting them explicitly, you can create very fast kernels.\nHardware Abstraction Maintained: You still work within the Python language and JAX ecosystem. You can use features like jit and grad as-is.\nMulti-Backend Support: Supports both GPUs and TPUs, providing flexibility to run the same kernel definition on multiple hardware platforms.\nConsiderations Experimental Stage: Pallas is still in an experimental stage with frequent changes. Breaking changes may occur with version updates. Some features may be incomplete or throw \u0026ldquo;unimplemented\u0026rdquo; errors.\nPractical Portability: While Pallas has backends for both GPU and TPU platforms, identical code does not run optimally on both systems. Pallas\u0026rsquo;s portability today is closer to functional compatibility.\nNeed for Debugging and Optimization Tools: Tools for detailed analysis of MXU utilization, memory bandwidth utilization, etc. are lacking compared to the CUDA ecosystem.\nAdoption Barrier: If you have only GPUs without TPUs, using Triton with PyTorch is more practical. Pallas, which is JAX-based and typically requires GCP for TPU access, is difficult to adopt in the general ML stack (PyTorch + AWS or on-premise servers).\nConclusion In this article, we explored the Pallas programming model that enables writing custom kernels on TPU.\nFirst, we examined how TPU\u0026rsquo;s Systolic Array structure differs from CPU/GPU. We also looked at how these hardware characteristics affect the programming model. TPU adopts a tensor-centric execution model. It has the structural feature of performing the entire operation sequence through a single data load.\nPallas emerged to address areas difficult for the XLA compiler to automatically optimize. It enables direct control over hardware memory hierarchy and data tiling through the key abstractions of Grid, BlockSpec, and Ref. It provides memory models tailored to TPU and GPU (e.g., TPU\u0026rsquo;s VMEM/SMEM, GPU\u0026rsquo;s Shared Memory). It maintains an abstraction level that\u0026rsquo;s relatively easy to use in a Python environment.\nThrough comparison with CUDA, we confirmed that Pallas provides a higher level of abstraction while maintaining hardware control. We also saw how Pallas has become a key tool for \u0026ldquo;Extreme performance\u0026rdquo; in the Ironwood generation.\nPallas is an important tool that bridges the gap between automated compilers and manual optimization. It enables efficient handling of the complex memory access patterns required by cutting-edge algorithms. As non-standard operation patterns like Flash Attention, MoE, and sparse operations become increasingly important, the role of custom kernel tools like Pallas will only grow.\nReference Pallas Documentation Pallas TPU Details Inside the Ironwood TPU codesigned AI stack PyTorch/XLA 2.4 improves Pallas and adds eager mode TPU Architecture (qsysarch.com) TPU v7 Documentation P.S. HyperAccel is Hiring! \u0026ldquo;Know your enemy, know yourself,\u0026rdquo; but to win every battle, we need many talented people!\nIf you\u0026rsquo;re interested in the technologies we work with, please apply at HyperAccel Career!\nHyperAccel has many excellent and brilliant engineers. We\u0026rsquo;re waiting for your application.\n","date":"2026.02.25","permalink":"https://hyperaccel.ai/techblog/en/posts/pallas-programming-model/","series":["Know Your Enemy, Know Yourself"],"style":"--c:#5BC5F6;--on:#02121c","summary":"Learn about Pallas programming model that enables writing custom kernels on TPU.","title":"[Know Your Enemy - Extra] Pallas Programming Model"},{"authors":["Jaewon Lim"],"category":"Memory \u0026 Storage","content":" \u0026ldquo;If you know the enemy and know yourself, you need not fear the result of a hundred battles.\u0026rdquo;\nThis series aims to deeply understand competitors\u0026rsquo; hardware for the purpose of AI accelerator design.\nIn this fourth installment, we cover ICMS (Inference Context Memory Storage), announced by NVIDIA CEO Jensen Huang at his CES keynote earlier this year, and the DPU (Data Processing Unit) that manages it.\nHello! I\u0026rsquo;m Jaewon Lim, a hardware verification engineer on the DV team at HyperAccel.\nToday\u0026rsquo;s post begins with NVIDIA CEO Jensen Huang\u0026rsquo;s keynote session at CES 2026 last month. Most of the content I\u0026rsquo;ll be introducing is covered in the latter half of this keynote video. I\u0026rsquo;ve included the link in the references section below, so if you\u0026rsquo;d like to learn more, please refer to the full video.\nThe Rubin Platform Every year, Jensen Huang\u0026rsquo;s keynote serves as an important indicator of the direction of AI advancement and what the semiconductor industry needs to deliver. In particular, the latest NVIDIA GPU lineup set to launch that year and the evolution of the various components that make up the GPU pod are introduced together.\nWhat CEO Jensen Huang emphasized in this year\u0026rsquo;s keynote is the Rubin Platform — a datacenter product built around the Rubin GPU, scheduled for official release in the second half of this year.\nThe section highlighted with a rectangle in the image above is the NVL72 rack, equipped with 36 Vera CPUs and 72 Rubin GPUs. This rack contains 18 compute trays stacked together, with each tray housing 2 CPUs and 4 GPUs. Additionally, each tray contains other chips beyond CPUs and GPUs, designed for communication with other trays or devices such as storage.\nLooking at all the chips that make up the Vera Rubin Platform together, we get the image below. While the performance improvements in CPU and GPU are certainly remarkable, what we\u0026rsquo;re focusing on today is not the CPU or GPU, but the chip in the 5th position in the figure below: the Bluefield-4 DPU. To understand why we should pay attention to the DPU, we need to explore the new bottleneck in AI — a topic Jensen Huang introduced after the Rubin Platform presentation.\nContext Is the New Bottleneck: When Memory Capacity Itself Becomes the Bottleneck Memory bandwidth is not the only bottleneck. Memory capacity itself has become a new bottleneck. In the previous post, we explained why LLM computation is memory-bound and the background behind the increasing demand for HBM. Since the KV cache must be fetched from memory every time a token is generated, memory bandwidth became the bottleneck.\nHowever, a new problem has recently emerged: the KV cache has grown so large that it can no longer fit in GPU memory. This is because recent LLM development has been moving in the direction of increasing context length.\nRepresentative examples include reasoning models, first introduced by OpenAI in late 2024, and agentic AI such as Claude\u0026rsquo;s collaborative workflows.\nReasoning Models Reasoning models, led by OpenAI\u0026rsquo;s o1 and o3, have gained significant attention for their remarkable performance improvements. They solve complex problems by taking extended \u0026rsquo;thinking time.\u0026rsquo; During this process, the model internally generates numerous \u0026rsquo;thinking tokens\u0026rsquo; before producing the final answer.\nInternal reasoning tokens: Even if not visible to the user, the \u0026rsquo;thinking\u0026rsquo; tokens generated by the model to follow logical steps occupy a significant portion of the overall context. Self-correction: As the model repeatedly discovers and corrects errors during the reasoning process, even a question consisting of a few hundred tokens can rapidly accumulate thousands of tokens of context. Agentic AI Agentic AI goes beyond simply answering questions — it autonomously makes plans, uses tools, and interacts with its environment. This process is carried out through repeated loops of \u0026lsquo;observe-judge-act.\u0026rsquo;\nMaintaining interaction history: For consistent behavior, the agent must keep all records in its context — what tools it used, what the results were, and what the next plan is. Large volumes of environmental data: The sheer volume of data an agent must process — web page screenshots, entire codebases, extensive documents — causes the context window requirements to skyrocket. Iterative refinement: When the agent doesn\u0026rsquo;t succeed on the first try, the process of analyzing failure causes and retrying accumulates, consuming tens of times more tokens compared to a typical chatbot. To give you a sense of the capacity requirements, here is a numerical example.\nThe formula for calculating the KV cache size per token in a Transformer architecture is as follows:\n$$\\text{Total KV Cache Memory} = 2 \\times B \\times S \\times L \\times H_{kv} \\times D_h \\times P$$\n$$B : batch\\ size$$ $$S : sequence\\ length$$ $$L : num\\ layer$$ $$H_{kv} : num\\ head$$ $$D_{h} : head\\ dimension$$ $$P : precision(BF16: 2B,\\ FP8: 1B)$$\nApplying this formula to the Llama 3.1 405B model (whose model config is publicly available), approximately 516KB of KV cache is used per token. If we assume a context length of around 100K per user, the KV cache required per user is 48GB. If 128 users are active simultaneously, the instantaneous KV cache requirement grows to 6TB.\nThat\u0026rsquo;s 6TB of memory needed for the KV cache alone, excluding model parameters. Even by the standards of the yet-to-be-released Rubin GPU, this amounts to the HBM of approximately 20 GPUs. In practice, memory-offloading techniques are already being used to address this — storing KV cache in the CPU\u0026rsquo;s DRAM alongside the GPU, or using other GPUs\u0026rsquo; HBM to expand the available storage.\nThen why not offload KV cache to neighboring compute tray GPUs connected via NVLink? With 72 GPUs in a single rack, theoretically over 20TB of HBM memory could be shared. While this may seem like the fastest approach at first glance, there are several issues at datacenter scale:\nOpportunity cost of HBM: HBM is too expensive and scarce a resource to use as simple storage. The moment you offload your KV cache to another GPU\u0026rsquo;s HBM, that GPU loses space to handle its own new requests — essentially stealing its KV cache capacity. From a datacenter operator\u0026rsquo;s perspective, GPU utilization rates must be kept high to generate revenue, making it an inefficient choice to use HBM as mere storage. NVLink traffic congestion: NVLink traffic is already dominated by communication for Model Parallelism. If KV cache sharing between GPUs becomes frequent, it encroaches on the communication bandwidth needed for actual model computation, causing traffic congestion. Failure propagation risk: If the neighboring GPU node holding your KV cache encounters a problem, your inference task stops too — creating a dependency issue. Beyond NVLink-based sharing, there is yet another problem. So far, we\u0026rsquo;ve only considered the scenario where a user sends a single request. But users don\u0026rsquo;t send just one request. What happens when they send multiple requests? What if one of those requests requires the same KV cache as a previous request? If the previously generated KV cache was already evicted due to capacity constraints, the GPU must recompute tens to hundreds of gigabytes of KV cache from scratch. As we discussed last time, recomputing this volume of data is an enormous waste of both computation and time.\nThe Need for KV Cache Storage Ultimately, in a datacenter environment, we need a dedicated shared storage solution that allows each GPU to focus solely on computation while also being able to persistently retain large volumes of KV cache without eviction.\nThis means we have no choice but to leverage memory from lower tiers of the memory hierarchy. The first option that comes to mind is storing KV cache on the local SSDs installed inside each server tray. However, local SSDs have two fundamental limitations:\nCapacity constraints: The number of SSDs that can fit inside a single server tray is very limited due to space and power constraints. Sharing problem: KV cache stored on one server\u0026rsquo;s local SSD cannot be directly read by another server. It must be transferred over the network with CPU involvement, which is far too slow. On the other hand, using general-purpose remote network storage (such as NAS) from across the datacenter provides sufficient capacity, but network latency is too high to meet the bandwidth demands of LLM serving. Local SSDs are fast but too small and cannot be shared; network storage is large but too slow.\nThis is precisely the problem NVIDIA aims to solve with the DPU.\nNVIDIA\u0026rsquo;s Solution: ICMS (Inference Context Memory Storage) NVIDIA introduced ICMS — referred to as the G3.5 tier — between local SSDs (G3 tier) and network storage (G4 tier) to solve this problem. It combines the fast bandwidth of local SSDs with the massive sharing capability of network storage.\nAlthough ICMS resides in a dedicated separate rack, it is directly connected via BlueField-4 DPU and PCIe Gen 6, delivering speeds nearly indistinguishable from reading local SSDs. At the same time, it serves as a massive shared memory pool that all GPUs can access through the DPU as if it were their own local memory. NVIDIA\u0026rsquo;s goal is to extend the memory available for GPU inference computation from HBM and DRAM all the way to SSD (NAND Flash) through ICMS. The highlighted section of the Rubin Platform rack shown earlier is this ICMS rack. Through this, each GPU in the Rubin Platform can utilize 1TB of existing HBM + an additional 16TB of storage as if it were its own HBM memory. The Bluefield-4 DPU, mounted alongside the GPU in the compute tray, manages the data communication between GPU and ICMS.\nSoftware Optimization to Overcome Bandwidth Limitations However, looking at the DPU\u0026rsquo;s spec sheet raises a question. While HBM bandwidth is 2.2 TBps, DPU bandwidth is 100 GBps — approximately 1/20th the level. Even if the memory capacity available to each GPU has increased, wouldn\u0026rsquo;t a memory bandwidth bottleneck occur if data from that memory arrives at a speed far slower than HBM?\nThis is a perfectly reasonable concern. However, this limitation can be overcome by leveraging the usage characteristics of KV cache — the very memory that ICMS targets. KV cache has the characteristic of repeatedly reusing previously generated KV cache during the decode phase. Therefore, prefetching the KV cache needed for the next token computation into the GPU ahead of time poses no significant issue. This is a domain where software-based prediction is entirely feasible.\nInference Optimization Frameworks: vLLM, SGLang, and LMCache Various frameworks have been developed to optimize inference computation by leveraging the KV cache characteristics described above. SGLang, which Jaewoo Kim (Author, LinkedIn) introduced in a previous post, is one of them.\nvLLM \u0026amp; SGLang vLLM and SGLang are inference acceleration engines that provide functionality for efficiently managing KV cache within GPU memory. One such feature is prefix caching: when the the beginning(prefix) of a sequence in a given request overlaps with the beginning of a sequence in the next request, the KV cache generated from the previous request can be directly reused. Reusing KV cache generated within the same request during decoding — for each token generation — has always been possible by default. This feature goes one step further: since the same input tokens will produce the same KV cache regardless of which request they come from, KV cache can be reused across different requests without any additional computation.\nHowever, the limitations of vLLM and SGLang are that their KV cache management scope is confined to GPU memory (+ CPU host memory), and prefix caching can only be used when input tokens are identical from the very beginning. Even if the same subsequence appears in the middle, it cannot be reused if the preceding prefix differs. This is because of the nature of causal attention: the KV cache generated for a given token is influenced by all tokens that come before it. This is also why the feature is called \u0026ldquo;prefix\u0026rdquo; caching.\nLMCache LMCache is a distributed cache management framework dedicated to KV cache, created to overcome these limitations. It provides the ability to prefetch the KV cache needed for the next computation from device memory concurrently during the current computation.\nFurthermore, it supports non-prefix cache reuse using the CacheBlend technique, which overcomes the limitations of prefix caching. By applying additional computation for token position and prior context to KV cache generated at different positions, it can compute the KV cache with far less computation than recalculating it from scratch.\nLMCache is also integrated into NVIDIA\u0026rsquo;s inference framework Dynamo, and Dynamo with LMCache functionality is utilized within ICMS. HyperAccel is also developing software to support LMCache on the LPU.\nThe Role of Bluefield-4 DPU The DPU is a co-processor that uses this software to perform computation for data transfer and memory offload between the GPU and other devices, efficiently managing data movement. The DPU internally houses a Grace CPU — the predecessor to the Vera CPU — enabling it to run the storage management software described above. Previously, the CPU handled most of these tasks, but with the DPU taking over data movement operations, the existing CPU can reduce its computational burden and focus its resources on LLM scheduling and other computations. Additionally, the DPU accelerates data encryption functionality essential for AI cloud services, helping to build a confidential computing environment.\nSummary Today, we explored the emerging storage bottleneck in LLMs, along with NVIDIA\u0026rsquo;s new ICMS platform designed to address it, the DPU — a new processor that manages it — and the software frameworks that run on the DPU. Through this, we gained insight into the hardware and software technologies NVIDIA employs to extend GPU-accessible memory all the way to flash memory.\nBut what if we could place that flash memory right next to the chip for even faster access? HBF (High Bandwidth Flash) is an emerging technology that does exactly that. We\u0026rsquo;ll cover HBF in another series that focuses on new technology trends.\nIn the next post, we\u0026rsquo;ll return to the topic of accelerators. We\u0026rsquo;ll look at Cerebras — the company that drew attention for serving OpenAI\u0026rsquo;s code generation model Codex-Spark — and their Wafer Scale Engine, an AI accelerator startup that builds chips the size of an entire wafer.\nP.S.: HyperAccel is Hiring! HyperAccel is on the verge of launching its first datacenter LPU product, and we are developing solutions through hardware and software optimization to address the critical LLM inference bottlenecks.\nIf our technical journey interests you, please apply now through HyperAccel Career!\nHyperAccel is waiting for your application.\nReference CES 2026 Jensen Huang Keynote LMCache KV Cache Calculator Inside the NVIDIA Rubin Platform Introducing Bluefield-4 powered ICMS LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion NVIDIA Dynamo LMCache integration ","date":"2026.02.24","permalink":"https://hyperaccel.ai/techblog/en/posts/nvidia-icms-dpu/","series":["Know Your Enemy, Know Yourself"],"style":"--c:#009EE2;--on:#02121c","summary":"We explore the technical principles behind NVIDIA\u0026rsquo;s ICMS — a new storage tier designed to solve the KV cache capacity bottleneck in LLMs — and the Bluefield-4 DPU that manages it.","title":"[Know Your Enemy - Part 04] Memory Capacity Bottleneck and NVIDIA ICMS"},{"authors":["Taeseo Um"],"category":"Engineering Culture","content":"Hello, my name is Taeseo Um, currently on the ML team at HyperAccel.\nThere is a running joke among developers that our job has always been about googling the issue, clicking the first link that pops up, then copying from Stack Overflow. In some ways, the arrival of AI coding tools didn\u0026rsquo;t change what we do — it just made us dramatically better at it. Over the past few months, our team has ridden almost every wave of AI-assisted development, from the earliest interactions to the autonomous agents we rely on today. Here\u0026rsquo;s what that journey looked like, what we learned, and where we think it\u0026rsquo;s all heading.\nThe Early Days: AI as a Smarter Search Engine When large language models first became accessible to the world, nobody had a playbook, especially developers. The general mass treated LLMs in various ways: friend, teacher, personal therapist, a generic search engine. What about developers? The initial use case was embarrassingly simple: copy an error message, paste it into ChatGPT, and hope for a useful answer. It was Stack Overflow with a conversational interface, and honestly, it worked surprisingly well. Instead of scrolling through forum threads to find the one answer that matched your exact Python version, you could get a tailored explanation in seconds.\nFrom there, usage naturally expanded. Developers started asking syntax questions, requesting code snippets, and getting explanations of unfamiliar APIs. The mental model was still very much \u0026ldquo;I have this specific question, give me an answer that matches what I need.” But a subtle shift was already underway. Once people realized these models could hold context, they started feeding in larger chunks of code — sometimes entire files — and asking higher-level questions about architecture, design patterns, and debugging strategies. The AI wasn\u0026rsquo;t writing all our code yet, but it was becoming a surprisingly competent pair-programming partner (one where I don’t feel embarrassed asking dumb questions to).\nTab Complete Changed Everything The real inflection point came with tools like GitHub Copilot and Cursor. Autocomplete had existed in IDEs for decades, but this was fundamentally different. Instead of suggesting variable names and method signatures, these tools were predicting entire blocks of logic. You\u0026rsquo;d start typing something, maybe even just a comment, and the rest would appear like magic. Press tab. Accept. Move on. Forget.\nIt\u0026rsquo;s hard to overstate how much this changed the rhythm of coding. Writing boilerplate, repetitive patterns, and standard implementations went from minutes to seconds. Developers got faster, and perhaps more importantly, they stayed in flow longer. The friction of context-switching to documentation or search engines basically vanished.\nThen the agentic experience arrived. Tools like Cursor evolved beyond simple autocomplete into something more ambitious. You could describe a problem in natural language, and the LLM / IDE would traverse your codebase, identify the relevant files, propose changes across multiple locations, and ask you to review and accept. The developer\u0026rsquo;s role shifted from writing code to directing and reviewing it. It felt like having a junior engineer who could read your entire repository in milliseconds.\nCLI Agents and the Rise of Structured AI Workflows As powerful as IDE-integrated tools were, we found they still lacked structure. When Cursor introduced the concept of rules — markdown files that defined coding standards, commenting conventions, and general information — we jumped on it. We added rule files to nearly every repository so that anyone using AI-assisted tooling would at least produce code that followed our team\u0026rsquo;s standards. It was a simple idea with substantial impact.\nThe next leap was MCPs (Model Context Protocols), which let AI tools connect to external data sources. Suddenly, an agent could pull context from our Confluence documentation, understand our internal documents, and reference design decisions that lived outside the codebase. The AI wasn\u0026rsquo;t just reading code anymore — it was reading our institutional knowledge.\nHowever, what truly attached a rocket booster to our team’s “let’s improve productivity with AI” mindset was Claude Code. Claude Code introduced the concept of skills, which go far beyond static rules and instead enable structured, multi-step workflows.\nIn one of our team repositories, we now maintain a .claude directory to manage skills for planning, feature implementation, writing tests, opening PRs, fetching and incorporating PR review comments, and more. We have also embedded platform-specific hardware constraints, anti-patterns our team learned the hard way, and SDK-specific conventions into either skills or Claude Code rules.\nWe will cover in depth how we designed our Claude Code skills files and how we optimized other Claude Code use cases in future blog posts.\nThe difference between rule files and skills is similar to the difference between a style guide and a “trained teammate.” A rule says, “Use pytest, do not use unittest.” A skill says, “Write a test suite appropriate for this type of module, execute it, automatically fix failures up to five times if needed, and summarize the results in a report.”\nSkills like /implement-tested, /pr-review, and /open-pr have transformed tasks that used to take hours into workflows that now take minutes under human supervision.\nWhile we had previously experimented with using LLMs for code generation, most approaches were limited to a “single prompt–single response” pattern. Claude Code’s skills allowed us to structure the entire execution flow — from planning → implementation → testing → fixing → reporting — into reusable units. In particular, they significantly reduced the cognitive load of repetitive but mentally taxing tasks, which made a tangible difference in our day-to-day productivity.\nThe Code Review Bottleneck Here\u0026rsquo;s the part nobody warned us about. AI and agentic tools made developers dramatically faster at writing code. Pull requests that used to take a day were being opened in an hour. But code review — the part that requires a human to carefully read, understand, and verify changes \u0026ndash; didn\u0026rsquo;t get any faster. If anything, the bottleneck got worse. The faster code was produced, the longer the review queue grew, and the overall speed of shipping features didn\u0026rsquo;t increase as much as we expected.\nWe started using AI for code review in GitHub PRs, and it helped catch surface-level issues. But the deeper question remained: how do you maintain quality at higher velocity? Our answer has been evolving. We\u0026rsquo;re moving toward a model where AI agents produce a plan.md file before writing any code — a structured document outlining the approach, the files to be changed, and the reasoning behind each decision. Human reviewers then review the plan rather than scrutinizing every line of the implementation. The key constraint is that the changed code must pass a comprehensive test suite. This shifts the burden of correctness from line-by-line review to test coverage, and it stresses the importance of writing excellent tests more than ever.\nThere\u0026rsquo;s a critical caveat to this approach: whenever someone creates new tests or modifies existing test code, that work must be human-reviewed. Tests are the contract that makes plan-based review trustworthy. If the tests are weak or wrong, the entire workflow model breaks down. So while we\u0026rsquo;re trying to gradually adopt this leaner review strategy for implementation code, test code remains firmly in the domain of careful human judgment.\nWhat Does the Future Hold? We\u0026rsquo;re still early. Not only our team, our company, but the entire software world trying to take advantage of these AI tools are still at the early stages to experience what these black boxes are truly capable of, at least that is my belief. The tools are improving monthly. You know what, no, they are improving daily. Whenever it seems like the AI workflow is stabilizing, there is always a shift. As developers, it is important to always keep an eye on these changes, always be ready to adapt. If you are not ready to adapt to these changes, you will be felt behind. You might be the best programmer in the world, but there will be someone who is a lot less talented than you in programming but knows just enough to fly past you just by knowing exactly how to take advantage of these AI tools. Our team is very open minded to AI tools and understands that this is the present and the future, and because of this collective belief, the speed of our team’s development is accelerating. We see a future where AI agents handle increasingly complex tasks and we are just there to make sure the thing that is getting done fast, is getting done right. People always talk about ‘this model’s context length’. That is important, yes. But as developers who are using these tools to do the dirty work for us, we need to be the ‘context masters’. The ones who fully get the big picture, down to the tiniest details of what needs to be done. The human role will continue shifting from writing and reviewing code toward defining intent, validating architecture, curating test suites, and making smart, fast, important judgement calls.\nThe teams that will thrive are the ones investing now in the infrastructure that makes this possible: well-structured skill definitions, comprehensive test coverage, clear architectural documentation, and a culture that treats AI tooling not as a shortcut but as a force multiplier that demands new kinds of discipline.\nWe\u0026rsquo;re not replacing developers. We\u0026rsquo;re redefining what it means to be one.\nUpcoming\u0026hellip; This post was shared as an introduction outlining the overall direction and key questions we are exploring. In future posts, I plan to cover Skills in more depth, Kernel Agent(s), and other AI tool use cases, detailing not only how to use them but also how we designed and optimized them internally.\nAs the teams at HyperAccel continues to find new ways AI melts into our workflow, we will share our journey as we navigate this new paradigm of how we develop code and amazing software.\nHyperAccel is Hiring! The reason why we are writing operating this tech blog is largely too attract top talent!\nIf you are interest in the technologies we work with, and want to join this revolution, please apply at HyperAccel Career!\nHyperAccel is full of brilliants engineers. We will be waiting.\n","date":"2026.02.19","permalink":"https://hyperaccel.ai/techblog/en/posts/how-we-use-ai/","series":["AI Agent Series"],"style":"--c:#8EE8E0;--on:#02121c","summary":"Hello, my name is Taeseo Um, currently on the ML team at HyperAccel.\nThere is a running joke among developers that our job has always been about googling the issue, clicking the first link that pops up, then copying from Stack Overflow. In some ways, the arrival of AI coding tools didn\u0026rsquo;t change what we do — it just made us dramatically better at it. Over the past few months, our team has ridden almost every wave of AI-assisted development, from the earliest interactions to the autonomous agents we rely on today. Here\u0026rsquo;s what that journey looked like, what we learned, and where we think it\u0026rsquo;s all heading.\n","title":"[AI Agent - Part 01] From Copy-Paste to Autonomous Agents: A Subtle Introduction"},{"authors":["Namyoon Kim"],"category":"Infra \u0026 DevOps","content":"Hello. I\u0026rsquo;m Namyoon Kim, working on the ML team at HyperAccel.\nIn Part 1, we covered the motivation and overall design direction for building a Kubernetes-based development environment. This article continues from there, focusing on how we redesigned our CI/CD infrastructure.\nOnce the development environment lives on Kubernetes, running CI/CD pipelines on the same environment is the architecturally consistent choice. This article structurally analyzes why traditional self-hosted runners were unsustainable long-term, then documents the key technical decisions made during ARC (Actions Runner Controller) adoption — the selection criteria between DinD mode and Kubernetes mode, Rook-Ceph based Ephemeral PVC strategy, secret lifecycle management through Vault, and pipeline observability through a custom-built GitHub Actions Exporter.\nThe Structural Limitations of Self-hosted Runners Previously, we operated by installing self-hosted runners as Docker containers directly on servers. Runner labels were set to hostnames (e.g., ha-xxx), and GPU test workflows used the --all-gpu flag to utilize all GPUs on a given server.\nThe reasons we didn\u0026rsquo;t use GitHub\u0026rsquo;s Hosted Runners (ubuntu-latest, etc.) were as follows:\nHardware dependency: FPGA synthesis and GPU-based testing can only run on nodes with these devices physically installed. Network isolation: Internal infrastructure such as Harbor registry and Vault server are unreachable from external runners. Image transfer costs: The round-trip of building, pushing, and pulling multi-GB Docker images through external networks is inefficient in both bandwidth and time. While self-hosted runners resolved these constraints, fundamental problems with the structure itself emerged over time.\nFirst, tight coupling between servers and workflows. Since runner labels were bound to hostnames, replacing or renaming a server required modifying every workflow referencing it. Additionally, the --all-gpu flag approach meant that when two jobs were scheduled on the same server simultaneously, GPU resource contention couldn\u0026rsquo;t be controlled, causing one job to fail unpredictably.\nSecond, execution environment contamination. Runners execute directly on the host OS, so packages installed or system configurations changed during Build A can affect Build B\u0026rsquo;s behavior. This means build reproducibility cannot be guaranteed — undermining the fundamental purpose of a CI pipeline.\nThird, inability to scale elastically. The number of concurrent workflows varies significantly by time of day, but the number of server-installed runners is static. Under-provisioning increases queue wait times; over-provisioning wastes idle resources.\nAll three problems arise because runners exist outside Kubernetes\u0026rsquo; scheduling and lifecycle management framework. In an environment where a Kubernetes cluster is already established, converting runners to Pods and placing them under cluster control was the logical decision.\nARC (Actions Runner Controller) Architecture ARC is a GitHub-officially-supported Kubernetes Operator that manages the complete lifecycle of self-hosted runners on Kubernetes. Understanding ARC\u0026rsquo;s architecture requires examining the AutoScalingRunnerSet CRD (Custom Resource Definition) — the core resource that controls runner creation, scaling, and deletion.\nOperational Flow A Listener Pod maintains an HTTPS Long Poll connection with the GitHub Actions Service to watch for new jobs. Since this is polling-based rather than webhook-based, no inbound network configuration is required.\nWhen a job is detected, the Listener patches the EphemeralRunnerSet replica count via the Kubernetes API to request a scale-up.\nThe EphemeralRunner Controller obtains a JIT (Just-in-Time) configuration token and creates a Runner Pod, which registers itself with the GitHub Actions Service.\nUpon job completion, the EphemeralRunner Controller verifies with the GitHub API and deletes the pod. This ephemeral execution model fundamentally prevents residual artifacts from previous builds from affecting subsequent ones.\nPurpose-specific AutoScalingRunnerSet Design HyperAccel\u0026rsquo;s CI workloads span a wide resource spectrum — from general builds to FPGA synthesis to GPU testing. We deploy 7 AutoScalingRunnerSets organized by purpose:\n$ kubectl get autoscalingrunnerset -n arc-systems NAME MIN MAX CURRENT RUNNING runner-base 1 10 1 1 # DinD Mode runner-cpu 1 10 3 3 # Kubernetes Mode runner-cpu-largememory 1 10 1 1 # Kubernetes Mode runner-fpga 1 3 1 1 # Kubernetes Mode runner-gpu 1 3 1 1 # Kubernetes Mode runner-highcpu 1 10 1 1 # Kubernetes Mode runner-hybrid 1 4 1 1 # Kubernetes Mode Each Scale Set is scheduled only on appropriate nodes via nodeAffinity or tolerations:\nrunner-gpu: Placed on GPU nodes with label nvidia.com/gpu.present=true runner-fpga: Placed only on specific FPGA-equipped nodes runner-cpu / runner-cpu-largememory: Placed on nodes with label ci=large-memory Workflows select the appropriate runner via the runs-on key:\njobs: gpu-test: runs-on: runner-gpu # GPU Runner fpga-synth: runs-on: runner-fpga # FPGA Runner build: runs-on: runner-cpu # General CPU Runner A critical observation: only runner-base uses DinD mode, while the remaining 6 all use Kubernetes mode. The difference between these two modes is one of the most important architectural decisions in ARC operations.\nDinD Mode vs Kubernetes Mode: Architectural Differences When a Runner Pod needs to execute containers within a workflow (the container: key or container actions), ARC provides two modes: DinD (Docker-in-Docker) mode and Kubernetes mode. These modes differ fundamentally in their container execution mechanism, volume management, and security model.\nDinD Mode (runner-base) Runs a Docker daemon as a sidecar container inside the Runner Pod. At HyperAccel, runner-base operates in this mode.\n# runner-base core configuration (DinD Mode) spec: containers: - name: runner env: - name: DOCKER_HOST value: unix:///var/run/docker.sock - name: RUNNER_WAIT_FOR_DOCKER_IN_SECONDS value: \u0026#34;120\u0026#34; volumeMounts: - mountPath: /var/run name: dind-sock # Docker socket sharing - mountPath: /home/runner/_work name: work initContainers: - name: init-dind-externals # Copy runner externals command: [\u0026#34;cp\u0026#34;] args: [\u0026#34;-r\u0026#34;, \u0026#34;/home/runner/externals/.\u0026#34;, \u0026#34;/home/runner/tmpDir/\u0026#34;] - name: dind # Docker daemon (Sidecar) image: docker:dind securityContext: privileged: true # ⚠️ Privileged required restartPolicy: Always args: [\u0026#34;dockerd\u0026#34;, \u0026#34;--host=unix:///var/run/docker.sock\u0026#34;] volumeMounts: - mountPath: /var/run name: dind-sock - mountPath: /home/runner/externals name: dind-externals volumes: - name: dind-sock emptyDir: {} # Volatile volume - name: dind-externals emptyDir: {} - name: work emptyDir: {} The structural characteristics of DinD mode are as follows.\nFirst, the docker:dind image runs as an initContainer with restartPolicy: Always, operating in the Sidecar pattern. The runner container accesses this Docker daemon\u0026rsquo;s Unix socket via the DOCKER_HOST environment variable.\nSecond, since the Docker daemon handles container layer management, image pulls, and network creation entirely within the Pod, it cannot leverage the node\u0026rsquo;s containerd image cache. Even images already cached on the node must be pulled again by the DinD daemon.\nThird, privileged: true is mandatory for Docker daemon execution. This grants the Pod access to nearly all host kernel capabilities, requiring careful consideration in environments with strict security policies.\nFourth, all volumes are configured as emptyDir — data disappears when the Pod is deleted. Docker build cache is also not preserved.\nKubernetes Mode (runner-cpu, runner-gpu, runner-fpga, etc.) The Runner Pod calls the Kubernetes API to create workflow container steps as separate Pods. ARC\u0026rsquo;s Container Hook (runner-container-hooks) mediates this process.\n# runner-cpu core configuration (Kubernetes Mode) spec: containers: - name: runner env: - name: ACTIONS_RUNNER_CONTAINER_HOOKS value: /home/runner/k8s/index.js - name: ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE value: /home/runner/k8s/worker-podspec.yaml - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER value: \u0026#34;true\u0026#34; volumeMounts: - mountPath: /home/runner/_work name: work - mountPath: /home/runner/k8s/worker-podspec.yaml name: hook-template subPath: worker-podspec.yaml volumes: - name: work ephemeral: volumeClaimTemplate: # Ephemeral PVC spec: accessModes: [\u0026#34;ReadWriteOnce\u0026#34;] storageClassName: rook-ceph-block resources: requests: storage: 15Gi - name: hook-template configMap: name: arc-hook-cpu # worker-podspec ConfigMap The core mechanisms of Kubernetes mode are as follows.\nACTIONS_RUNNER_CONTAINER_HOOKS specifies the Container Hook entry point (index.js). When a workflow uses the container: key, instead of executing directly, the Hook calls the Kubernetes API to create a separate Workflow Pod.\nACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE specifies the Workflow Pod\u0026rsquo;s spec template (worker-podspec.yaml). This template is managed as a ConfigMap, with different ConfigMaps referenced per runner type (arc-hook-cpu, arc-hook-gpu, arc-hook-fpga, etc.).\nACTIONS_RUNNER_REQUIRE_JOB_CONTAINER=true enforces that all Jobs must execute within a container via the container: key, preventing direct execution on the Runner Pod itself.\nWhen a job runs, a separate Workflow Pod is created alongside the Runner Pod, separating the runner from the actual workflow execution environment.\nStructural Comparison of Both Modes Aspect DinD Mode (runner-base) Kubernetes Mode (runner-cpu, etc.) Container execution Docker daemon inside Pod Separate Pod via K8s API Privileged mode Required (Docker daemon) Optional (per worker-podspec) Image cache Isolated in DinD (no node cache) Shares node\u0026rsquo;s containerd cache Work volume emptyDir (volatile) Ephemeral PVC (rook-ceph-block, 15Gi) Step isolation Same Docker network Independent Pod per step possible Docker CLI Fully compatible Incompatible (via Container Hook) Config complexity Low High (worker-podspec, RBAC, etc.) Best for Docker build/push workflows In-container build/test runner-base is maintained in DinD mode because workflows requiring direct Docker CLI usage (image builds, registry pushes, etc.) exist. In Kubernetes mode, there is no Docker daemon, so docker build cannot be executed directly.\nVolume Strategy: Ephemeral PVCs and Cache Layers Runner volume design directly impacts build performance and stability. The volume strategies differ qualitatively between DinD mode and Kubernetes mode.\nDinD Mode Volumes: emptyDir # runner-base volumes (DinD Mode) volumes: - name: dind-sock # Docker socket sharing (Runner ↔ DinD daemon) emptyDir: {} - name: dind-externals # Runner externals copy emptyDir: {} - name: work # Workspace (checkout, build artifacts) emptyDir: {} - name: harbor-ca # Harbor CA certificate configMap: name: harbor-ca All work volumes are emptyDir, so data is lost when the Pod is deleted. Docker build cache is also not preserved — previously cached layers cannot be reused. This is disadvantageous for large image builds.\nKubernetes Mode Volumes: Ephemeral PVC + Cache Layers # runner-cpu volumes (Kubernetes Mode) volumes: - name: work ephemeral: volumeClaimTemplate: spec: accessModes: [\u0026#34;ReadWriteOnce\u0026#34;] storageClassName: rook-ceph-block # Ceph block storage resources: requests: storage: 15Gi # 15Gi per runner - name: hook-template configMap: name: arc-hook-cpu # worker-podspec template In Kubernetes mode, the work directory (/home/runner/_work) uses Ephemeral PVCs. Unlike emptyDir, these are backed by Rook-Ceph block storage, providing stable I/O independent of node local disk conditions with a dedicated 15Gi volume per runner. Ephemeral PVCs are bound to the Runner Pod\u0026rsquo;s lifecycle — they are automatically deleted when the Pod is deleted, ensuring consistent operation without storage leaks.\nCache Volume Strategy in worker-podspec Workflow Pods created by Container Hook have additional cache volumes mounted. These are defined in each runner type\u0026rsquo;s ConfigMap (arc-hook-cpu, arc-hook-gpu, etc.) within worker-podspec.yaml.\n# arc-hook-cpu ConfigMap (worker-podspec.yaml) — excerpt spec: containers: - name: \u0026#34;$job\u0026#34; env: - name: HF_HOME value: /mnt/cache/huggingface - name: CCACHE_DIR value: /mnt/cache/ccache - name: UV_CACHE_DIR value: /mnt/cache/uv resources: limits: cpu: \u0026#34;32\u0026#34; memory: \u0026#34;128Gi\u0026#34; volumeMounts: - name: huggingface-cache mountPath: /mnt/cache/huggingface - name: ccache-cache mountPath: /mnt/cache/ccache - name: uv-cache mountPath: /mnt/cache/uv volumes: - name: huggingface-cache persistentVolumeClaim: claimName: huggingface-runner-pvc # 1Ti NFS (shared across all runners) - name: ccache-cache hostPath: path: /tmp/ccache # Node-local cache - name: uv-cache hostPath: path: /tmp/uv # Node-local cache The volume strategy organized by layer:\nLayer Volume Type Capacity Lifecycle Purpose Work directory Ephemeral PVC (rook-ceph-block) 15Gi / Runner Pod-bound Checkout, build artifacts Model cache PVC (huggingface-runner-pvc) Persistent (shared) HuggingFace models, datasets Build cache hostPath Node disk Node-bound ccache, uv package cache Notably, huggingface-runner-pvc is a PVC shared across all runners, preventing multi-GB LLM models from being downloaded on every build. ccache and uv caches use hostPath to share between runners scheduled on the same node.\nHowever, hostPath caches can cause lock contention when multiple runners execute simultaneously on the same node. We actually encountered this issue with uv cache, which we resolved by separating cache paths per runner using the UV_CACHE_DIR environment variable.\nVault: Secret Lifecycle Management Secret management (registry credentials, API keys, signing keys, etc.) in CI/CD pipelines is critical from both security and operational perspectives.\nGitHub Secrets Limitations GitHub\u0026rsquo;s Repository Secrets and Organization Secrets suffice for small-scale environments. However, as repositories scale to dozens, the following problems emerge:\nDuplicate management: Identical secrets registered across multiple repositories → full manual updates required on rotation No audit trail: No visibility into when a secret was last updated or by whom No privilege separation: Repository Admin permissions required for secret access The most critical trigger for adopting Vault was devcontainer image tag management. Previously, we managed devcontainer image tags via GitHub Repository Variables (e.g., vars.DEVCONTAINER_IMAGE_AIDA_CU126). Every time a new image was built, a developer had to manually update the variable, and frequent omissions meant workflows ran with outdated images. After adopting Vault, CI pipelines automatically record the latest image tag in Vault upon build completion, and subsequent workflows and the Developer Portal reference the latest value via needs.fetch-secrets.outputs. The human step of managing/updating image version information is completely eliminated.\nVault with Dual Auth Strategy We deployed HashiCorp Vault on the Kubernetes cluster and applied different Auth Methods depending on the access subject.\nARC Runners use the JWT Auth Method. GitHub Actions\u0026rsquo; OIDC provider issues an ID Token, which is submitted to Vault for authentication. By declaring permissions: id-token: write in the workflow, GitHub automatically issues an OIDC token, and hashicorp/vault-action forwards it to Vault.\n# Vault secret injection in workflows (JWT Auth) permissions: id-token: write # Enable GitHub OIDC token issuance contents: read steps: - name: Import Secrets from Vault uses: hashicorp/vault-action@v3 with: url: ${{ secrets.VAULT_ACTION_URL }} method: jwt # GitHub OIDC JWT role: ${{ secrets.VAULT_ACTION_ROLE }} exportToken: true secrets: | secret/data/harbor username | HARBOR_USERNAME ; secret/data/harbor password | HARBOR_PASSWORD In contrast, the Developer Portal runs as a Pod directly inside the Kubernetes cluster, so it authenticates via the Kubernetes Auth Method using its ServiceAccount token. By applying separate Auth Methods suited to each access subject\u0026rsquo;s characteristics, we optimize the security model for each path.\nThe key benefits of this architecture are twofold:\nSingle management point: Secret rotation requires only a single update in Vault, immediately reflected across all pipelines.\nAudit logging: Every secret access records who, when, and what — satisfying security audit requirements.\nfetch-secrets: Secret Centralization via Reusable Workflows Even with Vault in place, if each workflow individually implements Vault authentication and secret retrieval logic, duplicate code proliferates. To prevent this, we designed a fetch-secrets reusable workflow (GitHub Actions workflow_call). Vault authentication (JWT) and secret retrieval logic are encapsulated in this single workflow, and callers simply reference the outputs.\n# docker-build-push.yml — Caller side jobs: fetch-secrets: uses: ./.github/workflows/fetch-secrets.yml # Delegate Vault auth/retrieval secrets: inherit build: needs: [fetch-secrets] steps: - name: Log in to Harbor uses: docker/login-action@v3 with: registry: ${{ needs.fetch-secrets.outputs.harbor_registry_url }} username: ${{ needs.fetch-secrets.outputs.harbor_username }} password: ${{ needs.fetch-secrets.outputs.harbor_password }} Build workflows don\u0026rsquo;t even need to know Vault exists — they simply retrieve values from needs.fetch-secrets.outputs. Even if Vault\u0026rsquo;s secret paths change, updating fetch-secrets.yml alone propagates to all pipelines.\nGitHub Actions Exporter: Pipeline Observability To elevate the operational maturity of CI/CD infrastructure, observability is essential. GitHub Actions\u0026rsquo; web UI is adequate for checking individual workflow statuses, but falls short for real-time identification of cross-repository trends, bottleneck points, and anomalies.\nTo address this, we developed a GitHub Actions Exporter in Go.\nDevelopment Rationale Three metrics not provided by existing open-source exporters were needed:\nQueue wait time by runner label: To identify bottleneck points in a mixed ARC + Hosted Runner environment Consecutive failure tracking: For real-time detection and early response to serial workflow failures Branch-level analysis: Separate analysis needed since main and feature branch build patterns differ Architecture and Key Metrics GitHub REST API ──→ Collector ──→ /metrics endpoint │ Prometheus Scrape │ Grafana Dashboard + AlertManager Metric Description Usage workflow_runs_total Total workflow run count Usage trend analysis workflow_failure_rate Failure rate (0.0 ~ 1.0) Quality monitoring workflow_duration_seconds Execution time histogram Performance regression detection workflow_queue_time_seconds Queue wait time Runner shortage detection workflow_consecutive_failures Consecutive failure count Immediate alert trigger workflow_runs_in_progress Currently running workflows Real-time status workflow_runs_by_branch_total Runs per branch Branch strategy analysis Deployed as a Kubernetes Deployment and scraped via Prometheus ServiceMonitor. ARC\u0026rsquo;s Listener Pods also have Prometheus metric annotations configured, enabling collection of runner scaling metrics.\nDecision-making Through Observability Issues detected and addressed through the dashboard:\nQueue time spikes: Concurrent job surges hitting maxRunners limits → adjusted Scale Set maximum runner counts 50%+ failure rate on a specific workflow: Identified Docker layer cache expiration pattern → revised caching strategy Gradual build time increase: Growing test cases extended builds from 30 to 45 minutes → applied test parallelization # Identify workflows with failure rate \u0026gt; 20% github_actions_workflow_failure_rate \u0026gt; 0.2 # Queue wait exceeding 60s — runner shortage signal github_actions_workflow_queue_time_seconds_avg \u0026gt; 60 # 3+ consecutive failures — immediate response needed github_actions_workflow_consecutive_failures \u0026gt;= 3 Infrastructure Operations \u0026amp; Maintenance Automation We automated operational tasks to maintain the stability and currency of the infrastructure.\nAutomated Vault Backups Secrets and policies stored in Vault are critical data for recovery in case of cluster failure. A Raft snapshot is created every Sunday morning and backed up to two storage locations (AWS S3 and on-premise MinIO). Old backups are automatically pruned according to the retention policy (default 6 days).\nARC \u0026amp; Runner Version Tracking GitHub Actions Runner and Container Hooks are continuously updated. A version tracking workflow runs every Monday to check for the latest releases and compare them with the current versions. If a new version is detected, it automatically opens a PR to notify administrators, ensuring the runner environment stays up-to-date. Full Architecture Bringing all components together yields the following architecture:\nComponent Role AutoScalingRunnerSet (x7) Purpose-specific runner scaling policies (1 DinD + 6 K8s mode) Listener Pod Job detection via GitHub Long Poll, Prometheus metrics exposure EphemeralRunner JIT token registration → job execution → auto-deletion Container Hook + worker-podspec Workflow Pod creation with volume/resource injection in K8s mode Vault Centralized secret management with JWT (ARC) / K8s Auth (Portal) Rook-Ceph Ephemeral PVC backend (runner work directories) GitHub Actions Exporter Workflow metric collection, Prometheus exposure Quantitative Impact Metric Before After Build queue wait time 3+ minutes average Under 15 seconds Secret management Manual per-repository Single Vault management Incident detection latency Post-inquiry confirmation Real-time Grafana dashboard Hardware resource management Manual server allocation nodeAffinity auto-scheduling Build reproducibility Non-deterministic Guaranteed via Ephemeral Pods For hardware with limited availability like GPUs and FPGAs, maxRunners (3 each) caps concurrent job counts to match available hardware, preventing resource contention.\nClosing This article covered the full journey from the structural limitations of self-hosted runners to a complete CI/CD infrastructure redesign with ARC. In particular — the architectural differences between DinD mode and Kubernetes mode, the Rook-Ceph based Ephemeral PVC and multi-layer cache strategy, secret lifecycle management through Vault, and pipeline observability through a custom-built Exporter — we aimed to share not just the tools adopted, but the technical rationale behind each design decision.\nThank you for reading!\nP.S.: HyperAccel is Hiring! Vault manages secrets, ARC schedules workloads, Rook abstracts storage, and Prometheus observes everything. Each plays a different role, but when combined within a single cluster, they form a complete system. HyperAccel works the same way — experts across HW, SW, and AI come together, moving toward one goal. If you\u0026rsquo;d like to be part of this combination, visit HyperAccel Career.\nReference Actions Runner Controller (ARC) Documentation Runner Container Hooks (GitHub) HashiCorp Vault — JWT/OIDC Auth Method GitHub Actions — OIDC Token for Vault Rook-Ceph Documentation Building a Dev Environment with Kubernetes Part 1 ","date":"2026.02.12","permalink":"https://hyperaccel.ai/techblog/en/posts/arc-setup-guide/","series":["Building an In-House Dev Environment on Kubernetes"],"style":"--c:#0052CC;--on:#ffffff","summary":"The complete journey of redesigning HyperAccel\u0026rsquo;s CI/CD infrastructure based on Actions Runner Controller (ARC), including Vault JWT/Kubernetes Auth dual integration, and achieving pipeline observability through a custom-built Go-based Prometheus Exporter.","title":"[K8s Dev Environment - Part 02] CI/CD Infrastructure Modernization with ARC"},{"authors":["Younghoon Jun"],"category":"Infra \u0026 DevOps","content":"Hello! I\u0026rsquo;m Younghoon Jun, a DevOps Engineer on the ML team at HyperAccel.\nFor those of you reading this, what kind of environment do you develop in? I imagine you\u0026rsquo;re working in various environments—local setups, SSH server access, cloud services, and more.\nHyperAccel develops on an environment built on top of a Kubernetes cluster. We spin up Pods based on devcontainers pre-configured with the packages needed for development, then connect to the container to do our work. To make things easier for our internal developers, we\u0026rsquo;ve created and provided a Devcontainer Portal.\nHowever, we didn\u0026rsquo;t always develop in a Kubernetes environment from the start. In this Building an Internal Development Environment with Kubernetes series, we want to share our journey of building a Kubernetes-based development environment to address developer pain points and provide an efficient development process.\nIn this first post of the series, we\u0026rsquo;ll cover the limitations of our previous development environment before Kubernetes and the path that led us to adopt it.\nBefore Container-Based Development HyperAccel is a startup that began small, founded by members of KAIST\u0026rsquo;s CAST Lab.\nGiven the nature of an early-stage startup, development moved at an incredibly fast pace, making it difficult to establish a systematic development environment. Since this was before I joined HyperAccel, I learned about the development environment at that time through conversations with Hyunjun Park (Author, LinkedIn), one of our founding members on the ML team.\nBack then, we were a really small team of about 10 people, and we couldn\u0026rsquo;t pay much attention to the development environment while trying to meet our goals within tight deadlines. Everyone created their own accounts on the company servers and logged in to use them. If someone was using too many resources, you\u0026rsquo;d just walk over to their desk and ask when they\u0026rsquo;d be done. (laughs)\nLet\u0026rsquo;s take a closer look at the challenges that arise when there\u0026rsquo;s no shared development environment.\nServer Management Challenges If you have 10 servers and 10 developers, you need a total of 100 accounts for everyone to access all servers. Of course, you might think account creation isn\u0026rsquo;t that difficult if you use automation scripts. However, developing in this environment makes it extremely hard for developers to track whether their work is in sync across different servers. (Even with GitHub, it\u0026rsquo;s very inconvenient.)\nSecurity and server stability are also concerns. For example, you need additional policies for granting sudo privileges. Furthermore, if a server gets damaged due to a mistake during development (think of drastic examples like sudo rm -rf /\u0026hellip;), there\u0026rsquo;s a cost to recovery.\nPackage Version Consistency Challenges When multiple people are working on code together, integrating everyone\u0026rsquo;s work is essential. If everyone develops in their own isolated environment, version conflicts can arise during integration. You might run into Torch version conflicts, Clang version mismatches, and so on. This is where everyone starts claiming \u0026ldquo;It Works on my Machine.\u0026rdquo;\nResource Usage Challenges HyperAccel\u0026rsquo;s first-generation chip was built on FPGA. Since FPGA servers are connected in a ring topology, to use them in complete isolation within a server, you had to either use just one server or all of them.\nFor this reason, without additional configuration, it was difficult for multiple developers to use them simultaneously. (Currently, this works smoothly on our Kubernetes environment. We\u0026rsquo;ll cover this in detail in a future post on Kubernetes Device Plugins.)\nAdditionally, for GPUs, checking availability required running commands like nvidia-smi to see running processes or manually checking a dashboard—quite inconvenient.\nIntroducing Devcontainers As the company grew and the number of developers increased, we needed a more systematic development environment. To address this, Minho Park (Author, LinkedIn), the ML team lead, built HyperAccel-Devcontainer, a containerized development environment (which we\u0026rsquo;ll refer to as devcontainer in this post for convenience).\nCurrently, the devcontainer is provided in two versions. This is to accommodate the different configuration needs of our developers. Each version is built on top of a base image with commonly required packages installed, with version-specific packages and environments configured on top. This allowed us to resolve issues like clang and torch version conflicts that had been problematic before.\nProviding a container-based development environment has the advantage that all developers work in the same environment. No more \u0026ldquo;It Works on my Machine.\u0026rdquo; Also, since development happens in an isolated environment, we can largely avoid situations where a server gets damaged due to individual mistakes.\nHowever, this container-based development environment still had clear limitations.\nIndividual Server Accounts Still Required We still needed to provide individual server access accounts. From a developer\u0026rsquo;s perspective, development depended on having a server account, and from an administrator\u0026rsquo;s perspective, account management became another task whenever team membership changed.\nAdditionally, since individuals had server access, the possibility of server damage from mistakes still existed.\nLack of Development Environment Flexibility First, developers still needed to manage their own directories on each server, so they remained tied to the server environment. If a specific server went down, developers using that server might face work disruptions if they didn\u0026rsquo;t have backups and couldn\u0026rsquo;t track their work. Since we also use servers at external IDCs, if an administrator had to travel to fix a server, work could be delayed even longer.\nAdditionally, developers need time and effort to learn how to use the container environment. From basic container knowledge to execution procedures and precautions, this can be challenging for developers who aren\u0026rsquo;t familiar with DevOps.\nUnresolved Resource Usage Challenges FPGA and GPU usage still had constraints. The issues mentioned earlier could still occur in the container environment.\nTo overcome these limitations and provide a more comfortable development environment, we decided to adopt Kubernetes.\nAdopting Kubernetes-Based Devcontainer Development Environment If you\u0026rsquo;ve read this far, you might be wondering:\n\u0026ldquo;What exactly is Kubernetes, and why should we use it?\u0026rdquo;\nBefore diving into the Kubernetes-based development environment, let me introduce what Kubernetes is and the benefits of building a development environment on top of it.\nContainer Orchestration Kubernetes is a Container Orchestration Tool. It\u0026rsquo;s an open-source platform that makes it easy to deploy and scale containers quickly while automating their management. Beyond being a simple container platform, it aims to be a microservices and cloud platform, serving as a vessel that can easily hold and manage everything built with containers. To put it in one sentence:\nA tool that codifies and automates complex infrastructure operations, allowing anyone to consistently deploy, scale, and operate services\nLet me explain the concept of Container Orchestration a bit more. In a container-based environment, services are delivered to users in the form of containers. If the number of containers to manage is small, one person can handle issues. But as the organization scales, one person handling all issues becomes impossible. Large-scale environments require operational techniques like:\nMonitoring systems that continuously check whether all services are running normally Scheduling, load balancing, and scaling to prevent work from piling up on specific clusters or containers Container Orchestration is a system that provides features to make managing and operating numerous containers a bit easier and more automated.\nAbout Kubernetes So far, we\u0026rsquo;ve looked at what Kubernetes and Container Orchestration are. Next, let\u0026rsquo;s briefly explore Kubernetes components. The diagram below shows an overview of the essential components that make up a Kubernetes cluster.\nA Kubernetes cluster consists of a Control Plane and one or more Worker Nodes.\nControl Plane Components These manage the overall state of the Kubernetes cluster. They analyze the resource requirements of requested workloads to schedule them on optimal nodes, and enforce the cluster\u0026rsquo;s overall availability and security policies to ensure system stability.\nThe components introduced below serve as the central logical layer that defines and manages the overall state (Desired State) of the cluster.\nkube-apiserver\nThe core server component that exposes the Kubernetes HTTP API etcd\nA consistent and highly available key-value store for all API server data kube-scheduler\nFinds pods not assigned to nodes and assigns them to appropriate nodes kube-controller-manager\nRuns controllers to implement Kubernetes API behavior Worker Node Components A Worker Node is a physical (or virtual) server that actually performs the tasks (pod execution) assigned by the Control Plane. Simply put, it is the server where pods actually run.\nThe components introduced below run on every node, maintaining running pods and providing the Kubernetes runtime environment.\nkubelet\nThe node agent that ensures pods and their containers are running kube-proxy (Optional)\nMaintains network rules on nodes to implement services container-runtime\nSoftware responsible for running containers We\u0026rsquo;ve now looked at Kubernetes and its role at the cluster level. So why did we choose Kubernetes for our internal development environment? Next, I\u0026rsquo;ll explain our reasoning based on the advantages of managing a development environment as a Kubernetes cluster.\nAdvantages of a Kubernetes-Based Development Environment What advantages do you get from providing a development environment based on Kubernetes? Let me explain, focusing on how we overcame the limitations of the container-based development environment mentioned earlier.\nFully Isolated Development Environment With the Kubernetes cluster in place, developers no longer need to access servers directly. Developers simply run their devcontainer in the Kubernetes environment and connect to the container to develop! Infrastructure concerns like node selection and resource management are all handled at the cluster level.\nAdditionally, you can specify node usage policies at the cluster level to control user access permissions and resource usage. By separating nodes within the cluster by purpose, many pain points for users are eliminated, and abnormal resource usage can be controlled to prevent node downtime.\nFlexible Development Environment Operations Development environments are no longer tied to specific nodes! Even if some nodes go down, pods can be scheduled to other nodes within the cluster. Developer home directories are provided via NFS provisioner on NAS, so you can confidently continue development on a pod running on a different node.\nAdditionally, since we provide a portal interface to users (which I\u0026rsquo;ll explain in detail below), once users are granted cluster access, they can start using it with no initial learning curve.\nEasy Resource Management Custom resources like FPGAs and GPUs can be used exclusively at the container level. Kubernetes treats any resources beyond CPU and Memory as custom resources. To use these resources in Kubernetes, you need Device Plugins.\nOur cluster currently has Device Plugins configured, and by allocating these resources to containers, you can use them in isolation without interference from other containers.\nBuilding the Kubernetes Cluster and Introducing the Development Environment In the end, we built a Kubernetes cluster and introduced our internal development environment based on it!!\nLet me briefly introduce some of the components currently applied to our Kubernetes cluster that aim to enhance development environment convenience.\nHigh Availability\nIf you maintain only one Control Plane in Kubernetes, you have a SPOF (Single Point of Failure). If the cluster goes down, all developer work stops—a disaster. To prevent this, we designated 3 nodes as Control Plane and applied KeepAlived and HAProxy so that even if a specific node goes down, the cluster continues to operate normally.\nStorage Management\nFor management convenience and minimizing network latency, we introduced Rook-Ceph and Harbor to operate an internal OCI Registry. We introduced Nexus to provide an internal PyPi server environment. CI/CD Enhancement\nWhen running CI/CD in the GitHub environment, we previously ran GitHub Action Runners directly on compute nodes. With the Kubernetes cluster in place, we introduced ARC (Actions Runner Controller) for more stable runner execution.\nWe introduced Vault for centralized secret management and to improve developer access to secrets.\nPortal Provision\nWe created the Devcontainer Portal and provide it to our developers. Let me explain the features of the Devcontainer Portal in more detail.\nContainer Creation and Deletion\nWith just a single button click, you can easily create, restart, and delete containers. Error Log Checking\nThe Portal allows access to the terminal inside the container and viewing Logs. This makes it possible to check errors as soon as they occur. Kubernetes Cluster Node Monitoring\nIt provides information on node hardware resource usage and available remaining resources, so you can check the resources needed for development. Since HyperAccel is a company that builds HW chips, devices are essential for development, and monitoring them properly is important. Before providing the Portal, we used Makefile and .env files to run pods. This approach caused initial entry barriers and inconvenience for users. To solve this, we applied a GUI approach through the Portal to provide the pod execution environment, and the required specifications are provided using go-template and ConfigMap. Developers can now easily launch development environment infrastructure, and monitoring makes log checking convenient, creating an environment where developers can focus more on development.\nDuring my master\u0026rsquo;s program, I researched scheduling for efficient GPU usage in distributed training environments. I modified the Kubernetes scheduler directly for my research, but only used limited Kubernetes features. Through this project, I gained valuable experience going through the entire process from cluster setup to operation.\nBuilding a Kubernetes cluster for the development environment and figuring out how to increase productivity and convenience for our internal developers was challenging, but it was truly rewarding and enjoyable.\nIn Summary\u0026hellip; In this post, we covered the first topic in the Building an Internal Development Environment with Kubernetes series: the limitations of our previous development environment and the motivation and process for adopting Kubernetes.\nWe provided a container-based environment for isolated development under common conditions and adopted Kubernetes to manage it. From the developer\u0026rsquo;s perspective, infrastructure concerns are minimized, allowing them to focus comfortably on development. From the administrator\u0026rsquo;s perspective, the environment can be stably managed and issues addressed even as scale increases, greatly improving work efficiency.\nIn upcoming posts in this series, we\u0026rsquo;ll introduce what frameworks we\u0026rsquo;ve added to the Kubernetes cluster to enhance the development environment and boost productivity!\nFurthermore, our ML team is developing a software stack to support using HyperAccel\u0026rsquo;s upcoming ASIC chip on Kubernetes. Kubernetes plays an important role as the foundational infrastructure for LLM (Large Language Model) Serving, one of the most prominent technologies today. We\u0026rsquo;ll share more about this topic in future posts!\nThank you for your continued interest in our upcoming articles, and thank you for reading to the end!\nP.S.: HyperAccel is Hiring! HyperAccel is a company working on HW, SW, and AI to launch an LLM acceleration ASIC chip. We have talented people across all areas, and in this environment, we\u0026rsquo;re learning broadly (and deeply!) beyond just one domain, sharing knowledge, and growing together at a rapid pace!\nThe DevOps part within our ML team provides development environments and management to boost internal developer productivity, and develops software stacks to effectively support LPU chip utilization at the cloud level.\nIf you\u0026rsquo;re interested in the technologies we work with at HyperAccel, please apply at HyperAccel Career!\nReference Kubernetes Docs What is Kubernetes? Why Kubernetes? HyperAccel Builds High-Efficiency LLM Inference Service with LPU Based on Amazon EC2 F2 Instance Hyperdex Toolchain Software Stack Actions Runner Controller ","date":"2026.02.09","permalink":"https://hyperaccel.ai/techblog/en/posts/development-environment-with-k8s-ch1/","series":["Building an In-House Dev Environment on Kubernetes"],"style":"--c:#0052CC;--on:#ffffff","summary":"Hello! I\u0026rsquo;m Younghoon Jun, a DevOps Engineer on the ML team at HyperAccel.\nFor those of you reading this, what kind of environment do you develop in? I imagine you\u0026rsquo;re working in various environments—local setups, SSH server access, cloud services, and more.\nHyperAccel develops on an environment built on top of a Kubernetes cluster. We spin up Pods based on devcontainers pre-configured with the packages needed for development, then connect to the container to do our work. To make things easier for our internal developers, we\u0026rsquo;ve created and provided a Devcontainer Portal.\n","title":"[K8s Dev Environment - Part 01] Why Kubernetes?"},{"authors":["Jaewon Lim"],"category":"AI Hardware","content":" \u0026ldquo;If you know the enemy and know yourself, you need not fear the result of a hundred battles.\u0026rdquo;\nThis series aims to deeply understand competitors\u0026rsquo; hardware for AI accelerator design.\nIn this third article, we cover the LPU (Language Processing Unit) from Groq, an American startup acquired by NVIDIA for approximately $20 billion in late last year.\nA Bold Challenge from an Engineer Who Left Google To understand the background of Groq\u0026rsquo;s founding, we need to learn about its founder, Jonathan Ross. He was a key member who initiated the TPU (Tensor Processing Unit) project at Google. While working on the TPU development team, he sensed the growth potential of the future AI inference market and left Google to found Groq. He then introduced the world to LPU (Language Processing Unit), a dedicated inference chip.\nThe market they targeted is very similar to what we at HyperAccel are pursuing.\nGPU / TPU: Support both training and inference, but are inherently optimized for large-scale computation-intensive training LPU: Boldly abandons training and goes all-in on optimizing inference for Large Language Models (LLMs) Those unfamiliar with the concepts of training and inference may have this question:\n??? : Training? Inference? Aren\u0026rsquo;t they all the same AI operations? Can\u0026rsquo;t GPUs do everything? Why do we need inference-specific chips?\nFor those readers, let\u0026rsquo;s briefly touch on the difference between training and inference in AI operations.\nTraining and Inference: The Limits of General-Purpose Chips and the Opportunity for Dedicated Chips AI workloads can be broadly divided into training, which updates weights, and inference, which uses fixed weights.\nLet\u0026rsquo;s assume we\u0026rsquo;re developing an AI model that identifies what an image is from a given image. To do this, we first need to train the model with a large amount of image data. During the training process, the model looks at the image and outputs an answer based on the current weights (Forward path), then compares the error with the actual answer according to a defined learning mechanism and updates the current weights (Backward path). Repeating this process continuously updates the model\u0026rsquo;s weights, and after sufficient time, the model\u0026rsquo;s error decreases and it becomes able to identify images well. Once training has progressed to some extent, we need to actually use this model, right? Unlike training, the inference process does not require weight updates. This is because updating the weights of an already-trained model is unnecessary computation in the actual usage stage. Therefore, the inference process only involves the Forward path.\nWhat we can learn from this is that inference does not require as much computation as training.\nHowever, this alone is insufficient to justify dedicated chips. If inference simply requires less computation, we could just run many inference operations on hardware with massive computational capacity like GPUs or TPUs, even if it\u0026rsquo;s over-spec. In reality, AI companies use top-tier GPU lineups (B200, etc.) for training and relatively lower-tier GPUs (H200, etc.) for inference. To understand the need for inference chips in the current generative AI market led by LLMs, we need to examine the structural limitations of currently used LLMs.\nStructural Limitations of LLMs: The Bottleneck is Memory, Not Computing The fundamental characteristic of the Transformer architecture, which forms the basis of current LLM models like GPT, Gemini, and Grok, is that it does not generate results as complete sentences at once, but rather generates words (hereafter tokens) one by one sequentially. This is expressed as having auto-regressive properties. This is because the Transformer model does not generate results solely from input values, but recalculates the relationships between input tokens and all previously generated output tokens each time an output token is generated to create the next output token. This requires an enormous amount of computation. However, the problem is not just the amount of computation. If we recalculate all values every time we generate a token, computational inefficiency increases, so reusable intermediate values (KV cache) are stored in memory and read from memory when computing the next token for recalculation. The problem is that this requires reloading past data from memory every time a token is generated, so the amount of memory movement increases as more tokens are generated.\nIn computer architecture, we use a measurement technique called the Roofline Model to diagnose hardware performance limits. This checks the maximum possible performance of an operation through the computational characteristics of software used on hardware and hardware specifications.\nThe Roofline Model graph consists of two axes. The x-axis is computational intensity (Operational Intensity). This is an indicator of how much computation is performed with data fetched once, and it increases as data reusability increases. The y-axis is performance, typically expressed as floating-point operations per second (FLOPS/s).\nEach line on the graph represents the hardware\u0026rsquo;s performance limit. The sloped line represents the performance limit due to memory bandwidth, and when computational intensity is low (left region), the maximum performance is determined along this line. In this region, no matter how excellent the hardware\u0026rsquo;s computational capability is, performance is limited by the speed of fetching data from memory, so only performance proportional to memory bandwidth can be obtained (Memory bound). The slope of this line equals performance (FLOPS/s) / computational intensity (FLOPS/BYTE), which is the same as the hardware\u0026rsquo;s memory bandwidth (BYTE/s).\nOn the other hand, the horizontal line represents the hardware\u0026rsquo;s maximum computational performance limit. When computational intensity is sufficiently high (right region) to reach this horizontal line, the performance of that operation equals the maximum computational performance the hardware can deliver (Compute bound). The point where the two lines meet is called the Ridge Point, where memory and computational performance are balanced. Beyond this point, even if computational intensity increases, maximum performance can no longer increase due to hardware limitations.\nCompute Bound : The required computation is too much compared to hardware specs, causing a bottleneck due to the physical limits of the chip\u0026rsquo;s computation speed Memory Bound : Computation units are sufficient, but the speed of fetching data from memory is slow relative to that, causing a bottleneck The training process is close to compute bound because it requires a lot of computation and can be parallelized. However, in the LLM inference process, especially after calculating the first output token for the initial input tokens (Prefill Stage), the process of extracting output tokens one by one (Decoding Stage) requires reading past data from memory every time to output individual tokens. Moreover, the size of this data increases as the output token length increases. This lowers computational intensity, so inference operations are close to memory bound.\nThis is exactly why GPUs and other datacenter-oriented chips must use expensive HBM (High Bandwidth Memory). No matter how fast the chip\u0026rsquo;s computation speed is, it\u0026rsquo;s useless if it can\u0026rsquo;t overcome the Memory Wall. As mentioned earlier, since the slope of the graph in the memory-bound region equals memory bandwidth, using HBM to increase bandwidth allows us to extract higher performance at the same computational intensity, as shown in the graph below.\nInference chips target these bottlenecks in LLMs. If performance limits are inevitably reached due to memory bound no matter how high the computational performance, we compete with chips that have architectures suited to LLM structures or maximize memory bandwidth. General-purpose chips like GPUs are also showing such movements, but it\u0026rsquo;s difficult to make bold hardware innovations and structural improvements. This is because while GPUs are currently widely used for LLM operations, GPUs are not only used for LLM but also for training and computation of various AI model structures, and have become general-purpose chips used not only for AI operations but also for various tasks like image rendering. This is why dedicated chips that remove unnecessary parts and target specific workloads can show strengths.\nNext, let\u0026rsquo;s explore the bold attempts Groq has shown in the hardware market through this dedicated chip.\nGroq\u0026rsquo;s Bold Gamble: Shatter the Conventional Wisdom of Memory Hierarchy The biggest feature of Groq\u0026rsquo;s LPU is its memory structure. As I mentioned earlier, memory bandwidth is very important. Looking at the A100 GPU\u0026rsquo;s memory structure, it looks like the diagram below.\nAn individual GPU\u0026rsquo;s memory hierarchy consists of on-chip SRAM inside the GPU chip, off-chip memory (HBM) attached to the GPU chip, and system memory (DRAM) allocated to individual nodes when running on GPU servers. In the memory hierarchy, as you go down, capacity increases, bandwidth decreases, and price per capacity becomes cheaper. During LLM operations, weights and intermediate computation values are mostly stored in HBM and then read again, and data read from HBM is stored in cache form in GPU SRAM near the compute units for reuse, increasing memory efficiency. This is a format that has been adopted in CPUs and other traditional computer architectures as well.\nWhile large companies including Nvidia and many inference chip startups have been thinking about ways to use HBM and memory more efficiently, Groq questioned this memory hierarchy and asked a game-changing question.\n??? : Why not just put all data inside the chip (On-chip)?\nThis is not simply about increasing SRAM capacity. It means not using off-chip memory. Groq boldly removed HBM, which corresponds to the middle of the hierarchy above, from their chip. Instead, they use SRAM, which is used as cache memory inside the chip, as main memory.\nHBM bandwidth : 8 TB/s (B200 standard) LPU SRAM bandwidth : 80 TB/s or more (at least 10-20x difference) Using this overwhelming bandwidth, Groq proved superior performance to GPUs in image classification models like ResNet at ISCA (International Symposium on Computer Architecture) in 2020.\nProblem: Cost and Capacity However, using only on-chip memory doesn\u0026rsquo;t solve all problems. If using only SRAM solved everything, many hardware companies would have already adopted that method. The problem is cost. SRAM and DRAM may look like the same memory, but their structures are very different.\nThis is an image that those who majored in electrical engineering may have seen at least once. Simply put in numbers, DRAM requires 1 transistor and 1 capacitor per individual cell that stores one bit, while SRAM requires 6 transistors per cell. This is a 3x difference in just component count, and considering wiring complexity, the area difference can be tens of times. Therefore, increasing SRAM capacity increases chip size, and there are limits to increasing capacity due to the reticle limit mentioned in a previous article. Moreover, increasing SRAM capacity means the proportion of SRAM in the chip increases. This means less space for logic needed for computation. No matter how important memory is, you can\u0026rsquo;t compute with just memory. Due to these limitations, the reality is that individual chips can only hold SRAM at most about 1/10 of HBM capacity (hundreds of MB).\nLatest LLM models basically have parameters in the billions. Even with a simple calculation assuming no compression, storing weights alone requires tens of GB of memory, so using only SRAM makes it impossible to even load one model on one chip.\nSolution: Split and Store, Run Like a Conveyor Belt Groq solved this problem with Scale-out. If one chip can\u0026rsquo;t hold a model, they split (Shard) the model\u0026rsquo;s weights and store them across hundreds of chips. This is where AI parallelization techniques come in.\nTensor Parallelism (TP) : Decomposes one large matrix operation into multiple smaller matrix operations. Weights needed for partial matrix operations are distributed and stored in the main memory of multiple chips, and each chip performs partial matrix operations Pipeline Parallelism (PP) : Divide the model\u0026rsquo;s layers into groups for processing. In computation order, each chip receives computation results from the previous chip and performs computation for the next layer Groq likens the system they built by applying these parallelization techniques to a giant conveyor belt. When input data enters the first chip (the start of the belt), each chip only performs its assigned computation and passes data to the next chip. The final token pops out from the last chip. This can be visualized in the animation below.\nOn the other hand, if a GPU\u0026rsquo;s single chip were to perform the same task,\nMemory load from HBM -\u0026gt; computation -\u0026gt; memory load from HBM again -\u0026gt; computation\nThis process would need to be repeated. This is because while HBM stores data needed for computation, there are still limits to the amount of data that can be loaded near the compute units (SRAM). Groq\u0026rsquo;s claim is that bottlenecks occur in this process of repeated memory loads. This can be visualized in the animation below.\nFor Groq\u0026rsquo;s system to operate successfully, Chip-to-Chip (C2C) communication speed is critical. If the middle of the conveyor belt breaks, the entire factory stops. Groq tied 8 or more chips together like one giant node through their proprietary C2C interconnect technology (RealScale) and Dragonfly Topology. They made multiple chips share each other\u0026rsquo;s SRAM, appearing like one giant single chip, converging communication overhead to almost zero.\nGPUs are not incapable of using this method. The TP and PP explained earlier are parallelization methods widely used in distributed training with GPUs. However, when GPUs use these techniques (especially TP), work is needed to synchronize data between devices. GPUs support inter-GPU communication through the runtime library NCCL (Nvidia collective communication library). In the process of using TP mentioned earlier, all-reduce is needed, where matrix multiplication results computed on each GPU are shared with other GPUs to obtain summation.\nHowever, in Groq\u0026rsquo;s LPU, runtime libraries like NCCL used by Nvidia are unnecessary. This is because in LPU, inter-device communication is predictable at compile time, not through runtime libraries like NCCL. Let\u0026rsquo;s learn how this is possible through another architectural philosophy of Groq.\nSoftware-defined Hardware: The Compiler Decides Everything Another characteristic of Groq\u0026rsquo;s LPU is that software (especially the compiler) takes precedence over hardware design. In fact, looking at the official documents containing LPU\u0026rsquo;s design philosophy, they say they didn\u0026rsquo;t even touch chip design until the compiler architecture was designed. This is to make the compiler capable of controlling all hardware and make individual hardware operations predictable at the compiler level.\nDeterministic Execution CPUs and GPUs don\u0026rsquo;t know when or what data will come in, so they use complex schedulers (Branch Prediction, Cache Managing, Warp Scheduling, etc.) at the hardware level. This is flexible but creates unpredictable delays (Tail Latency).\nLet\u0026rsquo;s briefly revisit the warp scheduling example from GPU that appeared in Part 1.\n(The examples explained from now on are for simple explanation and may differ from actual hardware operation.)\nIn the warp scheduling introduced earlier, load/store commands that read and write data to memory occur asynchronously. This is fundamentally because we don\u0026rsquo;t know when memory load/store commands will complete. During this time, the processor has no choice but to wait until the asynchronous command completes. But what if we could know when this command completes? What if we could accurately predict the time we need to stall? The processor could calculate what it can do during that time and optimize the system by doing other work, minimizing idle processes.\nLPU\u0026rsquo;s static scheduling aims to apply this to deep learning and LLMs, especially inference operations. As mentioned earlier, inference operations are a process of calculating final results (output tokens in LLMs) through fixed weights. In this process, if the model is fixed, the computation order is generally fixed. A fixed computation order is one of the optimal conditions for applying static scheduling. This is because we can remove scheduling that might occur during computation. This allows us to utilize limited hardware resources for other units like compute units or memory instead of scheduling, increasing hardware efficiency.\nThe effects of this static scheduling shine even more when using multiple chips simultaneously.\nThe process of using all-reduce/all-gather through Nvidia\u0026rsquo;s NCCL library mentioned just before requires synchronization between devices. This is because all devices must finish their work before the next task can be performed, but we don\u0026rsquo;t know when each device will finish its work.\nBut what if the compiler could know when work completes on each device? It could perform not only the synchronization work needed during NCCL operations but also the work needed by NCCL at the compiler level.\nFor this reason, Groq minimized the scheduling function, which corresponds to the hardware\u0026rsquo;s \u0026lsquo;brain\u0026rsquo;, and delegated this function to the compiler. Let\u0026rsquo;s summarize the static scheduling of Groq\u0026rsquo;s LPU explained so far:\nThe compiler pre-calculates data flow inside chips and across all chips to be used at compile time in hardware time units (clock cycles). When and which memory address of which chip data will arrive at is 100% predictable (Deterministic). Hardware simply executes as the compiler instructs without complex deliberation. LPU secured predictability of execution time and was able to focus hardware resources that would have been used for scheduling on computation. Why Did NVIDIA Acquire Groq? Groq\u0026rsquo;s LPU was innovative and had clear strengths technically, but faced business challenges. From a customer\u0026rsquo;s perspective, even running just one LLM model on LPU requires hundreds of chips (rack-level), so initial adoption costs reach a tens to hundreds of millions of USD. Perhaps due to this problem, Groq initially sold chips (GroqChip) but later diversified their business to provide cloud API rental services through GroqCloud. They provide APIs that allow use of servers or racks they directly built with Groq chips.\nSo what is the background behind NVIDIA\u0026rsquo;s acquisition of Groq? In general economic articles, the prevailing opinion is that it\u0026rsquo;s for technology licensing to expand dominance in the inference market. However, as an engineer, I\u0026rsquo;ll add a bit of technical imagination and conclude this article by discussing several technical scenarios that might unfold after Groq\u0026rsquo;s acquisition.\nHypothesis 1: Building a New Heterogeneous Computing Platform One recent trend in AI inference operations is the separation of Prefill and Decoding operations (prefill-decode disaggregation). This is because operations with different characteristics coexist even within inference operations.\nPrefill(Context) phase : Input prompt is inserted at once, requiring much computation → Compute Bound Decoding(Generation) phase : After the first token is generated, generated tokens are input sequentially one by one → Memory Bound NVIDIA has previously announced a heterogeneous hardware platform that uses different compute units for prefill and decode. They announced Rubin CPX, specialized for the context-phase and based on the next-generation GPU architecture Rubin, along with the platform equipped with it. To summarize the idea of this platform, Prefill stage runs on Rubin CPX → Decoding stage runs on Rubin GPU. If we utilize Groq\u0026rsquo;s LPU architecture in this platform, we can imagine new platforms in the following 2 ways:\nInstall GPU + LPU hybrid hardware (Rubin LPU) instead of Rubin GPU on the current platform Rubin CPX → Prefill Rubin LPU → Decoding Add the hardware mentioned in 1 to the current platform Rubin CPX → Prefill Rubin LPU → Decoding Rubin GPU → Prefill + Decoding However, realistically, building such platforms is expected to be very difficult. This is because the hardware architecture differences between LPU and GPU are large, making integration within a single platform not easy. Therefore, I think the scenario to be introduced next is more likely than this scenario.\nHypothesis 2: Building a Hybrid AI Giga Factory NVIDIA is moving beyond being a simple hardware company to directly building datacenters. The so-called AI Giga Factory that NVIDIA is preparing is a platform where they directly build datacenters that hyperscalers (Meta, Google, Amazon) build and provide them to other customers. Think of it as an expanded version of the GroqCloud API that Groq provided.\nBuilding datacenters requires not just hardware knowledge but understanding of systems overall, customer experience, and various tangible/intangible assets, so there aren\u0026rsquo;t many companies (especially among hardware companies) with the technical capability to do this. Usually, hyperscalers have built their know-how and technical capabilities while building their own datacenters. Groq is a hardware company that actually built datacenters with their inference chips and provided real APIs. From NVIDIA\u0026rsquo;s perspective, acquiring Groq would allow them to expand the hardware lineup for their giga factory.\nIf we expand the heterogeneous computing mentioned earlier from server or node level to datacenter level:\nGPU-specialized tasks including training are computed in GPU clusters Tasks that can proceed faster on LPU are computed in LPU clusters Then what tasks can LPU proceed faster with? From the perspective of LLM inference operations mentioned earlier, we can think of separating Prefill/Decoding tasks between GPU and LPU clusters. This time, let me introduce a few workloads where LPU can show strengths from a task-level perspective.\n(From here on, my personal views are heavily reflected.)\nSpeculative Decoding\nOne recent trend in LLM serving is Speculative Decoding. As model sizes grow and computation time becomes longer, a small and fast model (Draft Model) that distills or is trained to behave similarly to the existing model (Target Model) quickly generates the latter part of a sentence in advance, then the Target Model verifies this in parallel. Groq\u0026rsquo;s LPU clusters can be used for small-sized Draft Model computation here. This is because LPU boasts overwhelming token generation speed in small-sized models. From an overall perspective, the roles of LPU/GPU clusters can be divided as follows:\nLPU : Generate candidate tokens at high speed GPU : Verify at once whether the tokens thrown by LPU are correct However, such systems fundamentally have limitations. This is due to communication overhead between GPU - LPU clusters. If LPU only throws the tokens it generated to GPU, overhead wouldn\u0026rsquo;t be large, but Speculative decoding\u0026rsquo;s Target Model needs data generated by the Draft Model (KV cache) for token verification. If communication occurs within one chip or node, overhead would be relatively small, but if large-capacity KV cache movement is needed in inter-cluster communication, the overhead becomes a big problem. This is because it can actually degrade the service\u0026rsquo;s performance itself. This is a problem that must be considered not only for speculative decoding but also when performing prefill-decode disaggregation at the cluster level.\nMoreover, recent speculative decoding is transforming into a method applied through internal optimization in a single model rather than separating models in the target-draft way explained above, changing to an environment where heterogeneous computing is difficult to use.\nAgentic AI\nAnother scenario is Agentic AI. AI is evolving beyond simple chatbots that only answer questions into agent forms that make their own plans, use tools, and go through multiple stages of reasoning. In each stage of making plans and using tools, large and small models exchange data they each generated to create final results. From the perspective that various models are used, we can think of a scenario using heterogeneous computing similar to Speculative Decoding mentioned just before.\nLPU : Handles light agent tasks (Worker) requiring fast response such as simple judgment, tool selection, repetitive routine work, routing, etc. GPU : Handles heavy tasks requiring massive computation such as complex logical reasoning, code generation, image rendering, etc. Agentic AI systems communicate based on text (JSON) rather than tensor forms like KV cache, so physical hardware separation is free. This is because concerns about communication overhead, the bottleneck mentioned in Speculative decoding, are low. Therefore, combining LPU + GPU clusters could form a datacenter platform optimized for Agentic AI serving.\nSummary In this article, we explored:\nThe background of LPU\u0026rsquo;s emergence and the difference between general-purpose and dedicated chips Groq\u0026rsquo;s hardware/software design philosophy Several technical scenarios expected after NVIDIA\u0026rsquo;s acquisition of Groq Personally, this acquisition seemed like NVIDIA\u0026rsquo;s move to maintain a monopolistic position in the hardware market with capital, which was both intimidating, while also feeling ambivalent because the term LPU, which our company also uses, seemed to become more known to the market and public. I also felt anticipation that if we prove our products in the market, great opportunities would open up for our company as well.\nThroughout this three-part series, we\u0026rsquo;ve explored various AI accelerators, from GPUs to TPUs and LPUs. While we\u0026rsquo;ve focused on accelerator architectures and computational structures so far, in the next article, we\u0026rsquo;ll expand the topic to storage, which plays an equally important role as accelerators outside of computation. In the next installment, we\u0026rsquo;ll examine NVIDIA\u0026rsquo;s BlueField, which garnered attention at this year\u0026rsquo;s CES, and explore storage and DPUs (Data Processing Units) that play crucial roles in AI datacenters.\nP.S. : HyperAccel is Hiring We at HyperAccel are on the verge of launching our first LPU product. Not only for this product but also for other products to come, HyperAccel needs more excellent engineers.\nIf you\u0026rsquo;re interested after seeing the technologies we work with, please apply at HyperAccel Career!\nHyperAccel has many truly excellent and smart engineers. We\u0026rsquo;re waiting for your application.\nReference Jonathan Ross Interview\nThink Fast: A Tensor Streaming Processor (TSP) for Accelerating Deep Learning Workloads, ISCA 2020\nA Software-defined Tensor Streaming Multiprocessor for Large-scale Machine Learning, ISCA 2022\nWhat is Language Processing Unit? - groq blog\ngroq whitepapers\nSpeculative Decoding - Nvidia blog\nRubin CPX platform - Nvidia blog\nNCCL Collective Operations\n","date":"2026.02.03","permalink":"https://hyperaccel.ai/techblog/en/posts/lpu-deep-dive/","series":["Know Your Enemy, Know Yourself"],"style":"--c:#00C2A8;--on:#02121c","summary":"We explore the background of Groq and LPU, their hardware/software design philosophy, and analyze NVIDIA\u0026rsquo;s intentions behind acquiring Groq.","title":"[Know Your Enemy - Part 03] Groq's LPU (Acquired by NVIDIA for $20B)"},{"authors":["Seungbin Shin"],"category":"Engineering Culture","content":"SDD: Adapting to the Changing Meaning of Work Table of Contents Introduction SDD? How is it different from BDD, TDD, RDD? The Era of Vibe Coding Ended Before It Began, Engineering Returns A Taste of LLM: What are Context Window and Attention? Context Window: A Movie Seen Through a Window Attention Mechanism: Cocktail Party Effect Why SDD: Structural Limitations and Collapse of Chat-Based Coding Emergence and Standardization of SDD Workflow Next-Gen Tools for SDD Conclusion: Where Should We Go? P.S. Reference Introduction Hello, I am Seungbin Shin, working as an RTL Designer at HyperAccel.\nI want to do my job well, but I don\u0026rsquo;t want to invest more time than necessary to achieve that goal.\nIn other words, I am always thinking about \u0026ldquo;how to increase the cost-effectiveness of work.\u0026rdquo; If I can improve work quality and do more work in the same amount of time, it is a gain for both me and the company.\n2025 was a year where the form of the software development industry changed significantly, enough for many reading this to feel it, along with the development and advancement of generative AI, which was more active than in \u0026lsquo;23 and \u0026lsquo;24.\nEven I, a hardware guy with zero knowledge of software development, could create the app I wanted after a few conversations, and I could whip up a homepage too.\nHowever, when I tried to add features I wanted to the app one by one, or modify the design to look better, after a few conversations, bugs I had solved before would reappear, or it would do random things like placing buttons wherever it wanted.\nEventually, I could reach \u0026ldquo;something in the form I wanted\u0026rdquo;, but I faced many situations where it was difficult to improve further from a level that was roughly 2%, no, 10% lacking, and the progress saturated.\nWhen this happened, since I am not an expert in that language, I couldn\u0026rsquo;t modify it based on that codebase, and even if I were an expert, it would have taken a lot of time to understand the spaghetti code tangled by multiple conversations.\nI could compromise and use it myself, but thinking about servicing or selling this to people, I thought it would be absolutely unmanageable, and eventually, I felt that this form of development had clear limits.\nOf course, it could be a problem because I am not an expert, but even for experts, this type of work process would likely consume more time in code review, refactoring, complying with rules, generating tests, etc., and ultimately has a much higher possibility of leading to a decrease rather than an increase in productivity.\nIt might look like I\u0026rsquo;m doing a lot of work, but I\u0026rsquo;m just producing a lot of useless code and spending a lot of time on maintenance, so the value produced per time could be lower.\nI think many of you reading this post have had similar experiences.\nLet\u0026rsquo;s learn about SDD, a methodology to avoid this vicious cycle and obtain the results we want by extracting high-quality tokens from LLMs as efficiently and quickly as possible.\nThe content of this post is based on what I have personally studied and experienced. If there are any errors, please let me know in the comments at any time.\nSDD? How is it different from BDD, TDD, RDD? In the past, and perhaps still today, many developers use various methodologies to improve process and quality while increasing collaboration efficiency when working with people. Let me introduce a few popular methodologies.\nBDD (Behavior-Driven Development) BDD (Behavior-Driven Development) is a methodology where you first write a scenario defining user behavior in natural language, and then convert it into test code for development. For example, instead of saying \u0026ldquo;Implement login function\u0026rdquo;, you write a requirements specification based on concrete user actions like \u0026ldquo;The user enters an ID and enters a password. Then, when the login button is clicked, it checks the password; if correct, it moves to the main page, and if incorrect, it shows an error message and does not move to the main page.\u0026rdquo;\nTDD (Test-Driven Development) TDD (Test-Driven Development) is a methodology where you write the minimum test first, write code that passes it, and constantly refactor while developing. Similarly, if you had to implement a login function, development proceeds in a direction of implementing increasingly complex functions starting from \u0026ldquo;a function that returns false when the ID is empty\u0026rdquo;, \u0026ldquo;a function that handles exceptions when the password is wrong\u0026rdquo;.\nRDD (README Driven Development) RDD (README Driven Development) is about writing the most perfect software manual before writing code, imagining how the user will install and use it while developing. It is especially important in open-source projects, and the point is that implementation takes place after writing a guide for other developers who will use this function.\nThe biggest difference between the 3 methodologies described above and SDD (Spec-Driven Development) lies in \u0026ldquo;who the promise is for\u0026rdquo;.\nBasically, existing methodologies were tools for \u0026ldquo;me\u0026rdquo; or \u0026ldquo;our team\u0026rdquo; to work well. When writing documents, corporate culture or internal characteristics might melt in, making it difficult for other teams to understand. However, since the purpose is to improve internal quality in the first place, it doesn\u0026rsquo;t matter much.\nNow we spend much more time creating results by talking to LLMs than wrestling with code with people. However, even within coding IDEs like Cursor or Antigravity, we change code generation models, start in different sessions, and sometimes scrape code to ask models on other web pages. Every time this happens, LLMs, which are probabilistic generative models, inevitably produce slightly different results. And the biggest problem is that LLMs are not interested in \u0026ldquo;corporate culture\u0026rdquo;. In other words, they lack \u0026ldquo;context understanding\u0026rdquo; of our development environment and style.\nSDD is a strict promise between \u0026ldquo;me\u0026rdquo; and \u0026ldquo;LLM (AI Agent)\u0026rdquo;. It operates as a rule that any model must follow, and plays a role in guaranteeing the connection and compatibility of generated codes. The era of treating LLMs like magic wands is passing quickly, and the era where methodologies to handle LLMs well are needed has arrived. Of course, SDD is not a panacea. But at least it gives direct help in productivity improvement in that you can continuously verify with one document and obtain results closest to the desired form.\nWe briefly looked at what SDD feels like, but does that mean TDD is a dead methodology and we should only use SDD? No. SDD is not a methodology that replaces TDD, but a methodology that complements parts that TDD cannot fill while using LLMs more efficiently.\nYou might still question why SDD is needed, hearing that it\u0026rsquo;s about generating code after reading a manual, and why this is needed in principle. From the next section, let\u0026rsquo;s look in more detail from the background of SDD\u0026rsquo;s emergence to theoretical parts.\nThe Era of Vibe Coding Ended Before It Began, Engineering Returns \u0026ldquo;Huh? This works?\u0026rdquo;\nI remember the thrill when I first asked ChatGPT 3 to write code during my undergraduate days.\nThe experience of it understanding perfectly and spitting out code even when I spoke roughly was truly new. In the industry, they call this Vibe Coding. Because we could create results with just a feeling (Vibe) through communication with AI.\nBut this period was short.\nAs I experienced before, when the project scale grew even a little, or after going through a few conversations, the AI started becoming stupid rapidly.\nIt forgets variable names, recreates bugs I fixed earlier, and eventually falls into the Doom Loop of modification that makes you think \u0026ldquo;I\u0026rsquo;ll just write it myself\u0026rdquo;.\nEven when I realized something was wrong, the code was often too messy and twisted that refactoring or modifying it was almost impossible.\nThe failure of Vibe Coding is not simply a performance issue of AI models.\nThis was an accident that inevitably occurs when the intrinsic limitations of LLMs relying on probabilistic generation meet the absence of engineering.\nNow is the time to look at AI from the perspective of \u0026ldquo;Engineering\u0026rdquo;, not \u0026ldquo;Magic\u0026rdquo;.\nA Taste of LLM: What are Context Window and Attention? Why does AI become stupid when the conversation gets long? To understand this, we need to understand Context Window and Attention Mechanism, which are the brain structures of LLMs.\nContext Window: A Movie Seen Through a Window LLM\u0026rsquo;s memory is not infinite. You must have had times when you thought \u0026ldquo;It forgot this?\u0026rdquo; while using chat models.\nHave you seen this toy? I remember playing with it when I was young; it\u0026rsquo;s a toy that uses optical illusion where you can see a horse running through a slit when spun around.\nLLM\u0026rsquo;s Context Window can be thought of as similar to the size of the slit in this toy. In other words, the amount of information it can see at once is limited.\nEven the memory to store that information is finite, so it cannot be increased infinitely. Eventually, information that goes out the window is unprocessed or forgotten.\nAttention Mechanism: Cocktail Party Effect Even if the window mentioned above became really wide and infinite, there is another problem. It becomes very difficult to compare and choose what is important.\nWhen we have ten friends, we can remember their faces, names, personalities, and maybe even birthdays and phone numbers.\nAnd all ten friends are precious. But what if there are ten thousand friends? A hundred thousand? It becomes difficult to even remember their names.\nCocktail Party Effect is a phenomenon where party attendees selectively focus on and understand the conversation with their interlocutor despite the surrounding noise. This selective perception or attention, where one ignores the surrounding environment and selectively accepts only meaningful information, refers to the psychological phenomenon that occurs. It is also called \u0026lsquo;self-referential effect\u0026rsquo;, \u0026lsquo;banquet hall effect\u0026rsquo;, and \u0026lsquo;feast house effect\u0026rsquo;.\nThe Attention Mechanism is quite similar to the Cocktail Party Effect. We\u0026rsquo;ve all had the experience of hearing a specific friend\u0026rsquo;s voice well in a noisy bar.\nLLMs also have a mathematical mechanism to find important content, that is, key requirements, in tens of thousands of lines of code and chat logs, and that is the Attention Mechanism. It is also the concept presented in the famous paper \u0026ldquo;Attention is all you need\u0026rdquo; that you might have heard of or read.\nWhen the conversation is short, AI accurately finds important sections. But if the conversation continues for hundreds of turns, the party hall becomes a mess, and the AI slows down, makes mistakes, and spits out different answers.\nWhy SDD: Structural Limitations and Collapse of Chat-Based Coding Once we know this technical background, it becomes clear why we suffered when writing code in chat windows.\nLost in the Middle Phenomenon: According to research, LLMs remember the beginning and end of text well, but amazingly forget the content in the middle. If data structures or utility functions defined early in the chat are buried in the middle of the conversation, the AI starts creating already existing functions anew (duplicate code) or creating random variables to use.\nAttention Dilution: As the conversation gets longer, the \u0026lsquo;concentration\u0026rsquo; that AI can pour into each piece of information decreases. Important requirements get diluted mixed between small talk or error logs. Simply typing \u0026ldquo;No, not that\u0026rdquo; made things harder for our LLM. Emotionally and memory-wise\u0026hellip;\nDoom Loop: When an error occurs and you give the log, usually AI gives correction code. But the same error occurs again. The AI\u0026rsquo;s context window becomes filled with \u0026lsquo;failed attempts\u0026rsquo; and \u0026rsquo;error logs\u0026rsquo;, becoming a state where it has learned wrong answer patterns rather than finding the right answer. It\u0026rsquo;s rare that it actually learns and gives that answer forever, but finding the question itself becomes difficult in a window filled with wrong answers that aren\u0026rsquo;t even correct.\nEventually, \u0026ldquo;using chat logs as specs\u0026rdquo; itself was a game structurally bound to fail. So we must switch to SDD, providing refined documents while minimizing chat. We must minimize chat logs, organize necessary contents into documents, and switch to a method where AI writes code based on documents.\nEmergence and Standardization of SDD Workflow The core of SDD is simple. It\u0026rsquo;s the philosophy that Code is volatile, Spec is permanent. Code is treated as an \u0026lsquo;Artifact\u0026rsquo; that can be extracted from the spec at any time, and the \u0026lsquo;Source of Truth\u0026rsquo; we must manage becomes the \u0026lsquo;Natural Language Specification\u0026rsquo;. Especially as the performance of LLM models improves geometrically, rather than worrying about \u0026ldquo;code quality\u0026rdquo;, we should value \u0026ldquo;the quality of the spec document\u0026rdquo; I write more. The literary literacy I lack the most is becoming necessary again\u0026hellip;\nThe general SDD workflow is as follows.\nIntent: Define \u0026ldquo;What do you want to make?\u0026rdquo;.\nSpec Writing: Write spec.md or PRD.md files conversing with AI. Fill this with features, data models, and constraints tightly. This file is continuously referenced, mutually modified, and continuously updated and preserved by LLM.\nGeneration: Command \u0026ldquo;Implement according to this spec\u0026rdquo;. AI sets up an implementation plan (Plan) looking at the spec and writes code. I felt Antigravity is especially good at this part.\nVerification: Is the result strange? Don\u0026rsquo;t fix the code. Fix the spec and generate again, or write a document for testing.\nThis process is like giving user source code to a compiler to extract an executable file. Here, \u0026lsquo;source code\u0026rsquo; is \u0026rsquo;natural language spec\u0026rsquo;, and \u0026lsquo;compiler\u0026rsquo; becomes \u0026lsquo;AI Agent\u0026rsquo;. Wait, are you curious about what a compiler is? Then try reading this post.\nNext-Gen Tools for SDD You probably have a thought about what SDD is, why it appeared, and that you should use it. I\u0026rsquo;ll introduce coding IDEs where you can simply try SDD. Fortunately, starting from 2025 and 2026, powerful tools supporting SDD are pouring out.\nCursor \u0026amp; .cursorrules: The most accessible tool. If you make a .cursorrules file at the top of the project, any model references this rule whenever it runs. This file works like a kind of prefix for LLMs. I usually set up rules related to RTL Design or C++ coding and used them; example_cursorrules.txt is a rule file I actually used. At the time, I didn\u0026rsquo;t know well so I found and pasted from https://cursorrules.org/category. Strictly speaking, rather than SDD, it can be used for the purpose of unifying the overall code generation style. It\u0026rsquo;s a Spec definition in a broad sense.\nGoogle Antigravity An \u0026lsquo;Agent-First\u0026rsquo; IDE released by Google. Here, AI doesn\u0026rsquo;t just code, but creates documents (Artifact) like Implementation Plan or Task List first and gets approval. It feels like systemizing the mentoring of a senior saying \u0026ldquo;Plan before coding\u0026rdquo;. If writing a spec document is actually difficult, it might be okay to start based on this implementation plan document.\nOh-My-OpenCode (OpenCode) A counterattack from the open-source camp. It works in the form of a plugin like oh-my-zsh, but the interesting thing is \u0026lsquo;Ultrawork\u0026rsquo; mode. Should I call it \u0026ldquo;Don\u0026rsquo;t go home until this works\u0026rdquo; mode. If an error occurs, the AI agent (Sisyphus) finds documents, debugs, and retries on its own. Especially since various models can be used, I think we can get a glimpse of an AI Agent closer to the form of \u0026ldquo;operating on its own\u0026rdquo;. It\u0026rsquo;s a bit different from how I work so I don\u0026rsquo;t use it well, but it\u0026rsquo;s receiving a lot of attention recently.\nThe reason all this is possible is because technologies like SGLang or Guidance force the results spat out by AI into JSON or specific formats. Thanks to this, hallucinations of AI are suppressed, and a structure where systems can talk to each other has been created. Since I covered SGLang in detail in a previous post, please refer to it if you are curious.\nConclusion: Where Should We Go? I think SDD is a point where the identity of the developer changes rather than simply a change in development methodology. Of course, I am not a SW developer generally called a developer, but I feel something is changing or possibilities are showing in HW development (RTL Design) as well.\nNow all of us who code must transform into small PMs directing smart AI workers.\nThe ability to organize ambiguous thoughts into clear writing will become much more important than the ability to type code fast.\nThe era where we have to do Literature to do Engineering is blooming.\nFurthermore, since the majority of LLMs are trained more on English input, it seems almost obvious that writing documents in English will bring better results.\nSome say that the foundation for the birth of a \u0026lsquo;1-Person Unicorn\u0026rsquo; company has been laid. A future where one person operates planning, design, development, and deployment by delegating to dozens of AI agents. Of course, it\u0026rsquo;s a future that can absolutely not be achieved with vibe coding alone, and we must ponder how to achieve the best efficiency and the goal I want without too much paperwork in spec writing.\nRecently, Jensen Huang, the CEO of Nvidia, said in many interviews that \u0026lsquo;use AI as much as possible\u0026rsquo;. This interview is in stark contrast to the many concerns about AI bubbles and job losses.\nPersonally, I also find it interesting that such an interview is interesting from the perspective of RTL developers who also code. However, I think it\u0026rsquo;s important to interpret it based on one\u0026rsquo;s own perspective, rather than just being optimistic.\nWhen the calculator first appeared, most people who used to do manual calculations lost their jobs. However, it\u0026rsquo;s not that the concept of \u0026ldquo;calculation\u0026rdquo; itself disappeared. In the end, those people were at least one step ahead of others in terms of using calculators.\nI also think that the current situation is not so different. The entry barrier for coding has become much lower. However, can a person who has never coded create a Google, Amazon, or YouTube-like mega-service? Even if we assume infinite time, in the end, there will only be a heap of spaghetti and failure. I can confidently say this. I have many more things I want to say about this, but in summary, I think the world where \u0026ldquo;people who know well do better\u0026rdquo; is already here, and we need to take advantage of the advantage we have now, which is one step ahead.\nPerhaps we are passing through a period. A new protocol where humans and AI create systems together, SDD is at that starting point. What features or services are you developing right now? The moment you open a markdown file and define the content you are imagining as a spec, a new engineering will begin.\nP.S. I am continuing to worry about \u0026ldquo;what kind of engineer should I grow into?\u0026rdquo;, and I got to know SDD in that process. Personally, I think SDD will become a mainstream development methodology in the near future, and the ability to write spec documents itself can become a competitiveness. Eventually, people will ask generative AI models to even write these spec documents someday, but even then, you\u0026rsquo;ll be able to get the \u0026ldquo;best spec document\u0026rdquo; only if you can write \u0026ldquo;specs for spec documents\u0026rdquo; well. Naturally, I should be able to explain well what I am doing, but the ability to accurately convey what I want to the LLM that does not know the Context of this development will become important.\nThe era of writing code from a pure empty main.cpp has passed. Now is the time to ponder where to find my competitiveness, and I think the start is developing the ability to write spec documents. What do you think?\nFor individual growth, writing spec documents well is important, but I think \u0026ldquo;who you work with\u0026rdquo; is also very important. In this regard, I pride myself that HyperAccel is the best company in this area. Whether it becomes an LLM or a person, knowledge in various fields is required to give work instructions. We are a company that handles all of HW, SW, and AI for the launch of LLM acceleration ASIC chips, gathering outstanding talents across all fields. If you want to grow together sharing knowledge and learning broadly, even deeply, not confined to one field in this environment, please apply to HyperAccel anytime!\nRecruiting Site: https://hyperaccel.career.greetinghr.com/ko/guide\nReference Cocktail party effect Zoetrope Why Your AI Coding Agent Keeps Making Bad Decisions(And How To Fix It) Spec-Driven Development: Building Better Software, Faster with Kiro Lost in the Middle: How Language Models Use Long Contexts Attention Dilution Reasons I Found Why AIs Struggle With Coding introducing-google-antigravity The Ultimate Guide to AI-Powered Development with Cursor: From Chaos to Clean Code Oh-My-OpenCode The Rise of the “Zero-Employee” Unicorn: The Comprehensive Guide to the Post-Labor Startup Extract, Edit, Apply – a design pattern for AI ","date":"2026.02.03","permalink":"https://hyperaccel.ai/techblog/en/posts/what-is-sdd/","series":[],"style":"--c:#8EE8E0;--on:#02121c","summary":"Spec-Driven Development (SDD) is a new development methodology for the AI era, moving beyond prompt engineering to spec engineering.","title":"Specs are All You Need: What is Spec-Driven Development (SDD)?"},{"authors":["Jaewoo Kim"],"category":"Compilers \u0026 Kernels","content":"Today I brought a somewhat different topic. Have you ever thought it would be great if a compiler could automatically transform loops for optimization or parallelization while using it? Compiler engineers have had the same concerns. One approach that emerged is the polyhedral compiler, which is used by LLVM\u0026rsquo;s Polly project and MLIR\u0026rsquo;s affine dialect. Today I\u0026rsquo;ll introduce this approach.\nBasic concepts Before going into detail, let\u0026rsquo;s go over some basic concepts. Most of these are easy to encounter with a little linear algebra.\nAffine function An affine function is a function that can be defined as a linear transformation plus a translation. When you apply an affine transformation, points that formed a line move to the same line, and the midpoint of a line segment remains the midpoint after the transformation. In other words, you can think of an affine transformation as allowing a constant term in a linear transformation.\n$f(\\vec{v})=M_f\\vec{v}+\\vec{f}_0$\nHere, $M$ is a matrix and $\\vec{f}_0$ is a constant vector.\nAffine hyperplane In an n-dimensional space, an affine subspace of dimension n-1 is called an affine hyperplane.\nA hyperplane is the set of vectors ($\\vec{v}$) that satisfy:\n$k = h\\vec{v}$ (k is a constant)\nSo if n is 3, the hyperplane is a 2D plane.\nPolyhedron A polyhedron is the intersection of half-spaces split by a finite number of affine hyperplanes. A bounded polyhedron is called a polytope (meaning it is a closed region).\nIn the figure below, there are four 1D hyperplanes (since it is a 2D space, the hyperplanes are 1D), and they define a polytope. That is, it is the set of $\\vec{x}$ defined by:\n${\\vec{x} \\in \\mathbb{R} | A\\vec{x} + \\vec{b} \\ge 0 }$\n$A \\in \\mathbb{R}^{m \\times n}, \\vec{b} \\in \\mathbb{R}^m$ (when the polyhedron is bounded by m hyperplanes in n dimensions)\nFarkas Lemma Let the domain $D$ be a polyhedron defined by half-spaces.\nFarkas\u0026rsquo;s lemma states that an affine form (affine function) inside domain D can be expressed as a linear combination of the half-spaces that define D.\nIn other words, when half-spaces are defined as:\n$a_k\\vec{x} + b_k ≥ 0, k = 1,p$\nit is equivalent that an affine function $\\psi(\\vec{x})$ is non-negative inside D and that $\\psi(\\vec{x})$ can be expressed as a linear combination of those non-negative half-spaces.\n$\\psi(\\vec{x}) = \\lambda_0 + \\sum_{k=1}^{p}\\lambda_k(a_k\\vec{x} + b_k)\\ where \\ \\lambda_0, \\lambda_1, \u0026hellip;, \\lambda_p \\ge 0$\nTo spell it out, suppose we define an affine function and it lies in domain D. If this function is non-negative, then it can be expressed as a linear combination of the half-spaces that define D, and the converse also holds.\nUsing this lemma, when transforming a schedule defined by affine functions, we can convert it into a linear combination of half-spaces and solve the problem more easily with ILP.\nInterpreting program components Now that we\u0026rsquo;ve introduced the basic concepts, let\u0026rsquo;s see how we can represent a program.\nLet\u0026rsquo;s take a simple example program.\nfor(int i = 0; i \u0026lt; N; ++i){ for(int j = 0; j \u0026lt; N; ++j){ x[i] = x[j]*2 + y[j]; // Statement 1, iteration vector : (i, j) } for(int k = 0; k \u0026lt; N; ++k){ y[i] = y[i]*2; // Statement 2, iteration vector : (i, k) } } This program consists of three loops and two statements. Each statement has an iteration vector, which is a vector representation of the loop induction variables (iterator variables. i, j and k in the above example.) that affect that statement.\nThat is, the iteration vector for Statement 1 x[i] = x[j]*2 + y[j] is (i, j), and the iteration vector for Statement 2 y[i] = y[i]*2 is (i, k).\nSchedule vector A schedule vector indicates when each statement is executed. The schedule vector contains information about the statement\u0026rsquo;s position and the loops that enclose it.\nThe schedule vector represents the execution time of a statement, and in polyhedral analysis it is used to check statement conditions or perform transformations.\nIn the example above, the schedule vector for Statement 1 is (i, 0, j), and for Statement 2 it is (i, 1, k).\nThe way to set a schedule vector is as follows:\nDescribe it from the outermost scope If there is a loop, include that loop\u0026rsquo;s induction variable If multiple statements (including loops) exist in a scope, insert a scalar value to separate them (this is called a scalar dimension) In the example above, there are two loops inside loop i, so we separated them with a scalar dimension. By 1,2,3, the maximum length of a schedule vector is 2m + 1 when there are m nested loops. \u0026lt;scalar dimension of the outermost scope\u0026gt; + \u0026lt;iteration vector of each loop\u0026gt;*m + \u0026lt;scalar dimension inside each loop\u0026gt; = 2m + 1 Generally, if a scalar dimension is not needed (i.e., there is only one statement in a scope), the scalar dimension is omitted. Let\u0026rsquo;s look at another example.\nSuppose we have the following code.\nfor (int i = 0; i \u0026lt; N; ++i) { for (int j = 0; j \u0026lt; N; ++j) { A[i][j] = A[i][j] + u[i] * v[j] + u2[i] * v2[j]; // S0 } B[i] = A[i][0]; // S1 } for (int k = 0; k \u0026lt; N; ++k) { for (int l = 0; l \u0026lt; N; ++l) { x[k] = x[k] + beta * A[l][k] * y[l]; // S2 } } Then each statement has the following schedule vector:\nS0 : $\\begin{pmatrix}0\u0026amp;i\u0026amp;0\u0026amp;j\\end{pmatrix}$ → the only statement in loop j (index 0) inside loop i (index 0)\nS1 : $\\begin{pmatrix}0\u0026amp;i\u0026amp;1\\end{pmatrix}$ → the first statement (index 1) inside loop i (index 0)\nS2 : $\\begin{pmatrix}1\u0026amp;k\u0026amp;l\\end{pmatrix}$ → the only statement in loop l inside loop k (index 1)\nPolyhedral analysis Now let\u0026rsquo;s perform polyhedral analysis. First, let\u0026rsquo;s analyze whether loops can be parallelized.\nParallelism analysis Consider the following code. Can it be parallelized?\n// Q. can loop i and loop j be parallelized? for(int i = 0; i \u0026lt; N; ++i){ for(int j = 0; j \u0026lt; N; ++j){ if(i \u0026gt; j) b[i][j] = b[j][i]; // S0 } } What conditions are required to enable parallelization? To parallelize loop i or loop j, there must be no relations between iterations. If each iteration runs independently, that\u0026rsquo;s fine, but if something written in one iteration must be read in another iteration (or vice versa), you cannot parallelize the loop. So what about the example above?\nThis code transposes matrix B. It moves the upper-triangular matrix to the lower-triangular matrix. Therefore, there is no relation between loop iterations (no dependency), so it can be parallelized.\nThen how can we tell this with polyhedral analysis? We can determine it through a few steps.\nCompute the iteration vector for each statement. Represent the domain based on the iteration vector. Check whether the domain is empty. The iteration vector of S0 is (i, j). And S0 has two operations: read and write.\nSo, viewing S0 from the read and write perspectives:\nRead_s0(i, j) = (j, i)\nWrite_s0(i, j) = (i, j)\nAnd the polyhedron (domain $D_{s0}$) of S0 is defined as:\ni ≥ 0 \u0026amp;\u0026amp; i \u0026lt; N j ≥ 0 \u0026amp;\u0026amp; j \u0026lt; N i \u0026gt; j For a memory dependency to exist, the following must hold:\n$$ \\exists(\\vec{s}, \\vec{t}) : \\begin{cases} \\vec{s} \\in D_{s0} \\newline \\vec{t}\\in D_{s0} \\newline W(\\vec{s}) = R(\\vec{t})\\end{cases} $$\nHere $\\vec{s}$ and $\\vec{t}$ are different iteration vectors, and each iteration interprets statement S0 in terms of read and write. That is, to parallelize each loop, when we use different iteration vectors, $R(\\vec{t})$ (READ_s0(i, j)) must not equal $W(\\vec{s})$ (WRITE_s0(i, j))).\nNow, let $\\vec{s} = (i, j)$ and $\\vec{t} = (i\u0026rsquo; ,j\u0026rsquo;)$ and substitute them directly.\nExpanding $W(\\vec{s}) = R(\\vec{t})$ gives $(i\u0026rsquo;, j\u0026rsquo;) = (j, i)$. Substituting this into the read condition yields $i\u0026rsquo; \u0026gt; j\u0026rsquo; \\implies j \u0026gt; i$, and the write condition is $i \u0026gt; j$.\nBut $i \u0026gt; j$ and $j \u0026gt; i$ clearly cannot be satisfied at the same time.\nTherefore, the dependence polyhedron is empty, so there is no dependency between iterations, and both loop i and loop j can be parallelized. (Since the domain is empty in the first place, there\u0026rsquo;s no need to build a schedule vector.)\nLet\u0026rsquo;s look at one more example.\nfor(int i = 0; i \u0026lt; N; ++i){ for(int j = 0; j \u0026lt; M; ++j){ a[i][j] = a[i][j-1]; // S0 } } In this case, how can the code be parallelized?\niteration domain\ni ≥ 0 \u0026amp;\u0026amp; i \u0026lt; N j ≥ 0 \u0026amp;\u0026amp; j \u0026lt; M iteration vector\n$$ \\exists(s, t) : \\begin{cases} \\vec{s} \\in D_{s0} \\newline \\vec{t}\\in D_{s0} \\newline W(\\vec{s}) = R(\\vec{t})\\end{cases} $$\nLet\u0026rsquo;s build a schedule vector with this. Parallel means there is no dependency between schedules, so for all loops to be parallelizable, we must satisfy the following. (If only some are 0, only the loops corresponding to those dimensions are parallelizable.)\n($\\phi(\\vec{s}), \\phi(\\vec{t})$ are the schedule vectors for each iteration.)\n$$ \\phi_{s0}(\\vec{t}) - \\phi_{s0}(\\vec{s}) =\\vec{0} $$\nIf we omit scalar dimensions,\n$$ \\begin{pmatrix}i \\newline j \\end{pmatrix} - \\begin{pmatrix} i\u0026rsquo; = i \\newline j\u0026rsquo; = j-1 \\end{pmatrix} = \\begin{pmatrix}0 \\newline 1 \\end{pmatrix} $$\nThat is, loop i gives 0 and loop j gives 1, so we can see a dependency exists only in loop j. Therefore, in the example above, only loop i can be parallelized.\nWe also created an example of analyzing parallelism by constructing schedule vectors in this way.\nPolyhedral transformation Now let\u0026rsquo;s learn about polyhedral transformation. Polyhedral transformation is the process of changing analyzed code into a more efficient form, and it is executed by libraries like llvm-polly for optimization.\nThe key is to transform the code in a direction that minimizes cost by defining a cost function while preserving correctness.\nFor the i-th dimension of the schedule vector of statement $S^k$, we can write part of the schedule vector as follows:\n$$ \\phi_{S_i^k}(\\vec{t})=\\begin{pmatrix}c_0^i\u0026amp;c_1^i\u0026amp;\\cdots\u0026amp;c_n^i\\end{pmatrix}\\begin{pmatrix}i_0\\newline i_1\\newline i_2\\newline \\cdots\\newline i_{n-1}\\newline 1\\end{pmatrix} $$\n$c_0^i \u0026hellip;c_n^i$ are the transformation parameters found by polyhedral optimization. $c_n^i$ is used to represent a scalar dimension.\nExpressing this for all dimensions:\n$$ \\tau_s \\vec{t}=\\begin{pmatrix}\\phi_{S_i^0}(\\vec{t}) \\newline\\phi_{S_i^1}(\\vec{t}) \\newline\\vdots \\newline\\phi_{S_i^{k-1}}(\\vec{t})\\end{pmatrix}=\\begin{pmatrix}c_1^0 \u0026amp; c_2^0 \u0026amp; \\cdots \u0026amp; c_n^0 \\newline c_1^1 \u0026amp; c_2^1 \u0026amp; \\cdots \u0026amp; c_n^1 \\newline\\vdots \u0026amp; \\vdots \u0026amp; \\ddots \u0026amp; \\vdots \\newline c_1^{k-1} \u0026amp; c_2^{k-1} \u0026amp; \\cdots \u0026amp; c_n^{k-1}\\end{pmatrix}\\vec{t}+\\begin{pmatrix}c_0^0 \\newline c_0^1 \\newline \\vdots \\newline c_0^{k-1}\\end{pmatrix} $$\nHere, $\\vec{t}$ is the iteration vector.\nOnce we determine the parameters in the transformation matrix, we get a new schedule vector. If we then build the program according to the new schedule vector while keeping the previously defined constraints, the program will look a bit different (the loop structure or statement indices may change), but it will still execute correctly after the transformation. In a new structure, it might become parallelizable or might not require temporary variables.\nTransformation parameters are typically found by minimizing a cost function via ILP, but we can also consider other methods such as genetic algorithms, Bayesian optimization, and RL.\nLet\u0026rsquo;s think through this in more detail with an example.\nfor(i = 0; i \u0026lt; N; ++i){ for(j = 0; j \u0026lt; N; ++j){ for(k = 0; k \u0026lt; N; ++k){ c[i][j] = c[i][j] + a[i][k] * b[k][j]; // S1 } } } for(i = 0; i \u0026lt; N; ++i){ for(j = 0; j \u0026lt; N; ++j){ for(k = 0; k \u0026lt; N; ++k){ d[i][j] = d[i][j] + e[i][k]*c[k][j]; // S2 } } } Now let\u0026rsquo;s look at a simple transformation example for this code.\nTransformation of Statement S1:\n$$ \\tau_{s_1}(\\vec{i}_{s_1}) = \\begin{pmatrix} 0 \u0026amp; 1 \u0026amp; 0 \\newline 1 \u0026amp; 0 \u0026amp; 0 \\newline 0 \u0026amp; 0 \u0026amp; 0 \\newline 0 \u0026amp; 0 \u0026amp; 1\\end{pmatrix}\\begin{pmatrix}i \\newline j \\newline k \\end{pmatrix} + \\begin{pmatrix} 0 \\newline 0 \\newline 0 \\newline 0\\end{pmatrix} $$\nTherefore, the schedule for each dimension is:\n$$ (\\phi_{S_1}^1, \\phi_{S_1}^2, \\phi_{S_1}^3, \\phi_{S_1}^4) = (j, i, 0, k) $$\nTransformation of Statement S2:\n$$ \\tau_{s_2}(\\vec{i}_{s_2}) = \\begin{pmatrix} 0 \u0026amp; 1 \u0026amp; 0 \\newline 0 \u0026amp; 0 \u0026amp; 1 \\newline 0 \u0026amp; 0 \u0026amp; 0 \\newline 1 \u0026amp; 0 \u0026amp; 0\\end{pmatrix}\\begin{pmatrix}i \\newline j \\newline k \\end{pmatrix} + \\begin{pmatrix} 0 \\newline 0 \\newline 1 \\newline 0\\end{pmatrix} $$\nTherefore, the schedule for each dimension is:\n$$ (\\phi_{S_2}^1, \\phi_{S_2}^2, \\phi_{S_2}^3, \\phi_{S_2}^4) = (j, k, 1, i) $$\nThis is the new schedule vector. If we name the loops of the vector in order as t0, t1, t3 and construct new loops, we get the following code:\nfor(t0 = 0; t0 \u0026lt; N; ++t0){ for(t1 = 0; t1 \u0026lt; N; ++t1){ for(t3 = 0; t3 \u0026lt; N; ++t3){ // i -\u0026gt; t1, j -\u0026gt; t0, k -\u0026gt; t3 c[t1][t0] = c[t1][t0] + a[t1][t3]*b[t3][t0]; // S1 } for(t3=0; t3 \u0026lt; N; ++t3){ // i -\u0026gt; t3, j -\u0026gt; t0, k -\u0026gt; t1 d[t3][t0] = d[t3][t0] + e[t3][t1]*c[t1][t0]; // S2 } } } This code behaves the same as the original, but the loop structure is slightly different (the outermost loop has been merged). The fact that it produces the same result is guaranteed because we find the transformation matrix parameters with the assumption that they satisfy the constraints (dependencies, etc.) of the original program.\nIn the end, the core is to find the transformation matrix $\\tau_s$ that satisfies all domain constraints while minimizing the cost function, and usually we do this via ILP. For example, in the above example, we can add constraints that the part reading C in statement S2 must execute after the part writing C in statement S1, and constraints on the loop ranges of i, j, k. We then solve the ILP with these constraints.\nA more concrete algorithm for finding the optimal transformation matrix will be explained in a later post.\nReferences Effective Automatic Parallelization AND Locality Optimization Using The Polyhedral Model (Uday, 2008)\n","date":"2026.01.29","permalink":"https://hyperaccel.ai/techblog/en/posts/polyhedral-compiler-analysis/","series":["Crafting Compilers"],"style":"--c:#5BC5F6;--on:#02121c","summary":"Today I brought a somewhat different topic. Have you ever thought it would be great if a compiler could automatically transform loops for optimization or parallelization while using it? Compiler engineers have had the same concerns. One approach that emerged is the polyhedral compiler, which is used by LLVM\u0026rsquo;s Polly project and MLIR\u0026rsquo;s affine dialect. Today I\u0026rsquo;ll introduce this approach.\nBasic concepts Before going into detail, let\u0026rsquo;s go over some basic concepts. Most of these are easy to encounter with a little linear algebra.\n","title":"[Crafting Compilers - Part 03] Polyhedral Analysis"},{"authors":["Hyunjun Park"],"category":"Conference Reports","content":"CES 2026 Key Highlights from a 2025 Participant (feat. SAMSUNG, SK, LG) Hello! I\u0026rsquo;m Hyunjun Park from the HyperAccel ML Team.\nCES, which started in Las Vegas on January 6th, successfully concluded on January 9th. I participated in last year\u0026rsquo;s (2025) CES and operated the HyperAccel booth, and based on that experience, I\u0026rsquo;d like to review this year\u0026rsquo;s (2026) CES. Some might wonder why I\u0026rsquo;m posting this now instead of last year.\nThe reason is simple: we didn\u0026rsquo;t have a tech blog last year. As someone who participated last year, reading articles about CES 2026 made me curious about how this year went, and the excuse of writing for our tech blog gave me the opportunity to learn about this year\u0026rsquo;s CES. This post will be helpful for those who:\nWant to know CES statistics and key highlights Are curious about CES booth preparation and operation experience Want to learn about Korean conglomerates\u0026rsquo; (LG, SAMSUNG, SK) exhibits and concepts CES Statistics and Key Highlights CES is the world\u0026rsquo;s largest consumer electronics exhibition held annually since 1967. From a company\u0026rsquo;s perspective, it\u0026rsquo;s an opportunity to showcase technology for marketing and branding, while visitors can understand the latest technology trends. South Korea ranks third in company participation after the United States and China, and holds the world record for the most innovation awards, demonstrating Korean technological capabilities to the world.\n2025 2026 Slogan DIVE IN - Connect. Solve. Discover innovators show up Keywords AI \u0026amp; Robotics, Mobility, Extended Reality, Smart Home, Digital Healthcare Physical AI, Spatial Computing, Digital Health, Mobility, Smart Home Participating Countries \u0026amp; Companies 160+ countries, 4800+ companies 150+ countries, 4500+ companies Participating Korean Companies 1000+ companies 1000+ companies Korean Best of Innovation Awards 15 out of 34 (44%) 15 out of 30 (50%) Korean Innovation Awards 219 out of 458 (47.8%) 222 out of 367 (60.5%) My first impression of CES was that it was much larger than I expected. It felt like walking around a university campus. Even though I had organized the booths I wanted to visit beforehand, it was difficult to find the booths I wanted, and moving to different campuses required shuttle buses with long waiting lines, making route planning challenging.\nWhile pondering this, I applied on-site for a booth tour organized by the K-STARTUP integrated pavilion, which allowed me to see notable global companies\u0026rsquo; booths with an optimized route. It was great to easily get an overview of global company booths, but on the other hand, I was disappointed that I couldn\u0026rsquo;t focus on the booths I wanted to see. I recommend using booth tours if you don\u0026rsquo;t have specific needs, but if you have specific needs, I recommend planning your route thoroughly in advance.\nKOTRA provides annual analysis on viewing points and routes. Although I didn\u0026rsquo;t write it, I think it\u0026rsquo;s well-written and would like to recommend and share it.\nCES 2025 Guide: https://dl.kotra.or.kr/pyxis-api/2/digital-files/51efe51a-718d-4f3a-9182-12c424384f7b\nCES 2026 Guide: https://dl.kotra.or.kr/pyxis-api/2/digital-files/719ffe16-6ece-4214-969f-b663985315d3\nCES 2025 Participation Experience The Ministry of SMEs and Startups is conducting the \u0026ldquo;Super Gap Startup 1000+\u0026rdquo; project to foster startups, and as part of this project, they gather Korean startups to operate a CES exhibition pavilion annually. It was well-organized so that even small startups could participate comfortably.\nThanks to lowering the barrier to entry for CES, many companies have been participating in the K-STARTUP integrated pavilion both last year and this year. This year, 470 companies from 38 organizations participated. Looking at Korea as a whole, about 1,000 companies participated. I believe this support is one of the key driving forces behind Korea receiving the most CES innovation awards.\nFrom an exhibitor\u0026rsquo;s perspective, thanks to the K-STARTUP branding, visitors of various nationalities and fields came to our booth. We were able to talk with numerous clients, investors, and partners. From a company standpoint, it was a valuable opportunity to build relationships with partners and brand ourselves to clients, and personally, it was a great opportunity to gain insights.\nThanks to their great support, despite being our first time, we were able to focus on \u0026ldquo;exhibiting\u0026rdquo; rather than just \u0026ldquo;participating\u0026rdquo; and successfully completed our CES booth operation. This is a photo of our booth from last year. The server shown in the photo is our first-generation FPGA-based server, which was the first step in developing next-generation AI chips for LLM inference.\nKorean Conglomerates\u0026rsquo; CES Exhibition Trends There were many global companies that surprised everyone with new technologies, but when I was actually at the venue, I was more excited to see Korean companies standing shoulder to shoulder with global companies on the world stage. (National pride!) Therefore, in this section, I\u0026rsquo;d like to explore how Korean conglomerates Samsung, LG, and SK operated their CES exhibitions by comparing 2025 and 2026.\nCompany Category 2025 2026 LG Innovation Awards 24 products including LG OLED TV 18 products including webOS platform Highlights Structure made of 28 OLED TVs Home robot CLOi that helps with housework Samsung Innovation Awards 29 products including Galaxy Buds3 Pro 27 products including quantum security chip Highlights Smart home with all appliances connected World\u0026rsquo;s first 130-inch Micro RGB TV SK Innovation Awards HBM3E, etc. HBM4, etc. Highlights Sole Best of Innovation Award in memory category World\u0026rsquo;s first HBM4 16-layer 48GB reveal LG in CES I felt that LG strives to make a visual impression on the audience. Among all the exhibitions I saw last year, LG\u0026rsquo;s exhibition was the most impressive.\nLast year, LG displayed 28 transparent/wireless OLED TVs in a circular formation to create a single structure showcasing their technological prowess. This LG showcase not only overwhelmed the surrounding space but also graced the cover of the official CES magazine.\nWhen you entered the booth, you could see the actual products up close, and they looked as if images were appearing on windows rather than TVs, making them aesthetically excellent. For reference, LG Signature OLED TV has won the Best of Innovation Award for four consecutive years from 2023 to 2026.\nThis year, LG launched the home assistant CLOi and once again made headlines.\nIn the actual demo, they showcased various demonstrations such as taking milk out of the refrigerator, folding towels, and putting in laundry. Personally, I thought it wasn\u0026rsquo;t yet at a level of completion for actual commercialization, but it was meaningful as an early version of a home robot. Given the speed of AI development, I\u0026rsquo;m really looking forward to seeing how much more complete it will be next year.\nSAMSUNG in CES Samsung presented the story of smart homes with a museum concept at CES both last year and this year.\nLast year, they displayed how AI connects appliances, increases productivity, and enables efficient energy use. They particularly emphasized the smart home ecosystem where refrigerators, washing machines, air conditioners, and all appliances are connected through the SmartThings platform.\nThis year, they exhibited how AI can provide enjoyment, convenience, and care in daily life.\nThe world\u0026rsquo;s first 130-inch Micro RGB TV displayed in the main hall was enough to capture the audience\u0026rsquo;s attention. Additionally, they showcased their vision of AI permeating daily life through the Bespoke AI refrigerator with Google Gemini and wearable-based health management solutions. Samsung\u0026rsquo;s approach of integrating AI across all areas from displays to home appliances to healthcare was impressive.\nSK in CES SK showcased HBM as their main technology both last year and this year. I\u0026rsquo;d like to carefully share my personal opinion: SK is relatively at a disadvantage (?) when it comes to demos because it\u0026rsquo;s difficult to visually showcase core technologies due to the nature of memory semiconductors.\nHowever, last year\u0026rsquo;s SK exhibition hall entrance gave a feeling of entering a cave behind a waterfall, which was sufficient to make people stop and attract their attention. I found out that they implemented graphics that visualized bits like waves to represent that it\u0026rsquo;s a memory company. And the mere existence of HBM is already the hottest topic in the world, so many people visited.\nSince I work at a semiconductor company, I was naturally more interested. I was able to see the actual products and specifications of various memory semiconductors including HBM, which was great. To briefly explain HBM, which has captured Jensen Huang\u0026rsquo;s love: HBM (High Bandwidth Memory) is a product that vertically connects multiple DRAMs to increase bandwidth, and its key feature is that it has a revolutionary faster data transfer speed compared to conventional DRAM. HBM3E, displayed at the top, means the Extension version of HBM3, and it\u0026rsquo;s such a hot model that more than half of HBM shipments in 2026 are expected to be HBM3E.\nThis year, they first revealed HBM4 16-layer. HBM4 goes beyond simple performance improvements and changed the entire architecture. They doubled the bandwidth and added logic processes to the bottom layer of memory, enabling computation functions near the memory.\nIt was impressive to see SK Hynix, which is leading HBM—a key component in the AI era—receive global attention at CES. I look forward to SK Hynix\u0026rsquo;s continued success in the memory semiconductor field. (Please make memory cheaper)\nConclusion CES once again proved it\u0026rsquo;s the World\u0026rsquo;s Best Show through its scale, the efforts of many companies, and tremendous attention from around the world.\nAt CES 2025, Jensen Huang said that the next frontier for AI is Physical AI. Now, one year later, looking at CES 2026, what stood out most was that AI is no longer confined to software but is expanding into the physical world. As a prime example, LG\u0026rsquo;s CLOi demonstrated its use in actual homes, and although not covered in this post, dancing and working robots were easily found at various booths.\nAnd at CES 2026, Lisa Su said that a world is coming where billions of AI agents will run 24/7 in the background for all of humanity. As such, AI is increasingly penetrating human life, and to support this demand, hardware development is rapidly progressing—not only NVIDIA GPUs but also SAMSUNG and SK\u0026rsquo;s HBM4. This trend will accelerate further.\nI believe that for a company to succeed (=achieve large revenue), three elements are needed: \u0026ldquo;clear market,\u0026rdquo; \u0026ldquo;innovative product,\u0026rdquo; and \u0026ldquo;great team.\u0026rdquo; And looking at this year\u0026rsquo;s CES, I felt that the current AI market is growing at an unprecedented pace, and many Korean companies are receiving global attention in this environment. Sending respect to the many companies that achieved remarkable technological advances in just one year, I look forward to Korean companies\u0026rsquo; activities at CES 2027.\nPostscript HyperAccel Hiring Notice This opportunity also served as a moment to reflect on HyperAccel. For us, an NPU company, the \u0026ldquo;clear market\u0026rdquo; couldn\u0026rsquo;t be better, and the \u0026ldquo;innovative product\u0026rdquo; LPU has already completed functionality and performance verification through FPGA and is waiting for ASIC chip release. We are waiting for you to join us in building the final element—a \u0026ldquo;great team.\u0026rdquo;\nCareer Site: https://hyperaccel.career.greetinghr.com/ko/guide\nIf you\u0026rsquo;re interested, please feel free to contact us anytime!\n","date":"2026.01.12","permalink":"https://hyperaccel.ai/techblog/en/posts/ces2026-report/","series":[],"style":"--c:#A7C4FF;--on:#02121c","summary":"CES 2026 Key Highlights from a 2025 Participant (feat. SAMSUNG, SK, LG) Hello! I\u0026rsquo;m Hyunjun Park from the HyperAccel ML Team.\nCES, which started in Las Vegas on January 6th, successfully concluded on January 9th. I participated in last year\u0026rsquo;s (2025) CES and operated the HyperAccel booth, and based on that experience, I\u0026rsquo;d like to review this year\u0026rsquo;s (2026) CES. Some might wonder why I\u0026rsquo;m posting this now instead of last year.\n","title":"CES 2026 Key Highlights from a 2025 Participant (feat. SAMSUNG, SK, LG)"},{"authors":["Jaewon Lim"],"category":"AI Hardware","content":"Know Your Enemy, Know Yourself, Part 1: GPU History and Fundamentals\nKnow Your Enemy, Know Yourself\nThis means that if you know your enemy and know yourself, you will not be imperiled in a hundred battles.\nThis series aims to deeply understand competitors\u0026rsquo; hardware for AI accelerator design.\nIn this second article, we cover Google\u0026rsquo;s TPU (Tensor Processing Unit), which has recently declared its entry into the AI accelerator market.\nWhat is TPU? TPU has become one of the hottest keywords in the AI accelerator market. Last November, Google introduced Ironwood, the 7th-generation TPU architecture, announcing improved training and inference performance, cluster scalability, and collaboration with AI startup Anthropic. Both domestic and international media have covered this extensively, making TPU a known term to the general public, and this news has significantly impacted the stock prices of both Google and NVIDIA.\nHowever, TPU is not a product that suddenly appeared. The recently announced TPU is the 7th generation. Google has been releasing new generations of TPU every year since 2016. Initially used only for internal model development and research, including AlphaGo, Google has also sold services through Google Cloud that allow other companies and research institutions to use TPU.\nBut when you think about it, it\u0026rsquo;s strange. Google is clearly a software company, so why did they create their own chip instead of using GPUs? And where does the confidence to enter the hardware market come from? Today, we will explore the origins of TPU and the latest generation Ironwood to understand Google\u0026rsquo;s \u0026lsquo;well-founded confidence\u0026rsquo; in the AI semiconductor market.\nThe Emergence of TPU To understand the TPU structure, we must first understand the background of TPU development. This is detailed in the TPU 1st generation paper.\nIn the 2000s, Google\u0026rsquo;s services could handle even tasks requiring special types of computation using idle resources in existing data centers without needing custom ASICs. However, as AI technology was fully integrated into Google services in the 2010s, the situation changed. When assuming people use voice search for just 3 minutes per day, predictions showed that running deep neural networks (DNN) for voice recognition would require doubling the scale of existing data centers. As general CPUs could no longer handle this, Google internally started a custom ASIC project, the TPU project. The remarkable thing is that design, verification, manufacturing, and deployment were completed in just 15 months. The TPU produced at that time was used only internally at Google until 2016, when it was revealed that TPU was used in the match between Lee Sedol and AlphaGo, making TPU known to the outside world for the first time.\nTPU Hardware Architecture Now let\u0026rsquo;s examine the TPU architecture in detail.\nSystolic Array As mentioned earlier, TPU was designed specifically for AI operations. The biggest characteristic of AI operations is performing large-scale matrix multiplication. This is a common feature seen in the process of using artificial neural network weights and computing other values during training and inference, regardless of model type (CNN, RNN, Transformer).\nTPU uses a special unit called Systolic array, which cannot be found in general processors (CPUs), to efficiently execute this matrix multiplication. The term \u0026ldquo;Systolic\u0026rdquo; is derived from \u0026ldquo;systole,\u0026rdquo; the contraction phase of the heart. Just as the heart rhythmically beats and sends blood to various parts of the body, data moves rhythmically and regularly between computational units within the array structure, performing operations - hence the name. Systolic array optimizes data flow and maximizes parallel processing, making it efficient for large-scale operations like matrix multiplication. The process of Systolic array performing matrix multiplication can be visualized as an animation below.\nYour browser does not support the video tag. Next, to explain the effectiveness of Systolic array in more detail, let\u0026rsquo;s compare the operation method of general processors with TPU\u0026rsquo;s systolic array operation method.\nThe process of performing arithmetic operations such as addition or multiplication in a general processor is as follows:\nLoad data into register Controller sends commands to ALU (Arithmetic Logic Unit) ALU reads data from register, performs operation, and writes result back to register In other words, every time an ALU performs an individual operation, it needs to read data from and write to registers. Large-scale matrix operations used in AI require significantly more computation than general operations. If we could eliminate this process:\nWe could increase area efficiency by reducing required register capacity We could reduce power consumption by decreasing data movement between register ↔ ALU Systolic array is one alternative that can eliminate this process. Instead of storing the results computed by individual ALUs in registers, it passes them to other ALUs. This achieves the effects mentioned above.\nHowever, to effectively use Systolic array, several conditions must be met:\nThe timing for passing data between ALU → ALU must be precisely synchronized It must be guaranteed that results computed by individual ALUs are used in the next ALU While the first condition can be controlled appropriately at the hardware level, the second condition must be satisfied by the nature of the operation itself. Interestingly, matrix multiplication satisfies this condition, making Systolic array the most efficient for this purpose.\nMatrix multiplication can be decomposed into multiple vector dot products. A vector dot product is the accumulated sum of individual element multiplications. By precisely synchronizing the timing of elements entering individual ALUs as input and passing the multiplication values to the next ALU while sequentially accumulating, matrix multiplication can be performed without communication with registers.\nThis Systolic array is a representative feature used in TPU and has had a significant impact on GPUs and other AI accelerators (NPU, LPU) that perform AI operations.\nTPU Building Block Next, let\u0026rsquo;s examine the computation units used in individual TPU chips. As we saw earlier, GPUs consist of hundreds of SMs (Streaming Multiprocessors). The hardware scheduler selects warps (32-thread bundles) that are ready for execution in real-time and feeds them into computational resources. This is not about executing instructions randomly, but rather a method of maximizing efficiency by filling waiting time with other tasks (Latency Hiding). In other words, GPUs are optimized for maximizing overall throughput by densely managing numerous small computational units.\nIn contrast, TPU (Tensor Processing Unit) takes a contrasting approach. Instead of numerous finely divided units, TPU is equipped with a small number of powerful dedicated cores capable of processing massive matrix operations at once. It also operates by pushing large amounts of data into computational blocks at once through higher-level instructions (VLIW, Very Long Instruction Word) that combine multiple instructions. Rather than relying on complex dynamic scheduling, TPU architecture is designed to perform large-scale matrix multiplication—the core of AI operations—in a single pass, like a massive pipeline flowing through, which is the essence of TPU architecture.\nThe cores used in individual TPU chips are divided into TensorCore, specialized for linear algebra operations, and SparseCore, specialized for embedding operations. (Note: While the terminology is the same as GPU\u0026rsquo;s Tensor Core, their roles differ.)\nTensorCore\nTensorCore is a computation pipeline that performs operations on data delivered through Systolic array and DMA (direct memory access) units. Similar to GPU\u0026rsquo;s CUDA cores or Tensor cores, while GPUs cannot specify exactly which core to use at which time (automatically filling available execution units), TPU\u0026rsquo;s TensorCore is statically allocated according to dataflow optimized at compile time. The MXU (Matrix Multiplication Unit) for matrix operations mentioned earlier is the most representative, and it also includes operators for scalar and vector operations, not just matrix operations. Depending on the generation, 1 or 2 TensorCores are mounted per chip.\nSparseCore\nSparseCore is a core architecture applied from the 4th generation TPU. SparseCore is a unit specialized for embedding operations, one of the operations used in AI besides matrix operations. While embedding operations used in LLMs have relatively small table sizes and don\u0026rsquo;t have excessive computational overhead, in recommendation systems like DLRM (Deep Learning Recommendation Model), embedding table sizes can account for most of the total parameters, making the overhead very large, so there\u0026rsquo;s a very high probability of computational bottlenecks in embedding operations.\nThe proportion of SparseCore in TPU (based on 4th generation) is about 5%, small compared to TensorCore, but this unit achieved 5-7x performance improvement compared to before. Depending on the generation, 2 or 4 SparseCores are mounted per chip.\nScale-up/Scale-out System TPU performs operations with a small number of massive cores, 1-2 per chip. While it\u0026rsquo;s possible to perform operations with a single chip using massive cores, connecting multiple chips and using them in parallel is necessary for larger operations. TPU uses the following network technologies for this purpose.\nTorus Topology\nTPU uses torus topology for scale-up, connecting multiple chips to build one massive server. When thinking about connecting multiple chips, one typically imagines a grid structure like a Go board (mesh topology) as shown in the first figure. The disadvantage of this structure is that nodes at both ends are not connected, making the maximum distance between nodes longer. In the torus structure, nodes at both ends are connected with wrap-around links, increasing wiring complexity but reducing the maximum distance between nodes from N to N/2. In the 4th generation TPU, this torus structure was extended to 3D, expanding the communication range between nodes and the number of TPUs grouped together. A group of 4 * 4 * 4 (=64) TPUs is called a cube, and a cluster of multiple cubes is called a pod.\nOptical Circuit Switch (OCS)\nA network switch is needed to mediate communication between chips. TPU introduced OCS, an optical switch, to allow the connection structure between torus networks to be reconfigured in software. This enables software to determine the optimal connection structure according to computation characteristics.\nTPU Software Stack Next, let\u0026rsquo;s examine the software used in TPU.\nTensorFlow As mentioned in the previous post, GPUs can be controlled through CUDA, a programming model. Instead of developing a kernel language like CUDA, Google developed a deep learning framework called TensorFlow that can be used regardless of GPU or TPU.\nThe characteristic of frameworks like TensorFlow and PyTorch is that, unlike CUDA, developers can execute the same code on various accelerators like GPU and TPU immediately without understanding the detailed structure of the hardware. Google provided strong flexibility so that the same TensorFlow source code could run on various hardware platforms (CPU, GPU, TPU) and be deployed anywhere from mobile devices to large-scale distributed systems for ecosystem expansion. This made TensorFlow a powerful tool for industry/deployment. However, precise hardware control is difficult, so there are limitations from an optimization perspective.\nJAX However, there is a way to optimize TPU hardware at the software level. This is possible using JAX, a Python library. This is a Python library developed by Google internal developers who actively use TPU to use TPU more efficiently. TPU can use XLA, a machine learning acceleration compiler that will be explained shortly. XLA is a compiler that can be used in both TensorFlow and JAX. JAX is a kind of Python interface created for software developers to use this XLA more directly and flexibly. It can be used regardless of GPU/TPU, and provides functional programming paradigms and automatic differentiation capabilities, making it more suitable for research and experimentation. Google\u0026rsquo;s AI team actively uses it for research and model training, but it has a somewhat high barrier to entry because operations must be optimized at the software level.\nXLA (Accelerated Linear Algebra) Google began to consider ways to overcome the limitations of TensorFlow mentioned earlier and further improve TensorFlow\u0026rsquo;s performance in machine learning. Google likely had two main directions to choose from:\nCreate a separate low-level language (e.g., TPU-C) that can directly control hardware similar to CUDA, and create optimized kernels that can be called from high-level languages like TensorFlow Use a compiler specialized for machine learning to generate machine code optimized for that hardware The former approach gives developers direct hardware control, providing high freedom. Additionally, whenever new features are added to hardware or new algorithms (FlashAttention, Mixture-of-Expert (MoE)) are developed, developers must write new CUDA kernels for optimization. The demand for optimization always exists, and each time, the CUDA ecosystem expands. This acts as a huge technical moat in the CUDA ecosystem that makes it difficult for other hardware companies to enter the market.\nGoogle chose the latter approach, creating the XLA compiler specialized for linear algebra operations, which are central to machine learning operations. While they avoided directly challenging the CUDA ecosystem, you could say they shifted the burden of developers having to learn new languages to the compiler. Let\u0026rsquo;s examine how the XLA compiler optimizes operations through some AI operations.\n(* This explanation is an example to illustrate the optimal approach of the XLA compiler and does not match actual operation.)\nimport math # 1. Linear: z = weight * x + bias def linear_op(x, weights, bias): # write result to main memory z = [sum(xj * wj for xj, wj in zip(x, weights[i])) + bias[i] for i in range(len(weights))] return z # 2. Activation: ReLU def relu_op(z): # Read z value from main memory -\u0026gt; write back result to main memory a = [max(0, val) for val in z] return a # 3. Softmax def softmax_op(a): # read a from main memory -\u0026gt; find max -\u0026gt; summation -\u0026gt; div -\u0026gt; write back result to main memory max_val = max(a) exp_vals = [math.exp(val - max_val) for val in a] sum_exp = sum(exp_vals) return [ev / sum_exp for ev in exp_vals] Simply executing the above 3 functions sequentially requires writing intermediate results of each function to main memory and reading them again. The core of the XLA compiler is to integrate these operations, remove unnecessary computations or memory allocations, and perform operations with fast speed and low memory usage.\nGPUs can also integrate multiple operation kernels through kernel fusion. While GPUs have two methods - manually creating integrated CUDA kernels and automatically fusing kernels using PyTorch or TensorRT internal features - XLA differs in that it analyzes individual operations at compile time and generates machine code optimized for each hardware (CPU/GPU/TPU).\nXLA was supported in TensorFlow and JAX, and recently its base has expanded as PyTorch also supports XLA through pytorch/xla.\nPallas XLA is a powerful optimization compiler, but it has limitations. When new operation algorithms emerge, it\u0026rsquo;s difficult for manually created custom kernels to match performance until the compiler is updated to a version that can optimize them.\nTo address this, Google has been providing a kernel language API called Pallas (jax.experimental.pallas) as an experimental extension of JAX since around 2023. This corresponds to the first of the two methods mentioned earlier for performance improvement. Pallas can be compared to Triton, a high-level kernel language for GPUs born in 2021.\nTriton is an open-source GPU programming language created by OpenAI in 2021. It overcomes CUDA\u0026rsquo;s limitations of high barriers to entry and requiring understanding of hardware structure by providing a more abstracted interface than CUDA, enabling machine learning developers to more easily optimize various operations and algorithms on GPUs. It abstracts away from Hopper/Blackwell and other architectures to increase developer convenience, and despite the high level of abstraction, there are benchmarks showing that programs created with Triton reach similar performance levels to manually created CUDA kernels.\nIntegration to use Triton language in JAX has also progressed, but since Triton only works on GPUs, it was difficult to perform such optimizations on TPU. Pallas is a TPU kernel-language API created to overcome this, and unlike Triton, it provides a common kernel language that supports both GPU and TPU, which is its major feature. Pallas has been integrated not only in JAX but also in PyTorch/XLA.\nIronwood Launch: Google\u0026rsquo;s Declaration of Entry After the 4th generation TPU, there isn\u0026rsquo;t much detailed technical documentation, so we\u0026rsquo;ll discuss what changed in the 7th generation TPU compared to previous generations and what it implies for the AI semiconductor industry from a technical perspective.\nHardware Perspective Chiplet Architecture\nIronwood is the first TPU architecture to adopt chiplet structure. Chiplet structure is a process technology introduced to overcome the limitations of semiconductor manufacturing processes called the reticle limit.\nReticle Limit: The reticle limit refers to the maximum size of an individual die that can be produced in a single exposure, determined by the reduction ratio of the lens in the lithography equipment used to draw circuits on the photomask (reticle), which is standardized and fixed in the semiconductor industry. When this limit (currently approximately 858mm²) is exceeded, single exposure becomes impossible, requiring a stitching technique that connects multiple exposure patterns. However, in this case, defect rates increase rapidly and manufacturing costs rise. For these economic reasons, existing chips were manufactured so that the processor area (area other than memory) did not exceed this size.\nHowever, as the demand specifications for chips used in data centers increased, larger chips became necessary, and instead of increasing single chip size, chiplet structure was introduced, where chips are split into multiple chips of the same structure and connected at the packaging stage. This allows individual chip sizes to increase, enabling individual product performance to increase.\nNVIDIA GPUs introduced this technology starting with the Blackwell architecture B100/B200 product line released in 2024, and other accelerator companies making data center accelerators are also adopting this structure. TPU applied chiplet structure starting with this generation\u0026rsquo;s product line, and we can see that single product performance (peak compute, memory bandwidth, etc.) is significantly higher than previous generations.\nIn this chiplet structure, since 2 individual dies show performance beyond individual chips of previous products, Ironwood treats individual dies as one node in torus topology, enabling individual control. For this, 4D torus topology was introduced by adding one more axis to 3D torus.\nSoftware Perspective vLLM Support and the Emergence of tpu-inference\nGoogle integrated a new hardware plugin called tpu-inference into vLLM last October. This plugin unifies JAX and PyTorch into a single lowering path, generating optimized TPU machine code regardless of framework. The biggest advantage of this plugin is that it enables high-performance LLM model serving on TPU using the same vLLM interface used on GPU.\nTensorFlow Support Discontinuation\nGoogle boldly discontinued TensorFlow support starting with this generation TPU. This is interpreted as having two meanings: acknowledging PyTorch\u0026rsquo;s ecosystem dominance while integrating XLA into the existing ecosystem to optimize TPU performance within the PyTorch ecosystem (PyTorch/XLA), and focusing more on the JAX framework, which is more optimized for performance improvement than TensorFlow for independent framework development.\nPallas\u0026rsquo;s Establishment in the Software Ecosystem\nAlthough Pallas is still documented as an experimental feature in JAX official documentation, Google officially introduces Pallas as a means to maximize TPU performance in their blog. This is interpreted as encouraging developers to write optimized kernels directly without waiting for XLA compiler updates. The freedom of kernel optimization that existed in CUDA/Triton can now be enjoyed on TPU as well.\nGoogle\u0026rsquo;s Ambition in the AI Semiconductor Market This expansion of the software ecosystem and dramatic improvement in hardware performance means more than just technical improvement - it represents Google\u0026rsquo;s strategic shift. This is a move to formally enter the AI semiconductor market that NVIDIA has monopolized. Let\u0026rsquo;s examine Google\u0026rsquo;s market entry strategy below.\nDirect TPU Sales Possibility\nUntil now, TPU could only be leased through Google Cloud (GCP). For security reasons, detailed hardware technical documentation was not publicly disclosed, and direct sales were not conducted, so TPU existed only in Google\u0026rsquo;s internal data centers. However, it\u0026rsquo;s been reported that Google is considering strategies to directly supply TPU to third-party data centers or sell the hardware itself. This is a declaration to shed the image of a closed Google-exclusive chip and compete head-on with NVIDIA as an independent AI semiconductor vendor.\nSoftware Openness and Completion of Vertical Integration\nNVIDIA\u0026rsquo;s strength comes not only from hardware but also from the CUDA software ecosystem. Google is countering this with XLA automation and Pallas openness as weapons. They\u0026rsquo;re sending the message: \u0026ldquo;GPUs require manual kernel optimization for best performance, but TPU automatically optimizes through the compiler and developers can also tune directly if needed.\u0026rdquo; Moreover, Google\u0026rsquo;s software stack, unlike CUDA, is not limited to specific hardware, enabling competitive advantage in the emerging heterogeneous computing market.\nFurthermore, Google is the only company that has vertically integrated the entire AI stack: models (Gemini), frameworks (JAX), compilers (XLA), and hardware (TPU). In the AI semiconductor market where software and hardware integration is essential, Google has secured an advantageous position through this vertical integration.\nSummary In this article, we explored:\nThe background of TPU\u0026rsquo;s emergence TPU\u0026rsquo;s hardware/software architecture Features of the latest TPU product, Ironwood Google\u0026rsquo;s semiconductor market strategy The conclusions we can draw from this are summarized below:\nGoogle has been developing TPU, an ASIC optimized for AI workloads, for 10 years. They continue their technical efforts to break down the technical moat of GPU and CUDA and build a software ecosystem that is not hardware-dependent. Beyond simple cloud business, they aim to directly enter the AI semiconductor market and establish equal status with NVIDIA. Did you know that members who participated in the original TPU project had founded another semiconductor startup? Groq, an AI semiconductor startup that recently signed a contract worth approximately $20 billion (30 trillion KRW) with NVIDIA, is that company. What technology did NVIDIA pay such an enormous amount for? In the next article, we\u0026rsquo;ll explore Groq\u0026rsquo;s LPU, which uses the same terminology as us.\nReference An in-depth look at Google\u0026rsquo;s first Tensor Processing Unit (TPU) From silicon to softmax: Inside the Ironwood AI stack Announcing Ironwood TPUs General Availability and new Axion VMs to power the age of inference In-Datacenter Performance Analysis of a Tensor Processing Unit TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings P.S.: HyperAccel is Hiring! Knowing your enemy and yourself is important, but to truly win we also need great people on the team.\nIf you’re interested in the kinds of technologies we work on, please apply via HyperAccel Career.\nWe’re looking forward to hearing from talented engineers like you.\n","date":"2026.01.03","permalink":"https://hyperaccel.ai/techblog/en/posts/tpu-deep-dive/","series":["Know Your Enemy, Know Yourself"],"style":"--c:#00C2A8;--on:#02121c","summary":"We explore the background of TPU\u0026rsquo;s emergence and analyze Google\u0026rsquo;s AI semiconductor strategy by examining its hardware and software architecture.","title":"[Know Your Enemy - Part 02] TPU Emergence and Rise"},{"authors":["Donghyeon Choi"],"category":"AI Hardware","content":" Know Your Enemy, Know Yourself\nIf you know the enemy and know yourself, you need not fear the result of a hundred battles.\nThis series aims to deeply understand competitors\u0026rsquo; hardware for designing AI accelerators.\nThis first post covers NVIDIA GPUs, our most formidable competitor.\nWhen we talk about hot topics in the 2020s, it is almost impossible to avoid talking about NVIDIA GPUs.\nIf we are building AI accelerators that will compete with NVIDIA, we first need to understand how NVIDIA got here and what kind of device a GPU is.\nThis first post in the series looks at how the NVIDIA GPU was born, which architectural choices brought it to its current position,\nand summarizes the strengths and structural characteristics of this design from a hardware engineer\u0026rsquo;s point of view.\nNVIDIA was founded in 1993 and entered the graphics market in 1995 with the NV1.\nWith RIVA 128 (1997) and GeForce 256 (1999), the company firmly established itself in the PC graphics card market,\nbut then hit a major crisis when the GeForce FX series failed.\nFrom that failure, NVIDIA rebuilt its GPU architecture, introducing the G80 Tesla architecture and CUDA.\nThat pivot turned the GPU from \u0026ldquo;just\u0026rdquo; a graphics chip into a platform that could also dominate the HPC and GPGPU markets.\nIn 2012, AlexNet – a convolutional neural network trained on GPUs – won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC),\nfirmly tying GPUs to deep learning.\nIn 2022, the release of ChatGPT pushed generative AI into the mainstream,\nand the AI hardware device market effectively consolidated around NVIDIA; the company would go on to become the world\u0026rsquo;s most valuable by market cap.\nToday, NVIDIA GPUs are no longer \u0026ldquo;just graphics accelerators\u0026rdquo; – they are a de‑facto standard parallel computing platform that underpins the AI infrastructure layer,\nand, from our point of view, a baseline that any new accelerator must beat.\nSo how did GPUs emerge, and how did they expand from graphics into the AI market?\nThe Birth of the Graphics Card Early mass‑market computers used command line interfaces (CLI). The screen was mostly green text on a black background,\nand there was very little graphics work to do. The CPU could easily handle both computation and screen updates.\nBut CLIs required users to remember commands, which limited usability and mass‑market appeal.\nSystems like Xerox Alto and Apple Lisa kicked off the era of graphical user interfaces (GUI) with windows, icons and a mouse.\nThis greatly improved usability, but it also put more load on the CPU.\nIn response, vendors like IBM steadily beefed up the graphics circuits on the motherboard, and by the time we reached VGA (Video Graphics Array),\ngraphics had become a first‑class component in the PC.\nIn the 1990s, demand for 3D games exploded and the need for graphics compute grew with it.\nDOOM (1993) was a breakthrough 3D game and required around ten million operations per second.\nThe required compute can be thought of as:\nframes per second × resolution × physics simulation × texture effects\nAs each of these factors improved, the required operations grew explosively.\nHere\u0026rsquo;s how the compute requirements evolved across major games:\nDOOM (1993): ~10 million operations per second Super Mario 64 (1996): ~100M operations per second Half‑Life (1998): ~500M operations per second World of Warcraft (2004): ~2.2B operations per second Minecraft (2011): ~100G operations per second Call of Duty: Modern Warfare III (2023): 30–40 trillion operations per second In just 30 years, compute requirements increased by roughly 3–4 million times.\nWhat are these \u0026ldquo;graphics operations\u0026rdquo; in practice?\nThe CPU first sends geometric information from 3D space – a set of vertices.\nTo transform these vertices into final pixel colors on the screen, the GPU runs small programs called shaders.\nShaders provide programmable control over each stage of the rendering pipeline,\nperforming operations such as transformations, lighting, and color calculations on vertices or pixels.\nFirst, a vertex shader places the vertices in screen space.\nThen primitive generation and rasterization map them to pixels.\nFor each pixel, fragment shaders compute the final color.\nThese computed pixel colors are written into the framebuffer for display.\nOn top of this basic graphics pipeline, additional effects like anti‑aliasing, blending, transparency and shadow mapping were applied for more realistic rendering.\nIf we look at an NVIDIA FX‑series GPU, early GPUs essentially hard‑wired this graphics pipeline into hardware.\nThere was a separate vertex shader block for vertex shading, a separate fragment shader block for fragment shading.\nIn other words, a fixed‑function accelerator dedicated to the graphics pipeline.\nThis strictly feed‑forward pipeline had an important drawback:\nif an intermediate stage took a long time, upstream stages would stall, leading to bubbles and poor utilization.\nThis problem became more apparent with the arrival of programmable shaders.\nCustom shader programs often took longer to run, which hurt the overall hardware utilization,\nand as the pipeline got deeper, it became harder to identify where the bottlenecks were.\nIn this situation, NVIDIA came up with an innovative solution: the G80 Tesla architecture with unified shaders.\nInstead of separate vertex and fragment shader hardware, all shader work could run on a common pool of cores.\nComputation that had been processed as small fixed‑width vectors of pixels was broken down into per‑pixel work,\nand many such pieces were scheduled together across the cores.\nThis architectural shift meant that even if some shader invocations took a long time,\nother shader work could still make progress on the same cores.\nThe classic issues of a strict one‑way pipeline were greatly reduced.\nThis shift opened the door to a new way of thinking about GPUs: GPGPU for HPC.\nThe Rise of GPGPU and CUDA As GPUs evolved as graphics processors, researchers started asking a simple question:\n“If GPUs can do so many operations per second, can we use them for general‑purpose math?”\nAs GPUs evolved as graphics processors, researchers started asking a simple question.\nAround 2003, two independent research teams showed that\n\u0026ldquo;general linear algebra problems could be solved faster on GPUs than on high-performance CPUs\u0026rdquo;,\nwhich kicked off the first wave of interest in GPGPU.\nTo understand the background: DirectX 8 (2000) introduced programmable shaders,\nallowing developers to write custom shader programs that run on dedicated shader hardware.\nDirectX 9 (2002) added the shader language HLSL (High-Level Shader Language),\nmaking programmable shaders truly practical.\nAt the same time, the fundamental nature of the GPU workload – \u0026ldquo;the same operation on many pixels\u0026rdquo; –\nlooked a lot like linear algebra problems such as matrix multiplication – \u0026ldquo;the same operation on many data elements\u0026rdquo;.\nInput vertices could be interpreted as function inputs, and the framebuffer as function outputs.\nThat is how the concept of a \u0026ldquo;general‑purpose GPU (GPGPU)\u0026rdquo; took shape.\nHowever, as mentioned earlier, the hardware of that era was still built as a graphics pipeline.\nDoing GPGPU back then was effectively a \u0026ldquo;hack\u0026rdquo;:\nyou had to reinterpret your linear algebra problem as a graphics problem,\nfit it into the graphics pipeline, and rewrite existing C code in shader languages like HLSL or GLSL (OpenGL Shading Language).\nThis was powerful but very cumbersome.\nTesla (2006) and CUDA (Compute Unified Device Architecture, 2007) solved this problem.\nCUDA was designed not for graphics but for general‑purpose parallel computation,\nand it mapped naturally onto the unified shader architecture.\nConceptually:\nwork on a single data element becomes a thread a group of threads that run together on one SM becomes a block and all the threads launched by a kernel form a grid CUDA lets you express kernels in a C‑like language instead of in shader languages,\nand the Tesla architecture executes them efficiently. This is where the GPU truly became a GPGPU platform.\nNow let\u0026rsquo;s look more closely at the CUDA programming model on top of a modern GPU like Hopper.\nModern GPU Architecture GPU (Device) The full chip, including GPCs, memory controllers (HBM (High Bandwidth Memory)/GDDR (Graphics Double Data Rate)), L2 (Level 2) cache, PCIe (Peripheral Component Interconnect Express)/NVLink (NVIDIA NVLink) interfaces, etc. The GigaThread Engine at the top receives kernel launch requests and distributes thousands of thread blocks across GPCs and SMs. GPC (Graphics Processing Cluster) A higher‑level grouping of multiple SMs. Shares common resources for graphics (e.g., raster engines) and, in Hopper, forms the boundary for thread‑block clusters. Within a GPC, high‑bandwidth links connect SMs, enabling Distributed Shared Memory (DSMEM) access. SM (Streaming Multiprocessor) The main compute building block of the GPU – the place where thread blocks actually run.\nIt is analogous to a CPU core but designed to keep many more threads in flight. Key components: 4 SM sub-partitions (SMSPs): groups of execution units. Unified shared memory / L1 (Level 1) cache (256 KB on Hopper): fast on-chip storage for data sharing and caching. TMA (Tensor Memory Accelerator): an async copy engine introduced with Hopper to offload memory movement from compute units. A single SM can have tens of warps active at the same time.\nWhen one warp stalls on memory, the scheduler quickly swaps in another to keep the pipelines busy (latency hiding). SM Sub‑partition (Processing Block / SMSP) Each SM is split into 4 SMSPs in modern NVIDIA GPUs. Each SMSP has its own warp scheduler, dispatch unit, register file (64 KB), and a set of CUDA Cores and Tensor Cores. This partitioning makes the scheduling problem more tractable and allows each SMSP to independently issue work, improving parallelism. So how is work structured on top of this hardware? Let\u0026rsquo;s take a closer look at the CUDA programming model.\nCUDA Programming Model Structure On top of this hardware architecture, the GPU groups parallel work into five main levels:\nThread The smallest unit of parallel work A CUDA thread runs the kernel function on one piece of data.\nAll threads execute the same program (SPMD: Single Program, Multiple Data),\nbut each has its own thread ID and can follow slightly different control flow and access different memory.\nPhysically, threads execute by occupying CUDA Core (ALU, Arithmetic Logic Unit) pipelines.\nWarp The minimum unit of execution in hardware A warp is a group of 32 consecutive threads. The warp is the unit at which instructions are issued:\nall 32 threads in a warp execute the same instruction at the same time.\nEach warp has a program counter in the instruction cache. When the warp is issued,\nthe instruction at that PC is sent to the dispatch unit, executed, and then the PC advances.\nIf threads in a warp branch in different directions (branch divergence), the hardware serializes the different paths and reconverges later.\nTo get good performance, you want threads within a warp to follow the same control path.\nThe warp scheduler rapidly context‑switches between warps to hide memory latency.\nThread Block (CTA) The unit of cooperation and shared memory A block is a group of threads (up to 1024) that can cooperate closely.\nThreads in the same block share shared memory and can synchronize with __syncthreads().\nFor that reason, a block is always scheduled entirely on a single SM for its lifetime.\nRegister and shared memory limits on an SM constrain the maximum block size.\nBlocks can be 1D, 2D or 3D (blockDim.x/y/z), which maps naturally to images and volume data.\nThread Block Cluster Introduced in Hopper, a cluster is a group of blocks that can communicate more efficiently.\nTraditionally, blocks could communicate only via global memory, which is slow.\nIn a cluster, blocks mapped to the same GPC can access each other\u0026rsquo;s shared memory via DSMEM without going through L2 (Level 2),\nenabling faster inter‑block cooperation.\nGrid The whole kernel launch All the blocks created by one kernel launch together form a grid.\nBlocks in a grid are independent: their execution order is not guaranteed,\nand they may run in parallel across many SMs or sequentially on a smaller number of SMs.\nThis independence is what gives CUDA its scalability: the same kernel can run on a GPU with 10 SMs or 144 SMs without code changes.\nHow Does a GPU Actually Run Your Code? Now let’s walk through a concrete scheduling example to see what happens when we actually run code on the GPU.\nLet\u0026rsquo;s consider a simple fp32 (32-bit floating-point) add kernel.\nWe launch a grid with a single block of 96 threads.\nIn practice, the kernel is compiled down to machine code (SASS, Shader Assembly), but we can think of it as four logical steps:\nLoad operand A from memory (SMEM/Shared Memory, L1, L2, global) into register file (LD, Load) Load operand B into registers (LD) Perform the floating‑point add FADD (Floating-Point Add) R3, R1, R2 and store the result in a register Store the result back to memory (SMEM/L1/L2/global) (ST, Store) The load and store operations are asynchronous from the perspective of the compute pipelines.\nLD/ST units compute addresses and send memory requests to the memory subsystem,\nwhich completes them later.\nOur 96 threads are split into three warps (A, B, C) of 32 threads each, and all belong to one block.\nFor simplicity, assume that this block is mapped entirely onto a single SMSP.\nWarp Scheduling Hardware on Hopper Before we dive into the timeline diagrams, it helps to understand what the warp scheduler and dispatch unit actually do on Hopper.\nThe warp scheduler uses a scoreboard to track dependencies for each warp and picks eligible warps that are ready to run.\nWhen it does so, it must respect the capacity of the dispatch unit – on Hopper, the effective issue width is roughly 32 threads/clk –\nso warps that cannot be dispatched due to resource limits are deferred to later cycles.\nThe dispatch unit then maps the issued warp\u0026rsquo;s instruction onto the actual execution pipelines.\nEven though the warp scheduler issues work at warp granularity, the dispatch unit can spread the 32 threads across multiple cycles,\nand mix different pipeline types (LD/ST, INT (Integer), FP (Floating-Point), etc.) in the same cycle (co‑issue)\nto keep the hardware as full as possible.\nWith that in mind, let’s now follow how the first warp, Warp A, is scheduled step by step.\nSingle Warp Scheduling Instead of starting with the full three‑warp picture, let’s first follow Warp A in isolation.\nWarp A is issued by the warp scheduler (WS, Warp Scheduler). Based on its PC (Program Counter), the first instruction LD R1, [A+tid] is sent to the dispatch unit.\nThe dispatch unit scans the SMSP for available LD/ST units – suppose there are 8.\nIt then dispatches the 32 threads of Warp A across these 8 LD/ST units over 4 cycles (8 threads per cycle).\nEach LD/ST unit computes addresses and sends load requests to the memory subsystem;\nthese loads complete asynchronously.\nOn the fifth cycle, the LD/ST activity finishes and the WS releases Warp A, advancing its PC.\nOn the next cycle, the WS can issue Warp A again for the second LD, which proceeds in the same way.\nOnce both loads are issued, Warp A cannot immediately execute FADD R3, R1, R2,\nbecause R1 and R2 are still being filled by the asynchronous loads.\nThe warp must wait until both operands are ready. During this wait, the FP units would sit idle – a stall.\nMemory load times are roughly:\nL1 (Level 1) cache: ~20 cycles L2 (Level 2) cache: ~100 cycles HBM (High Bandwidth Memory): hundreds of cycles (300–800) When the loads complete and the dependencies are resolved, Warp A becomes eligible again.\nThe WS (Warp Scheduler) issues it; FADD is sent to the FP32 (32-bit Floating-Point) pipeline.\nOn Hopper, there are 32 FP32 units per SMSP, so all 32 threads in the warp can be processed in one go.\nAfter the FADD completes, the warp is released again, and the final ST instruction proceeds similarly to the LD.\nDouble Warp Scheduling Now add Warp B to the picture.\nIn the single‑warp scenario, when the first LD finished, there was a cycle where Warp A\u0026rsquo;s PC (Program Counter) had not yet advanced,\nso the scheduler could not immediately re‑issue Warp A, leaving one idle cycle.\nWith Warp B around, this idle slot can be used.\nWhile Warp A is using all LD/ST units, Warp B cannot run.\nBut as soon as A\u0026rsquo;s LD is fully dispatched, the WS (Warp Scheduler) can issue B and perform its LD in the same fashion.\nWe alternate between A and B, issuing four LD instructions in total (two per warp),\nand then both warps wait for their loads to complete before FADD.\nCompared to the single‑warp case, the key difference is that the memory latency of Warp A\u0026rsquo;s loads is overlapped with Warp B\u0026rsquo;s loads,\nreducing the visible stall time.\nWhen A\u0026rsquo;s loads finish, its FADD can run immediately.\nIf B\u0026rsquo;s loads also complete soon after, B\u0026rsquo;s FADD can follow, with the pipeline stages overlapping like a classic CPU pipeline,\neffectively hiding latency.\nMultiple Warp Scheduling Finally, add Warp C.\nWith three warps in flight, the GPU can overlap memory operations and computation even more effectively,\nfilling in most of the stalls that would otherwise occur.\nIn the end, CUDA programming on Hopper is all about packing enough independent work into the SMSP pipelines –\nissuing loads, arithmetic and stores from many warps so that asynchronous memory communication time is well overlapped.\nTo Sum Up… In this post we:\nLooked at NVIDIA’s history from a graphics‑card vendor to the dominant AI GPU provider, Traced how Tesla and CUDA turned the GPU into a GPGPU platform, and Examined the Hopper‑generation hardware and a scheduling example to see how it actually runs code. Taken together, these show that “a GPU is a machine that hides slow memory with many warps and aggressive concurrency.”\nThe memory hierarchy, scheduling policies and execution pipelines are all designed around that goal.\nIn the next post, we’ll turn our attention to Google’s TPU, which has been rapidly gaining share in the AI hardware market.\nWe’ll walk through the history of TPU and analyze the latest TPU architecture, Ironwood, from a hardware engineer’s perspective.\nSee you again in Know Your Enemy, Know Yourself, Part 2: TPU Emergence and Rise.\nP.S.: HyperAccel is Hiring! Knowing your enemy and yourself is important, but to truly win we also need great people on the team.\nIf you’re interested in the kinds of technologies we work on, please apply via HyperAccel Career.\nWe’re looking forward to hearing from talented engineers like you.\n","date":"2025.12.29","permalink":"https://hyperaccel.ai/techblog/en/posts/how-gpu-works/","series":["Know Your Enemy, Know Yourself"],"style":"--c:#00C2A8;--on:#02121c","summary":"An overview of how NVIDIA GPUs evolved and how the Hopper architecture hides memory latency with massive parallelism.","title":"[Know Your Enemy - Part 01] GPU History and Fundamentals"},{"authors":["Minho Park"],"category":"Engineering Culture","content":"The Motivation Behind Starting Our Tech Blog In our company\u0026rsquo;s SW group, we have a developer named Jaewoo Kim (Author, LinkedIn). Jaewoo is developing legato, a language for our HW kernel development. For several months, Jaewoo has been consistently requesting something from us.\nShouldn\u0026rsquo;t we have a company tech blog? (from Jaewoo)\nJaewoo mentioned that he had experience posting projects from his previous company on a tech blog, and thanks to those posts, many talented developers became interested in the company and even joined.\nI also had been thinking a lot about wanting to share with external developers the many challenges we face in building high-performance LLM Inference Chips and creating Software Stacks to support those Chips.\nBut\u0026hellip; who will build and operate it?\nI knew that we had plenty of content to post on the blog in our company\u0026rsquo;s Wiki (operated on Confluence). However, having a lot of content to post and actually building and operating a blog are completely different challenges.\nOpen source analysis for HyperAccel\u0026rsquo;s LLM Inference Engine support (Coming Soon\u0026hellip;)\nMonths had passed since Jaewoo suggested operating a tech blog, but no one had taken the initiative. That\u0026rsquo;s because no one had experience operating one!\nBut I actually had experience\u0026hellip; I had been posting on my personal blog for several years.\nHowever, I was pretending not to know, with a vague expectation that \u0026ldquo;someone else will step up, right?\u0026rdquo; and the thought that \u0026ldquo;I\u0026rsquo;m already too busy\u0026hellip;\u0026rdquo;\nBut in October of this year, I had the opportunity to attend PyTorch Conference with Jaewoo (see previous post).\nSeeing Jaewoo continuously discussing and developing the compiler with team members in Korean time even during his business trip to the US, I thought, \u0026ldquo;Is being busy just an excuse for me?\u0026rdquo;\nAnd there was something I always tell my team members as a team lead.\n\u0026ldquo;If you think you can do it, just do it\u0026rdquo; - that\u0026rsquo;s the mindset.\nSo I decided. I\u0026rsquo;ll do it\nChoosing a Tech Blog Platform When building a tech blog, there are several options.\nUsing blog platforms Velog Medium Tistory \u0026hellip; Self-hosting (using Static Site Generator + GitHub Pages) Jekyll Hugo \u0026hellip; Of course, the simplest approach would be using a blog platform. You can just sign up and create a blog.\nHowever, I had several important factors in mind for operating a tech blog.\nMust have comment functionality Must be able to integrate Google Analytics Must have good SEO Must be easy to maintain Multiple users should be able to easily submit their posts While other factors could be sufficiently supported by using blog platforms, ease of maintenance and allowing multiple users to easily submit their posts seemed difficult to achieve with blog platforms.\nWait, but shouldn\u0026rsquo;t using a blog platform make maintenance easier?\nYes. For personal blogs, using a blog platform would be much more convenient. However, there was a critical point we had to consider.\n\u0026ldquo;Our tech blog should be operated by an Editor Group, not by a single employee.\u0026rdquo;\nI really like the book Software Engineering at Google, and I apply many things I learned from it in team management. This book introduces the term Bus Factor.\nBus Factor: An index that represents how many team members working on a project can suddenly leave without proper handover procedures before the project is halted or faces a similarly serious situation.\nThe higher the Bus Factor, the more stable the project becomes, but if we operate the blog using a blog platform, if the blog operator (probably me?) falls into a situation where they cannot operate the blog, our tech blog will become a ghost blog.\nHowever, if we self-host using GitHub, while I would need to do most of the blog setup, the operation can be handled by the Editor Group.\nAnd so it began. Building a tech blog with Hugo\nBuilding the Tech Blog! My previous blog was built with Jekyll. However, Jekyll was difficult to modify themes, and being Ruby-based, it sometimes caused dependency issues. Since I had never used Ruby, when dependency problems occurred, I would just turn a blind eye and ignore them.\nBut when I looked into modern SSGs (Static Site Generators), I found Hugo, built with Go, which had diverse themes and an active community, so I started building the blog based on it.\nAnd we also recruited people who wanted to participate in the Editor Group.\nAs many as 9 applicants!\nAnd we completed implementing comment functionality, multilingual support (Korean, English), search functionality, Google Search registration, Google Analytics registration, Author functionality\u0026hellip; everything!\nOperating the Tech Blog If you want to write a post on our tech blog, you can do so through a GitHub Pull Request. Since it operates through GitHub PRs, it\u0026rsquo;s a system where anyone who wants to write can freely submit posts, not just a specific administrator.\nAnd, the Editor Group can freely leave comments with their opinions on posts in PRs.\nThe Editor Group has one very important role. That is to encourage developers to write tech blog posts.\nI once posted a blog post called BERT 논문정리 after reading the BERT paper around 2018. Thankfully, so many people read this post that it appeared at the top of Google search results for a while.\nSharing what you\u0026rsquo;ve studied and researched externally is good for promoting the company\u0026rsquo;s technical capabilities, but I believe it should also serve as personal promotion, so I encourage developers to actively blog post content from our internal wiki that can be shared externally.\nUpcoming Posts\u0026hellip; At HyperAccel, we develop everything from HW design for LLM Inference to the entire Software Stack for it.\nThat\u0026rsquo;s why we deal with a really wide range of technologies within one company. So the posts we\u0026rsquo;ll put on the tech blog will cover really diverse topics.\nHow to build a Compiler? GPU characteristics (Know your enemy and know yourself) Open source analysis of LLM Inference Frameworks (vLLM, SGLang, \u0026hellip;) Development of Kubernetes Components for Hardware support in Cluster environments Building an internal development environment based on Kubernetes \u0026hellip; We\u0026rsquo;re planning to share really diverse and high-quality posts, so please stay tuned!\nHyperAccel is Hiring! The biggest purpose of operating this tech blog is talent recruitment!\nIf you\u0026rsquo;re interested in the technologies we work with, please apply at HyperAccel Career!\nHyperAccel has many excellent and brilliant engineers. We\u0026rsquo;re waiting for your applications.\n","date":"2025.12.26","permalink":"https://hyperaccel.ai/techblog/en/posts/tech-blog-operation/","series":[],"style":"--c:#8EE8E0;--on:#02121c","summary":"The Motivation Behind Starting Our Tech Blog In our company\u0026rsquo;s SW group, we have a developer named Jaewoo Kim (Author, LinkedIn). Jaewoo is developing legato, a language for our HW kernel development. For several months, Jaewoo has been consistently requesting something from us.\nShouldn\u0026rsquo;t we have a company tech blog? (from Jaewoo)\nJaewoo mentioned that he had experience posting projects from his previous company on a tech blog, and thanks to those posts, many talented developers became interested in the company and even joined.\n","title":"Starting Our Tech Blog"},{"authors":["Jaewoo Kim"],"category":"Compilers \u0026 Kernels","content":"Before building a compiler, we need to define the language it will compile. A programming language isn’t just syntax and semantics—it’s the user interface between humans and computers.\nWhat Is a Programming Language? Abstraction is one of the core ideas in computer science. Without abstraction, interacting with computers would require understanding electrical signals, memory layout, registers, and countless hardware details.\nA programming language simplifies this complexity. It provides a human-friendly way to express ideas while hiding the low-level mechanisms that make them work. In this sense, a programming language functions as a UI for computing—a layer that lets us focus on building logic, applications, and systems, rather than manually manipulating hardware.\nWhatever language you choose—Python, Rust, C++, Go, or something entirely new—its purpose is the same: turn human intention into machine behavior. First, let\u0026rsquo;s see what options we\u0026rsquo;ve got for designin our own programming language.\nCompiled vs. Interpreted Programming language implementations are often categorized by how they execute code:\nCompiled languages Interpreted languages A compiled language translates the entire source code into machine instructions before execution. Once compiled, the program can run directly on the hardware, resulting in high performance and efficient CPU usage.\nAn interpreted language, on the other hand, executes code step-by-step. The interpreter reads each instruction and performs it immediately, without producing a standalone machine binary.\nA simple analogy:\nImagine you have a recipe book written in French.\nWith an interpreter, you translate each sentence as you read it and cook immediately. With a compiler, you translate the entire cookbook into your language first and then cook smoothly without stopping. Compiled languages (like C, C++, or Rust) typically offer speed, safety analyses, and optimization, but require recompilation when the code changes. Interpreted languages (like Python, Ruby, or Perl) prioritize convenience, interactivity, and flexibility—especially during early development—but generally run slower. This is why C is generally faster than Python. C can have REPL-like environments, but they are uncommon and not part of the language’s standard toolchain.\nLanguages can be compiled \u0026amp; interpreted at same time\nActually, being “compiled language” or “interpreted language” is not property of language itself, but it is more accurate to view this distinction as a property of the language implementation rather than the language itself.\nLanguages based on JVM (Java Virtual Machine) can be both “compiled” and “interpreted”. Meaning, they can be compiled into machine code on the fly at runtime using a Just-In-Time compiler, or they can be interpreted. Typically, Java (or any other JVM based language) programs are first compiled to something called “bytecode”. Which is instruction for JVM, but not hardware-specific. This bytecode can be deployed anywhere that runs JVM, and it is JVM’s choice to compile them down to target-specific code, or interpret bytecode.\nMoreover, Python is often described as an interpreted language, but in practice, its source code is first compiled into Python bytecode (.pyc files), which is then executed by the Python virtual machine. Some implementation like PyPy can compile Python down to machine code, just like JVMs do.\nManaged vs. Unmanaged Another useful distinction between languages is how they manage memory.\nPrograms need memory to run. They request memory from the operating system and should return it when finished. If they fail to do so, we get issues such as:\nSegmentation faults when accessing memory the program doesn’t own. Memory leaks when unused memory is never returned. Managed languages automate memory handling, often using garbage collection or runtime analysis. This reduces the chance of memory-related mistakes and simplifies development. They typically use program called “Garbage Collector (GC)”, which automatically detects unreachable memory regions and frees them, so operating system can allocate that memory for another program (process). Some garbage collectors may stop the entire program (stop-the-world) during certain phases, although modern collectors often reduce or avoid long pauses using concurrent or incremental techniques. Garbage collectors are convenient, but might make programs run slower.\nExample of managed programming language (Python)\n# my_list is internally allocated size = input() # Python internally allocates memory space for storing list. my_list = [x for x in range(0, int(size))] # Do some work using my_list # my_list is automatically deallocated after use Unmanaged languages, like C/C++ give developers direct control. They provide APIs such as malloc and free, which ultimately request memory from the operating system through lower-level system calls. Programmers are responsible for making sure memory is allocated and freed correctly. Since unmanaged languages do not rely on garbage collectors, they avoid GC-related pauses such as stop-the-world events, but this shifts the responsibility of correctness and safety to the programmer or the compiler. Also, they give more optimization headroom since programmers can hand-optimize when to allocate and free memory. But managing this makes programs more complicated, and easier to break. (Rust uses a static ownership and borrowing system enforced by the compiler to achieve memory safety without garbage collection.)\nExample of unmanaged programming language (C)\n#include \u0026lt;stdio.h\u0026gt; #include \u0026lt;stdlib.h\u0026gt; int main(){ int size; scanf(\u0026#34;%d\u0026#34;, \u0026amp;size); // Explicitly allocate aray int* my_list = malloc(size * sizeof(int)); for(int i = 0; i \u0026lt; size; ++i){ my_list[i] = i; } // Do some work with my_list // my_list has to be explicitly freed, or memory would leak. free(my_list); return 0; } Functional Programming Finally, I\u0026rsquo;d like to briefly introduce functional programming, another paradigm of programming languages. While less commonly used than imperative languages, it\u0026rsquo;s a programming paradigm with significant advantages, which is why I want to introduce it. Functional programming allows you to write programs that produce significantly fewer bugs. Even if you\u0026rsquo;re not actively doing functional programming, knowing about it as a programmer helps you build more stable programs.\nFunctional programming languages are languages that compose programs out of mathematical functions, emphasizing function purity, value immutability, and how functions are composed. Functional languages express computation as the evaluation of expressions. The functions we define in mathematics can never produce different results when given the same input, right? Functional programming follows this way of thinking.\nHere are simple examples of adding all numbers in a list.\nExample of adding all values in a list in Python\nmy_list = [1,2,3,4,5] sum = 0 for elem in my_list: sum += elem print(f\u0026#34;Sum : {sum}\u0026#34;) # Prints \u0026#34;Sum : 15\u0026#34; Example of functional language (F#)\nlet my_list = [1;2;3;4;5] let sum = List.fold (fun acc elem -\u0026gt; acc + elem) 0 my_list printfn \u0026#34;Sum : %A\u0026#34; sum // Prints \u0026#34;Sum : 15\u0026#34; They both do the same thing, but the way they are written is a bit different, right?\nPure functional languages have constraints that other languages don\u0026rsquo;t have.\nAll values (let bindings) are immutable No for loops (all iteration is done by recursion) However, the compiler can transform recursion to loops for optimization if the function is \u0026ldquo;tail-recursive\u0026rdquo; (we\u0026rsquo;ll explain more about this later) Multi-paradigm languages that support functional programming but aren\u0026rsquo;t purely functional may also provide for loops. Functions are \u0026ldquo;Pure\u0026rdquo; By \u0026ldquo;Pure\u0026rdquo;, we mean that functions don\u0026rsquo;t perform operations with side effects that the programmer isn\u0026rsquo;t aware of. Side effects refer to internal operations that aren\u0026rsquo;t explicitly visible, such as file I/O. Since there are no side effects, it\u0026rsquo;s safe to remove the function if its result is not used (Compilers can do aggressive dead code elimination) Functions always give the same result when given the same input Functional programming languages generally have a steeper learning curve, and many programmers aren\u0026rsquo;t familiar with functional languages. However, they come with some clear advantages.\nAdvantages of Functional Programming\nSince there are no side effects, programs are safer and easier to analyze for stability. Makes it easier for compilers to detect potential errors. Programs can be expressed more clearly. It focuses on \u0026ldquo;what to compute\u0026rdquo; rather than execution itself. (This might be a bit hard to grasp at first, but as you see implementations in functional languages, you\u0026rsquo;ll understand it more easily) Purity (pure functions) and immutability (value immutability) make it easier to write safe concurrent programs. Explaining all the reasons listed above is beyond the scope of this chapter, but as we continue building our programming language, you\u0026rsquo;ll understand why these problems are important and what advantages functional languages have compared to regular programming languages.\nOf course, functional languages are not a silver bullet. For certain tasks, especially those requiring complex side effects or where performance is critical, it may be more natural or efficient to not use a functional style.\nLet’s build a programming language! So, what kind of language are we going to build?\nWe will build a functional-style language (though we won\u0026rsquo;t make it 100% functional) and compiler that doesn\u0026rsquo;t use GC (meaning the compiler or user manages memory). However, we will implement ownership control (just like Rust!) so programmers don\u0026rsquo;t need to manually manage data lifetimes. Why are we doing this? It\u0026rsquo;s easier to focus on the compiler itself (we don\u0026rsquo;t have to build interpreters or garbage collectors!), and there are already many good articles about implementing interpreters, so here we want to focus more on compilers.\nWhy a functional language?\nFunctional languages are harder to learn for most people, and it\u0026rsquo;s a type of language that many programmers aren\u0026rsquo;t familiar with. So why build a functional language? First, from a language designer\u0026rsquo;s perspective, they\u0026rsquo;re easier and clearer to define. Second, because most programmers aren\u0026rsquo;t familiar with functional programming, I wanted to introduce it to you. We will see how it behaves differently from regular programming languages, and what advantages and disadvantages it has. However, even if you don\u0026rsquo;t build a functional language, the concepts explained here will be helpful.\nHoya Language. Our language is called \u0026ldquo;Hoya\u0026rdquo;. The rest of this article will focus on implementing Hoya. But before we start typing away at the keyboard, let\u0026rsquo;s first take a look at what Hoya looks like, shall we? Let\u0026rsquo;s define what we\u0026rsquo;re building before we start.\nHello world! Let\u0026rsquo;s make Hoya print \u0026ldquo;Hello world!\u0026rdquo;. Here is the simplest example of the Hoya language.\nfunc main(){ print(\u0026#34;Hello world!\u0026#34;) 0 } This program can be represented in a simple tree format.\nThis is what our program looks like (where the suffix “E” stands for “Expression”).\nThe CallE (call expression) consists of three children. The first one represents the callee (the function being called). The second one is the argument, “Hello world”, which represents a string. The last one is NextE, which represents the next expression to execute. In our case, the next expression is the constant zero.\nBy representing a program in a tree-like format, compilers or interpreters can traverse it and compile or execute it. This kind of tree is called an AST (short for Abstract Syntax Tree). Converting a program—which is just a string from the computer\u0026rsquo;s perspective—into an AST is typically the first process a compiler performs. Once the program is converted into an AST, the compiler can start translating it into machine code, or the interpreter can execute it directly.\nFrom the next chapter onwards, we will start designing all parts of Hoya. Let\u0026rsquo;s explore how a language can be defined and how we define its behavior as each expression is evaluated.\n","date":"2025.12.13","permalink":"https://hyperaccel.ai/techblog/en/posts/crafting-compilers-ch1-1/","series":["Crafting Compilers"],"style":"--c:#5BC5F6;--on:#02121c","summary":"Before building a compiler, we need to define the language it will compile. A programming language isn’t just syntax and semantics—it’s the user interface between humans and computers.\nWhat Is a Programming Language? Abstraction is one of the core ideas in computer science. Without abstraction, interacting with computers would require understanding electrical signals, memory layout, registers, and countless hardware details.\nA programming language simplifies this complexity. It provides a human-friendly way to express ideas while hiding the low-level mechanisms that make them work. In this sense, a programming language functions as a UI for computing—a layer that lets us focus on building logic, applications, and systems, rather than manually manipulating hardware.\n","title":"[Crafting Compilers - Part 02] Building a Programming Language"},{"authors":["Sinhyun Park"],"category":"Conference Reports","content":"NAVER DAN 2025 Report Introduction Hello! I\u0026rsquo;m Sinhyun Park from the HyperAccel ML team. I\u0026rsquo;d like to share my thoughts after attending NAVER DAN 2025, held at COEX on November 6-7, 2025.\nMotivation for Attending The main objectives for attending this conference were:\nGaining insights on scalable storage infrastructure: Beyond sharing data between nodes within a computing cluster, I wanted to acquire technical insights for building storage infrastructure that enables data exchange across different clusters.\nLearning efficient AI cluster resource utilization: At HyperAccel, we operate AI clusters composed of GPUs and FPGAs for AI workloads. To efficiently utilize resources within these AI clusters, I wanted to learn how other companies manage their AI clusters.\nNAVER DAN 2025 Key Highlights IDC Seamless HDFS I\u0026rsquo;d like to share the contents of the IDC Seamless HDFS session, which I attended to gain insights on distributed storage infrastructure. This session covered the limitations NAVER\u0026rsquo;s search platform faced while handling massive data and traffic in a multi-IDC environment, and the distributed storage technology they built to overcome these challenges.\nProblem: Data Silos and Data Recovery NAVER operates large-scale Hadoop clusters (C3) across multiple IDCs including Pyeongchon, Pangyo, and Sejong. Due to the physical distance between IDCs, they encountered the following limitations:\nData Silo Since clusters are configured independently for each IDC, accessing data in other IDCs requires going through the Data Center Interconnect (DCI). If the traffic exceeds the DCI bandwidth, it can cause outages for all other services using that connection, making it extremely risky. Data Recovery Since data is stored in isolation at specific IDCs, there\u0026rsquo;s a risk of permanent data loss if that IDC is completely destroyed by events like fires. Solution: IDC Seamless HDFS NAVER introduced a new architecture called IDC Seamless HDFS to solve these problems.\nLogical Single Namespace DataNodes located in different IDCs were logically unified into a single HDFS Namespace. From the user\u0026rsquo;s perspective, they can use it conveniently like local storage without worrying about which IDC the data is in. Even if one IDC experiences an outage, services and data access continue uninterrupted through other IDCs. Replica Placement Strategy (2+2 Structure) They set the default replica count to 4 and developed a new BlockPlacementPolicy to enforce placing 2 replicas in each IDC. Rack information was structured in 2 depths as IDC Code + Rack format so that the NameNode can recognize physical locations. Read Operation Optimization When clients read data, they are guided to preferentially read from replicas (one of the 2) within their own IDC. This ensures zero inter-IDC traffic during read operations, with no performance degradation. Write Operation Optimization Sync Write: Used when data safety is critical. Writes are performed simultaneously to both IDCs with immediate synchronization. Async Write: Used when performance is critical. Writes first to the local IDC, then replicates to the other IDC in the background. MLXP - Large-Scale MLOps Platform Leading GPU Efficiency Next, I\u0026rsquo;ll cover NAVER\u0026rsquo;s MLOps platform, MLXP. I\u0026rsquo;ve organized how NAVER, operating thousands of GPUs, maximizes resource efficiency and reliably handles large-scale training/serving workloads into 2 key parts.\nCompany-wide GPU Resource Integration and Quota System NAVER was operating GPU clusters using a Private Zone approach, allocating physical nodes to each organization.\nImbalances occurred where some teams had idle GPUs while others were waiting due to shortages. Additionally, resource fragmentation worsened as heterogeneous GPUs like A100 and H100 were mixed. They broke away from the physical monopoly structure and consolidated all GPUs into a single shared resource pool (Public Zone) to minimize idle resources. Priority Class: Beyond simple consolidation, they developed and applied a systematic quota system. Provisioning Type: Workloads are classified as Reserved (guaranteed) or Spot (low-cost/preemptible) based on their characteristics. Category \u0026amp; Purpose: Priorities are subdivided by purpose—Serving (high availability), Training (batch), Interactive (development)—to ensure scheduling fairness. Network Topology-Aware Scheduling Optimized for AI Workloads Kubernetes\u0026rsquo; default scheduler is not optimized for AI workloads (Batch, Gang Scheduling), causing performance degradation. To address this, they enhanced functionality based on the Volcano scheduler.\nGang Scheduling: Resources are allocated only when all Pods required for training are ready (All-or-Nothing), preventing deadlocks. In distributed training, inter-node communication speed determines overall training speed. However, the default scheduler places Pods randomly without knowledge of the physical network structure (InfiniBand Switch connections, etc.), causing severe performance degradation.\nHyperNode CRD: They implemented HyperNode CRD so that the scheduler recognizes physical network topology and groups Pods on nodes under the same switch for high-speed communication. HyperNode: A CRD (Custom Resource Definition) containing network topology information was defined for the scheduler to reference. Takeaways Technical Insights The most impressive part of this conference was IDC Seamless HDFS. Since HyperAccel\u0026rsquo;s development environment is also distributed across multiple clusters, building storage infrastructure that can share data regardless of physical location is an important challenge. NAVER\u0026rsquo;s 2+2 Replica placement strategy and Read/Write optimization methods will serve as good references for our future infrastructure design.\nThe MLXP session provided practical insights on operating large-scale AI clusters. In particular, network topology-aware scheduling considering InfiniBand is a key factor that directly affects distributed training performance. Running large-scale LLMs requires nodes equipped with our chips to be connected via high-speed networks, so random placement ignoring topology causes severe performance degradation.\nUltimately, making good AI chips alone is not enough. We reaffirmed that developing the SW stack including schedulers is essential to efficiently utilize our chips.\nReference IDC Seamless HDFS: Reaching the World of Data Integration Beyond IDC Limitations\nMLXP: Large-Scale MLOps Platform Leading GPU Efficiency\nP.S. HyperAccel Hiring HyperAccel is developing next-generation AI chips for LLM inference. We have completed FPGA verification and will soon release ASIC chips and server solutions. We are creating solutions that developers can easily use through integration with the PyTorch ecosystem.\nCareer Site: https://hyperaccel.career.greetinghr.com/ko/guide\nIf you\u0026rsquo;re interested, please feel free to contact us!\n","date":"2025.12.05","permalink":"https://hyperaccel.ai/techblog/en/posts/naverdan2025-report/","series":[],"style":"--c:#A7C4FF;--on:#02121c","summary":"NAVER DAN 2025 Report Introduction Hello! I\u0026rsquo;m Sinhyun Park from the HyperAccel ML team. I\u0026rsquo;d like to share my thoughts after attending NAVER DAN 2025, held at COEX on November 6-7, 2025.\nMotivation for Attending The main objectives for attending this conference were:\nGaining insights on scalable storage infrastructure: Beyond sharing data between nodes within a computing cluster, I wanted to acquire technical insights for building storage infrastructure that enables data exchange across different clusters.\n","title":"NAVER DAN 2025 Report"},{"authors":["Jaewoo Kim"],"category":"LLM \u0026 Models","content":"Philosophy of SGLang Since the introduction of LLMs, they have been used to solve complex tasks in various fields—including problem-solving, code writing, and answering questions. Today, they are expanding their ability to be agentic, completing tasks users request without human interference.\nThis requires a wide range of prompting techniques, such as skeleton of thought or tree of thought. In other words, we structure LLMs to follow certain patterns to fit our needs, and we require programmability to control and guide them to meet our requirements.\nThe authors argue this process has been inefficient in current solutions for the following reasons:\nProgramming LM(Language Model) programs is difficult since LLMs are non-deterministic. We cannot predict what LLMs will emit before running a command. This reduces readability of LLM outputs. Executing LM programs is inefficient due to redundant computation and memory usage. Current solution (Solutions like vLLM or TGI at the time when this paper was written) lack effective mechanisms to reuse KV-cache. When output format is fixed or follows certain grammar rules, current solutions cannot effectively leverage this, since they always emit tokens one at a time. The core idea of the SGLang is to simplify LM programs into structure using new programming language (in Python-embedded DSL) which can be compiled.\nSGLang DSL Here\u0026rsquo;s a DSL example from the paper.\nLet\u0026rsquo;s say we\u0026rsquo;re building an AI application that evaluates essays about images.\nWe want our system to:\nRead the essay and image, then judge whether the essay is about the image. If not, terminate. If the essay matches the image, evaluate it across multiple dimensions (such as clarity, grammar, readability, and structure). Based on these evaluations, assign a grade from A+ to D-. Output the grade and summary in JSON format. This process can be described in SGLang DSL as follows.\n@function def multi_dimensional_judge(s, path, essay): s += system(\u0026#34;Evaluate an essay about an image.\u0026#34;) s += user(image(path) + \u0026#34;Essay:\u0026#34; + essay) s += assistant(\u0026#34;Sure!\u0026#34;) # Return directly if it is not related s += user(\u0026#34;Is the essay related to the image?\u0026#34;) s += assistant(select(\u0026#34;related\u0026#34;, choices=[\u0026#34;yes\u0026#34;, \u0026#34;no\u0026#34;])) if s[\u0026#34;related\u0026#34;] == \u0026#34;no\u0026#34;: return # Judge multiple dimensions in parallel forks = s.fork(len(dimensions)) for f, dim in zip(forks, dimensions): f += user(\u0026#34;Evaluate based on the following dimension:\u0026#34; + dim + \u0026#34;. End your judgment with the word \u0026#39;END\u0026#39;\u0026#34;) f += assistant(\u0026#34;Judgment:\u0026#34; + gen(\u0026#34;judgment\u0026#34;, stop=\u0026#34;END\u0026#34;)) # Fetch the judgements results \u0026amp; merge the judgments judgment = \u0026#34;\\n\u0026#34;.join(f[\u0026#34;judgment\u0026#34;] for f in forks) # Generate a summary and a grade. Return in the JSON format. s += user(\u0026#34;Provide the judgment, summary, and a letter grade\u0026#34;) s += assistant(judgment + \u0026#34;In summary,\u0026#34; + gen(\u0026#34;summary\u0026#34;, stop=\u0026#34;.\u0026#34;) + \u0026#34;The grade of it is\u0026#34; + gen(\u0026#34;grade\u0026#34;)) schema = r\u0026#39;\\{\u0026#34;summary\u0026#34;: \u0026#34;[\\w\\d\\s]+\\.\u0026#34;, \u0026#34;grade\u0026#34;: \u0026#34;[ABCD][+-]?\u0026#34;\\}\u0026#39; s += user(\u0026#34;Return in the JSON format.\u0026#34;) # Runtime optimize for fast constrained decoding with regex s += assistant(gen(\u0026#34;output\u0026#34;, regex=schema)) state = multi_dimensional_judge.run(...) print(state[\u0026#34;output\u0026#34;]) # Runs an SGLang program With SGLang, this can be written with simple expressions, which is much shorter than previous solutions.\nIn the program, we can observe SGLang language primitives such as fork, join, regex ,etc.\nfork will generate multiple requests to judge the essay in multiple dimensions, (it is going to judge the essay in different fields) while join will merge all judgements. regex accepts regular expressions, and it can constrain output of the LLM to be in specific format. += operator is used to append the string.\nSome primitives will instruct SGLang execute in asynchronous stream, such as select, gen or extend. Authors say we can view this as launching CUDA kernels with CUDA streams asynchronously.\nRadixAttention SGLang introduces a technique for managing KV cache called RadixAttention. This algorithm uses a radix tree to efficiently manage KV caches in the context of SGLang DSL.\n(Note: A radix tree is a tree where a node that is an only child is merged with its parent node, saving space.)\nThe algorithm is simple. It maintains prompts as a tree that branches when a different prompt or answer is given. It evicts leaf nodes using an LRU (Least Recently Used) policy when no running requests are using them.\nHere is an example of a radix tree, which can express multiple possible contexts. It naturally represents multiple states of the prompts and reuses all previous KV caches when a new request matches stored nodes in the tree.\nDecision Process\nTree is empty Request \u0026ldquo;Hello\u0026rdquo; is accepted, and LLM answers \u0026ldquo;Hi\u0026rdquo;. This sequence is stored as new node a. New request arrives (\u0026ldquo;Solve this problem\u0026rdquo;), and the conversation matches. We can safely reuse the KV cache. Another request arrives. The request matches up to \u0026ldquo;You are a helpful assistant\u0026rdquo; (the system prompt), but we must branch out the tree since the user did not say \u0026ldquo;hello\u0026rdquo;. Due to memory limits, we evict one least-recently-used node. New request arrives, but it did not use the previous system prompt. Therefore, we branch out from the root. We have more queries coming in, but they share the same prompts as the previous request without a system prompt, so we branch out from the last node. We receive a new message after \u0026ldquo;Hello\u0026rdquo; and \u0026ldquo;Hi!\u0026rdquo;, so we append a new node. However, due to lack of KV cache space, we evict other least-recently-used nodes. Now, we have more requests sharing the same prompts up to \u0026ldquo;Answer 3\u0026rdquo;, but we must evict other nodes until we have enough space for them. We evict them using LRU policy. If you\u0026rsquo;ve read this far, you might wonder: what if cache performance goes down due to suboptimal request ordering? You\u0026rsquo;re very sharp—the authors thought of this too. They designed a cache-aware scheduling algorithm to maximize performance. It sorts requests by the prefix length that best matches the radix-tree KV cache (essentially pre-evaluating cache hit rate before processing incoming requests).\nConstrained decoding with compressed FSM (Finite state machine) Some LM programs require output in a specific format (such as JSON in our example). If we\u0026rsquo;re building an AI program for generating presentation slides, we want the LLM output to follow a specific format. SGLang supports this efficiently using FSMs.\nDuring decoding, the correct LLM output is sometimes always fixed. For example, if an LLM is prompted to output JSON, we always know when to output colons, commas, or braces—which can be expressed as regex in SGLang DSL.\nExample: In the figure above, we always know we must emit {\\n \u0026quot;name\u0026quot;: } before we emit \u0026ldquo;Harry\u0026rdquo;.\nThe problem with regex is that it\u0026rsquo;s defined at the character and string level, which doesn\u0026rsquo;t align with LLM tokens. Therefore, we must make the LLM emit small tokens one at a time.\nExample: The LLM decodes {, \\n, \u0026quot; one-by-one, which is very inefficient.\nHowever, SGLang can re-tokenize these sequences so that fixed consecutive expressions are \u0026ldquo;compressed\u0026rdquo; into a single expression. The FSM tracks which compressed token to output, allowing SGLang to \u0026ldquo;jump forward\u0026rdquo; over multiple expressions and prevent the LLM from inferencing them one by one—accelerating the decoding process.\nIn the example above, { \\n ____ \u0026quot;name\u0026quot; is merged into a single compressed expression. We can skip decoding that part entirely and simply paste it in, allowing the LLM to proceed directly to the next output.\nEvaluations SGLang is very effective in agent-based AI workloads due to inference context-aware KV-cache mechanisms, which will be most of the inference workloads in the future.\nPerformance speedup comes from Radix attention, FSM based decoding. Authors say cache hit rate was between 50-90% in DsPy RAG pipeline that showed best performance.\nMy personal thoughts We are living in a world of agentic AI, and agentic AI workloads are expanding. AI workloads are integrated into various services, including coding, creating slides, creating music, and work tools. These workloads require many system prompts and output constraints to control the model. I personally think SGLang will be very efficient in these cases since they fit really well with radix attention. Also, I liked the idea of saving time and energy using FSMs with regex when tokens don\u0026rsquo;t need to be generated. While SGLang won\u0026rsquo;t always be better than vLLM, it would be much more suitable for agentic AI workloads.\nReference SGLang: Efficient Execution of Structured Language Model Programs\n","date":"2025.11.29","permalink":"https://hyperaccel.ai/techblog/en/posts/sglang-review/","series":[],"style":"--c:#003A70;--on:#ffffff","summary":"Philosophy of SGLang Since the introduction of LLMs, they have been used to solve complex tasks in various fields—including problem-solving, code writing, and answering questions. Today, they are expanding their ability to be agentic, completing tasks users request without human interference.\nThis requires a wide range of prompting techniques, such as skeleton of thought or tree of thought. In other words, we structure LLMs to follow certain patterns to fit our needs, and we require programmability to control and guide them to meet our requirements.\n","title":"SGLang paper review"},{"authors":["Jaewoo Kim"],"category":"Compilers \u0026 Kernels","content":"Compilers This will be a series of posts describing what compilers are, how they are crafted, and how to build your own compiler. Compilers are complicated (sort of) programs that translate high-level programs (written in English mostly) into binary formats that computers can understand. Engineering compilers means designing the way that programs are translated.\nLet\u0026rsquo;s first think about what \u0026ldquo;programs\u0026rdquo; are, and how they look. I would define a program as a sequence of instructions for hardware to execute. So at the base level, programs are just sets of instructions composed of \u0026ldquo;1\u0026quot;s and \u0026ldquo;0\u0026quot;s. One of the original forms of programs was the assembly language used for computers such as the IBM 360. Programmers in the old days wrote hardware instructions directly to make computers run mission-critical programs such as calculating the trajectory of space rockets or managing bank accounts. These directly expressed what the computer should do at each step, and programmers had to be aware of all the hardware details the computer had. They had to manually calculate register usage, memory state, and every hardware detail, or their program would malfunction and blow up their spaceships.\nHowever, as people started to rely more on computers, the complexity and range of tasks grew rapidly. The size of programs became massive, making it almost impossible to write them in assembly. Now, computers run almost everything we use today—from the elevator you take to work to AI applications that even surpass human abilities.\nSince programs became too complicated, we could no longer give manual instructions to hardware directly. It is simply infeasible for most modern applications. Instead, computer scientists developed programming languages that humans can easily understand by providing necessary abstractions. A programming language can be thought of as a user interface (UI) that helps humans write programs by providing abstractions. They are designed to be closer to human language and aligned with how people usually think. As a result, today\u0026rsquo;s programmers can focus on writing programs themselves rather than thinking about the precise internal state of computers, such as register usage, memory usage, and so on. Modern programming languages tend to become languages for humans.\nCompilers are the bridge that connects human-written programs to machine-level programs (series of instructions). Good compilers translate a programmer\u0026rsquo;s intention without errors or side effects, and make the program execute on computers as fast and efficiently as possible. As I mentioned before, the programming languages we use every day (such as C/C++, Java, Python, etc.) are abstracted. They hide hardware details that they don\u0026rsquo;t want to expose to programmers. In other words, compilers must take care of them. Abstracted expressions include things like register allocation, memory allocation, and instruction selection. These are decisions made by compilers without asking humans—similar to an English–Japanese translator who chooses the correct Japanese words that best express English expressions.\nSo these are what compilers do. What I am going to tell you from now on is how compilers actually perform these tasks in a step-by-step manner—from English-written programs to executable binaries! I am going to explain some theoretical stuff, but I\u0026rsquo;ll lean toward real-world implementation. I\u0026rsquo;m going to use popular compiler infrastructures such as LLVM and the MLIR compiler stack, and show you how to build a decent programming language from scratch!\nTable of Contents Chapter 1. Programming Language 1.1 Building a Programming Language What is a programming language? Compiled vs. Interpreted Managed vs. Unmanaged Functional vs. Imperative Introducing the Hoya Language Chapter 2. Parsers Abstract syntax tree What is an abstract syntax tree Optimizing ASTs Lexers, parsers Parsing algorithms Recursive-descent parsing Pratt parsing LR parsing Chapter 3. Intermediate Representations What is an intermediate representation, and why it matters SSA (Single Static Assignment) Blocks and control flow Chapter 4. Optimizations Scalar optimizations Control-flow optimizations Vectorization Loop analysis Chapter 5. Instruction Selection Choosing the right instruction Register allocation Chapter 6. Compilers for Accelerators GPU compilers Special compilers for AI accelerators ","date":"2025.11.16","permalink":"https://hyperaccel.ai/techblog/en/posts/crafting-compilers/","series":["Crafting Compilers"],"style":"--c:#5BC5F6;--on:#02121c","summary":"Compilers This will be a series of posts describing what compilers are, how they are crafted, and how to build your own compiler. Compilers are complicated (sort of) programs that translate high-level programs (written in English mostly) into binary formats that computers can understand. Engineering compilers means designing the way that programs are translated.\nLet\u0026rsquo;s first think about what \u0026ldquo;programs\u0026rdquo; are, and how they look. I would define a program as a sequence of instructions for hardware to execute. So at the base level, programs are just sets of instructions composed of \u0026ldquo;1\u0026quot;s and \u0026ldquo;0\u0026quot;s. One of the original forms of programs was the assembly language used for computers such as the IBM 360. Programmers in the old days wrote hardware instructions directly to make computers run mission-critical programs such as calculating the trajectory of space rockets or managing bank accounts. These directly expressed what the computer should do at each step, and programmers had to be aware of all the hardware details the computer had. They had to manually calculate register usage, memory state, and every hardware detail, or their program would malfunction and blow up their spaceships.\n","title":"[Crafting Compilers - Part 01] Series Introduction"},{"authors":["Minho Park"],"category":"Conference Reports","content":"PyTorch Conference 2025 Report Introduction Hello, I\u0026rsquo;m Minho Park, Lead of the ML Team at HyperAccel. I attended the PyTorch Conference 2025 held in San Francisco on October 22-23, and I\u0026rsquo;d like to share my observations and insights.\nAbout HyperAccel HyperAccel is a company that designs AI chips for efficient inference of Large Language Models (LLMs). We have completed IP verification with FPGA, and our ASIC chip and server are scheduled for release next year. For ASIC, we plan to provide an SDK that supports PyTorch for inference, along with vLLM and Kubernetes support.\nMotivation for Attendance The main objectives of attending this conference were:\nGaining Insights for Torch Hardware Backend Implementation: We wanted to understand PyTorch\u0026rsquo;s hardware backend implementation methods and latest trends to gain technical insights for developing an optimized backend for our chip.\nPyTorch Foundation Membership Consideration: HyperAccel is considering joining the Foundation membership to contribute to the PyTorch ecosystem and collaborate with the community. At PyTorch Conference 2025, we met with Meredith Roach, Membership Solutions at Linux Foundation, to discuss PyTorch membership.\nLearning from Other Hardware Vendors: We wanted to gather information on how other hardware vendors such as Google TPU, AMD, Rebellions, and Furiosa AI approach and integrate with PyTorch, vLLM, SGLang, etc.\nKey Highlights of PyTorch Conference 2025 Scalable Scalability solutions for large-scale model training and inference received significant attention. Previously, third-party software was often required for large-scale distributed training/inference (Megatron, DeepSpeed, Horovod, \u0026hellip;). However, at this PyTorch Conference, PyTorch Organization introduced its own distributed training frameworks, communication libraries, and abstraction concepts.\nMonarch Monarch is a Distributed Programming Framework for PyTorch, designed to support efficient distributed training and inference of large-scale models. Looking at the API, it appears to be heavily influenced by Ray\u0026rsquo;s Actor and Task concepts. It provides automated partitioning and communication optimization without the need to manually configure complex parallelization strategies. This significantly reduces code complexity and communication overhead compared to existing distributed training frameworks. DTensor: Simplifies model parallelization across multiple devices. It\u0026rsquo;s an abstraction layer that enables efficient execution of large-scale models without complex distributed setup. TorchComms A new API to support PyTorch\u0026rsquo;s next-generation distributed communication. The existing PyTorch Distributed\u0026rsquo;s c10d::Backend has significant technical debt, making modernization and extension difficult, which created challenges for adding new hardware. By separating the communication layer from PyTorch core, new collective communication or backends can be implemented more quickly and independently. The ease of implementing new backends is extremely important for us at HyperAccel. For LLM Inference, implementing Tensor Parallelism is essential to reduce latency, and at this point, implementing a communication backend for our chip is necessary. We believe this can be implemented independently through TorchComms. Ray A computing framework for distributed Python processing that enables easy scaling of large-scale ML workloads through integration with PyTorch. Python functions are treated as Tasks and classes as Actors, which can be executed anywhere within a Ray Cluster. Ray operates as a Distributed Compute Engine on top of Kubernetes, a Container Orchestrator, and helps efficiently distribute workloads provided by PyTorch-based training \u0026amp; inference frameworks such as vLLM, SGLang, and Megatron. Even faster for advanced users Performance optimization tools for advanced users were introduced.\nHelion PyTorch\u0026rsquo;s eDSL (Embedded Domain-Specific Language) that maximizes runtime performance through compile-time optimization. It enables more efficient execution of complex computation graphs. To run Transformers quickly, various companies use an enormous amount of Custom Kernels, but supporting Custom Kernels on new hardware is difficult, which can quickly become technical debt. It\u0026rsquo;s a Higher Level DSL that automates manual tasks when writing Triton kernels. The actual result of Helion DSL becomes a Triton Kernel, and by adding a Triton Kernel Backend, various hardware support becomes possible. It appears to be an essential Abstraction Layer that can support various hardware agnostically. From a PyTorch user\u0026rsquo;s perspective, it could be a means to write more efficient kernels. CuTile As GPU Architecture evolves, various features have been added (e.g., TMA in Hopper, TensorCore from Turing Architecture). When implementing through GPU Kernel PTX, there\u0026rsquo;s a problem where existing kernels have difficulty supporting new hardware specs when they are added. Additionally, PTX-level implementation requires users to manually divide work into blocks, divide data into tiles, and even map to threads directly. Such abstraction leads to extensive code writing. In the case of Tile IR, the system performs the work of mapping work to threads on behalf of the user. This Tile Level abstraction was introduced in Triton-lang, and Helion mentioned above also supports DSL through Tile Level abstraction. By supporting Tile Level DSL and IR in CUDA as well, an appropriate abstraction layer has been established between the User and System. Runs Everywhere Projects were announced to enable PyTorch to run on various platforms and devices.\nPrivateUse1 and torch.accelerator torch.accelerator A unified accelerator abstraction API that enables using various hardware backends through a consistent interface. Previously, torch.cuda.* was frequently used in PyTorch code under the assumption that only NVIDIA GPU would be used. However, such code reduces hardware portability. Now, PyTorch supports torch.accelerator, a Device generic API, enabling hardware agnostic code writing. PrivateUse1 An extensible interface for custom hardware backends. It\u0026rsquo;s an important feature that enables AI chip vendors like us to port PyTorch to their hardware. PyTorch support is possible through an Out-Of-Tree plugin format without needing our backend to be included in the PyTorch main branch. (PyTorch support method of Rebellions, Huawei Ascend) Executorch: A lightweight PyTorch runtime for edge devices that enables efficient execution of PyTorch models on mobile and IoT devices. LLM Inference Engine There were many sessions on LLM inference engines, and real-world industry application cases were shared.\nvLLM: vLLM is becoming increasingly integrated with PyTorch, and vLLM plays a significant role in improving PyTorch\u0026rsquo;s features, while vLLM is being further accelerated by PyTorch\u0026rsquo;s new features. vLLM serves as a Hub that can support new models and hardware as an LLM Inference Framework. At PyTorch Conference 2025, it was one of the most important projects, with 7 related presentations in the keynote and standalone sessions. At HyperAccel, supporting vLLM\u0026rsquo;s features to the maximum extent is the Software Group\u0026rsquo;s biggest goal. SGLang SGLang is also an LLM Inference Framework widely used in the industry, similar to vLLM. It\u0026rsquo;s a framework being actively adopted by xAI Grok, AMD, TogetherAI, etc. Plugin functionality to attach non-GPU hardware as a backend was not yet supported, but at this Conference, plans were shared to support various hardware through multi-platform abstraction refactoring. Industry Application Cases Spotify presented a case study of applying LLM to a large-scale music recommendation system by combining vLLM with Google TPU. Performance optimization and operational experiences in actual production environments were shared, which was very useful. In Prefill Heavy situations, TPU was significantly better than L4, A100 in terms of Performance and Cost Efficiency. System Level LLM Inference There were presentations on LLM inference optimization at the system level.\nllm-d llm-d is a Kubernetes-native distributed inference serving stack that provides a Production Ready solution for efficiently serving large-scale generative AI models. It\u0026rsquo;s configured by integrating vLLM as the base model server, Inference Gateway as a request scheduler and balancer, and Kubernetes as an infrastructure orchestrator. Key features: Intelligent Inference Scheduler: Provides P/D separation, KV cache, SLA, and load-aware routing through smart load balancing based on Envoy proxy. Disaggregated Serving: Separates Prefill and Decode stages into independent instances to optimize GPU utilization and improve TTFT (Time To First Token) and TPOT (Time Per Output Token). Disaggregated Prefix Caching: Provides a Pluggable KV Cache layer that off-loads KV cache to host memory, remote storage, LMCache, etc. Variant Autoscaling: An autoscaler that recognizes traffic and hardware, calculating optimal instance combinations for Prefill, Decode, and latency-tolerant requests. Supports various hardware accelerators (NVIDIA GPU, AMD GPU, Google TPU, Intel XPU) and provides production-validated Helm charts and guides. Dynamo Dynamo is NVIDIA\u0026rsquo;s high-throughput, low-latency inference framework, designed to serve generative AI and inference models in multi-node distributed environments. It\u0026rsquo;s inference engine agnostic, supporting TRT-LLM, vLLM, SGLang, etc., and is implemented in Rust and Python, considering both performance and scalability. Key features: Disaggregated Prefill \u0026amp; Decode Inference: Separates Prefill and Decode to maximize GPU throughput and balance throughput and latency. Dynamic GPU Scheduling: Provides performance optimization based on real-time demand. LLM-aware Request Routing: Routes requests to workers with the highest KV cache hit rate, eliminating unnecessary KV cache recomputation. Accelerated Data Transfer: Reduces inference response time using NIXL (NVIDIA Inference Transfer Library). KV Cache Offloading: Utilizes multiple memory tiers such as HBM, DDR, NVMe, or remote storage to increase system throughput and reduce latency. Post-Conference Reflections Technical Insights The most impressive part of this conference was the PrivateUse1 and torch.accelerator APIs. These extensible interfaces are essential for porting PyTorch to the AI chip we\u0026rsquo;re developing. It was particularly useful to understand the details of implementing custom hardware backends through PrivateUse1.\nI gained a lot of inspiration especially from the vLLM sessions. I was able to see how closely vLLM is integrated with the PyTorch ecosystem and how it serves as a Hub for supporting new hardware. This made me realize that it\u0026rsquo;s important not just to implement a PyTorch backend, but to systematically structure the entire software stack. The key challenge is to properly structure the overall SW stack, considering not only integration with high-level inference engines (vLLM, SGLang, etc.) but also system-level optimization, so that users can easily use our chip.\nCommunity Engagement Through meetings with PyTorch Foundation officials, we were able to discuss membership procedures and contribution methods. We are positively considering joining the PyTorch Foundation to participate in the Torch ecosystem together. I was also able to see the technical capabilities of other hardware vendors such as AMD, Rebellions, and Furiosa AI and gained a lot of inspiration.\nFuture Plans Based on the insights gained from the conference, we plan to proceed with the following tasks:\nPrivateUse1 Backend Implementation: Implement a PyTorch backend for our chip through the PrivateUse1 interface Complete SW Stack Configuration: Systematic software stack configuration including vLLM/SGLang integration PyTorch Foundation Membership Review: Review official membership to participate in the Torch ecosystem Open Source Contribution: Share our hardware backend implementation experience with the community Postscript HyperAccel Recruitment HyperAccel is developing next-generation AI chips for LLM inference. We have completed FPGA verification and will soon release ASIC chips and server solutions. We are creating solutions that developers can easily use through integration with the PyTorch ecosystem.\nCareer Site: https://hyperaccel.career.greetinghr.com/en/guide\nIf you\u0026rsquo;re interested, please feel free to contact us!\n","date":"2025.11.10","permalink":"https://hyperaccel.ai/techblog/en/posts/pytorchcon2025-report/","series":[],"style":"--c:#A7C4FF;--on:#02121c","summary":"PyTorch Conference 2025 Report Introduction Hello, I\u0026rsquo;m Minho Park, Lead of the ML Team at HyperAccel. I attended the PyTorch Conference 2025 held in San Francisco on October 22-23, and I\u0026rsquo;d like to share my observations and insights.\nAbout HyperAccel HyperAccel is a company that designs AI chips for efficient inference of Large Language Models (LLMs). We have completed IP verification with FPGA, and our ASIC chip and server are scheduled for release next year. For ASIC, we plan to provide an SDK that supports PyTorch for inference, along with vLLM and Kubernetes support.\n","title":"Pytorchcon2025 Report"},{"authors":[],"category":"","content":"HyperAccel ","date":"2025.10.30","permalink":"https://hyperaccel.ai/techblog/en/about/","series":[],"style":"--c:#009EE2;--on:#02121c","summary":"HyperAccel","title":"About"}]