{"id":5040,"date":"2026-10-26T17:18:53","date_gmt":"2026-10-27T00:18:53","guid":{"rendered":"https:\/\/www.flamingspork.com\/blog\/?p=5040"},"modified":"2026-10-26T17:18:53","modified_gmt":"2026-10-27T00:18:53","slug":"zig-self-hosted-backend-lowers-a-seq_cst-atomicrmw-xchg-with-a-discarded-result-to-a-plain-mov","status":"publish","type":"post","link":"https:\/\/www.flamingspork.com\/blog\/2026\/10\/26\/zig-self-hosted-backend-lowers-a-seq_cst-atomicrmw-xchg-with-a-discarded-result-to-a-plain-mov\/","title":{"rendered":"Zig self-hosted backend lowers a seq_cst @atomicRmw(.Xchg) with a discarded result to a plain mov"},"content":{"rendered":"\n

Finding codegen bugs for atomic operations is fun! I’ll note that I am not a Zig developer. I don’t write code in the language, and I don’t work on the language. I’m just some guy who decided to learn a bit about it.<\/p>\n\n\n\n

The Zig programming language has two compilers that are widely used: one generates LLVM IR that libLLVM then builds into native code, and the other is a self-hosted compiler which eschews LLVM and does its own codegen.<\/p>\n\n\n\n

It’s important to note that the self-hosted compiler is only currently (as of late October 2026) used on x86_64 Debug targets. All other targets use the LLVM backend by default.<\/p>\n\n\n\n

_ = @atomicRmw(u32, p, .Xchg, 1, .seq_cst);<\/code> is a sequentially consistent exchange. On x86 that is xchg<\/code> with a memory operand, which is implicitly locked and is a full barrier. When the result is discarded, Zig’s self-hosted x86_64 backend emits a plain mov<\/code> store instead. A plain store on x86 lets a later load be reordered before it (StoreLoad), so the operation loses its seq_cst ordering. Any build that uses the self-hosted x86_64 backend is affected. That backend is the default for -O Debug<\/code> on x86_64-linux, and -fno-llvm<\/code> selects it explicitly.<\/p>\n\n\n\n

The same backend emits xchg<\/code> when the result is used. So does a lock acquired by testing the exchange’s return value, which is not affected. Zig’s LLVM backend and gcc emit xchg<\/code> in both cases.<\/p>\n\n\n\n

This affects softwhare when: code discards the exchange’s result, and then relies on access of a different variable to be executed by the processor after that xchg. An example could be code sets its own flag, then reads another thread’s flag.<\/p>\n\n\n\n

Affected:<\/strong> Zig 0.16.0 (Fedora 44 package), 0.17.0 and (current at the time) master 0.18.0-dev.367+187d0bf9e<\/code> (commit 187d0bf9e483<\/code>, 2026-10-19), all with the self-hosted x86_64 backend (the default for -O Debug -target x86_64-linux<\/code>). The LLVM backend (-fllvm<\/code>) is not affected.<\/p>\n\n\n\n

Reproducing the codegen bug<\/h2>\n\n\n\n

In order to see the codegen bug, you can build a small source file and disassemble it:<\/p>\n\n\n\n

export fn xchg_unused(p: *u32) void {\n    _ = @atomicRmw(u32, p, .Xchg, 1, .seq_cst);    \/\/ BUG: result discarded -> plain mov\n}\nexport fn xchg_used(p: *u32) u32 {                 \/\/ control\n    return @atomicRmw(u32, p, .Xchg, 1, .seq_cst);\n}<\/code><\/pre>\n\n\n\n
We can then compare the result with some C code built with a variety of different toolchains, but gcc -O0<\/code> gives us the closest result to the self-hosted Zig compiler’s output.<\/p>\n\n\n\n
void xchg_unused(unsigned *p) { (void)__atomic_exchange_n(p, 1, __ATOMIC_SEQ_CST); }<\/code><\/pre>\n\n\n\n
The (annotated) gcc output for the C code looks like this:<\/p>\n\n\n\n
+ x86_64-linux-gnu-gcc -O0 -c \/repro\/src\/xchg.c -o \/tmp\/xchg\/gcc-O0\/xchg.o\n+ cd \/tmp\/xchg\/gcc-O0 && x86_64-linux-gnu-objdump -d --no-addresses --disassemble=xchg_unused xchg.o > xchg_unused.objdump\n<xchg_unused>:\n\t55                   \tpush   %rbp\n\t48 89 e5             \tmov    %rsp,%rbp\n\t48 89 7d f8          \tmov    %rdi,-0x8(%rbp)\n\t48 8b 45 f8          \tmov    -0x8(%rbp),%rax\n\tba 01 00 00 00       \tmov    $0x1,%edx\n\t87 10                \txchg   %edx,(%rax)      # opcode 87: xchg, implicitly locked: correct\n\t90                   \tnop\n\t5d                   \tpop    %rbp\n\tc3                   \tret\nRESULT[gcc -O0 (16.2.1)]: OK, 1 locked\/fencing instruction(s)<\/code><\/pre>\n\n\n\n
While we can see the difference in the code generated by the Zig compiler below, annotated as to where the bug is:<\/p>\n\n\n\n
+ \/opt\/zig-0.17.0\/zig build-obj \/repro\/src\/xchg.zig -target x86_64-linux -O Debug -femit-bin=\/tmp\/xchg\/zig-0.17.0\/xchg.o\n+ cd \/tmp\/xchg\/zig-0.17.0 && x86_64-linux-gnu-objdump -d --no-addresses --disassemble=xchg_unused xchg.o > xchg_unused.objdump\n+ diff -u \/tmp\/xchg\/gcc-O0\/xchg_unused.objdump \/tmp\/xchg\/zig-0.17.0\/xchg_unused.objdump\n@@ -7,10 +7,12 @@\n <xchg_unused>:\n \t55                   \tpush   %rbp\n \t48 89 e5             \tmov    %rsp,%rbp\n-\t48 89 7d f8          \tmov    %rdi,-0x8(%rbp)\n-\t48 8b 45 f8          \tmov    -0x8(%rbp),%rax\n-\tba 01 00 00 00       \tmov    $0x1,%edx\n-\t87 10                \txchg   %edx,(%rax)      # gcc: the locked exchange\u2026\n-\t90                   \tnop\n+\t53                   \tpush   %rbx\n+\t48 83 ec 08          \tsub    $0x8,%rsp\n+\t48 89 3c 24          \tmov    %rdi,(%rsp)\n+\tbe 01 00 00 00       \tmov    $0x1,%esi\n+\t89 37                \tmov    %esi,(%rdi)      # \u2026Zig: opcode 89, a plain store, and no lock or fence anywhere\n+\t48 8d 65 f8          \tlea    -0x8(%rbp),%rsp\n+\t5b                   \tpop    %rbx\n \t5d                   \tpop    %rbp\n \tc3                   \tret\nRESULT[zig-0.17.0]: BUG, no xchg, lock or mfence: the exchange is a plain store<\/code><\/pre>\n\n\n\n
While there’s a bunch of boilerplate around function entry\/exit that differs between the two, the xchg<\/code> and mov<\/code> are fairly visible in the middle. If we diff the output of the xchg_used<\/code> and xchg_unused<\/code> from the self-hosted Zig compiler, we remove the boiler plate delta:<\/p>\n\n\n\n
+ diff -u \/tmp\/xchg\/zig-0.17.0\/xchg_unused.objdump \/tmp\/xchg\/zig-0.17.0\/xchg_used.objdump\n-<xchg_unused>:\n+<xchg_used>:\n \u2026\n \tbe 01 00 00 00       \tmov    $0x1,%esi\n-\t89 37                \tmov    %esi,(%rdi)\n+\t87 37                \txchg   %esi,(%rdi)\n+\t89 f0                \tmov    %esi,%eax<\/code><\/pre>\n\n\n\n
Reproducing at runtime<\/h2>\n\n\n\n
Since it is always a pile of fun to reproduce concurrency bugs, catching this at runtime means running a bunch of iterations and looking for a fail condition. The following does that, with two threads:<\/p>\n\n\n\n
\/\/   thread A: exchange(x, 1) seq_cst; r1 = load(y) seq_cst\n\/\/   thread B: exchange(y, 1) seq_cst; r2 = load(x) seq_cst\n\/\/\n\/\/ seq_cst forbids r1 == 0 and r2 == 0 together: whichever exchange comes first in the\n\/\/ single total order is visible to the other thread's load. The program counts how often\n\/\/ both loads read 0 in three trials:\n\/\/\n\/\/   relaxed    monotonic store, then monotonic load. Both-zero is allowed here, and is what\n\/\/              x86 does for a plain `mov` store followed by a load. Shows the test can see it.\n\/\/   discarded  seq_cst exchange, result discarded. The bug: the self-hosted x86_64 backend\n\/\/              emits a plain `mov`, so this behaves like `relaxed`.\n\/\/   used       seq_cst exchange, result used. Control: emitted as `xchg`, never both-zero.\n\/\/\n\/\/ Exits 1 if both-zero was seen in either seq_cst trial.\nconst std = @import(\"std\");\n\nconst iterations = 1_000_000;\n\nvar x: u32 = 0;\nvar y: u32 = 0;\nvar go: u32 = 0; \/\/ iteration the threads should run next; set by main\nvar done: u32 = 0; \/\/ threads that finished the current iteration\n\nconst Mode = enum { relaxed, discarded, used };\n\nfn Worker(comptime mode: Mode) type {\n    return struct {\n        fn run(mine: *u32, other: *u32, seen: *u32, old: *u32) void {\n            var i: u32 = 1;\n            while (i <= iterations) : (i += 1) {\n                while (@atomicLoad(u32, &go, .acquire) != i) std.atomic.spinLoopHint();\n                switch (mode) {\n                    .relaxed => @atomicStore(u32, mine, 1, .monotonic),\n                    .discarded => _ = @atomicRmw(u32, mine, .Xchg, 1, .seq_cst), \/\/ BUG: plain `mov`\n                    .used => old.* = @atomicRmw(u32, mine, .Xchg, 1, .seq_cst), \/\/ control: `xchg`\n                }\n                seen.* = @atomicLoad(u32, other, if (mode == .relaxed) .monotonic else .seq_cst);\n                _ = @atomicRmw(u32, &done, .Add, 1, .acq_rel);\n            }\n        }\n    };\n}\n\nfn trial(comptime mode: Mode) !u32 {\n    const W = Worker(mode);\n    var r1: u32 = 0;\n    var r2: u32 = 0;\n    var old1: u32 = 0;\n    var old2: u32 = 0;\n    go = 0;\n    done = 0;\n    const a = try std.Thread.spawn(.{}, W.run, .{ &x, &y, &r1, &old1 });\n    const b = try std.Thread.spawn(.{}, W.run, .{ &y, &x, &r2, &old2 });\n    var both_zero: u32 = 0;\n    var i: u32 = 1;\n    while (i <= iterations) : (i += 1) {\n        @atomicStore(u32, &x, 0, .seq_cst);\n        @atomicStore(u32, &y, 0, .seq_cst);\n        @atomicStore(u32, &go, i, .release); \/\/ start both threads\n        while (@atomicLoad(u32, &done, .acquire) != 2 * i) std.atomic.spinLoopHint();\n        if (r1 == 0 and r2 == 0) both_zero += 1; \/\/ forbidden in the two seq_cst trials\n    }\n    a.join();\n    b.join();\n    return both_zero;\n}\n\npub fn main() !void {\n    const relaxed = try trial(.relaxed);\n    const discarded = try trial(.discarded);\n    const used = try trial(.used);\n    std.debug.print(\"relaxed store+load (allowed):        both loads read 0 in {d} of {d} iterations\\n\", .{ relaxed, iterations });\n    std.debug.print(\"seq_cst xchg, result discarded:      both loads read 0 in {d} of {d} iterations\\n\", .{ discarded, iterations });\n    std.debug.print(\"seq_cst xchg, result used (control): both loads read 0 in {d} of {d} iterations\\n\", .{ used, iterations });\n    if (discarded != 0 or used != 0) std.process.exit(1);\n}\n<\/code><\/pre>\n\n\n\n
This will reproduce on x86-64 systems fairly easily. I’ll note that I didn’t get qemu to reproduce when doing usermode emulation on an aarch64 host, so be sure to try on a real x86-64 system (and almost certainly a multi-core one). Below, we have the output of running the binary produced by the self hosted Zig compiler, and then the output of the binary built with the LLVM based one:<\/p>\n\n\n\n
\n    sb-0.17.0 Worker(.discarded): 89 3e   mov    %edi,(%rsi)      # self-hosted: plain store\n    sb-0.17.0 Worker(.used): 87 3e        xchg   %edi,(%rsi)\n    sb-0.17.0-llvm Worker(.discarded): 87 16   xchg   %edx,(%rsi) # LLVM: xchg either way\n+ \/tmp\/sb\/sb-0.17.0\nrelaxed store+load (allowed):        both loads read 0 in 96990 of 1000000 iterations\nseq_cst xchg, result discarded:      both loads read 0 in 128430 of 1000000 iterations   # forbidden, seen 13% of the time\nseq_cst xchg, result used (control): both loads read 0 in 0 of 1000000 iterations\nRESULT[sb-0.17.0]: BUG, both loads read 0 under seq_cst\n\n+ \/tmp\/sb\/sb-0.17.0-llvm\nrelaxed store+load (allowed):        both loads read 0 in 41808 of 1000000 iterations\nseq_cst xchg, result discarded:      both loads read 0 in 0 of 1000000 iterations      # same source, LLVM backend\nseq_cst xchg, result used (control): both loads read 0 in 0 of 1000000 iterations\nRESULT[sb-0.17.0-llvm]: OK, seq_cst ordering held<\/code><\/pre>\n\n\n\n
I’ve reproduced this with the 0.16 version shipped in Fedora 44, the upstream 0.17 binaries, and upstream nightly build 0.18.0-dev.367+187d0bf9e<\/code>.<\/p>\n\n\n\n
Okay, but get Volume 3A of the Intel\u00ae 64 and IA-32 Architectures Software Developer\u2019s Manual nerdy with me<\/h2>\n\n\n\n
Specifically, “11.2.2 Memory Ordering in P6 and More Recent Processor Families” in the Intel\u00ae 64 and IA-32 Architectures Software Developer\u2019s Manual Volume 3A<\/a> has two relevant rules:<\/p>\n\n\n\n
    \n
  1. “Reads may be reordered with older writes to different locations but not with older writes to the same location.”<\/strong> This is what the plain mov<\/code> allows: a later load can complete before the store is visible to other processors.<\/li>\n\n\n\n
  2. “Reads or writes cannot be reordered with I\/O instructions, locked instructions, or serializing instructions.”<\/strong> This is what a locked instruction guarantees, and seq_cst requires it.<\/li>\n<\/ol>\n\n\n\n
    The Zig language appears to have a big TODO in their language documentation around Atomics<\/a>, and the std<\/code> documentation for what the enum<\/code> is for AtomicOrder<\/a>. The upstream LLVM docs are rather complete<\/a>.<\/p>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    \n
    “Additionally, the code generator must enforce ordering between SequentiallyConsistent stores followed by SequentiallyConsistent loads” – https:\/\/llvm.org\/docs\/Atomics.html#sequentiallyconsistent<\/a><\/p>\n<\/blockquote>\n\n\n\n
    Did I report this upstream?<\/h2>\n\n\n\n
    No, because doing so would be a violation of the Code of Conduct<\/a>. Specifically, let’s look at the LLM\/AI policy part<\/a>: <\/p>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    Since I used an LLM to help me learn a bit about Zig and how it works, and then got it to help isolate this down, the “No LLMs for finding bugs”<\/strong> undoubtedly applies. Later on, there’s also the explicit example of unacceptable behavior being “Discussing this Code of Conduct<\/strong>“.<\/p>\n\n\n\n
    I’ve written this up because others may find it interesting, and maybe this helps someone debug their concurrent Zig application at some point.<\/p>\n","protected":false},"excerpt":{"rendered":"
    Finding codegen bugs for atomic operations is fun! I’ll note that I am not a Zig developer. I don’t write code in the language, and I don’t work on the language. I’m just some guy who decided to learn a … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[1],"tags":[],"class_list":["post-5040","post","type-post","status-publish","format-standard","hentry","category-general"],"jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5a6n8-1ji","jetpack-related-posts":[{"id":4896,"url":"https:\/\/www.flamingspork.com\/blog\/2024\/01\/13\/using-llvm-mca-for-predicting-cpu-cycle-impact-of-code-changes\/","url_meta":{"origin":5040,"position":0},"title":"Using llvm-mca for predicting CPU cycle impact of code changes","author":"Stewart Smith","date":"2024-01-13","format":false,"excerpt":"Way back in the distant past, when the Apple ][ and the Commodore 64 were king, you could read the manual for a microprocessor and see how many CPU cycles each instruction took, and then do the math as to how long a sequence of instructions would take to execute.\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3912,"url":"https:\/\/www.flamingspork.com\/blog\/2014\/12\/03\/running-skiboot-opal-on-the-power8-simulator\/","url_meta":{"origin":5040,"position":1},"title":"Running skiboot (OPAL) on the POWER8 Simulator","author":"Stewart Smith","date":"2014-12-03","format":false,"excerpt":"skiboot is open source boot and runtime firmware for OpenPOWER. On real POWER8 hardware, you will also need HostBoot to do this (basically, to make the chip work) but in a functional simulator (such as this one released by IBM) you don't need a bunch of hardware procedures to make\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3989,"url":"https:\/\/www.flamingspork.com\/blog\/2015\/08\/28\/doing-nothing-on-modern-cpus\/","url_meta":{"origin":5040,"position":2},"title":"doing nothing on modern CPUs","author":"Stewart Smith","date":"2015-08-28","format":false,"excerpt":"Sometimes you don't want to do anything. This is understandably human, and probably a sign you should either relax or get up and do something. For processors, you sometimes do actually want to do absolutely nothing. Often this will be while waiting for a lock. You want to do nothing\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1587,"url":"https:\/\/www.flamingspork.com\/blog\/2009\/04\/08\/using-dtrace-to-find-out-why-solaris-is-so-slow-and-if-its-actually-solaris\/","url_meta":{"origin":5040,"position":3},"title":"Using Dtrace to find out if the hardware or Solaris is slow (but really just working around the problem)","author":"Stewart Smith","date":"2009-04-08","format":false,"excerpt":"A little while ago, I was the brave soul tasked with making sure Drizzle was working properly and passing all tests on Solaris and OpenSolaris. Brian recently blogged about some of the advantages of also running on Solaris and the SunStudio compilers - more warnings from the compiler is a\u2026","rel":"","context":"In "drizzle"","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4166,"url":"https:\/\/www.flamingspork.com\/blog\/2016\/09\/12\/compiling-your-own-firmware-for-the-s822lc-for-hpc\/","url_meta":{"origin":5040,"position":4},"title":"Compiling your own firmware for the S822LC for HPC","author":"Stewart Smith","date":"2016-09-12","format":false,"excerpt":"IBM (my employer) recently announced\u00c2\u00a0 the new S822LC for HPC POWER8+NVLINK NVIDIA P100 GPUs server (press release, IBM Systems Blog, The Register). The \"For HPC\" suffix on the model number is significant, as the S822LC is a different machine. What makes the \"for HPC\" variant different is that the POWER8\u2026","rel":"","context":"In "IBM"","block_context":{"text":"IBM","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/ibm-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":218,"url":"https:\/\/www.flamingspork.com\/blog\/2004\/02\/18\/memberdb-01\/","url_meta":{"origin":5040,"position":5},"title":"memberdb 0.1!","author":"Stewart Smith","date":"2004-02-18","format":false,"excerpt":"Oh yes, time for me to abuse my position and spam a bunch of lists! :) memberdb is the membership database software i've been hacking off and on for a while now, and is being used by Linux Australia. In it's current form, it's visually not very pretty - but\u2026","rel":"","context":"In "linux-aus"","block_context":{"text":"linux-aus","link":"https:\/\/www.flamingspork.com\/blog\/category\/linux-aus\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_likes_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/5040","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/comments?post=5040"}],"version-history":[{"count":3,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/5040\/revisions"}],"predecessor-version":[{"id":5047,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/5040\/revisions\/5047"}],"wp:attachment":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/media?parent=5040"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/categories?post=5040"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/tags?post=5040"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}