Double-width atomics aren’t atomic if using “zig cc” to build C/C++ code

The Zig project heavily advocates the use case of using its toolchain to build C/C++ code, “Use Zig as a zero-dependency, drop-in C/C++ compiler that supports cross-compilation out-of-the-box.” (Zig home page https://ziglang.org/ fetched 2026-10-21). Back in 2020, the announcement from Zig’s creator was “zig cc: a Powerful Drop-In Replacement for GCC/Clang”. As far as I can work out, the vast majority of the usage of the Zig toolchain is as a C/C++ compiler, and since it is LLVM based, one may reasonably assume that it is pretty much the same as using clang.

Unfortunately, this is not true if you rely on some atomic operations being, well, atomic.

I think going through the details on this bug is a pretty interesting learning exercise (I certainly learned some things), so here goes….

A compiler (at least in the Linux world, I assume it’s similar on other platforms) implements an atomic operation by emitting a lock-free instruction, or by calling a library function (e.g. __atomic_load_16, __atomic_load_8, etc) if building for a target which lacks a suitable lock-free instruction. This enables a compiler to emit efficient native code for the target, as well as have a library that can be used in two cases: 1) The base processor being targeted doesn’t have an instruction for the operation, and 2) the implementation of the library calls can be optimal for the CPU the program is running on, thus a program can target an older CPU but get (most of) the benefit of improved atomic instructions on newer processors.

The lock-free instruction and the implementation of the equivalent library function needs to be compatible as either form can be used on the same object in the same program. This allows building software that works across a large range of processors but has the hot paths built for different micro-architectures and selected at runtime. This is effectively the same thing as supporting linking in objects built with a prior version of the compiler that didn’t know about the lock-free instruction. Thus, LLVM and GCC require that the library implementing the atomic functions to be lock-free for every size the compiler can emit inline, and both link to GNU libatomic for the library functions.

What the toolchains require

The LLVM Atomics guide, “Libcalls: __atomic_*“:

The set of atomic sizes supported lock-free must be a superset of the sizes any compiler can emit. That is: if a new compiler introduces support for inline-lock-free atomics of size N, the __atomic_* functions must also have a lock-free implementation for size N. This is a requirement so that code produced by an old compiler (which will have called the __atomic_* function) interoperates with code produced by the new compiler (which will use native the atomic instruction).

GCC’s design for its atomic library, Atomic/GCCMM/LIbrary on the GCC wiki (archived copy, 2024-12-30):

Implementation of the external routines must agree with the compiler for any given object size. If the compiler generates lock-free code for an object, the library must also implement lock free for that size as well. It is conforming for the compiler to simply always call the library, so both must agree on the implementation.

The same page gives the 32-bit test below as its example:

Compile a library for 386. The 386 does not support a lock-free atomic int, so the compiler generates function calls. Compile an application for 486. The 486 does support a lock-free atomic int, so the compiler generates machine instructions. Link those two and run them on a 486. The operating system was built knowing that it’s architectural level is 486, so its atomic ABI library implements the atomic int with lock-free machine instructions.

The problem with “zig cc”

However, when you build your C/C++ code using zig cc, it will emit lock-free instructions and calls to library functions, but it does not link to GNU libatomic, instead it links to Zig’s bundled compiler_rt library, which contains its own implementations.

In Zig’s compiler_rt library, which is linked to your C/C++ code, __atomic_load_16 on x86_64 and ppc64le, and __atomic_load_8 on 32-bit x86, are implemented by taking a spinlock and then doing two separate half-size loads. This obviously breaks if the compiler can emit a lock-free instruction for these operations, which it does. Thus, loads tear and your atomic operations are not atomic. While it is possible to pass -latomic to zig cc, this makes no difference.

Reproducing with a single C file (x86-64, 16-byte atomics)

We will use a small C program where a writer thread and a reader thread access one _Atomic unsigned __int128. We use the __attribute__((target("cx16"))) feature of the compiler as a shortcut for situations where you’d have different implementations of one function optimized for different CPUs or for linking in different builds.

If we build the source with the default options, since writer() has __attribute__((target("cx16"))), its atomic_store will be an inline lock cmpxchg16b, and since reader() has no attribute, its atomic_load is a call to __atomic_load_16.

(spoiler) if you use clang or gcc to compile and link, this will work perfectly.

here’s the code:

/*
 * Test: the writer only stores values whose two 8-byte halves are equal;
 *       a load with unequal halves is torn, and should NEVER happen.
 */
#include <stdatomic.h>
#include <stdio.h>
#include <pthread.h>
#include <time.h>

static _Atomic unsigned __int128 shared;
static atomic_int stop;

__attribute__((target("cx16"))) static void *writer(void *arg) {
    (void)arg;
    for (unsigned long long i = 1; !atomic_load(&stop); i++)
        atomic_store(&shared, ((unsigned __int128)i << 64) | i);       /* inline lock cmpxchg16b */
    return NULL;
}

static void *reader(void *arg) {
    unsigned long long *torn = arg, loads = 0;
    while (!atomic_load(&stop)) {
        unsigned __int128 v = atomic_load(&shared);                    /* call __atomic_load_16 */
        if ((unsigned long long)(v >> 64) != (unsigned long long)v) (*torn)++;
        loads++;
    }
    torn[1] = loads;
    return NULL;
}

int main(void) {
    unsigned long long res[2] = {0, 0};
    pthread_t w, r;
    pthread_create(&w, NULL, writer, NULL);
    pthread_create(&r, NULL, reader, res);
    struct timespec t = {2, 0}; nanosleep(&t, NULL);
    atomic_store(&stop, 1);
    pthread_join(w, NULL); pthread_join(r, NULL);
    printf("torn loads: %llu of %llu\n", res[0], res[1]);
    return res[0] != 0;
}

If we now build and run that code (run it 3x for 2s each time), we can see how the different compilers fare:


+ clang -O2 atomic16.c -lpthread -latomic -o out/atomic16-clang-latomic
#    writer: cmpxchg16b   reader: call <__atomic_load_16@plt>
#    Links to: [libatomic.so.1] [libc.so.6]
#      __atomic_load_16: U __atomic_load_16@LIBATOMIC_1.0
+ zig cc -target x86_64-linux-gnu -O2 atomic16.c -lpthread -o out/atomic16-zigcc
#    writer: cmpxchg16b   reader: call <__atomic_load_16>
#    Links to: [libc.so.6] [libpthread.so.0]
#     __atomic_load_16: t __atomic_load_16
+ zig cc -target x86_64-linux-gnu -O2 atomic16.c -lpthread -latomic -o out/atomic16-zigcc-latomic
#    writer: cmpxchg16b   reader: call <__atomic_load_16>
#    Links to: [libc.so.6] [libpthread.so.0]
#    __atomic_load_16: t __atomic_load_16

+ ./out/atomic16-clang-latomic
torn loads: 0 of 682030142
torn loads: 0 of 645687188
torn loads: 0 of 692854822
+ ./out/atomic16-zigcc
torn loads: 720 of 128380947
torn loads: 382 of 138503698
torn loads: 13108 of 131805641
+ ./out/atomic16-zigcc-latomic
torn loads: 455 of 137924614
torn loads: 20909 of 132735414
torn loads: 425 of 135945438

I used Zig 0.17.0 for these tests.

The inability to report upstream

I wrote about it in more detail in my first post on Zig, but the tl;dr is that since an LLM was involved in finding this bug, the “No LLMs for finding bugs” part of their Code of Conduct obviously applies.

I’ve written this up because others may find it interesting, and maybe this helps someone debug their concurrent Zig application at some point, or learn something about how compilers do (or do not) work.

Zig self-hosted compiler can’t compile a 16-byte @atomicLoad

Previously, I wrote about how the Zig self-hosted backend lowers a seq_cst @atomicRmw(.Xchg) with a discarded result to a plain mov, but this isn’t the only issue with the self-hosted backend and atomics.

A valid bit of Zig code is @atomicLoad(u128, p, .seq_cst) – an atomic load of an unsigned 128 bit value.

A trivial example in Zig:

export fn load128(p: *u128) u128 {
    return @atomicLoad(u128, p, .seq_cst);
}

The equivalent trivial example in C:

unsigned __int128 load128(unsigned __int128 *p) { return __atomic_load_n(p, __ATOMIC_SEQ_CST); }

When building for a target that has the cmpxchg16b instruction (such as the x86-64v2 target, the base x86-64 processor level for several Linux distributions, or by specifying -mcx16 to enable just that instruction), this should compile down to the lock cmpxchg16b instruction. If using the LLVM backend for building Zig, then this works. However, if you try and build the same Zig with the self-hosted compiler (the default for x86-64 debug builds, or selected by passing -fno-llvm), then it doesn’t work:

/opt/zig-0.17.0/zig build-obj /repro/src/load128.zig -target x86_64-linux -mcpu=x86_64_v2 -O Debug -femit-bin=/tmp/load128/zig-0.17.0/load128.o
repro/src/load128.zig:2:8: error: genSetReg called with a value larger than dst_reg
export fn load128(p: *u128) u128 {
~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
exit status: 1

It’s possible that the Codeberg issue #37289 issue, “”x86_64 backend: compiler segfault on @cmpxchgStrong(u128, ...) when the new value is an expression”” is related (this was an open issue as of 2026-10-17).

I have not reported this upstream, as the Zig Code of Conduct is unambiguous about bugs found with the assistance of LLMs (see my previous post for details).

Zig self-hosted backend lowers a seq_cst @atomicRmw(.Xchg) with a discarded result to a plain mov

Finding codegen bugs for atomic operations is fun! I’ll note that I am not a Zig developer. I don’t write code in the language, and I don’t work on the language. I’m just some guy who decided to learn a bit about it.

The Zig programming language has two compilers that are widely used: one generates LLVM IR that libLLVM then builds into native code, and the other is a self-hosted compiler which eschews LLVM and does its own codegen.

It’s important to note that the self-hosted compiler is only currently (as of late October 2026) used on x86_64 Debug targets. All other targets use the LLVM backend by default.

_ = @atomicRmw(u32, p, .Xchg, 1, .seq_cst); is a sequentially consistent exchange. On x86 that is xchg with a memory operand, which is implicitly locked and is a full barrier. When the result is discarded, Zig’s self-hosted x86_64 backend emits a plain mov store instead. A plain store on x86 lets a later load be reordered before it (StoreLoad), so the operation loses its seq_cst ordering. Any build that uses the self-hosted x86_64 backend is affected. That backend is the default for -O Debug on x86_64-linux, and -fno-llvm selects it explicitly.

The same backend emits xchg when the result is used. So does a lock acquired by testing the exchange’s return value, which is not affected. Zig’s LLVM backend and gcc emit xchg in both cases.

This affects softwhare when: code discards the exchange’s result, and then relies on access of a different variable to be executed by the processor after that xchg. An example could be code sets its own flag, then reads another thread’s flag.

Affected: Zig 0.16.0 (Fedora 44 package), 0.17.0 and (current at the time) master 0.18.0-dev.367+187d0bf9e (commit 187d0bf9e483, 2026-10-19), all with the self-hosted x86_64 backend (the default for -O Debug -target x86_64-linux). The LLVM backend (-fllvm) is not affected.

Reproducing the codegen bug

In order to see the codegen bug, you can build a small source file and disassemble it:

export fn xchg_unused(p: *u32) void {
    _ = @atomicRmw(u32, p, .Xchg, 1, .seq_cst);    // BUG: result discarded -> plain mov
}
export fn xchg_used(p: *u32) u32 {                 // control
    return @atomicRmw(u32, p, .Xchg, 1, .seq_cst);
}

We can then compare the result with some C code built with a variety of different toolchains, but gcc -O0 gives us the closest result to the self-hosted Zig compiler’s output.

void xchg_unused(unsigned *p) { (void)__atomic_exchange_n(p, 1, __ATOMIC_SEQ_CST); }

The (annotated) gcc output for the C code looks like this:

+ x86_64-linux-gnu-gcc -O0 -c /repro/src/xchg.c -o /tmp/xchg/gcc-O0/xchg.o
+ cd /tmp/xchg/gcc-O0 && x86_64-linux-gnu-objdump -d --no-addresses --disassemble=xchg_unused xchg.o > xchg_unused.objdump
<xchg_unused>:
	55                   	push   %rbp
	48 89 e5             	mov    %rsp,%rbp
	48 89 7d f8          	mov    %rdi,-0x8(%rbp)
	48 8b 45 f8          	mov    -0x8(%rbp),%rax
	ba 01 00 00 00       	mov    $0x1,%edx
	87 10                	xchg   %edx,(%rax)      # opcode 87: xchg, implicitly locked: correct
	90                   	nop
	5d                   	pop    %rbp
	c3                   	ret
RESULT[gcc -O0 (16.2.1)]: OK, 1 locked/fencing instruction(s)

While we can see the difference in the code generated by the Zig compiler below, annotated as to where the bug is:

+ /opt/zig-0.17.0/zig build-obj /repro/src/xchg.zig -target x86_64-linux -O Debug -femit-bin=/tmp/xchg/zig-0.17.0/xchg.o
+ cd /tmp/xchg/zig-0.17.0 && x86_64-linux-gnu-objdump -d --no-addresses --disassemble=xchg_unused xchg.o > xchg_unused.objdump
+ diff -u /tmp/xchg/gcc-O0/xchg_unused.objdump /tmp/xchg/zig-0.17.0/xchg_unused.objdump
@@ -7,10 +7,12 @@
 <xchg_unused>:
 	55                   	push   %rbp
 	48 89 e5             	mov    %rsp,%rbp
-	48 89 7d f8          	mov    %rdi,-0x8(%rbp)
-	48 8b 45 f8          	mov    -0x8(%rbp),%rax
-	ba 01 00 00 00       	mov    $0x1,%edx
-	87 10                	xchg   %edx,(%rax)      # gcc: the locked exchange…
-	90                   	nop
+	53                   	push   %rbx
+	48 83 ec 08          	sub    $0x8,%rsp
+	48 89 3c 24          	mov    %rdi,(%rsp)
+	be 01 00 00 00       	mov    $0x1,%esi
+	89 37                	mov    %esi,(%rdi)      # …Zig: opcode 89, a plain store, and no lock or fence anywhere
+	48 8d 65 f8          	lea    -0x8(%rbp),%rsp
+	5b                   	pop    %rbx
 	5d                   	pop    %rbp
 	c3                   	ret
RESULT[zig-0.17.0]: BUG, no xchg, lock or mfence: the exchange is a plain store

While there’s a bunch of boilerplate around function entry/exit that differs between the two, the xchg and mov are fairly visible in the middle. If we diff the output of the xchg_used and xchg_unused from the self-hosted Zig compiler, we remove the boiler plate delta:

+ diff -u /tmp/xchg/zig-0.17.0/xchg_unused.objdump /tmp/xchg/zig-0.17.0/xchg_used.objdump
-<xchg_unused>:
+<xchg_used>:
 …
 	be 01 00 00 00       	mov    $0x1,%esi
-	89 37                	mov    %esi,(%rdi)
+	87 37                	xchg   %esi,(%rdi)
+	89 f0                	mov    %esi,%eax

Reproducing at runtime

Since it is always a pile of fun to reproduce concurrency bugs, catching this at runtime means running a bunch of iterations and looking for a fail condition. The following does that, with two threads:

//   thread A: exchange(x, 1) seq_cst; r1 = load(y) seq_cst
//   thread B: exchange(y, 1) seq_cst; r2 = load(x) seq_cst
//
// seq_cst forbids r1 == 0 and r2 == 0 together: whichever exchange comes first in the
// single total order is visible to the other thread's load. The program counts how often
// both loads read 0 in three trials:
//
//   relaxed    monotonic store, then monotonic load. Both-zero is allowed here, and is what
//              x86 does for a plain `mov` store followed by a load. Shows the test can see it.
//   discarded  seq_cst exchange, result discarded. The bug: the self-hosted x86_64 backend
//              emits a plain `mov`, so this behaves like `relaxed`.
//   used       seq_cst exchange, result used. Control: emitted as `xchg`, never both-zero.
//
// Exits 1 if both-zero was seen in either seq_cst trial.
const std = @import("std");

const iterations = 1_000_000;

var x: u32 = 0;
var y: u32 = 0;
var go: u32 = 0; // iteration the threads should run next; set by main
var done: u32 = 0; // threads that finished the current iteration

const Mode = enum { relaxed, discarded, used };

fn Worker(comptime mode: Mode) type {
    return struct {
        fn run(mine: *u32, other: *u32, seen: *u32, old: *u32) void {
            var i: u32 = 1;
            while (i <= iterations) : (i += 1) {
                while (@atomicLoad(u32, &go, .acquire) != i) std.atomic.spinLoopHint();
                switch (mode) {
                    .relaxed => @atomicStore(u32, mine, 1, .monotonic),
                    .discarded => _ = @atomicRmw(u32, mine, .Xchg, 1, .seq_cst), // BUG: plain `mov`
                    .used => old.* = @atomicRmw(u32, mine, .Xchg, 1, .seq_cst), // control: `xchg`
                }
                seen.* = @atomicLoad(u32, other, if (mode == .relaxed) .monotonic else .seq_cst);
                _ = @atomicRmw(u32, &done, .Add, 1, .acq_rel);
            }
        }
    };
}

fn trial(comptime mode: Mode) !u32 {
    const W = Worker(mode);
    var r1: u32 = 0;
    var r2: u32 = 0;
    var old1: u32 = 0;
    var old2: u32 = 0;
    go = 0;
    done = 0;
    const a = try std.Thread.spawn(.{}, W.run, .{ &x, &y, &r1, &old1 });
    const b = try std.Thread.spawn(.{}, W.run, .{ &y, &x, &r2, &old2 });
    var both_zero: u32 = 0;
    var i: u32 = 1;
    while (i <= iterations) : (i += 1) {
        @atomicStore(u32, &x, 0, .seq_cst);
        @atomicStore(u32, &y, 0, .seq_cst);
        @atomicStore(u32, &go, i, .release); // start both threads
        while (@atomicLoad(u32, &done, .acquire) != 2 * i) std.atomic.spinLoopHint();
        if (r1 == 0 and r2 == 0) both_zero += 1; // forbidden in the two seq_cst trials
    }
    a.join();
    b.join();
    return both_zero;
}

pub fn main() !void {
    const relaxed = try trial(.relaxed);
    const discarded = try trial(.discarded);
    const used = try trial(.used);
    std.debug.print("relaxed store+load (allowed):        both loads read 0 in {d} of {d} iterations\n", .{ relaxed, iterations });
    std.debug.print("seq_cst xchg, result discarded:      both loads read 0 in {d} of {d} iterations\n", .{ discarded, iterations });
    std.debug.print("seq_cst xchg, result used (control): both loads read 0 in {d} of {d} iterations\n", .{ used, iterations });
    if (discarded != 0 or used != 0) std.process.exit(1);
}

This will reproduce on x86-64 systems fairly easily. I’ll note that I didn’t get qemu to reproduce when doing usermode emulation on an aarch64 host, so be sure to try on a real x86-64 system (and almost certainly a multi-core one). Below, we have the output of running the binary produced by the self hosted Zig compiler, and then the output of the binary built with the LLVM based one:


    sb-0.17.0 Worker(.discarded): 89 3e   mov    %edi,(%rsi)      # self-hosted: plain store
    sb-0.17.0 Worker(.used): 87 3e        xchg   %edi,(%rsi)
    sb-0.17.0-llvm Worker(.discarded): 87 16   xchg   %edx,(%rsi) # LLVM: xchg either way
+ /tmp/sb/sb-0.17.0
relaxed store+load (allowed):        both loads read 0 in 96990 of 1000000 iterations
seq_cst xchg, result discarded:      both loads read 0 in 128430 of 1000000 iterations   # forbidden, seen 13% of the time
seq_cst xchg, result used (control): both loads read 0 in 0 of 1000000 iterations
RESULT[sb-0.17.0]: BUG, both loads read 0 under seq_cst

+ /tmp/sb/sb-0.17.0-llvm
relaxed store+load (allowed):        both loads read 0 in 41808 of 1000000 iterations
seq_cst xchg, result discarded:      both loads read 0 in 0 of 1000000 iterations      # same source, LLVM backend
seq_cst xchg, result used (control): both loads read 0 in 0 of 1000000 iterations
RESULT[sb-0.17.0-llvm]: OK, seq_cst ordering held

I’ve reproduced this with the 0.16 version shipped in Fedora 44, the upstream 0.17 binaries, and upstream nightly build 0.18.0-dev.367+187d0bf9e.

Okay, but get Volume 3A of the Intel® 64 and IA-32 Architectures Software Developer’s Manual nerdy with me

Specifically, “11.2.2 Memory Ordering in P6 and More Recent Processor Families” in the Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 3A has two relevant rules:

  1. “Reads may be reordered with older writes to different locations but not with older writes to the same location.” This is what the plain mov allows: a later load can complete before the store is visible to other processors.
  2. “Reads or writes cannot be reordered with I/O instructions, locked instructions, or serializing instructions.” This is what a locked instruction guarantees, and seq_cst requires it.

The Zig language appears to have a big TODO in their language documentation around Atomics, and the std documentation for what the enum is for AtomicOrder. The upstream LLVM docs are rather complete.

“Additionally, the code generator must enforce ordering between SequentiallyConsistent stores followed by SequentiallyConsistent loads” – https://llvm.org/docs/Atomics.html#sequentiallyconsistent

Did I report this upstream?

No, because doing so would be a violation of the Code of Conduct. Specifically, let’s look at the LLM/AI policy part:

Since I used an LLM to help me learn a bit about Zig and how it works, and then got it to help isolate this down, the “No LLMs for finding bugs” undoubtedly applies. Later on, there’s also the explicit example of unacceptable behavior being “Discussing this Code of Conduct“.

I’ve written this up because others may find it interesting, and maybe this helps someone debug their concurrent Zig application at some point.

Swift code for talking to the Shotwell database

A flashback to a few years in the past, this has sat as a draft for too long. See previously: Getting your photos out of Shotwell and Every time I program a Mac…

So…. what does the code to read the Shotwell database look like?

First, we’re going to have to import some of the modules we need:

import Foundation
import SQLite3
import SwiftUI
import Photos

I forget what SwiftUI here is for… I think it’s for my bit of integration with the UI I have.

From here I can start to model the bits of the Shotwell database that I’ll possibly need, but using more idiomatic Swift than raw SQL queries in random places. This is what I ended up for having information about a Photo:

class ShotwellPhotoInfo : Codable, Identifiable, Hashable {
    static func == (lhs: ShotwellPhotoInfo, rhs: ShotwellPhotoInfo) -> Bool {
        return lhs.id == rhs.id
    }
    
    func hash(into hasher: inout Hasher) {
        hasher.combine(id)
    }
    
    var id : Int32 = 0
    var filename : String = ""
    var translated_filename : String = ""
    var orientation : Int32 = 0
    var original_orientation : Int32 = 0
    var import_id : Int32 = 0
    var event_id : Int32 = 0
    var event : ShotwellEventInfo?
    var transformations : String = ""
    var md5 : String = ""
    var flags: Int32 = 0
    var rating: Int32 = 0
    var title: String = ""
    var raw_backlinks : String = ""
    var backlinks : ShotwellBacklinks?
    var comment : String = ""
    var has_gps : Bool = false
    var gps_lat : Double = 0.0
    var gps_lon : Double = 0.0
}

The comparison function is just comparing the id, which since that’s a primary key in the database, are unique. The hash function also just acts on id because of this.

The translated_filename is not in the Shotwell database, but rather for me to do various transformations on the filename in the shotwell database so that I can actually find the file. The natural translation is going to be for wherever I have mounted my photos, but there were some other weirdness likely due to some symlinks and such I’ve had.

The Shotwell Database Architecture documentation says this:

  • backlinks are persistent links to container objects, i.e. Events and Tags. These are used when a Photo has been removed from one (for example, it’s been trashed or marked offline). If the Photo is returned to the main heap of photos (i.e. untrashed), these backlinks are used to restore its connections to the containers.
https://wiki.gnome.org/Apps/Shotwell/Architecture/Database

So it appears that Backlinks may be rather useful to find out everything I need to know about a Photo that I am looking to import. Thus, we will need to model Events and Tags as well, which are familiar concepts from the Shotwell UI.

Since there are Tags in Shotwell, I need to model them as well:

class ShotwellTag : Codable, Identifiable, Hashable {
    static func == (lhs: ShotwellTag, rhs: ShotwellTag) -> Bool {
        return lhs.id == rhs.id
    }
    
    func hash(into hasher: inout Hasher) {
        hasher.combine(id)
    }
    let id : Int32
    let name : String
    let photo_id_list : [Int32]
    let time_created : Int32
    let photos : [ShotwellPhotoInfo]
    init(id: Int32, name: String, photo_id_list: [Int32], time_created: Int32, photos: [ShotwellPhotoInfo]) {
        self.id = id
        self.name = name
        self.photo_id_list = photo_id_list
        self.time_created = time_created
        self.photos = photos
    }
}

Again, we have an id as a primary key, and the photos in a tag are referenced as a list of IDs.

For an Event, this is what I ended up with:

struct ShotwellEventInfo : Codable, Identifiable {
    let id : Int32
    let name : String
    let primary_photo_id : Int32
    let time_created : Int32
    let primary_source_id : Int32
    let comment : String
}

and for backlinks:

class ShotwellBacklinks : Codable {
    var tag : [ShotwellTag?] = []
    var event : [String] = []
    var other : [String] = []
}

But as for just getting some basic information out of the database? That we can start off with by having a class for opening it, and run some simple queries, and even display them in the UI just to prove we can.

class ShotwellDatabase {
    var db: OpaquePointer?
    var import_query: OpaquePointer?
    var dbPath : String?
    
    struct db_version {
        var schema_version: Int32 = 0
        var app_version: String = ""
        init() {
            schema_version = 0
            app_version = ""
        }
    }
    
    func open_db(shotwellDB: String) -> Bool {
        dbPath = shotwellDB
        if sqlite3_open(shotwellDB, &db) == SQLITE_OK {
            print("Opened Shotwell Database!")
        }
        return db != nil
    }
    
    func get_db_version() -> db_version {
        let query = "SELECT schema_version, app_version FROM VersionTable"
        var queryStatement: OpaquePointer?
        var r = db_version()
        
        if sqlite3_prepare_v2(db, query, -1, &queryStatement, nil) == SQLITE_OK {
            if sqlite3_step(queryStatement) == SQLITE_ROW {
                r.schema_version = sqlite3_column_int(queryStatement, 0)
                
                guard let app_version = sqlite3_column_text(queryStatement, 1) else {
                    print("Couldn't get app version")
                    r.app_version = ""
                    return r
                }
                
                r.app_version = String(cString: app_version)
            }
        }
        return r
    }

The database version is more about just verifying I could link things together than anything actually useful. Continuing the ShotwellDatabase class, getting the total number of photos is pretty easy (added to the ShotwellDatabase class above):

    
    func get_photos_count() -> Int64 {
        let query = "SELECT COUNT(id) FROM PhotoTable"
        var queryStatement: OpaquePointer?

        if sqlite3_prepare_v2(db, query, -1, &queryStatement, nil) == SQLITE_OK {
            if sqlite3_step(queryStatement) == SQLITE_ROW {
                return sqlite3_column_int64(queryStatement, 0)
            }
        }
        return 0
    }

Again, that’s mostly just useful for some nice “oh look, I linked up the UI to the database” rather than anything of use (although progress bars may want to use it).

So, now that we can talk to the database, we can go and set things up to iterate over all the photos and load them up into memory. I’m not doing anything too fancy here as this is all very quick and dirty, and was literally the first time writing any Swift code.

The below code starts by opening the Database, then get_next_photo will iterate over everything before done_all_photos can close things up. The commented out block of code below does something pretty neat, and that’s print out a JSON representation of our photo. I found this incredibly easy to set up in Swift.

    func start_get_all_photos() -> OpaquePointer? {
        var db: OpaquePointer?
        if sqlite3_open(dbPath, &db) == SQLITE_OK {
            print("Opened Shotwell Database!")
        }
        start_import(db: db)
        return db
    }
    func get_next_photo(db: OpaquePointer?) -> ShotwellPhotoInfo? {
        var photo: ShotwellPhotoInfo?
            do {
                photo = db_get_next_photo(db: db)
                if photo != nil {
/*                    let encoder = JSONEncoder()
                    encoder.outputFormatting = .prettyPrinted
                    let jsonData = try encoder.encode(photo)
                    let jsonString = String(data: jsonData, encoding: .utf8)!
                    photo_info = jsonString
                    print(jsonString)
                    do_some_photo_things()
                    progress += 1 */
                    all_photos[photo!.id] = photo
                }
            } catch { print(error) }
        return photo
    }
    func done_all_photos(db: OpaquePointer?) {
        sqlite3_close(db)
    }

For parsing that row, I have the following class:

class ShotwellPhotoInfo : Codable, Identifiable, Hashable {
    static func == (lhs: ShotwellPhotoInfo, rhs: ShotwellPhotoInfo) -> Bool {
        return lhs.id == rhs.id
    }
    
    func hash(into hasher: inout Hasher) {
        hasher.combine(id)
    }
    
    var id : Int32 = 0
    var filename : String = ""
    var translated_filename : String = ""
    var orientation : Int32 = 0
    var original_orientation : Int32 = 0
    var import_id : Int32 = 0
    var event_id : Int32 = 0
    var event : ShotwellEventInfo?
    var transformations : String = ""
    var md5 : String = ""
    var flags: Int32 = 0
    var rating: Int32 = 0
    var title: String = ""
    var raw_backlinks : String = ""
    var backlinks : ShotwellBacklinks?
    var comment : String = ""
    var has_gps : Bool = false
    var gps_lat : Double = 0.0
    var gps_lon : Double = 0.0
}

and function in the ShotwellDatabase class:

    func db_get_next_photo(db: OpaquePointer?) -> ShotwellPhotoInfo? {
        var r = ShotwellPhotoInfo()
        if sqlite3_step(import_query) == SQLITE_ROW {
            r.id = sqlite3_column_int(import_query, 0)
            r.filename = String(cString: sqlite3_column_text(import_query, 1))
            r.orientation = sqlite3_column_int(import_query, 2)
            r.original_orientation = sqlite3_column_int(import_query, 3)
            r.import_id = sqlite3_column_int(import_query, 4)
            r.event_id = sqlite3_column_int(import_query, 5)
// FIXME:            r.event = get_event_info(db: db, id: r.event_id)
            let transformation = sqlite3_column_text(import_query, 6)
            if let t = transformation {
                r.transformations = String(cString: t)
            }
            let md5 = sqlite3_column_text(import_query, 7)
            if let t = md5 {
                r.md5 = String(cString: t)
            }
            r.flags = sqlite3_column_int(import_query, 8)
            r.rating = sqlite3_column_int(import_query, 9)
            let title = sqlite3_column_text(import_query, 10)
            if let t = title {
                r.title = String(cString: t)
            }
            let backlinks = sqlite3_column_text(import_query, 11)
            if let t = backlinks {
                r.raw_backlinks = String(cString: t)
                r.backlinks = process_backlinks(all_backlinks: r.raw_backlinks)
            }
            let comment = sqlite3_column_text(import_query, 12)
            if let t = comment {
                r.comment = String(cString: t)
            }
            r.has_gps = sqlite3_column_int(import_query, 13) != 0
            r.gps_lat = sqlite3_column_double(import_query, 14)
            r.gps_lon = sqlite3_column_double(import_query, 15)
        } else {
            return nil
        }
        return r
    }

So with that combined, I ended up with my global var all_photos: [Int32: ShotwellPhotoInfo] = [:] containing all the photos in the shotwell database able to be looked up by ID. I’m sure there’s a more idiomatic way to do this in Swift… but the learning as I went, and need to only ever run this once certainly encouraged me to take some shortcuts.

There’s two interesting bits in that code. There’s the process_backlinks function, which does the thing of processing this giant text string that Shotwell stores that points the photo back to everywhere it has been referenced, as well as the FIXME around event information. I ended up giving up on translating Events, as these were never something I used much.

Parsing the backlinks I did with this:

    func process_backlinks(all_backlinks : String) -> ShotwellBacklinks {
        var r = ShotwellBacklinks()
        let backlinks = all_backlinks.split(separator: "\n").map( {String($0)})
        
        for backlink in backlinks {
            let kv = backlink.split(separator: "=")
            switch kv.first {
            case "tag":
                let tag_ids = kv[1].split(separator: "|").map( {Int32($0)})
                for tag_id in tag_ids {
                    if tag_id != nil {
                        if tag_id! > 0 {
                            r.tag.append(all_tags[tag_id!])
                        }
                    }
                }
                break
            case "event":
                let events = kv[1].split(separator: "|").map( {String($0)})
                r.event = events
                break
            default:
                r.other.append(contentsOf: kv.map({String($0)}))
            }
        }
        
        return r
    }

and the very simple ShotwellBacklinks class:

class ShotwellBacklinks : Codable {
    var tag : [ShotwellTag?] = []
    var event : [String] = []
    var other : [String] = []
}

But to start to get a view as to what Tags exist in the shotwell database, we can just go through the TagTable in the Shotwell SQLite database (added to the ShotwellDatabase class above):

    func get_all_tags() {
        var db: OpaquePointer?
        if sqlite3_open(dbPath, &db) == SQLITE_OK {
            print("Opened Shotwell Database!")
        }
        let query = "SELECT id, name, photo_id_list, time_created FROM TagTable"
        var q : OpaquePointer?
        
        if sqlite3_prepare_v2(db, query, -1, &q, nil) != SQLITE_OK {
            print("Failed to prepare query")
            assert(false)
        }
        
        while sqlite3_step(q) == SQLITE_ROW {
            let r = tag_row_to_ShotwellTag(q: q)
            
            all_tags[r.id] = r
            for photo_id in r.photo_id_list {
                photoToTags[photo_id]?.append(r)
            }
        }
        sqlite3_finalize(q)
        sqlite3_close(db)
    }

I did the hacky thing and just added var all_tags: [Int32: ShotwellTag] = [:] as a global variable, that I then used in the UI element showing them (well… via a wrapper, but that’s how that goes).

We just now need the tag_row_to_ShotwellTag that’s called above:

    func tag_row_to_ShotwellTag(q: OpaquePointer?) -> ShotwellTag {
        let id = sqlite3_column_int(q, 0)
        let c_name = sqlite3_column_text(q, 1)
        let c_photo_id_list = sqlite3_column_text(q, 2)
        let time_created = sqlite3_column_int(q, 3  )
        var name : String = ""
        var photo_id_list : [Int32] = []
        var photos : [ShotwellPhotoInfo] = []
        
        if let c_name = c_name {
            name = String(cString: c_name)
        }
        
        if let c_photo_id_list = c_photo_id_list {
            let photo_ids = String(cString: c_photo_id_list).split(separator: ",")
            photo_id_list.reserveCapacity(photo_ids.count)
            for photo_id in photo_ids {
                if photo_id.hasPrefix("thumb") {
                    var id = photo_id
                    id.removeFirst(5)
                    photo_id_list.append(Int32(id, radix: 16)!)
                    let photo = all_photos[Int32(id, radix: 16)!]
                    if photo == nil {
                        print("Couldn't find photo id " + id)
                    } else {
                        photos.append(photo!)
                    }
                } else {
                    print("UNKNOWN PREFIX FOR " + photo_id)
                }
            }
        }
        return ShotwellTag(id: id, name: name, photo_id_list: photo_id_list, time_created: time_created, photos: photos)
    }
    

Using llvm-mca for predicting CPU cycle impact of code changes

Way back in the distant past, when the Apple ][ and the Commodore 64 were king, you could read the manual for a microprocessor and see how many CPU cycles each instruction took, and then do the math as to how long a sequence of instructions would take to execute. This cycle counting was used pretty effectively to do really neat things such as how you’d get anything on the screen from an Atari 2600. Modern CPUs are… complex. They can do several things at once, in a different order than what you wrote them in, and have an interesting arrangement of shared resources to allocate.

So, unlike with simpler hardware, if you have a sequence of instructions for a modern processor, it’s going to be pretty hard to work out how many cycles that could take by hand, and it’s going to differ for each micro-architecture available for the instruction set.

When designing a microprocessor, simulating what a series of existing instructions will take to execute compared to the previous generation of microprocessor is pretty important. The aim should be for it to take less time or energy or some other metric that means your new processor is better than the old one. It can be okay if processor generation to generation some sequence of instructions take more cycles, if your cycles are more frequent, or power efficient, or other positive metric you’re designing for.

Programmers may want this simulation too, as some code paths get rather performance critical for certain applications. Open Source tools for this aren’t as prolific as I’d like, but there is llvm-mca which I (relatively) recently learned about.

llvm-mca is a performance analysis tool that uses information available in LLVM (e.g. scheduling models) to statically measure the performance of machine code in a specific CPU.

the llvm-mca docs

So, when looking at an issue in the IPv6 address and connection hashing code in Linux last year, and being quite conscious of modern systems dealing with a LOT of network packets, and thus this can be quite CPU usage sensitive, I wanted to make sure that my suggested changes weren’t going to have a large impact on performance – across the variety of CPU generations in use.

There’s two ways to do this: run everything, throw a lot of packets at something, and measure it. That can be a long dev cycle, and sometimes just annoying to get going. It can be a lot quicker to simulate the small section of code in question and do some analysis of it before going through the trouble of spinning up multiple test environments to prove it in the real world.

So, enter llvm-mca and the ability to try and quickly evaluate possible changes before testing them. Seeing as the code in question was nicely self contained, I could easily get this to a point where I could easily get gcc (or llvm) to spit out assembler for it separately from the kernel tree. My preference was for gcc as that’s what most distros end up compiling Linux with, including the Linux distribution that’s my day job (Amazon Linux).

In order to share the results of the experiments as part of the discussion on where the code changes should end up, I published the code and results in a github project as things got way too large to throw on a mailing list post and retain sanity.

I used a container so that I could easily run it in a repeatable isolated environment, as well as have others reproduce my results if needed. Different compiler versions and optimization levels will very much produce different sequences of instructions, and thus possibly quite different results. This delta in compiler optimization levels is partially why the numbers don’t quite match on some of the mailing list messages, although the delta of the various options was all the same. The other reason is learning how to better use llvm-mca to isolate down the exact sequence of instructions I was caring about (and not including things like the guesswork that llvm-mca has to do for branches).

One thing I learned along the way is how to better use llvm-mca to get the results that I was looking for. One trick is to very much avoid branches, as that’s going to be near complete guesswork as there’s not a simulation of the branch predictor (at least in the version I was using.

The big thing I wanted to prove: is doing the extra work having a small or large impact on number of elapsed cycles. The answer was that doing a bunch of extra “work” was essentially near free. The CPU core could execute enough things in parallel that the incremental cost of doing extra work just… wasn’t relevant.

This helped getting a patch deployed without impact to performance, as well as get a patch upstream, fixing an issue that was partially fixed 10 years prior, and had existed since day 1 of the Linux IPv6 code.

Naturally, this wasn’t a solo effort, and that’s one of the joys of working with a bunch of smart people – both at the same company I work for, and in the broader open source community. It’s always humbling when you’re looking at code outside your usual area of expertise that was written (and then modified) by Really Smart People, and you’re then trying to fix a problem in it, while trying to learn all the implications of changing that bit of code.

Anyway, check out llvm-mca for your next adventure into premature optimization, as if you’re going to get started with evil, you may as well start with what’s at the root of all of it.

Personal Finance Apps

I (relatively) recently went down the rabbit hole of trying out personal finance apps to help get a better grip on, well, the things you’d expect (personal finances and planning around them).

In the past, I’ve had an off-again-on-again relationship with GNUCash. I did give it a solid go for a few months in 2004/2005 it seems (I found my old files) and I even had the OFX exports of transactions for a limited amount of time for a limited number of bank accounts! Amazingly, there’s a GNUCash port to macOS, and it’ll happily open up this file from what is alarmingly close to 20 years ago.

Back in those times, running Linux on the desktop was even more of an adventure than it has been since then, and I always found GNUCash to be strange (possibly a theme with me and personal finance software), but generally fine. It doesn’t seem to have changed a great deal in the years since. You still have to manually import data from your bank unless you happen to be lucky enough to live in the very limited number of places where there’s some kind of automation for it.

So, going back to GNUCash was an option. But I wanted to survey the land of what was available, and if it was possible to exchange money for convenience. I am not big on the motivation to go and spend a lot of time on this kind of thing anyway, so it had to be easy for me to do so.

For my requirements, I basically had:

  • Support multiple currencies
  • Be able to import data from my banks, even if manually
  • Some kind of reporting and planning tools
  • Be easy enough to use for me, and not leave me struggling with unknown concepts
  • The ability to export data. No vendor lock-in

I viewed a mobile app (iOS) as a Nice to Have rather than essential. Given that, my shortlist was:

GNUCash

I’ve used it before, its web site at https://www.gnucash.org/ looks much the same as it always has. It’s Free and Open Source Software, and is thus well aligned with my values, and that’s a big step towards not having vendor lock-in.

I honestly could probably make it work. I wish it had the ability to import transactions from banks for anywhere I have ever lived or banked with. I also wish the UI got to be a bit more consistent and modern, and even remotely Mac like on the Mac version.

Honestly, if the deal was that a web service would pull bank transactions in exchange for ~$10/month and also fund GNUCash development… I’d struggle to say no.

Quicken

Here’s an option that has been around forever – https://www.quicken.com/ – and one that I figured I should solidly look at. It’s actually one I even spent money on…. before requesting a refund. It’s Import/Export is so broken it’s an insult to broken software everywhere.

Did you know that Quicken doesn’t import the Quicken Interchange Format (QIF), and hasn’t since 2005?

Me, incredulously, when trying out quicken

I don’t understand why you wouldn’t support as many as possible formats that banks export your transaction data as. It cannot possibly be that hard to parse these things, nor can it possibly be code that requires a lot of maintenance.

This basically meant that I couldn’t import data from my Australian Banks. Urgh. This alone ruled it out.

It really didn’t build confidence in ever getting my data out. At every turn it seemed to be really keen on locking you into Quicken rather than having a good experience all-up.

Moneywiz

This one was new to me – https://www.wiz.money/ – and had a fancy URL and everything. I spent a bunch of time trying MoneyWiz, and I concluded that it is pretty, but buggy. I had managed to create a report where it said I’d earned $0, but you click into it, and then it gives actual numbers. Not being self consistent and getting the numbers wrong, when this is literally the only function of said app (to get the numbers right), took this out of the running.

It did sync from my US and Australian banks though, so points there.

Intuit Mint

Intuit used to own Quicken until it sold it to H.I.G. Capital in 2016 (according to Wikipedia). I have no idea if that has had an impact as to the feature set / usability of Quicken, but they now have this Cloud-only product called Mint.

The big issue I had with Mint was that there didn’t seem to be any way to get your data out of it. It seemed to exemplify vendor lock-in. This seems to have changed a bit since I was originally looking, which is good (maybe I just couldn’t find it?). But with the cloud-only approach I wasn’t hugely comfortable with having everything there. It also seemed to be lacking a few features that I was begging to find useful in other places.

It is the only product that links with the Apple Card though. No idea why that is the case.

The price tag of $0 was pretty unbeatable, which does make me wonder where the money is made from to fund its development and maintenance. My guess is that it’s through commission on the various financial products advertised through it, and I dearly hope it is not through selling data on its users (I have no reason to believe it is, there’s just the popular habit of companies doing this).

Banktivity

This is what I’ve settled on. It seemed to be easy enough for me to figure out how to use, sync with an iPhone App, be a reasonable price, and be able to import and sync things from accounts that I have. Oddly enough, nothing can connect and pull things from the Apple Card – which is really weird. That isn’t a Banktivity thing though, that’s just universal (except for Intuit’s Mint).

I’ve been using it for a bit more than a year now, and am still pretty happy. I wish there was the ability to attach a PDF of a statement to the Statement that you reconcile. I wish I could better tune the auto match/classification rules, and a few other relatively minor things.

Fitness watches and my descent into madness

Periodically in life I’ve had the desire to be somewhat fit, or at least have the benefits that come with that such as not dying early and being able to navigate a mountain (or just the city of Seattle) on foot without collapsing. I have also found that holding myself accountable via data is pretty vital to me actually going and repeatedly doing something.

So, at some point I got myself a Garmin watch. The year was 2012 and it was a Garmin Forerunner 410. It had a standard black/grey LCD screen, GPS (where getting a GPS lock could be utterly infuriatingly slow), a sensor you attached to your foot, a sensor you strap to your chest for Heart Rate monitoring, and an ANT+ dongle for connecting to a PC to download your activities. There was even some open source software that someone wrote so I could actually get data off my watch on my Linux laptops. This wasn’t a smart watch – it was exclusively for wearing while exercising and tracking an activity, otherwise it was just a watch.

However, as I was ramping up to marathon distance running, one huge flaw emerged: I was not fast enough to run a marathon in the time that the battery in my Garmin lasted. IIRC it would end up dying around 3hr30min into something, which at the time was increasingly something I’d describe as “not going for too long of a run”. So, the search for a replacement began!

The year was 2017, and the Garmin fenix 5x attracted me for two big reasons: a battery life to be respected, and turn-by-turn navigation. At the time, I seldom went running with a phone, preferring a tiny SanDisk media play (RIP, they made a new version that completely sucked) and a watch. The attraction of being able to get better maps back to where I started (e.g. a hotel in some strange city where I didn’t speak the language) was very appealing. It also had (what I would now describe as) rudimentary smart-watch features. It didn’t have even remotely everything the Pebble had, but it was enough.

So, a (non-trivial) pile of money later (even with discounts), I had myself a shiny and virtually indestructible new Garmin. I didn’t even need a dongle to sync it anywhere – it could just upload via its own WiFi connection, or through Bluetooth to the Garmin Connect app to my phone. I could also (if I ever remembered to), plug in the USB cable to it and download the activities to my computer.

One problem: my skin rebelled against the Garmin fenix 5x after a while. Like, properly rebelled. If it wasn’t coming off, I wanted to rip it off. I tried all of the tricks that are posted anywhere online. Didn’t help. I even got tested for what was the most likely culprit (a Nickel allergy), and didn’t have one of them, so I (still) have no idea what I’m actually allergic to in it. It’s just that I cannot wear it constantly. Urgh. I was enjoying the daily smart watch uses too!

So, that’s one rather expensive watch that is special purpose only, and even then started to get to be a bit of an issue around longer activities. Urgh.

So the hunt began for a smart watch that I could wear constantly. This usually ends in frustration as anything I wanted was hundreds of $ and pretty much nobody listed what materials were in it apart from “stainless steel”, “may contain”, and some disclaimer about “other materials”, which wasn’t a particularly useful starting point for “it is one of these things that my skin doesn’t like”. As at least if the next one also turned out to cause me problems, I could at least have a list of things that I could then narrow down to what I needed to avoid.

So that was all annoying, with the end result being that I went a long time without really wearing a watch. Why? The search resumed periodically and ended up either with nothing, or totally nothing. That was except if I wanted to get further into some vendor lock-in.

Honestly, the only manufacturer of anything smartwatch like which actually listed everything and had some options was Apple. Bizarre. Well, since I already got on the iPhone bandwagon, this was possible. Rather annoyingly, they are very tied together and thus it makes it a bit of a vendor-lock-in if you alternate phone and watch replacement and at any point wish to switch platforms.

That being said though, it does work well and not irritate my skin. So that’s a bonus! If I get back into marathon level distance running, we’ll see how well it goes. But for more common distances that I’ve run or cycled with it… the accuracy seems decent, HR monitor never just sometimes decides I’m not exerting myself, and the GPS actually gets a lock in reasonable time. Plus it can pair with headphones and be the only thing I take out with me.

Getting your photos out of Shotwell

Somewhat a while ago now, I wrote about how every time I return to write some software for the Mac, the preferred language has changed. The purpose of this adventure was to get my photos out of the aging Shotwell and onto my (then new) Mac and the Apple Photos App.

I’ve had a pretty varied experience with photo management on Linux over the past couple of decades. For a while I used f-spot as it was the new hotness. At some point this became…. slow and crashy enough that it was unusable. Today, it appears that the GitHub project warns that current bugs include “Not starting”.

At some point (and via a method I have long since forgotten), I did manage to finally get my photos over to Shotwell, which was the new hotness at the time. That data migration was so long ago now I actually forget what features I was missing from f-spot that I was grumbling about. I remember the import being annoying though. At some point in time Shotwell was no longer was the new hotness and now there is GNOME Photos. I remember looking at GNOME Photos, and seeing no method of importing photos from Shotwell, so put it aside. Hopefully that situation has improved somewhere.

At some point Shotwell was becoming rather stagnated, and I noticed more things stopping to work rather than getting added features and performance. The good news is that there has been some more development activity on Shotwell, so hopefully my issues with it end up being resolved.

One recommendation for Linux photo management was digiKam, and one that I never ended up using full time. One of the reasons behind that was that I couldn’t really see any non manual way to import photos from Shotwell into it.

With tens of thousands of photos (~58k at the time of writing), doing things manually didn’t seem like much fun at all.

As I postponed my decision, I ended up moving my main machine over to a Mac for a variety of random reasons, and one quite motivating thing was the ability to have Photos from my iPhone magically sync over to my photo library without having to plug it into my computer and copy things across.

So…. how to get photos across from Shotwell on Linux to Photos on a Mac/iPhone (and also keep a very keen eye on how to do it the other way around, because, well, vendor lock-in isn’t great).

It would be kind of neat if I could just run Shotwell on the Mac and have some kind of import button, but seeing as there wasn’t already a native Mac port, and that Shotwell is written in Vala rather than something I know has a working toolchain on macOS…. this seemed like more work than I’d really like to take on.

Luckily, I remembered that Shotwell’s database is actually just a SQLite database pointing to all the files on disk. So, if I could work out how to read it accurately, and how to import all the relevant metadata (such as what Albums a photo is in, tags, title, and description) into Apple Photos, I’d be able to make it work.

So… is there any useful documentation as to how the database is structured?

Semi annoyingly, Shotwell is written in Vala, a rather niche programming language that while integrating with all the GObject stuff that GNOME uses, is largely unheard of. Luckily, the database code in Shotwell isn’t too hard to read, so was a useful fallback for when the documentation proves inadequate.

So, I armed myself with the following resources:

Programming the Mac side of things, it was a good excuse to start looking at Swift, so knowing I’d also need to read a SQLite database directly (rather than use any higher level abstraction), I armed myself with the following resources:

From here, I could work on getting the first half going, the ability to view my Shotwell database on the Mac (which is what I posted a screenshot of back in Feb 2022).

But also, I had to work out what I was doing on the other end of things, how would I import photos? It turns out there’s an API!

A bit of SwiftUI code:

import SwiftUI
import AppKit
import Photos

struct ContentView: View {
    @State var favorite_checked : Bool = false
    @State var hidden_checked : Bool = false
    var body: some View {
        VStack() {
            Text("Select a photo for import")
            Toggle("Favorite", isOn: $favorite_checked)
            Toggle("Hidden", isOn: $hidden_checked)
            Button("Import Photo")
            {
                let panel = NSOpenPanel()
                panel.allowsMultipleSelection = false
                panel.canChooseDirectories = false
                if panel.runModal() == .OK {
                    let photo_url = panel.url!
                    print("selected: " + String(photo_url.absoluteString))
                    addAsset(url: photo_url, isFavorite: favorite_checked, isHidden: hidden_checked)
                }
            }
            .padding()
        }
    }
}

struct ContentView_Previews: PreviewProvider {
    static var previews: some View {
        ContentView()
    }
}

Combined with a bit of code to do the import (which does look a bunch like the examples in the docs):

import SwiftUI
import Photos
import AppKit

@main
struct SinglePhotoImporterApp: App {
    var body: some Scene {
        WindowGroup {
            ContentView()
        }
    }
}

func addAsset(url: URL, isFavorite: Bool, isHidden: Bool) {
    // Add the asset to the photo library.
    let path = "/Users/stewart/Pictures/1970/01/01/1415446258647.jpg"
    let url = URL(fileURLWithPath: path)
    PHPhotoLibrary.shared().performChanges({
        let addedImage = PHAssetChangeRequest.creationRequestForAssetFromImage(atFileURL: url)
        addedImage?.isHidden = isHidden
        addedImage?.isFavorite = isFavorite
    }, completionHandler: {success, error in
        if !success { print("Error creating the asset: \(String(describing: error))") } else
        {
            print("Imported!")
        }
    })
}

This all meant I could import a single photo. However, there were some limitations.

There’s the PHAssetCollectionChangeRequest to do things to Albums, so it would solve that problem, but I couldn’t for the life of me work out how to add/edit Titles and Descriptions.

It was so close!

So what did I need to do in order to import Titles and Descriptions? It turns out you can do that via AppleScript. Yes, that thing that launched in 1993 and has somehow survived the transition of m68k based Macs to PowerPC based Macs to Intel based Macs to ARM based Macs.

The Photos dictionary for AppleScript

So, just to make it easier to debug what was going on, I started adding code to my ShotwellImporter tool that would generate snippets of AppleScript I could run and check that it was doing the right thing…. but then very quickly ran into a problem…. it appears that the AppleScript language interpreter on modern macOS has limits that you’d be more familiar with in 1993 than 2023, and I very quickly hit limits where the script would just error out before running (I was out of dictionary size allegedly).

But there’s a new option! Everything you can do with AppleScript you can now do with JavaScript – it’s just even less documented than AppleScript is! But it does work! I got to the point where I could generate JavaScript that imported photos, into all the relevant albums, and set title and descriptions.

A useful write up of using JavaScript rather than AppleScript to do things with Photos: https://mudge.name/2019/11/13/scripting-photos-for-macos-with-javascript/

More recent than when I was doing my hacking, https://alexwlchan.net/2023/managing-albums-in-photos/ is a good read.

With luck I’ll find some time to write up a bit of a walkthrough of my code, and push it up somewhere.

Adventures in the Apple Partition Map (Part 2 of the continuing adventures with the Apple Power Macintosh 7200/120 PC Compatible)

I “recently” wrote about obtaining a new (to me, actually quite old) computer over in The Apple Power Macintosh 7200/120 PC Compatible (Part 1). This post is a bit of a detour, but may help others understand why some images they download from the internet don’t work.

Disk partitioning is (of course) a way to divide up a single disk into multiple volumes (partitions) for different uses. While the idea is similar, computer platforms over the ages have done this in a variety of different ways, with varying formats on disk, and varying limitations. The ones that you’re most likely to be familiar with are the MBR partitioning scheme (from the IBM PC), and the GPT partitioning scheme (common for UEFI systems such as the modern PC and Mac). One you’re less likely to be familiar with is the Apple Partition Map scheme.

The way all IBM PCs and compatibles worked from the introduction of MS-DOS 2.0 in 1983 until some time after 2005 was the Master Boot Record partitioning scheme. It was outrageously simple: of the first 512 byte sector of a disk, the first 446 bytes was for the bootstrapping code (the “boot sector”), the last 2 bytes were for the magic two bytes telling the BIOS this disk was bootable, and the other 64 bytes were four entries of 16 bytes, each describing a disk partition. The Wikipedia page is a good overview of what it all looks like. Since “four partitions should be enough for anybody” wasn’t going to last, DOS 3.2 introduced “extended partitions” which was just using one of those 4 partitions as another similar data structure that could point to more partitions.

In the 1980s (similar to today), the Macintosh was, of course, different. The Apple Partition Map is significantly more flexible than the MBR on PCs. For a start, you could have more than four partitions! You could actually have a lot more than four partitions, as the Apple Partition Map is a single 512-byte sector for each partition, and the partition map is itself a partition. Instead of being block 0 (like the MBR is), it actually starts at block 1, and is contiguous (The Driver Descriptor Record is what’s at block 0). So, once created, it’s hard to extend. Typically it’d be created as 64×512-byte entries, for 32kb… which turns out is actually about enough for anyone.

The Inside Macintosh reference on the SCSI Manager goes through more detail as to these structures. If you’re wondering what language all the coding examples are in, it’s Pascal – which was fairly popular for writing Macintosh applications in back in the day.

But the actual partition map isn’t the “interesting” part of all this (and yes, the quotation marks are significant here), because Macs are pretty darn finicky about what disks to boot off, which gets to be interesting if you’re trying to find a CD-ROM image on the internet from which to boot, and then use to install an Operating System from.

An Unearthly Child

So, this idea has been brewing for a while now… try and watch all of Doctor Who. All of it. All 38 seasons. Today(ish), we started. First up, from 1963 (first aired not quite when intended due to the Kennedy assassination): An Unearthly Child. The first episode of the first serial.

A lot of iconic things are there from the start: the music, the Police Box, embarrassing moments of not quite remembering what time one is in, and normal humans accidentally finding their way into the TARDIS.

I first saw this way back when a child, where they were repeated on ABC TV in Australia for some anniversary of Doctor Who (I forget which one). Well, I saw all but the first episode as the train home was delayed and stopped outside Caulfield for no reason for ages. Some things never change.

Of course, being a show from the early 1960s, there’s some rougher spots. We’re not about to have the picture of diversity, and there’s going to be casual racism and sexism. What will be interesting is noticing these things today, and contrasting with my memory of them at the time (at least for episodes I’ve seen before), and what I know of the attitudes of the time.

“This year-ometer is not calculating properly” is a very 2020 line though (technically from the second episode).

libeatmydata v129

Every so often, I release a new libeatmydata. This has not happened for a long time. This is just some bug fixes, most of which have been in the Debian package for some time, I’ve just been lazy and not sat down and merged them.

git clone https://github.com/stewartsmith/libeatmydata.git

Download the source tarball from here: libeatmydata-129.tar.gz and GPG signature: libeatmydata-129.tar.gz.asc from my GPG key.

Or, feel free to grab some Fedora RPMs:

Releases published also in the usual places:

Photos from Taiwan

A few years ago we went to Taiwan. I managed to capture some random bits of the city on film (and also some shots on my then phone, a Google Pixel). I find the different style of art on the streets around the world to be fascinating, and Taiwan had some good examples.

I’ve really enjoyed shooting Kodak E100VS film over the years, and some of my last rolls were shot in Taiwan. It’s a film that unfortunately is not made anymore, but at least we have a new Ektachrome to have fun with now.

Words for our time: “Where there is democracy, equality and freedom can exist; without democracy, equality and freedom are merely empty words”.

This is, of course, only a small number of the total photos I took there. I’d really recommend a trip to Taiwan, and I look forward to going back there some day.

Why you should use `nproc` and not grep /proc/cpuinfo

There’s something really quite subtle about how the nproc utility from GNU coreutils works. If you look at the man page, it’s even the very first sentence:

Print the number of processing units available to the current process, which may be less than the number of online processors.

So, what does that actually mean? Well, just because the computer some code is running on has a certain number of CPUs (and here I mean “number of hardware threads”) doesn’t necessarily mean that you can spawn a process that uses that many. What’s a simple example? Containers! Did you know that when you invoke docker to run a container, you can easily limit how much CPU the container can use? In this case, we’re looking at the --cpuset-cpus parameter, as the --cpus one works differently.

$ nproc
8

$ docker run --cpuset-cpus=0-1 --rm=true -it  amazonlinux:2
bash-4.2# nproc
2
bash-4.2# exit

$ docker run --cpuset-cpus=0-2 --rm=true -it  amazonlinux:2
bash-4.2# nproc
3

As you can see, nproc here gets the right bit of information, so if you’re wanting to do a calculation such as “Please use up to the maximum available CPUs” as a parameter to the configuration of a piece of software (such as how many threads to run), you get the right number.

But what if you use some of the other common methods?

$ /usr/bin/lscpu -p | grep -c "^[0-9]"
8
$ grep -c 'processor' /proc/cpuinfo 
8

$ docker run --cpuset-cpus=0-1 --rm=true -it  amazonlinux:2
bash-4.2# yum install -y /usr/bin/lscpu
......
bash-4.2# /usr/bin/lscpu -p | grep -c "^[0-9]"
8
bash-4.2# grep -c 'processor' /proc/cpuinfo 
8
bash-4.2# nproc
2

In this case, if you base your number of threads off grepping lscpu you take another dependency (on the util-linux package), which isn’t needed. You also get the wrong answer, as you do by grepping /proc/cpuinfo. So, what this will end up doing is just increase the number of context switches, possibly also adding a performance degradation. It’s not just in docker containers where this could be an issue of course, you can use the same mechanism that docker uses anywhere you want to control resources of a process.

Another subtle thing to watch out for is differences in /proc/cpuinfo content depending on CPU architecture. You may not think it’s an issue today, but who wants to needlessly debug something?

tl;dr: for determining “how many processes to run”: use nproc, don’t grep lscpu or /proc/cpuinfo

Photos from long ago….

It’s strange to get unexpected photos from a while ago. It’s also joyous.

These photos above are from a park down the street from where we used to live. I believe it was originally a quarry, and a number of years ago the community got together and turned it into a park. It’s a quite decent size (Parkrun is held there), and there’s plenty of birds (and ducks!) to see.

Moorabbin Station

It’s a very strange feeling seeing photos from both the before time, and from where I used to live. I’m sure that if the world wasn’t the way it was now, and there wasn’t a pandemic, it would feel different.

All of the above were shot on a Nikon F80 with 35mm Fuji Velvia 50 film.

Refurbishing my Macintosh Plus

Somewhere in the mid to late 1990s I picked myself up a Macintosh Plus for the sum of $60AUD. At that time there were still computer Swap Meets where old and interesting equipment was around, so I headed over to one at some point (at the St Kilda Town Hall if memory serves) and picked myself up four 1MB SIMMs to boost the RAM of it from the standard 1MB to the insane amount of 4MB. Why? Umm… because I could? The RAM was pretty cheap, and somewhere in the house to this day, I sometimes stumble over the 256KB SIMMs as I just can’t bring myself to get rid of them.

This upgrade probably would have cost close to $2,000 at the system’s release. If the Macintosh system software were better at disk caching you could have easily held the whole 800k of the floppy disk in memory and still run useful software!

One of the annoying things that started with the Macintosh was odd screws and Apple gear being hard to get into. Compare to say, the Apple ][ which had handy clips to jump inside whenever. In fitting my massive FOUR MEGABYTES of RAM back in the day, I recall using a couple of allen keys sticky-taped together to be able to reach in and get the recessed Torx screws. These days, I can just order a torx bit off Amazon and have it arrive pretty quickly. Well, two torx bits, one of which is just too short for the job.

My (dusty) Macintosh Plus

One thing had always struck me about it, it never really looked like the photos of the Macintosh Plus I saw in books. In what is an embarrassing number of years later, I learned that a lot can be gotten from the serial number printed on the underside of the front of the case.

So heading over to the My Old Mac Serial Number Decoder I can find out:

Manufactured in: F => Fremont, California, USA
Year of production: 1985
Week of production: 14
Production number: 3V3 => 4457
Model ID: M0001WP => Macintosh 512K (European Macintosh ED)

Your Macintosh 512K (European Macintosh ED) was the 4457th Mac manufactured during the 14th week of 1985 in Fremont, California, USA.

Pretty cool! So it is certainly a Plus as the logic board says that, but it’s actually an upgraded 512k! If you think it was madness to have a GUI with only 128k of RAM in the original Macintosh, you’d be right. I do not envy anybody who had one of those.

Some time a decent (but not too many, less than 10) years ago, I turn on the Mac Plus to see if it still worked. It did! But then… some magic smoke started to come out (which isn’t so good), but the computer kept working! There’s something utterly bizarre about looking at a computer with smoke coming out of it that continues to function perfectly fine.

Anyway, as the smoke was coming out, I decided that it would be an opportune time to turn it off, open doors and windows, and put it away until I was ready to deal with it.

One Global Pandemic Later, and now was the time.

I suspected it was going to be a capacitor somewhere that blew, and figured that I should replace it, and probably preemptively replace all the other electrolytic capacitors that could likely leak and cause problems.

First thing’s first though: dismantle it and clean everything. First, taking the case off. Apple is not new to the game of annoying screws to get into things. I ended up spending $12 on this set on Amazon, as the T10 bit can actually reach the screws holding the case on.

Cathode Ray Tubes are not to be messed with. We’re talking lethal voltages here. It had been many years since electricity went into this thing, so all was good. If this all doesn’t work first time when reassembling it, I’m not exactly looking forward to discharging a CRT and working on it.

The inside of my Macintosh Plus, with lots of grime.

You can see there’s grime everywhere. It’s not the worst in the world, but it’s not great (and kinda sticky). Obviously, this needs to be cleaned! The best way to do that is take a lot of photos, dismantle everything, and clean it a bit at a time.

There’s four main electronic components inside a Macintosh Plus:

  1. The CRT itself
  2. The floppy disk drive
  3. The Logic Board (what Mac people call what PC people call the motherboard)
  4. The Analog Board

There’s also some metal structure that keeps some things in place. There’s only a few connectors between things, which are pretty easy to remove. If you don’t know how to discharge a CRT and what the dangers of them are you should immediately go and find out through reading rather than finding out by dying. I would much prefer it if you dyed (because creative fun) rather than died.

Once the floppy connector and the power connector is unplugged, the logic board slides out pretty easily. You can see from the photo below that I have the 4MB of RAM installed and the resistor you need to snip is, well, snipped (but look really closely for that). Also, grime.

Macintosh Plus Logic Board

Cleaning things? Well, there’s two ways that I have used (and considering I haven’t yet written the post with “hurray, it all works”, currently take it with a grain of salt until I write that post). One: contact cleaner. Two: detergent.

Macintosh Plus Logic Board (being washed in my sink)

I took the route of cleaning things first, and then doing recapping adventures. So it was some contact cleaner on the boards, and then some soaking with detergent. This actually all worked pretty well.

Logic Board Capacitors:

  • C5, C6, C7, C12, C13 = 33uF 16V 85C (measured at 39uF, 38uF, 38uF, 39uF)
  • C14 = 1uF 50V (measured at 1.2uF and then it fluctuated down to around 1.15uF)

Analog Board Capacitors

  • C1 = 35V 3.9uF (M) measured at 4.37uF
  • C2 = 16V 4700uF SM measured at 4446uF
  • C3 = 16V 220uF +105C measured at 234uF
  • C5 = 10V 47uF 85C measured at 45.6uF
  • C6 = 50V 22uF 85C measured at 23.3uF
  • C10 = 16V 33uF 85C measured at 37uF
  • C11 = 160V 10uF 85C measured at 11.4uF
  • C12 = 50V 22uF 85C measured at 23.2uF
  • C18 = 16V 33uF 85C measured at 36.7uF
  • C24 = 16V 2200uF 105C measured at 2469uF
  • C27 = 16V 2200uF 105C measured at 2171uF (although started at 2190 and then went down slowly)
  • C28 = 16V 1000uF 105C measured at 638uF, then 1037uF, then 1000uF, then 987uF
  • C30 = 16V 2200uF 105C measured at 2203uF
  • C31 = 16V 220uF 105C measured at 236uF
  • C32 = 16V 2200uF 105C measured at 2227uF
  • C34 = 200V 100uF 85C measured at 101.8uF
  • C35 = 200V 100uF 85C measured at 103.3uF
  • C37 = 250V 0.47uF measured at <exploded>. wheee!
  • C38 = 200V 100uF 85C measured at 103.3uF
  • C39 = 200V 100uF 85C mesaured at 99.6uF (with scorch marks from next door)
  • C42 = 10V 470uF 85C measured at 556uF
  • C45 = 10V 470uF 85C measured at 227uF, then 637uF then 600uF

I’ve ordered an analog board kit from https://console5.com/store/macintosh-128k-512k-plus-analog-pcb-cap-kit-630-0102-661-0462.html and when trying to put them in, I learned that the US Analog board is different to the International Analog board!!! Gah. Dammit.

Note that C30, C32, C38, C39, and C37 were missing from the kit I received (probably due to differences in the US and International boards). I did have an X2 cap (for C37) but it was 0.1uF not 0.47uF. I also had two extra 1000uF 16V caps.

Macintosh Repair and Upgrade Secrets (up to the Mac SE no less!) holds an Appendix with the parts listing for both the US and International Analog boards, and this led me to conclude that they are in fact different boards rather than just a few wires that are different. I am not sure what the “For 120V operation, W12 must be in place” and “for 240V operation, W12 must be removed” writing is about on the International Analog board, but I’m not quite up to messing with that at the moment.

So, I ordered the parts (linked above) and waited (again) to be able to finish re-capping the board.

I found https://youtu.be/H9dxJ7uNXOA video to be a good one for learning a bunch about the insides of compact Macs, I recommend it and several others on his YouTube channel. One interesting thing I learned is that the X2 cap (C37 on the International one) is before the power switch, so could blow just by having the system plugged in and not turned on! Okay, so I’m kind of assuming that it also applies to the International board, and mine exploded while it was plugged in and switched on, so YMMV.

Additionally, there’s an interesting list of commonly failing parts. Unfortunately, this is also for the US logic board, so the tables in Macintosh Repair and Upgrade Secrets are useful. I’m hoping that I don’t have to replace anything more there, but we’ll see.

But, after the Nth round of parts being delivered….

Note the lack of an exploded capacitor

Yep, that’s where the exploded cap was before. Cleanup up all pretty nicely actually. Annoyingly, I had to run it all through a step-up transformer as the board is all set for Australian 240V rather than US 120V. This isn’t going to be an everyday computer though, so it’s fine.

Macintosh Plus booting up (note how long the memory check of 4MB of RAM takes. I’m being very careful as the cover is off. High, and possibly lethal voltages exposed.

Woohoo! It works. While I haven’t found my supply of floppy disks that (at least used to) work, the floppy mechanism also seems to work okay.

Macintosh Plus with a seemingly working floppy drive mechanism. I haven’t found a boot floppy yet though.

Next up: waiting for my Floppy Emu to arrive as it’ll certainly let it boot. Also, it’s now time to rip the house apart to find a floppy disk that certainly should have made its way across the ocean with the move…. Oh, and also to clean up the mouse and keyboard.

My POWER9 CPU Core Layout

So, following on from my post on Sensors on the Blackbird (and thus Power9), I mentioned that when you look at the temperature sensors for each CPU core in my 8-core POWER9 chip, they’re not linear numbers. Let’s look at what that means….

stewart@blackbird9$ sudo ipmitool sensor | grep core
 p0_core0_temp            | na                                                                                                               
 p0_core1_temp            | na                                                                                                               
 p0_core2_temp            | na                                                                                                               
 p0_core3_temp            | 38.000                                                                                                           
 p0_core4_temp            | na          
 p0_core5_temp            | 38.000      
 p0_core6_temp            | na          
 p0_core7_temp            | 38.000      
 p0_core8_temp            | na          
 p0_core9_temp            | na          
 p0_core10_temp           | na          
 p0_core11_temp           | 37.000      
 p0_core12_temp           | na          
 p0_core13_temp           | na          
 p0_core14_temp           | na          
 p0_core15_temp           | 37.000      
 p0_core16_temp           | na          
 p0_core17_temp           | 37.000      
 p0_core18_temp           | na          
 p0_core19_temp           | 39.000      
 p0_core20_temp           | na          
 p0_core21_temp           | 39.000      
 p0_core22_temp           | na          
 p0_core23_temp           | na        

You can see I have eight CPU cores in my Blackbird system. The reason the 8 CPU cores are core 3, 5, 7, 11, 15, 17, 19, and 21 rather than 0-8 or something is that these represent the core numbers on the physical die, and the die is a 24 core die. When you’re making a chip as big and as complex as modern high performance CPUs, not all of the chips coming out of your fab are going to be perfect, so this is how you get different models in the line with only one production line.

Weirdly, the output from the hwmon sensors and why there’s a “core 24” and a “core 28”. That’s just… wrong. What it is, however, is right if you think of 8*4=32. This is a product of Linux thinking that Thread=Core in some ways. So, yeah, this numbering is the first thread of each logical core.

[stewart@blackbird9 ~]$ sensors|grep -i core
 Chip 0 Core 0:            +39.0°C  (lowest = +25.0°C, highest = +71.0°C)
 Chip 0 Core 4:            +39.0°C  (lowest = +26.0°C, highest = +66.0°C)
 Chip 0 Core 8:            +39.0°C  (lowest = +27.0°C, highest = +67.0°C)
 Chip 0 Core 12:           +39.0°C  (lowest = +26.0°C, highest = +67.0°C)
 Chip 0 Core 16:           +39.0°C  (lowest = +25.0°C, highest = +67.0°C)
 Chip 0 Core 20:           +39.0°C  (lowest = +26.0°C, highest = +69.0°C)
 Chip 0 Core 24:           +39.0°C  (lowest = +27.0°C, highest = +67.0°C)
 Chip 0 Core 28:           +39.0°C  (lowest = +27.0°C, highest = +64.0°C)

But let’s ignore that, go from the IPMI sensors (which also match what the OCC shows with “occtoolp9 -LS” (see below).

$ ./occtoolp9 -SL
Sensor Details: (found 86 sensors, details only for Status of 0x00)                                           
     GUID Name             Sample     Min    Max U    Stat   Accum     UpdFreq   ScaleFactr   Loc   Type 
....
   0x00ED TEMPC03………     47      29     47 C    0x00 0x00037CF2 0x00007D00 0x00000100 0x0040 0x0008
   0x00EF TEMPC05………     37      26     39 C    0x00 0x00014E53 0x00007D00 0x00000100 0x0040 0x0008
   0x00F1 TEMPC07………     46      28     46 C    0x00 0x0001A777 0x00007D00 0x00000100 0x0040 0x0008
   0x00F5 TEMPC11………     44      27     45 C    0x00 0x00018402 0x00007D00 0x00000100 0x0040 0x0008
   0x00F9 TEMPC15………     36      25     43 C    0x00 0x000183BC 0x00007D00 0x00000100 0x0040 0x0008
   0x00FB TEMPC17………     38      28     41 C    0x00 0x00015474 0x00007D00 0x00000100 0x0040 0x0008
   0x00FD TEMPC19………     43      27     44 C    0x00 0x00016589 0x00007D00 0x00000100 0x0040 0x0008
   0x00FF TEMPC21………     36      30     40 C    0x00 0x00015CA9 0x00007D00 0x00000100 0x0040 0x0008

So what does that mean for physical layout? Well, like all modern high performance chips, the POWER9 is modular, with a bunch of logic being replicated all over the die. The most notable duplicated parts are the core (replicated 24 times!) and cache structures. Less so are memory controllers and PCI hardware.

P9 chip layout from page 31 of the POWER9 Register Specification

See that each core (e.g. EC00 and EC01) is paired with the cache block (EC00 and EC01 with EP00). That’s two POWER9 cores with one 512KB L2 cache and one 10MB L3 cache.

You can see the cache layout (including L1 Instruction and Data caches) by looking in sysfs:

$ for i in /sys/devices/system/cpu/cpu0/cache/index*/; \
  do echo -n $(cat $i/level) $(cat $i/size) $(cat $i/type); \
  echo; done
 1 32K Data
 1 32K Instruction
 2 512K Unified
 3 10240K Unified

So, what does the layout of my POWER9 chip look like? Well, thanks to the power of graphics software, we can cross some cores out and look at the topology:

My 8-core POWER9 CPU in my Raptor Blackbird

If I run some memory bandwidth benchmarks, I can see that you can see the L3 cache capacity you’d assume from the above diagram: 80MB (10MB/core). Let’s see:

[stewart@blackbird9 lmbench3]$ for i in 5M 10M 20M 30M 40M 50M 60M 70M 80M 500M; \
  do echo -n "$i   "; \
  ./bin/bw_mem -N 100  $i rd; \
done
  5M    5.24 63971.98
 10M   10.49 31940.14
 20M   20.97 17620.16
 30M   31.46 18540.64
 40M   41.94 18831.06
 50M   52.43 17372.03
 60M   62.91 16072.18
 70M   73.40 14873.42
 80M   83.89 14150.82
 500M 524.29 14421.35

If all the cores were packed together, I’d expect that cliff to be a lot sooner.

So how does this compare to other machines I have around? Well, let’s look at my Ryzen 7. Specifically, a “AMD Ryzen 7 1700 Eight-Core Processor”. The cache layout is:

$ for i in /sys/devices/system/cpu/cpu0/cache/index*/; \
  do echo -n $(cat $i/level) $(cat $i/size) $(cat $i/type); \
  echo; \
done
 1 32K Data
 1 64K Instruction
 2 512K Unified
 3 8192K Unified

And then the performance benchmark similar to the one I ran above on the POWER9 (lower numbers down low as 8MB is less than 10MB)

$ for i in 4M 8M 16M 24M 32M 40M 48M 56M 64M 72M 80M 500M; \
  do echo -n "$i   "; ./bin/x86_64-linux-gnu/bw_mem -N 10  $i rd;\
done
  4M    4.19 61111.04
  8M    8.39 28596.55
 16M   16.78 21415.12
 24M   25.17 20153.57
 32M   33.55 20448.20
 40M   41.94 20940.11
 48M   50.33 20281.39
 56M   58.72 21600.24
 64M   67.11 21284.13
 72M   75.50 20596.18
 80M   83.89 20802.40
 500M 524.29 21489.27

And my laptop? It’s a four core part, specifically a “Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz” with a cache layout like:

$ for i in /sys/devices/system/cpu/cpu0/cache/index*/; \
   do echo -n $(cat $i/level) $(cat $i/size) $(cat $i/type); \
     echo; \
   done
   1 32K Data
   1 32K Instruction
   2 256K Unified
   3 6144K Unified 
$ for i in 3M 6M 12M 18M 24M 30M 36M 42M 500M; \
  do echo -n "$i   "; ./bin/x86_64-linux-gnu/bw_mem -N 10  $i rd;\
done
  3M    3.15 48500.24
  6M    6.29 27144.16
 12M   12.58 18731.80
 18M   18.87 17757.74
 24M   25.17 17154.12
 30M   31.46 17135.87
 36M   37.75 16899.75
 42M   44.04 16865.44
 500M 524.29 16817.10

I’m not sure what performance conclusions we can realistically draw from these curves, apart from “keeping workload to L3 cache is cool”, and “different chips have different cache hardware”, and “I should probably go and read and remember more about the microarchitectural characteristics of the cache hardware in Ryzen 7 hardware and 10th gen Intel Core hardware”.

OCC and Sensors on the Raptor Blackbird (and other POWER9 systems)

This post we’re going to look at three different ways to look at various sensors in the Raptor Blackbird system. The Blackbird is a single socket uATX board for the POWER9 processor. One advantage of the system is completely open source firmware, so you can (like I have): build your own firmware. So, this is my Blackbird running my most recent firmware build (the BMC is running the 2.00 release from Raptor).

Sensors over IPMI

One way to get the sensors is over IPMI. This can be done either in-band (as in, from the OS running on the blackbird), or over the network.

stewart@blackbird9$ sudo ipmitool sensor |head
occ                      | na         | discrete   | na    | na        | na        | na        | na        | na        | na        
 occ0                     | 0x0        | discrete   | 0x0200| na        | na        | na        | na        | na        | na        
 occ1                     | 0x0        | discrete   | 0x0100| na        | na        | na        | na        | na        | na        
 p0_core0_temp            | na         |            | na    | na        | na        | na        | na        | na        | na        
 p0_core1_temp            | na         |            | na    | na        | na        | na        | na        | na        | na        
 p0_core2_temp            | na         |            | na    | na        | na        | na        | na        | na        | na        
 p0_core3_temp            | 38.000     | degrees C  | ok    | na        | -40.000   | na        | 78.000    | 90.000    | na        
 p0_core4_temp            | na         |            | na    | na        | na        | na        | na        | na        | na        
 p0_core5_temp            | 38.000     | degrees C  | ok    | na        | -40.000   | na        | 78.000    | 90.000    | na        
 p0_core6_temp            | na         |            | na    | na        | na        | na        | na        | na        | na    

It’s kind of annoying to read there, so standard unix tools to the rescue!

stewart@blackbird9$ sudo ipmitool sensor | cut -d '|' -f 1,2
 occ                      | na                                                                                                               
 occ0                     | 0x0                                                                                                              
 occ1                     | 0x0                                                                                                              
 p0_core0_temp            | na                                                                                                               
 p0_core1_temp            | na                                                                                                               
 p0_core2_temp            | na                                                                                                               
 p0_core3_temp            | 38.000                                                                                                           
 p0_core4_temp            | na          
 p0_core5_temp            | 38.000      
 p0_core6_temp            | na          
 p0_core7_temp            | 38.000      
 p0_core8_temp            | na          
 p0_core9_temp            | na          
 p0_core10_temp           | na          
 p0_core11_temp           | 37.000      
 p0_core12_temp           | na          
 p0_core13_temp           | na          
 p0_core14_temp           | na          
 p0_core15_temp           | 37.000      
 p0_core16_temp           | na          
 p0_core17_temp           | 37.000      
 p0_core18_temp           | na          
 p0_core19_temp           | 39.000      
 p0_core20_temp           | na          
 p0_core21_temp           | 39.000      
 p0_core22_temp           | na          
 p0_core23_temp           | na          
 p0_vdd_temp              | 40.000 
 dimm0_temp               | 35.000      
 dimm1_temp               | na          
 dimm2_temp               | na          
 dimm3_temp               | na          
 dimm4_temp               | 38.000      
 dimm5_temp               | na          
 dimm6_temp               | na          
 dimm7_temp               | na          
 dimm8_temp               | na          
 dimm9_temp               | na          
 dimm10_temp              | na          
 dimm11_temp              | na          
 dimm12_temp              | na          
 dimm13_temp              | na          
 dimm14_temp              | na          
 dimm15_temp              | na          
 fan0                     | 1200.000    
 fan1                     | 1100.000    
 fan2                     | 1000.000    
 p0_power                 | 33.000      
 p0_vdd_power             | 5.000       
 p0_vdn_power             | 9.000       
 cpu_1_ambient            | 30.600      
 pcie                     | 27.000      
 ambient                  | 26.000  

You can see that I have 3 fans, two DIMMs (although why it lists 16 possible DIMMs for a two DIMM slot board is a good question!), and eight CPU cores. More on why the layout of the CPU cores is the way it is in a future post.

The code path for reading these sensors is interesting, it’s all from the BMC, so we’re having the OCC inside the P9 read things, which the BMC then reads, and then passes back to the P9. On the P9 itself, each sensor is a call all the way to firmware and back! In fact, we can look at it in perf:

$ sudo perf record -g ipmitool sensor
$ sudo perf report --no-children
“ipmitool sensors” perf report

What are the 0x300xxxxx addresses? They’re the OPAL firmware (i.e. skiboot). We can look up the symbols easily, as the firmware exposes them to the kernel, which then plonks it in sysfs:

[stewart@blackbird9 ~]$ sudo head /sys/firmware/opal/symbol_map 
[sudo] password for stewart: 
0000000000000000 R __builtin_kernel_end
0000000000000000 R __builtin_kernel_start
0000000000000000 T __head
0000000000000000 T _start
0000000000000010 T fdt_entry
00000000000000f0 t boot_sem
00000000000000f4 t boot_flag
00000000000000f8 T attn_trigger
00000000000000fc T hir_trigger
0000000000000100 t sreset_vector

So we can easily look up exactly where this is:

[stewart@blackbird9 ~]$ sudo grep '18e.. ' /sys/firmware/opal/symbol_map 
 0000000000018e20 t .__try_lock.isra.0
 0000000000018e68 t .add_lock_request

So we’re managing to spend a whole 12% of execution time spinning on a spinlock in firmware! The call stack of what’s going on in firmware isn’t so easy, but we can find the bt_add_ipmi_msg call there which is probably how everything starts:

[stewart@blackbird9 ~]$ sudo grep '516.. ' /sys/firmware/opal/symbol_map   0000000000051614 t .bt_add_ipmi_msg_head  0000000000051688 t .bt_add_ipmi_msg  00000000000516fc t .bt_poll

OCCTOOL

This is the most not-what-you’re-meant-to-use method of getting access to sensors! It’s using a debug tool for the OCC firmware! There’s a variety of tools in the OCC source repositiory, and one of them (occtoolp9) can be used for a variety of things, one of which is getting sensor data out of the OCC.

$ sudo ./occtoolp9 -SL
     Sensor Type: 0xFFFF
 Sensor Location: 0xFFFF
     (only displaying non-zero sensors)
 Sending 0x53 command to OCC0 (via opal-prd)…
   MFG Sub Cmd: 0x05  (List Sensors)
   Num Sensors: 50
     [ 1] GUID: 0x0000 / AMEintdur…….  Sample:     20  (0x0014)
     [ 2] GUID: 0x0001 / AMESSdur0…….  Sample:      7  (0x0007)
     [ 3] GUID: 0x0002 / AMESSdur1…….  Sample:      3  (0x0003)
     [ 4] GUID: 0x0003 / AMESSdur2…….  Sample:     23  (0x0017)

The odd thing you’ll see is “via opal-prd” – and this is because it’s doing raw calls to the opal-prd binary to talk to the OCC firmware running things like “opal-prd --expert-mode htmgt-passthru“. Yeah, this isn’t a in-production thing :)

Amazingly (and interestingly), this doesn’t go through host firmware in the way that an IPMI call will. There’s a full OCC/Host firmware interface spec to read. But it’s insanely inefficient way to monity sensors, a long bash script shelling out to a whole bunch of other processes… Think ~14.4 billion cycles versus ~367million cycles for the ipmitool option above.

But there are some interesting sensors at the end of the list:

Sensor Details: (found 86 sensors, details only for Status of 0x00)                                                  
     GUID Name             Sample     Min    Max U    Stat   Accum     UpdFreq   ScaleFactr   Loc   Type   
....
   0x014A MRDM0………..    688       3  15015 GBs  0x00 0x0144AE6C 0x00001901 0x000080FB 0x0008 0x0200
   0x014E MRDM4………..    480       3  14739 GBs  0x00 0x01190930 0x00001901 0x000080FB 0x0008 0x0200
   0x0156 MWRM0………..    560       4  16605 GBs  0x00 0x014C61FD 0x00001901 0x000080FB 0x0008 0x0200
   0x015A MWRM4………..    360       4  16597 GBs  0x00 0x014AE231 0x00001901 0x000080FB 0x0008 0x0200

is that memory bandwidth? Well, if I run the STREAM benchmark in a loop and look again:

0x014A MRDM0………..  15165       3  17994 GBs  0x00 0x0C133D6C 0x00001901 0x000080FB 0x0008 0x0200
   0x014E MRDM4………..  17145       3  18016 GBs  0x00 0x0BF501D6 0x00001901 0x000080FB 0x0008 0x0200
   0x0156 MWRM0………..   8063       4  24280 GBs  0x00 0x07C98B88 0x00001901 0x000080FB 0x0008 0x0200
   0x015A MWRM4………..   1138       4  24215 GBs  0x00 0x07CE82AF 0x00001901 0x000080FB 0x0008 0x0200

It looks like it! Are these exposed elsewhere? Well, another blog post at some point in the future is where I should look at that.

lm-sensors

$ rpm -qf /usr/bin/sensors
 lm_sensors-3.5.0-6.fc31.ppc64le

Ahhh, old faithful lm-sensors! Yep, a whole bunch of sensors are just exposed over the standard interface that we’ve been using since ISA was a thing.

[stewart@blackbird9 ~]$ sensors                                                                  
 ibmpowernv-isa-0000                                       
 Adapter: ISA adapter                                      
 Chip 0 Vdd Remote Sense:  +1.02 V  (lowest =  +0.72 V, highest =  +1.02 V)
 Chip 0 Vdn Remote Sense:  +0.67 V  (lowest =  +0.67 V, highest =  +0.67 V)
 Chip 0 Vdd:               +1.02 V  (lowest =  +0.73 V, highest =  +1.02 V)
 Chip 0 Vdn:               +0.68 V  (lowest =  +0.68 V, highest =  +0.68 V)
 Chip 0 Core 0:            +47.0°C  (lowest = +25.0°C, highest = +71.0°C)            
 Chip 0 Core 4:            +47.0°C  (lowest = +26.0°C, highest = +66.0°C)            
 Chip 0 Core 8:            +48.0°C  (lowest = +27.0°C, highest = +67.0°C)            
 Chip 0 Core 12:           +48.0°C  (lowest = +26.0°C, highest = +67.0°C)            
 Chip 0 Core 16:           +47.0°C  (lowest = +25.0°C, highest = +67.0°C)                      
 Chip 0 Core 20:           +47.0°C  (lowest = +26.0°C, highest = +69.0°C)            
 Chip 0 Core 24:           +48.0°C  (lowest = +27.0°C, highest = +67.0°C)                     
 Chip 0 Core 28:           +51.0°C  (lowest = +27.0°C, highest = +64.0°C)                     
 Chip 0 DIMM 0 :           +40.0°C  (lowest = +34.0°C, highest = +44.0°C)                     
 Chip 0 DIMM 1 :            +0.0°C  (lowest =  +0.0°C, highest =  +0.0°C)                     
 Chip 0 DIMM 2 :            +0.0°C  (lowest =  +0.0°C, highest =  +0.0°C)
 Chip 0 DIMM 3 :            +0.0°C  (lowest =  +0.0°C, highest =  +0.0°C)
 Chip 0 DIMM 4 :            +0.0°C  (lowest =  +0.0°C, highest =  +0.0°C)
 Chip 0 DIMM 5 :            +0.0°C  (lowest =  +0.0°C, highest =  +0.0°C)
 Chip 0 DIMM 6 :            +0.0°C  (lowest =  +0.0°C, highest =  +0.0°C)
 Chip 0 DIMM 7 :            +0.0°C  (lowest =  +0.0°C, highest =  +0.0°C)
 Chip 0 DIMM 8 :            +0.0°C  (lowest =  +0.0°C, highest =  +0.0°C)
 Chip 0 DIMM 9 :            +0.0°C  (lowest =  +0.0°C, highest =  +0.0°C)
 Chip 0 DIMM 10 :           +0.0°C  (lowest =  +0.0°C, highest =  +0.0°C)
 Chip 0 DIMM 11 :           +0.0°C  (lowest =  +0.0°C, highest =  +0.0°C)
 Chip 0 DIMM 12 :          +43.0°C  (lowest = +36.0°C, highest = +47.0°C)
 Chip 0 DIMM 13 :           +0.0°C  (lowest =  +0.0°C, highest =  +0.0°C)
 Chip 0 DIMM 14 :           +0.0°C  (lowest =  +0.0°C, highest =  +0.0°C)
 Chip 0 DIMM 15 :           +0.0°C  (lowest =  +0.0°C, highest =  +0.0°C)
 Chip 0 Nest:              +48.0°C  (lowest = +27.0°C, highest = +64.0°C)
 Chip 0 VRM VDD:           +47.0°C  (lowest = +39.0°C, highest = +66.0°C)
 Chip 0 :                  44.00 W  (lowest =  31.00 W, highest = 132.00 W)
 Chip 0 Vdd:               15.00 W  (lowest =   4.00 W, highest = 104.00 W)
 Chip 0 Vdn:               10.00 W  (lowest =   8.00 W, highest =  12.00 W)
 Chip 0 :                 227.11 kJ
 Chip 0 Vdd:               44.80 kJ
 Chip 0 Vdn:               58.80 kJ
 Chip 0 Vdd:              +21.50 A  (lowest =  +6.50 A, highest = +104.75 A)
 Chip 0 Vdn:              +14.88 A  (lowest = +12.63 A, highest = +18.88 A)

The best thing? It’s really quick! The hwmon interface is fast and efficient.

Yet another near-upstream Raptor Blackbird firmware build

In what is coming a month occurance, I’ve put up yet another firmware build for the Raptor Blackbird with close-to-upstream firmware (see here and here for previous ones).

Well, I’ve done another build! It’s current op-build (as of yesterday), but my branch with patches for the Raptor Blackbird. The skiboot patch is there, the SBE speedup patch is now upstream. The machine-xml which is straight from Raptor but in my repo.

Here’s the current versions of everything:

$ lsprop /sys/firmware/devicetree/base/ibm,firmware-versions/
skiboot          "v6.5-228-g82aed17a-p4360f95"
bmc-firmware-version
                 "0.00"
occ              "3ab2921"
hostboot         "acdff8a-pe7e80e1"
buildroot        "2019.05.3-15-g3a4fc2a888"
capp-ucode       "p9-dd2-v4"
machine-xml      "site_local-stewart-a0efd66"
hostboot-binaries
                 "hw013120a.opmst"
sbe              "c318ab0-p1ddf83c"
hcode            "hw030220a.opmst"
petitboot        "v1.12"
phandle          0000064c (1612)
version          "blackbird-v2.4-514-g62d1a941"
linux            "5.4.22-openpower1-pdbbf8c8"
name             "ibm,firmware-versions"

If we compare this to the last build I put up, we have:

Componentoldnew
skibootv6.5-209-g179d53df-p4360f95v6.5-228-g82aed17a-p4360f95
linux5.4.13-openpower1-pa361bec5.4.22-openpower1-pdbbf8c8
occ3ab2921no change
hostboot779761d-pe7e80e1acdff8a-pe7e80e1
buildroot2019.05.3-14-g17f117295f2019.05.3-15-g3a4fc2a888
capp-ucodep9-dd2-v4no change
machine-xmlsite_local-stewart-a0efd66no change
hostboot-binarieshw011120a.opmsthw013120a.opmst
sbe166b70c-p06fc80cc318ab0-p1ddf83c
hcodehw011520a.opmsthw030220a.opmst
petitbootv1.11v1.12
versionblackbird-v2.4-415-gb63b36efblackbird-v2.4-514-g62d1a941

So, what do those changes mean? Not too much changed over the past month. Kernel bump, new petitboot (although I can’t find release notes but it doesn’t look like there’s a lot of changes), and slight bumps to other firmware components.

Grab blackbird.pnor from https://www.flamingspork.com/blackbird/stewart-blackbird-4-images/ and give it a whirl!

To flash it, copy blackbird.pnor to your Blackbird’s BMC in /tmp/ (important! the /tmp filesystem has enough room, the home directory for root does not), and then run:

pflash -E -p /tmp/blackbird.pnor

Which will ask you to confirm and then flash:

About to erase chip !
WARNING ! This will modify your HOST flash chip content !
Enter "yes" to confirm:yes
Erasing... (may take a while)
[==================================================] 99% ETA:1s      
done !
About to program "/tmp/blackbird.pnor" at 0x00000000..0x04000000 !
Programming & Verifying...
[==================================================] 100% ETA:0s   

Booting temporary firmware on the Raptor Blackbird

In a future post, I’ll detail how to build my ported-to-upstream Blackbird firmware. Here though, we’ll explore booting some firmware temporarily to experiment.

Step 1: Copy your new PNOR image over to the BMC.
Step 2: …
Step 3: Profit!

Okay, not really, once you’ve copied over your image, ensure the computer is off and then you can tell the daemon that provides firmware to the host to use a file backend for it rather than the PNOR chip on the motherboard (i.e. yes, you can boot your system even when the firmware chip isn’t there – although I’ve not literally tried this).

root@blackbird:~# mboxctl --backend file:/tmp/blackbird.pnor 
SetBackend: Success
root@blackbird:~# obmcutil poweron

If we look at the serial console (ssh to the BMC port 2200) we’ll see Hostboot start, realise there’s newer SBE code, flash it, and reboot:

--== Welcome to Hostboot hostboot-b284071/hbicore.bin ==--

  3.02606|secure|SecureROM valid - enabling functionality
  5.14678|Booting from SBE side 0 on master proc=00050000
  5.18537|ISTEP  6. 5 - host_init_fsi
  5.47985|ISTEP  6. 6 - host_set_ipl_parms
  5.54476|ISTEP  6. 7 - host_discover_targets
  6.56106|HWAS|PRESENT> DIMM[03]=8080000000000000
  6.56108|HWAS|PRESENT> Proc[05]=8000000000000000
  6.56109|HWAS|PRESENT> Core[07]=1511540000000000
  6.61373|ISTEP  6. 8 - host_update_master_tpm
  6.61529|SECURE|Security Access Bit> 0x0000000000000000
  6.61530|SECURE|Secure Mode Disable (via Jumper)> 0x8000000000000000
  6.61543|ISTEP  6. 9 - host_gard
  7.20987|HWAS|FUNCTIONAL> DIMM[03]=8080000000000000
  7.20988|HWAS|FUNCTIONAL> Proc[05]=8000000000000000
  7.20989|HWAS|FUNCTIONAL> Core[07]=1511540000000000
  7.21299|ISTEP  6.11 - host_start_occ_xstop_handler
  8.28965|ISTEP  6.12 - host_voltage_config
  8.47973|ISTEP  7. 1 - mss_attr_cleanup
  9.07674|ISTEP  7. 2 - mss_volt
  9.35627|ISTEP  7. 3 - mss_freq
  9.63029|ISTEP  7. 4 - mss_eff_config
 10.35189|ISTEP  7. 5 - mss_attr_update
 10.38489|ISTEP  8. 1 - host_slave_sbe_config
 10.45332|ISTEP  8. 2 - host_setup_sbe
 10.45450|ISTEP  8. 3 - host_cbs_start
 10.45574|ISTEP  8. 4 - proc_check_slave_sbe_seeprom_complete
 10.48675|ISTEP  8. 5 - host_attnlisten_proc
 10.50338|ISTEP  8. 6 - host_p9_fbc_eff_config
 10.50771|ISTEP  8. 7 - host_p9_eff_config_links
 10.53338|ISTEP  8. 8 - proc_attr_update
 10.53634|ISTEP  8. 9 - proc_chiplet_fabric_scominit
 10.55234|ISTEP  8.10 - proc_xbus_scominit
 10.56202|ISTEP  8.11 - proc_xbus_enable_ridi
 10.57788|ISTEP  8.12 - host_set_voltages
 10.59421|ISTEP  9. 1 - fabric_erepair
 10.65877|ISTEP  9. 2 - fabric_io_dccal
 10.66048|ISTEP  9. 3 - fabric_pre_trainadv
 10.66665|ISTEP  9. 4 - fabric_io_run_training
 10.66860|ISTEP  9. 5 - fabric_post_trainadv
 10.67060|ISTEP  9. 6 - proc_smp_link_layer
 10.67503|ISTEP  9. 7 - proc_fab_iovalid
 11.10386|ISTEP  9. 8 - host_fbc_eff_config_aggregate
 11.15103|ISTEP 10. 1 - proc_build_smp
 11.27537|ISTEP 10. 2 - host_slave_sbe_update
 11.68581|sbe|System Performing SBE Update for PROC 0, side 0
 34.50467|sbe|System Rebooting To Complete SBE Update Process
 34.50595|IPMI: Initiate power cycle
 34.54671|Stopping istep dispatcher
 34.68729|IPMI: shutdown complete

One of the improvements is we now get output from the SBE! This means that when we do things like mess up secure boot and non secure boot firmware (I’ll explain why/how this is a thing later), we’ll actually get something useful out of a serial port:

--== Welcome to SBE - CommitId[0x8b06b5c1] ==--
istep 3.19
istep 3.20
istep 3.21
istep 3.22
istep 4.1
istep 4.2
istep 4.3
istep 4.4
istep 4.5
istep 4.6
istep 4.7
istep 4.8
istep 4.9
istep 4.10
istep 4.11
istep 4.12
istep 4.13
istep 4.14
istep 4.15
istep 4.16
istep 4.17
istep 4.18
istep 4.19
istep 4.20
istep 4.21
istep 4.22
istep 4.23
istep 4.24
istep 4.25
istep 4.26
istep 4.27
istep 4.28
istep 4.29
istep 4.30
istep 4.31
istep 4.32
istep 4.33
istep 4.34
istep 5.1
istep 5.2
SBE starting hostboot

And then we’re back into normal Hostboot boot (which we’ve all seen before) and end up at a newer petitboot!

Petitboot 1.11 on a Raptor Blackbird

One notable absence from that screenshot is my installed Fedora is missing. This is because there appears to be a bug in the 5.3.7 kernel that’s currently upstream, and if we drop to the shell and poke at lspci and dmesg, we can work out what could be the culprit:

Exiting petitboot. Type 'exit' to return.
You may run 'pb-sos' to gather diagnostic data
No password set, running as root. You may set a password in the System Configuration screen.
# lspci
0000:00:00.0 PCI bridge: IBM Device 04c1
0001:00:00.0 PCI bridge: IBM Device 04c1
0001:01:00.0 Non-Volatile memory controller: Intel Corporation Device f1a8 (rev 03)
0002:00:00.0 PCI bridge: IBM Device 04c1
0002:01:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9235 PCIe 2.0 x2 4-port SATA 6 Gb/s Controller (rev 11)
0003:00:00.0 PCI bridge: IBM Device 04c1
0003:01:00.0 USB controller: Texas Instruments TUSB73x0 SuperSpeed USB 3.0 xHCI Host Controller (rev 02)
0004:00:00.0 PCI bridge: IBM Device 04c1
0004:01:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0004:01:00.1 Ethernet controller: Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0004:01:00.2 Ethernet controller: Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0005:00:00.0 PCI bridge: IBM Device 04c1
0005:01:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04)
0005:02:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
# dmesg|grep -i nvme
[    2.991038] nvme nvme0: pci function 0001:01:00.0
[    2.991088] nvme 0001:01:00.0: enabling device (0140 -> 0142)
[    3.121799] nvme nvme0: Identify Controller failed (19)
[    3.121802] nvme nvme0: Removing after probe failure status: -5
# uname -a
Linux skiroot 5.3.7-openpower1 #2 SMP Sat Dec 14 09:06:20 PST 2019 ppc64le GNU/Linux

If for some reason the device didn’t show up in lspci, then I’d look at the skiboot firmware log, which is /sys/firmware/opal/msglog.

Looking at upstream stable kernel patches, it seems like 5.3.8 has a interesting looking patch when you realize that ppc64le uses a 64k page size:

commit efac0f186ea654e8389f5017c7f643ef48cb4b93
Author: Kevin Hao <haokexin@gmail.com>
Date:   Fri Oct 18 10:53:14 2019 +0800

    nvme-pci: Set the prp2 correctly when using more than 4k page
    
    commit a4f40484e7f1dff56bb9f286cc59ffa36e0259eb upstream.
    
    In the current code, the nvme is using a fixed 4k PRP entry size,
    but if the kernel use a page size which is more than 4k, we should
    consider the situation that the bv_offset may be larger than the
    dev->ctrl.page_size. Otherwise we may miss setting the prp2 and then
    cause the command can't be executed correctly.
    
    Fixes: dff824b2aadb ("nvme-pci: optimize mapping of small single segment requests")
    Cc: stable@vger.kernel.org
    Reviewed-by: Christoph Hellwig <hch@lst.de>
    Signed-off-by: Kevin Hao <haokexin@gmail.com>
    Signed-off-by: Keith Busch <kbusch@kernel.org>
    Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

So, time to go try 5.3.8. My yaks are getting quite smooth.

Oh, and when you’re done with your temporary firmware, either fiddle with mboxctl or restart the systemd service for it, or reboot your BMC or… well, I gotta leave you something to work out on your own :)

Building OpenPOWER firmware on Fedora 31

One of the challenges with Fedora 31 is that /usr/bin/python is now Python 3 rather than Python 2. Just about every python script in existence relies on /usr/bin/python being Python 2 and not anything else. I can’t really recall, but this probably happened with the 1.5 to 2 transition as well (although IIRC that was less breaking).

What this means is that for projects that are half-way through converting to python 3, everything breaks.

op-build is one of these projects.

So, we need:

After all that, you can actually build a pnor image on Fedora 31. Even on Fedora 31 ppc64le, which is literally what I’ve just done.