Thanks to my most recent PR being merged, op-build v2.5 will have full support for the Raptor Blackbird! This includes support for the “IPL Monitor” that’s required to get fan control going.
Note that if you’re running Fedora 32 then you need some patches to buildroot to have it build, but if you’re building on something a little older, then upstream should build and work straight out of the box (err… git tree).
I also note that the work to get Secure Boot for an OS Kernel going is starting to make its way out for code reviews, so that’s something to look forward to (although without a TPM we’re going to need extra code).
I have done a few builds of firmware for the Raptor Blackbird since I got mine, each of them based on upstream op-build plus a few patches. The previous one was Yet another near-upstream Raptor Blackbird firmware build that I built a couple of months ago. This new build is based off the release candidate of op-build v2.5. Here’s what’s changed:
There’s two differences from upstream op-build: my pull request to op-build, and the fixing of the (old) buildroot so that it’ll build on Fedora 32. From discussions on the openpower-firmware mailing list, it seems that one hopeful thing is to have all the Blackbird support merged in before the final op-build v2.5 is tagged. The previous op-build release (v2.4) was tagged in July 2019, so we’re about 10 months into what was a 2 month release cycle, so speculating on when that final release will be is somewhat difficult.
It goes without saying that using this build is a At Your Own Risk and I make zero warranty. AFAIK it can’t physically destroy your system.
My GitHub op-build branch stewart-blackbird-v1 has all the changes built into this build (the VERSION displayed in firmware will be slightly weird as I did the tagging afterwards… this is not meant to be “howto release firmware to the public”). Follow op-build pull 3341 for the state of upstreaming everything.
Linux v5.4.3-openpower1 because 5.3.7 in the current op-build repo has an NVME driver bug that means it didn’t recognize my Intel NVME drive, and thus prevented me from actually booting an OS.
Apart from that, I was all happy as Larry. Except then I went into the room with the Blackbird in it an went “huh, that’s loud”, and since it was bedtime, I decided it could all wait until the morning.
It is now the morning. Checking fan speeds over IPMI, one fan stood out (fan2, sitting at 4300RPM). This was a bit of a surprise as what’s silkscreened on the board is that the rear case fan is hooked up to ‘fan2″, and if we had a “start from 0/1” mix up, it’d be the front case fan. I had just assumed it’d be maybe OCC firmware dying or something, but this wasn’t the case (I checked – thanks occtoolp9!)
After a bit of digging around, I worked out this mapping:
IPMI fan0
Rear Case Fan
Motherboard Fan 2
IPMI fan1
Front Case Fan
Motherboard Fan 3
IPMI fan2
CPU Fan
Motherboard Fan 1
Which is about as surprising and confusing as you’d think.
After a bunch of digging around the Raptor ports of OpenBMC and Hostboot, it seems that the IPL Observer which is custom to Raptor controls if the BMC decides to do fan control or not.
You can get its view of the world from the BMC via the (incredibly user friendly) poking at DBus:
Which if you just have the Hostboot patch in (like I first did) you end up with:
s "IPL_RUNNING"
s "21,3"
Which is where Hostboot exits the IPL process (as you see on the screen) and hands over to skiboot. But if you start digging through their op-build tree, you find that there’s a signal_linux_start_complete script which calls pnv-lpc to write two values to LPC ports 0x81 and 0x82. The pnv-lpc utility is the external/lpc/ binary from skiboot, and these two ports are the “extended lpc port 80h” state.
So, to get back fan control? First, build the lpc utility:
git clone git@github.com:open-power/skiboot.git
cd skiboot/external/lpc
make
and then poke the magic values of “IPL complete and linux running”:
$ sudo ./lpc io 0x81.b=254
[io] W 0x00000081.b=0xfe
$ sudo ./lpc io 0x82.b=254
[io] W 0x00000082.b=0xfe
You get a friendly beep, and then your fans return to sanity.
Of course, for that to work you need to have debugfs mounted, as this pokes OPAL debugfs to do direct LPC operations.
Next up: think of a smarter way to trigger that than “stewart runs it on the command line”. Also next up: work out the better way to determine that fan control should be on and patch the BMC.
In a future post, I’ll detail how to build my ported-to-upstream Blackbird firmware. Here though, we’ll explore booting some firmware temporarily to experiment.
Step 1: Copy your new PNOR image over to the BMC. Step 2: … Step 3: Profit!
Okay, not really, once you’ve copied over your image, ensure the computer is off and then you can tell the daemon that provides firmware to the host to use a file backend for it rather than the PNOR chip on the motherboard (i.e. yes, you can boot your system even when the firmware chip isn’t there – although I’ve not literally tried this).
One of the improvements is we now get output from the SBE! This means that when we do things like mess up secure boot and non secure boot firmware (I’ll explain why/how this is a thing later), we’ll actually get something useful out of a serial port:
And then we’re back into normal Hostboot boot (which we’ve all seen before) and end up at a newer petitboot!
Petitboot 1.11 on a Raptor Blackbird
One notable absence from that screenshot is my installed Fedora is missing. This is because there appears to be a bug in the 5.3.7 kernel that’s currently upstream, and if we drop to the shell and poke at lspci and dmesg, we can work out what could be the culprit:
Exiting petitboot. Type 'exit' to return.
You may run 'pb-sos' to gather diagnostic data
No password set, running as root. You may set a password in the System Configuration screen.
# lspci
0000:00:00.0 PCI bridge: IBM Device 04c1
0001:00:00.0 PCI bridge: IBM Device 04c1
0001:01:00.0 Non-Volatile memory controller: Intel Corporation Device f1a8 (rev 03)
0002:00:00.0 PCI bridge: IBM Device 04c1
0002:01:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9235 PCIe 2.0 x2 4-port SATA 6 Gb/s Controller (rev 11)
0003:00:00.0 PCI bridge: IBM Device 04c1
0003:01:00.0 USB controller: Texas Instruments TUSB73x0 SuperSpeed USB 3.0 xHCI Host Controller (rev 02)
0004:00:00.0 PCI bridge: IBM Device 04c1
0004:01:00.0 Ethernet controller: Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0004:01:00.1 Ethernet controller: Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0004:01:00.2 Ethernet controller: Broadcom Limited NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
0005:00:00.0 PCI bridge: IBM Device 04c1
0005:01:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04)
0005:02:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
# dmesg|grep -i nvme
[ 2.991038] nvme nvme0: pci function 0001:01:00.0
[ 2.991088] nvme 0001:01:00.0: enabling device (0140 -> 0142)
[ 3.121799] nvme nvme0: Identify Controller failed (19)
[ 3.121802] nvme nvme0: Removing after probe failure status: -5
# uname -a
Linux skiroot 5.3.7-openpower1 #2 SMP Sat Dec 14 09:06:20 PST 2019 ppc64le GNU/Linux
If for some reason the device didn’t show up in lspci, then I’d look at the skiboot firmware log, which is /sys/firmware/opal/msglog.
Looking at upstream stable kernel patches, it seems like 5.3.8 has a interesting looking patch when you realize that ppc64le uses a 64k page size:
commit efac0f186ea654e8389f5017c7f643ef48cb4b93
Author: Kevin Hao <haokexin@gmail.com>
Date: Fri Oct 18 10:53:14 2019 +0800
nvme-pci: Set the prp2 correctly when using more than 4k page
commit a4f40484e7f1dff56bb9f286cc59ffa36e0259eb upstream.
In the current code, the nvme is using a fixed 4k PRP entry size,
but if the kernel use a page size which is more than 4k, we should
consider the situation that the bv_offset may be larger than the
dev->ctrl.page_size. Otherwise we may miss setting the prp2 and then
cause the command can't be executed correctly.
Fixes: dff824b2aadb ("nvme-pci: optimize mapping of small single segment requests")
Cc: stable@vger.kernel.org
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Kevin Hao <haokexin@gmail.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
So, time to go try 5.3.8. My yaks are getting quite smooth.
Oh, and when you’re done with your temporary firmware, either fiddle with mboxctl or restart the systemd service for it, or reboot your BMC or… well, I gotta leave you something to work out on your own :)
Now that I can actually boot the machine, I could test and send my patch upstream for Blackbird support in skiboot. One thing I noticed with the current firmware from Raptor is that the PCIe slot names were wrong. While a pretty minor point, it’s a bit funny that there’s only two slots and the names were wrong.
The PCIe slot names are used to call out the physical location of PCIe cards in the system, so if you, say, hit a bunch of errors, OS/firmware can say “It’s this card in the slot labeled BLAH on the board”.
With my patch, the slot table from skiboot is spat out looking like this:
If you want to give it a go, grab the patch, build skiboot, and flash it on. Alternatively, you can download a built skiboot here. To flash it, do this:
# Copy to your BMC for the Blackbird
scp skiboot-v6.5-146-g376bed3f.lid.xz.stb root@blackbird:/tmp/
# then, ssh to the BMC
$ ssh root@blackbird
# ensure the machine is off
obmcutil poweroff --wait
# Now, make a backup copy (remember to copy it off /tmp on the bmc)
pflash -P PAYLOAD -r /tmp/skiboot-backup
# and flash the new skiboot:
pflash -e -P PAYLOAD -p /tmp/skiboot.lid.xz.stb
# now, power on the box
obmcutil poweron
Way back when Raptor Computer Systems was doing pre-orders for the microATX Blackboard POWER9 system, I put in a pre-order. Since then, I’ve had a few life changes (such as moving to the US and starting to work for Amazon rather than IBM), but I’ve finally gone and done (most of) the setup for my own POWER9 system on (or under) my desk.
An 8 core POWER9 CPU, in bubble wrap and plastic packaging.
Everything came in a big brown box, all rather well packed. I had the board, CPU, heatsink assembly and the special tool to attach the heatsink to the board. Although unique to POWER9, the heatsink/fan assembly was one of the easier ones I’ve ever attached to a board.
The board itself looks pretty much as you’d expect – there’s a big spot for the CPU, a couple of PCI slots, a couple of DIMM slots and some SATA connectors.
The bits that are a bit unusual for a micro-ATX board are the big space reserved for FlexVer, the ASPEED BMC chip and the socketed flash. FlexVer is something I’m not ever going to use, and instead wish that there was an on-board m2 SSD slot instead, even if it was just PCIe. Having to sacrifice a PCIe slot just for a SSD is kind of a bummer.
The Blackbird POWER9 board
The POWER9 chip in socket
If you’re in the know, you know the pain these two chips and their firmware cause you. Luckily, the ASPEED2500 is running OpenBMC, so there is hope, plus there’s a “write open source firmware for it” effort for the Broadcom.
One annoying thing is my DIMMs are taking their sweet time in getting here, so I couldn’t actually populate the board with any memory.
Even without memory though, you can start powering it on and see that everything else works okay (i.e. it’s not completely boned). So, even without DIMMs, I could plug it in, and observe the Hostboot firmware complaining about insufficient hardware to IPL the box.
It Lives!
Yep, out the console (via ssh) you clearly see where things fail:
--== Welcome to Hostboot hostboot-3beba24/hbicore.bin ==--
3.03104|secure|SecureROM valid - enabling functionality
6.67619|Booting from SBE side 0 on master proc=00050000
6.85100|ISTEP 6. 5 - host_init_fsi
7.23753|ISTEP 6. 6 - host_set_ipl_parms
7.71759|ISTEP 6. 7 - host_discover_targets
11.34738|HWAS|PRESENT> Proc[05]=8000000000000000
11.34739|HWAS|PRESENT> Core[07]=1511540000000000
11.69077|ISTEP 6. 8 - host_update_master_tpm
11.73787|SECURE|Security Access Bit> 0x0000000000000000
11.73787|SECURE|Secure Mode Disable (via Jumper)> 0x8000000000000000
11.76276|ISTEP 6. 9 - host_gard
11.96654|HWAS|FUNCTIONAL> Proc[05]=8000000000000000
11.96655|HWAS|FUNCTIONAL> Core[07]=1511540000000000
12.07554|================================================
12.07554|Error reported by hwas (0x0C00) PLID 0x90000007
12.10289| checkMinimumHardware found no functional dimm cards.
12.10290| ModuleId 0x03 MOD_CHECK_MIN_HW
12.10291| ReasonCode 0x0c06 RC_SYSAVAIL_NO_MEMORY_FUNC
12.10292| UserData1 HUID of node : 0x0002000000000000
12.10293| UserData2 number of present, non-functional dimms : 0x0000000000000000
12.10294|------------------------------------------------
12.10417| Callout type : Procedure Callout
12.10417| Procedure : EPUB_PRC_FIND_DECONFIGURED_PART
12.10418| Priority : SRCI_PRIORITY_HIGH
12.10419|------------------------------------------------
12.10420| Hostboot Build ID: hostboot-3beba24/hbicore.bin
12.10421|================================================
12.51718|================================================
12.51719|Error reported by hwas (0x0C00) PLID 0x90000007
12.51720| Insufficient hardware to continue.
12.51721| ModuleId 0x03 MOD_CHECK_MIN_HW
12.51722| ReasonCode 0x0c04 RC_SYSAVAIL_INSUFFICIENT_HW
12.54457| UserData1 : 0x0000000000000000
12.54458| UserData2 : 0x0000000000000000
12.54458|------------------------------------------------
12.54459| Callout type : Procedure Callout
12.54460| Procedure : EPUB_PRC_FIND_DECONFIGURED_PART
12.54461| Priority : SRCI_PRIORITY_HIGH
12.54462|------------------------------------------------
12.54462| Hostboot Build ID: hostboot-3beba24/hbicore.bin
12.54463|================================================
12.73660|System shutting down with error status 0x90000007
12.75545|================================================
12.75546|Error reported by istep (0x1700) PLID 0x90000007
12.77991| IStep failed, see other log(s) with the same PLID for reason.
12.77992| ModuleId 0x01 MOD_REPORTING_ERROR
12.77993| ReasonCode 0x1703 RC_FAILURE
12.77994| UserData1 eid of first error : 0x9000000800000c04
12.77995| UserData2 Reason code of first error : 0x0000000100000609
12.77996|------------------------------------------------
12.77996| host_gard
12.77997|------------------------------------------------
12.77998| Callout type : Procedure Callout
12.77998| Procedure : EPUB_PRC_HB_CODE
12.77999| Priority : SRCI_PRIORITY_LOW
12.78000|------------------------------------------------
12.78001| Hostboot Build ID: hostboot-3beba24/hbicore.bin
12.78002|================================================
Looking forward to getting some DIMMs to show/share more.
