{"id":4200,"date":"2016-10-20T16:24:00","date_gmt":"2016-10-20T06:24:00","guid":{"rendered":"https:\/\/www.flamingspork.com\/blog\/?p=4200"},"modified":"2016-10-20T16:54:20","modified_gmt":"2016-10-20T06:54:20","slug":"workaround-for-opal-prd-using-100-cpu","status":"publish","type":"post","link":"https:\/\/www.flamingspork.com\/blog\/2016\/10\/20\/workaround-for-opal-prd-using-100-cpu\/","title":{"rendered":"Workaround for opal-prd using 100% CPU"},"content":{"rendered":"

opal-prd<\/code> is the Processor RunTime Diagnostics daemon, the userspace process that on OpenPower systems is responsible for some of the runtime diagnostics. Although a userspace process, it memory maps (as in mmap) in some code loaded by early firmware (Hostboot) called the HostBoot RunTime (HBRT) and runs it, using calls to the kernel to accomplish any needed operations (e.g. reading\/writing registers inside the chip). Running this in user space gives us benefits such as being able to attach gdb, recover from segfaults etc.<\/p>\n

The reason this code is shipped as part of firmware rather than as an OS package is that it is very<\/strong> system specific, and it would be a giant pain to update a package in every Linux distribution every time a new chip or machine was introduced.<\/p>\n

Anyway, there’s a bug in the HBRT code that means if there’s an ECC error in the HBEL (HostBoot Error Log) partition in the system flash (“bios” or “pnor”… the flash where your system firmware lives), the opal-prd process may get stuck chewing up 100% CPU and not doing anything useful. There’s https:\/\/github.com\/open-power\/hostboot\/issues\/67<\/a> for this.<\/p>\n

You will notice a problem if the opal-prd process is using 100% CPU and the last log messages are something like:<\/p>\n

HBRT: ERRL:>>ErrlManager::ErrlManager constructor.\r\nHBRT: ERRL:iv_hiddenErrorLogsEnable = 0x0\r\nHBRT: ERRL:>>setupPnorInfo\r\nHBRT: PNOR:>>RtPnor::getSectionInfo\r\nHBRT: PNOR:>>RtPnor::readFromDevice: i_offset=0x0, i_procId=0 sec=11 size=0x20000 ecc=1\r\nHBRT: PNOR:RtPnor::readFromDevice: removing ECC...\r\nHBRT: PNOR:RtPnor::readFromDevice> Uncorrectable ECC error : chip=0,offset=0x0\r\n<\/pre>\n
(the parameters to readFromDevice may differ)<\/p>\n
Luckily, there’s a simple workaround to fix it all up! You will need the pflash utility. Primarily, pflash is meant only for developers and those who know what they’re doing. You\u00c2\u00a0can<\/strong> turn your computer into a brick using it.<\/p>\n
pflash is packaged in Ubuntu 16.10 and RHEL 7.3, but you can otherwise build it from source easily enough:<\/p>\n
git clone https:\/\/github.com\/open-power\/skiboot.git\r\ncd skiboot\/external\/pflash\r\nmake<\/pre>\n
Now that you have pflash, you just need to erase the HBEL partition and write (ECC) zeros:<\/p>\n
dd if=\/dev\/zero of=\/tmp\/hbel bs=1 count=147456\r\npflash -P HBEL -e\r\npflash -P HBEL -p \/tmp\/hbel<\/pre>\n
Note: you cannot just erase the partition or use the pflash option to do an ECC erase, you may render your system unbootable if you get it wrong.<\/p>\n
After that, restart opal-prd however your distro handles restarting daemons (e.g. systemctl restart opal-prd.service<\/code>) and all should<\/strong> be well.<\/p>\n","protected":false},"excerpt":{"rendered":"
opal-prd is the Processor RunTime Diagnostics daemon, the userspace process that on OpenPower systems is responsible for some of the runtime diagnostics. Although a userspace process, it memory maps (as in mmap) in some code loaded by early firmware (Hostboot) … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":true,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[570,588,615],"tags":[697,585,636,696,587,694,695,55],"class_list":["post-4200","post","type-post","status-publish","format-standard","hentry","category-ibm-work-et-al","category-opal","category-powerpc","tag-hbrt","tag-hostboot","tag-opal","tag-opal-prd","tag-openpower","tag-ppc64el","tag-rhel","tag-ubuntu"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5a6n8-15K","jetpack-related-posts":[{"id":3775,"url":"https:\/\/www.flamingspork.com\/blog\/2014\/07\/17\/openpower-firmware-up-on-github\/","url_meta":{"origin":4200,"position":0},"title":"OpenPower firmware up on github!","author":"Stewart Smith","date":"2014-07-17","format":false,"excerpt":"With the whole OpenPower thing, a lot of low level firmware is being open sourced, which is really exciting for the platform - the less proprietary code sitting in memory the better in my books. If you go to https:\/\/github.com\/open-power you'll see code for a bunch of the low level\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3912,"url":"https:\/\/www.flamingspork.com\/blog\/2014\/12\/03\/running-skiboot-opal-on-the-power8-simulator\/","url_meta":{"origin":4200,"position":1},"title":"Running skiboot (OPAL) on the POWER8 Simulator","author":"Stewart Smith","date":"2014-12-03","format":false,"excerpt":"skiboot is open source boot and runtime firmware for OpenPOWER. On real POWER8 hardware, you will also need HostBoot to do this (basically, to make the chip work) but in a functional simulator (such as this one released by IBM) you don't need a bunch of hardware procedures to make\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4028,"url":"https:\/\/www.flamingspork.com\/blog\/2016\/02\/08\/my-linux-conf-au-2016-talk-adventures-in-openpower-firmware-is-up\/","url_meta":{"origin":4200,"position":2},"title":"My linux.conf.au 2016 talk “Adventures in OpenPower Firmware” is up!","author":"Stewart Smith","date":"2016-02-08","format":false,"excerpt":"Thanks to the absolutely amazing efforts of the LCA video team, they've already (only a few days after I gave it) got the video from my linux.conf.au 2016 talk up! Abstract In mid 2014, IBM released the first POWER8 based systems with the new Free and Open Source OPAL firmware.\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/www.flamingspork.com\/blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3965,"url":"https:\/\/www.flamingspork.com\/blog\/2015\/06\/12\/gcov-code-coverage-for-openpower-firmware\/","url_meta":{"origin":4200,"position":3},"title":"gcov code coverage for OpenPower firmware","author":"Stewart Smith","date":"2015-06-12","format":false,"excerpt":"For skiboot (which provides the OPAL boot and runtime firmware for OpenPower machines), I've been pretty interested at getting some automated code coverage data for booting on real hardware (as well as in a simulator). Why? Well, it's useful to see that various test suites are actually testing what you\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4345,"url":"https:\/\/www.flamingspork.com\/blog\/2017\/12\/11\/a-simplified-view-of-openpower-firmware-development\/","url_meta":{"origin":4200,"position":4},"title":"A (simplified) view of OpenPOWER Firmware Development","author":"Stewart Smith","date":"2017-12-11","format":false,"excerpt":"I've been working on trying to better document the whole flow of code that goes into a build of firmware for an OpenPOWER machine. This is partially to help those not familiar with it get a better grasp of the sheer scale of what goes into that 32\/64MB of flash.\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2017\/12\/devflow-legend.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2017\/12\/devflow-legend.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2017\/12\/devflow-legend.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2017\/12\/devflow-legend.png?resize=700%2C400&ssl=1 2x"},"classes":[]},{"id":4544,"url":"https:\/\/www.flamingspork.com\/blog\/2020\/02\/01\/another-close-to-upstream-blackbird-firmware-build\/","url_meta":{"origin":4200,"position":5},"title":"Another close-to-upstream Blackbird Firmware Build","author":"Stewart Smith","date":"2020-02-01","format":false,"excerpt":"A few weeks ago (okay, close to six), I put up a firmware build for the Raptor Blackbird with close-to-upstream firmware (see here). Well, I've done another build! It's current op-build (as of this morning), but my branch with patches for the Raptor Blackbird. The skiboot patch is there, as\u2026","rel":"","context":"In "OPAL"","block_context":{"text":"OPAL","link":"https:\/\/www.flamingspork.com\/blog\/category\/opal\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/4200","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/comments?post=4200"}],"version-history":[{"count":4,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/4200\/revisions"}],"predecessor-version":[{"id":4204,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/4200\/revisions\/4204"}],"wp:attachment":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/media?parent=4200"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/categories?post=4200"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/tags?post=4200"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}