{"id":4290,"date":"2017-08-22T13:19:35","date_gmt":"2017-08-22T03:19:35","guid":{"rendered":"https:\/\/www.flamingspork.com\/blog\/?p=4290"},"modified":"2017-08-22T13:19:35","modified_gmt":"2017-08-22T03:19:35","slug":"op-test-framework-lets-break-the-console","status":"publish","type":"post","link":"https:\/\/www.flamingspork.com\/blog\/2017\/08\/22\/op-test-framework-lets-break-the-console\/","title":{"rendered":"op-test-framework: Let’s break the console!"},"content":{"rendered":"

One of the things I’ve been working on fairly quietly is the test suite for OpenPOWER firmware: op-test-framework<\/a>. I’ve approach things I’m hacking on from the goal of “when I merge patches into skiboot<\/a>, can I be confident that I haven’t merged something that’s broken existing functionality?”<\/p>\n

By testing host firmware, we incidentally (as well as on purpose) test a whole bunch of BMC functionality. One bit of functionality we rely on a lot is the host “serial” console. Typically, this is exposed to the user over IPMI Serial Over LAN (SOL), or on OpenBMC<\/a> it’s also exposed as something you can ssh to (which proves to be both faster and more reliable than IPMI, not to mention there’s some remote semblance of security).<\/p>\n

When running through some tests, I noticed something pretty odd, it appeared as though we were sometimes missing some console output on larger IOs. This usually isn’t a problem as when we’re using expect<\/a>(1) (or the python equivalent pexpect<\/a>) we end up having all sorts of delays here there and everywhere to work around all the terrible things you hope you never learn. So… how could I test that? Well.. what about checking the output of something like dd if=\/dev\/zero bs=1024 count=16|hexdump -C<\/code> to see if we get the full output?<\/p>\n

Time to add a test to op-test-framework<\/a>! Adding such a test is pretty easy. If we look at the source of the test I added, we can see what happens (source here<\/a>).<\/p>\n

class Console():\r\n    bs = 1024\r\n    count = 8\r\n    def setUp(self):\r\n        conf = OpTestConfiguration.conf\r\n        self.bmc = conf.bmc()\r\n        self.system = conf.system()\r\n\r\n    def runTest(self):\r\n        self.system.goto_state(OpSystemState.PETITBOOT_SHELL)\r\n        console = self.bmc.get_host_console()\r\n        self.system.host_console_unique_prompt()\r\n        bs = self.bs\r\n        count = self.count\r\n        self.assertTrue( (bs*count)%16 == 0, \"Bug in test writer. Must be multiple of 16 bytes: bs %u count %u \/ 16 = %u\" % (bs, count, (bs*count)%16))\r\n        try:\r\n            zeros = console.run_command(\"dd if=\/dev\/zero bs=%u count=%u|hexdump -C -v\" % (bs, count), timeout=120)\r\n        except CommandFailed as cf:\r\n            self.assertEqual(cf.exitcode, 0)\r\n        self.assertTrue( len(zeros) == 3+(count*bs)\/16, \"Unexpected length of zeros %u\" % (len(zeros)))<\/pre>\n
First thing you’ll notice is that this looks like a Python unittest. It’s because it is. The unittest infrastructure was a path of least resistance, so we started with it. This class isn’t the one that’s actually run, we do a little bit of inheritance magic in order to run the same test with different parameters (see https:\/\/github.com\/open-power\/op-test-framework\/blob\/6c74fb0fb0993ae8ae1a7aa62ec58e57c0080686\/testcases\/Console.py#L50<\/a>)<\/p>\n
class Console8k(Console, unittest.TestCase):\r\n    bs = 1024\r\n    count = 8\r\n\r\nclass Console16k(Console, unittest.TestCase):\r\n    bs = 1024\r\n    count = 16\r\n\r\nclass Console32k(Console, unittest.TestCase):\r\n    bs = 1024\r\n    count = 32\r\n<\/pre>\n
The setUp()<\/code> function is pure boiler plate, we grab some objects from the configuration of the test run, namely information about the BMC and the system itself, so we can do things to both. The real magic happens in runTest().<\/p>\n
op-test-framework tracks the state of the machine being tested across each test. This means that if we’re executing 101 tests in the petitboot shell, we don’t need to do 101 separate boots to petitboot. The self.system.goto_state(OpSystemState.PETITBOOT_SHELL)<\/code> statement says “Please ensure the system is booted to the petitboot shell”. Other states include OFF (obvious) and OS, which is when the machine is booted to the OS.<\/p>\n
The next two lines ensure we can run commands on the console (where console is IPMI Serial over LAN or other similar connection, such as the SSH console provided by OpenBMC):<\/p>\n
console = self.bmc.get_host_console()\r\nself.system.host_console_unique_prompt()\r\n<\/pre>\n
The host_console_unique_prompt() call is a bit ugly, and I’m hoping we fix the APIs so that this isn’t needed. Basically, it sets things up so that pexpect will work properly.<\/p>\n
The bit that does the work is the try\/except block along with the assertTrue. We don’t currently check that the content is all correct, we just check we got the right *amount* of content.<\/p>\n
It turns out, this check is enough to reveal a bug that turns out to be deep in the core Linux TTY layer, and has caused Jeremy<\/a> some amount of fun (for certain values of fun).<\/p>\n
Want to know more about how the console works? Jeremy blogged on it<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
One of the things I’ve been working on fairly quietly is the test suite for OpenPOWER firmware: op-test-framework. I’ve approach things I’m hacking on from the goal of “when I merge patches into skiboot, can I be confident that I … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[76,1,615],"tags":[636,587,590],"class_list":["post-4290","post","type-post","status-publish","format-standard","hentry","category-code","category-general","category-powerpc","tag-opal","tag-openpower","tag-testing"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5a6n8-17c","jetpack-related-posts":[{"id":4311,"url":"https:\/\/www.flamingspork.com\/blog\/2017\/10\/20\/zmodem-saves-the-day-or-why-my-firmware-for-a-machine-with-a-cpu-from-2017-contains-a-serial-file-transfer-protocol-from-the-1980s\/","url_meta":{"origin":4290,"position":0},"title":"ZMODEM saves the day! Or, why my firmware for a machine with a CPU from 2017 contains a serial file transfer protocol from the 1980s","author":"Stewart Smith","date":"2017-10-20","format":false,"excerpt":"Recently, I added the package lrzsz to op-build in this commit. This package provides the rz and sz commands - for receive zmodem and send zmodem respectively. For those who don't know, op-build builds a firmware image for OpenPOWER machines, and adding this package adds the commands to the petitboot\u2026","rel":"","context":"In "IBM"","block_context":{"text":"IBM","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/ibm-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4440,"url":"https:\/\/www.flamingspork.com\/blog\/2019\/01\/23\/cve-2019-6260-gaining-control-of-bmc-from-the-host-processor\/","url_meta":{"origin":4290,"position":1},"title":"CVE-2019-6260: Gaining control of BMC from the host processor","author":"Stewart Smith","date":"2019-01-23","format":false,"excerpt":"This is details for CVE-2019-6260 - which has been nicknamed \"pantsdown\" due to the nature of feeling that we feel that we've \"caught chunks of the industry with their...\" and combined with the fact that naming things is hard, so if you pick a bad name somebody would have to\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/www.flamingspork.com\/blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4526,"url":"https:\/\/www.flamingspork.com\/blog\/2019\/12\/15\/are-you-fans-of-the-blackbird-speak-up-i-cant-hear-you-over-the-fan\/","url_meta":{"origin":4290,"position":2},"title":"Are you Fans of the Blackbird? Speak up, I can’t hear you over the fan.","author":"Stewart Smith","date":"2019-12-15","format":false,"excerpt":"So, as of yesterday, I started running a pretty-close-to-upstream op-build host firmware stack on my Blackbird. Notable yak-shaving has included: Update README.md for Fedora 31 build-deps because READMEs are importantSupport Python3 as default \/usr\/bin\/python (so I could compile it at all on Fedora 31)RaptorCS Blackbird support the actual minimal firmware\u2026","rel":"","context":"In "cool gadgets"","block_context":{"text":"cool gadgets","link":"https:\/\/www.flamingspork.com\/blog\/category\/cool-gadgets\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4522,"url":"https:\/\/www.flamingspork.com\/blog\/2019\/12\/14\/booting-temporary-firmware-on-the-raptor-blackbird\/","url_meta":{"origin":4290,"position":3},"title":"Booting temporary firmware on the Raptor Blackbird","author":"Stewart Smith","date":"2019-12-14","format":false,"excerpt":"In a future post, I'll detail how to build my ported-to-upstream Blackbird firmware. Here though, we'll explore booting some firmware temporarily to experiment. Step 1: Copy your new PNOR image over to the BMC.Step 2: ...Step 3: Profit! Okay, not really, once you've copied over your image, ensure the computer\u2026","rel":"","context":"In "cool gadgets"","block_context":{"text":"cool gadgets","link":"https:\/\/www.flamingspork.com\/blog\/category\/cool-gadgets\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2019\/12\/Screenshot-from-2019-12-14-09-59-38.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2019\/12\/Screenshot-from-2019-12-14-09-59-38.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2019\/12\/Screenshot-from-2019-12-14-09-59-38.png?resize=525%2C300&ssl=1 1.5x"},"classes":[]},{"id":3940,"url":"https:\/\/www.flamingspork.com\/blog\/2015\/02\/03\/building-openpower-firmware-for-use-in-power8-simulator\/","url_meta":{"origin":4290,"position":4},"title":"Building OpenPower firmware for use in POWER8 Simulator","author":"Stewart Smith","date":"2015-02-03","format":false,"excerpt":"Previously, I blogged on how to Run skiboot (OPAL) on the POWER8 Simulator. If you want to build the full Open Power firmware environment, including the Petitboot bootloader and kernel, you can now do so! My pull request for an op-build target for the simulator has been merged, so you\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4508,"url":"https:\/\/www.flamingspork.com\/blog\/2019\/12\/10\/looking-at-the-state-of-blackbird-firmware\/","url_meta":{"origin":4290,"position":5},"title":"Looking at the state of Blackbird firmware","author":"Stewart Smith","date":"2019-12-10","format":false,"excerpt":"Having been somewhat involved in OpenPOWER firmware, I have a bunch of experience and opinions on maintaining firmware trees for products, what working with upstream looks like and all that. So, with my new Blackbird system I decided to take a bit of a look as to what the firmware\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/4290","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/comments?post=4290"}],"version-history":[{"count":1,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/4290\/revisions"}],"predecessor-version":[{"id":4297,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/4290\/revisions\/4297"}],"wp:attachment":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/media?parent=4290"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/categories?post=4290"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/tags?post=4290"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}