{"id":4057,"date":"2016-05-19T08:56:58","date_gmt":"2016-05-18T22:56:58","guid":{"rendered":"https:\/\/www.flamingspork.com\/blog\/?p=4057"},"modified":"2016-05-19T08:56:58","modified_gmt":"2016-05-18T22:56:58","slug":"fuzzing-firmware-afl-fuzz-skiboot","status":"publish","type":"post","link":"https:\/\/www.flamingspork.com\/blog\/2016\/05\/19\/fuzzing-firmware-afl-fuzz-skiboot\/","title":{"rendered":"Fuzzing Firmware – afl-fuzz + skiboot"},"content":{"rendered":"

In what is likely to be a series on how firmware makes some normal tools harder to use, first I’m going to look at american fuzzy lop<\/a> – a tool for fuzz testing that if you’re not using then you most certainly have bugs it’ll find for you.<\/p>\n

I first got interested in afl-fuzz during Erik de Castro Lopo’s excellent linux.conf.au 2016 in Geelong earlier this year: “Fuzz all the things!<\/a>“. In a previous life, the Random Query Generator<\/a> managed to find a heck of a lot of bugs in MySQL (and Drizzle). For randgen info, see Philip Stoev’s talk on it<\/a> from way back in 2009, a recent (2014) blog post on how Tokutek uses it<\/a> and some notes on how it was being used at Oracle from 2013<\/a>. Basically, the randgen was a specialized fuzzer that (given a grammar) would randomly generate SQL queries, and then (if the server didn’t crash), compare the result to some other database server (e.g. your previous version).<\/p>\n

The afl-fuzz fuzzer takes a different approach – it’s a much more generic fuzzer rather than a targeted tool. Also, while tools such as the random query generator are extremely powerful and find specialized bugs, they’re hard to get started with. A huge benefit of afl-fuzz is that it’s really, really<\/strong> simple to get started with.<\/p>\n

Basically, if you have a binary that takes input on stdin or as a (relatively small) file, afl-fuzz will just work and find bugs for you – read the Quick Start Guide<\/a> and you’ll be finding bugs in no time!<\/p>\n

For firmware of course, we’re a little different than a simple command line program as, well, we aren’t one! Luckily though, we have unit tests<\/strong>. These are just standard binaries that include a bunch of firmware code and get run in user space as part of “make check”. Also, just like unit tests for any project, people do send me patches that break tests (which I reject).<\/p>\n

Some of these tests act on data we get from a place – maybe reading other parts of firmware off PNOR or interacting with data structures we get from other bits of firmware. For testing this code, it can be relatively easy to (for the test), read these off disk.<\/p>\n

For skiboot, there’s a data structure we get from the service processor on FSP machines called HDAT. Basically, it’s just like the device tree<\/a>, but different. Because yet another binary format is always a good idea (yes, that is laced with a heavy dose of sarcasm). One of the steps in early boot is to parse the HDAT data structure and convert it to a device tree. Luckily, we structured our code so that creating a unit test that can run in userspace was relatively easy, we just needed to dump this data structure out from a running machine. You can see the test case here<\/a>. Basically, hdat_to_dt is a binary that reads the HDAT structure out of a pair of files and prints out a device tree. One of the regression tests we have is that we always produce the same output from the same input.<\/p>\n

So… throwing that into AFL yielded a couple of pretty simple bugs, especially around aborting out on invalid data (it’s better to exit the process with failure rather than hit an assert). Nothing too<\/strong> interesting here on my simple input file, but it does mean that our parsing code exits “gracefully” on invalid data.<\/p>\n

Another utility we have is actually a userspace utility for accessing the gard records in the flash. A GARD record is a record of a piece of hardware that has been deconfigured due to a fault (or a suspected fault). Usually this utility operates on PNOR flash through \/dev\/mtd – but really what it’s doing is talking to the libflash library, that we also use inside skiboot (and on OpenBMC) to read\/write from flash directly, via \/dev\/mtd or just from a file. The good news? I haven’t been able to crash this utility yet!<\/p>\n

So I modified the pflash utility to read from a file to attempt to fuzz the partition reading code we have for the partitioning format that’s on PNOR. So far, no crashes – although to even get it going I did have to fix a bug in the file handling code in pflash, so that’s already a win!<\/p>\n

But crashing bugs aren’t the only type of bugs – afl-fuzz has exposed several cases where we act on uninitialized data. How? Well, we run some test cases under valgrind! This is the joy of user space unit tests for firmware – valgrind becomes a tool that you can run! Unfortunately, these bugs have been sitting in my “todo” pile (which is, of course, incredibly long).<\/p>\n

Where to next? Fuzzing the firmware calls themselves would be nice – although that’s going to require a targeted tool that knows about what to pass each of the calls. Another round of afl-fuzz running would also be good, I’ve fixed a bunch of the simple things and having a better set of starting input files would be great (and likely expose more bugs).<\/p>\n","protected":false},"excerpt":{"rendered":"

In what is likely to be a series on how firmware makes some normal tools harder to use, first I’m going to look at american fuzzy lop – a tool for fuzz testing that if you’re not using then you … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[76,1,570,588],"tags":[660,633,658,586,659,664,636,662,663,584,661],"class_list":["post-4057","post","type-post","status-publish","format-standard","hentry","category-code","category-general","category-ibm-work-et-al","category-opal","tag-afl-fuzz","tag-code","tag-debugging","tag-firmware","tag-fuzzing","tag-hdat","tag-opal","tag-pflash","tag-pnor","tag-skiboot","tag-valgrind"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5a6n8-13r","jetpack-related-posts":[{"id":4076,"url":"https:\/\/www.flamingspork.com\/blog\/2016\/07\/12\/using-smatch-static-analysis-on-openpower-opal-firmware\/","url_meta":{"origin":4057,"position":0},"title":"Using Smatch static analysis on OpenPOWER OPAL firmware","author":"Stewart Smith","date":"2016-07-12","format":false,"excerpt":"For Skiboot, I'm always looking at new automated systems to find bugs in the code. A little while ago, I read about the Smatch tool developed by some folks at Oracle (they also wrote about using it on the Linux kernel). I was eager to try it with skiboot to\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3940,"url":"https:\/\/www.flamingspork.com\/blog\/2015\/02\/03\/building-openpower-firmware-for-use-in-power8-simulator\/","url_meta":{"origin":4057,"position":1},"title":"Building OpenPower firmware for use in POWER8 Simulator","author":"Stewart Smith","date":"2015-02-03","format":false,"excerpt":"Previously, I blogged on how to Run skiboot (OPAL) on the POWER8 Simulator. If you want to build the full Open Power firmware environment, including the Petitboot bootloader and kernel, you can now do so! My pull request for an op-build target for the simulator has been merged, so you\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3775,"url":"https:\/\/www.flamingspork.com\/blog\/2014\/07\/17\/openpower-firmware-up-on-github\/","url_meta":{"origin":4057,"position":2},"title":"OpenPower firmware up on github!","author":"Stewart Smith","date":"2014-07-17","format":false,"excerpt":"With the whole OpenPower thing, a lot of low level firmware is being open sourced, which is really exciting for the platform - the less proprietary code sitting in memory the better in my books. If you go to https:\/\/github.com\/open-power you'll see code for a bunch of the low level\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3972,"url":"https:\/\/www.flamingspork.com\/blog\/2015\/06\/16\/opal-firmware-specification-conformance-and-documentation\/","url_meta":{"origin":4057,"position":3},"title":"OPAL firmware specification, conformance and documentation","author":"Stewart Smith","date":"2015-06-16","format":false,"excerpt":"Now that we have an increasing amount of things that run on top of OPAL: Linux hello_world (in skiboot tree) ppc64le_hello (as I wrote about yesterday) FreeBSD and that the OpenPower ecosystem is rapidly growing (especially around people building OpenPower machines), the need for more formal specification, conformance testing and\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4072,"url":"https:\/\/www.flamingspork.com\/blog\/2016\/06\/20\/building-opal-firmware-for-power9\/","url_meta":{"origin":4057,"position":4},"title":"Building OPAL firmware for POWER9","author":"Stewart Smith","date":"2016-06-20","format":false,"excerpt":"Recently, we merged into the op-build project (the build scripts for OpenPOWER Firmware) a defconfig for building OPAL for (certain) POWER9 simulators. I won't bother linking over to articles on the POWER9 chip or schedule (there's search engines for that), but with this commit - if you happen to be\u2026","rel":"","context":"In "IBM"","block_context":{"text":"IBM","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/ibm-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3965,"url":"https:\/\/www.flamingspork.com\/blog\/2015\/06\/12\/gcov-code-coverage-for-openpower-firmware\/","url_meta":{"origin":4057,"position":5},"title":"gcov code coverage for OpenPower firmware","author":"Stewart Smith","date":"2015-06-12","format":false,"excerpt":"For skiboot (which provides the OPAL boot and runtime firmware for OpenPower machines), I've been pretty interested at getting some automated code coverage data for booting on real hardware (as well as in a simulator). Why? Well, it's useful to see that various test suites are actually testing what you\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/4057","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/comments?post=4057"}],"version-history":[{"count":1,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/4057\/revisions"}],"predecessor-version":[{"id":4062,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/4057\/revisions\/4062"}],"wp:attachment":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/media?parent=4057"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/categories?post=4057"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/tags?post=4057"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}