{"id":3906,"date":"2014-11-14T10:53:02","date_gmt":"2014-11-14T00:53:02","guid":{"rendered":"https:\/\/www.flamingspork.com\/blog\/?p=3906"},"modified":"2014-11-13T15:12:27","modified_gmt":"2014-11-13T05:12:27","slug":"c-bitfields-considered-harmful","status":"publish","type":"post","link":"https:\/\/www.flamingspork.com\/blog\/2014\/11\/14\/c-bitfields-considered-harmful\/","title":{"rendered":"C bitfields considered harmful"},"content":{"rendered":"

In C (and C++) you can specify that a variable should take a specific number of bits of storage by doing “uint32_t foo:4;” rather than just “uint32_t foo”. In this example, the former uses 4 bits while the latter uses 32bits. This can be useful to pack many bit fields together.<\/p>\n

Or, that’s what they’d like<\/strong> you to think.<\/p>\n

In reality, the C spec allows the compiler to do just about anything it wants<\/strong> with these bitfields – which usually means it’s something you didn’t expect.<\/p>\n

For a start, in a struct -e.g. “struct foo { uint32_t foo:4; uint32_t blah; uint32_t blergh:20; }” the compiler could go and combine foo and blergh into a single uint32_t and place it somewhere… or it could not. In this case, sizeof(struct foo) isn’t defined and may vary based on compiler, platform, compiler version, phases of the moon or if you’ve washed your hands recently.<\/p>\n

Where this can get interesting is in network protocols (OMG DO NOT DO IT), APIs (OMG DO NOT DO IT), protecting different parts of a struct with different mutexes (EEP, don’t do it!) and performance.<\/p>\n

I recently filed MySQL bug 74831<\/a> which relates to InnoDB performance on POWER8. InnoDB uses C bitfields which are themselves bitfields (urgh) for things like “flag to say if this table is compressed”. At various parts of the code, this flag is checked.<\/p>\n

When you apply this simple patch:<\/p>\n

--- mysql-5.7.5-m15.orig\/storage\/innobase\/include\/dict0mem.h\r\n+++ mysql-5.7.5-m15\/storage\/innobase\/include\/dict0mem.h\r\n@@ -1081,7 +1081,7 @@ struct dict_table_t {\r\n        Use DICT_TF_GET_COMPACT(), DICT_TF_GET_ZIP_SSIZE(),\r\n        DICT_TF_HAS_ATOMIC_BLOBS() and DICT_TF_HAS_DATA_DIR() to parse this\r\n        flag. *\/\r\n-       unsigned                                flags:DICT_TF_BITS;\r\n+       unsigned                                flags;\r\n<\/pre>\n
I get 10,000 key lookups\/sec more than without it!<\/p>\n
Why is this? If you go and read the bug, you’ll see that the amount of CPU time spent on the instruction checking the bit flag is actually about the same… and this puzzled me for a while. That is, until Anton<\/a> reminded me that the PMU can be approximate and perhaps I should look at the loads.<\/p>\n
Sure enough, the major difference is that with the bitfield in place (i.e. MySQL 5.7.5 as it stands today), there is a ld instruction doing the load – which is a 64bit load. In my patched version, it’s a lwx instruction – which is a 32bit load.<\/p>\n
So, basically, we were loading 8 bytes instead of 4 every time we were checking if it was a compressed table.<\/p>\n
So, along with yesterday’s lesson of never, ever, ever use volatile<\/strong><\/a>, today’s lesson is never, ever, ever use bitfields<\/strong>.<\/p>\n","protected":false},"excerpt":{"rendered":"
In C (and C++) you can specify that a variable should take a specific number of bits of storage by doing “uint32_t foo:4;” rather than just “uint32_t foo”. In this example, the former uses 4 bits while the latter uses … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[76,570,14],"tags":[257,628,568,637,39],"class_list":["post-3906","post","type-post","status-publish","format-standard","hentry","category-code","category-ibm-work-et-al","category-mysql","tag-innodb","tag-mysql","tag-power8","tag-powerpc","tag-programming"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5a6n8-110","jetpack-related-posts":[{"id":2416,"url":"https:\/\/www.flamingspork.com\/blog\/2011\/11\/21\/pandora-build-autotools-made-easy\/","url_meta":{"origin":3906,"position":0},"title":"pandora-build: autotools made easy","author":"Stewart Smith","date":"2011-11-21","format":false,"excerpt":"Way back in 2009, Monty Taylor got fed up with maintaining a set of common autotools foo across several projects (one of which was Drizzle) and started the pandora-build project. \u00c2\u00a0Basically, it's a collection of the foo you need for autotools to do things like: use it properly, detect a\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":499,"url":"https:\/\/www.flamingspork.com\/blog\/2005\/10\/27\/__packed-2\/","url_meta":{"origin":3906,"position":1},"title":"__packed","author":"Stewart Smith","date":"2005-10-27","format":false,"excerpt":"Blog | rml struct __packed s { ... }This attribute tells GCC that a type or variable should be packed into memory, using the minimum amount of space possible, potentially disregarding alignment requirements. If specified on a struct or union, all variables therein are so packed. If specified on just\u2026","rel":"","context":"In "mysql"","block_context":{"text":"mysql","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/mysql\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1008,"url":"https:\/\/www.flamingspork.com\/blog\/2008\/02\/11\/getting-a-file-size-on-windows\/","url_meta":{"origin":3906,"position":2},"title":"Getting a file size (on Windows)","author":"Stewart Smith","date":"2008-02-11","format":false,"excerpt":"The first point I'd like to make is that you're using a Microsoft Windows API, so you have already lost. You are just not quite aware of how much you have lost. A quick look around and you say \"Ahh... GetFileSize, that's what I want to do!\" Except, of course,\u2026","rel":"","context":"In "grumble"","block_context":{"text":"grumble","link":"https:\/\/www.flamingspork.com\/blog\/category\/grumble\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1855,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/03\/17\/stored-proceduresfunctions-for-drizzle\/","url_meta":{"origin":3906,"position":3},"title":"Stored Procedures\/Functions for Drizzle","author":"Stewart Smith","date":"2010-03-17","format":false,"excerpt":"Previously, in \"Thoughts on Thoughts on Drizzle\" I theorized that one of the major reasons why we did not see lots of people jumping at stored procedures in MySQL was that it wasn't in their native language (for lack of a better term). We've seen External Language Stored Procedures for\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":562,"url":"https:\/\/www.flamingspork.com\/blog\/2006\/01\/20\/mysql-can-mysql-be-run-from-a-cdrom\/","url_meta":{"origin":3906,"position":4},"title":"MySQL: Can MySQL be run from a CDROM?","author":"Stewart Smith","date":"2006-01-20","format":false,"excerpt":"MySQL: Can MySQL be run from a CDROM? The question is asked. The answer is - yes! Temporary tables can be store anywhere - e.g. the system's temporary directory (\/tmp on unix, probably c:\\temp or something on windows... i'm sure there's one of those %FOO% things for it). IIRC you\u2026","rel":"","context":"In "mysql"","block_context":{"text":"mysql","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/mysql\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3912,"url":"https:\/\/www.flamingspork.com\/blog\/2014\/12\/03\/running-skiboot-opal-on-the-power8-simulator\/","url_meta":{"origin":3906,"position":5},"title":"Running skiboot (OPAL) on the POWER8 Simulator","author":"Stewart Smith","date":"2014-12-03","format":false,"excerpt":"skiboot is open source boot and runtime firmware for OpenPOWER. On real POWER8 hardware, you will also need HostBoot to do this (basically, to make the chip work) but in a functional simulator (such as this one released by IBM) you don't need a bunch of hardware procedures to make\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/3906","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/comments?post=3906"}],"version-history":[{"count":1,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/3906\/revisions"}],"predecessor-version":[{"id":3907,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/3906\/revisions\/3907"}],"wp:attachment":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/media?parent=3906"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/categories?post=3906"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/tags?post=3906"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}