{"id":3752,"date":"2014-06-03T12:17:49","date_gmt":"2014-06-03T02:17:49","guid":{"rendered":"https:\/\/www.flamingspork.com\/blog\/?p=3752"},"modified":"2014-06-03T12:03:14","modified_gmt":"2014-06-03T02:03:14","slug":"mysql-5-6-on-power-patch-available","status":"publish","type":"post","link":"https:\/\/www.flamingspork.com\/blog\/2014\/06\/03\/mysql-5-6-on-power-patch-available\/","title":{"rendered":"MySQL 5.6 on POWER (patch available)"},"content":{"rendered":"<p>The following sentence is brought to you by IBM Legal. The postings on this site are my own and don&#8217;t necessarily represent IBM&#8217;s positions, strategies or opinions.<\/p>\n<p>Okay, now that is out of the way&#8230;.<\/p>\n<p>If you&#8217;re the kind of person who follows the <a href=\"http:\/\/bugs.mysql.com\/bug.php?id=47213\">MySQL<\/a> <a href=\"http:\/\/bugs.mysql.com\/bug.php?id=72754\">bugs<\/a> <a href=\"http:\/\/bugs.mysql.com\/bug.php?id=72718\">database<\/a> closely or subscribes to the <a href=\"http:\/\/lists.mysql.com\/internals\/38786\">MySQL Internals<\/a> mailing list, you may have worked out that I&#8217;ve spent a small amount of time poking at MySQL on modern POWER systems.<\/p>\n<p>Unlike Intel CPUs, POWER CPUs require explicit memory barriers to synchronize memory state between different CPUs. This means that when you&#8217;re implementing synchronization primitives, you have one extra thing to get right.<\/p>\n<p>Luckily, if you use straight pthread mutexes, this is already taken care of. Unluckily, there are some optimizations in MySQL that don&#8217;t use straight pthread mutexes and so may be problematic on non-Intel CPUs. A few of these issues have sneaked into MySQL over the past few years. The most problematic area was around the optimized mutexes in InnoDB (you can use the pthread_mutex fallback code, but it&#8217;s less performant).<\/p>\n<p>Luckily, I both knew where to look and there are good asserts throughout InnoDB code to help spot any other areas that I may not have initially thought of to look at. Coding defensively with a good amount of asserts is a <strong>good thing<\/strong>.<\/p>\n<p>After not too much work, I have a set of patches that I&#8217;m fairly confident is correct and performs near as well as possible. Initially, I had a different patch that used heavyweight memory barriers in a lot of places, but big kudos to Yasufumi for posting a better patch than mine to <a href=\"http:\/\/bugs.mysql.com\/bug.php?id=47213\">bug 47213<\/a> &#8211; using the lighter weight barriers gives a decent performance boost.<\/p>\n<p>One of the key patches is in the InnoDB mutex code to change the thread priority &#8211; i.e. a POWER equivalent to the x86 pause instruction. These are hints to the CPU that the thread being executed is in a spinloop and CPU resources should be allocated to other threads to make betterr forward progress.<\/p>\n<p>After dragging <a href=\"http:\/\/antonblanchardfacts.com\/\">Anton<\/a> in to have a look and a think, this code may have motivated him to have a go at getting kernel support for adaptive mutexes, thus removing the need for this spin\/sleep\/yield\/eep loop in InnoDB (at least on Linux).<\/p>\n<p>So&#8230; I&#8217;ve spent the appropriate time filing bugs in the MySQL bug tracker for the things I&#8217;ve found. Feel free to track them yourself, they are:<\/p>\n<ul>\n<li><a href=\"http:\/\/bugs.mysql.com\/bug.php?id=72715\">Bug 72715<\/a>: character set code endianness dependent on CPU type rather than endianness of CP\n<ul>\n<li>I don&#8217;t think this is an issue for us&#8230; or it could be that this is actually just incredibly untested code in the MySQL Server. It&#8217;s also not POWER specific, although was caught by the Migration Assistant which is part of the Advanced Toolchain from IBM.<\/li>\n<\/ul>\n<\/li>\n<li><a href=\"http:\/\/bugs.mysql.com\/bug.php?id=72718\">Bug 72718<\/a>: CACHE_LINE_SIZE in innodb should be 128 on POWER\n<ul>\n<li>I contributed a patch that&#8217;s a simple #ifdef for CPU type. Those who care about other CPU architectures should chime in with the correct value for them.<\/li>\n<li>There&#8217;s other places in InnoDB where there&#8217;s some padding that don&#8217;t use this define, I need to file a bug for that.<\/li>\n<\/ul>\n<\/li>\n<li><a href=\"http:\/\/bugs.mysql.com\/bug.php?id=72754\">Bug 72754<\/a>: Set thread priority in InnoDB mutex spinloop\n<ul>\n<li>This makes a big difference when you have mutex contention and SMT (Symmetric Multi-Threading) enabled (on POWER, you can dynamically change SMT levels at runtime).<\/li>\n<li>I&#8217;ve contributed a preliminary patch that isn&#8217;t generic. I should go and fix that.<\/li>\n<\/ul>\n<\/li>\n<li><a href=\"http:\/\/bugs.mysql.com\/bug.php?id=72755\">Bug 72755<\/a>: InnoDB mutex spin loop is missing GCC barrier\n<ul>\n<li>This also applies to x86 (and indeed all platforms). If GCC gets a bit smarter, the current code could compile down to nothing, which is exactly what you don&#8217;t want from a spinloop. The correct thing to do is to have a GCC memory barrier (not CPU one) to ensure that the compiler doesn&#8217;t optimize away the spinning.<\/li>\n<li>I&#8217;ve contributed a patch, may need #ifdef GCC added.<\/li>\n<\/ul>\n<\/li>\n<li><a href=\"http:\/\/bugs.mysql.com\/bug.php?id=72809\">Bug 72809<\/a>: InnoDB Linux native aio setup missing barrier after setup\n<ul>\n<li>This appears to be a &#8220;POWER8 is fast&#8221; related bug :)<\/li>\n<li>Patch contributed.<\/li>\n<\/ul>\n<\/li>\n<li><a href=\"http:\/\/bugs.mysql.com\/bug.php?id=72811\">Bug 72811<\/a>: Set NUMA mempolicy for optimum mysqld performance\n<ul>\n<li>Not POWER specific.<\/li>\n<li>I&#8217;ve contributed a patch that sets NUMA memory allocation policy inside mysqld rather than having to run &#8220;numactl&#8221; manually<\/li>\n<\/ul>\n<\/li>\n<li><a href=\"http:\/\/bugs.mysql.com\/bug.php?id=47213\">Bug 47213<\/a>: InnoDB mutex\/rw_lock should be conscious about memory ordering other than Intel\n<ul>\n<li>Originally filed by Yasufumi back in 2009.<\/li>\n<li>Some good discussion going on here to ensure the patch is correct. This is the kind of patch that requires more review\u00c2\u00a0 than it takes to write it.<\/li>\n<li>This patch would fix the majority of problems for non-Intel CPU architectures.<\/li>\n<li>Thanks to Yasufumi for providing an updated patch, it helped a lot!<\/li>\n<\/ul>\n<\/li>\n<li><a href=\"http:\/\/bugs.mysql.com\/bug.php?id=72544\">Bug 72544<\/a>: Incorrect locking for global_query_id\n<ul>\n<li>I found a bug. Rather benign and not POWER specific.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>Want to run MySQL 5.6.17 on POWER? Get my MySQL 5.6.17 patch here: <a href=\"https:\/\/flamingspork.com\/mysql\/mysql-5.6.17-POWER.patch\">https:\/\/flamingspork.com\/mysql\/mysql-5.6.17-POWER.patch<\/a><\/p>\n<p>My accumulation of 5.6 patches seems fairly reliable. I&#8217;d test before putting into production, and I&#8217;d certainly love to know any problems you hit.<\/p>\n<p>Get the quilt series of patches here: <a href=\"https:\/\/flamingspork.com\/mysql\/mysql-5.6.17-POWER-patches.tar.gz\">https:\/\/flamingspork.com\/mysql\/mysql-5.6.17-POWER-patches.tar.gz<\/a><\/p>\n<p>I have, of course, done the legal wrangling for the Oracle Contributor Agreement (remarkably painless) and am working on making the patches completely acceptable to be merged into MySQL.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The following sentence is brought to you by IBM Legal. The postings on this site are my own and don&#8217;t necessarily represent IBM&#8217;s positions, strategies or opinions. Okay, now that is out of the way&#8230;. If you&#8217;re the kind of &hellip; <a href=\"https:\/\/www.flamingspork.com\/blog\/2014\/06\/03\/mysql-5-6-on-power-patch-available\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[76,1,570,14],"tags":[86,578,628,577,562],"class_list":["post-3752","post","type-post","status-publish","format-standard","hentry","category-code","category-general","category-ibm-work-et-al","category-mysql","tag-gcc","tag-memory-barrier","tag-mysql","tag-mysql-5-6","tag-power"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5a6n8-Yw","jetpack-related-posts":[{"id":3758,"url":"https:\/\/www.flamingspork.com\/blog\/2014\/06\/03\/mysql-5-7-on-power\/","url_meta":{"origin":3752,"position":0},"title":"MySQL 5.7 on POWER","author":"Stewart Smith","date":"2014-06-03","format":false,"excerpt":"In a previous post, I covered porting MySQL 5.6 to POWER and subsequently, some new record performance numbers with MySQL 5.6.17 on POWER8. Well, those following at home will be aware that not only is the next sentence sponsored by IBM Legal, but that MySQL 5.7 alleviates a bunch of\u2026","rel":"","context":"In &quot;code&quot;","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3884,"url":"https:\/\/www.flamingspork.com\/blog\/2014\/10\/14\/mysql-5-7-5-on-power-thread-priority\/","url_meta":{"origin":3752,"position":1},"title":"MySQL 5.7.5 on POWER &#8211; thread priority","author":"Stewart Smith","date":"2014-10-14","format":false,"excerpt":"Good news everyone! MySQL 5.7.5 is out with a bunch more patches for running well on POWER in the tree. I haven't yet gone and tried it all out, but since I'm me, I look at bugs database and git\/bzr history first. On Intel CPUs, when you're spinning on a\u2026","rel":"","context":"In &quot;IBM&quot;","block_context":{"text":"IBM","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/ibm-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3901,"url":"https:\/\/www.flamingspork.com\/blog\/2014\/11\/12\/preliminary-mysql-cluster-benchmark-results-on-power8\/","url_meta":{"origin":3752,"position":2},"title":"Preliminary MySQL Cluster benchmark results on POWER8","author":"Stewart Smith","date":"2014-11-12","format":false,"excerpt":"Yesterday, I got the basics going for MySQL Cluster on POWER. Today, I finished up a couple more patches to improve performance and ran some benchmarks. This is on a 3.7Ghz POWER8 machine with non-balanced memory (only 2 of the 4 NUMA nodes have memory, so we have less total\u2026","rel":"","context":"In &quot;code&quot;","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4019,"url":"https:\/\/www.flamingspork.com\/blog\/2015\/12\/18\/power8-accelerated-crc32-merged-in-mariadb-10-1\/","url_meta":{"origin":3752,"position":3},"title":"POWER8 Accelerated CRC32 merged in MariaDB 10.1","author":"Stewart Smith","date":"2015-12-18","format":false,"excerpt":"Earlier on in benchmarking MySQL and MariaDB on POWER8, we noticed that on write workloads (or read workloads involving a lot of IO) we were spending a bunch of time computing InnoDB page checksums. This is a relatively well known MySQL problem and has existed for many years and Percona\u2026","rel":"","context":"In &quot;IBM&quot;","block_context":{"text":"IBM","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/ibm-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3778,"url":"https:\/\/www.flamingspork.com\/blog\/2014\/07\/17\/update-on-mysql-on-power8\/","url_meta":{"origin":3752,"position":4},"title":"Update on MySQL on POWER8","author":"Stewart Smith","date":"2014-07-17","format":false,"excerpt":"About 1.5 months ago I blogged on MySQL 5.6 on POWER andtalked about what I had to poke at to make modern MySQL versions run and run well on shiny POWER8 systems. One of those bugs, MySQL bug 47213 (InnoDB mutex\/rw_lock should be conscious of memory ordering other than Intel)\u2026","rel":"","context":"In &quot;code&quot;","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3755,"url":"https:\/\/www.flamingspork.com\/blog\/2014\/06\/03\/mysql-5-6-performance-on-power8\/","url_meta":{"origin":3752,"position":5},"title":"MySQL 5.6 Performance on POWER8","author":"Stewart Smith","date":"2014-06-03","format":false,"excerpt":"The following sentence is brought to you by IBM Legal: The postings on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. My previous post covered the work needed to get MySQL 5.6.17 running reliably on modern POWER systems. The patch to MySQL 5.6.17 that's\u2026","rel":"","context":"In &quot;IBM&quot;","block_context":{"text":"IBM","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/ibm-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/3752","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/comments?post=3752"}],"version-history":[{"count":2,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/3752\/revisions"}],"predecessor-version":[{"id":3756,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/3752\/revisions\/3756"}],"wp:attachment":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/media?parent=3752"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/categories?post=3752"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/tags?post=3752"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}