{"id":1876,"date":"2010-04-02T00:14:32","date_gmt":"2010-04-01T14:14:32","guid":{"rendered":"http:\/\/www.flamingspork.com\/blog\/?p=1876"},"modified":"2010-04-02T00:15:30","modified_gmt":"2010-04-01T14:15:30","slug":"the-drizzle-and-mysql-key-tuple-format","status":"publish","type":"post","link":"https:\/\/www.flamingspork.com\/blog\/2010\/04\/02\/the-drizzle-and-mysql-key-tuple-format\/","title":{"rendered":"The Drizzle (and MySQL) Key tuple format"},"content":{"rendered":"<p>Here&#8217;s something that&#8217;s not really documented anywhere (unless you count ha_innodb.cc as a source of server documentation). You may have some idea about the MySQL\/Drizzle row buffer format. This is passed around the storage engine interface: in for write_row and update_row and out for the various scan and index read methods.<\/p>\n<p>If you want to see the docs for it that exist in the code, check out store_key_val_for_row in ha_innodb.cc.<\/p>\n<p>However, there is another format that is passed to your engine (and that your engine is expected to understand) and for lack of a better name, I&#8217;m going to call it the key tuple format. The first place you&#8217;ll probably see this is when implementing the index_read function for a Cursor (or handler in MySQL speak).<\/p>\n<p>You get two things: a pointer to the buffer and the length of the buffer. Since a key can be made up of multiple parts, some of which can be NULL and some of which can be of variable length, this buffer is not (usually) a simple value. If you are starting out in your engine development, you can use this buffer blindly as a single value for non-nullable indexes with only 1 column.<\/p>\n<p>The basic format is this:<\/p>\n<ul>\n<li>The buffer is in-order of the index. First column in the index is first in the buffer, second second etc.<\/li>\n<li>The buffer must be zero-filled. The server kernel will use memcmp to compare two key values.<\/li>\n<li>If the column is NULLable, then the first byte is set to 1 if the column is null. Else, 0 means not-null.<\/li>\n<li>From ha_innodb.cc (for BLOBs, which I haven&#8217;t put in embedded_innodb yet):<em> If the column is of a BLOB type (it must be a column prefix field\u00c2\u00a0in this case), then we put the length of the data in the field to the\u00c2\u00a0next 2 bytes, in the little-endian format. If the field is SQL NULL,\u00c2\u00a0then these 2 bytes are set to 0. Note that the length of data in the\u00c2\u00a0field is &lt;= column prefix length.<\/em><\/li>\n<li>For fixed length fields (such as int), the next max field length bytes are for that field.<\/li>\n<li>For VARCHAR, there is always a 2 byte (in little endian) length. This is different to the row format, which may have 1 or 2 bytes. In the key tuple format it is ALWAYS two bytes.<\/li>\n<\/ul>\n<p>I&#8217;ll discuss the use of this for rnd_pos() and position() in a later post&#8230;<\/p>\n<p>This blog post (but not the whole blog) is published under the <a href=\"http:\/\/creativecommons.org\/licenses\/by-sa\/3.0\/\">Creative Commons Attribution-Share Alike License<\/a>. Attribution is by linking back to this post and mentioning my name (Stewart Smith).<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Here&#8217;s something that&#8217;s not really documented anywhere (unless you count ha_innodb.cc as a source of server documentation). You may have some idea about the MySQL\/Drizzle row buffer format. This is passed around the storage engine interface: in for write_row and &hellip; <a href=\"https:\/\/www.flamingspork.com\/blog\/2010\/04\/02\/the-drizzle-and-mysql-key-tuple-format\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[75,1,14],"tags":[70,278,276,628,281,280,279,80,242,277],"class_list":["post-1876","post","type-post","status-publish","format-standard","hentry","category-drizzle-work-et-al","category-general","category-mysql","tag-drizzle","tag-format","tag-key","tag-mysql","tag-position","tag-rnd_init","tag-storage-engine","tag-storage-engine-api","tag-storageengine","tag-tuple"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5a6n8-ug","jetpack-related-posts":[{"id":2168,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/10\/25\/storage-engine-api-state-graph\/","url_meta":{"origin":1876,"position":0},"title":"Storage Engine API state graph","author":"Stewart Smith","date":"2010-10-25","format":false,"excerpt":"Drizzle still has a number of quirks inherited from the MySQL Storage Engine API (e.g. BLOBs, row buffer, CREATE SELECT and lack of DDL transaction boundaries, key tuple format). One of the things we fixed a long time ago was to have proper methods for StorageEngines to be called for:\u2026","rel":"","context":"In &quot;code&quot;","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"State transitions for a transaction. Transaction can be empty OR have one or more statements","src":"https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2010\/10\/edge30-300x163.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":1409,"url":"https:\/\/www.flamingspork.com\/blog\/2009\/02\/20\/improving-the-storage-engine-api\/","url_meta":{"origin":1876,"position":1},"title":"Improving the Storage Engine &#8220;API&#8221;","author":"Stewart Smith","date":"2009-02-20","format":false,"excerpt":"I increasingly enclose the API part of \"Storage Engine API\" in quotes as it does score a rather large number on the API Design Rusty levels (Coined by Rusty Russell). I give it a 15 (out of 18. lower is better) in this case \"The obvious use is wrong\". The\u2026","rel":"","context":"In &quot;code&quot;","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3234,"url":"https:\/\/www.flamingspork.com\/blog\/2013\/04\/04\/refactoring-internal-temporary-tables-another-stab-at-it\/","url_meta":{"origin":1876,"position":2},"title":"Refactoring Internal temporary tables (another stab at it)","author":"Stewart Smith","date":"2013-04-04","format":false,"excerpt":"A few weekends ago, I started to again look at the code in Drizzle for producing internal temporary tables. Basically, we have a few type of tables: Standard Temporary (from CREATE TEMPORARY TABLE) Temporary (from ALTER TABLE) Internal temporary (to help with query execution) If you're lucky enough to be\u2026","rel":"","context":"In &quot;code&quot;","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1879,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/05\/24\/using-the-row-buffer-in-drizzle-and-mysql\/","url_meta":{"origin":1876,"position":3},"title":"Using the row buffer in Drizzle (and MySQL)","author":"Stewart Smith","date":"2010-05-24","format":false,"excerpt":"Here's another bit of the API you may need to use in your storage engine (it also seems to be a rather unknown. I believe the only place where this has really been documented is ha_ndbcluster.cc, so here goes.... Drizzle (through inheritance from MySQL) has its own (in memory) row\u2026","rel":"","context":"In &quot;drizzle&quot;","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2010\/04\/row-300x85.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":2253,"url":"https:\/\/www.flamingspork.com\/blog\/2011\/01\/05\/is-your-storage-engine-buggy-or-the-database-server\/","url_meta":{"origin":1876,"position":4},"title":"Is your Storage Engine buggy or the database server?","author":"Stewart Smith","date":"2011-01-05","format":false,"excerpt":"If your storage engine returns an error from rnd_init (or doStartTableScan as it's named in Drizzle) and does not save this error and return it in any subsequent calls to rnd_next, your engine is buggy. Namely it is buggy in that a) an error may not be reported back to\u2026","rel":"","context":"In &quot;code&quot;","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1813,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/03\/12\/writing-a-storage-engine-for-drizzle-part-2-create-table\/","url_meta":{"origin":1876,"position":5},"title":"Writing A Storage Engine for Drizzle, Part 2: CREATE TABLE","author":"Stewart Smith","date":"2010-03-12","format":false,"excerpt":"The DDL code paths for Drizzle are increasingly different from MySQL. For example, the embedded_innodb StorageEngine CREATE TABLE code path is completely different than what it would have to be for MySQL. This is because of a number of reasons, the primary one being that Drizzle uses a protobuf message\u2026","rel":"","context":"In &quot;drizzle&quot;","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/1876","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/comments?post=1876"}],"version-history":[{"count":3,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/1876\/revisions"}],"predecessor-version":[{"id":1882,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/1876\/revisions\/1882"}],"wp:attachment":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/media?parent=1876"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/categories?post=1876"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/tags?post=1876"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}