{"id":1879,"date":"2010-05-24T17:21:31","date_gmt":"2010-05-24T07:21:31","guid":{"rendered":"http:\/\/www.flamingspork.com\/blog\/?p=1879"},"modified":"2014-10-08T09:16:07","modified_gmt":"2014-10-07T23:16:07","slug":"using-the-row-buffer-in-drizzle-and-mysql","status":"publish","type":"post","link":"https:\/\/www.flamingspork.com\/blog\/2010\/05\/24\/using-the-row-buffer-in-drizzle-and-mysql\/","title":{"rendered":"Using the row buffer in Drizzle (and MySQL)"},"content":{"rendered":"

Here’s another bit of the API you may need to use in your storage engine (it also seems to be a rather unknown. I believe the only place where this has really been documented is ha_ndbcluster.cc, so here goes….<\/p>\n

Drizzle<\/a> (through inheritance from MySQL<\/a>) has its own (in memory) row format (it could be said that it has several, but we’ll ignore that for the moment for sanity). This is used inside the server for a number of things. When writing a Storage Engine all you really need to know is that you’re expected to write these into your engine and return them from your engine.<\/p>\n

The row buffer format itself is kind-of documented (in that it’s mentioned in the MySQL Internals documentation<\/a>) but everywhere that’s ever pointed to makes the (big) assumption that you’re going to be implementing an engine that just uses a more compact variant of the in-memory row format. The notable exception is the CSV engine, which only ever cares about textual representations of data (calling val_str() on a Field is pretty simple).<\/p>\n

The basic layout is a NULL bitmap plus the data for each non-null column:<\/p>\n

\"\"<\/a>Except that the NULL bitmap is byte aligned. So in the above diagram, with four nullable columns, it would actually be padded out to 1 byte:<\/p>\n

\"\"<\/a><\/p>\n

Each column is stored in a type-specific way.<\/p>\n

Each Table (an instance of an open table which a Cursor is used to iterate over parts of) has two row buffers in it: record[0] and record[1]. For the most part, the Cursor implementation for your Storage Engine only ever has to deal with record[0]. However, sometimes you may be asked to read a row into record[1], so your engine must deal with that too.<\/p>\n

A Row (no, there’s no object for that… you just get a pointer to somewhere in memory) is made up of Fields (as in Field objects). It’s really made up of lots of things, but if you’re dealing with the row format, a row is made up of fields. The Field objects let you get the value out of a row in a number of ways. For an integer column, you can call Field::val_int() to get the value as an integer, or you can call val_str() to get it as a string (this is what the CSV engine does, just calls val_str() on each Field).<\/p>\n

The Field objects are not part of a row in any way. They instead have a pointer to record[0] stored in them. This doesn’t help you if you need to access record[1] (because that can be passed into your Cursor methods). Although the buffer passed into various Cursor methods is usually<\/strong> record[0] it is not always<\/strong> record[0]. How do you use the Field objects to access fields in the row buffer then? The answer is the Field::move_field_offset(ptrdiff_t) method. Here is how you can use it in your code:<\/p>\n

\r\n
ptrdiff_t row_offset= buf - table->record[0];<\/div>\r\n
(**field).move_field_offset(row_offset);<\/div>\r\n
(do things with field)<\/div>\r\n
(**field).move_field_offset(-row_offset);<\/div><\/pre>\n
Yes, this API completely sucks and is very easy to misuse and abuse – especially in error handling cases. We’re currently discussing some alternatives for Drizzle.<\/p>\n
This blog post (but not the whole blog) is published under the\u00c2\u00a0Creative Commons Attribution-Share Alike License<\/a>. Attribution is by linking back to this post and mentioning my name (Stewart Smith).<\/p>\n","protected":false},"excerpt":{"rendered":"
Here’s another bit of the API you may need to use in your storage engine (it also seems to be a rather unknown. I believe the only place where this has really been documented is ha_ndbcluster.cc, so here goes…. Drizzle … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[75,14],"tags":[285,70,146,628,284,80,242],"class_list":["post-1879","post","type-post","status-publish","format-standard","hentry","category-drizzle-work-et-al","category-mysql","tag-documentation","tag-drizzle","tag-internals","tag-mysql","tag-row-format","tag-storage-engine-api","tag-storageengine"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5a6n8-uj","jetpack-related-posts":[{"id":1876,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/04\/02\/the-drizzle-and-mysql-key-tuple-format\/","url_meta":{"origin":1879,"position":0},"title":"The Drizzle (and MySQL) Key tuple format","author":"Stewart Smith","date":"2010-04-02","format":false,"excerpt":"Here's something that's not really documented anywhere (unless you count ha_innodb.cc as a source of server documentation). You may have some idea about the MySQL\/Drizzle row buffer format. This is passed around the storage engine interface: in for write_row and update_row and out for the various scan and index read\u2026","rel":"","context":"In "drizzle"","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1409,"url":"https:\/\/www.flamingspork.com\/blog\/2009\/02\/20\/improving-the-storage-engine-api\/","url_meta":{"origin":1879,"position":1},"title":"Improving the Storage Engine “API”","author":"Stewart Smith","date":"2009-02-20","format":false,"excerpt":"I increasingly enclose the API part of \"Storage Engine API\" in quotes as it does score a rather large number on the API Design Rusty levels (Coined by Rusty Russell). I give it a 15 (out of 18. lower is better) in this case \"The obvious use is wrong\". The\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1931,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/05\/26\/blobs-in-the-drizzlemysql-storage-engine-api\/","url_meta":{"origin":1879,"position":2},"title":"BLOBS in the Drizzle\/MySQL Storage Engine API","author":"Stewart Smith","date":"2010-05-26","format":false,"excerpt":"Another (AFAIK) undocumented part of the Storage Engine API: We all know what a normal row looks like in Drizzle\/MySQL row format (a NULL bitmap and then column data): Nothing that special. It's a fixed sized buffer, Field objects reference into it, you read out of it and write the\u2026","rel":"","context":"In "drizzle"","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2010\/05\/rect28161-300x67.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":1745,"url":"https:\/\/www.flamingspork.com\/blog\/2009\/12\/09\/drizzle-frm-replacement-the-table-proto\/","url_meta":{"origin":1879,"position":3},"title":"Drizzle FRM replacement: the table proto","author":"Stewart Smith","date":"2009-12-09","format":false,"excerpt":"Drizzle originally inherited the FRM file from MySQL (which inherited it from UNIREG). The FRM file stores metadata about a table; what columns it has, what type those columns are, what indexes, any default values, comments etc are all stored in the FRM. In the days of MyISAM, this worked\u2026","rel":"","context":"In "drizzle"","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2168,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/10\/25\/storage-engine-api-state-graph\/","url_meta":{"origin":1879,"position":4},"title":"Storage Engine API state graph","author":"Stewart Smith","date":"2010-10-25","format":false,"excerpt":"Drizzle still has a number of quirks inherited from the MySQL Storage Engine API (e.g. BLOBs, row buffer, CREATE SELECT and lack of DDL transaction boundaries, key tuple format). One of the things we fixed a long time ago was to have proper methods for StorageEngines to be called for:\u2026","rel":"","context":"In "code"","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"State transitions for a transaction. Transaction can be empty OR have one or more statements","src":"https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2010\/10\/edge30-300x163.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":2303,"url":"https:\/\/www.flamingspork.com\/blog\/2011\/03\/16\/fixed-in-drizzle-no-more-gotchas\/","url_meta":{"origin":1879,"position":5},"title":"Fixed in Drizzle: No more “GOTCHA’s”","author":"Stewart Smith","date":"2011-03-16","format":false,"excerpt":"\u00a0 At the upcoming MySQL Conference and Expo, I'm going to give a Thursday afternoon (2pm) session entitled Fixed in Drizzle: No more \"GOTCHA's\". I plan to have a lot of fun with this session.. If you go back to the very start of when I started submitting code to\u2026","rel":"","context":"In "drizzle"","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/1879","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/comments?post=1879"}],"version-history":[{"count":6,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/1879\/revisions"}],"predecessor-version":[{"id":3837,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/1879\/revisions\/3837"}],"wp:attachment":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/media?parent=1879"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/categories?post=1879"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/tags?post=1879"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}