{"id":1874,"date":"2010-03-30T15:06:17","date_gmt":"2010-03-30T05:06:17","guid":{"rendered":"http:\/\/www.flamingspork.com\/blog\/?p=1874"},"modified":"2014-10-08T09:16:07","modified_gmt":"2014-10-07T23:16:07","slug":"storing-the-table-message-in-embedded-innodb","status":"publish","type":"post","link":"https:\/\/www.flamingspork.com\/blog\/2010\/03\/30\/storing-the-table-message-in-embedded-innodb\/","title":{"rendered":"Storing the table message in Embedded InnoDB"},"content":{"rendered":"<p>One of the exciting things[1] about working on a storage engine in <a href=\"http:\/\/www.drizzle.org\">Drizzle<\/a> is that you get to manage your own metadata. When the database engine you&#8217;re writing the storage engine interface for has a pretty complete data dictionary (e.g. <a href=\"http:\/\/www.innodb.com\/wp\/products\/embedded-innodb\/\">Embedded InnoDB<\/a>) you could just directly use it. At some point I plan to do this for the embedded_innodb engine for Drizzle so that you could just point Drizzle at an existing Embedded InnoDB database and run SQL queries on it.<\/p>\n<p>The Drizzle table message does have some things in it that aren&#8217;t in the InnoDB data dictionary though (e.g. table and column comments). We want to preserve these (and also things like there may be several data types in Drizzle that map to the same data type in InnoDB). Since the Embedded InnoDB API allows us to do things within the DDL transaction (such as insert a row into a table), we store the serialized table message in a table as part of the DDL transaction. This means we can have <strong>fully crash safe DDL<\/strong>! There is no way the table definition can get out of sync with what is in InnoDB; we are manipulating them both in the same transaction!<\/p>\n<p>The table structure we&#8217;re using is pretty simple. There is two columns: table_name VARCHAR(IB_MAX_TABLE_NAME_LEN) and message BLOB.<\/p>\n<p>The operations we need are:<\/p>\n<ul>\n<li>store the table message in doCreateTable (INSERT)<\/li>\n<li>rename the table message in doRenameTable (UPDATE the table_name column)<\/li>\n<li>delete the table message in doDropTable (DELETE)<\/li>\n<li>list tables in a database (SELECT with prefix)<\/li>\n<li>get table message (SELECT using key lookup)<\/li>\n<\/ul>\n<p>All of which are pretty easy to implement using the Embedded InnoDB API.<\/p>\n<p>[1] Maybe I need to get out more&#8230;.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>One of the exciting things[1] about working on a storage engine in Drizzle is that you get to manage your own metadata. When the database engine you&#8217;re writing the storage engine interface for has a pretty complete data dictionary (e.g. &hellip; <a href=\"https:\/\/www.flamingspork.com\/blog\/2010\/03\/30\/storing-the-table-message-in-embedded-innodb\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[75],"tags":[70,275,257,205],"class_list":["post-1874","post","type-post","status-publish","format-standard","hentry","category-drizzle-work-et-al","tag-drizzle","tag-embedded_innodb","tag-innodb","tag-table-message"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5a6n8-ue","jetpack-related-posts":[{"id":1903,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/04\/21\/storage-engine-api-write_row-create-select-and-ddl\/","url_meta":{"origin":1874,"position":0},"title":"Storage Engine API: write_row, CREATE SELECT and DDL","author":"Stewart Smith","date":"2010-04-21","format":false,"excerpt":"(this probably applies exactly the same for MySQL and Drizzle... but I'm just speaking about current Drizzle here) In my current merge request for the embedded-innodb\u00ef\u00bb\u00bf-create-select-transaction-arrgh branch (also see this specific revision), you'll notice an odd hoop that we have to jump through to make CREATE SELECT statements work with\u2026","rel":"","context":"In &quot;drizzle&quot;","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1849,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/03\/16\/embedded-innodb-querying-the-configuration\/","url_meta":{"origin":1874,"position":1},"title":"Embedded InnoDB: querying the configuration","author":"Stewart Smith","date":"2010-03-16","format":false,"excerpt":"I am rather excited about being able to do awesome things such as this to get the current configuration of your server: drizzle> SELECT NAME,VALUE -> FROM DATA_DICTIONARY.INNODB_CONFIGURATION -> WHERE NAME IN (\"data_file_path\", \"data_home_dir\"); +----------------+-------+ | NAME | VALUE | +----------------+-------+ | data_file_path | NULL | | data_home_dir | .\/\u2026","rel":"","context":"In &quot;drizzle&quot;","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1813,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/03\/12\/writing-a-storage-engine-for-drizzle-part-2-create-table\/","url_meta":{"origin":1874,"position":2},"title":"Writing A Storage Engine for Drizzle, Part 2: CREATE TABLE","author":"Stewart Smith","date":"2010-03-12","format":false,"excerpt":"The DDL code paths for Drizzle are increasingly different from MySQL. For example, the embedded_innodb StorageEngine CREATE TABLE code path is completely different than what it would have to be for MySQL. This is because of a number of reasons, the primary one being that Drizzle uses a protobuf message\u2026","rel":"","context":"In &quot;drizzle&quot;","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1888,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/04\/08\/embedded-innodb-is-in-the-tree\/","url_meta":{"origin":1874,"position":3},"title":"Embedded InnoDB is in the tree!","author":"Stewart Smith","date":"2010-04-08","format":false,"excerpt":"Well... the start of it :) I've taken the approach of taking tiny incremental steps (and getting review for each step) in implementing a Storage Engine based on the Embedded InnoDB library. What hit lp:drizzle (the trunk branch, for the 2010-04-07 milestone tarball) is only a handful of these small\u2026","rel":"","context":"In &quot;drizzle&quot;","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1677,"url":"https:\/\/www.flamingspork.com\/blog\/2009\/07\/29\/table-discovery-for-drizzle-take-2-now-merged\/","url_meta":{"origin":1874,"position":4},"title":"Table discovery for Drizzle (take 2, now merged!)","author":"Stewart Smith","date":"2009-07-29","format":false,"excerpt":"Table discovery looks a bit different from the previous time I blogged about it. Everything is now just hanging off the StorageEngine. If you want to not have dfe files on disk and just use your own data dictionary, you need to implement two things: A method to get table\u2026","rel":"","context":"In &quot;drizzle&quot;","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2155,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/10\/14\/second-drizzle-beta-and-innodb-update\/","url_meta":{"origin":1874,"position":5},"title":"Second Drizzle Beta (and InnoDB update)","author":"Stewart Smith","date":"2010-10-14","format":false,"excerpt":"We just released the latest Drizzle tarball (2010-10-11 milestone). There are a whole bunch of bug fixes, but there are two things that are interesting from a storage engine point of view: The Innobase plugin is now based on innodb_plugin 1.0.6 The embedded_innodb engine is now named HailDB and requires\u2026","rel":"","context":"In &quot;code&quot;","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/1874","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/comments?post=1874"}],"version-history":[{"count":2,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/1874\/revisions"}],"predecessor-version":[{"id":3836,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/1874\/revisions\/3836"}],"wp:attachment":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/media?parent=1874"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/categories?post=1874"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/tags?post=1874"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}