{"id":1811,"date":"2010-03-01T23:40:27","date_gmt":"2010-03-01T13:40:27","guid":{"rendered":"http:\/\/www.flamingspork.com\/blog\/?p=1811"},"modified":"2013-04-05T09:27:17","modified_gmt":"2013-04-04T23:27:17","slug":"writing-a-storage-engine-for-drizzle-part-1-plugin-basics","status":"publish","type":"post","link":"https:\/\/www.flamingspork.com\/blog\/2010\/03\/01\/writing-a-storage-engine-for-drizzle-part-1-plugin-basics\/","title":{"rendered":"Writing A Storage Engine for Drizzle, Part 1: Plugin basics"},"content":{"rendered":"<p>So, you&#8217;ve decided to write a Storage Engine for Drizzle. This is excellent news! The API is continually being improved and if you&#8217;ve worked on a Storage Engine for MySQL, you&#8217;ll notice quite a few differences in some areas.<\/p>\n<p>The first step is to create a skeleton StorageEngine plugin.<\/p>\n<p>You can see my skeleton embedded_innodb StorageEngine plugin in its merge request.<\/p>\n<p>The important steps are:<\/p>\n<h2>1. Create the plugin directory<\/h2>\n<p>e.g. <code>mkdir plugin\/embedded_innodb<\/code><\/p>\n<h2>2. Create the plugin.ini file describing the plugin<\/h2>\n<p>create the plugin.ini file in the plugin directory (so it&#8217;s <code>plugin\/plugin_name\/plugin.ini<\/code>)<br \/>\nAn example plugin.ini for embedded_innodb is.<\/p>\n<pre>[plugin]\r\ntitle=InnoDB Storage Engine using the Embedded InnoDB library\r\ndescription=Work in progress engine using libinnodb instead of including it in tree.\r\nsources=embedded_innodb_engine.cc\r\nheaders=embedded_innodb_engine.h<\/pre>\n<p>This gives us a title and description, along with telling the build system what sources to compile and what headers to make sure to include in any source distribution.<\/p>\n<h2>3. Add plugin dependencies<\/h2>\n<p>Your plugin may require extra libraries on the system. For example, the embedded_innodb plugin uses the Embedded InnoDB library (libinnodb).<\/p>\n<p>Other examples include the MD5 function requiring either openssl or gnutls, the gearman related plugins requiring gearman libraries, the <code>UUID()<\/code> function requiring libuuid and BlitzDB requiring Tokyo Cabinet libraries.<\/p>\n<p>For embedded_innodb, pandora-build has a macro for finding libinnodb on the system. We want to run this configure check, so we create a plugin.ac file in the plugin directory (i.e. <code>plugin\/plugin_name\/plugin.ac<\/code>) and add the check to it.<\/p>\n<p>For embedded_innodb, the plugin.ac file just contains this one line:<\/p>\n<pre>PANDORA_HAVE_LIBINNODB<\/pre>\n<p>We also want to add two things to plugin.ini; one to tell the build system only to build our plugin if libinnodb was found and the other to link our plugin with libinnodb. For embedded_innodb, it&#8217;s these two lines:<\/p>\n<pre>build_conditional=\"x${ac_cv_libinnodb}\" = \"xyes\"\r\nldflags=${LTLIBINNODB}<\/pre>\n<div>Not too hard at all! This should look relatively familiar for those who have seen autoconf and automake in the past.<\/div>\n<p>Some plugins (such as the md5 function) have a bit more custom auto-foo in plugin.ini and plugin.ac (as one of two libraries can be used). You can do pretty much anything with the plugin system, but you&#8217;re a lot more likely to keep it simple like we have here.<\/p>\n<h2>4. Add skeleton source code for your StorageEngine<\/h2>\n<p>While this will change a little bit over time (and is a little long to just paste into here), you can see what I did for embedded_innodb in the <a href=\"https:\/\/code.launchpad.net\/~stewart\/drizzle\/skeleton-embedded-innodb-engine\">skeleton-embedded-innodb-engine<\/a> tree.<\/p>\n<h2>5. Build!<\/h2>\n<p>You will need to re-run <code>.\/config\/autorun.sh<\/code> so the build system picks up your new plugin. When you run <code>.\/configure --help<\/code> afterwards, you should see options for building with\/without your new plugin.<\/p>\n<h2>6. Add a test<\/h2>\n<p>You will probably want to add a test to see that your plugin loads successfully. When your plugin is built, the test suite automatically picks up any tests you have in the <code>plugin\/plugin_name\/tests<\/code> directory. This is in the same format as general MySQL and Drizzle tests: tests go in a <code>t\/<\/code> directory, expected results in a <code>r\/<\/code> directory.<\/p>\n<p>Since we are loading a plugin, we will also need some server options to make sure that plugin is loaded. These are stored in the rather inappropriately named <code>test-master.opt<\/code> file (that&#8217;s the test name with &#8220;<code>-master.opt<\/code>&#8221; appended to the end instead of &#8220;<code>.test<\/code>&#8220;). For the embedded_innodb plugin_load test, we have a <code>plugin\/embedded_innodb\/tests\/t\/plugin_load-master.opt<\/code> file with the following content:<\/p>\n<pre>--plugin_add=embedded_innodb<\/pre>\n<p>You can have pretty much anything in the plugin_load.test file&#8230; if you&#8217;re fancy, you&#8217;ll have a <code>SELECT<\/code> query on <code>data_dictionary.plugins<\/code> to check that the plugin really is there. Be sure to also add a <code>r\/plugin_load.result<\/code> file (My preferred method is to just create an empty result file, run the test suite and examine the rejected output before renaming the <code>.reject<\/code> file to <code>.result<\/code>)<\/p>\n<p>Once you&#8217;ve added your test, you can run it either by just typing &#8220;<code>make test<\/code>&#8221; (which will run the whole test suite), or you can go into the main <code>tests\/<\/code> directory and run <code>.\/test-run.pl --suite=plugin_name<\/code> (which will just run the tests for your plugin).<\/p>\n<h2>7. Check the code in, feel good about self<\/h2>\n<p>and you&#8217;re done. Well&#8230; the start of a Storage Engine plugin is done :)<\/p>\n<p>This blog post (but not the whole blog) is published under the\u00c2\u00a0<a href=\"http:\/\/creativecommons.org\/licenses\/by-sa\/3.0\/\">Creative Commons Attribution-Share Alike License<\/a>. Attribution is by linking back to this post and mentioning my name (Stewart Smith).<\/p>\n","protected":false},"excerpt":{"rendered":"<p>So, you&#8217;ve decided to write a Storage Engine for Drizzle. This is excellent news! The API is continually being improved and if you&#8217;ve worked on a Storage Engine for MySQL, you&#8217;ll notice quite a few differences in some areas. The &hellip; <a href=\"https:\/\/www.flamingspork.com\/blog\/2010\/03\/01\/writing-a-storage-engine-for-drizzle-part-1-plugin-basics\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[1],"tags":[70,628,196,242],"class_list":["post-1811","post","type-post","status-publish","format-standard","hentry","category-general","tag-drizzle","tag-mysql","tag-plugin","tag-storageengine"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5a6n8-td","jetpack-related-posts":[{"id":2155,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/10\/14\/second-drizzle-beta-and-innodb-update\/","url_meta":{"origin":1811,"position":0},"title":"Second Drizzle Beta (and InnoDB update)","author":"Stewart Smith","date":"2010-10-14","format":false,"excerpt":"We just released the latest Drizzle tarball (2010-10-11 milestone). There are a whole bunch of bug fixes, but there are two things that are interesting from a storage engine point of view: The Innobase plugin is now based on innodb_plugin 1.0.6 The embedded_innodb engine is now named HailDB and requires\u2026","rel":"","context":"In &quot;code&quot;","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1849,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/03\/16\/embedded-innodb-querying-the-configuration\/","url_meta":{"origin":1811,"position":1},"title":"Embedded InnoDB: querying the configuration","author":"Stewart Smith","date":"2010-03-16","format":false,"excerpt":"I am rather excited about being able to do awesome things such as this to get the current configuration of your server: drizzle> SELECT NAME,VALUE -> FROM DATA_DICTIONARY.INNODB_CONFIGURATION -> WHERE NAME IN (\"data_file_path\", \"data_home_dir\"); +----------------+-------+ | NAME | VALUE | +----------------+-------+ | data_file_path | NULL | | data_home_dir | .\/\u2026","rel":"","context":"In &quot;drizzle&quot;","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2118,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/08\/18\/storage-engine-independent-test-suite\/","url_meta":{"origin":1811,"position":2},"title":"Storage Engine independent test suite","author":"Stewart Smith","date":"2010-08-18","format":false,"excerpt":"This is something that has never existed in the MySQL world. Nothing to help you start developing your engine. Sure, you could start running the whole test suite against your engine.... but where it wasn't specifically MyISAM, you'd certainly hit things that were.. well... as simple as having to change\u2026","rel":"","context":"In &quot;drizzle&quot;","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1813,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/03\/12\/writing-a-storage-engine-for-drizzle-part-2-create-table\/","url_meta":{"origin":1811,"position":3},"title":"Writing A Storage Engine for Drizzle, Part 2: CREATE TABLE","author":"Stewart Smith","date":"2010-03-12","format":false,"excerpt":"The DDL code paths for Drizzle are increasingly different from MySQL. For example, the embedded_innodb StorageEngine CREATE TABLE code path is completely different than what it would have to be for MySQL. This is because of a number of reasons, the primary one being that Drizzle uses a protobuf message\u2026","rel":"","context":"In &quot;drizzle&quot;","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3345,"url":"https:\/\/www.flamingspork.com\/blog\/2013\/05\/23\/mysql-vs-drizzle-plugin-api\/","url_meta":{"origin":1811,"position":4},"title":"MySQL vs Drizzle plugin APIs","author":"Stewart Smith","date":"2013-05-23","format":false,"excerpt":"There's a big difference in how plugins are treated in MySQL and how they are treated in Drizzle. The MySQL way has been to create a C API in front of the C++-like (I call it C- as it manages to take the worst of both worlds) internal \"API\". The\u2026","rel":"","context":"In &quot;code&quot;","block_context":{"text":"code","link":"https:\/\/www.flamingspork.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2041,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/06\/29\/enum-now-works-properly-in-drizzle\/","url_meta":{"origin":1811,"position":5},"title":"ENUM now works properly (in Drizzle)","author":"Stewart Smith","date":"2010-06-29","format":false,"excerpt":"Over at the Drizzle blog, the recent 2010-06-07 tarball was announced. This tarball release has my fixes for the ENUM type, so that it now works as it should. I was quite amazed that such a small block of code could have so many bugs! One of the most interesting\u2026","rel":"","context":"In &quot;drizzle&quot;","block_context":{"text":"drizzle","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/drizzle-work-et-al\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/1811","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/comments?post=1811"}],"version-history":[{"count":7,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/1811\/revisions"}],"predecessor-version":[{"id":3283,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/1811\/revisions\/3283"}],"wp:attachment":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/media?parent=1811"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/categories?post=1811"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/tags?post=1811"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}