Better disk allocation with MythTV and XFS

Posted on by
Reply

Running MythTV on XFS? Noticed that all your recordings end up rather fragmented? (use xfs_bmap to find out) Well, the culprit is MythTV not being too nice to the file system. Good news is, it’s rather fixable.

From the MythTV source code, edit libs/libmythtv/ThreadedFileWrite.cpp and look for the following:

void ThreadedFileWriter::Sync(void)
{
if (fd >= 0)
{
#ifdef HAVE_FDATASYNC
fdatasync(fd);
#else
fsync(fd);
#endif
}
}
You then want to, after the first squiggly bracket (a { ) put in a “return;” so that it looks like this:

void ThreadedFileWriter::Sync(void)
{
return;
if (fd >= 0)
{
#ifdef HAVE_FDATASYNC
fdatasync(fd);
#else
fsync(fd);
#endif
}
}

Recompile MythTV and go!

(Before anybody says, yes – this patch could be neater and all that… I just haven’t had time yet).

mysql-5.1.22-stew2

Posted on by
1

New:

  • Updated NDB Compressed LCP and BACKUP patches (now with O_DIRECT support)
  • InnoDB patch for Windows that should give ~5x improvement on commits/sec (Bug31876)
  • Everything in current telco-6.3 tree (ndb ~6.3.5)
    • Lots of NDB improvements and new features over regular 5.1.
        • WL3686 Remove read before update
        • WL2680 NDB Batched Update
        • WL2679 NDB Batched Delete
        • WL4108 NDB Handler statistics
        • WL4096 NDB Realtime performance and settings
        • WL3126 and WL3127 Client and Replication bind address
        • NDB Online ALTER TABLE ADD COLUMN
        • NDB Multi-Master replication conflict resolution (limitations apply :)
        • NDB prepare for endian independence
        • NDB micro-gcp (reduces replication lag)
        • NDB SendBuffer throttling
        • NDB MySQL Server TC selection (improve performance)

    Old (In previous patchset too):

    • Remove ndb_use_exact_count giving up to 300% performance improvements on Joins in NDB
    • INFORMATION_SCHEMA table for NDB node status
    • NDB Cluster Log as CSV file (suitable for ENGINE=CSV)
    • Skeleton Engine (build from storage/skeleton)
    • MyHTTP Engine (build from storage/myhttp)
    • PBXT Engine (build from storage/pbxt)
    • Make ARCHIVE faster at compressing (at slight expense of space usage)

    Availability:

    • Patch (apply with -p1 to mysql 5.1.22) 4.0 MB
      • Applies cleanly on a BK source tree… a few files don’t exist in the tarball on dev.mysql.com (due to the way it’s built)… so when asked for “file to patch” just hit enter and then choose y to skip that patch.
    • README (list of patches, descriptions) 13kb
    • quilt patch series tarball (individual patches) 4.1MB
    • diffstat 228k

    Feedback much appreciated.

    Speaking at VITTA (Victorian IT Teachers Association Inc) Conference

    Posted on by
    1

    I’m speaking at the upcoming VITTA conference.

    Title:MySQL database administration for non DBAs

    Abstract: MySQL is incredibly ubiquitous. MySQL database administrators are not everywhere; MySQL is. Often MySQL is run to power a small web site or two, an application or two, or run on a machine purely for someone else’s use (and the install made MySQL just work so you don’t have to care). This session goes over the things you need to know about your MySQL installations to keep them healthy without burdening you with work, including MySQL Basics, installation, security, backup, restore, performance and upgrades.

    When: 12:15 PM, Wednesday 21 November 2007

    Should be fun!

    MySQL User Conference 2008 – MySQL Cluster Sessions

    Posted on by
    1

    So… stealing the idea from Peter, does anybody have any suggestions for MySQL Cluster related things to have at the UC next year (April)? Either leave a comment or email me (first name at mysql dot com).

    Practical MythTV for $14.95US

    Posted on by
    1

    Julie at Apress let me know that over at Bookpool: Practical MythTV: Building a PVR and Media Center PC they’re having a big sale of Apress titles, including Practical MythTV.

    So over there now for (a bit) less than $15US you can get Practical MythTV. Pretty neat. So everybody should go buy two copies (buy one get one free… right? :)

    At some point soon, I’ll even post how I go with upgrading my MythTV box to Ubuntu 7.10 (Gutsy)… once I gather up the courage to do so :)

    Problems with Tracker (and why I’m back to Beagle)

    Posted on by
    5
    1. If you have more than 8192 directories, it can’t monitor them (max inotify limit)
    2. No search results
    3. access(), stat() and then lstat() called on *every* file that it’s going to index… this takes a long time.
    4. Did I mention that search doesn’t actually give you any results?
    5. Even when you run strings on the database and then search for something there, you don’t get any results
    6. Even when you search for the name of an application, you get no results.

    Everything that’s wrong (and right) with Ubuntu Gutsy

    Posted on by
    5

    So, I’ve upgraded two boxes: my laptop and my mail server.

    Wrong:

    • courier-imap-ssl broke. My cell phone could no longer pull mail.
      I got something like in the log:
      imapd-ssl: couriertls: connect: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
      The fix for this is to change the TLS_PROTOCOL option in /etc/courier/imapd-ssl to SSL23 (an option not listed in the “possible verions” list in the config file). So mail now works.
    • Evolution now seems to put a blank line at the top of emails when replying (before quoting the message). GRRRRRRR…
    • The Window Manager (metacity) has changed how it arranges new windows. Instead of cascading them down, so if you’re opening a bunch you can easily switch between them, it now puts them right on top of each other GGRRRRRR.
    • It set up my Xorg to *not* to X monitor hotpluggy goodness. I’ve managed
    • Login seems to take a fair bit longer than before
    • Tracker
      • trackerd will index while on battery power… GRRR
      • trackerd will hog disk and CPU
      • I haven’t been able to get any results out of tracker.
      • There is no tracker-status or something to check that it hasn’t just stopped working
      • Tracker will use up all your remaining disk space… (I initially only had a few GB free.. and trackerd filled it up)
      • I haven’t actually had a query work at all. i.e. it seems to be useless. I just get a “trackerd exited with status 0”.
    • There is no “remember passphrase” button in Evolution for GPG signing anymore. I think this is now set in the global preferences… but the behaviour change is annoying.
    • The pidgin icon is different than the gaim icon in the Notification Area… this is annoying as I can never find it anymore
    • The Window List applet changed behaviour to now *not* group windows together at all… making having lots of windows open (as I often do) completely unmanagable as opposed to (previously) just annoying.

    The Awesome:

    • X monitor hotpluggy goodness (known to others as xrandr 1.2). VGA out works with the free radeon driver now! *YAY*. (I can rearrange screens using xrandr… awesome)
    • Pidgin is quite nice… the likelyhood of reply plugin is neat. More compact IM windows (also cool).
    • Syncing between Pidgin and Evolution with contacts seems to work okay now.
    • Suspend and resume still works
    • New version of f-spot – goodness!
    • New OpenOffice.org (2.3)  sucks less.
    • emacs22 (shiny)
    • New WINE does real neat stuff (such as have menu items for browse C drive, remove software)
    • Evolution doesn’t seem to leak memory as much anymore. Only 316MB RSS at the moment!
    • Lots of neat Liferea bug fixes – much nicer! (read items in the planet.linux.org.au feed stay marked read now!)

    MySQL 5.1.22(ish)-stew1

    Posted on by
    Reply

    I’ve decided to publish my patch series. The goal of the -stew patches is to collect things I find interesting and that at some point could (should) make it into the main MySQL tree (even if others don’t think so).

    It’s not designed for use in production.. I don’t really care if there’s failing test cases…. if it builds it’s perfect.

    It includes the following which could be interesting:

    • Removal of ndb_use_exact_count (performance for NDB)
    • NDB node status in an INFORMATION_SCHEMA table
    • Compressed Backup and LCP for NDB
    • Cluster log as CSV
    • Skeleton Engine
    • MyHTTP Engine
    • PBXT Engine
    • Skeleton of MyBS support for NDB
      • (in the hope that somebody finishes it)

    Currently the additional engines have to be built separately in their storage/ENGINE directories. I have some preliminary patches to get them to build in-tree via the plug.in file, but it’s not finished (patches welcome).

    This is all currently based off the 5.1-ndb tree. In future, it will likely be based off 5.1-telco.

     http://www.flamingspork.com/mysql/patch-5.1-ndb-stew1-20071016.patch.gz

    and broken out in:
    http://www.flamingspork.com/mysql/patch-5.1-ndb-stew1-20071016/
    (including the not-quite-working ENGINE_in_tree_build patches)

    Known to apply against this tree:

    http://www.flamingspork.com/mysql/mysql-5.1-ndb-20071016.tar.bz2

    Comments, thoughts, patches to include, all welcome!