{"id":1201,"date":"2008-09-08T22:13:41","date_gmt":"2008-09-08T12:13:41","guid":{"rendered":"http:\/\/www.flamingspork.com\/blog\/?p=1201"},"modified":"2008-09-08T22:13:41","modified_gmt":"2008-09-08T12:13:41","slug":"setfilevaliddata-function-windows-now-with-added-fail","status":"publish","type":"post","link":"https:\/\/www.flamingspork.com\/blog\/2008\/09\/08\/setfilevaliddata-function-windows-now-with-added-fail\/","title":{"rendered":"SetFileValidData Function (Windows) – Now with added FAIL"},"content":{"rendered":"

SetFileValidData Function (Windows)<\/a><\/p>\n

There seems to be two options on Win32 for preallocating disk space to files.<\/p>\n

Basically, I want a equivilent to posix_fallocate or the ever wonderful xfsctl XFS_IOC_RESVSP64 call.<\/p>\n

The idea being to (quickly) create a large file on disk that is stored efficiently (i.e. isn’t fragmented).<\/p>\n

From SQL, you’d do something like “CREATE LOGFILE GROUP lg1 ADD UNDOFILE ‘uf1’ INITIAL_SIZE 1G;” and expect a 1GB file on disk. One way of getting this is calling write() (or WriteFile() on Win32) repeatedly until you’ve written a 1GB file full of zeros. This means you’re generating approximately 1GB of IO.<\/p>\n

Except it’s worse than that: every time you extend the file, you’re going to be changing the metadata (file and free space information). If you’re lucky, you won’t be using a file system that writes a new transaction to the journal for each time you do this.<\/p>\n

If your file system allocator doesn’t like you today (even more likely when you’ve got more than one process doing IO), you may end up with rather fragmented files as well – especially if you’re doing synchronous IO. So you want some method of saying “this file will be size X, please allocate disk space to it in the most efficient way for a file size of X” as it’s not possible to infer this from everyday IO calls (I guess the Win32 CopyFile and CopyFileEx calls could though).<\/p>\n

It probably doesn’t do it, but having a CopyFile call would be neat for copy on write file systems and saving space… although I wonder how many Win32 apps would cope with ENOSPC on a write to an existing part of a file.<\/p>\n

On IRIX we used the magic xfsctl() with the XFS_IOC_RESVSP64 argument. On Linux (with XFS), we use the same. On ext2\/ext3 the only way to get the same has been to (with the file system unmounted), parse the file system and implement it yourself. Although (and this just in) the brand new fallocate() call should help with this. The posix_fallocate() call in GNU libc has just been a wrapper around the simple method of writing 0 to a file from start to end (albeit rather efficiently).<\/p>\n

XFS implements something called “unwritten extents”. An unwritten extent says “this range of blocks is allocated to this file. If reading from this range, return a zero page. If writing, split the unwritten extent into 3 parts: before, the newly written extent (which isn’t unwritten: i.e. now valid data), and the after extent.” Simple, rather efficient and gets really good allocation as XFS gets to search the free space btrees based on size.<\/p>\n

So what to do on Win32 (apart from drink heavily to try and make it all go away)?<\/p>\n

There’s SetFileValidData, but that needs special permissions and may expose previously deleted data from other users. i.e. massive security hole. FAIL<\/p>\n

There’s SetEndOfFile which, quoting the MS docs: “If the file is extended, the contents of the file between the old end of the file and the new end of the file are not defined.” Not exactly reassuring… but introduced in W2k, so rather safe to use today. Doesn’t save you from having to fill the file with zeros as part of initialisation though.<\/p>\n

There’s SetFileInformationByHandle, which looks like it may do exactly what I want… if you read between the lines of the documentation. But it’s only supported starting with Vista. Which you all use of course, so that’s not a problem.<\/p>\n","protected":false},"excerpt":{"rendered":"

SetFileValidData Function (Windows) There seems to be two options on Win32 for preallocating disk space to files. Basically, I want a equivilent to posix_fallocate or the ever wonderful xfsctl XFS_IOC_RESVSP64 call. The idea being to (quickly) create a large file … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[14],"tags":[51,50,36,52],"class_list":["post-1201","post","type-post","status-publish","format-standard","hentry","category-mysql","tag-fallocate","tag-file-allocation","tag-win32","tag-xfs"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5a6n8-jn","jetpack-related-posts":[{"id":1229,"url":"https:\/\/www.flamingspork.com\/blog\/2008\/10\/14\/mysql-cluster-ndb-on-win32-progress\/","url_meta":{"origin":1201,"position":0},"title":"MySQL Cluster (NDB) on Win32 progress","author":"Stewart Smith","date":"2008-10-14","format":false,"excerpt":"Many things have been happenning in the land of NDB on Win32 as of late. I've fixed about 700 compiler warnings (some of which were real bugs) leaving about 161 to go on Win32 (VS2003). We're getting a few more warnings on Win64 (some of which look merely semantic, while\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/www.flamingspork.com\/blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1935,"url":"https:\/\/www.flamingspork.com\/blog\/2010\/05\/27\/a-warning-to-solaris-users-fsync-possibly-doesnt\/","url_meta":{"origin":1201,"position":1},"title":"A warning to Solaris users…. (fsync possibly doesn’t)","author":"Stewart Smith","date":"2010-05-27","format":false,"excerpt":"Read the following: Oracle\/Sun ZFS Data Loss - Still Vulnerable OpenSolaris Bug 6880764 Data loss running Oracle on ZFS on Solaris 10, pre 142900-09 Linux has its fair share of dumb things with data too (ext3 not defaulting to using write barriers is a good one). This is however particularly\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/www.flamingspork.com\/blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":894,"url":"https:\/\/www.flamingspork.com\/blog\/2007\/09\/20\/ndb_mgmd-on-win32-an-alpha\/","url_meta":{"origin":1201,"position":2},"title":"ndb_mgmd on Win32 (an Alpha)","author":"Stewart Smith","date":"2007-09-20","format":false,"excerpt":"So, here is an Alpha quality port of the MySQL Cluster management server to Win32 based on the current MySQL 5.0 tree. This isn't going into 5.0, so don't expect to ever have that. This isn't going into 5.1 either, so don't expect it there. It'll go into some future\u2026","rel":"","context":"In "mysql"","block_context":{"text":"mysql","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/mysql\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1215,"url":"https:\/\/www.flamingspork.com\/blog\/2008\/09\/18\/ndb-windows-port-shaping-up\/","url_meta":{"origin":1201,"position":3},"title":"NDB Windows port shaping up…","author":"Stewart Smith","date":"2008-09-18","format":false,"excerpt":"It's getting there. The tree should now pretty much always compile, and (at least mostly) doesn't break anything on other platforms. It even works on win32... at least basic functionality. There will be monsters (bugs.. but scarier, becuase it's win32).","rel":"","context":"In "mysql"","block_context":{"text":"mysql","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/mysql\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1008,"url":"https:\/\/www.flamingspork.com\/blog\/2008\/02\/11\/getting-a-file-size-on-windows\/","url_meta":{"origin":1201,"position":4},"title":"Getting a file size (on Windows)","author":"Stewart Smith","date":"2008-02-11","format":false,"excerpt":"The first point I'd like to make is that you're using a Microsoft Windows API, so you have already lost. You are just not quite aware of how much you have lost. A quick look around and you say \"Ahh... GetFileSize, that's what I want to do!\" Except, of course,\u2026","rel":"","context":"In "grumble"","block_context":{"text":"grumble","link":"https:\/\/www.flamingspork.com\/blog\/category\/grumble\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1217,"url":"https:\/\/www.flamingspork.com\/blog\/2008\/09\/19\/visual-studio-2008-unreferenced-local-variable-bug\/","url_meta":{"origin":1201,"position":5},"title":"Visual Studio 2008 unreferenced local variable bug","author":"Stewart Smith","date":"2008-09-19","format":false,"excerpt":"screenshot 'cause typing is for wusses UPDATE: not actually VS bug. Nasty macro defining strtok_r to strtok on Win32. ouch.","rel":"","context":"In "mysql"","block_context":{"text":"mysql","link":"https:\/\/www.flamingspork.com\/blog\/category\/work-et-al\/mysql\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.flamingspork.com\/blog\/wp-content\/uploads\/2008\/09\/vs_bug.png?resize=350%2C200","width":350,"height":200},"classes":[]}],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/1201","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/comments?post=1201"}],"version-history":[{"count":1,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/1201\/revisions"}],"predecessor-version":[{"id":1202,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/1201\/revisions\/1202"}],"wp:attachment":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/media?parent=1201"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/categories?post=1201"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/tags?post=1201"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}