{"id":511,"date":"2005-11-23T09:40:01","date_gmt":"2005-11-22T23:40:01","guid":{"rendered":"http:\/\/www.flamingspork.com\/blog\/?p=511"},"modified":"2012-05-30T10:15:39","modified_gmt":"2012-05-30T00:15:39","slug":"disk-space-allocation-part-1-seeing-whats-happenned","status":"publish","type":"post","link":"https:\/\/www.flamingspork.com\/blog\/2005\/11\/23\/disk-space-allocation-part-1-seeing-whats-happenned\/","title":{"rendered":"disk space allocation (part 1: seeing what’s happenned)"},"content":{"rendered":"

(a little while ago I was writing a really long entry on everything possible. I realised that this would be a long read for people and that less people would look at it, so I’ve split it up).<\/p>\n

This sprung out of doing work on the NDB<\/a> disk data tree. Anything where efficient use of the filesystem is concerned tickles my fancy, so I went to have a look at what was going on.<\/p>\n

Filesystems store what part of the disk belongs to what file in one of two ways. The first is to keep a list of every disk block (typically 4kb) that’s being used by the file. A 400kb file will have 100 block numbers. The second way is to store a range (extent). That is, a 400kb file could use 100 blocks starting at disk block number 1000.<\/p>\n

XFS<\/a> has a tool called xfs_bmap. It gives you a list of the extents allocated to a file.<\/p>\n

So, let’s have a look at what it tells us about some recordings on my MythTV<\/a> box.<\/p>\n

\r\nmyth@orpheus:~$ ls -lah myth-recordings\/10_20050912183000_20050912190000.nuv\r\n -rw-r--r--  1 myth myth 452M 2005-09-12 19:00 myth-recordings\/10_20050912183000_20050912190000.nuv\r\nmyth@orpheus:~$ xfs_bmap -v myth-recordings\/10_20050912183000_20050912190000.nuv\r\nmyth-recordings\/10_20050912183000_20050912190000.nuv:\r\n EXT: FILE-OFFSET       BLOCK-RANGE          AG AG-OFFSET             TOTAL\r\n   0: [0..639]:         228712176..228712815  7 (21106232..21106871)    640\r\n   1: [640..1663]:      83674040..83675063    2 (24358056..24359079)   1024\r\n   2: [1664..923519]:   83675368..84597223    2 (24359384..25281239) 921856\r\n   3: [923520..924031]: 84631272..84631783    2 (25315288..25315799)    512\r\n<\/pre>\n
Just to make things fun, this is all in 512byte blocks. But anyway, the real interesting thing is the number of extents. Ideally, every file would have one extent as this means that we avoid disk seeks – *the* most expensive disk operation.<\/p>\n
XFS also provides the xfs_fsr tool (File System Repacker) that can defragment files (even on a mounted file system). On IRIX this used to run out of cron – fun when a bunch of machines hit a CXFS volume all at the same time.<\/p>\n","protected":false},"excerpt":{"rendered":"
(a little while ago I was writing a really long entry on everything possible. I realised that this would be a long read for people and that less people would look at it, so I’ve split it up). This sprung … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[8,14],"tags":[],"class_list":["post-511","post","type-post","status-publish","format-standard","hentry","category-linux-kernel","category-mysql"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5a6n8-8f","jetpack-related-posts":[{"id":514,"url":"https:\/\/www.flamingspork.com\/blog\/2005\/11\/29\/disk-space-allocation-part-3-storing-extents-on-disk\/","url_meta":{"origin":511,"position":0},"title":"disk space allocation (part 3: storing extents on disk)","author":"Stewart Smith","date":"2005-11-29","format":false,"excerpt":"Here I'm going to talk about how file systems store what part of the disk a part of the file occupies. If your database files are very fragmented, performance will suffer. How much depends on a number of things however. XFS can store some extents directly in the inode (see\u2026","rel":"","context":"In "linux-kernel"","block_context":{"text":"linux-kernel","link":"https:\/\/www.flamingspork.com\/blog\/category\/linux-kernel\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":512,"url":"https:\/\/www.flamingspork.com\/blog\/2005\/11\/23\/disk-space-allocation-part-2-examining-your-database-files\/","url_meta":{"origin":511,"position":1},"title":"disk space allocation (part 2: examining your database files)","author":"Stewart Smith","date":"2005-11-23","format":false,"excerpt":"memberdb\/log.MYD: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL 0: [0..943]: 5898248..5899191 3 (36536..37479) 944 1: [944..1023]: 6071640..6071719 3 (209928..210007) 80 2: [1024..1127]: 6093664..6093767 3 (231952..232055) 104 3: [1128..1279]: 6074800..6074951 3 (213088..213239) 152 4: [1280..1407]: 6074672..6074799 3 (212960..213087) 128 5: [1408..1423]: 6074264..6074279 3 (212552..212567) 16 memberdb\/log.MYI: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL\u2026","rel":"","context":"In "linux-kernel"","block_context":{"text":"linux-kernel","link":"https:\/\/www.flamingspork.com\/blog\/category\/linux-kernel\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":759,"url":"https:\/\/www.flamingspork.com\/blog\/2006\/11\/13\/disk-allocation-xfs-ndb-disk-data-and-more\/","url_meta":{"origin":511,"position":2},"title":"Disk allocation, XFS, NDB Disk Data and more…","author":"Stewart Smith","date":"2006-11-13","format":false,"excerpt":"I've talked about disk space allocation previously, mainly revolving around XFS (namely because it's what I use, a sensible choice for large file systems and large files and has a nice suite of tools for digging into what's going on).Most people write software that just calls write(2) (or libc things\u2026","rel":"","context":"In "linux-kernel"","block_context":{"text":"linux-kernel","link":"https:\/\/www.flamingspork.com\/blog\/category\/linux-kernel\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":515,"url":"https:\/\/www.flamingspork.com\/blog\/2005\/11\/29\/disk-space-allocation-part-4-allocating-an-extent\/","url_meta":{"origin":511,"position":3},"title":"disk space allocation (part 4: allocating an extent)","author":"Stewart Smith","date":"2005-11-29","format":false,"excerpt":"For XFS, in normal operation, an extent is only allocated when data has to be written to disk. This is called delayed allocation. If we are extending a file by 50MB - that space is deducted from the total free space on the filesystem, but no decision on where to\u2026","rel":"","context":"In "linux-kernel"","block_context":{"text":"linux-kernel","link":"https:\/\/www.flamingspork.com\/blog\/category\/linux-kernel\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1655,"url":"https:\/\/www.flamingspork.com\/blog\/2009\/06\/09\/drizzle-pluggable-metadatastore-or-no-table-definition-file-on-disk\/","url_meta":{"origin":511,"position":4},"title":"Drizzle pluggable MetadataStore (or: no table definition file on disk)","author":"Stewart Smith","date":"2009-06-09","format":false,"excerpt":"My code is shaping up rather nicely (see https:\/\/code.launchpad.net\/~stewart\/drizzle\/discovery) and I'm planning to submit a merge-request for it later today. I'm about to commit code that implements a MetadataStore for the ARCHIVE engine. This means that for ARCHIVE tables, you only have the .ARZ file on disk. The table definition\u2026","rel":"","context":"In "General"","block_context":{"text":"General","link":"https:\/\/www.flamingspork.com\/blog\/category\/general\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":228,"url":"https:\/\/www.flamingspork.com\/blog\/2004\/04\/10\/tightly-packing-onodes\/","url_meta":{"origin":511,"position":5},"title":"Tightly packing onodes","author":"Stewart Smith","date":"2004-04-10","format":false,"excerpt":"The current problem is that an onode, however much we can pack forks into a block, still takes up a minimum of one disk block. A disk block typically being 4kb, and a tendancy to want to be bigger (think large media files), and also being the unit of atomicity\u2026","rel":"","context":"In "fcfs"","block_context":{"text":"fcfs","link":"https:\/\/www.flamingspork.com\/blog\/category\/fcfs\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/511","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/comments?post=511"}],"version-history":[{"count":1,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/511\/revisions"}],"predecessor-version":[{"id":2962,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/posts\/511\/revisions\/2962"}],"wp:attachment":[{"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/media?parent=511"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/categories?post=511"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.flamingspork.com\/blog\/wp-json\/wp\/v2\/tags?post=511"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}