Pre-allocating disk space? (1 Viewer)

[Nippywiffle]

Hi,
I've noticed that the main drive I use for recording get's totally fragmented, very quickly. It gets to a point where windows defrag is unable to even do anything. Multiple video files have 1000's of fragmentations, and defrag just says "windows was unable to defrag some of your files".

For now, I've started moving them to another disk, so I can maybe defrag, then copy them back one at a time, to see if it copies them into contiguous disk space.
Then I plan to set a defrag command to run, whenever powerscheduler is going to hibernate the PC...

I was wondering if there is any way to get tvserver to pre-allocate chunks of disk space for recordings, like, say maybe 500mb at a time, so that the file gets written into that chunk, unstead of fragmented all over the place... and at worst might have 3 or 4 fragments, instead of 1000's.

Anyone have any idea if thats possible, and how it could be done?

Thanks,
Dan

 

[gxtracker]

Thats really not the best solution to the problem you're having.

I dont know if you have done this already or not, but I've broken up my drive into 2 volumes - one for the OS and programs, and the other media storage. if you do this, fragmentation can be reduced since you're not mixing tiny windows files with larger media files on the same drive.

Plus, you dont have to worry about the hibernate cache or virtual memory cache from ruining your contiguous storage blocks on the media volume.

I've been using this system for about 3 years now, and having disk defrag run once a week really keeps the drives in order - considering the HTPC is recording about 4 hours worth of shows every night.

 

[Nippywiffle]

Thanks for the response.

I actually already have that setup... I have about 15gb partitioned as c:. and the other 475gb for video. I also have 2 other drives for music, photos and dvd images.

maybe once I get it defragged once, I'll just run the defrag once a week like you said, and hopefully it'll work.

 

[and-81]

Thats really not the best solution to the problem you're having.

I actually like the idea of pre-allocating the space... Out of interest what makes you say that it's not the best solution?

I think it would cut down on fragmentation and might make the recording process a bit smoother.

I already have a two driver setup and I still get a lot of fragmentation on the recording drive.

Cheers,

 

[gxtracker]

Out of interest what makes you say that it's not the best solution?

I dont feel its the best solution because you're using an application (MP) to solve a problem that exists at the OS layer.

what if your disk is so fragmented that you dont have a large enough contiguous block of space to allocate? then you cant save anything until you defragment the volume anyways.

one solution could be to take your media partition and increase the cluster size, since the majority of your media will be large video files. that should help assist with the fragmentation issue.

Im not really sure why you guys are having fragmentation issues and im not. I run a command line defrag on each volume once a week as an automated task. The computer wakes up early in the morning, and defrags all volumes, then powerscheduler shuts it down in 5 minutes. when I check the completed task log, the defrag only takes several minutes in most cases on a 400GB volume.

 

[and-81]

I guess the fragmentation could be because I have 2 tv cards and often record two shows at the same time.

But yes, you're right, MediaPortal shouldn't really be trying to solve what are essentially OS issues. I'll double check my cluster size, I think I've maxed it out, but I might not have.

I'll look into it a bit further and see if I can find a simple solution ... google time now...

Cheers,

 

[grubi]

I guess the fragmentation could be because I have 2 tv cards and often record two shows at the same time.

But yes, you're right, MediaPortal shouldn't really be trying to solve what are essentially OS issues. I'll double check my cluster size, I think I've maxed it out, but I might not have.

I'll look into it a bit further and see if I can find a simple solution ... google time now...

Cheers,

Preallocation is a really good idea and many other programs creating large files follow exactly this way and this is by far the only reliable solution to the problem (popular examples are DBM systems, BitTorrent and even Windows file copy).

@gxtracker - i think you are confusing two things. It has nothing to do with fixing problems in an application which are OS layer problems. The point is to give the OS more information about what you intend to do so that it can act more intelligent.

IMHO what Nippywiffle suggested is worth thinking about implementing it into MP cause it reduces fgramentation esp. if you do recording while watching live tv. Also the fact that timeshift buffers are not reused but deleted and recreated very often could lead to free space fragmentation.

AFAIK one solution would be the combination of SetFileValidData() and SetEndOfFile().
The downside is that IIRC SetFileValidData() is only available on OS >= XP and needs SE_MANAGE_VOLUME_NAME priviledges to run.


Regards,
grubi.

 

[gxtracker]

It has nothing to do with fixing problems in an application which are OS layer problems.

But it does. Thats exactly what pre-allocating space in an application is doing. Im all about less is more, and having MP do less of the things it doesnt need to do, means that in reality it does more of the things it's supposed to do. Managing physical storage is something the MP suite doesnt need to do - that's why there is un underlying operating system.

Now, if you were using volume information to make an intelligent defragmentation schedule that would be run at the OS layer, then I would agree that its purpose is to give the OS more information about what you intend to do with your storage - but having MP pre-allocate storage space just trades fixing one problem with creating several others.

Even if you perform an initial full defrag, then defrag on a daily basis (have your HTPC wake up early in the morning to not disrupt its usage), You shouldnt have any problem with fragmentation on the drive unless you're close to filling up the volume.

I guess the fragmentation could be because I have 2 tv cards and often record two shows at the same time.

Thats very true, and I can see fragmenation becoming an issue for users of multiple TV cards recording at the same time, but frequent disk maintence should help alleviate the issue. I dont know how often you gents are defraging your systems, but maybe try a daily setup? It's worked quite well in my testing.

Below is a screenshot of the fragmented drive of my HTPC in the bedroom. we're looking at close to 30% fragmentation on the drive, and it took just over 20 minutes to defrag the volume. The HTPC in the bedroom isnt used as much as the one in the living room, but it still does quite a bit of work - records anywhere from 2 - 3 hour long television shows a night, is on daily, and recieves regular file activity with shows being deleted when watched.

I havent defraged this volume since MP 0.2.2.0 - If I set it to defrag daily, that time could be reduced from 20 minutes to only 2 or 3 minutes.

 

[uggabugga]

Not to change subject, but I run this defrag tool JkDefrag once a week in a schedule, same as gxtracker does.
Works quit well I must say.
Just my 2 cents

 

[and-81]

I understand what you guys are saying about defragmenting the hard disk. I used JkDefrag before I reinstalled Windows, I haven't bothered to put the scheduled task back in place yet because I wanted to see how bad fragmentation could be.

I guess I'm coming at this from the opposite direction to you. Instead of removing the fragmentation after it's happened I'd like to stop it from happening...

There should be significantly less hard disk activity if we preallocate recordings, for the following reasons:

1. Less hard disk seeking when recording.
2. Less hard disk seeking when playing back.
3. Little or no defragmenting required.

This should have the following benefits:

1. Less wear and tear on the hard disk (This is a big deal for me because I do a lot of recording).
2. Less hard disk noise (doesn't bother me, but I know it effects some).

Now let me explain the way I use my HTPC so you can understand why a daily defrag doesn't really solve it for me.

To watch all the shows we like I have the HTPC record them all so we can watch them when we get home.

My family and I are rarely home in the early evening, we usually watch a few hours of TV between ~8pm and midnight.

By the time we sit down to watch TV there is usually around 2 to 4 shows recorded from the afternoon/evening. We will start watching one show while another one or two are recording.

By the end of the night we might have watched somewhere between half of the days shows to perhaps all of the days shows. And will either delete some or all of what we've watched depending on who was there (eg. My brother might not be home yet and will want to watch something the next day).

I didn't put the defragment scheduled task back in when I re-installed windows last month, take a look at the screenshot attached to see how badly fragmented some of my recordings are...

You can see that the top 2 fragmented files are over 2000 fragments and the next 5 are around 1500 fragments. That's just plain crazy!

Now even if I was defragmenting every morning I would still end up with significant fragmentation by the end of that night because of the recordings that occur. I get a few glitches in my recordings from time to time and I'm starting to wonder if it's because the hard disk can't keep up when it's recording and playing back such heavily fragmented files.

The answer is not magical or dangerous, it's pretty simple. Just preallocate large chunks when recording. In C# it's a one liner:

FileStream.SetLength(size);

In C++ (to put it in the TS File Writer filter) it wouldn't be much harder, maybe 1 or 2 lines.

As has already been said, download managers, bittorrent clients, DBMS' all do it for a reason.

All we'd be doing with preallocation is telling the OS the expected size of the file ahead of time ... the OS still does the file system voodoo, we just give it a heads-up on what to expect.

We could do it a few ways:

A) Preallocate large chunks, say 5gb each. Nearly all of my recordings will fit under that size.
B) Preallocate smaller chunks as we go, say 512mb each.

If we increased recording sizes by 512mb at a time as we went then my average recording could be expected to have between 1 and maybe 8 fragments ... that's a lot better than 2466 fragments

Of course if the hard disk is approaching full up the OS will still fragment a preallocated chunk, but by keeping a bit of free space available at all times we should be able to cut down on fragmentation (thereby reducing hard disk wear and noise) significantly.

Also, keep in mind that the average user will not have a scheduled defrag task and will likely suffer from heavy fragmentation of the hard disk. With all that in mind, why wouldn't we preallocate recordings?

I'll pitch this to the guys that handle the filters, I know they will not put it in before MP 1.0 is released but I might see if I can be a tester for this change between now and then.

Cheers,