Pausing TV corrupted recording (1 Viewer)

[doveman]

MediaPortal Version: v1.2.1
MediaPortal Skin: StreamedMP
Windows Version: Win7 Ult 64
CPU Type: Phenom II X4 955
HDD: 2TB Samsung F4
Memory: 8GB DDR3 1600Mhz
Motherboard: MSI 990FXA-GD80
Video Card: HD6950 2GB
Video Card Driver: 11.10
Sound Card: onboard Realtek AC97 (ALC892)
Sound Card AC3: no AC3
Sound Card Driver: 6.01.6482
1. TV Card: Hauppauge Nova-T 500
1. TV Card Type: DVB-T
1. TV Card Driver: 4.3.27240
MPEG2 Video Codec: Cyberlink PDVD11
MPEG2 Audio Codec: ffdshow
h.264 Video Codec: ffdshow
Satelite/CableTV Provider:
HTPC Case: Custom
Cooling: Scythe 120mm Bottom Intake Fan, TRUE Rev. c CPU HSF
Power Supply: Antec CP-850
Remote: Nova-T 500
TV: Sony XBR800 36"
TV - HTPC Connection: DVI


I paused the programme (Black Mirror) I was watching (and recording, but I was watching from the Live point) for a few minutes whilst I got my dinner. When I unpaused it, it played for maybe 30-60secs, before the sound and picture completely disintegrated (i.e. it became blocky, only updated about once every 1-2 seconds, completely unwatchable in other words) before jumping to the current live point, which was the next programme.

OK I thought, I guess the buffer must have run out, I'll watch the end from the recording. However, this does the same thing (completely reproducible), so I can't watch the end of the programme, which is rather annoying. I can't understand why pausing, even if the buffer did run out, should damage the recording but it has.

 

[mm1352000]

Hi doveman

I paused the programme (Black Mirror) I was watching (and recording, but I was watching from the Live point) for a few minutes whilst I got my dinner. When I unpaused it, it played for maybe 30-60secs, before the sound and picture completely disintegrated (i.e. it became blocky, only updated about once every 1-2 seconds, completely unwatchable in other words) before jumping to the current live point, which was the next programme.

OK I thought, I guess the buffer must have run out, I'll watch the end from the recording. However, this does the same thing (completely reproducible), so I can't watch the end of the programme, which is rather annoying. I can't understand why pausing, even if the buffer did run out, should damage the recording but it has.

In my opinion it is not pausing that caused the problem. We would get many complaints about this problem if that were the case. I can clearly see that the issue with this particular recording starts to present on the TsWriter side at 21:31:27.709. The message is "Discontinuity header bit set!" - that usually means that the tuner driver has recognised that there is unrecoverable corruption in the stream and is warning the downstream processors and decoders (ie. TsWriter and TsReader). After that I see various bad messages:
MultiFileWriter: failed to create file R:\\live5-0.ts.tsbuffer2.ts
Failed to reopen old file. It's currently in use. Dropping data!
Recorderid 230 Continuity error... 0 ( prev 4 ) - bad signal?

I'm not surprised that the recording is totally unwatchable because it looks like the stream itself is absolutely and totally corrupt. Those messages are not entirely isolated either. There are "discontinuity header bit set" and continuity error messages occuring sporadically throughout the day.

There are a few potential causes/sources of corruption:
- bad signal quality (which could be caused by bad wiring, interference, dish/aerial alignment... etc.)
- stressed HDD
- overheating components
- spin up/spin down of other HDDs

mm

 

[doveman]

I'm fairly sure it's connected to my pausing as

a) I've never had anything like this happen before or since (not recently or that I can recall anyway) and so it's seems an unlikely coincidence that my aerial/wiring would develop a temporary fault for just the period (or part of) that I had it paused

b) my timeshift folders are on RAM drive, so the HD shouldn't have been overstressed. Certainly no other recordings have been corrupted like this

c) I take cooling very seriously and monitor all my temps and they're fine. I don't even have my PC in a case at the moment, it's sitting on a bench

d) I only have one HD

Surely the

MultiFileWriter: failed to create file R:\\live5-0.ts.tsbuffer2.ts
Failed to reopen old file. It's currently in use. Dropping data!

errors can't have anything to do with the corrupted recording, which is separate from the timeshift files?

 

[mm1352000]

a) I've never had anything like this happen before or since (not recently or that I can recall anyway) and so it's seems an unlikely coincidence that my aerial/wiring would develop a temporary fault for just the period (or part of) that I had it paused

I'm not saying it was a temporary fault. You have relatively frequent discontinuities in your logs for that recording - sometimes as close as 5-10 seconds but other times as far apart as 3 minutes or so. This is not a once off. I can see similar issues from earlier in the day. If I had more logs then I'm reasonably confident that they would also show the pattern.

b) my timeshift folders are on RAM drive, so the HD shouldn't have been overstressed. Certainly no other recordings have been corrupted like this

So how big is the RAM drive? Maybe it filled up while you had TV paused (how long was TV paused for?) and that was why I see the...

11-12-2011 22:46:32.177 MultiFileWriter: failed to create file R:\\live5-0.ts.tsbuffer2.ts

11-12-2011 22:46:32.177 Failed to reopen old file. It's currently in use. Dropping data!

...messages.

c) I take cooling very seriously and monitor all my temps and they're fine. I don't even have my PC in a case at the moment, it's sitting on a bench

Okay.

d) I only have one HD

Okay.

Surely the

MultiFileWriter: failed to create file R:\\live5-0.ts.tsbuffer2.ts
Failed to reopen old file. It's currently in use. Dropping data!

errors can't have anything to do with the corrupted recording, which is separate from the timeshift files?

From the logs I would guess they are connected - the continuity errors on recorder PIDs start when data starts being dropped. If data is being dropped for timeshifting then I think it is entirely possible that it is being dropped for the recording too.

mm

 

[doveman]

I'm not saying it was a temporary fault. You have relatively frequent discontinuities in your logs for that recording - sometimes as close as 5-10 seconds but other times as far apart as 3 minutes or so. This is not a once off. I can see similar issues from earlier in the day. If I had more logs then I'm reasonably confident that they would also show the pattern.

The point I'm trying to make is that even if there is a signal problem (and I'm sure you're right that my logs would show more discontinuities) it's never caused corrupted recordings (or timeshifting) before, so it seems strange that it's only done so on this occassion, when I happened to have paused timeshifting.

So how big is the RAM drive? Maybe it filled up while you had TV paused (how long was TV paused for?) and that was why I see the...

It's set to min 3, max 6, Filesize 40MB, so it should give me a max of 240MB (or approx 6mins). I don't think I had it paused that long, but let's assume I did and that's why the timeshifting went kaputt.

From the logs I would guess they are connected - the continuity errors on recorder PIDs start when data starts being dropped. If data is being dropped for timeshifting then I think it is entirely possible that it is being dropped for the recording too.

I think they're connected too, but by that I mean that perhaps the timeshifting run out of space and for some reason this also corrupted the recording. I just find it too hard to believe that this fault, which has never happened before, was not due to the paused timeshifting (I can't recall when I last used this feature) but just a signal problem that happened to manifest exactly and only whilst timeshifting was paused. I guess I could test by recording something I don't care about whilst watching it and pause it until the timeshift buffer runs out again.

 

[Paranoid Delusion]

It's set to min 3, max 6, Filesize 40MB, so it should give me a max of 240MB (or approx 6mins)

I have never seen such a small amount being used, but imo it is asking for failure and I'm not even going to ask why you changed things so drastically.

 

[doveman]

It's set to min 3, max 6, Filesize 40MB, so it should give me a max of 240MB (or approx 6mins)

I have never seen such a small amount being used, but imo it is asking for failure and I'm not even going to ask why you changed things so drastically.

I know you didn't ask but it's quite simple. I've got two tuners and didn't want to give up more than 512MB of RAM for my timeshifting/RAMDrive (which is then unavailable for anything else) and I can't imagine myself needing to pause for more than 6 minutes anyway. I might increase it now I've upgraded to 8GB. Do you agree then that the timeshift drive running out of space does wreck recordings?

 

[elliottmc]

It's set to min 3, max 6, Filesize 40MB, so it should give me a max of 240MB (or approx 6mins)

I have never seen such a small amount being used, but imo it is asking for failure and I'm not even going to ask why you changed things so drastically.

I know you didn't ask but it's quite simple. I've got two tuners and didn't want to give up more than 512MB of RAM for my timeshifting/RAMDrive (which is then unavailable for anything else) and I can't imagine myself needing to pause for more than 6 minutes anyway. I might increase it now I've upgraded to 8GB. Do you agree then that the timeshift drive running out of space does wreck recordings?

Well, if you can reproduce the problem using those settings, then maybe pausing is related. However, if you can then I would try changing back to a normal HD for timeshifting. See if that works better.

Mark

 

[doveman]

I just did another test and paused whilst recording from 20:56 to 21:04 and didn't notice any breakup when I resumed playback, but the recording does show a couple of spots of breakup, that last about 10 seconds each. The second set of attached logs are when I was testing playback of the recording, which eventually crashed MP.

I also note the bulk of Continuity errors are when I had it paused. There are some at other places (as well as Discontinuity errors) but the vast majority are when it was paused.

So it does seem that the timeshift running out of space (if that's what happened, I'm assuming it did) corrupts recordings unfortunately, at least if the programme being recorded is the same one being timeshifted, which I'm surprised by as I'd have thought they were two separate processes.

Perhaps a solution would be (and I think I've suggested this before) to allow the use of a RAMdisk for timeshift but have it switch to using the HD if it's paused, which I think would be ideal as I only occasionally use the timeshift feature to skip back a few seconds to check something I missed or misheard, so don't need a large buffer and using the RAMdisk for this prevents constant writes to the HD, with all the implications for wear but more importantly noise that entails, whereas when I pause it's normally just so I can make a cup of tea or get my dinner (which doesn't require long either) but could potentially be because someone phones, in which case I may need it paused for longer than my RAMdisk allows.

Incidentally, I'm not sure if I've misunderstood the TS settings as I was under the impression that I needed to double the space requirements for two tuners, but thinking about it I can only be timeshifting one channel/tuner at a time, so can't I just allocate most of my RAMdrive, thus doubling the amount of time it can currently pause for?

The estimates seem wrong as well, as it only seems to go by the filesize*minimum (+1*filesize overhead), so for min 3, max 6, filesize 80MB it says the space needed is 320MB, when 80MB*6 is in fact 480MB (and if I need to allow an overhead as well, that takes it to 560MB, in which case I'd need to reduce the max or the filesize to keep it within my 512MB RAMdisk).

The estimated time for SD recording seems wrong as well, as my 17min recording is just under 400MB, so around 23MB/min, but with min 3, max 6, filesize 70MB, which it says requires 280MB (which I think should be 490MB), it estimates the maximum SD timeshifting time as 5.775min.

 

[mm1352000]

Hi again doveman

I just did another test and paused whilst recording from 20:56 to 21:04 and didn't notice any breakup when I resumed playback, but the recording does show a couple of spots of breakup, that last about 10 seconds each. The second set of attached logs are when I was testing playback of the recording, which eventually crashed MP.

The first logs show that the tuner is setting discontinuity flags almost as soon as you start playing.

13-12-2011 20:48:08.816 Recorderid 231 : Discontinuity header bit set!

Then at at 20:49:

13-12-2011 20:49:11.630 MultiFileWriter: failed to create file R:\\live5-0.ts.tsbuffer1.ts

I would guess that the TS drive was out of space at this point already.

At 20:56 we start seeing the dropping data messages:

13-12-2011 20:56:07.160 Failed to reopen old file. It's currently in use. Dropping data!

Then about 5 seconds later we start to see discontinuities on the recorder PIDs:

13-12-2011 20:56:13.696 Recorderid 230 Continuity error... 9 ( prev b ) - bad signal?

I also note the bulk of Continuity errors are when I had it paused. There are some at other places (as well as Discontinuity errors) but the vast majority are when it was paused.

I'd agree.

So it does seem that the timeshift running out of space (if that's what happened, I'm assuming it did) corrupts recordings unfortunately, at least if the programme being recorded is the same one being timeshifted, which I'm surprised by as I'd have thought they were two separate processes.

Well it absolutely depends how you're watching the "recording". If you start timeshifting on the same channel then they're independent processes, however you go into the recordings section and start watching the recording from the live point then they're not separate.

Perhaps a solution would be (and I think I've suggested this before) to allow the use of a RAMdisk for timeshift but have it switch to using the HD if it's paused, which I think would be ideal as I only occasionally use the timeshift feature to skip back a few seconds to check something I missed or misheard, so don't need a large buffer and using the RAMdisk for this prevents constant writes to the HD, with all the implications for wear but more importantly noise that entails, whereas when I pause it's normally just so I can make a cup of tea or get my dinner (which doesn't require long either) but could potentially be because someone phones, in which case I may need it paused for longer than my RAMdisk allows.

Yes, maybe that would be a solution in an ideal world, however the complexity in supporting that would be substantial. Further we seem to only have a limited number of devs who are able and interested to work on TsWriter and TsReader. Practically speaking I don't see this feature being added in the foreseeable future.

Incidentally, I'm not sure if I've misunderstood the TS settings as I was under the impression that I needed to double the space requirements for two tuners, but thinking about it I can only be timeshifting one channel/tuner at a time, so can't I just allocate most of my RAMdrive, thus doubling the amount of time it can currently pause for?

I actually thought the timeshift settings are applied over all tuners and users, not per-tuner. I don't think I've ever actually looked into that code though.

The estimates seem wrong as well, as it only seems to go by the filesize*minimum (+1*filesize overhead), so for min 3, max 6, filesize 80MB it says the space needed is 320MB, when 80MB*6 is in fact 480MB (and if I need to allow an overhead as well, that takes it to 560MB, in which case I'd need to reduce the max or the filesize to keep it within my 512MB RAMdisk).

Personally I totally agree with Ray - 512 MB seems an inadequate amount of space. I think you can ignore the overhead to some degree since you're not using a traditional HDD that gets significantly slower as it gets very close to full.

The estimated time for SD recording seems wrong as well, as my 17min recording is just under 400MB, so around 23MB/min, but with min 3, max 6, filesize 70MB, which it says requires 280MB (which I think should be 490MB), it estimates the maximum SD timeshifting time as 5.775min.

*It is an estimate* - if you work back the maths it seems to be using a relatively high bitrate of 10 Mbps for that calculation. Your recording (which may be missing data) is only 3.14 Mbps.

mm