1.15.0 Recording and timeshift ts files saved by System account with backgound priority (1 Viewer)

[Andy_2639]

MediaPortal Version: 1.15.0

Description
Media Portal TV server runs as Windows service under System account.
The ts files for recordings and time shifting are written to HDD with background priority. This causes lost packets when other IO is done at the same time.
The processor priority is set to "Above Normal". I would expect that the IO priority also is higher than normal or at least normal. I see TV recording as soft real time process (which therefore needs priority over other "normal" stuff) because missed data is lost.

I made a screen shot in which resmon shows the IO priority with which the recording is written to disk. I use a German Windows:
E/A-Priorität means IO priority, Hintergrund means background.
The screen shot is taken in a situation in which only Media Portal and the TV service were used.
The screen shot was made on 2639draco (MP 1.16 pre-release, Windows 10 Anniversary Update).

This also appears on 2639capricornus (MP 1.15 final) which often has background activity due to other running services or Windows doing some stuff.

Steps to Reproduce:
Start recording in Media Portal.
Use "resmon" (shipped with Windows) on Disk tab to see with which priority ts file is written to disk.

 

[mm1352000]

Hello

Media Portal TV server runs as Windows service under System account.

This is the normal default. You can change the account in the service configuration if you want to.

The processor priority is set to "Above Normal".

Are you referring to the TV Server Config -> general -> TVService priority setting here, or...?
I would have thought that threads started by the service would inherit the service priority. This may not be the case if you manually set the process priority in some other way.

 

[Andy_2639]

Are you referring to the TV Server Config -> general -> TVService priority setting here, or...?

Yes, I use the GUI option TV Server Config -> general -> TVService priority to adjust the CPU priority. I don't use other ways to set the priority.
This priority change is applied correctly to the TVServer process (at least this is what the Taskmanager says).

Generally, I like to increase the priority of media players a bit - if they officially have this option.

 

[mm1352000]

Please allow me to be up-front with you.

1. I'm a little doubtful that your problem is caused by I/O contention.
Packet loss could be caused by other factors like signal strength/quality issues, a badly behaved driver that causes high DPC latency, etc.

2. In my opinion it's not practical for us to set the I/O priority.
Results of a quick Internet search suggest that Microsoft doesn't intend to make it possible for programs to increase their I/O priority. It's possible (via SetProcessPriorityClass with PROCESS_MODE_BACKGROUND_BEGIN/PROCESS_MODE_BACKGROUND_END) for programs to lower their I/O priority when doing background work (we're definitely not doing this!)... but in all other cases, Windows gets to choose the I/O priority.
How to change I/O priority of a process or thread in Win7?
Fun With Process/Thread and IO Priorities

The links above do mention an "NTSetInformationProcess" Windows API function which can be used to select any preferred I/O priority value. However, that function is undocumented. That's why I said "Microsoft doesn't intend...". In my opinion it's best that we avoid using undocumented functions.

3. I'm a little doubtful that setting the I/O priority would be a reliable solution for an I/O contention problem.
Increasing the priority doesn't make the contention go away. You'd still be stuck trying to get good performance while multi-tasking with a single, half full (therefore, probably reasonably fragmented) 5400 rpm HDD.

If you haven't already done so, my advice would be to try to confirm that the problem is really caused by I/O contention. In other words:

• check signal strength/quality
• check DPC latency
• try to use Process Lasso or similar to increase the I/O priority and check if it makes any difference
  
• defragment your HDD and check if that makes any difference
  
• if possible, reduce multi-tasking while recording and check if that makes any difference

If the problem is caused by I/O contention, probably the best solution may involve upgrading your HDD (eg. get an SSD and/or additional/faster HDD) and/or use of Process Lasso.

 

[Owlsroost]

...Also make sure Windows 'Write-caching' is enabled for the disk: Device Manager -> Disk drives -> (select disk) -> Policies tab

 

[Andy_2639]

I used 2639draco (my laptop) just as test system whether the IO priority background still occurs with 1.16.0 pre-release. I use 2639capricornus to watch and record TV.

I can set the service account only to System or administrator. With other accounts with less priviledges I get the error Graph building failed (see attached logs). (2639draco and 2639capricornus)


The following is about 2639capricornus.

The signal strength and quality look good.
The artifacts/dropouts correlated pretty good with IO happening.

I used DPC latency checker (see attached screenshots). I didn't know the thing about dpc latency before and glad to learn about it, thanks!
The latency check from 2017-02-27 ran a few minutes.
The check from 2017-03-02 ran about 20 hours. It was 1.5 ms until I did a little stress test:
Recording 2 HD channels at the same time + watching a 3rd HD channel at the same time (all on one DVB-S2 transponder as I only have one TV card on this system) + FTPES file transfer with 11 MB/s and after the FTP file transfer a SMB file transfer with 11 MB/s (both file transfers wrote 700 MB to 2639capricornus which did the TV playing and recording).
The file transfers are with 11 MB/s only because my laptop is only capable of 100 MBit Ethernet (something with the GBit part of the network card is broken).
After the stress test the maximum DPC latency was by 2.1 ms.
During the stress test I noticed a short audio artefact in the live TV stream. But there were no artefacts or dropouts in the 2 recordings.
This is strange because earlier FTP or SMB traffic would cause many dropouts making the recoding unusable.

I tried to increase the IO priority before I wrote the initial post with IOPriorityV1.1. This changed nothing on resmon reporting the TS being written with background priority. So I did not continue with changing the IO priority.
Resmon also reports the TS file written by System (PID 4) regardless whether the tv server runs unter System or Administrator account.
I checked the IO priority for System (PID 4) and TVServer with ProcessExplorer. Both have priority normal. Despite this, the TS file is written with background priority. I don't know what Windows thinks of this ...
I attached screenshots of ProcessExplorer showing details about the 2 processes while recording one channel. The screenshots are taken without IO priority modification by IOPriorityV1.1.

Write-Caching is enabled.

Reducing multi-tasking isn't an option as tv is only a small part of the use-case of the system. Also starting MediaPortal (client) while the TVServer did recording was enough to get artifacts or dropouts. This suddenly diappeared ...

IO priorization doesn't eliminate IO congestion but it can soothe the impact: I don't care whether windows backgound stuff or file transfers take a bit longer because something different is more important - but I do care whether tv recordings have dropouts.

I don't know why my system now behaves better ... I don't like bugs which duck away when I try to investigate them ...
Maybe it is because I disabled one RAM drive. I had 2: one 4 GB and one 3 GB. Now I have just one 4 GB. The amount of RAM didn't change and is 16 GB.

 

[mm1352000]

Thanks for the update Andy_2639

I used 2639draco (my laptop) just as test system whether the IO priority background still occurs with 1.16.0 pre-release.

Nothing has changed with 1.16 PR. As explained in my previous reply, Windows chooses TV Server's I/O priority. The fact that Windows chooses "background" is not considered a bug/problem. In other words: don't expect TV Server's I/O priority to change.

I can set the service account only to System or administrator. With other accounts with less priviledges I get the error Graph building failed (see attached logs). (2639draco and 2639capricornus)

You've set the TV Server log level to "warning":
[2017-02-26 12:51:26,739] [Log ] [4 ] [INFO ] - Set loglevel to: Warning

That means we get almost nothing in the log files, and so it's hard to say anything about the error. I can only guess that the account you were trying to use doesn't have permission to write to c:\ProgramData\Team MediaPortal\MediaPortal TV Server.


...but why are you trying to use a different account in the first place? The default is recommended and shouldn't be changed unless you have good reason to do so.

The signal strength and quality look good.

Are you basing this conclusion on the tuner's signal strength/quality readings?
If yes, please be aware that tuner readings tend not to show temporal issues (short/intermittent glitches/jitter).
Your "dropouts" are temporal issues so you shouldn't necessarily trust your tuner's signal readings.

The artifacts/dropouts correlated pretty good with IO happening.

If the artefacts correlated with I/O, shouldn't we have expected almost continuous artefacts during your test?

I used DPC latency checker...

The system specifications in your forum profile say that you're using Windows 10. If that's the case these results should be disregarded. As it says on the DPC Latency Checker home/download page:

Windows 8 Compatibility: The DPC latency utility runs on Windows 8 but does not show correct values.

If you want to check DPC latency on W8 or newer, you need to use another program such as LatencyMon.

I tried to increase the IO priority before I wrote the initial post with IOPriorityV1.1. This changed nothing on resmon reporting the TS being written with background priority. So I did not continue with changing the IO priority.

Have you considered that resmon might be reporting incorrect details?

Reducing multi-tasking isn't an option as tv is only a small part of the use-case of the system.

If you insist that the system is to be used for multi-tasking and you're convinced the problem is I/O-related, maybe you need to seriously consider an additional/new HDD/SSD as previously suggested. That 5400 rpm HDD imposes real constraints. You have to be realistic about what you can achieve within those constraints.

IO priorization doesn't eliminate IO congestion but it can soothe the impact: I don't care whether windows backgound stuff or file transfers take a bit longer because something different is more important - but I do care whether tv recordings have dropouts.

I don't know why my system now behaves better ... I don't like bugs which duck away when I try to investigate them ...

This is exactly why I said that I didn't think setting I/O priority would be a reliable solution.
I can only repeat: if you're convinced the problem is I/O-related (I'm not yet) and you care about drop-outs in recordings, in my opinion your focus on I/O priority is misplaced.

Maybe it is because I disabled one RAM drive.

Note that RAM drive drivers can affect DPC latency.

 

[Andy_2639]

...but why are you trying to use a different account in the first place? The default is recommended and shouldn't be changed unless you have good reason to do so.

I don't want to give a program which allows incoming network connections more rights than it has to have. TVServer just can communicate inside my LAN so this is not that critical to me. I will investigate this further and start a new thread when I have time to create a log with debug level.

Are you basing this conclusion on the tuner's signal strength/quality readings?

Yes, I used the signal strength/quality data provided by MediaPortal (F9 - 8 Empfangs-Details, I think something like reception details in English) - and I guess this is what my tuner reports to MediaPortal.

If the artefacts correlated with I/O, shouldn't we have expected almost continuous artefacts during your test?

Yes, and this is what I meant with bugs ducking away ...
I cannot reproduce the behavior of my system I observed earlier.

Thanks for the link to LatencyMon. As I already said, DPC latency is new to me and I didn't know this tool.
A 28 hour measurement report is attached. 8 ms peak don't look good ...
But the tool didn't get my CPU speed correctly - it runs with 800 MHz.

Have you considered that resmon might be reporting incorrect details?

resmon is a tool built in Windows (C:\Windows\System32\resmon.exe) and I trust Microsoft knowing their system. However, I cannot explain why the ts files should be written with background priority ...

and you're convinced the problem is I/O-related

As my last tests showed completely different results than I observed earlier I already began to doubt.

Note that RAM drive drivers can affect DPC latency.

I'm still using the same software for the one RAM drive that I used before for the two. So there should be the same drivers/kernel modules loaded. I don't know whether 1 or 2 drives can make a difference ...

Just to be sure that I get the thing with DPC latency right:
DVB stream is continuous. TVService can only get the stream in packets. So there must be a buffer in my USB tv card. When the latency is to high, this buffer overflows and a part of the DVB stream gets lost.
Please correct me when I'm wrong.

I still would like to see the IO priority increased. But as you stated earlier that the API is not documented, I also think that using non-API isn't a good idea. So refusing this request is perfectly fine to me.

As I can't reproduce the earlier behavior of my system and DPC latency doesn't look good, I think this issue can be closed.

Thank you very much for your help!

Kind regards
Andy

 

[mm1352000]

I guess this is what my tuner reports to MediaPortal.

Yes it is, so the points I mentioned earlier about accuracy and/or usefulness for trouble-shooting with "bursty"/"spikey" noise do apply.

But the tool didn't get my CPU speed correctly - it runs with 800 MHz.

As you probably know, CPUs don't run at constant speed anymore. I suspect your CPU was "clocked down" for power saving. Such power saving can also cause problems for TV Server.

Just to be sure that I get the thing with DPC latency right...

Your explanation pretty much matches my understanding.
TV Server receives the stream in chunks. The chunks are larger than just one DVB/MPEG 2 packet (188 bytes), but the same principles apply. When other drivers take a long time to execute their interrupt procedures or deferred procedure calls, the tuner driver and/or TV Server (because we use synchronous I/O) doesn't get enough time for stream processing. This causes parts of the stream to be lost as you said.

 

[Andy_2639]

As you probably know, CPUs don't run at constant speed anymore. I suspect your CPU was "clocked down" for power saving. Such power saving can also cause problems for TV Server.

I know about this enery saving feature. capri's CPU constantly runs at 800 MHz because I configured Windows to use the lowest possible frequency.
Are the problems you are talking about caused by the frequency switching or by running the CPU at low frequencies?

Regarding DPC latency: Are TV cards known which have less (or more) trouble with DPC latency (e.g. by bigger/smaller buffers)?