1.27.0 Memory leak in TVService (1 Viewer)

[PRW]

There appears to be a memory leak in TvService.exe of about 15KB per hour. After restarting my system (W10, MP 1.27) Windows task manager showed 17KB of active and 32KB of committed memory assigned to TvService. Two hours later with no recording or playback the numbers were 50KB active and 65KB committed. Five and a half hours after the restart with no recording or playback they were 113KB and 128KB. About 24 hours later with some recording and playback they were 432KB and 457KB.

The numbers continue to climb until I run TV-Server Configuration, Manual Control, Stop Service, then Start Service, to set the memory allocation back to about 20KB active and start over.


I have 4 USB tuners.


This problem is not new to MP 1.27. I noticed it several versions ago.

 

[PRW]

Correction: All instances of KB should be MB.

 

[mm1352000]

Hello

First, thanks for reporting.

It's possible there is a leak. Always possible. The tricky thing about reporting a memory leak based on task manager's memory details is that TvService is a .NET process. That means garbage collection (freeing of memory that's no longer required) is an automatic and unpredictable background process. From a code perspective TvService can have finished using a particular chunk of memory, but task manager will report that the memory is still being used until the .NET garbage collector cleans up. This makes it difficult to determine whether a leak has actually occurred, and also difficult to determine when and where (in code) the leak is.

Some clarification questions...

After restarting my system (W10, MP 1.27) Windows task manager showed 17KB of active and 32KB of committed memory assigned to TvService. Two hours later with no recording or playback the numbers were 50KB active and 65KB committed. Five and a half hours after the restart with no recording or playback they were 113KB and 128KB.

Would it be fair to say that you expect TvService's memory usage to have remained the same over those 7.5 hours (because live TV wasn't used and there were no recordings in that timeframe)?

I note that even when TvService isn't streaming live TV or recording, it may perform various other background tasks. The most common is possibly updating program guide data. Is it possible that your TvService ran background tasks in the 7.5 hours you referred to? (If yes, this - in combination with the .NET garbage collection mentioned above - may explain the task manager numbers.)

About 24 hours later with some recording and playback they were 432KB and 457KB.

Would it be fair to say that after the recordings and playback finished you expect TvService's memory usage to have returned to the usage observed shortly after system restart?

I note that there's at least one reason - in addition to the .NET garbage collection mentioned above - that this may not happen. That is: tuners are not "unloaded" until the TvService is stopped. By default TvService loads tuners as required to enable streaming and/or recording. Loading each tuner requires some memory usage. That memory won't be released until the tuners are unloaded when the TvService process is stopped. In other words, it's pretty normal for memory usage to be XYZ immediately after the TvService starts... then for memory usage to increase when the TvService is streaming and/or recording... and for memory usage to not go back down to XYZ when streaming and recording stop.

This problem is not new to MP 1.27. I noticed it several versions ago.

Are you saying that at one time TvService did not leak for you, and now it does?
...or is it possible that TvService has always leaked, and you didn't notice until several versions ago?

Do you remember - or are you able to determine - which version of MediaPortal you were using when you first noticed the leak?

 

[PRW]

Thanks for your reply.


I had a hunch the problem was deeper than I could see just from the numbers in Task Manager.


My system is up 24/7 so memory allocation is not routinely reduced to initial values. There is a very severe memory leak somewhere other than TvService (I assume) that forces me to restart my system every 5 or 6 days to avoid out of memory errors. As a result I keep a close eye on the the amount of committed memory and restart to avoid crashes/hangs. (Yes, I have expanded my virtual memory file.)


A couple of days following a system restart TvService is ALWAYS the single process with the most memory allocated. As far as I can tell the amount of memory assigned to TvS never goes down unless I manually stop and restart it. This would seem to suggest either that automated garbage collection doesn't happen or that it does happen but doesn't find all the unused memory or that some memory is never marked unused.


To answer your questions: I would not necessarily expect TvS to return to its initial memory usage. But I would expect the usage to stop growing at some point, which does not seem to happen.


As to background activities: I had been using EPG-Buddy to get program guide data one day each week. The memory allocated to TvS increases regardless of the day of the week.


As to recording and playback: Again, I would not necessarily expect TvS to return to its initial memory allocation after some recording and playback. But I would expect the usage to stop growing at some point.


I cannot say what version of MP I was using when I first noticed the problem. I only noticed the memory usage for TvS while I was keeping tabs on a different memory leak.

 

[mm1352000]

Okay.

I suspect the best way for us to make progress on this issue is to start by eliminating TvS background tasks. The goal is to control and understand exactly what TvS is doing at any point in time in the hope that this may clarify when and why TvS is using memory.

EPG Grabber
Is the built-in EPG grabber enabled?

1. Open TV Server Configuration
2. Go to the "DVB EPG" section.
3. The EPG Grabber is enabled if either of the 2 "enabled" ("EPG grabbing while timeshifting/recording" and/or "EPG grabbing while idle") checkboxes are ticked.

If you're based in the USA as your forum profile indicates, you almost certainly don't need the built-in EPG grabber. I mean, this EPG grabber doesn't support grabbing North American guide data. It's doubly unnecessary if you're using EPG-Buddy.

If the built-in EPG grabber is enabled, I strongly recommend to disable it.
(Note that you need to restart TvS after unticking the "enabled" checkboxes for the change to take effect.)

Recording Thumbnail Creator
Is the thumbnail creator enabled?

1. Open TV Server Configuration
2. Go to the "Recording" section.
3. Go to the "Thumbs" tab.
4. The thumbnail creator is enabled if the "autocreate thumbs" checkbox is ticked.

Having thumbnails (frame-grabs for use as a preview/icon) for recordings can be helpful and/or desirable for various reasons. However, in the interests of identifying the source of the issue you're seeing I strongly recommend you disable it. At least for now.

(Note that you need to restart TvS after unticking the "autocreate thumbs" checkbox for the change to take effect.)

Plugins
Which TvS plugins are enabled?

You indicated you are (or were?) using EPG-Buddy as your guide data source. If I recall correctly, using EPG-Buddy requires the XMLTV plugin. Given how mature, stable, and widely used the XMLTV plugin is, I'd be very, very surprised if the issue you're seeing had anything to do with it. So..

If you're actively using EPG-Buddy then feel free to leave the XMLTV plugin enabled.
If you're no longer using EPG-Buddy, I recommend to disable the XMLTV plugin as a precaution.

Any other plugins you can disable - even if only temporarily - may help to identify the source of the issue.



The above covers all the questions and recommendations I can think of at present.
After following the recommendations you're willing to follow I suggest monitoring TvS's memory usage for 24 hours or so to see if the changes have made a difference.

 

[PRW]

I must modify one earlier statement. The memory allotted to TvS does go up and down a bit but goes down only by small amounts -- in the 10s of KB -- while the general trend is up -- about 18 MB per hour.
For the test you requested:
EPG Grabber was not enabled.
Recording thumbnail creator was enabled. I disabled it.
The only plugin enabled was XMLTV. I disabled it.
I stopped and restarted TvS.
The initial allocation was 16.7 MB. 24 hours later the allocation was 477 MB.

Start test
16.7 MB @ 9:15 after stopping and restarting TvS before any activity
50.5 MB @ 10:15 after some playback
68.7 MB @ 11:15 no new activity
87.6 MB @ 12:15 no new activity
106.3 MB @ 13:15 no new activity
126 MB @ 14:15 no new activity
144 MB @ 15:20 no new activity
161 MB @ 16:15 no new activity
256 MB @ 21:15 after some playback
one hour recorded at 22:00
Next day
477 MB @ 9:15

 

[mm1352000]

Interesting. Okay. So in short: apparently no significant difference compared to the numbers you initially reported.

I must confess I was hoping that we'd see some change in the outcome, and that this would lead to a relatively easy resolution. This result is taking us down a harder path.

For now I only have more questions...!

1. You've said previously that your system is normally up 24/7. To make sure I've understood that statement correctly: you mean that the system is normally on, and you don't use standby, hibernate, or any other power-saving scheme - correct?

2. Without asking you to monitor more closely (...yet!): is it your current understanding/observation that memory usage just slowly creeps up over the seconds and minutes? A dripping tap that slowly but surely leaks ~20 MB per hour, as opposed to static memory usage 99% of the time followed by a burst-increase at [for example] a particular time every hour?

3. Again, without asking you to monitor more closely: is it your current understanding/observation that memory usage starts creeping up from the moment TvS starts? That, as opposed to an issue that only starts after a particular interaction - for example, after the first recording starts or live TV usage?

4. Does this system also run MP?
5. In total how many MP (client/frontend) installations use this instance of TvS?
6. Roughly speaking, what is the usage pattern for the clients? In particular, are some/any always on... or do they generally sit in standby/hibernate unless they're being actively used?

I ask the above client-related questions because at this point I honestly can't think of any other purely TvS-driven and "disableable" background tasks. That would only leave us with client interactions (which obviously can't occur when the clients are in standby or not running - hence question 6) and non-disableable tasks such as:

• the recording/scheduler loop ("Is it time to start or stop a recording now?"), which if I recall correctly runs every 15 seconds
• the power state change monitor ("Does the system want to suspend or resume now?"), which does "stuff" every time Windows notifies us of an event we may or may not care about
• the tuner detector ("Has a new tuner been connected to the system, or has an existing tuner been disconnected from the system?"), which does "stuff" every time Windows notifies us that "a device connected to the system has changed", and when UPnP devices connect to or disconnect from the network
• the streaming server, which enables remote clients to stream live TV and recordings

All of these are very much core tasks, which is why they can't be disabled. A set of debug-level log files would potentially show how active the power state change monitor, tuner detector, and streaming server are. It's possible - for example - that on your particular system Windows is triggering the tuner detector more often than expected... and therefore the memory usage would be more obvious on your system than any other.

 

[PRW]

No, no significant change.
1. My system is powered on and running 24/7. (I participate in Folding@home so there is always work to do.)
2. I have not noticed big jumps in memory, just a steady increase.
3. Yes. memory usage starts to increase immediately after restarting TvS.
4. The system occasionally runs MP. The primary playback is through the client.
5. Only one client.
6. The client is normally powered on about 1600 and powered off about 2100. It does not use standby or hibernate.
I have had the same 4 USB turners attached to the same ports for at least 4 years.

I gather from your questions that you do not see a problem on your system.

 

[mm1352000]

I gather from your questions that you do not see a problem on your system.

Correct. However, I don't use MP as regularly as I once did, I'm currently not running the latest version, I don't have a dedicated server that runs 24/7, or separate clients. I just have the one PC with a single-seat installation that I use for a variety of other tasks (office, internet etc.), and which is turned off when I'm not using it. Very basic.

Given your answers to the most recent round of questions (thanks!), I think the next step is for me to ask you to provide debug-level TV Server log files. As I previously mentioned, the log files should (I hope!) give further insight into the nitty-gritty specifics of what TvS is doing when it's apparently idle. It's one thing for memory usage to increase when TvS is being used (which might even be as light as maintaining a connection to an idle instance of MP); it's quite another for memory usage to increase when TvS is sitting there doing nothing except the basics, with no client connections.

So, please could you:

1. Open TV Server Configuration.
2. Go to the "general" section.
3. Set log level to debug.
4. Go to the "manual" section.
5. Click "stop service"... then "start service".
6. Note the date/time.
7. Wait 24 hours or so, noting any significant periods of expected activity. Doesn't have to be in great detail. For example, "watched recordings and/or live TV for ~1 hour at ~7pm". (This is just to help me match what I see in the log files with what actually happened in the real world.)
8. Open TV Server Configuration.
9. Click "open log directory" in the top left corner.
10. Zip up all the files in that folder, and attach the zip file to your next update... along with the details from (6) and (7).

 

[PRW]

Here are the logs for about 21 hours. (A Windows feature update did not go well.)
1545 restart TvS with log level = debug
1600 Power up client
2030 Start MP on client
Playback 30 min of recorded program on client
2100 Exit MP and power down client
Next day
0910 Power up client No MP
Update Windows on client
Update Windows on server
1300 restart server to install Windows update
1505 Server back up after update