1.27.0 Memory leak in TVService (3 Viewers)

[CyberSimian]

I have never used the "record every time on every channel" option. So at the moment it looks to me that that option is the culprit (but I have not tested this yet).

I have tested it now.

• I used my test system that I had not booted for several weeks, so the EPG was empty. I used "Refresh DVB EPG" in "Manual Control" in "TV Server Config" to grab the EPG using the idle grabber for 15 minutes (and waiting a further 20 minutes for the database updates to finish).
• I then started the MP client and scheduled 10 series using "every time on every channel". A couple of the series each had about 50 episodes in the 7-day EPG, the remainder had typically 2 to 4 episodes in the EPG.
• I then grabbed the EPG again using the idle grabber for 15 minutes.
• Before/after each step I took a screen shot of "Resource Manager" showing how much memory TV Server was using.

The outcome of all this is that no memory leak was indicated -- at all points in the test sequence the memory consumed by TV Server was less than 100 MB (which I think is typical for TV Server). I did not test the timeshift grabber, so it is possible that the memory leak exists there.

@Berney : If you frequently encounter this problem, all that I can suggest is that you try changing some of the settings and seeing what effect they have; e.g. disable the timeshift grabber and use only the idle grabber for a day or two, and then reverse those settings and try that for a day or two.

It would be useful to identify the cause of this problem, even if currently there is no one available to fix it.

-- from CyberSimian in the UK

 

[joecrow]

I'll try what joecrow suggested once I know what is meant by 'provider'. My guess is it means 'MUX/Transponder'?

Sorry about that, seems there is a difference in that respect between UK and German TV.

 

[Berney]

Hi everyone, I hope you all had a good weekend.

Joecrow, yes it would seem there are differences between Germany and the Uk because there us no data in my provider column.

CyberSimian, I absolutely agree with you that it would be useful to
a) be able to recreate the problem easily so it can be a quality control test.
And
B) Narrow down where the problem is occurring as much as possible so that it can be put on a 'to fix' list when someone with the required skills comes along, making their job easier.

I've done the following test over the weekend:-

As suggested by CyberSimian I changed all the 'Record every time on every channel' (ETEC) instances to 'Record Every time on this channel ' (ETTC). There were 9 occurances.
I've run the test over the weekend including up till now which is approximately 60 hours. The TVServer no longer locked up. The memory still filled up at around the same rate and up to the same amount full (approximately 1.6 gig). And the server has crashed 3 times, but it was able to restart each time due to windows service settings I have put in (i.e. restart the service in a crash). Where as previously the TVServer locked and was unable to easily restart it easily. So there has been some improvement but its not stopped the problem of a memory leak of some sort occurring.
Now my next thought is what is the difference between CyberSimian's (CS) testing and my pre-test situation? If I understand CS correctly had the same number of ETEC recordings as I did (9). CS recordings happened successfully and no apparent memory leaks. Now here is the difference as I see it:-
In my 9 recordings setup none of them occurred (because none of the 9 ETEC recordings were broadcast). So could it be that is the problem that yet to be recorded settings are left in memory and gradually build up till the memory is full ?
This would seem to fit in with the way I record in general which I will explain. Generally I like to collect recordings not just for what I like to watch but to gather for when family and friends turn up and want to watch that they find interesting from my 'library' that I have built up. Often a friend will say 'hey I've just become interested in a series and I missed the first few episodes'. So I'll put in a scheduled recording for either ETEC or more likely ETTC. Over the years this has built up to many scheduled recordings that might happen say a couple of years in the future or not at all ever. The way I use Mediaportal means there are very many scheduled recordings that rarely get actually recorded, possibly using a lot of memory that some how builds up till it totally fills up.

I've been trying to figure out the source code and from what I've deduced so far it would seem to me that the epg is grabbed from a broadcast stream in the file EpgDBUpdater.cs in the procedure 'ImportPrograms'. This is put into the sql database. Checking of is it time to actually do a recording is done in the file 'Scheduler.cs' in a procedure 'IsTimeToRecord' which I believe is triggered on a timer event every so often. In that procedure there is a switch statement that chooses one program path depending upon Recording Type (e.g. record once, record daily, record every time this channel etc.). Now what stands out to me in that switch statement is that the type for ETEC calls a procedure 'IsTimeToRecordEveryTimeOnEveryChannel' is to my eyes significantly different from all other recording types. If I had been writing it I probably would have based it similarly to the ETTC and simply checked through in a loop of every channel. The procedure being different to the rest might not catch an error condition the same as the other recording types and not handle an error as well as them hence the TVServer lock up being caused?

Ok thats it from me today as my tummy is rumbling and I need to go cook dinner!

 

[doskabouter]

It could get quite technical really fast, and I don't have any experience with them, but since it's all .net, there are probably some good memory profilers to be found.
Perhaps that would shine a light on things?

 

[CyberSimian]

I changed all the 'Record every time on every channel' (ETEC) instances to 'Record Every time on this channel ' (ETTC)... ...The memory still filled up at around the same rate and up to the same amount full (approximately 1.6 gig)

This suggests that the memory leak is not related to the schedule type (ETEC or ETTC). My test result of no memory leak with ETEC is consistent with this. My test was not comprehensive, and a different test might provoke the memory leak:

• Perhaps a single EPG grab is not sufficient, and there needs to be hundreds of EPG grabs for the memory leak to accumulate and be noticeable.
• Perhaps it is the timeshift grabber that causes the memory leak.
• Perhaps the memory leak occurs when a scheduled recording starts, and not when the EPG is grabbed.

My test did not test any of these possibilities.

Your current setup is valuable because it seems to be able to reproduce the error consistently. The suggestion from @doskabouter of using a memory profiler to see if it is possible to identify where the memory leak is occurring is a good one. But as I have never used such a tool, I don't know how easy it would be to set it up for MP. If you have some programming experience (and the time available!), the memory profiler might be the best way to investigate this problem.

If you want simply to be able to use MP for enjoyment, and not spend hours investigating its inner workings, I would suggest that you change your EPG grabber settings, and use the ones that I suggested earlier, to make the EPG processing as simple as possible:

• Use the idle grabber once per day, grabbing from only one channel.
• Disable the timeshift grabber (at least, for a few days).

If this avoids the memory leak, you will have a usable work-around. However, it might cause the memory leak to disappear forever from your system, and no longer be reproducible (since we don't know what exact combination of settings causes the error to occur).

I'll put in a scheduled recording for either ETEC or more likely ETTC. Over the years this has built up to many scheduled recordings that might happen say a couple of years in the future or not at all ever. The way I use Mediaportal means there are very many scheduled recordings that rarely get actually recorded, possibly using a lot of memory that some how builds up till it totally fills up.

I do this too. I currently have around 150 series scheduled to record, but only a small number of them are actually being broadcast at the moment. I have not encountered the memory leak with this number of series, but if you have a significantly larger number of series scheduled, that cannot be ruled out as a cause of the problem.

-- from CyberSimian in the UK

 

[CyberSimian]

I have just read the first two pages of this thread, and I see that this post from @RonD describes a careful study of the memory used by TV Server. If I have understood his test scenario correctly, the HTPC was never put into standby (sleep or hibernation) during these tests.

My HTPC always hibernates when it is not being used, which may be why I have not noticed the memory leak problem. I would guess that @Berney is leaving his HTPC powered-on all of the time.

My conclusion is that a possible work-around for this problem is to sleep or hibernate the TV Server system when it is not being used. This will prevent the memory used by TV Server increasing unnecessarily, and normal activities such as grabbing the EPG should trigger the .NET memory garbage collector at appropriate times (well, maybe).

-- from CyberSimian in the UK

 

[Berney]

Hi everyone

Doskabouter, I've no experience with memory profilers so know virtually nothing about them. Would you (or anyone else) know how detailed they profile memory, e.g. down to variable level ? If so, would a special build of TVServer be required in order to provide info on the memory locations to the profiler?

CyberSimian, Yes it does seems difficult to reproduce the error conditions for the memory leak to become obvious. The original poster seems to me to be so similar as to be identical to my problems, it would be useful to compare notes with the original poster with respect to system setup and tvserver setup this might then provide a clue as to a way to regularly cause the problem and thus can be put under the microscope more fully.

I've only got the one windows pc and thus can't setup a test pc. I currently manage the memory leak problem by looking at task manager every day (often multiple times) to see how full memory is. With having removed ETEC scheduled recordings at least if it does crash it will automatically restart the TVServer within a few minutes, whereas before it just locked up and almost impossible to restart with out rebooting the PC, so there has been a work around of sorts that works for me.

I plan to study the source code at length in order to understand it as much as I can. I might get lucky and spot the problem, though some how I doubt it.

My current thinking is that there are 2 ways the epg is grabbed in the source code and that they conflict with each other, causing memory to remain taken up. I think it would be nice if there was just one basic set of procedures to grab the EPG from a broadcast stream and have that set of basic procedures called as and when needed either while timeshifting/recording or as a regular timed event but undoubtedly there is probably a good reason as to why it is not done that way that I am currently unaware of.

I have approximately 420 scheduled recordings setup, they are currently a mixture of single and ETTC.

I'm very busy over the next 3 weeks so I'll not be able to devote much time to the problem. However this nemory leak does irritate me and I will be keeping an eye open on this forum thread for any ideas that people might have.

Btw the second way the EPG is grabbed is in the source file 'TimeShiftingEPGGrabber.cs' should anyone care to take a look at it.

 

[Berney]

I have just read the first two pages of this thread, and I see that this post from @RonD describes a careful study of the memory used by TV Server. If I have understood his test scenario correctly, the HTPC was never put into standby (sleep or hibernation) during these tests.

My HTPC always hibernates when it is not being used, which may be why I have not noticed the memory leak problem. I would guess that @Berney is leaving his HTPC powered-on all of the time.

My conclusion is that a possible work-around for this problem is to sleep or hibernate the TV Server system when it is not being used. This will prevent the memory used by TV Server increasing unnecessarily, and normal activities such as grabbing the EPG should trigger the .NET memory garbage collector at appropriate times (well, maybe).

-- from CyberSimian in the UK

CyberSimian, yes that is correct, my pc is left on 24/7/365 because it does not hibernate/sleep well.

 

[CyberSimian]

I don't know what is causing the "memory leak" problem, and I have no knowledge of the internals of TV Server and no experience of .Net programming . However...

Earlier posts from those who know more than I have suggested that the problem may be caused by the .Net Garbage Collector not collecting garbage soon enough. This made me wonder whether there is a SetGarbageCollectionThreshhold() function that could be used to set a lower threshhold for garbage collection (1 GB, or perhaps even 512 MB). I did not find such a function (but I performed only a cursory search). However, I did find a GC.Collect() function that can be used to trigger garbage collection. This suggests that TV Server could be modified to contain logic something like this:

if "n" minutes have elapsed since the previous call to GC.Collect(),
and TV Server is not streaming content to an MP client,
and TV Server is not recording TV or radio,
and a recording is not about to start within the next 5 minutes,
then call GC.Collect();

"n" should probably be a configurable setting (since we don't know what would be an appropriate value).

-- from CyberSimian in the UK

 

[Berney]

I don't know what is causing the "memory leak" problem, and I have no knowledge of the internals of TV Server and no experience of .Net programming . However...

Earlier posts from those who know more than I have suggested that the problem may be caused by the .Net Garbage Collector not collecting garbage soon enough. This made me wonder whether there is a SetGarbageCollectionThreshhold() function that could be used to set a lower threshhold for garbage collection (1 GB, or perhaps even 512 MB). I did not find such a function (but I performed only a cursory search). However, I did find a GC.Collect() function that can be used to trigger garbage collection. This suggests that TV Server could be modified to contain logic something like this:

if "n" minutes have elapsed since the previous call to GC.Collect(),
and TV Server is not streaming content to an MP client,
and TV Server is not recording TV or radio,
and a recording is not about to start within the next 5 minutes,
then call GC.Collect();

"n" should probably be a configurable setting (since we don't know what would be an appropriate value).

-- from CyberSimian in the UK

Hi CyberSimian,

From my observations via Windows Task Manager it seems to me that the Private Working Set of memory for TVService.exe grows steadily and then it plumets and starts once more to grow steadily. This tends to repeat again and again but each time of the cycle it pushes up the Peak Working Set of memory. So this suggests to me that the garbage collector is working to clear up no longer used memory. However it also suggests to me that gradually more and more memory is being used for variables and 1 or more variables are not being freed to be garbage collected for when the garbage collector kicks in. The garbage collector will only return memory into available memory for variables that have been freed. So your suggestion is a good one in that it would mean the TVServer would crash less often, but it would still eventually crash when the new limit was reached.

When I have some free time I will try the earlier suggestion of switching off either Timeshifting/recording epg grab or epg grab while idle and see if this makes a difference.