1.27.0 Memory leak in TVService (4 Viewers)

[Berney]

Hello

I've had a quick peruse of the above conversation and I too have the same problem.
I use a Windows 7 pc that is as up to date as I can make it. It has 32 Gb of memory in it. I run little else on it apart from MediaPortal 1.25.

This problem is definitely in version 1.21 and I believe it is also in version 1.19 when I started to narrow down the problem.

Last night I started and stopped the TV service from the TV Server configuration program because starting and stopping the TV Service is the only way I have discovered to minimise the problem of MediaPortal locking up. No programs were scheduled to be recorded and I do not use MediaPortal to play back recordings. This morning MediaPortal had locked up. Checking the memory usage of the TVService*32.exe showed approximately 1.6 GB Peak Working Set Memory and a little less for the Private Working set. There was no CPU usage on the TVService*32.exe. I tried to start and stop the TVService but was unable to. I tried to use the Task Manager to End the Process Tree, but was not able to. I tried to shutdown the PC and that failed too. Eventually I had to pull the plug.

So thats some background to the problems I have with MediaPortal. My past investigations led me to posts previously about the same subject. From what I read the problem appears to be due to scheduling recordings but only a specific type of schedued recording. When you schedule a recording via 'Advanced Record' and then select 'Record every time on EVERY channel' it starts to cause the problems. I had quite a few such scheduled recordings (approximately 10) and MediaPortal would lock up frequently. When I reduced the scheduled recordings for every channel to a more meagre 4 the lockups occur approximately once a day unless I start and stop the TVService.

So may I suggest the following test:-

Start and stop the TVservice.
Put a whole load of scheduled recordings in for programmes with the 'every time on EVERY channel'. The more set up to record the quicker the problem will manifest itself.
Then keep an eye on the memory usage in Task Manager.
I think you will find that the memory can then be seen to keep on climbing at a rate proportional to the number of said scheduled recordings.
The problem might also be proportional to the number of available channels on the transmitter being recorded from.

So that's the proposed test. Whats the cause ? I speculate that its due to a basic problem of a memory leak on the schedule a recording. Its too tiny a memory leak for it to be noticed when using 'Record once' or 'Record every time on this channel' but because 'Record every time on EVERY CHANNEL' simply magnifies the tiny memory leak problem to a noticeable one and having too many of such scheduled recordings causes it to be a major problem causing MediaPortal to lock and for the operating system to be unable to gracefully shutdown without pulling the plug.

I find the 'Record every time on every Channel' to be a very useful feature and I really would like this problem to be solved and not solved by simply removing the feature.

I'd like to help, but because I use Windows 7 I do not connect it to the Internet and I find that the current system of having to have a connection to the internet to perform an update very irritating because while my PC is connected to the Net it is vulnerable.

I hope my above thoughts are helpful and help lead to a fix of this extremely irritating problem.

Finally my thanks to everyone involved in creating MediaPortal!

 

[Berney]

Just to add to the information I gave:-

I too have two usb tv sticks that both have twin tuners. One is an IT9135 k-world usb stick and the other is hauppauge win-tv dualHD usn stick.

 

[CyberSimian]

I speculate that its due to a basic problem of a memory leak on the schedule a recording... ...'Record every time on EVERY CHANNEL' simply magnifies the tiny memory leak problem to a noticeable one

This is an interesting observation. I will test this, but I need to setup a test system first (I don't want to use my production system).

One question: how are you obtaining the EPG? The reason that I ask this is that I would guess that TV Server actions "record every time on every channel" only when the EPG is obtained. This is when TV Server identifies the programmes to record, and is most likely to be the point at which the memory leak occurs.

If you are using the EPG transmitted via DVB-T/T2 or DVB-S/S2, you may have the settings set inappropriately, causing TV Server to grab the EPG repeatedly, exacerbating the memory-leak problem. As an example, I use the DVB-T/T2 EPG, but I grab it only once per day, from a single channel (not from every channel). I have never used "record every time on every channel", so I don't know whether grabbing the EPG once per day would avoid the problem.

Be that as it may, it is not likely that this problem will be fixed in the foreseeable future, as MP currently lacks anyone who has expertise in this area. Sadly, the most that can be hoped for is to identify a work-around. I regret that this is probably not the outcome that you wanted.

-- from CyberSimian in the UK

 

[Berney]

This is an interesting observation. I will test this, but I need to setup a test system first (I don't want to use my production system).

One question: how are you obtaining the EPG? The reason that I ask this is that I would guess that TV Server actions "record every time on every channel" only when the EPG is obtained. This is when TV Server identifies the programmes to record, and is most likely to be the point at which the memory leak occurs.

If you are using the EPG transmitted via DVB-T/T2 or DVB-S/S2, you may have the settings set inappropriately, causing TV Server to grab the EPG repeatedly, exacerbating the memory-leak problem. As an example, I use the DVB-T/T2 EPG, but I grab it only once per day, from a single channel (not from every channel). I have never used "record every time on every channel", so I don't know whether grabbing the EPG once per day would avoid the problem.

Be that as it may, it is not likely that this problem will be fixed in the foreseeable future, as MP currently lacks anyone who has expertise in this area. Sadly, the most that can be hoped for is to identify a work-around. I regret that this is probably not the outcome that you wanted.

-- from CyberSimian in the UK

Thanks for showing interest in the problem.

I only use dvb to get the epg. There are 2 usb sticks, the IT9135 cannot receive HD, where as the Hauppauge can.

Details of the DVB EPG page in the TV Server Configuration are as follows:-

Always try to fill holes ON
Always try to update existing entries ON
Enable CRC check OFF
Grab EPG only for channels on same transponder OFF

EPG grabbing while timeshifting/recording ON
Timeout 2 mins

Enable grabbing while idle ON
Timeout 2 mins
Refresh every 720 mins.

The TV Epg grabber page:-

Store data only for selected channels ON

The channels ticked are around 90% of the list of channels.

Having looked at the debug log produced it would appear that the actual grab of the data off air occurs every 6 hours (so it would seem the 720 minutes setting is working correctly.).
What I did find interesting is that when it does do the update it produces a log entry for each channel. There is a log entry for inserting X epg entries for channel Y. However there is no mention in the log for removing old entries (which I think would be due to having become out of date, i.e. in the past). Now this may be simply the programmer thinking there is no reason to insert a log entry and in fact does a delete of out of date epg entries, but if old out of date entries are not deleted then this could be a cause of concern and while not actually a memory leak it would cause the data to keep on growing until memory limits are reached!
To find the point in the log I refer to then set logging level to debug in the TV Server under the general page.
Run the server for a long enough time to do an epg grab and then use a text editor on the log file TVServer.log and search for 'EPG Update Thread' where you will see entries that I am referring to.
(Apologies for spelling it out in detail, but I do so in the hope of other people perhaps needing to know how to do these things and helping them).

Finally I note that the EPG is grabbed in two ways.
A) while idle (at set interval)
B) while time shifting/recording.

From the debug log it would appear that
B) EPG grabbing is stopped when A) starts at the set interval.

Its a gut feeling of mine that the memory fills up quicker during the night when there is typically no time shifting/recording occuring. Which makes me wonder if interval EPG method is where the problem can be narrowed down to.

 

[joecrow]

Store data only for selected channels ON

The channels ticked are around 90% of the list of channels.

I am sure @CyberSimian will have more to say about those settings but I would strongly suggest you untick "Store data only for selected channels" and only select one channel per provider from the list. Providers are listed under TV Server Config/TV Channels.

 

[Berney]

Ignore my gut feeling of memory filling up quicker when not timeshifting/recording. My usual rough testing is as follows:-

I normally stop/start the server just before 7pm before I watch tv during the evening to ensure viewing via time shifting has no problems. Then in the morning I check memory usage (which is typically reaching around 1,500,000 K for peak working set by then) and again reset so it will not crash at around 1,600, 000K allowing day time recording to work without problems.
So last night I left it timeshifting and as usual it reached around the 1,500,000K this morning. So I think its safe to say there is no difference in how quickly the memory fills and my gut feeling was wrong.

I'll try what joecrow suggested once I know what is meant by 'provider'. My guess is it means 'MUX/Transponder'?

 

[Berney]

I should add that in the TV Server Config page for TV Channels that there are no values in the column 'Provider' on my pc.

 

[CyberSimian]

I only use dvb to get the epg... ...Details of the DVB EPG page in the TV Server Configuration are as follows:-

For the DVB EPG, TV Server makes available various settings that provide a great deal of flexibility. This is partly to satisfy users' different needs, and partly to accommodate the different EPG arrangements used by different countries around the world. But the result is that there is a great deal of scope for picking a combination of settings that is not optimal. Below I suggest settings for DVB-T/T2 in the UK. For other countries, the considerations will be similar, but the details may not be identical.

Overview
The broadcast EPG can be regarded as a carousel that rotates continuously. To obtain a complete 7-day EPG, TV Server needs to grab the EPG for one complete rotation of the carousel. When digital TV started in the UK, one rotation took around 4 minutes, but with the explosion in the number of channels, one rotation nowadays take around 8 to 10 minutes. The exact time needed varies from day to day (depending on the content of the EPG), so it can take less than 8 minutes or more than 10 minutes. It will also depend on where you are in the UK. Some locations may receive fewer channels, and so have a smaller EPG.

Because TVs mostly do not store the EPG across standby, the programme info for the next several hours needs to be repeated in the broadcast EPG every few seconds. This is so that the user can see what is on in the near future without having to wait several minutes for the EPG to populate. On the other hand, those programmes which occur 7 days in the future are likely to occur only once in the carousel. Consequently, if you grab the EPG for only 2 minutes, you are unlikely to receive the programme info for all of the programmes that are 6 to 7 days in the future. You may notice this by finding that there are gaps in the MP EPG for 6 to 7 days in the future. These gaps will populate as the days pass, but of course the gaps are inconvenient of you want to schedule a programme to record that is 7 days in the future but not yet present in the MP EPG.

Idle EPG grabber
For the idle EPG grabber, the grab duration specified is the maximum time TV Server will operate a tuner and extract the EPG from the datastream. However, TV Server is able to determine when it has seen one complete rotation of the carousel, and stop grabbing . So you could specify a grab duration of 60 minutes, but TV Server would still grab only for 8 to 10 minutes.

The grabbing phase extracts the programme info from the broadcast datastream, and caches it in RAM (at least, I think that is how it works). After grabbing is complete, database update starts. The duration of database update will depend on the speed of your hard disk. I use a slowish Western Digital Green drive for Windows, and the database update phase takes longer than the grabbing phase: 8 to 10 minutes for grabbing, 15 to 20 minutes for database update.

To ensure that you always have a full EPG (no gaps), I would recommend performing one idle grab each day, with a grab duration of 15 minutes (for the UK). You can use "PowerScheduler" to cause your HTPC to wake at a time when your HTPC is not in use, grab the EPG, and then return to sleep. I would suggest a refresh time that is 24 hours or just under. A refresh time of 24 hours will stop the idle grabber running during the day when you don't want it to run.

Timeshift EPG grabber
If you use the timeshift grabber, you need to understand a difference between the MUXes. In the UK, each MUX broadcasts the EPG for its own channels, plus the EPGs for all of the other MUXes. However, each MUX transmits differently-ordered data. The best MUXes to use for grabbing are the "BBC1 SD" MUX and the "BBC1 HD" MUX. These MUXes treat the different channels equally, and do not favour the BBC channels unduly. In contrast, the non-BBC MUXes favour their own channels. So if you use only the timeshift grabber, and mostly watch channels which are not in the BBC MUXes, you are likely once again to find gaps in the EPG.

The timeshift grabber is useful for receiving last-minute changes in the EPG (for example, when live sport overruns, or news broadcasts are extended due to world events). But you only need to run the timeshift grabber for a short time, to receive programme info for the next hour or two. A grab time of 1 to 2 minutes is appropriate for this. However, there is a "gotcha"...

Settings with undesirable side effects
I don't understand the details of the TV Server implementation, but there are two settings that can have undesirable consequences:

(1) Grab EPG only for channels on the same transponder
(2) Store data only for selected channels

If neither (1) nor (2) is enabled, TV Server can grab the entire EPG from a single MUX (transponder) in one pass.

If (1) is enabled, TV Server needs to tune each MUX in turn. So at my location where I can receive 6 MUXes, EPG grabbing using the idle grabber would take around 6 times longer.

If (2) is enabled, TV Server tunes each selected channel in turn in order to grab its EPG. So at my location with around 80 channels enabled, EPG grabbing using the idle grabber would take around 80 times longer.

Because of (1) and (2), the recommendation is to select only one channel for grabbing. The "gotcha" is that if you select only one channel for grabbing, the timeshift grabber will work only when you watch or record that channel -- it won't run when watching/recording any other channel.

Settings that I use
In the 7 years that I have been using MP, my EPG settings have evolved several times. Conceptually I use the idle grabber once each day (at 06:00 hours), and don't use the timeshift grabber. But in practice I actually do this:

(1) Disable the idle grabber.
(2) Enable the timeshift grabber, with one channel in the BBC MUX selected.
(3) Record that channel at 06:00 hours for 40 minutes each day.

This gives me a complete EPG grab each day, but allows me to receive last-minute changes to the EPG by recording or watching that channel for 1 to 2 minutes at any time during the day (although I don't often do this). The channel that I use is a BBC radio channel, but a BBC TV channel is equally good. Another advantage of using the timeshift grabber is that the grab still occurs, even if a recording is taking place at the scheduled grab time (the idle grabber would be prevented from running if a recording was occurring or live TV being viewed).

My settings are shown below. Notice that no channels are selected on the "TV EPG Grabber" panel, and one channel is selected on the "Radio EPG Grabber" panel:



Memory leak
With the different EPG settings that I have used over the years, I have not encountered the memory leak. But then I have never used the "record every time on every channel" option. So at the moment it looks to me that that option is the culprit (but I have not tested this yet).

-- from CyberSimian in the UK

 

[CyberSimian]

So last night I left it timeshifting and as usual it reached around the 1,500,000K this morning.

Before hibernating my HTPC last night, I checked "Resource Monitor" to see what memory TV Server was using, and again this morning after my 06:00 hours EPG grab had occurred. The values were not greatly different -- about 150 MB for committed memory, and 75 to 90 MB for the working set. After rebooting, these values were 36 MB and 54 MB respectively. I use a variety of recording options:

Every time on this channel
Every time on this day of the week
Daily at this time of day
Monday-Friday at this time of day

But I don't use "every time on every channel". Since you seem to be able to reproduce the memory leak easily, may I suggest that as a test you delete the "every time on every channel" items and see if that stops the memory leak? (Obviously, you don't want to miss any recordings, so you would need to use "every time on this channel" for those channels that are most likely to broadcast the programmes that you want to record.)

The MP Wiki contains a lot of information, although it is of variable quality, and not always up-to date. For information about the DVB EPG, see this page.

-- from CyberSimian in the UK

 

[Berney]

Hi CyberSimian

Thanks for your excellent reply! There is a lot of really useful info there to work through and understand fully. Unfortunately I won't be able to implement your recommendations till next week.

If you are interested in having a look at the source code that seems to implement the epg update you can find it in the file 'EpgDBUpdater.cs' which is under the branch TVEngine3 TVLibrary TVDatabase TvBusinessLayer. When looking in the file EpgDBUpdater.cs search for the procedure 'ImportPrograms'. I think you will find it fascinating.
Something I did notice at the bottom of the procedure is that '_layer.StartResetProgramStatesThread(....)' is commented out and it leaves me wondering why it was commented out and left in the code. Is it because it was commented out temporarily for testing and forgotten to be reinstated ? Perhaps I will never know.
Anyway it would seem that the EPG Database is updated per channel in a loop within the ImportPrograms Procedure. I've not had time to study it enough myself but intend to when free time is available.
Have a great weekend!