1.14.0 On XmlTV update, TV guide keeps programs that have been removed, if nothing else is in their place. (1 Viewer)

[Blue Lightning]

MediaPortal Version: 1.13.0

Description
If you have an EPG supplied by Xmltv data (possibly other data sources but haven't tested), and the data is updated, and as part of that update a program has been removed but nothing new has been put in its place yet, the old program will still be included in MediaPortal's TV guide. This applies even if "Remove old programs on import" is selected.

I'm having this issue with my 1.11.0 install but I've installed 1.13.0 in a VM to confirm it's not specific to my system and that it is still a bug in 1.13.0. The following details/reports/steps are from that VM and how to reproduce it in a standalone environment.

Here is the SetupTv.log output for each import: http://pastebin.com/raw/H8YCiZJY (the SetupTv-error.log files are empty)

Attached are 3 xml files representing a sample of how the data might update which results in this bug.
** tvguide-import1.xml - Has programs A B C D.
** tvguide-import2.xml - The supplier has now removed 'D' as it is no longer airing, nothing put in place (TBD), and has added E and F after that. Mediaportal incorrectly keeps D in the guide even though it doesn't exist in this xml file.
** tvguide-import3.xml - The supplier has decided E is going to start earlier and run longer, it now overlaps with where D was located, causing Mediaportal to correctly remove D.

Steps to Reproduce:

• Install MP 1.13.0, as singleseat with Default Wide.
• TV Server config: Create a channel called "MPTEST" (no tuning details or tuner needed).
• Plugins > XmlTV (Enable) > point it to a folder that has a dtd.
• Add "tvguide-import1.xml" to that folder, rename it "tvguide.xml". Go to mappings and map the channel, do the guide import. Save settings (TV server config will close).
• Open MediaPortal, save settings, and get MediaPortal loaded.
• Open TV Guide, view Tuesday 1st March, you'll see items A B C D (this is correct so far). Close Mediaportal.
• Remove tvguide.xml, rename tvguide-import2.xml to tvguide.xml, do an import in TV Server config.
• When complete go back into Mediaportal, view TV Guide, you'll see items A B C D E F. This is incorrect, it should be A B C E F. (View the program descriptions and compare with the xml file).
• Close Mediaportal, remove tvguide.xml, rename tvguide-import3.xml to tvguide.xml, do an import. View TV guide again, you'll see A B C E F (which is correct).

 

[mm1352000]

Hello

Thanks for such a detailed report.

From what I can see in the code, I think the behaviour you've identified may be by design.

Can you help me to understand why it is important that program D is deleted?
From my perspective it seems harmless to leave it in the guide until the start time for program E is updated...

 

[Blue Lightning]

I use a custom data grabber (which I won't detail because Australian TV networks are really protective of their schedule for some reason..), but it could stop working at any time. So I have it insert a fake "TV Guide Data Expiry" program at the end of the availability for each channel (runtime 1 minute) and set that as a series recording. So if I see this coming up in the guide in the next few days then I know the data is running out or my grabber isn't running for some reason, and I can look into it.
The issue is on channels where they've left a "TBA" at the end of the day until maybe the day or two before airing, so this still displays the "TV Guide Data Expiry" program until then (as well as when the data actually ends) even though the XML data only has the entry once for that channel. This also happens on channels that don't run 24 hours, for example if a channel ends at 9pm then my XmlTv file has "TV Guide Data Expiry" at 9pm on the final day, but after 1-2 weeks I'd end up with "TV Guide Data Expiry" on at 9pm every day (even though the XmlTv file only has it at the end). Some of them disappear of course if the program before ends up being changed to 9:02pm or something like that.

Those cases might be specific to my situation and not feasible to fix, but there's another issue with it that I noticed recently which I believe is related to the same feature/bug.
TEN is launching a simulcast of their primary channel in HD (TEN HD) next week. It's available for tuning now so I've tuned it but disabled it until it actually starts (at which point I'll disable TEN and re-enable TEN HD). I told TEN HD to use the guide data for TEN, which resulted in suddenly getting every TEN scheduled recording listed twice (once for TEN, again for TEN HD) even though I disabled the channel. I removed the TEN guide mapping for TEN HD but the data continued to display in my schedule for a day or two (and cancelling the TEN HD recording also cancelled the TEN recording, so I was stuck with it recording both channels even though one is just a "coming soon" promo screen). I suspect once TEN HD starts and I disable TEN (and un-map the guide for it), I'll probably keep getting TEN scheduled recordings for a few days.

Would it be possible to add a manual data purge button for specific channels? This would resolve the re-mapping issue retaining old data. I feel like it's probably doing this in the background after a day or two but a button for it in the TV Server Config would be useful.

About my issue with "TV Guide Data Expiry" not clearing, is there something I can provide in the XmlTv data that would force Mediaportal to dump the cached version of the entry when it imports a newer one (such as providing a unique ID for the program)? Or does it have some sort of flag for informational programming that causes them to be treated differently (eg "TBA", "Programs Resume 6am" "Channel starts on Wednesday", etc)?

 

[mm1352000]

I use a custom data grabber...

Hmmm, I understand what you mean. Using programs as markers in the way that you described is quite a specific and unusual scenario. TV Server and the XMLTV plugin assume every program is a real program - there's no concept of markers. They're also optimised to minimise gaps in the guide data. The combination of those 2 factors obviously doesn't fit very well with your scheme.

Those cases might be specific to my situation and not feasible to fix...

Yes, I confess I do have that feeling. However I wonder if there's an alternative solution that would be more robust in the long term. Two questions come to mind:

1. Could you use a different mechanism (for example, email - ie. have the grabber send you an email when it runs into problems) to monitor whether the grabber is working or not differently?
2. From an even wider perspective: is there a reason you've chosen to use this custom grabber over more reliable solutions such as EPG Collector or even TV Server's built in DVB EPG grabber?

I told TEN HD to use the guide data for TEN, which resulted in suddenly getting every TEN scheduled recording listed twice (once for TEN, again for TEN HD) even though I disabled the channel.

Are your schedules "record every time on every channel" type schedules?
If yes, that would conveniently explain all the behaviour you described. You could avoid the problem by using "record every time on this channel" schedules instead. It should also be possible to cancel the TEN HD recordings using the cancel episode function (rather than deleting the schedules completely).
If no, I have no idea how or why MediaPortal would have behaved that way.

Would it be possible to add a manual data purge button for specific channels? This would resolve the re-mapping issue retaining old data. I feel like it's probably doing this in the background after a day or two but a button for it in the TV Server Config would be useful.

Does it need to be for specific channels?
I'm pretty sure the "refresh DVB EPG" button in the "manual control" section of TV Server Configuration purges all EPG data... but it does it for all channels.

About my issue with "TV Guide Data Expiry" not clearing, is there something I can provide in the XmlTv data that would force Mediaportal to dump the cached version of the entry when it imports a newer one (such as providing a unique ID for the program)?

AFAIK the only thing that would force MP to dump a program is a newer overlapping program. That's why item E in tvguide-import3.xml triggers deletion of item D. Gaps in guide data (like the gap between items C and E in tvguide-import2.xml) don't cause existing programs within the gap to be deleted.

Or does it have some sort of flag for informational programming that causes them to be treated differently (eg "TBA", "Programs Resume 6am" "Channel starts on Wednesday", etc)?

No, definitely not. As mentioned above: both TV Server and the XMLTV plugin assume every program is a real program. All programs are treated the same.


A final alternative possibility...
Instead of inserting your "TV Guide Data Expiry" marker programs after the last real program, could you instead insert it in the last minute of the last real program?
I think that may work as a solution because it would change when/how the overlaps work. The full explanation is too complex!

 

[Blue Lightning]

• Could you use a different mechanism (for example, email - ie. have the grabber send you an email when it runs into problems) to monitor whether the grabber is working or not differently?

• From an even wider perspective: is there a reason you've chosen to use this custom grabber over more reliable solutions such as EPG Collector or even TV Server's built in DVB EPG grabber?

I suppose there's no reason I need it to be in the guide, I could try getting it to display that information somewhere else.
I've never tried EPG Collector, but the over-the-air EPG data provided by Australian networks is quite horrible. I haven't checked them for a while but last I checked they were getting title additions such as "Double Episode" "New Time" "*Includes lotto draw at 10:35pm*" "Encore", and absolutely no inclusion of the episode title. I'm unsure if a lack of episode title is a limitation of the EPG system over DVB-T or simply a case of the networks not caring to provide it. That's why I'm using a custom grabber/scraper, so I can get the details from a more accurate source which includes more accurate program titles (I still have to strip a few "- Double Episode" though) and includes episode titles for most episode-based programs. (It was also a decent learning experience to code). I should perhaps look into the over-the-air data again and see if it's still as bad as when I last checked.

Are your schedules "record every time on every channel" type schedules?
If yes, that would conveniently explain all the behaviour you described. You could avoid the problem by using "record every time on this channel" schedules instead.

Yes. That's the behaviour I expected from it, the oddity was that trying to cancel the specific airing on TEN HD also cancelled the specific airing on TEN (not the entire schedule).

I'm pretty sure the "refresh DVB EPG" button in the "manual control" section of TV Server Configuration purges all EPG data

Thanks, I'll try that when I do some channel remapping and need old data cleared. I forgot that option existed.

Instead of inserting your "TV Guide Data Expiry" marker programs after the last real program, could you instead insert it in the last minute of the last real program?
I think that may work as a solution because it would change when/how the overlaps work. The full explanation is too complex!

I think I understand what you mean, MP would adjust the actual final program to be 1 minute shorter and then when re-importing the next day it'll be 1 minute longer than cached and would wipe out the cached "TV Guide Data Expiry" program? I'll give that a try and see how it goes.

Thank you for your help!

 

[mm1352000]

I suppose there's no reason I need it to be in the guide, I could try getting it to display that information somewhere else...

First a disclaimer: I'm based in NZ and don't have first-hand experience with Aussie TV.
From what I've heard, the EIT data suffers from all of the problems you've described, but the MHEG 5 data is better. EPG Collector supports grabbing both formats. It also supports enhancing the data with episode information via TVDB lookup.

You might also want to look at this plugin:
https://forum.team-mediaportal.com/threads/plugin-epg-cleaner-schedule-repair.133416/

...developed by an Aussie to combat these exact same problems.

So, there are definitely alternative solutions out there if you take a look.

Yes. That's the behaviour I expected from it, the oddity was that trying to cancel the specific airing on TEN HD also cancelled the specific airing on TEN (not the entire schedule).

Hmmm, maybe the cancellation is time-based, and not tied to a specific channel. That might be something for somebody to look into fixing...

I think I understand what you mean, MP would adjust the actual final program to be 1 minute shorter and then when re-importing the next day it'll be 1 minute longer than cached and would wipe out the cached "TV Guide Data Expiry" program? I'll give that a try and see how it goes.

Yep, that's what I was thinking. Well put!
Even if the program ended up getting scheduled as one minute shorter than it should be, you could simply extend the post-record time to compensate.

Thank you for your help!

No problem, happy to be of assistance.