Gaps in displayed EPG from WebEPG and OzTiVo (Australia)

[Taipan]

I use WebEPG to get the EPG data from OzTiVo and I have noticed that sometimes there are gaps (missing programs) in the EPG that is displayed in Mediaportal. The gaps always begin at midnight and they seem to only occur at the point where new data is added to existing data.

For example, I run WebEPG on Friday afternoons to get the next 7 day’s data (that's all the data available) – which is through to midnight on the following Friday. All that data appears fine in the EPG.

Then I run WebEPG again on the next Tuesday afternoon, to pick up additional data for the following Saturday (and Sunday, if it is available).

This is when the gaps appear – any programs (like Rage on ABC) that commence before midnight on the Friday, do not connect up correctly with the remainder of the program on Saturday morning.

Here’s a typical example taken directly from the tvguide.xml file that is generated by WebEPG. To make it easier to read (and follow my explanation), I have bolded the times for each program.

In this example, Rage is shown as starting on October 13 at 23:55 and then finishing the next morning (on October 14) at 00:00 (midnight). The next entry shows the same program (Rage) commencing at 05:00 and going through until 08:00. There is a gap in the data for Rage from 00:00 to 05:00…..and this shows as no program in the EPG.

- <programme start="20061013235500" stop="20061014000000" channel="abc-vic@abc.net.au" clumpidx="0/1">
  <title>Rage</title> 
  <desc>Continuous music programming</desc> 
  <category>Music</category> 
  </programme>
- <programme start="20061014050000" stop="20061014080000" channel="abc-vic@abc.net.au" clumpidx="0/1">
  <title>Rage</title> 
  <desc>Continuous music programming</desc> 
  <category>Music</category> 
  </programme>

Here is the corresponding data in the WebEPG.log file, and it appears to me that all the correct data has been “grabbed” from OzTiVo.

I have highlighted the line that shows Rage commencing at 23:55 and concluding at 05:00 – with no gap.

10/10/2006 5:25:43 PM [Debug] WebEPG: Guide, Program title: Rage
10/10/2006 5:25:43 PM [Debug] WebEPG: Guide, Program start: 13:55 - 13/10/2006
10/10/2006 5:25:43 PM [Debug] WebEPG: Guide, Program end  : 19:0 - 13/10/2006
10/10/2006 5:25:43 PM [Debug] WebEPG: Guide, Program desc.: Continuous music programming
10/10/2006 5:25:43 PM [Debug] WebEPG: Guide, Program genre: Music
10/10/2006 5:25:43 PM [Debug] WebEPG: TimeZone, Adjusting from start Guide Time: 1:55 PM 13/10/2006
10/10/2006 5:25:43 PM [Debug] WebEPG: TimeZone, Adjusting to   start Local Time: 11:55 PM 13/10/2006
10/10/2006 5:25:43 PM [Debug] WebEPG: TimeZone, Adjusting from end Guide Time: 7:00 PM 13/10/2006
10/10/2006 5:25:43 PM [Debug] WebEPG: TimeZone, Adjusting to   end Local Time: 5:00 AM 14/10/2006
10/10/2006 5:25:43 PM [Info.] WebEPG: Guide, Program Info: 20061013235500 / 20061014050000 - Rage
10/10/2006 5:25:43 PM [Info.] WebEPG: Program in db copying it
10/10/2006 5:25:43 PM [Debug] WebEPG: Guide, Program title: Rage
10/10/2006 5:25:43 PM [Debug] WebEPG: Guide, Program start: 19:0 - 13/10/2006
10/10/2006 5:25:43 PM [Debug] WebEPG: Guide, Program end  : 22:0 - 13/10/2006
10/10/2006 5:25:43 PM [Debug] WebEPG: Guide, Program desc.: Continuous music programming
10/10/2006 5:25:43 PM [Debug] WebEPG: Guide, Program genre: Music
10/10/2006 5:25:43 PM [Debug] WebEPG: TimeZone, Adjusting from start Guide Time: 7:00 PM 13/10/2006
10/10/2006 5:25:43 PM [Debug] WebEPG: TimeZone, Adjusting to   start Local Time: 5:00 AM 14/10/2006
10/10/2006 5:25:43 PM [Debug] WebEPG: TimeZone, Adjusting from end Guide Time: 10:00 PM 13/10/2006
10/10/2006 5:25:43 PM [Debug] WebEPG: TimeZone, Adjusting to   end Local Time: 8:00 AM 14/10/2006
10/10/2006 5:25:43 PM [Info.] WebEPG: Guide, Program Info: 20061014050000 / 20061014080000 - Rage
10/10/2006 5:25:43 PM [Debug] WebEPG: Guide, Program title: Rage
10/10/2006 5:25:43 PM [Debug] WebEPG: Guide, Program start: 22:0 - 13/10/2006
10/10/2006 5:25:43 PM [Debug] WebEPG: Guide, Program end  : 23:0 - 13/10/2006
10/10/2006 5:25:43 PM [Debug] WebEPG: Guide, Program desc.: Continuous music programming
10/10/2006 5:25:43 PM [Debug] WebEPG: Guide, Program genre: Music
10/10/2006 5:25:43 PM [Debug] WebEPG: TimeZone, Adjusting from start Guide Time: 10:00 PM 13/10/2006
10/10/2006 5:25:43 PM [Debug] WebEPG: TimeZone, Adjusting to   start Local Time: 8:00 AM 14/10/2006
10/10/2006 5:25:43 PM [Debug] WebEPG: TimeZone, Adjusting from end Guide Time: 11:00 PM 13/10/2006
10/10/2006 5:25:43 PM [Debug] WebEPG: TimeZone, Adjusting to   end Local Time: 9:00 AM 14/10/2006
10/10/2006 5:25:43 PM [Info.] WebEPG: Guide, Program Info: 20061014080000 / 20061014090000 – Rage

I hope you can follow my explanation, and someone can throw some light on what is causing these gaps…..

[zombiepig]

taipan - i get the gaps too with oztivo/webepg. if you need extra info to help troubleshoot it, i can send through my logs or whatever...

[Taipan]

Thanks for the update zombiepig - I am glad that I am not alone - I was begining to think I had screwed something up.....

Some more information (for James?):-

I am using MP 0.2.0.4 - no SVN.

I also noticed that if I "Remove all programs form TV database" (in MP Setup / Television / Program Guide) and then run WebEPG again (no other changes), the gaps are now filled correctly....

So, it looks to me that WebEPG checks the existing EPG data in the TV database and only "grabs" new data, but it then fails to correctly join the old data with the new data for programs that overlap midnight?

When I deleted all the old data, then WebEPG was forced to grab all new data and thus the gaps around midnight no longer existed. Remember that these gaps only appear where new data is added to existing data....I hope my explanation is clear.....

[squidina]

I have been playing around with a grabber for myself ( I just wanted to teach myself C# and this seemed to be the best way of doing it ) and I have come across the same problem, so I don't think it's limited to the WEBEpg code. I suspect that the importer is not overwriting "old" data.

I tried it out with an xml file which had a show going from 9:00am until 5:00am and then a new xml file which changed the show to go from 9 until 10 and other shows after 10. The old data was retained, and the new data was not updated.

From memory the epg log did say that it had imported all the shows correctly ( although I may be wrong on this point ). I'm using a patched version of 0.2.0.4, so it may be different in the latest SVN.

If I get some time on the weekend, I will investigate further, if no one has a solution by then. I just figured my code was to blame ( wouldn't be the first time )

[Marcusb]

Yep, I get them too.
I'm using the latest SVN's of different ages on each of two PC's

[sonome]

Hmmm.... strange, I don't have any gaps, so I don't think it's an issue with MP's importer. I'm using Will Uther's script (http://www.cse.unsw.edu.au/~willu/xm...ab_au_reg.html) and OzTivo

[Taipan]

Originally Posted by sonome

Hmmm.... strange, I don't have any gaps, so I don't think it's an issue with MP's importer. I'm using Will Uther's script (http://www.cse.unsw.edu.au/~willu/xm...ab_au_reg.html) and OzTivo

No, I don't think the problem is with MP's importer either - I think the problem is with WebEPG itself. And since you are not using WebEPG, and you are not getting any gaps, then that tends to support my assumption that WebEPG is doing something odd here....

[James]

I have been trying to reproduce this problem without success

Can someone who is having this problem save a copy of the WebEPG log, xmltv file and TV database before and after the problem?

So with a clean tv db ... import a new xmltv file (copy files)
update from site and re-import xmltv file (copy files)

This would help. I will try over the next few days as more guide data appear on the site.

[Taipan]

Hi James,

You probably haven't been able to reproduce the problem over the last few days as the data at OzTiVo hasn't changed.

The problem only seems to occur after new data is added to OzTiVo - for example, ABC-Vic has data through to midnight Friday 20th October, but OzTiVo will most likely add Saturday's (October 21st) data tomorrow (AU Tuesday) afternoon.

To reproduce the problem, grab all the current data from OzTiVo today, and then run WebEPG again after the Saturday data has been added to OzTiVo (on your Tuesday) - then you should see gaps around midnight Friday 20th where the new data does not join correctly with the existing data.

I can do this for you and send you the requested files, if you would prefer?

Cheers,

Taipan

[James]

I tried to reproduce it by loading the data and then deleting one day from my TV database (by hand) and then loading the data again, but no luck. I will try again, but it would be good if you could do it too and send me the files if I don't succeed.