EPG Collector for MHEG5, EIT, OpenTV & MediaHighway (2 Viewers)

[RonD]

ok, so your results are about what I saw. I think the FCC requires all/most channels support 12 hours of OTA/EPG and that is what I saw. A few channels do more than minimum.

The OTA/EPG listings have <title>The Big Bang Theory</title> fields, but I've never seen anything with <sub-title>EPISODE TITLE</sub-title>, so I don't think the tvdb/imdb lookup can find episode details for description, cast, or season/episode info.

I guess the <episode-num system="xmltv_ns">-1 . -1 . </episode-num> is really just a placeholder, the xmltv_ns number format is a little strange in that it is zero-based. For example The Big Bang Theory (The Countdown Reflection) is S5 E24 is coded as <episode-num system="xmltv_ns">4.23.</episode-num>

 

[Pat Clark]

The few I checked manually seem to run out on the 27th having been run on the 24th. Can't work on system right now because wife is using it.

It does seem like a default scheme for new MP installs could be developed, in which EPG Collector is installed in an MP folder, and then be referred to appropriately in the XMLTV plugin.

Since the channel IDs of EPGC and MP are both (derived / derivable) from the broadcast data, they could be made to match, so very little (if any) manual setup would be required. This all might require a bit of fiddling with all three: EPGC, MP, and XMLTV, but not much.

Who would know what the tv.guide properties are?

Who would know how the .dtd file is used?

 

[mm1352000]

Aside: a few comments from an interested dev who has attempted to implement ATSC EPG support for TVE 3.5...

I think the FCC requires all/most channels support 12 hours of OTA/EPG and that is what I saw. A few channels do more than minimum.

I've heard of such an FCC requirement, but apparently a lot of channels ignore it or the data quality is not great. I was quite surprised to read Pat's comments and was wondering whether the better data came from the TVDB/IMDB lookup that EPGC does.

The OTA/EPG listings have <title>The Big Bang Theory</title> fields...

Title is a mandatory field in the ATSC EPG format ("PSIP") as well as XMLTV format.

...but I've never seen anything with <sub-title>EPISODE TITLE</sub-title>

Not surprising. The ATSC EPG format does not have an episode name/title field. Only a title and generic text (which is usually a description, perhaps with other content stuffed into it). Season and episode number fields are also not defined in the format. Broadcasters could use custom descriptors to convey this information, but this seems unlikely since they don't have any integration with the TV or set-top-box manufacturers. In other words: they could broadcast the data, but the TVs and STBs wouldn't know it existed ...unless TV and STB manufacturers integrated support for the custom descriptors.

...so I don't think the tvdb/imdb lookup can find episode details for description, cast, or season/episode info.

That's my understanding too.

I guess the <episode-num system="xmltv_ns">-1 . -1 . </episode-num> is really just a placeholder

Agreed.

...the xmltv_ns number format is a little strange in that it is zero-based.

Zero-indexing is very common in code. It's not so strange for a developer.

For example The Big Bang Theory (The Countdown Reflection) is S5 E24 is coded as <episode-num system="xmltv_ns">4.23.</episode-num>

Correct.

 

[mm1352000]

It does seem like a default scheme for new MP installs could be developed, in which EPG Collector is installed in an MP folder, and then be referred to appropriately in the XMLTV plugin.

The problem with doing this is that EPGC and TV Server have no visibility over when each is or is not using tuners. It is better for a single application to have absolute control over a tuner... or not control it at all. That's why I "reimplemented" ATSC EPG support - along with various other formats that EPGC supports - inside TVE 3.5.

Since the channel IDs of EPGC and MP are both (derived / derivable) from the broadcast data, they could be made to match, so very little (if any) manual setup would be required. This all might require a bit of fiddling with all three: EPGC, MP, and XMLTV, but not much.

You're right. However, the above-mentioned tuner sharing is a significant problem that can't be overcome this way.

Who would know what the tv.guide properties are?

Possibly only the code.
Search for "GUIPropertyManager.SetProperty" here:
https://github.com/MediaPortal/Medi...e3/TVLibrary/TvPlugin/TvPlugin/TvGuideBase.cs

Who would know how the .dtd file is used?

Me. What exactly do you want to know?

 

[RonD]

Who would know how the .dtd file is used?

to add a little to mm comment, if you google "dtd file" there are lots of webpages with info, but it is a text file, you can open with notepad or another editor. At a high level a dtd file defines what a valid "xmltv.xml" file looks like. For some dtd info see
http://www.w3schools.com/xml/xml_dtd_intro.asp
https://en.wikipedia.org/wiki/Document_type_definition

The problem with doing this is that EPGC and TV Server have no visibility over when each is or is not using tuners. It is better for a single application to have absolute control over a tuner... or not control it at all. That's why I "reimplemented" ATSC EPG support - along with various other formats that EPGC supports - inside TVE 3.5

The sharing/collision problem is something I had when I experimented with EPGC. I ended up faking sharing using taskmanager. Since almost all USA shows start at HR:00 or HR:30, I made sure taskmanager only started EPGC at HR:05, HR:35 so MP recordings had 1st chance to grab the tuner and EPGC would lose access if all tuners were busy. I think EPGC needs 60 sec (maybe 30??) per channel, so I had to make sure it was done before next 00/30 boundary.

I only used/played with EPGC for a few weeks, I had weird hybrid setup going, WebEPG/imdb for the 10 channels I used for recordings, and EPGC/xmltv to add basic EPG data for all the "other channels".

 

[Pat Clark]

if you google "dtd file" there are lots of webpages with info, but it is a text file, you can open with notepad or another editor.

I had done that, but it wasn't what I thought I needed.

Possibly only the code.
Search for "GUIPropertyManager.SetProperty" here:
https://github.com/MediaPortal/Medi...e3/TVLibrary/TvPlugin/TvPlugin/TvGuideBase.cs

It appears some interesting items are defined but not populated. An image is not among them. (Thumb is populated with logo.)

Me. What exactly do you want to know?

I wasn't sure whether items I might want to forward to MP had to be defined there somehow. But it doesn't much matter, since the code above wouldn't handle them anyway, from what I can tell, so they can't get out to a skin.

 

[mm1352000]

It appears some interesting items are defined but not populated. An image is not among them.

Correct, no program image support. I'd speculate that's for a number of reasons:

• no support in broadcast EPG formats
• space in the guide screen/layout at a premium; more channel "timelines" generally preferred
  
• historically, low internet data caps => users don't want a ton of images downloaded
  
• historically, no/few good metadata databases available

I wasn't sure whether items I might want to forward to MP had to be defined there somehow. But it doesn't much matter, since the code above wouldn't handle them anyway, from what I can tell, so they can't get out to a skin.

There are a series of limitations that have to be overcome in order to have something displayed in the guide:

• acquisition - Don't underestimate this one; people often seem to assume that [for example] "new" flags, season and episode numbers, artwork etc. are readily available. Fact is they're not in broadcast EPG formats, which have traditionally been the main source for EPG content. That means such details usually have to be pulled in by matching against a metadata database. Frankly that's a pain point for TV Server because of how many EPG sources/plugins we have to co-ordinate and/or remain compatible with (XMLTV, WebEPG, SD, TV Movie ClickFinder, broadcast etc.).
• storage - At minimum this usually means adding a field in TV Server's database. However, that's usually followed by "shot gun surgery" through TV Server's various EPG sources to patch in support for the new content. With images we have a ton of questions due to the existence of client (MP or KODI) + server setups and the range of skins available. Does the data reside on the server? ...in which case, how does it get to the client? ...and how do we handle setups where multiple skins or low-powered clients are used (implying the need for different image styles/sizes? ...or does the data reside on the client, in which case how do we manage/update it?
  
• retrieval - As with storage: not particularly straightforward.
  
• display - Obviously MP has a fair few community skins available (and so does KODI), which is great for user choice. This makes it tricky to manage the compatibility issues and/or community expectations ("OMG, why isn't <new field XYZ> available in <favourite skin name here>!?! Useless @#$%@#% developers!!!") when we want to add, modify or remove skin properties.

So, the situation is "challenging".

 

[Pat Clark]

So, the situation is "challenging".

. . . to say the least. Impossible sounds more like it.

2. The ATSC EPG data provides 12 hours of data for most channels, a few channels provide 1-2 days of data. Since most channels only have 12 hours of data you need to run the grabber 2 or more times a day. See attached EPG_CollectorLog.txt file for more details.

I found that RonD was right. Most channels have just a few hours of data ahead. Some a few days, and some none at all.

I was fooled by the algorithm that XMLTV uses to update. I was worried about that, which is why I had asked how to clear the database out, in another thread. Now, having done so, it's much easier to see how inconsistent the channels are. It just so happened that the ones I checked manually were pretty good, but most are not.

There are 2 here with no data (out of 38) which ARE in zap2it's guide, and one here WITH data that are not in zap2it's. Neither alternative is complete. So I have to ask Ron, how did you manage to combine two sources in order to get the missing channel(s)?

 

[RonD]

or my tests with WebEPG/imdb and EPGC/xmltv. I setup WebEPG to write directly to database for the "core channels". For xmltv, EPGC wrote tvguide.xml and I carefully used the mapping setup to only use EPGC/xmltv for the "other channels" and skip the WebEPG channels. You can also combine 2 or more tvguide.xml files using the tvguide.lst option (disable tvguide.xml). WebEPG can also write a tvguide.xml file. For the tvguide.lst I've done the following:

FullPathToEPGC\tvguide.xml
FullPathToOtherSource\tvguide.xml

You need to be a little careful in xmltv mapping to keep track of which tvguide.xml file is used for each channel. Easy to get confused.

I'm a little surprised some channels do NOT have OTA/EPG. If you use a "real TV", do they at least show basic program name and maybe description if you press the "info" button on the remote. Almost all of my local SF Bay area channels show basic info. Some channels seem to ignore EPG, forget to flip standard/daylight time switches, clock off by 5-10 minutes, or on a few rare cases every program is "DTV-Program" every 30/60 minutes.

Also if you look at the EPGC log files, they did a pretty good job of showing what data they found for each channel. Maybe you have a glitch somewhere in your setup

 

[Edalex]

That's why I "reimplemented" ATSC EPG support - along with various other formats that EPGC supports - inside TVE 3.5.

So we will finally have ultimate EPG grabber, right?