WebEPG suggestions (1 Viewer)

[jameson_uk]

I have started using webEPG since it was incorporated into 1.1 and I have to say it is great and I think this could be a major advantage for MP over front ends.... but I think there is a little bit of work to polish it off so a few suggestions.

First of an easy one... could you adjust the logging so that the actual EPG grabbing side of things is either in a webEPG.log file or the EPG.log file. Trawling through the tv.log trying to find out what is happening with interleaved entries from other procesesses can be a little confusing.

Another small tweak which would give a major boost to usability is tweaking the channel matching. Currently it looks like it is just matching the strings and doing an ok job but on an automatch for 60 channels I think about 25 were wrongly allocated. A lot of these problems came down to either numbers in channel names (ie. the channel name in MP being BBC3 but the channel name in channels.xml being BBC Three and vice versa) or the +1 channels; most of the UK channels are called x +1 but channels.xml is storing these as x plus 1. If when doing the matching we could expand out numbers and the plus sign (guess it would be easiest to do it on both sides as I have seen the problems going both ways) then this would make automatching a lot more accurate and a lot more usable. I guess if you have BBC3 in MP and BBC THREE in channels.xml it would be simple enough to replace the numbers from 1-9 in both sides with the worse and then compare? Same with the timeshift channels. If we replace all occurrences of + with PLUS before matching this would auto match far more channels.

Then the bigger one it seems that the UK grabbers and channels.xml files are very out of date. This in itself is not an issue as the files can be updates. The issue is that I tried to do this today (and have been doing so for the last couple of days). There is so much data to enter and match up that it is very time consuming to update. Yes someone could update the files (I have posted on the help us EPG forum to start updating the UK stuff) but it will end up out of date again very soon and we will be no further forward.

I know I have read somewhere about moving channels.xml to a database and whilst this would help a little I am not sure it would help solve the issues completely. Lots of these issues are caused by new channels and others being renamed. new channels are easy enough and need new entries but renamed channels can be problematic. In the UK for example a channel was recently changed from five life to FIVER; the dilema here is what to update; we can just update the channel name in channels.xml and leave the identifier as life@five.tv but then this looks confusing as anyone looking at the grabber six or 12 months later will not thing FIVER is there and end up adding it to the channel list or if a new source comes along someone really has to know that a channel has been renamed in order to match the channel ID with the siteID in the grabber.

Not really sure what the solution is or how this would get addressed given that MP2 is on the horizon. I guess in an ideal world the community would keep the files up to date and there would be automatic updates (which would also adjust the webEPG mappings) but that is obvious a lot of work and probably something for MP2...

I think this is great but I am very wary that if other users come along and only manage to get 30 of their 60 channels mapped and then can not figure out how to map the others if the channels do not exist etc this will just turn them off this great idea

 

[arion_p]

First of an easy one... could you adjust the logging so that the actual EPG grabbing side of things is either in a webEPG.log file or the EPG.log file. Trawling through the tv.log trying to find out what is happening with interleaved entries from other procesesses can be a little confusing.

There has been a lot of talk about whether there should be separate logs or only one. Also TVServer logging currently has no log-levels and generally needs some rework. There has been some work done but this part cannot be touched until after 1.1.0 is out.

Another small tweak which would give a major boost to usability is tweaking the channel matching. Currently it looks like it is just matching the strings and doing an ok job but on an automatch for 60 channels I think about 25 were wrongly allocated. A lot of these problems came down to either numbers in channel names (ie. the channel name in MP being BBC3 but the channel name in channels.xml being BBC Three and vice versa) or the +1 channels; most of the UK channels are called x +1 but channels.xml is storing these as x plus 1. If when doing the matching we could expand out numbers and the plus sign (guess it would be easiest to do it on both sides as I have seen the problems going both ways) then this would make automatching a lot more accurate and a lot more usable. I guess if you have BBC3 in MP and BBC THREE in channels.xml it would be simple enough to replace the numbers from 1-9 in both sides with the worse and then compare? Same with the timeshift channels. If we replace all occurrences of + with PLUS before matching this would auto match far more channels.

Expanding numbers and symbols to their verbal representation cannot be done automatically, because you would have to consider every possible language available (e.g BBC1 -> BBC One, but RAI1 -> RAI Uno). It is however possible to allow multiple (alternative) names for each channel-id.

Then the bigger one it seems that the UK grabbers and channels.xml files are very out of date. This in itself is not an issue as the files can be updates. The issue is that I tried to do this today (and have been doing so for the last couple of days). There is so much data to enter and match up that it is very time consuming to update. Yes someone could update the files (I have posted on the help us EPG forum to start updating the UK stuff) but it will end up out of date again very soon and we will be no further forward.

I am afraid this is a tough one. It seems as if there is an endless battle between the epg providers (web sites) and grabber developers. Providers constantly change their site layout, making it increasingly more difficult to grab epg data from their site and grabber developers try to keep up. Sometimes I think these changes are made to intentionally prevent programs like WebEPG from using their site. Otherwise they could easily provide epg in some xml format (xmltv is pretty much standard) and everyone would be happy.

I know I have read somewhere about moving channels.xml to a database and whilst this would help a little I am not sure it would help solve the issues completely. Lots of these issues are caused by new channels and others being renamed. new channels are easy enough and need new entries but renamed channels can be problematic. In the UK for example a channel was recently changed from five life to FIVER; the dilema here is what to update; we can just update the channel name in channels.xml and leave the identifier as life@five.tv but then this looks confusing as anyone looking at the grabber six or 12 months later will not thing FIVER is there and end up adding it to the channel list or if a new source comes along someone really has to know that a channel has been renamed in order to match the channel ID with the siteID in the grabber.

Alternative names could solve this problem too. We could also allow alternative channel-ids for the same channel, where one of them is the primary channel-id and the rest are obsolete ids. The primary channel-id will be used to identify the channel in the grabber and when a new mapping is created. Obsolete channel-ids will offer backward compatibility so that if the primary channel-id changes the user does not need to remap WebEPG channels - the mapping to the obsolete channel-id will keep the configuration working. Also I would like to be able to comment each entry, so any changes can be documented (xml comments are fine for that).

Not really sure what the solution is or how this would get addressed given that MP2 is on the horizon. I guess in an ideal world the community would keep the files up to date and there would be automatic updates (which would also adjust the webEPG mappings) but that is obvious a lot of work and probably something for MP2...

There has been a talk about having an online repository of grabbers and WebEPG would check (periodically or on request) the repository for updates and new grabbers, much like what happens with movie info grabbers (IMDB etc.).

I have to say that all of these are good points, and I would like to see them implemented, but as these are all new features it won't happen within 1.1.0 time-frame.

Arion

 

[jameson_uk]

Expanding numbers and symbols to their verbal representation cannot be done automatically, because you would have to consider every possible language available (e.g BBC1 -> BBC One, but RAI1 -> RAI Uno). It is however possible to allow multiple (alternative) names for each channel-id.

but as we know the location of the grabber could this data not be used? In reality I think we are only talking about auto matching so it would be possible to store an xml file in each grabber location folder something like

<automatch location="GB">
  <replace what="1">ONE</replace>
  <replace what="2">TWO</replace>
  ..
  <replace what="6">Six</replace>
  <replace what="+">Plus</replace>
</automatch>

As you set the location on the front page of the webEPG config we know the location already so it would just be a case of reading the relevant xml file and doing the replacements.

Remember that if you replace on both sides it won't really make a difference if you use the wrong language. eg. using an English conversion would just change RAI1 =>RAIONE and if the grabber was setup with RAI1 already that would also be replaced with RAIONE so would still match. If the channel name was stored as RAI1 in the database and RAI UNO in the grabber then it would not match anyway so changing to RAI ONE would not really make a difference ? (and vice versa ?)

This would just need an xml file creating for each location (I guess six is high enough for channel numbers but might be just as well to go to 9?)

Just an idea that is fairly simple but I think would improve auto matching a great deal.

I am afraid this is a tough one. It seems as if there is an endless battle between the epg providers (web sites) and grabber developers. Providers constantly change their site layout, making it increasingly more difficult to grab epg data from their site and grabber developers try to keep up. Sometimes I think these changes are made to intentionally prevent programs like WebEPG from using their site. Otherwise they could easily provide epg in some xml format (xmltv is pretty much standard) and everyone would be happy.

Agreed, in the UK RadioTimes were being very funny about this for ages. Now they actually provide xmltv data which is nice I think the fact there is this fluidity just adds to the argument for making the system flexible enough to make these updates simple to handle. I think what was more frustrating is that the xml files numbers the channels and stores a total. This means that someone can not just copy and paste a few new channels into their channels.xml file as if they have added one themselves they would need to renumber everything and remember to go back and change the total number at the top.

Thinking about it there is a much simpler solution... If the list of channels was stored on the MP site then the config app could get the latest list of channels and give users this list to create their own channels.xml file only containing the channels they are interested in. This solution could even be extended to include a provider so users could select say "Freeview" (free DVB-T) or "Freesat" (Free DVB-S) or "Sky" (paid DVB-S) in the UK and have their channel list populated automatically. This would solve many of the issues as the channel IDs could be controlled in one place?

Just throwing out thoughts here, have not really thought this through...

There has been a talk about having an online repository of grabbers and WebEPG would check (periodically or on request) the repository for updates and new grabbers, much like what happens with movie info grabbers (IMDB etc.).

This would be ideal. The only difference I can see is that there is a webEPG.xml file storing the config. If something changes in the file these changes would need to be propogated to the webepg.xml file?

I have to say that all of these are good points, and I would like to see them implemented, but as these are all new features it won't happen within 1.1.0 time-frame.

I agree no point trying to do something for 1.1 when there are other issues outstanding. I do think though that a few improvements before MP2 would really help.

 

[yourmumquake]

Like jameson_uk am pleased (great work!) to see WebEPG incorporated to TV Server and over the last few weeks have made an effort to improve automating channel updates to RadioTimes and MyDigiguide with this prog:
https://forum.team-mediaportal.com/webepg-136/radiotimes-mydigiguide-xml-updater-75005/

Whilst taking a closer look at grabbers and seeing this thread I would like to add a suggestions to jameson_uk above list:

Incorporate both DVB and Web EPG grabbers to a single tab in TV Server config – so that a single ‘EPG’ tab opens say three sub selections: DVB (TV), DVB (Radio) and Web grabbers.
Further, make some of the functionality common amongst all grabbers, e.g. the DVB EPG ‘General’, ‘grabbing whilst timeshifting/recording or idle’ and ‘display’ options/selections work for both DVB and Web grabbers.

Also, allow GUI (for WebEPG and DVB EPG) to select all DB columns as some are in DVB EPG.

Finally in reference to auto mapping channels, I agree this functionality is begging improvements and have spent time thinking about it during development of the above RadioTimes_MyDigiGuide.exe but ran out of time to actually dive into generating code but my thinking was along the lines of exporting all the channel details (using Export in TV Server config) and updating the RadioTimes, MyDigiGuide and Channels xml in reference to the exported channel list (<channelgroup GroupName="All Channels" SortOrder="9999">). Obviously requires a lot of work to make sure accuracy and the amount of code would reflect the complex nature of achieving this accuracy. But you would end up with xml files that would guarantee correct hits in auto-mapping. I know there are very small discrepancies/inconsistencies across each channel provider and across each web site including additional spaces or capitalisation, let alone numbering, that makes generating code very involved and time consuming...