Webepg auto grab problem in MP1.1Beta (1 Viewer)

[clanmackenzi]

Great to see webepg integrated into TVserver configuration in 1.1Beta, however I am having some problems getting nightly program updates to work properly.

Basically when I set it up to grab about 100 channels of data from UK radiotimes for 14 days - it works when I do a manual grab. However, if I save and shutdown configuration after setting the schedule to grab every night at 4am, I get lost channels and limited data in the guide the following morning.

I have played around with this for a while - if I save and reopen the settings in general tab are lost - grab days default to 2 days, automap country defaults to US and all schedule settings are gone. If I setup again, do a manual grab - all OK but if I leave TVserver configuration open all night the auto schedule runs but again lost channel data results.

So it would appear to me that I have a problem with file locations - my configuration updates are not being restored and the auto grab is not using the correct parameters. I am trying this on two system, one Vist64 and the other Vista32 - thought it might be 64-bit file structure problem (program files(x86)) but both systems are the same.

This may be a simple configuration setting (file locations) that I am missing, as I cannot find any other threads that mention this problem.

Any help greatly appreciated.

Thanks

 

[clanmackenzi]

Update - just in case someone is interested...

Auto grab at 4am although the configuration tabs suggest that no schedule is configured.

98 channels imported
33,559 programs imported

However channels missing - BBC1, 3 and 4; ITVHD, 2, 3 and 4; Channel 4 and E4

Did a manual scan - 98 channels imported
34,972 programs imported

All channels back - manual grab found almost an additional 1500 programs

Clearly the auto grab is not using the same files as the manual grab and the configuration is not reading the current configuration file. However, I do not know what these files are called or where they reside.

Maybe I need to use XMLTV as I used in MP1.02. I see others are creating script files to update guide data - seems like a backward step to me. Without solid guide data the whole TV functionality of MP is completely useless.

I'll head of to another part of the forum to try and find some help

 

[winterescape]

something to check is how long it takes to update and what the timeout is for powerscheduler, make sure the system is not shutting down in the middle of updating

Just a thought as I use MP 1.0.2 and zap2xml with the XML plugin

 

[clanmackenzi]

Not sure what the power scheduler timeout is - I do not shutdown the TVserver at all. Maybe not great for global warming but solved many problems in the early days when coming out of sleep mode. Not sure this is what you mean - I will see what I can find though.

Thinking that the best possible solution in the short term may be to use the old standalone webepg progs that worked in MP1.0.2 to create a scheduled xmltv file and then use the xmltv section in plugins to update the TVserver database instead.

 

[winterescape]

Not sure what the power scheduler timeout is - I do not shutdown the TVserver at all. Maybe not great for global warming but solved many problems in the early days when coming out of sleep mode. Not sure this is what you mean - I will see what I can find though. .

that is what I meant, never mind if the pc is on all the time...

 

[yourmumquake]

clanmackenzi, no reply on your bug report?

I mentioned in another thread about programmes disappearing from 'Films' after a 'manual' 'grab' using WebEPG tool but more programmes appearing in 'Movies'.

I don't think this is just happening with Films but quite possibly other genre.

I believe what is happening after the manual grab (and possibly automatically if set in Power Scheduler), the DVB EPG grabber updates the genre, especially if the 'Always try to updating the existing entries' is selected, and possibly has a slightly different genre showing. For instance I just looked at a film on Film4 (The Land Girls) and it did not appear in the Film genre but was in the Movies, which is strange as the WebEPG grabber is using RadioTimes for Film4 and the film is set to 'Film' (and there is only one genre available in RadioTimes). After doing another grab using WebEPG it moved back from Movies to Film.

Checking the film details on uk-tv-guide.com, it shows the genres to be: Keywords: Drama, Based-on-Novel, WWII, WW2, World War, World War 2, War, Historical, History, Film, Movie, Movies, Films

So you can see it is quite likely DVB EPG could be set to anyone of a possible ten or more genres.... therefore not matching the WebEPG genre.

Not sure if this is the same issue as you noted or raised a bug against. If not, I wonder if this issue is worth raising on a separate post?

 

[arion_p]

I have played around with this for a while - if I save and reopen the settings in general tab are lost - grab days default to 2 days, automap country defaults to US and all schedule settings are gone. If I setup again, do a manual grab - all OK but if I leave TVserver configuration open all night the auto schedule runs but again lost channel data results.

There have been several fixes in WebEPG and supporting code since 1.1.0 beta. The above issue is described in 0002502: WebEPG config section fails to read all settings when radio channel mappings exist - MediaPortal Bugtracker, and has been resolved.
Other issues that have been resolved:
0002559: System may sleep while programs are being inserted to the database after an EPG grabber has finished - MediaPortal Bugtracker
0002518: WebEPG only grabs the last part of a merged channel on the second and subsequent grabs - MediaPortal Bugtracker
0002456: Auto Map button in General page of WebEPG plugin does nothing. - MediaPortal Bugtracker
0002455: Activating WebEPG plugin logs NullRef exception - MediaPortal Bugtracker
0002457: WebEPG merged channels don't get grabbed correctly or at all - MediaPortal Bugtracker
0002483: Program end time in merged channels is sometimes wrong - MediaPortal Bugtracker

Regarding your other issue it may be related to the time the automatic grab happens. Try scheduling the grab at a different time, e.g. at 10am and see if you get different results. And please provide full logs. Without logs it is nearly impossible to figure out what is wrong.

 

[yourmumquake]

arion_p – I can not find any logs that records what is being collected or updated as part of the DVB EPG grabbing, let me know if any other logs help, thanks.

When using WebEPG and DVB EPG grabbers, each stream (web site for WebEPG and channel provider for DVB EPG) contains conflicting data, especially for genres.

With the DVB EPG set to 'Always try to updating the existing entries', existing entries that were generated using WebEPG change genres. For example, after running WebEPG I have approx. 900 films in the 'film' genre and around 100 films in the 'Movie' genre (these movies maybe from earlier DVB EPG collection). After the scheduled EPG grabber (I set 06:00) this changes to 'Films' reducing to 400 films but 'Movies' increases to around 500 films. I don't see this as a bug per se but would prefer to keep films in the 'Film' genre and continue to use DVB EPG to provide updates to say descriptions (if more descriptive), times if these change, star rating, date made etc. and only change say the genre if one does not exist already.
For now, I have to un-check the DVB EPG's 'Always try to updating the existing entries' and rely on the daily updates from WebEPG only. This will mean if I want to make the information from RadioTimes richer in MP, I will need to look at trying to change the template in xml file. Any other suggestions?

The RadioTimes Film4 channel: http://xmltv.radiotimes.com/xmltv/160.dat has all genres set to ‘Film’.
XML template as follows with ~ set as delimitor: #TITLE~~#SUBTITLE~~~#ACTORS~~~#REPEAT~#SUBTITLES~~~~~~~#GENRE~#DESCRIPTION~~#DATE~#START~#END~

 

[arion_p]

yourmumquake:
If I understand what you are saying correctly, this is normal. MediaPortal does not have preset genres, if DVB EPG marks a program as "Movie" but the site used in the WebEPG grabber marks it as "Film", there is nothing MP can do about it. Since in DVB EPG you have enabled 'Always try to updating the existing entries' it will update genre along with all other fields of each program - you cannot specify which one to update and which not.

One thing that might help though would be to add a seach/replace expression in WebEPG grabber to replace the word "Film" in genre with the word "Movie" like this:

  <Actions>
    <Modify channel="*" field="#GENRE" search="(?<=^|\s)[Ff]ilm(?=\s|$)" action="Replace">Movie</Modify>
  </Actions>

 

[yourmumquake]

Thanks arion_p - picked up on another thread regards < should in fact be &lt;

Is there any reason why WebEPG can not use all the columns available in the db such as STARRATING or originalAirDate etc. - I note some are available for DVB EPG?