[Approved] [WebEPG] fixes for delay bugs (1 Viewer)

[benjerry]

Hi,

This patch fixes 2 bugs in WebEPG:

1) Delay for grabbing a sublinked page was always 0
The sublink delay is now properly equal to the one specified in Site tag or if a Link tag specified it has that value.

Minimum delays of 500ms are affecting grabspeed.

Default delays:

Before this fix it was like this:
Program list grab: delay 500ms minimum.
Sublink grab: delay 0ms. (bug)

With this fix:
Program list grab: delay 500ms minimum.
Sublink grab: delay 500ms (copied value)

It's still possible to make it behave like before this fix, by specifying a valid Link tag with delay="0" or delay="" (or just leave out). I've removed the minimum setting there.

2) Delay for grabbing a webpage using the internal method was always done twice - causing double delay resulting by using defaults in 1000ms.

gr,
Gijs

 

[arion_p]

Sorry, I have not responded earlier as I was busy with work and RC5.

The patch looks good to me, but have not yet tested it. Unfortunately at this stage only show-stopper will be fixed for 1.1.0. So this will be scheduled for 1.2.0 (or 1.1.1 if there is one after all).

Just to make sure: existing grabbers that do not specify any delay will still behave as before (min delay 500ms for main requests, no delay for sublinks) right?

I will be away for the next 2 weeks. When I get back I will bring the patch to the attention of the team to get it approved and add it to mantis.

 

[benjerry]

Sorry, I have not responded earlier as I was busy with work and RC5.

The patch looks good to me, but have not yet tested it. Unfortunately at this stage only show-stopper will be fixed for 1.1.0. So this will be scheduled for 1.2.0 (or 1.1.1 if there is one after all).

Just to make sure: existing grabbers that do not specify any delay will still behave as before (min delay 500ms for main requests, no delay for sublinks) right?.

No, with this patch the delay from the site/listing request (which is 500 minimum) will be copied along with the host, encoding, etc. when no link tag is existing.

If delay should be 0ms by default, making it like before, then an extra codeline has to be added wich explicit sets it 0.

see here (WebParser.cs) added the new line:

    public int ParseUrl(HTTPRequest site)
    {
      if (_sublinkParser != null && _sublinkRequest == null)
      {
        _sublinkRequest = new HTTPRequest(site);  // copies site/listing request information to sublink request
        _sublinkRequest.Delay = 0;  // sets default delay to 0ms
      }
      return _listingParser.ParseUrl(site);
    }

gr,
Gijs

 

[kkendall]

I've added the file 3-tvgids_upc_nl.xml to the grabber folder and I'm using this to grab epg info for about 17 channels that do not have dvb-epg data. But some of these channels still stay empty in the tvguide... for example 'E! Entertainment'. Do I need this patch to make webepg work for these channels? And if so, how do I use this patch?

 

[benjerry]

I've added the file 3-tvgids_upc_nl.xml to the grabber folder and I'm using this to grab epg info for about 17 channels that do not have dvb-epg data. But some of these channels still stay empty in the tvguide... for example 'E! Entertainment'. Do I need this patch to make webepg work for these channels? And if so, how do I use this patch?

Can you be more specific in which channels fail?

I've tested E!Entertainment as first grabbed channel and I received program data, except for descriptions at the end of the list because of the delay problem.

The patch will solve the issues, but you'll have to compile the webepg plugin and another module.

Except from this you might try to raise the grab delay of the programlist page.

find in the grabber file this line:

<Site url="http://tvgids.upc.nl/TV/Guide/Channel/[ID]/[DAY_NAME]/" />

and add delay tag:

<Site url="http://tvgids.upc.nl/TV/Guide/Channel/[ID]/[DAY_NAME]/" delay="1000" />

The delay will reduce chance on failed page requests, but will slow down grabbing. Check error.log for failures and try adjusting delay for optimum result.

edit: I shall make a patched webepg available.

 

[benjerry]

Hi,

Here are my personal build files with fixes for WebEPG based on final version 1.1

It fixes delays and adds a new optional array WeekDayNames (see https://forum.team-mediaportal.com/submit-code-patches-mediaportal-tv-server-etc-325/alt-fix-mantis-0002629-allow-weekday-names-configured-grabber-file-84413/)

Backup existing files before replacing them.

gr,
Gijs

 

[kkendall]

Except from this you might try to raise the grab delay of the programlist page.

find in the grabber file this line:

<Site url="http://tvgids.upc.nl/TV/Guide/Channel/[ID]/[DAY_NAME]/" />

and add delay tag:

<Site url="http://tvgids.upc.nl/TV/Guide/Channel/[ID]/[DAY_NAME]/" delay="1000" />

The delay will reduce chance on failed page requests, but will slow down grabbing. Check error.log for failures and try adjusting delay for optimum result.

Yes! This worked! First it still didn't grab all program info, but when I changed the delay from 1000 to 3000, grabbed programs for the 17 channels increased from about 175 to over 500 and all missing-epg-channels were filled
Thanks a lot for your help!

 

[benjerry]

Except from this you might try to raise the grab delay of the programlist page.

find in the grabber file this line:

<Site url="http://tvgids.upc.nl/TV/Guide/Channel/[ID]/[DAY_NAME]/" />

and add delay tag:

<Site url="http://tvgids.upc.nl/TV/Guide/Channel/[ID]/[DAY_NAME]/" delay="1000" />

The delay will reduce chance on failed page requests, but will slow down grabbing. Check error.log for failures and try adjusting delay for optimum result.

Yes! This worked! First it still didn't grab all program info, but when I changed the delay from 1000 to 3000, grabbed programs for the 17 channels increased from about 175 to over 500 and all missing-epg-channels were filled
Thanks a lot for your help!

Good to hear that. In the unmodified webepg you have a 2x3000 = 6 sec delay between programlist page requests.

Could you also try out the files from my last post?

It should solve all missing as well, by spreading out the delays over the sublink requests.
Site and link delay work with 425 in my case.

 

[kkendall]

Sure. Do I have to remove the extra delay in the upc grabber file again to properly test the modified files?

 

[benjerry]

Sure. Do I have to remove the extra delay in the upc grabber file again to properly test the modified files?

Yes, and perhaps change the 375 to 425 in this line:

<Link url="http://tvgids.upc.nl/" delay="375" />

You can try to finetune this amount.