[Pending] WebEPG - Patch for the ability to replace (1 Viewer)

pal.sch

Portal Member
June 13, 2008
18
2
Germany Germany
Summary:This patch adds a regex replace function to the html parser (indirect to webepg). The regex replaces will be done on the page source.

Problem:
There are situations, where a template don't match to the whole grabbing page because of some inconsistency on this page. The template goes through the page source and does the work till the inconsistency comes. After this template doesn't much anymore and ignores the rest of this page.
Inconsistency example:
page source:
Code:
...
<div>Time</div><div>ProgramName</div>
<div>Time</div><div>ProgramName</div>
<div>Time</div><div>ProgramName</div><div>Inconsistency</div>
<div>Time</div><div>ProgramName</div>
…
template:
Code:
<div>#TIME</div><div>#NAME</div>
result:
Code:
1. #TIME:Time; #NAME:ProgramName
2. #TIME:Time; #NAME:ProgramName
3. #TIME:Time; #NAME:ProgramName
4. #TIME:Inconsistency; #NAME:Time  ⇒ Error


Solution:
A solution for this problem is to replace/remove this inconsistency before parsing the page (send the page to the template parser).
Since there has been no way for it, I have implemented it myself.

Solution example:
page source:
Code:
...
<div>Time</div><div>ProgramName</div>
<div>Time</div><div>ProgramName</div>
<div>Time</div><div>ProgramName</div><div>Inconsistency</div>
<div>Time</div><div>ProgramName</div>
…
template:
Code:
<div>#TIME</div><div>#NAME</div>
replace:
Code:
<replace match=”<div>Inconsistency</div>” replace=”” />
page source before parsing:
Code:
...
<div>Time</div><div>ProgramName</div>
<div>Time</div><div>ProgramName</div>
<div>Time</div><div>ProgramName</div>
<div>Time</div><div>ProgramName</div>
…
result:
Code:
1. #TIME:Time; #NAME:ProgramName
2. #TIME:Time; #NAME:ProgramName
3. #TIME:Time; #NAME:ProgramName
4. #TIME:Time; #NAME:ProgramName

How this works:
Now you can add the following new tags to your grabber file.
Code:
<replaces>
    <replace match=”” replace=”” />
    <replace match=”” replace=”” />
    ...
</replaces>

Legends:
match: The part you want to replace. (Uses RegExp)
replace: The text you want to replace with. (no RegExp)

1. The replace function will do all the replace tasks in the order you write them in your file, from up to down.
2. This replacements will be done before the template parsing, but after the page was cut (template-start, template-end). So the replacement affects only the important part of the page.
3. Because of point 2 and because of the actual structure of “htmlparser” and “webepg” code, the replaces-tags must be placed inside the template-tag.

Code:
<template>
    <replaces>
    …
    </replaces>
</template>
Example Grabber File:
(see the attached file)

Git Pull request:
URL: git_pull_request

I hope you would add this code to the svn/git, and we can see that in the next MediaPortal release 1.2.3.

Edit: Added Binarys for Test (copy them to Team MediaPortal\MediaPortal TV Server directory)
 

Attachments

Last edited by a moderator:

pal.sch

Portal Member
June 13, 2008
18
2
Germany Germany
Is nobody interested in this feature?

I think this feature is also good for other epg sites, not only for www.vsetv.com.
For russian people, the www.vsetv.com site is very important, because it has all the russian channels. Without this patch, you cant parse this site without errors.

Take a look at this please.

Bye pal.sch
 

Edalex

Community Plugin Dev
  • Premium Supporter
  • January 3, 2008
    2,922
    1,228
    Saratov
    Russian Federation Russian Federation
    Country flag
    I thought webepg already had replace function which is used in avdanced vsetv grabber many times.
    So in this case regex replace not needed I think. Maybe it is ok in some other grabbers
     

    Attachments

    pal.sch

    Portal Member
    June 13, 2008
    18
    2
    Germany Germany
    Ok, you are right. Your vsetv grabber really works, and this grabber is really advanced.

    I just done some tests, and my solution have one important advantage.
    The tests i done were to get only the Titles with both grabber, my and yours. In your grabber i removed all actions for descriptions and other things, and leaved only the actions for the title.
    In this test was my grabber 3 times faster as yours.
    The slow things in your grabber were the Modify Actions for the Title.

    So it works with the current version of webepg, but we can make it better with my solution.
    I would say, that the speed of webepg is very important. Some people doesnt use this plugin because of the speed.
    With my solution, we can improve this disadvantage.
    We remove "bad" strings of the page source, and than parse the source with a light template. This makes it fast and easy to understand.

    bye

    Edit:
    I added to my grabber file all your functionality and then did some speed tests.
    Here are some results:

    My:
    - 4,016230 sec
    - 4,042231 sec
    - 3,924225 sec

    Your:
    - 5,286302 sec
    - 5,332305 sec
    - 5,098291 sec

    On the results you can see, that my grabber is more than 20% faster as yours. And the epg results are the same.

    See the attached parts of my and your grabber files.

    bye
     

    Attachments

    MaxT

    MP Donator
  • Premium Supporter
  • December 9, 2009
    112
    13
    Russian Federation Russian Federation
    Country flag
    Seems that this patch will significantly ease grabbers writing at least and I would definitely vote for it!
     

    pal.sch

    Portal Member
    June 13, 2008
    18
    2
    Germany Germany
    Hello @ all,
    it's over 6 month since my post and nothing changes.
    Why nobody looked at my patch that increase the speed and adds new functionality to WebEPG?

    I wanted to help this cool project and uploaded this patch, but nobody is interested in it.
    It really disappointing if you try to help and your help will be ignored!

    Maybe nobody of the developer need this patch, but i am sure there are many people that would be very happy with this.
     
    Last edited:

    pal.sch

    Portal Member
    June 13, 2008
    18
    2
    Germany Germany
    Hello again,
    i don't know how the patch-way works here, but this process discourages other developers from help this project, sorry.
     

    Users Who Are Viewing This Thread (Users: 0, Guests: 1)

    Top Bottom