[Pending] - WebEPG - Patch for the ability to replace

Discussion in 'Submit: code patches (MediaPortal/TV-Server/etc.)' started by pal.sch, March 22, 2012.

  1. pal.sch

    pal.sch Portal Member

    Joined:
    June 13, 2008
    Messages:
    18
    Likes Received:
    2
    Gender:
    Male
    Ratings:
    +2 / 0
    Home Country:
    Germany Germany
    Show System Specs
    Summary:This patch adds a regex replace function to the html parser (indirect to webepg). The regex replaces will be done on the page source.

    Problem:
    There are situations, where a template don't match to the whole grabbing page because of some inconsistency on this page. The template goes through the page source and does the work till the inconsistency comes. After this template doesn't much anymore and ignores the rest of this page.
    Inconsistency example:
    Show Spoiler
    page source:
    Code (Text):
    1. ...
    2. <div>Time</div><div>ProgramName</div>
    3. <div>Time</div><div>ProgramName</div>
    4. <div>Time</div><div>ProgramName</div><div>Inconsistency</div>
    5. <div>Time</div><div>ProgramName</div>
    template:
    Code (Text):
    1. <div>#TIME</div><div>#NAME</div>
    result:
    Code (Text):
    1. 1. #TIME:Time; #NAME:ProgramName
    2. 2. #TIME:Time; #NAME:ProgramName
    3. 3. #TIME:Time; #NAME:ProgramName
    4. 4. #TIME:Inconsistency; #NAME:Time  ⇒ Error



    Solution:
    A solution for this problem is to replace/remove this inconsistency before parsing the page (send the page to the template parser).
    Since there has been no way for it, I have implemented it myself.

    Solution example:
    Show Spoiler

    page source:
    Code (Text):
    1. ...
    2. <div>Time</div><div>ProgramName</div>
    3. <div>Time</div><div>ProgramName</div>
    4. <div>Time</div><div>ProgramName</div><div>Inconsistency</div>
    5. <div>Time</div><div>ProgramName</div>
    template:
    Code (Text):
    1. <div>#TIME</div><div>#NAME</div>
    replace:
    Code (Text):
    1. <replace match=”<div>Inconsistency</div>” replace=”” />
    page source before parsing:
    Code (Text):
    1. ...
    2. <div>Time</div><div>ProgramName</div>
    3. <div>Time</div><div>ProgramName</div>
    4. <div>Time</div><div>ProgramName</div>
    5. <div>Time</div><div>ProgramName</div>
    result:
    Code (Text):
    1. 1. #TIME:Time; #NAME:ProgramName
    2. 2. #TIME:Time; #NAME:ProgramName
    3. 3. #TIME:Time; #NAME:ProgramName
    4. 4. #TIME:Time; #NAME:ProgramName


    How this works:
    Now you can add the following new tags to your grabber file.
    Code (Text):
    1. <replaces>
    2.     <replace match=”” replace=”” />
    3.     <replace match=”” replace=”” />
    4.     ...
    5. </replaces>

    Legends:
    match: The part you want to replace. (Uses RegExp)
    replace: The text you want to replace with. (no RegExp)

    1. The replace function will do all the replace tasks in the order you write them in your file, from up to down.
    2. This replacements will be done before the template parsing, but after the page was cut (template-start, template-end). So the replacement affects only the important part of the page.
    3. Because of point 2 and because of the actual structure of “htmlparser” and “webepg” code, the replaces-tags must be placed inside the template-tag.

    Code (Text):
    1. <template>
    2.     <replaces>
    3.     …
    4.     </replaces>
    5. </template>
    Example Grabber File:
    (see the attached file)

    Git Pull request:
    URL: git_pull_request

    I hope you would add this code to the svn/git, and we can see that in the next MediaPortal release 1.2.3.



    Edit: Added Binarys for Test (copy them to Team MediaPortal\MediaPortal TV Server directory)
     

    Attached Files:

    • www_vsetv_com.xml
      File size:
      28.1 KB
      Uploaded:
      March 22, 2012
      Views:
      126
    • Binarys.zip
      File size:
      57.7 KB
      Uploaded:
      March 27, 2012
      Views:
      144
    Last edited by a moderator: March 15, 2013
    • Like Like x 2
  2. Google AdSense Guest Advertisement



    to hide all adverts.
  3. pal.sch

    pal.sch Portal Member

    Joined:
    June 13, 2008
    Messages:
    18
    Likes Received:
    2
    Gender:
    Male
    Ratings:
    +2 / 0
    Home Country:
    Germany Germany
    Show System Specs
    Is nobody interested in this feature?

    I think this feature is also good for other epg sites, not only for www.vsetv.com.
    For russian people, the www.vsetv.com site is very important, because it has all the russian channels. Without this patch, you cant parse this site without errors.

    Take a look at this please.

    Bye pal.sch
     
  4. Edalex
    • Premium Supporter

    Edalex Community Plugin Dev

    Joined:
    January 3, 2008
    Messages:
    2,907
    Likes Received:
    713
    Gender:
    Male
    Location:
    Saratov
    Ratings:
    +1,229 / 9
    Home Country:
    Russian Federation Russian Federation
    Show System Specs
    I thought webepg already had replace function which is used in avdanced vsetv grabber many times.
    So in this case regex replace not needed I think. Maybe it is ok in some other grabbers
     

    Attached Files:

    • vsetv_com.zip
      File size:
      7.4 KB
      Uploaded:
      March 31, 2012
      Views:
      157
  5. pal.sch

    pal.sch Portal Member

    Joined:
    June 13, 2008
    Messages:
    18
    Likes Received:
    2
    Gender:
    Male
    Ratings:
    +2 / 0
    Home Country:
    Germany Germany
    Show System Specs
    Ok, you are right. Your vsetv grabber really works, and this grabber is really advanced.

    I just done some tests, and my solution have one important advantage.
    The tests i done were to get only the Titles with both grabber, my and yours. In your grabber i removed all actions for descriptions and other things, and leaved only the actions for the title.
    In this test was my grabber 3 times faster as yours.
    The slow things in your grabber were the Modify Actions for the Title.

    So it works with the current version of webepg, but we can make it better with my solution.
    I would say, that the speed of webepg is very important. Some people doesnt use this plugin because of the speed.
    With my solution, we can improve this disadvantage.
    We remove "bad" strings of the page source, and than parse the source with a light template. This makes it fast and easy to understand.

    bye

    Edit:
    I added to my grabber file all your functionality and then did some speed tests.
    Here are some results:

    Show Spoiler
    My:
    - 4,016230 sec
    - 4,042231 sec
    - 3,924225 sec

    Your:
    - 5,286302 sec
    - 5,332305 sec
    - 5,098291 sec


    On the results you can see, that my grabber is more than 20% faster as yours. And the epg results are the same.

    See the attached parts of my and your grabber files.

    bye
     

    Attached Files:

  6. MaxT
    • Premium Supporter

    MaxT MP Donator

    Joined:
    December 9, 2009
    Messages:
    112
    Likes Received:
    13
    Ratings:
    +13 / 0
    Home Country:
    Russian Federation Russian Federation
    Seems that this patch will significantly ease grabbers writing at least and I would definitely vote for it!
     
    • Like Like x 1
  7. pal.sch

    pal.sch Portal Member

    Joined:
    June 13, 2008
    Messages:
    18
    Likes Received:
    2
    Gender:
    Male
    Ratings:
    +2 / 0
    Home Country:
    Germany Germany
    Show System Specs
    Hello @ all,
    it's over 6 month since my post and nothing changes.
    Why nobody looked at my patch that increase the speed and adds new functionality to WebEPG?

    I wanted to help this cool project and uploaded this patch, but nobody is interested in it.
    It really disappointing if you try to help and your help will be ignored!

    Maybe nobody of the developer need this patch, but i am sure there are many people that would be very happy with this.
     
    Last edited: September 11, 2012
  8. Paranoid Delusion
    • Premium Supporter

    Paranoid Delusion Moderation Manager

    Joined:
    June 13, 2005
    Messages:
    13,062
    Likes Received:
    1,481
    Gender:
    Male
    Occupation:
    Electrical Engineer
    Location:
    Cheshire
    Ratings:
    +1,482 / 0
    Home Country:
    United Kingdom United Kingdom
    Show System Specs
    Hi pal.sch

    Will try and get you a definitive answer, over 6 months is unacceptable as a wait...

    @arion_p
     
    • Like Like x 1
  9. pal.sch

    pal.sch Portal Member

    Joined:
    June 13, 2008
    Messages:
    18
    Likes Received:
    2
    Gender:
    Male
    Ratings:
    +2 / 0
    Home Country:
    Germany Germany
    Show System Specs
    Hello again,
    i don't know how the patch-way works here, but this process discourages other developers from help this project, sorry.
     
  10. Vasilich
    • Team MediaPortal

    Vasilich Test Group

    Joined:
    August 30, 2009
    Messages:
    3,392
    Likes Received:
    608
    Gender:
    Male
    Occupation:
    Developer
    Location:
    Germany, Mayence
    Ratings:
    +1,173 / 2
    Home Country:
    Russian Federation Russian Federation
    Show System Specs
    @arion_p have you had a look on this?
     
  11. Vasilich
    • Team MediaPortal

    Vasilich Test Group

    Joined:
    August 30, 2009
    Messages:
    3,392
    Likes Received:
    608
    Gender:
    Male
    Occupation:
    Developer
    Location:
    Germany, Mayence
    Ratings:
    +1,173 / 2
    Home Country:
    Russian Federation Russian Federation
    Show System Specs
    can it be considered for including in 1.3.1 or in 1.4 at least? arion_p isn't active on forum since mid 2012
     
Loading...

Users Viewing Thread (Users: 0, Guests: 0)

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice
  • About The Project

    The vision of the MediaPortal project is to create a free open source media centre application, which supports all advanced media centre functions, and is accessible to all Windows users.

    In reaching this goal we are working every day to make sure our software is one of the best.

             

  • Support MediaPortal!

    The team works very hard to make sure the community is running the best HTPC-software. We give away MediaPortal for free but hosting and software is not for us.

    Care to support our work with a few bucks? We'd really appreciate it!