Tweaking DVB EPG

Discussion in 'TV-Server Plugins' started by noonereallycares, March 17, 2014.

  1. noonereallycares

    noonereallycares Portal Pro

    Joined:
    August 3, 2005
    Messages:
    101
    Likes Received:
    2
    Location:
    Melbourne
    Ratings:
    +6 / 0
    Home Country:
    I apologise if this has already been discussed.

    But....



    Is there any way of creating a set of rules to override a poor quality DVB EPG. e.g. The networks in Australia like to spam the EPG with "New" and "Final" (yet never "Repeat") in the title of the shows. This means that it can be hit and miss when it comes to scheduled recordings.

    Is there a plugin that exists (short of reverting to an XMLTV guide) that would allow for removal of these useless parts of the EPG? While I am familiar with regular expressions and scripting languages such as PHP and JS, my skill set does not extend far enough to develop a plugin of my own for MP.
     
    • Great Idea! Great Idea! x 1
  2. Google AdSense Guest Advertisement



    to hide all adverts.
  3. mm1352000
    • Team MediaPortal

    mm1352000 Development Group

    Joined:
    September 1, 2008
    Messages:
    21,527
    Likes Received:
    4,738
    Ratings:
    +8,205 / 17
    Home Country:
    New Zealand New Zealand
    So to be clear: you want to tweak the actual EPG content?
    If so, I've never come across any plugin that can do that.
     
  4. DJBlu

    DJBlu Portal Pro

    Joined:
    August 14, 2007
    Messages:
    1,670
    Likes Received:
    789
    Location:
    Llanelli
    Ratings:
    +894 / 1
    Home Country:
    United Kingdom United Kingdom
    I think what @noonereallycares is asking for is a way to add words to ignore when the EPG grab happens. So when the title and descriptions are added to the database these ignored words don't appear. I think this would have to be added to the core EPG grabber.

    Sky UK have just started to add this annoyance too.
     
  5. Django.edwards

    Django.edwards Portal Pro

    Joined:
    October 22, 2009
    Messages:
    457
    Likes Received:
    47
    Gender:
    Male
    Location:
    Eupen
    Ratings:
    +55 / 0
    Home Country:
    Belgium Belgium
    Show System Specs
  6. hurley

    hurley Portal Pro

    Joined:
    February 2, 2015
    Messages:
    61
    Likes Received:
    26
    Ratings:
    +44 / 0
    Home Country:
    Australia Australia
    TV Guide spam has been driving me mad here (Australia) too. It's mainly Nine network.
    I decided to have a go at writing a plugin to filter the DVB guide data and also fix another issue that's always been a problem where Once recording schedules get broken if the broadcaster changes the start/end time or program title after the schedule has been set. This also tends to un-cancel any cancelled series schedules.

    The plugin EPGCleaner uses regex to filter the guide data so it's very configurable but also has a simple mode which uses wildcard matching like * and ? for less complex filtering.

    There's an unlimited number of filters for both program title and description text which can be mapped to all channels or selected channels only and the title filter matches can optionally be pre-pended to the description text. Useful for suffixes like New Episode or Premiere etc..

    Some other features include:

    • A filter tester to check if your regex pattern works as expected.
    • Convert upper-case titles to title case. (with ignore list)
    • Separate or remove the short description text.
    (Some networks send the title or episode name in the short description field which MediaPortal doesn't handle so it gets pre-pended to the description text without a space. This option removes the short description if it matches the title or inserts a separator if it's an episode name or something else.)

    Repair Schedules:
    • Repairs all schedules where titles are changed by EPGCleaner filters (whether Repair Schedules is enabled or not)
    • Repairs Once schedules and cancelled series schedules where: Start time, end time or title has changed.
    • Removes obsolete cancelled schedules from the database for pre MP 1.10 versions.
    After setting up your filters and options you won't see any changes until the guide is next updated. You can speed this process up by clicking Refresh EPG which currently just replicates TV-Server Configuration -> Manual Control -> Refresh DVB EPG but without any feedback. I intend to improve this function for the release version.

    I should note that this is an unofficial alpha version so enable logging and keep a close eye on things if testing it.
    There's a few tweaks plus language translations and documentation/help to be done yet but most options are tool-tipped.

    It would be good to get some field testing feedback as I'm stuck on WinXP and MP 1.3.0 for my HTPC here and have only tested Windows 7 MP 1.10.0 on a borrowed tuner-less laptop.

    Oh, for any Aussie users, these settings have been working well for me the past month or so:

    Upper Case to Title Case [Enabled] Channels: 7,9,10 networks (Ignore list: M*A*S*H, CSI, NY)
    Separate/Remove desc prefix [Enabled] Channels: 7,9,10 networks
    Repair Schedules [Enabled]
    Enable Logging [Enabled]

    Title Filters:
    1. "\(Inc.*\)?" (Replace with: empty) Channels: 9 network (Removes channel nine's spam)

    2. " - (New Episode|New|Final|Premiere)" (move to description) Channels: 7,9 networks
    3. "Movie: " (move to description) Channels: 10 networks (This ones just for consistency across channels)

    Description Filters:
    1. "\(Inc.*?\)" (Replace with: empty) Channels: 7 network

    Don't include the quotes!!! and note leading/trailing spaces.


    [​IMG]

    Removed version 0.1.0.32 of plugin. See post below for update.
     
    Last edited: May 1, 2015
    • Like Like x 5
  7. SciDoctor
    • Premium Supporter

    SciDoctor Retired Team Member

    Joined:
    February 2, 2005
    Messages:
    1,465
    Likes Received:
    128
    Location:
    England
    Ratings:
    +138 / 0
    Show System Specs
    I will give this a test over the next few days and give some feedback possibly next week.

    Thanks in advance for your work as the issues above are plaguing UK TV broadcst EPG.
     
  8. hurley

    hurley Portal Pro

    Joined:
    February 2, 2015
    Messages:
    61
    Likes Received:
    26
    Ratings:
    +44 / 0
    Home Country:
    Australia Australia
    Thanks SciDoctor, much appreciated.
    Between build 31 -> 32 I noticed some database errors in my logs, hopefully v33 fixes those. There's a stack of other changes in this update too. (32 was the first version I posted here)

    Updated: EPGCleaner Plugin v0.1.0.33 - 2015-05-01
    • Switched to bulk insert (single transaction) for database updates
    • Improvements to Schedule Repair engine
    • Improvements to Logging / Log display section
    • Corrected some missed regional date formatting
    • Re-enabled obsolete cancelled schedules cleanup (was disabled in 0.1.0.32)
    • Various minor changes/improvements
    00_EPGCleaner_SettingsMain.png 01_EPGCleaner_LogMain.png View attachment 164305 03_EPGCleaner_LogDebug.png
    Removed version 0.1.0.33 of plugin. See post below for update.
     
    Last edited: June 23, 2015
  9. hurley

    hurley Portal Pro

    Joined:
    February 2, 2015
    Messages:
    61
    Likes Received:
    26
    Ratings:
    +44 / 0
    Home Country:
    Australia Australia
    Updated: EPGCleaner Plugin v0.1.0.40 - 2015-06-23
    • Added sizable Channel Selector dialog with channel groups, filtering and sorting
    (Should make things a bit easier for regions with many 100's+ channels)​
    • Revised all Schedule Repair engine code and added new program title similarity algorithm
    • Schedule Repair now also repairs active recordings where the end time changes during recording
    (This only works for once and everytime type series schedules due to the different implementation of time based schedules in MediaPortal and works by offsetting the post rec interval)​
    • Added icons and other improvements to schedule repair log
    (log from pre 0.1.0.40 versions is no longer compatible)​
    • Reverted change to bulk insert of database updates
    (Was not really needed since no entries are deleted/added. I have in mind a method for throttling database access in case any one has issues in that area)

    Some more to be done yet. I've been really pressed for time lately.
    I'm curious how the plugin's working out for anyone who's tried it though.
    Any issues or area's that could be improved?
    Also would be interested in examples of guide data that need's cleaning in other regions.
    I've seen in another thread that user's in Finland have inconsistent classification rating suffix's like (S) or (18) which make scheduling difficult. Those are easy to fix with regex. :)

    * Removed version 0.1.0.40 of plugin. See post below for update.​
     
    Last edited: June 26, 2015
  10. Vasilich
    • Team MediaPortal

    Vasilich Test Group

    Joined:
    August 30, 2009
    Messages:
    3,392
    Likes Received:
    608
    Gender:
    Male
    Occupation:
    Developer
    Location:
    Germany, Mayence
    Ratings:
    +1,173 / 2
    Home Country:
    Russian Federation Russian Federation
    Show System Specs
    Hi @hurley
    just quick-checked your plugin on test install of MP. I cannot do any replacement because i don't know how have you realized it: i thied {$1} $1 and some other combinations that are common for matched string replacement, but none of those works.
    On your screenshot i have seen tabs "Regex reference" and "help", but i don't see those at my installation. Is there anything more to install?
     
    • Thank You! Thank You! x 1
  11. hurley

    hurley Portal Pro

    Joined:
    February 2, 2015
    Messages:
    61
    Likes Received:
    26
    Ratings:
    +44 / 0
    Home Country:
    Australia Australia
    Hi @Vasilich thanks for your feedback.
    I disabled the Help and Regex Guide tabs in this release because I haven't written the documentation yet and so they were just empty placeholders, but I will definitely add them back for official release version.

    I think I've had my head buried in the code so long I take for granted how to use it so it's good to remind me that it needs a user guide.

    Internally the plugin uses Regex matching to select a part of a program's title to remove or replace.
    If your EPG guide data is of good quality, the plugin may not be very useful but if your guide has spam or variable or inconsistent program naming, cleaning those parts can improve scheduling reliability and file naming of recordings.

    For example in Australia the worst of the spam we get looks like:

    "Friends (Includes: Sneak Peak Reno Rumble at 8:29 PM)"

    So I want to match and remove the part starting from "(Includes..blah blah.." [this spam often changes with each EPG update]

    The regex pattern which works in this case is: "\(Inc.*\)?"
    The backslashes escape the parenthesis because these are special characters in regex.
    The .* basically matches any text and the question mark at the end means the closing parenthesis is optional.
    (Sometimes the spam is so long it doesn't even fit in the length allowed for a program title)

    Some users in Finland have reported that networks append classification ratings to the end of program titles like:

    "Emmerdale (7)" or "Emmerdale (S)"

    but these classifications can be different for each episode of the same program which makes scheduling difficult.

    This regex pattern can remove the rating from titles: "\s\((S|7|12|16|18)\)$"
    It means a space \s followed by any of the characters split by | between parenthesis and at the end of the line $.

    Regex can be a real pain to work with but it's very powerful.
    If you have any problematic guide data like above I recommend googling on regex usage and use the 'Filter tester' to verify that your pattern works.
    Enter the offending program title in the text box next to the 'Test' button, create your filter, press test and the result will be shown in the text box below along with a 'Match/Failed' label.

    While the plugin configuration is open no changes to filters are saved so you can safely mess around testing what works and what doesn't.
    You can also disable any particular filter with the first checkbox.

    For simple cases you can switch to 'wildcard' mode using the option: 'Use simple filters instead of regex'

    ----------------------------------------------------------------------------------------------------------------------------------

    As a basic test of the plugins functionality you could pick a series program you're not going to ever record and create a crude filter for it.
    For example:

    Regex Expression: "Home Shopping" Replace with: "Home Shopping TEST"

    Select the channels this show is on or 'ALL' channels and update the EPG for that channel...
    You should see the modified program titles in the TV Guide and also entries in the 'Epg Cleaner' log if Logging is enabled.

    One other thing I have forgotten to mention. The filters are processed in sequence so it is possible to run 2 or more filters on the same program name if it's too difficult to achieve the result with a single regex pattern.

    Here are some regex reference pages I quickly googled:

    http://www.zytrax.com/tech/web/regex.htm
    https://msdn.microsoft.com/en-us/library/az24scfc(v=vs.110).aspx
    http://www.rexegg.com/regex-quickstart.html
    https://www.icewarp.com/support/online_help/203030104.htm

    Thanks again, hurley :)
     
    Last edited: October 30, 2015
Loading...

Users Viewing Thread (Users: 0, Guests: 0)

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice
  • About The Project

    The vision of the MediaPortal project is to create a free open source media centre application, which supports all advanced media centre functions, and is accessible to all Windows users.

    In reaching this goal we are working every day to make sure our software is one of the best.

             

  • Support MediaPortal!

    The team works very hard to make sure the community is running the best HTPC-software. We give away MediaPortal for free but hosting and software is not for us.

    Care to support our work with a few bucks? We'd really appreciate it!