Tweaking DVB EPG (1 Viewer)

[noonereallycares]

I apologise if this has already been discussed.

But....

Is there any way of creating a set of rules to override a poor quality DVB EPG. e.g. The networks in Australia like to spam the EPG with "New" and "Final" (yet never "Repeat") in the title of the shows. This means that it can be hit and miss when it comes to scheduled recordings.

Is there a plugin that exists (short of reverting to an XMLTV guide) that would allow for removal of these useless parts of the EPG? While I am familiar with regular expressions and scripting languages such as PHP and JS, my skill set does not extend far enough to develop a plugin of my own for MP.

 

[mm1352000]

So to be clear: you want to tweak the actual EPG content?
If so, I've never come across any plugin that can do that.

 

[DJBlu]

I think what @noonereallycares is asking for is a way to add words to ignore when the EPG grab happens. So when the title and descriptions are added to the database these ignored words don't appear. I think this would have to be added to the core EPG grabber.

Sky UK have just started to add this annoyance too.

 

[Django.edwards]

Hi noonereallycares, maybe you should take a look at https://forum.team-mediaportal.com/threads/webgrab-plus-a-new-xmltv-grabber.88425/
or over the website www.webgrabplus.com

A great tool to grab epg data from different websites and the possibility to manipulate them... so if you are familiar with regexp....

 

[hurley]

TV Guide spam has been driving me mad here (Australia) too. It's mainly Nine network.
I decided to have a go at writing a plugin to filter the DVB guide data and also fix another issue that's always been a problem where Once recording schedules get broken if the broadcaster changes the start/end time or program title after the schedule has been set. This also tends to un-cancel any cancelled series schedules.

The plugin EPGCleaner uses regex to filter the guide data so it's very configurable but also has a simple mode which uses wildcard matching like * and ? for less complex filtering.

There's an unlimited number of filters for both program title and description text which can be mapped to all channels or selected channels only and the title filter matches can optionally be pre-pended to the description text. Useful for suffixes like New Episode or Premiere etc..

Some other features include:

• A filter tester to check if your regex pattern works as expected.
• Convert upper-case titles to title case. (with ignore list)
• Separate or remove the short description text.

(Some networks send the title or episode name in the short description field which MediaPortal doesn't handle so it gets pre-pended to the description text without a space. This option removes the short description if it matches the title or inserts a separator if it's an episode name or something else.)​

Repair Schedules:​

• Repairs all schedules where titles are changed by EPGCleaner filters (whether Repair Schedules is enabled or not)
• Repairs Once schedules and cancelled series schedules where: Start time, end time or title has changed.
• Removes obsolete cancelled schedules from the database for pre MP 1.10 versions.

After setting up your filters and options you won't see any changes until the guide is next updated. You can speed this process up by clicking Refresh EPG which currently just replicates TV-Server Configuration -> Manual Control -> Refresh DVB EPG but without any feedback. I intend to improve this function for the release version.

I should note that this is an unofficial alpha version so enable logging and keep a close eye on things if testing it.
There's a few tweaks plus language translations and documentation/help to be done yet but most options are tool-tipped.

It would be good to get some field testing feedback as I'm stuck on WinXP and MP 1.3.0 for my HTPC here and have only tested Windows 7 MP 1.10.0 on a borrowed tuner-less laptop.

Oh, for any Aussie users, these settings have been working well for me the past month or so:

Upper Case to Title Case [Enabled] Channels: 7,9,10 networks (Ignore list: M*A*S*H, CSI, NY)
Separate/Remove desc prefix [Enabled] Channels: 7,9,10 networks
Repair Schedules [Enabled]
Enable Logging [Enabled]​

Title Filters:
1. "\(Inc.*\)?" (Replace with: empty) Channels: 9 network (Removes channel nine's spam)
2. " - (New Episode|New|Final|Premiere)" (move to description) Channels: 7,9 networks
3. "Movie: " (move to description) Channels: 10 networks (This ones just for consistency across channels)​

Description Filters:
1. "\(Inc.*?\)" (Replace with: empty) Channels: 7 network​

Don't include the quotes!!! and note leading/trailing spaces.

Removed version 0.1.0.32 of plugin. See post below for update.

 

[SciDoctor]

I will give this a test over the next few days and give some feedback possibly next week.

Thanks in advance for your work as the issues above are plaguing UK TV broadcst EPG.

 

[hurley]

Thanks SciDoctor, much appreciated.
Between build 31 -> 32 I noticed some database errors in my logs, hopefully v33 fixes those. There's a stack of other changes in this update too. (32 was the first version I posted here)

Updated: EPGCleaner Plugin v0.1.0.33 - 2015-05-01

• Switched to bulk insert (single transaction) for database updates
• Improvements to Schedule Repair engine
• Improvements to Logging / Log display section
• Corrected some missed regional date formatting
• Re-enabled obsolete cancelled schedules cleanup (was disabled in 0.1.0.32)
• Various minor changes/improvements

View attachment 164305
Removed version 0.1.0.33 of plugin. See post below for update.

 

[hurley]

Updated: EPGCleaner Plugin v0.1.0.40 - 2015-06-23

• Added sizable Channel Selector dialog with channel groups, filtering and sorting

(Should make things a bit easier for regions with many 100's+ channels)​

• Revised all Schedule Repair engine code and added new program title similarity algorithm

• Schedule Repair now also repairs active recordings where the end time changes during recording

(This only works for once and everytime type series schedules due to the different implementation of time based schedules in MediaPortal and works by offsetting the post rec interval)​

• Added icons and other improvements to schedule repair log

(log from pre 0.1.0.40 versions is no longer compatible)​

• Reverted change to bulk insert of database updates

(Was not really needed since no entries are deleted/added. I have in mind a method for throttling database access in case any one has issues in that area)

Some more to be done yet. I've been really pressed for time lately.
I'm curious how the plugin's working out for anyone who's tried it though.
Any issues or area's that could be improved?
Also would be interested in examples of guide data that need's cleaning in other regions.
I've seen in another thread that user's in Finland have inconsistent classification rating suffix's like (S) or (18) which make scheduling difficult. Those are easy to fix with regex.

* Removed version 0.1.0.40 of plugin. See post below for update.​

 

[Vasilich]

Hi @hurley
just quick-checked your plugin on test install of MP. I cannot do any replacement because i don't know how have you realized it: i thied {$1} $1 and some other combinations that are common for matched string replacement, but none of those works.
On your screenshot i have seen tabs "Regex reference" and "help", but i don't see those at my installation. Is there anything more to install?

 

[hurley]

Hi @Vasilich thanks for your feedback.
I disabled the Help and Regex Guide tabs in this release because I haven't written the documentation yet and so they were just empty placeholders, but I will definitely add them back for official release version.

I think I've had my head buried in the code so long I take for granted how to use it so it's good to remind me that it needs a user guide.

Internally the plugin uses Regex matching to select a part of a program's title to remove or replace.
If your EPG guide data is of good quality, the plugin may not be very useful but if your guide has spam or variable or inconsistent program naming, cleaning those parts can improve scheduling reliability and file naming of recordings.

For example in Australia the worst of the spam we get looks like:

"Friends (Includes: Sneak Peak Reno Rumble at 8:29 PM)"

So I want to match and remove the part starting from "(Includes..blah blah.." [this spam often changes with each EPG update]

The regex pattern which works in this case is: "\(Inc.*\)?"
The backslashes escape the parenthesis because these are special characters in regex.
The .* basically matches any text and the question mark at the end means the closing parenthesis is optional.
(Sometimes the spam is so long it doesn't even fit in the length allowed for a program title)

Some users in Finland have reported that networks append classification ratings to the end of program titles like:

"Emmerdale (7)" or "Emmerdale (S)"

but these classifications can be different for each episode of the same program which makes scheduling difficult.

This regex pattern can remove the rating from titles: "\s\((S|7|12|16|18)\)$"
It means a space \s followed by any of the characters split by | between parenthesis and at the end of the line $.

Regex can be a real pain to work with but it's very powerful.
If you have any problematic guide data like above I recommend googling on regex usage and use the 'Filter tester' to verify that your pattern works.
Enter the offending program title in the text box next to the 'Test' button, create your filter, press test and the result will be shown in the text box below along with a 'Match/Failed' label.

While the plugin configuration is open no changes to filters are saved so you can safely mess around testing what works and what doesn't.
You can also disable any particular filter with the first checkbox.

For simple cases you can switch to 'wildcard' mode using the option: 'Use simple filters instead of regex'

----------------------------------------------------------------------------------------------------------------------------------

As a basic test of the plugins functionality you could pick a series program you're not going to ever record and create a crude filter for it.
For example:

Regex Expression: "Home Shopping" Replace with: "Home Shopping TEST"

Select the channels this show is on or 'ALL' channels and update the EPG for that channel...
You should see the modified program titles in the TV Guide and also entries in the 'Epg Cleaner' log if Logging is enabled.

One other thing I have forgotten to mention. The filters are processed in sequence so it is possible to run 2 or more filters on the same program name if it's too difficult to achieve the result with a single regex pattern.

Here are some regex reference pages I quickly googled:

http://www.zytrax.com/tech/web/regex.htm
https://msdn.microsoft.com/en-us/library/az24scfc(v=vs.110).aspx
http://www.rexegg.com/regex-quickstart.html
https://www.icewarp.com/support/online_help/203030104.htm

Thanks again, hurley