WebEpg development (1 Viewer)

[doskabouter]

Don't know if it's ok to bump this thread (cant exactly call it hijacking ), but I have a proposal for some webepg params.
It's the #START and #END I'm struggling with.
The times I receive are UTC formatted (f.e. 2014-04-07T23:00:00Z) and there's not yet a possibility to parse those.
There's #STARTXMLTV which expects a long, and #START which boils down to HH:MM with minor variations.

I did think of just adding a DateTime.TryParse to the #START code, but then there's now way to just parse UTC times other that specifying the exact format.
If I don't specify the exact format, DateTime.TryParse would also correctly parse f.e. 23:45, with possibly unknown sideeffects (think timezones, offsets etc).

So my proposal is to add #STARTGENERIC and #ENDGENERIC and use a simple dt=datetime.parse and do a worlddatetime.SetFromDateTime(dt), in general let the .NET runtime handle it.

The names (#STARTGENERIC and #ENDGENERIC) are open for debate as I can't think of a proper name right now (see https://alpha.app.net/abraham/post/23590687 )

Any thoughts?

 

[Tony Wall]

So my proposal is to add #STARTGENERIC and #ENDGENERIC and use a simple dt=datetime.parse and do a worlddatetime.SetFromDateTime(dt), in general let the .NET runtime handle it.

The correct way to deal with this data is with DateTimeOffset for storage and TimeZoneInfo for calculation. Especially because this guide data very likely comes from different time zones (at least for satellite users) or will change across yearly offset changes.

Check this article out for a complete explanation. What I'm saying here IS the current recommended "default" by Microsoft:

http://msdn.microsoft.com/en-us/library/bb384267(v=vs.110).aspx

Regarding naming of any new parameter, please not "generic", there is a proper name for this XML format, I think it's "ISO" but double-check MSDN and wikipedia or something first. I'm surprised the guys at XMLTV didn't use the standard format either (their format looks like the "sortable" format without offset).

It's also going to be important to honour the time zone settings in each grabber file, and calculate the stop time so that recordings crossing an offset change really record the whole program, not one hour less or more for example. It also depends on whether the source listings are start+duration or start+stop times, either way it's important to use TimeZoneInfo to calculate the true length/stop-time whatever MediaPortal requires (still haven't programmed that myself yet, but soon...).

As many programmers forget or choose to ignore the necessary complexities of things like globalization and time zones, I wouldn't be surprised if during development some bugs in the TV server scheduler itself were found.

 

[doskabouter]

I think the main thing to decide first if it's going to be just the UTC notation (2014-04-07T23:00:00Z) that will be handled, or anything that .NET's DateTime.Parse will recognize.

 

[Tony Wall]

UTC is fine for ignorant storage of data, when the original time offset is safe to discard and is not useful to anyone. If we don't care about what the original web response was then that's okay. But any update to the whole Media Portal code generally would make sense to switch to DateTimeOffset and review code to make sure no calculation loopholes (i.e. offset changes) were overlooked (TimeZoneInfo used for all calculations).

But I don't see any point trying to avoid the use of the DateTimeOffset and TimeZoneInfo classes when they make things easier and for many good reasons are best to be utilized by default. So at least the source code should be using these classes to help with conversion, even if the time zone is never output to any file or API call.

On the point of files, actually it would make sense to write the offsets and time zones used in calculations into the debug log file. That would help diagnose problems and enable users to verify that their foreign schedules were being imported correctly. This could be especially useful to help diagnose likely issues with misconfiguration of template grabber files (forgetting to change the time zone) or web sites which try to project schedules in an unexpected time zone, e.g. based on the .NET web client/browser language/IP range (seen too many of those, living abroad a long time myself).

 

[wjw]

Hi,

I have just come across this thread - had given up all hope of finding anyone still doing things with WEBepg. Is there any possibility that you might be interested in fixing a problem in the importing of the XMLTV data into the database? It's caused by an error in the UK Radio Times feed which now seems to have become a permanent feature that is not going to be fixed (the BBC have farmed it out). Oddly it only seems to happen at one time in the day.

First, here is an extract from the XMLTV file after grabbing from the website:

<programme start="20140317193000 +0000" stop="20140317193000 +0000" channel="UK_RT_92">
<title lang="en">BBC News; Regional News</title>
<category lang="en">News</category>
<video>
<aspect>16:9</aspect>
</video>
<subtitles type="teletext">
<language lang="en">English</language>
</subtitles>
</programme>
<programme start="20140317193000 +0000" stop="20140317200000 +0000" channel="UK_RT_92">
<title lang="en">Bang Goes the Theory</title>

you can see that the first programme (BBC News) has a finish time that is the same as the start time (19.30). The second programme (which is the one that actually gets broadcast) has the same start time (19.30) and a correct finish time of 20.00.

The two screenshots attached show what actually gets imported, i.e both of them, but the first programme (actually of zero length and not broadcast) takes on the time of the second programme.

Love to get this fixed - simply don't have this kind of skill. I think that it should be possible when examining the data to simply ignore any programme where the start and end times are the same and not write it into the database.

Any chance please? It's really annoying because it means programmes in that time slot can't be recorded.

 

[doskabouter]

UTC is fine for ignorant storage of data, when the original time offset is safe to discard and is not useful to anyone. If we don't care about what the original web response was then that's okay. But any update to the whole Media Portal code generally would make sense to switch to DateTimeOffset and review code to make sure no calculation loopholes (i.e. offset changes) were overlooked (TimeZoneInfo used for all calculations).

But I don't see any point trying to avoid the use of the DateTimeOffset and TimeZoneInfo classes when they make things easier and for many good reasons are best to be utilized by default. So at least the source code should be using these classes to help with conversion, even if the time zone is never output to any file or API call.

On the point of files, actually it would make sense to write the offsets and time zones used in calculations into the debug log file. That would help diagnose problems and enable users to verify that their foreign schedules were being imported correctly. This could be especially useful to help diagnose likely issues with misconfiguration of template grabber files (forgetting to change the time zone) or web sites which try to project schedules in an unexpected time zone, e.g. based on the .NET web client/browser language/IP range (seen too many of those, living abroad a long time myself).

In my understanding, UTC is a standard time where timezones are taken into account when converting to local time, so it would be safe to parse it and do a DateTime.ToLocalTime to convert it, but I'm absolutely no expert on time/date issues, so I could be completely wrong here. Also don't know what the best way is to feed it to the WorldDateTime class.

Regarding all this, I'm leaning to the thought of doing a strict conversion of UTC to DateTime to avoid missing timezones and such. Have to figure if there's a better way than ParseExact with "yyyy-MM-ddTHH:mm:ss.ffK" though.
Would like to hear from other @Developers too...

 

[doskabouter]

Hi,

I have just come across this thread - had given up all hope of finding anyone still doing things with WEBepg. Is there any possibility that you might be interested in fixing a problem in the importing of the XMLTV data into the database? It's caused by an error in the UK Radio Times feed which now seems to have become a permanent feature that is not going to be fixed (the BBC have farmed it out). Oddly it only seems to happen at one time in the day.

First, here is an extract from the XMLTV file after grabbing from the website:

<programme start="20140317193000 +0000" stop="20140317193000 +0000" channel="UK_RT_92">
<title lang="en">BBC News; Regional News</title>
<category lang="en">News</category>
<video>
<aspect>16:9</aspect>
</video>
<subtitles type="teletext">
<language lang="en">English</language>
</subtitles>
</programme>
<programme start="20140317193000 +0000" stop="20140317200000 +0000" channel="UK_RT_92">
<title lang="en">Bang Goes the Theory</title>

you can see that the first programme (BBC News) has a finish time that is the same as the start time (19.30). The second programme (which is the one that actually gets broadcast) has the same start time (19.30) and a correct finish time of 20.00.

The two screenshots attached show what actually gets imported, i.e both of them, but the first programme (actually of zero length and not broadcast) takes on the time of the second programme.

Love to get this fixed - simply don't have this kind of skill. I think that it should be possible when examining the data to simply ignore any programme where the start and end times are the same and not write it into the database.

Any chance please? It's really annoying because it means programmes in that time slot can't be recorded.

Didn't dive into that (yet), mostly doing some small patches into the webepg parser.
Perhaps skip those in the parser, so the importer doesn't freak out on them?

 

[Tony Wall]

In my understanding, UTC is a standard time where timezones are taken into account when converting to local time, so it would be safe to parse it and do a DateTime.ToLocalTime to convert it, but I'm absolutely no expert on time/date issues, so I could be completely wrong here. Also don't know what the best way is to feed it to the WorldDateTime class.

Regarding all this, I'm leaning to the thought of doing a strict conversion of UTC to DateTime to avoid missing timezones and such. Have to figure if there's a better way than ParseExact with "yyyy-MM-ddTHH:mm:ss.ffK" though.
Would like to hear from other @Developers too...

Quite the opposite; UTC is not a time zone but a standard baseline which does NOT take ANY time zone into account, just used as a common reference. It's not only the baseline but also without any daylight saving time applied. It is often confused with GMT, which although similar for part of the year is not the same because GMT has British Summer Time applied. So for the first part GMT is UTC+00:00 then during BST it's UTC+01:00 then back to UTC+00:00 for the rest. If matters could not be worse, there are several other names for each time zone with each offset, e.g. Western European Time = GMT Winter, Western European Summer Time = GMT Summer.

So when one says UTC they are actually saying "baseline AND no time zone AND no offset adjustment/daylight saving". The DateTime structure also has no offset, just a "local" or "UTC" flag (the "kind" property). As you know already, if you have DateTime data from more than one source you need to baseline them all to UTC before doing any conversion, but it's no magic wand... The part you didn't seem to grasp is that you must really use TimeZoneInfo to do calculations adding or removing dates in the same zone, or changing to any other zone, else you will corrupt data across offset changes twice a year.

Now if that wasn't enough good reason to check out the current (not even new now) classes, because DateTimeOffset stores the offset (not a flag) you can skip the whole UTC conversion part and go directly between any time zone. That's why it's simply better. No data is lost, no need for UTC conversion, no problems if you need to convert the data in the future and understand what the user's perspective was, e.g. was it a Monday morning or Sunday night in Dubai when the problem occurred in a error report logged in the UK.

Go read Wikipedia if you want to kill a few hours. Better spend 15 minutes reading the MSDN link I sent you and very simple examples which underline the issue and how easy (actually almost the same) it is to use DateTimeOffset compared to the OLD DateTime structure.

If you really want to continue with just DateTime be careful with your UTC conversion; make sure you construct the DateTime with the Kind parameter specified every time, i.e. set it manually when parsing an exact format which doesn't specify an offset. Default is "unspecified" which usually means local system time (not UTC) but behaviour cannot be guaranteed because of other developers making similar mistakes.

Just trying to help. I've been a professional .NET developer since the first beta (14 years now) and gained loads of experience with XML standards and serialization, data analysis and reports for global users, even written my own full featured scheduler classes (no simple feat when done properly). Anyway good luck with whatever you do with whatever classes! If nobody else manages it I might delve in myself and submit a fix later, or even a new plug-in (I have a better design I've used for a another product which would fit the EPG web site scraping pattern well).

 

[mnmr]

Overall, I'm with Tony on this one, but do have a few remarks:

Working with a DateTime without knowing the associated time zone is much like reading a text file without knowing the encoding (you could get garbage or you could get the correct data, but you wouldn't know which one it was). Therefore, if you are parsing the date from a string, this either needs to include time zone information or you have to obtain the information elsewhere (e.g. configuration option for a particular EPG provider).

If you store the result of the parse operation in a DateTime, you need to know which time zone it is in. If you store the result in a DateTimeOffset, the time zone information will be preserved. I personally prefer to unify everything to UTC, because then you only need to consider time zone conversions when the DateTime "leaves your subsystem", such as when it is stored in a database or passed along to some other component not under your control.

 

[mm1352000]

I note that the TV Server database stores date/time info in the local timezone for the user/PC. Right now all the plugins have to handle timezone stuff themselves, which I don't like. Ideally I'd see us store UTC in the database (implying that plugins would convert any EPG they pull in to UTC) and then have one configuration option which allows people to adjust/set the timezone for display.