WebEPG / Data is grabbed from wrong day!! (1 Viewer)

alexirion

MP Donator
  • Premium Supporter
  • February 8, 2010
    170
    12
    50
    St. Georgen im Schwarzwald
    Home Country
    Germany Germany
    Hi!

    There is an issue with WebEPG. It is grabbing the EPG data from the day before! So at Friday, 01-21 I have the TV programm from Thursday 01-20 !!! The behaviour is 100% reproducible.

    I use WebEPG with this settings:

    Grab Days: 1
    Grab EPG on the following days at this time: 00:00, Monday - Sunday all checked.

    Yesterday I started grabbing manually at 2011-01-20 21:32:41.028320 (tv.bak).
    So far, everything good and the EPG was available until 2011-01-21 05:00:00.

    Before midnight the computer was hibernated.

    It woke up at 23:59 and started EPG grabbing. That's my first question, I thought the "Grab EPG on the following days" option would not wake up the system from hibernate, as the PowerScheduler Option should do in contrast. In my understanding, grabbing with this option should only be done, when the computer starts for any reason and when the time set with this option has expired or when the system is already running at this time.

    Grabbing had started at 2011-01-21 00:00:21.515625. This time the EPG data from the previous day is stored for the new day!
    In the logs you can see, that the EPG data for 01-21 (tv.log) is the same as for 01-20 (tv.log).

    At 2011-01-21 00:58:23.397460 grabbing started again (the computer woke up for a recording). The question: Why did it grab again?? In the Power Scheduler the option "Wakeup system for EPG grabbing" is not checked. Again, the logs show the wrong EPG data from the day before.

    At 2011-01-21 22:37:18.075195 I started grabbing manually. This time the correct data for this day was grabbed.

    Regards,
    --Alex












    MediaPortal Version: 1.1.2
    MediaPortal Skin: Blue3Wide
    Windows Version: Windows 7 64 Bit
    CPU Type: Intel Core2 Duo @ 3GHz
    HDD: WD15EVDS 1,5TB
    Memory: 4GB DDR2-800
    Motherboard: XFX GeForce 9300
    Video Card: ATI 5570
    Video Card Driver: ATI Catalyst™ 10.11 Suite for Windows 7 (64 bit)
    Sound Card:
    Sound Card AC3:
    Sound Card Driver:
    1. TV Card: Mystique SaTiX-S2 Dual V2
    1. TV Card Type: DVB-S2
    1. TV Card Driver: 1.1.0.28
    2. TV Card:
    2. TV Card Type:
    2. TV Card Driver:
    3. TV Card:
    3. TV Card Type:
    3. TV Card Driver:
    4. TV Card:
    4. TV Card Type:
    4. TV Card Driver:
    MPEG2 Video Codec: CyberLink PDVD 10(TV)
    MPEG2 Audio Codec: ffdShow(Video & TV)
    h.264 Video Codec: ffdShow (Video), CyberLink PDVD 10(TV)
    Satelite/CableTV Provider: Astra 19.2
    HTPC Case: HFX classic bundle
    Cooling: BorgFX Heatpipekühler
    Power Supply: Silverstone ST-30NF, 300W
    Remote: iMON VFD
    TV: Panasonic PT-AE4000
    TV - HTPC Connection: HDMI
     

    Attachments

    • tv.zip
      211.2 KB

    arion_p

    Retired Team Member
  • Premium Supporter
  • February 7, 2007
    3,373
    1,626
    Athens
    Home Country
    Greece Greece
    There is an issue with WebEPG. It is grabbing the EPG data from the day before! So at Friday, 01-21 I have the TV programm from Thursday 01-20 !!! The behaviour is 100% reproducible.
    This is a common issue. In all the cases I know of the issue was with the web site itself. Try browsing Fernsehprogramm von TVinfo - sehen was läuft - Ihr TV-Programm just after midnight - does is show correct data or is it shifted by one day (I bet the latter). See also your logs:
    Code:
    2011-01-21 00:00:45.092773 [WebEPGImporter(4)]: WebEPG: Grab Start 00:00 21.01.2011
    2011-01-21 00:00:45.094726 [WebEPGImporter(4)]: WebEPG: Reading http://www.tvinfo.de/exe.php3? POST: target=senderlist.inc&newD=[COLOR=Red][B]21[/B][/COLOR]&newM=[B][COLOR=Red]01[/COLOR][/B]&newY=[B][COLOR=Red]2011[/COLOR][/B]&showSenderID=36
    2011-01-21 00:00:45.360351 [SQL EPG importer(21)]: BusinessLayer: Inserted 40 programs to the database
    2011-01-21 00:00:45.628906 [SQL EPG importer(21)]: BusinessLayer: Inserted 40 programs to the database
    2011-01-21 00:00:46.657226 [WebEPGImporter(4)]: HTMLPage: GetInternal encoding: iso-8859-1
    2011-01-21 00:00:46.659179 [WebEPGImporter(4)]: WebEPG: Listing Count 32
    2011-01-21 00:00:46.663086 [WebEPGImporter(4)]: WebEPG: Guide, Program Info: 20110121053000 - Morgenmagazin
    2011-01-21 00:00:46.663086 [WebEPGImporter(4)]: WebEPG: SubLink Request http://www.tvinfo.de/exe.php3?target=popup&sidnr=126426139&sel=&comefrom=senderlist.inc&backQS=0x00x00x00x00x00x00x00x00x00x00x00x0360x00x00x00x00x00x00x00000000 POST: 
    2011-01-21 00:00:46.967773 [WebEPGImporter(4)]: HTMLPage: GetInternal encoding: iso-8859-1
    2011-01-21 00:00:46.968750 [WebEPGImporter(4)]: WebEPG: Getting sublinked data sucessful
    2011-01-21 00:00:46.971679 [WebEPGImporter(4)]: WebEPG: Guide, Program Info: 20110121090000 - Tagesschau
    The url requests data for 21.01.2011 but apparently the data returned is for 20.01.2011

    I use WebEPG with this settings:

    Grab Days: 1
    Grab EPG on the following days at this time: 00:00, Monday - Sunday all checked.
    ...
    Before midnight the computer was hibernated.

    It woke up at 23:59 and started EPG grabbing. That's my first question, I thought the "Grab EPG on the following days" option would not wake up the system from hibernate, as the PowerScheduler Option should do in contrast. In my understanding, grabbing with this option should only be done, when the computer starts for any reason and when the time set with this option has expired or when the system is already running at this time.
    If powerscheduler is enabled, WebEPG will wake up the system to grab EPG. This is by design and is what most users would expect.

    At 2011-01-21 00:58:23.397460 grabbing started again (the computer woke up for a recording). The question: Why did it grab again?? In the Power Scheduler the option "Wakeup system for EPG grabbing" is not checked.

    At that time WebEPG did not start a new grab. It merely resumed the grab that was interrupted because the system hibernated (at 00:05:27). This probably happened because you did not have "Prevent standby when grabbing EPG".
     

    alexirion

    MP Donator
  • Premium Supporter
  • February 8, 2010
    170
    12
    50
    St. Georgen im Schwarzwald
    Home Country
    Germany Germany
    AW: Re: WebEPG / Data is grabbed from wrong day!!

    Hi!

    Thank you very much arion_p for quick answering.

    Ok, then it should work if I grab later at the day and not in the night. But if grabbing was not finished the last time, and if I start the computer right after midnight, the EPG could be broken, or not? To prevent this, it would be good to have an option to prevent grabbing for a specific time spawn. Did some one already reported this problem to "tvinfo.de"?

    I would find it reasonable to have an option, that the computer does not wake up for just EPG grabbing. This morning it was running for 1h (I think this is a fixed timeout in the grabber?) and did not finish grabbing of 38 channels for 14 days. It would be more than enough, if the computer uses the time in the evening, when I'm watching a movie.

    Regards,
    --Alex






    There is an issue with WebEPG. It is grabbing the EPG data from the day before! So at Friday, 01-21 I have the TV programm from Thursday 01-20 !!! The behaviour is 100% reproducible.
    This is a common issue. In all the cases I know of the issue was with the web site itself. Try browsing Fernsehprogramm von TVinfo - sehen was läuft - Ihr TV-Programm just after midnight - does is show correct data or is it shifted by one day (I bet the latter). See also your logs:
    Code:
    2011-01-21 00:00:45.092773 [WebEPGImporter(4)]: WebEPG: Grab Start 00:00 21.01.2011
    2011-01-21 00:00:45.094726 [WebEPGImporter(4)]: WebEPG: Reading http://www.tvinfo.de/exe.php3? POST: target=senderlist.inc&newD=[COLOR=Red][B]21[/B][/COLOR]&newM=[B][COLOR=Red]01[/COLOR][/B]&newY=[B][COLOR=Red]2011[/COLOR][/B]&showSenderID=36
    2011-01-21 00:00:45.360351 [SQL EPG importer(21)]: BusinessLayer: Inserted 40 programs to the database
    2011-01-21 00:00:45.628906 [SQL EPG importer(21)]: BusinessLayer: Inserted 40 programs to the database
    2011-01-21 00:00:46.657226 [WebEPGImporter(4)]: HTMLPage: GetInternal encoding: iso-8859-1
    2011-01-21 00:00:46.659179 [WebEPGImporter(4)]: WebEPG: Listing Count 32
    2011-01-21 00:00:46.663086 [WebEPGImporter(4)]: WebEPG: Guide, Program Info: 20110121053000 - Morgenmagazin
    2011-01-21 00:00:46.663086 [WebEPGImporter(4)]: WebEPG: SubLink Request http://www.tvinfo.de/exe.php3?target=popup&sidnr=126426139&sel=&comefrom=senderlist.inc&backQS=0x00x00x00x00x00x00x00x00x00x00x00x0360x00x00x00x00x00x00x00000000 POST: 
    2011-01-21 00:00:46.967773 [WebEPGImporter(4)]: HTMLPage: GetInternal encoding: iso-8859-1
    2011-01-21 00:00:46.968750 [WebEPGImporter(4)]: WebEPG: Getting sublinked data sucessful
    2011-01-21 00:00:46.971679 [WebEPGImporter(4)]: WebEPG: Guide, Program Info: 20110121090000 - Tagesschau
    The url requests data for 21.01.2011 but apparently the data returned is for 20.01.2011

    I use WebEPG with this settings:

    Grab Days: 1
    Grab EPG on the following days at this time: 00:00, Monday - Sunday all checked.
    ...
    Before midnight the computer was hibernated.

    It woke up at 23:59 and started EPG grabbing. That's my first question, I thought the "Grab EPG on the following days" option would not wake up the system from hibernate, as the PowerScheduler Option should do in contrast. In my understanding, grabbing with this option should only be done, when the computer starts for any reason and when the time set with this option has expired or when the system is already running at this time.
    If powerscheduler is enabled, WebEPG will wake up the system to grab EPG. This is by design and is what most users would expect.

    At 2011-01-21 00:58:23.397460 grabbing started again (the computer woke up for a recording). The question: Why did it grab again?? In the Power Scheduler the option "Wakeup system for EPG grabbing" is not checked.

    At that time WebEPG did not start a new grab. It merely resumed the grab that was interrupted because the system hibernated (at 00:05:27). This probably happened because you did not have "Prevent standby when grabbing EPG".
     

    arion_p

    Retired Team Member
  • Premium Supporter
  • February 7, 2007
    3,373
    1,626
    Athens
    Home Country
    Greece Greece
    Re: AW: Re: WebEPG / Data is grabbed from wrong day!!

    Ok, then it should work if I grab later at the day and not in the night. But if grabbing was not finished the last time, and if I start the computer right after midnight, the EPG could be broken, or not? To prevent this, it would be good to have an option to prevent grabbing for a specific time spawn.
    Yes this would be welcome addition to help with those sites that do no play nice.

    Did some one already reported this problem to "tvinfo.de"?
    This issue has been reported for several web sites and each time it turned out to be a problem with the site itself, not WebEPG. I cannot remember if tvinfo.de was one of them.

    I would find it reasonable to have an option, that the computer does not wake up for just EPG grabbing. This morning it was running for 1h (I think this is a fixed timeout in the grabber?) and did not finish grabbing of 38 channels for 14 days. It would be more than enough, if the computer uses the time in the evening, when I'm watching a movie.

    Most people prefer EPG to be grabbed at night so it does not interfere with their normal use of the HTPC. You want EPG to be grabbed on a regular schedule, not whenever you happen to be watching a movie or else you might end up with empty TV Guide and scheduled recordings failing (no EPG data = no schedules).

    There is not 1h timeout in the grabber. I don't know why you say it was running for 1h, there is no such indication in your logs. The grabber took 9 min to download ~26MB of data in more than 1200 pages. It all depends on you internet connection speed and the web sites being grabbed. See logs for details:
    2011-01-21 01:03:34.960937 [WebEPGImporter(4)]: WebEPG: Database update finished.
    2011-01-21 01:03:34.960937 [WebEPGImporter(4)]: HTTP Statistics: Site http://www.tvinfo.de : Pages 1252 : Bytes 25806086 : Total Time 00:08:41.4248035 : Av. Rate 49,49148 KBps
    2011-01-21 01:03:34.960937 [WebEPGImporter(4)]: HTTP Statistics: Site http://www.tvtv.at: Pages 1 : Bytes 81247 : Total Time 00:00:02.2607422 : Av. Rate 35,9382 KBps
    2011-01-21 01:03:34.965820 [WebEPGImporter(4)]: plugin:webepg import done
     

    alexirion

    MP Donator
  • Premium Supporter
  • February 8, 2010
    170
    12
    50
    St. Georgen im Schwarzwald
    Home Country
    Germany Germany
    AW: Re: AW: Re: WebEPG / Data is grabbed from wrong day!!

    Hi arion_p,

    the logs I attached were for grabbing epg for only one day. But I want to have the epg for the following seven days, as DVB EPG does. This takes between one and two hours and I don't want the computer to be running each day so long, only to get the EPG one day further. (I think it grabs for all days each day again, also if the data was already downloaded).

    When I once have the EPG for let's say 7 days, the computer would have more than enough time to grab what it can, when I watch a movie. During playback CPU load is <10% and internet is not used for any other big downloads - so this would be the ideal time to keep the PC a bit busy for further getting EPG. Ok, when I don't use the system for some days, I don't have the full data immediately, but as soon as I switch it on, it continues filling up the data again - so this would be ok. This behaviour of WebEPG would be consistent with DVB EPG. It also grabs as much as long as the system is running and I can configure it to not wake up for grabbing.

    So without having an option to not resume for EPG grabbing, the only thing I can do is to disable "prevent standby when EPG grabbing". The system goes on at some time each day, grabs a little portion (normally no new data) and sleeps after 5 minutes (powerschedulers shutdown idle time). Then the real grabbing continues, when I start the computer for watching or when a recording was scheduled. The five minutes of grabbing each day are useless and so I would just need a checkbox to get rid of them. :)

    Once, I checked the option to prevent standby while EPG grabbing. The system woke up for grabbing at the specified time and was hibernating after one hour and I found such a line in the logs:

    plugin:webepg: Telling PowerScheduler standby is allowed: False, timeout is one hour

    I dont't have any timeout with one hour, so I thought the grabber is limited by this fixed amount of time?

    Kind regards,
    --Alex




    Ok, then it should work if I grab later at the day and not in the night. But if grabbing was not finished the last time, and if I start the computer right after midnight, the EPG could be broken, or not? To prevent this, it would be good to have an option to prevent grabbing for a specific time spawn.
    Yes this would be welcome addition to help with those sites that do no play nice.

    Did some one already reported this problem to "tvinfo.de"?
    This issue has been reported for several web sites and each time it turned out to be a problem with the site itself, not WebEPG. I cannot remember if tvinfo.de was one of them.

    I would find it reasonable to have an option, that the computer does not wake up for just EPG grabbing. This morning it was running for 1h (I think this is a fixed timeout in the grabber?) and did not finish grabbing of 38 channels for 14 days. It would be more than enough, if the computer uses the time in the evening, when I'm watching a movie.

    Most people prefer EPG to be grabbed at night so it does not interfere with their normal use of the HTPC. You want EPG to be grabbed on a regular schedule, not whenever you happen to be watching a movie or else you might end up with empty TV Guide and scheduled recordings failing (no EPG data = no schedules).

    There is not 1h timeout in the grabber. I don't know why you say it was running for 1h, there is no such indication in your logs. The grabber took 9 min to download ~26MB of data in more than 1200 pages. It all depends on you internet connection speed and the web sites being grabbed. See logs for details:
    2011-01-21 01:03:34.960937 [WebEPGImporter(4)]: WebEPG: Database update finished.
    2011-01-21 01:03:34.960937 [WebEPGImporter(4)]: HTTP Statistics: Site Fernsehprogramm von TVinfo - sehen was läuft - Ihr TV-Programm : Pages 1252 : Bytes 25806086 : Total Time 00:08:41.4248035 : Av. Rate 49,49148 KBps
    2011-01-21 01:03:34.960937 [WebEPGImporter(4)]: HTTP Statistics: Site tvtv.at &ndash; Alles andere ist nur Fernsehen Pages 1 : Bytes 81247 : Total Time 00:00:02.2607422 : Av. Rate 35,9382 KBps
    2011-01-21 01:03:34.965820 [WebEPGImporter(4)]: plugin:webepg import done
     

    Spooky

    MP Donator
  • Premium Supporter
  • February 14, 2005
    1,187
    47
    void 4tl
    Home Country
    Austria Austria
    The problem is with grabbing data from tvtv.at. All the data from there is from the wrong day. But I don't know if that is a problem of the grabber or the site or interface from which the grabber gets the data from tvtv.at.

    Weirdly enough, when grabbing on certain days, the data will be correct.
     

    Users who are viewing this thread

    Top Bottom