WebEPG trouble creating grabber for US radio stations (1 Viewer)

benicehavefun

Portal Member
September 12, 2014
42
10
Home Country
United States of America United States of America
I'm attempting to create a WebEPG grabber for US radio stations using DAR.fm as a source. I don't appear to be getting any errors during the grabbing process, but no programs are found for my test station. I suspect the problem is with my template -- but trial and error isn't getting me anywhere, so I'm hoping for some input from someone whose been down this road creating a new grabber.

Here's my US\www_dar_fm_com.xml file:

XML:
<?xml version="1.0" encoding="utf-8"?>
<Grabber>
  <Info language="en" availableDays="7" timezone="Pacific Standard Time" version="2.0" />
  <Channels>
    <Channel id="KNOW-FM" siteId="3740" />
  </Channels>
  <Listing type="Html">
    <Site url="http://www.dar.fm/user_search.php?search_type=stations&amp;station_id=[ID]&amp;list_type=station_detail" post="" external="false" encoding="" />
    <Html>
      <Template name="default" start="&apos;s Schedule" end="Schedule a recording">
        <SectionTemplate tags="TIA">
          <TemplateText>  &lt;tr&gt;&lt;td&gt;&lt;img&gt;&lt;/td&gt;&lt;td&gt;&lt;a&gt;&lt;img&gt;&lt;/a&gt;&lt;span&gt;&lt;a&gt;#TITLE&lt;/a&gt;&lt;/span&gt;&lt;/td&gt;&lt;td&gt;#TIME&lt;/td&gt;&lt;/tr&gt;
          </TemplateText>
        </SectionTemplate>
      </Template>
    </Html>
  </Listing>
</Grabber>

Here's a section of HTML source from DAR.fm that contains the program data I'm trying to extract:

CSS:
<tr>
                                                <td width="10%" style="text-align:center;">
                                                <img src='http://s.mp3tunes.com/images/l/p129.gif' style='min-height:30px;max-height:70px;max-width:70px;margin:0 auto;'/>                                                </td><td width="60%">
                                                                                                        <a href="#" title="Record The World Today" class="new_user_talk"  format_id="101" showinfo_id="6428" show_title="The World Today" station_id="3740"  start_time="2017-05-22 23:00:00"><img src="images/record.png" alt="" width="44" height="22" border="0" class="record" /></a><span>
                                                                                                                <a href="#" title="Details" search_t="The World Today" showinfo_id="6428" class="get_details nodec">
                                                                                                                The World Today                                                        </a>                                                        </span>
                                                                                                </td>
                                                <td width='20%'>11:00pm Pacific</td>
<td width='10%'>60 min</td>


                                        </tr>

I'm attempting to extract "The World Today" and "11:00pm" from the above snippet.

Here's the relevant section of my SetupTV.log file:

Code:
[2017-05-22 13:44:09,906] [Log    ] [SetupTv  ] [INFO ] - WebEPG Config: Loading Channels
[2017-05-22 13:44:09,909] [Log    ] [SetupTv  ] [INFO ] - WebEPG Config: Loading Existing channels.xml
[2017-05-22 13:44:10,781] [Log    ] [SetupTv  ] [INFO ] - WebEPG Config: Loading Grabbers
[2017-05-22 13:44:10,927] [Log    ] [SetupTv  ] [INFO ] - WebEPG Config: Loading Existing WebEPG.xml
[2017-05-22 13:44:29,686] [Log    ] [WebEPGImporter] [INFO ] - plugin:webepg importing
[2017-05-22 13:44:29,689] [Log    ] [WebEPGImporter] [INFO ] - WebEPG: Using directory C:\ProgramData\Team MediaPortal\MediaPortal TV Server
[2017-05-22 13:44:29,693] [Log    ] [WebEPGImporter] [INFO ] - Writing to tvguide.xml in C:\ProgramData\Team MediaPortal\MediaPortal TV Server\WebEPG
[2017-05-22 13:44:29,697] [Log    ] [WebEPGImporter] [INFO ] - Assembly versions:
[2017-05-22 13:44:29,701] [Log    ] [WebEPGImporter] [INFO ] - WebEPG 1.15.0.0
[2017-05-22 13:44:29,704] [Log    ] [WebEPGImporter] [INFO ] - TvLibrary.Interfaces 1.15.0.0
[2017-05-22 13:44:29,734] [Log    ] [WebEPGImporter] [INFO ] - Loading Config File: C:\ProgramData\Team MediaPortal\MediaPortal TV Server\WebEPG\WebEPG.xml
[2017-05-22 13:44:29,739] [Log    ] [WebEPGImporter] [INFO ] - WebEPG: Loading Channel Config
[2017-05-22 13:44:29,747] [Log    ] [WebEPGImporter] [INFO ] - WebEPG: Getting Channel ID: KNOW-FM
[2017-05-22 13:44:29,751] [Log    ] [WebEPGImporter] [INFO ] -         [1 of 1]
[2017-05-22 13:44:29,756] [Log    ] [WebEPGImporter] [INFO ] - WebEPG: Opening US\www_dar_fm_com.xml
[2017-05-22 13:44:29,760] [Log    ] [WebEPGImporter] [INFO ] - WebEPG: Version: 2.0
[2017-05-22 13:44:29,763] [Log    ] [WebEPGImporter] [INFO ] - WebEPG: TimeZone, Local: Central Standard Time
[2017-05-22 13:44:29,767] [Log    ] [WebEPGImporter] [INFO ] - WebEPG: TimeZone, Site : Pacific Standard Time
[2017-05-22 13:44:29,790] [Log    ] [WebEPGImporter] [INFO ] - WebEPG: ChannelId: KNOW-FM
[2017-05-22 13:44:30,064] [Log    ] [WebEPGImporter] [INFO ] - WebEPG: Reading http://www.dar.fm/user_search.php?search_type=stations&station_id=3740&list_type=station_detail POST:  - Delay: 500ms
[2017-05-22 13:45:11,575] [Log    ] [WebEPGImporter] [INFO ] - WebEPG: No Listings Found
[2017-05-22 13:45:11,579] [Log    ] [WebEPGImporter] [ERROR] - WebEPG: ChannelId: KNOW-FM grabber error - stopping
[2017-05-22 13:45:11,599] [Log    ] [WebEPGImporter] [INFO ] - WebEPG: Writing Channel: KNOW-FM
[2017-05-22 13:45:11,611] [Log    ] [WebEPGImporter] [INFO ] - HTTP Statistics: Site www.dar.fm : Pages 1 : Bytes 2392391 : Total Time 00:00:41.4713721 : Av. Rate 57.68777 KBps
[2017-05-22 13:45:11,620] [Log    ] [WebEPGImporter] [INFO ] - plugin:webepg import done

As you can see in the log there's one line that says "grabber error - stopping", but that's after "No Listings Found". It appears to be scrapping the page based on the "HTTP Statistics".

Any input would be appreciated!

Scott
 

RonD

Test Group
  • Team MediaPortal
  • December 20, 2011
    911
    278
    SillyValley CA
    Home Country
    United States of America United States of America
    Have you looked at Localize Mediaportal info for WebEPG in the following link. I'd suggest reading through the pages to see what is needed. Getting started with WebEPG has a steep learning curve takes some time and lots of patience.
    Web-EPG - MediaPortal 1 - MediaPortal Wiki

    I notice the download WebEPG Designer file WebEPG-Designer-v5.zip does not link to the zip, I was able to find a copy I downloaded back in 2012. The WebEPG-Designer tool helps you "experiment" with how your grabber decodes/parses the web page data.

    A different option is use WebGrab+Plus to make a tvguide.xml file and use the xmltv plugin to import the file. A different learning curve.
    WebGrab+Plus
     

    Attachments

    • WebEPG-Designer-v5.zip
      37.6 KB

    benicehavefun

    Portal Member
    September 12, 2014
    42
    10
    Home Country
    United States of America United States of America
    RonD -

    You're the man! I'd given up hope of finding the WebEPG Designer -- since the link was dead, and I could find few mentions of it. But that did the trick, I was able to load the page source and move much more quickly experimenting with possible templates. I've hit on one that works for loading one day's schedule for one radio station. Next I'll set my sights on adding more stations and importing the full 7 days that appears to be available.

    For anyone that's interested, here's the grabber for the current day, one station:

    XML:
    <?xml version="1.0" encoding="utf-8"?>
    <Grabber>
      <Info language="en" availableDays="7" timezone="Pacific Standard Time" version="0.1" />
      <Channels>
        <Channel id="KNOW-FM" siteId="3740" />
      </Channels>
      <Listing type="Html">
        <Site url="http://www.dar.fm/user_search.php?search_type=stations&amp;station_id=[ID]&amp;list_type=station_detail" post="" external="false" encoding="" />
        <Html>
          <Template name="default" start="" end="">
            <SectionTemplate tags="AT">
              <TemplateText>
                &lt;tr&gt;&lt;td&gt;&lt;img&gt;&lt;/td&gt;&lt;td&gt;&lt;a&gt;&lt;img&gt;&lt;/a&gt;&lt;a&gt;&lt;#TITLE&gt;&lt;/a&gt;&lt;/span&gt;&lt;/td&gt;&lt;td&gt;&lt;#START&gt; Pacific&lt;/td&gt;&lt;td&gt;&lt;/td&gt;&lt;/tr&gt;
              </TemplateText>
            </SectionTemplate>
          </Template>
        </Html>
      </Listing>
    </Grabber>

    Understanding how the "template tags" combine with the "template text" during parsing was where WebEPG Designer really made a difference for me. Plus, being able to work with the template text in "normal" form rather than "xml" was sweet.

    Cheers,

    Scott
     

    RonD

    Test Group
  • Team MediaPortal
  • December 20, 2011
    911
    278
    SillyValley CA
    Home Country
    United States of America United States of America
    glad you are making progress. 2 more comments

    1. you can set WebEPG to write a tvguide.xml file to a test folder then use a web browser or xml friendly editor to see what WebEPG is doing. When you are done, you can then write to the epg database, but for debug it helps if you can quickly see WebEPG output

    2. if you are writing to a tvguide.xml file, you can use other tags like #ACTORS etc, as scratch locations to see what your grabber is trying to do.
     
    Last edited:

    benicehavefun

    Portal Member
    September 12, 2014
    42
    10
    Home Country
    United States of America United States of America
    So, for anyone that would like EPG data for radio stations in the US, here's my finished grabber. You'll need to customize it much like what's required for the IMDb US TV grabber. Go to DAR.fm to figure out the "siteId" (a 4 digit number) for each station you'd like guide data for, and modify the www_dar_fm_com.xml file according to this Wiki: WebEPG - TVGuide - MediaPortal 1 - MediaPortal Wiki (though there's no zip code requirement). The modified file needs to be placed in the C:\ProgramData\Team MediaPortal\MediaPortal TV Server\WebEPG\grabbers\US on your MePo TV Server. You should find the www_imdb_com.xml file there already.

    Once you've scanned your stations on your tuner card and placed the xml file in the the above directory, you can map your scanned channels to the stations listed in the grabber file using the WebEPG plugin mapping feature for radio. DAR.fm seems to have data for most stations that have actual shows (talk, news, etc) -- not so much for stations that are mostly music. You can add stations and scheduling data to their site though, with a free account, if there's something you want that's missing. With a paid account (a tad expensive), you can can record in the cloud and download the recordings like podcasts. Obviously, you'll be able to do much the same on your HTPC with MediaPortal, a tuner card that does FM radio, and the EPG data you'll get with this grabber!
     

    Attachments

    • www_dar_fm_com.xml
      934 bytes

    Users who are viewing this thread

    Top Bottom