Shepherd- how to set up as guide source for Australian TV (1 Viewer)

[etheesdad]

After I switched from Mythtv to MP, the only thing I missed was the excellent TV Guide, Shepherd. One of its more notable features (apart from verbose tv data) is its ability to scrape IMDB for Movies on TV, and thetvdb for episode data. The result is very good indeed.

Features:

Australians have traditionally had two options for acquiring TV guide data:

1. Downloading pre-compiled data from a service like IceTV or OzTivo
2. Using a tv_grab_au script to read data from a web site like ninemsn.com.au

Each method has drawbacks. The quality of pre-compiled data varies, while tv_grab_au scripts tend to frequently break. Shepherd aims to solve the traditional problems of tv_grab_au scripts:

1. It employs multiple TV guide grabbers and data sources, switching between them as necessary when one fails
2. It auto-updates, eliminating the need for users to manually address problems

Shepherd knows enough about the capabilities of each grabber in order to make intelligent judgments about which is most appropriate for any given situation, maximizing data quality while minimizing bandwidth usage. It analyses the XML output from each grabber to determine whether any further grabbers are required to obtain a full dataset of the required channels. It then employs postprocessor components to further refine the data: the imdb_augment_data postprocessor adds movie information from IMDb.com, tvdb_augment_data adds series/episode details from TheTVDB.com, flag_aus_hdtv marks shows that are available in High Definition, and more

I was browsing xpmediacentre.com.au and came across a discussion of how to get it working in Windows. I jumped at the chance to set it up, and thought it might be useful to provide the a guide here on how to do it for anyone interested. The information below is pulled from that thread and from some other sources.

It requires a Linux box or a virtual machine with Linux running (in this case xubuntu). I used Virtualbox as the VM. Any VM would probably work though.

On my machine the VM only uses 38mb of ram and 0% cpu when running minimised. (increases slightly when it grabs), but does not noticeably affect the performance of the machine.

1. obtain xubuntu or another suitable distro

2. install and configure virtual machine in XP/Vista/W7.

(Steps for Virtualbox)

new
OS- Linux
Type - Ubuntu
memory - default (384mb)
Virtual Hard Disk - NEW, dynamically expanding (8gb)

3. boot xubuntu / virtual machine and add dependencies for shepherd

sudo apt-get install xmltv libxml-simple-perl libjavascript-perl libalgorithm-diff-perl \
libgetopt-mixed-perl libcompress-zlib-perl libdata-dumper-simple-perl \
libdate-manip-perl liblist-compare-perl libdatetime-format-strptime-perl \
libhtml-parser-perl libxml-dom-perl libgd-gd2-perl libdigest-sha1-perl \
libarchive-zip-perl libio-string-perl libdbi-perl libclass-dbi-mysql-perl

4. Install Guest Additions in the VM (Virtualbox) - this allows file sharing and a number of other useful features.

In the VM window:

Devices -Install -Install Guest Additions
(toggle from full-screen mode so you can see the VM menu bar).

cd /media/cdrom0
sudo sh ./VBoxLinuxAdditions-x86.run

restart

5. Get Shepherd and configure (script is self explanatory)

w-get http://www.whuffy.com/shepherd/shepherd
perl ./shepherd

(remove the '-' from w-get - its there because the forum blocks posts with this command)

6. Run shepherd to get guide data

perl ./shepherd

(Takes awhile for the first run)

The resulting file is stored at:

~/.shepherd/output.xmltv

7. make and mount a shared folder

In XP, make a folder and share it (ie, C:\shared)

Virtualbox:
devices
shared folders
machine folders
add (path)
tick "make permanent"

then mount the shared XP folder to the Linux share

sudo mount -t vboxsf (name of xp shared folder without path ie "shared") (name and path of linux shared folder ie "/home/usrname/shared)

ie:

sudo mount -t vboxsf shared /home/jim/shared

8. direct MP to the file in the Host's (XP) shared folder

TVserver config - XMLTV - Import c:\shared\tvguide.xml
(dont forget to map channels in xmltv)
- more info on the XMLTV plugin here

9. Set up a cron job to make shepherd run automatically once per day

Open a terminal window in xubuntu and enter:

export EDITOR=gedit
export VISUAL=gedit
crontab -e

add 2 lines to the file:

# m h  dom mon dow   command

[B]0 12 * * * ~/.shepherd/shepherd --quiet 2>&1 >/dev/null
0 14 * * * cp /home/(username)/.shepherd/output.xmltv /home/(username)/shared/tvguide.xml 2>&1 >/dev/null[/B]

line 1 runs shepherd once per day at midday (12)
line 2 copies and renames the output file to windows share folder at 2pm (14)

--if you dont have gedit, substitute the name for the editor you use ie 'nano'

10. Remove from taskbar

Lastly, I found it a bit annoying having the VM on my taskbar as I dont really want to know its there, so I found a neat little utility that puts it in the system tray, where its out of the way.

Powermenu

The results are great. Much more detail than any tvguide.xml generator Ive run in the past.

Thanks to Arkay and aanda6 for the information and support.

 

[CairnsFella]

Look forward to giving this a try as since setting up MP, the EPG is the only thing giving me grief...

Could you (or anyone else in the know) advise whether the EPG's grabbed by Shepherd tend to contain Series(Season) and Episode data. Although I appreciatte such data is not always available, of the solutions that I have tried so far, some I have not got to work, and others - although they work great - have not yet returned any series/episode info that I can use to get MP TV-Series working.

Any help in this matter, greatly appreciatted.

 

[etheesdad]

I just had a look and cant see anything in the tv guide about season or episode data - just plot details for the episode. As far as whether it can be configured to do this, I dont know as Ive never tried.

Its an interesting idea though and Id also be interested to know if its do-able.

 

[CairnsFella]

Just a FYI.

Whilst I have no doubt that had I pursued "Shepherd" as intended it would have been an exelent epg provider, I actually have managed to get the info I require through the oztivo data I already had. I do have to do a find/replace to tag extra info on to the episode/seried data already included (Im sure this wouldnt be a chore for those in the know, but I just do it in notepad).

As a result I am able to pick up the required info in my recordings so that the programmes are automatically picked up by MP TVSeries without further intevention. On the few programmes I have recorded so far, this has been accurate, though I am led to understand that the oztivo data is not complete and sometimes innacurate.

Still, for my needs, if I only occasionally have to perform a "rename" then thats better than doing it all the time.

I'll keep my ear to the ground though, and if Shepherd does at any stage contain the data required I still look forward to giving it a try.

Thanks

 

[S.O.P]

Hi etheesdad,

I gave it a shot, started your guide, installed xubuntu-desktop 10.04 in VirtualBox with your settings and everything else default.

Restarted xubuntu, logged in as my user, opened a terminal and entered your apt-get and failed on libjavascript-perl - couldn't find package.

It's late here so I don't have time to troubleshoot it at the moment, do you have any ideas?

Also, when I start my XP MP singleseat, how do I make Virtualbox and Xubuntu log in automatically and run Shepherd automatically when I get to that point?

Keep in mind, while I've used Linux before, I'm not a power user and the guide, while well-written, isn't aimed at new users as it does leave some information out. Example: Boot Xubuntu/VirtualBox. Does that mean install Xubuntu in Virtualbox etc.

Thanks in advance if your offering help to move me along or can you link to the threads where you collated all your info?

 

[MPBuddy]

Thanks etheesdad, below is a variation to your method that I use for paytv program information.
This method launches virtualbox/linux/shepherd once a day at 3am from the windows sheduler.

1. Install virtualbox with linux mint 12 and shepherd as you described.
2. Add a shared folder so that linux can write files directly to the windows xmltv folder.
   In virtualbox menu devices->shared folders add a share called xmltv and select the windows xmltv folder.
   Check "make permanent", but not automount. On win7 and MP1.2.3 the xmtv folder was "C:\program data\TeamMediaPortal\MediaPortal TV Server\xmltv"
3. From the linux command line make the directory //media/share-xmltv then add the following command to "//etc/init.d/rc.local" so it mounts on boot.
   

   mount -t vboxsf xmltv //media/share-xmltv

4. Give the linux user shutdown privilages. Add the following to //etc/sudoers
   

   <username> ALL=NOPASSWD: /sbin/shutdown -h now

5. In the linux users home folder add the following script called epg.sh
   

   perl shepherd
   echo copying epg to windows share
   cp ~/.shepherd/output.xmltv //media/share-xmltv/tvguide.xml
   echo Ctrl-C now to prevent shutdown.
   sleep 60
   echo shutting down now!
   sudo shutdown -h now
   sleep 10

6. Create the scipt epg-startup.sh so that shepherd runs in a terminal
   

   xterm -e ./epg.sh

7. Now you need to run this script on linux boot. In mint12 I did this using the "startup applications preferences" tool found in Applications->Other->Startup Applications. Add an item to run epg-startup.sh
8. Now you need to launch virtualdub and linux from the windows command line. I created epg.bat on the windows desktop.
   

   echo EPG run on %date% at %time% > epg.log
   "c:\program files\oracle\virtualbox\vboxmanage.exe" startvm <virtualmachine name>

9. In media portal configuration PowerScheduler++ add VirtualBox to the list of programs to prevent the computer from going to sleep while virtualbox is running.
10. Run the epg.bat script from the windows task sheduler, say every morning at 3am. In win7 I found this only worked if used "Run only when the user is logged in". Maybe a permissions issue?
11. In media portal configuration select to use xmltv. This plugin periodically checks to see if the tvguide.xml file has been updated.
12. Map your channels to the epg data in the xmltv plugin.

NB. The first time shepherd is run it can take a few hours.

Good luck.