FASTER music/movie browsing and DB access

[htpcNoob]

Here's a trick that should speed up MP music/video browsing and database access by using a ramdrive. For Music/video browsing, MP checks to see if there is an image in its own THUMB directory for the item. if its not there it will check the directory where the item is located for a file called folder.jpg. If the images are in MP's thumbs directory, then you will notice a speed increase in browsing. In general, database access will be faster.

Tools:
-------
Ramdisk: Freeware. If you're not familiar with ramdrives they basically emulate a local hard drive using your systems memory and thus gives you a MUCH MUCH MUCH faster "drive". The only problem is that upon reboot, all data on the ramdrive is erased (ram is cleared). There are some Ramdisk programs that'll automatically restore/backup everything at startup/shutdown but I had probelms with them. We will restore and backup the ramdrive contents during system start and shutdown. (batch scripts below).
link: http://www.codeguru.com/Cpp/W-P/syst...cle.php/c5789/

Sysinternal Junction: Freeware. This software creates a sort of symbolic link directory. If you are familiar with linux, then you know what I'm talking about.
link: http://www.sysinternals.com/Utilities/Junction.html

Considerations
-----------------
You can install MP entirely on a ramdisk. The problem is that MP has a large footprint because of the skins and such. Even deleting unused skins is not worth installing MP on a ramdrive yet. In my setup, I have stored the database and some of the thumbs directories on the ramdrive. Everything else is in the default MP install dir (C:\Program Files\MediaPortal).

Also, I wouldnt recommend putting the ENTIRE thumbs directory on the ramdrive. If you create thumbs for say "My Pictures" and you have a lot of pictures, then the ramdrive will fill up the ramdrive in no time. So its best to be selective about which thumb directories go in the ramdrive. For me I've put the Thumbs\Music\Albums and Thumbs\Videos\Title under the ramdrive.
The size of the ramdrive should be set to at least 32mb. This will support a lot of the thumbs that are downloaded by from IMDB and allmusic. The images that MP download, usually range from 1KB to 11KB with a couple of 30KB's as well. At 11KB each, you can store about 6000 thumbs on a 64MB ramdisk.

Currenlty MP doesnt allow you to designate where specific folders (such as database and thumbs) can reside. They are created in the MP install directory. This is where Sysinternals junction.exe comes in.

Here we go...

Steps:
-------
1) Install the ramdisk software and set it up. (You can set it up by going to your System Properties->Hardware->Device Manager and accessing properties of "Ram Disk" item after it is installed. Set it to the 64MB, specify the drive letter. I have mine set to Z. You may need to format the drive first by going into explorer, right-clicking the drive and choosing format.

2) Install junction.exe anywhere. I have mine installed under C:\Utils.

3) Create a folder that will copy the contents of the ramdrive to the hardrive during shutdown and restor them during startup. I have mine set to C:\RAMCACHE.

4) Its decision time! What would like to keep on the ramdrive? I put the entire database, Thumbs\Music\Albums, and Thumbs\Videos\Title on the ramdrive. So under C:\RAMCACHE, i created the following folders: database, Music-Albums, Videos-Title.

5) Go into the MP install directory (mine is C:\Program Files\MediaPortal) and look under the database folder. **MOVE ALL FILES and FOLDERS (if any)** from C:\Program Files\MediaPortal\database to C:\RAMCACHE\database. This will make the MediaPortal database directory empty. NOTE: Junctions can only be created on empty directories. If the folders have files, or other directories in them, then junctioning will fail. This is why we MOVE the contents.

6) Do the same thing with the Thumbs\Music\Albums. Go into C:\Program Files\MediaPortal\Thumbs\Music\Albums and MOVE EVERYTHING to C:\RAMCACHE\Music-Albums.

7) Again, go into C:\Program Files\MediaPortal\Thumbs\Videos\Title and move everything to C:\RAMCACHE\Videos-Title.

8) Next comes the script, startup.bat, which will be startup when system is booted up. This script will 1) copy the contents of C:\RAMCACHE to ramdrive Z:\ and 2) create the junctions from the mediaportal database, Thumbs\Music\Albums and Thumbs\Videos\Title directories. If you have set up the directories like me,

MP install directory = C:\Program Files\MediaPortal
ramcache directory = C:\RAMCACHE
ramdrive = Z:\
junction.exe = C:\UTILS\junction.exe

then you dont have to edit the startup.bat file. Otherwise, you need to make changes to it and make sure the directories are correct.

Code for startup.bat:

@echo off
mkdir Z:\Database
C:\utils\junction.exe "C:\Program Files\MediaPortal\database" Z:\Database

mkdir Z:\Music-Albums
C:\utils\junction.exe "C:\Program Files\MediaPortal\thumbs\music\albums" Z:\Music-Albums

mkdir Z:\Videos-Title
C:\utils\junction.exe "C:\Program Files\MediaPortal\thumbs\videos\title" Z:\Videos-Title

xcopy "C:\RAMCache\*" /e /c /q /h /r /y "Z:\"

9) Now comes the script shutdown.bat. Upon system shutdown, it will backup the ramdrive contents for Z:\ to C:\RAMCACHE. Again you may need to change the paths if they were different from mine.

Code for shutdown.bat:

@echo off
cd /d C:\
del /s /q /f "C:\RAMCache_BACKUP"
rmdir /q /s "C:\RAMCache_BACKUP"
mkdir "C:\RAMCache_TEMP"
xcopy "Z:\*" /e /c /q /h /r /y "C:\RAMCache_TEMP\"
rename "C:\RAMCache" "RAMCache_BACKUP"
rename "C:\RAMCache_TEMP" "RAMCache"

In this script you will notice that I create two OTHER directories C:\RAMCACHE_BACKUP and C:\RAMCACHE_TEMP. They are actually temporary directories used by the script to make sure copying is properly done. Call me paranoid if u want :-D :-D :-D but better safe than sorry I say.

10) After saving the two batch files, it is time to make sure that they are properly executed during startup and shutdown. Go to START->Run on the taskbar and type gpedit.msc. This will launch the group policy editor. Go to Computer Configuration->Windows Settings->Scripts (Startup/Shutdown).
On the right panel double-click Startup and add the startup.bat file to the list. I have mine as C:\startup.bat.

11) Now select Shutdown and add the shutdown.bat. I have mine as C:\shutdown.bat.

12) Double check to make sure the following directories are empty: C:\Program Files\MediaPortal\database, C:\Program Files\MediaPortal\Thumbs\Music\Albums, and C:\Program Files\MediaPortal\Thumbs\Videos\Title.
As I mentioned junctions can only be created on empty directories.

You are now set. You can test the scripts w/o rebooting. Just run the startup.bat file. If it was done properly, then everything in c:\ramcache should now exist the ramdrive z:\. Now run mediaportal and check your music/videoes see if see the albums/movie images. Check the tv to see if you have all the channels. This will confirm, that the database was properly moved and junctioned as well. If you do, then you're set and MP should run faster now.
when browing music and movies thumbs.

You may be thinking of putting the startup.bat in the startup folder of your programs or in the regisrty but that can lead to problems. If you do his and have mediaportal load at startup as well, then what could happen is mediaportal may start loading while the script is running and this could cause problems. The sections is group prolicy and scripts that are ran even earlier and guarantees that the copying takes place before users are logged on and other programs are started.

Word of caution: If your system BSOD's, hangs, or you use the reset button on the pc and windows is not properly shutdown, you will lose the thumbnails/database info that was added FOR THAT SESSION only. previous saved thumbnails and db changes will still be retained. Scripts can be written to call the shutdown.bat at specific intervals in the background or manually from "my programs" so that the ramdrive contents are saved often.

Happy htpc-ing and lemme know if you notice the speedup.

[rtv]

wow! this is just what i had in mind - not for thumbs but tv-database. I realized that by adding ALOT of EPG information via xmltv the channel switch gets slower and slower :-/

Maybe it'd be easy for you in your running setup to include this as well? I'd like to hear if it's worth the work

[htpcNoob]

It should definately speed up EPG loading/browsing as the file will reside on the ramdrive. I can now browse all my albums quickly as scrolling thru them one page at a time takes < one second. before it took about 2 seconds to scroll one page at a time.

[jadz]

Can't we implement this solution in the code base ? Prefetch images or something?

[htpcNoob]

An improved caching system for images would definately help. But this trick has to do with loading the images from the disk. The bottleneck here was the time it took to load the image into memory. The rendering of the images didnt seem to slow it down much.

I myself did other tweaks to the code for additional boost. All my mp3's are stored in folders as follows: <Genre>\<Artist - Album>\<Track> - <Title>.mp3. I took all my folder.jpg images and renamed to "<Artist - Album>.jpg" and put these album images on my ram drive. I changed the code in MP's GetCoverArt function so that it first looks in my ramdrive directory for "<Artist - Album>.jpg". If that is found, then just display it. The trick here was not to look for a folder.jpg on the directories corresponding to the mp3 file, but rather look for "<Artist - Album>.jpg" on one specific directory (this being a directory on my ramdrive).

[rtv]

Wow, you're not intending CVS-updates with this pile of work :shock:
But your changes would be interesting because I organize myself identically Would be fine if MP would have an option to choose any pic in the folder if it isn't named folder.jpg ..

[htpcNoob]

Originally Posted by rtv

Wow, you're not intending CVS-updates with this pile of work

Actually. I CVS update MP on a daily basis. Its all scripted.
1) Download CVS
2) Use PowerGREP to automatically add my code to the proper places
3) Compile the project
4) Send it to my HTPC machine via windows sharing (samba)
5) Run a script on HTPC machine using PSEXEC to close MP, update the files and restart MP.

All with just one click

[rtv]

You're a born linux user

Everybody complains tricky setup/fine-tune but it's also quite impressive everytime it works :lol:

To stupid i've no other windows machine beneath my htpc.. I really don't like the idea to have an IDE with all sdks laying around on my htpc usable for everyone.

[SpaceProtector]

I now use the RAMdisk it worked fine just as explained above. I also put the tvguide.xml file in the RAM. Thx for this neat trick.

[rtv]

It works great for me too!

well, do you have the complete xmltv directory on ramdisk? Since I know from previous versions that there were issues with the file Tvguide.xml being stored in different places..