Aibility to play Media without adding them to DB (1 Viewer)

Lehmden

Retired Team Member
  • Premium Supporter
  • December 17, 2010
    12,553
    3,934
    Lehmden
    Home Country
    Germany Germany
    Hi.
    I really would like to have a possibility to play media items without adding them to DB first. Similar as it's possible in MP1 right now.

    Let me explain why. In MP1 I simply can connect a SD card and watch the pictures on. In MP2 first I need to add them to DB and generating thumbs for them, before I can watch them. After I watched them they never would be watched from same path. My visitors are gone or I had them photoshoped and sorted them to my regular collection. So I need to add them a second time. In MP2 all those unnecessary DB action is needed over and over again.

    And the DB file itself grows bigger and bigger until it's not really usable at all. I've tried to add my 500.000 pictures collection several times without much luck. After having 25% of them in DB (this takes about 2 or 3 days to generate the Thumbs) the DB file is sooo big I can't use MP2 like normal. Every DB action lasts until eternity while the Server is working on HDD like a maniac.

    I really would like to add a path and simply browse it similar like in explorer. If I found a picture and I click on it, the picture will be displayed, If it's an audio file I can listen to it by clicking on it and same for video files. Without any need to add them all to DB.

    I really like DB for all "main stream" media like Movies, Series and so on. But there are tons of other stuff like demo videos, screenshots, raw pictures,... without any need to add them to DB at all. But I like to play them if needed.

    As it comes to this, MP2 could behave a bit more like MP1. This would be much better, I think.
     

    Takachulo

    New Member
    June 10, 2012
    2
    1
    50
    Home Country
    Norway Norway
    I agree 100%. I like the way "Videos" works in MP1 where files are separated in folders, and the files aren't all shown together.

    Also, the files added as "TV Series" and "Movies" should not then also show up in the "Videos" section. It just makes a huge mess of files.
     

    mbuzina

    Retired Team Member
  • Premium Supporter
  • April 11, 2005
    2,839
    726
    Germany
    Home Country
    Germany Germany
    You mix 2 issues here.

    1st: MP2 should be able to handle large (very large) collections gracefully. I have approx. 200.000 Pictures, 80.000 Songs & some films (lower number). MP should be able to handle 10-fold at least.
    2nd: Browse SD Card without importing it. That should be a client plugin (as you would plug the SD Card into the client I would suppose). I would like to have an SD Card plugin that can browse & Show media on the card, allow for importing pictures to where you store them (along with renaming, grouping, etc) and maybe some other features.

    Agree?
     

    morpheus_xx

    Retired Team Member
  • Team MediaPortal
  • March 24, 2007
    12,073
    7,459
    Home Country
    Germany Germany
    Did you test this feature already? When you are using "Browse Media" you can switch to local view, which will browse filesystem only (no DB)?
    01_local_view.jpg 02_local_view.jpg 03_local_view.jpg

    Does this work for you already better? What else is missing?
     

    Lehmden

    Retired Team Member
  • Premium Supporter
  • December 17, 2010
    12,553
    3,934
    Lehmden
    Home Country
    Germany Germany
    Hi.
    I've tried this and it seems to be an option. But there also Thumbs are generated (would be good if those thumbs are windows system "thumbs.db" which are generated anyway) and this takes a long(er) time. And I can't add local shares on MP2 Server machine... (see here)
    Main goal is to keep the DB responsible. It's enough already that I have 1500 movies and 16.000 series episodes inside. This significant slows down all DB handling already. Adding my 500.000 pictures makes the DB soooo slow I can't use it any longer. BTW where are the thumbs for local shares are stored? Did not find it jet.
     

    MJGraf

    Retired Team Member
  • Premium Supporter
  • January 13, 2006
    2,478
    1,385
    Hi Lehmden,

    First of all, even if you add a share as local share, the metadata of the media items in this share is stored in the database (I.e. MediaLibrary) of the MP2 server. So this is not a solution if you want to keep the size of your database low.

    But besides that it scares me a bit that you are not able to add a local share in your situation. Before I try to reproduce, could you please give me some more information on your configuration? I assume your MP2 Server is installed on computer A, your MP2 client is installed on computer B. You are using MP2 Client on computer B to add a local share with the NetworkNeighborhoodResourceProvider and you try to add a windows share located on computer A. Is that correct?

    Thanks,
    Michael
     

    Lehmden

    Retired Team Member
  • Premium Supporter
  • December 17, 2010
    12,553
    3,934
    Lehmden
    Home Country
    Germany Germany
    Hi.
    I assume your MP2 Server is installed on computer A, your MP2 client is installed on computer B. You are using MP2 Client on computer B to add a local share with the NetworkNeighborhoodResourceProvider and you try to add a windows share located on computer A. Is that correct?
    Yep. But if all metadata and thumbs are stored in main DB then local shares did not make much sense at all to me...

    My main goal is to keep the DB clean. Media items that are not "mainstream" I don't want to have in DB but want to play nevertheless. Same as it's possible on MP1 or XBMC. Example with MP1 you can add Video folders containing Movies to MyVideos DB so they are browsable in DB view, attached with Artwork and IMDB infos. And you can leave other folders containing e.g. holiday videos outside the DB so they are only browsable in filesystem view, not in DBview...

    Best I've seen here so far is XBMC. There you add folders containing videos and then decide which "scrapper" to use. If it's IMDB, then those videos are movies, if it's TVDB, the they are treated as Series, if it's LastFM then they are Music Videos,... Ad if you choose no scrapper at all, the videos are playable only without any metadata from online...

    Especially I need something without DB for my huge pictures collection. I've tried twice to add them all to DB, but after 48 hours of thumbnail generation the DB was sooo slow, no longer usable at all. Ad there only 20% of all pictures are added until then... So a complete import will last for about 10 days on my Celeron 847 server machine...
     

    morpheus_xx

    Retired Team Member
  • Team MediaPortal
  • March 24, 2007
    12,073
    7,459
    Home Country
    Germany Germany
    I understand your arguments, Lehmden. I think about a solution, maybe we add a flag to shares that they are not intended for DB imports, or define a special type "local browsing only".
     

    chefkoch

    Retired Team Member
  • Premium Supporter
  • October 5, 2004
    3,129
    1,634
    Dresden / Munich / Maastricht
    Home Country
    Germany Germany
    Especially I need something without DB for my huge pictures collection. I've tried twice to add them all to DB, but after 48 hours of thumbnail generation the DB was sooo slow, no longer usable at all. Ad there only 20% of all pictures are added until then... So a complete import will last for about 10 days on my Celeron 847 server machine...
    Besides having the possibility to add shares without importing them to the db and/or better support for removable media (sd-cards, ...), doesn't this also bring us back to the "thumbnails are stored in db" issue?

    In general I also would like to have a backup option to browse videos, pics and maybe even movies if something with import to DB failed, but if the thuimbnails in the db are an issue, we should not ignore this fact or only workaround it.
     

    MJGraf

    Retired Team Member
  • Premium Supporter
  • January 13, 2006
    2,478
    1,385
    I think we are talking about two different things here which we have to tackle separately:

    Browsing media without adding them to the db
    This is a good point and worth an improvement. But at least my feeling is that this should only be the way to go when you watch media that are "temporary" (i.e. on sd cards or something you downloaded, want to have a look at and delete afterwards). When you are talking about media you don't watch that often but you store them somewhere at a fixed location, I would nevertheless want to have the possibility to add them to the DB without having to worry about this being a problem with DB size and speed. The reason for me is easy: This is the whole point of MP2 to me - having a central DB which has as much information on all your media as possible. Not putting this information into the DB intentionally is IMO just a workaround for a completely different issue:

    Speed
    And this brings us to the second thing we are talking about: speed. Independently from the above, we should put a lot of effort into making MP2's speed (as far as possible) independent from its speed. The reason is that I think over the next years, the amount of media people have will grow - and so will the amount of metadata for these media. That means MP2 will have to cope with very big amounts of data - such as your collections, Lehmden (and this is why I have to say thousand :D to you for testing MP2. Your media collections are really a great test to make MP2 future-proof :) ). And I also think this should be possible, because although the amount of media and metadata will grow, the user can only handle a very small amount of data at the same time. So we are talking about very small parts of the huge data collection to be shown at the same time, which our system should be able to handle very quickly. Now when I'm thinking about speed improvements, five things come into my mind:

    Reduce the amount of data
    What you are reporting about XBMC sounds like they import it anyway into their db, but if you don't select a scraper, no additional information is imported making the import very fast, maybe hardly noticeable that they are actually imported into some kind of db. We currently only have the choice to tell the importer which type of media to expect (videos, movies, etc.). That makes it easier for the average user who doesn't know what a scraper is. But maybe we should have an "advanced" button there to enable the advanced user to choose the MetadataExtractors he wants to apply (or at least something like (a) all the information you can get - even from the internet, (b) only local information and (c) only local information without images such as covers, thumbs, etc. and maybe even (d) only path and file name).
    But in the end, this is again only a workaround. It is the purpose of MP2 to get as much info as possible - and to handle it as fast as possible.

    Speed up the collection of data
    This seems to be one of the issues you are reporting because it takes ages to generate the thumbnails for your pictures. Besides tackling this particular problem in detail it may be an option that you can choose whether the MP2 Server generates this data for all media automatically in the background (via the importworker) or whether this data is only generated when you watch a media item for the first time. It would then be slower to watch this particular media the first time, because we would only then generate the metadata, but when you watch them the second time, it would be faster because the metadata is already there.

    Speed up the retrieval of data
    This is what Valks is working on and it is not really a "speed-up" it is more an "only get the data we really need". It has to be seen with what I said above about the user only being able to handle a very small amount of data at the same time. Currently, when you search for all the mediaitems with "Eric Clapton" as a search string, you may get 200 mediaitems, which are all retrieved from the database - although the UI can only display maybe 10 of them at the same time. Valks is working on something called "data virtualization", which makes sure that only 10 (or maybe 30 - the 10 you see, 10 before and ten after to make scrolling smooth) are fetched from the database. I'm sure this will improve the speed a lot.
    The same idea - just for the UI (and therefore called UI virtualization) - is possible on the client side regarding the generation of the display views of the items to be shown. Instead of generating the views for all the items available, it makes sense to only generate the views for those you actually see (and some before and after). You can already see this effect by switching between list view and e.g. thumbs view. Our list view already virtualizes - the other views don't which is why they are much slower with large amounts of mediaitems.

    Speed up the transfer of data to the client
    Since we cannot speed-up the network we have available from the MP2 Server to the MP2 Client, this is again no real speed-up, but a way to make MP2 "feel faster". There are two options I can see:
    (a) First transfer the necessary (non image) data and display them and then in the background fetch the image data. This is what Morph has implemented for the FanArtService. The effect is that when you enter a screen with all your mediaitems of a certain kind, you see them immediately by name (but without images) and can already scroll, etc., but the images for the items appear one by one while your UI is responsive all the time and independent from the images. We should definitely find a way to make this happen not only for the FanArtService but also for the thumbnails in the MediaLibrary.
    (b) Having a cache for images on the client side. This seems to be the way XMBC does it. They maintain a small database on the client side, just holding the IDs, the path, a checksum and the "number this item was display within the last week or so" and a folder with the respective images. When they display an image, they use the checksum to check whether the local cache still is the same as the original image on the server. If yes, they first look into their local cache and only get it via network, if it's not in the cache. If the checksum doesn't match, they get it from the server in any case (to make sure the client immediately reacts to changes on the server). Everytime they fetch an image from the server, they put it into the local cache. If the image isn't used for some time, it is deleted from the local cache to keep it small. Maybe we should think about something like that...

    Speed up the backend
    Now this is certainly a very important part of the speed, but according to the above, it's only one of many options. And this is where we are talking about not putting BLOBs (i.e. images) into the database to make it faster. I personally like having all the metadata in the database - the images as well - for several reasons (not cluttering the directory structure with thousands of files having cryptic names, data integrity in the db, etc.). And this is the area, where I can help most in improving the speed.
    I already tried with the SQLite plugin and I'm perfectly willing to improve its speed further in particular with respect to BLOBs. The only thing I fear is that no one answered Morpheus' post here regarding the problem we have with SQLite and TVE3.5 and if this isn't solved by the syste.data.sqlite developers, we have a problem with SQLite... I therefore already had a look at Firebird (which we have as a very old and outdated plugin urgently needing an update). To me it looks promising as (a) it is possible to integrate it into MP2 Server without the need for a separate installation, (b) there is no size limit of the database and (c) they seem to be very far with their support of EF with respect to TVE3.5. See here - they already support EF 6 for .NET 4.0 and .NET 4.5. But I have no clue how fast this would be so the only chance is to try and error... (a minor drawback that checkoch won't like: They offer their .net-provider via nuget, but not the database itself so we would have to create our own package with the database for the MP-nuget-feed. If this wasn't the case, I would have removed the binaries already from our repository...).
    Now back to the question how we should deal with images on the storage side. I see five options:
    1. Storing images like any other metadata in the same tables in the database. This is what we currently do. What we can do to improve this is try different databases and different settings of the databases.
    2. Storing images in the same database but in a different table.
    3. Storing images in a second database (of the same database type), i.e. in a separate database file.
    4. Storing images in a second and different database which is optimized for BLOB storage (I thought about some kind of document database such as MongoDB or a simple but fast key-value-store only).
    5. Storing images in the filesystem and only the rest of the metadata in the database.
    Unfortunately, anything but option 1 requires changes to the MediaLibrary itself, because currently we treat BLOBs like any other data and to implement 2-5, we would have to introduce a separate MediaLibrary treatment for BLOBs first, before we can try these options.
    I found a very interesting test last night, which implies that with SQLite even option 3 could give as enormous speed improvements and option 3 would definitely be easier to implement than options 4 or 5.
    But before I start digging into all these options, I wanted to here more opinions about all this - so let your thoughts flow ;)

    Finally, as to your speed problem, Lehmden: Sorry for cluttering this thread with so much information, but your posts in this thread made me want to find more info on this whole speed thing and the above is the result of a lot of reading in the internet last night. However, I still cannot judge what makes your system slow when you import a lot of data in your database. To find our whether it is the database itself, please do whatever makes your system slow, post your SQLDebug.log here, tell me what you did and I will have a look at it to find out whether is has to do something with "speed up the backend".

    Thanks,
    Michael
     

    Users who are viewing this thread

    Top Bottom