Database: Refactor MP1 database for multi seat usage (2 Viewers)

wizard123

Retired Team Member
  • Premium Supporter
  • January 24, 2012
    2,568
    2,668
    Home Country
    United Kingdom United Kingdom
    Is this still being worked on ? Multi seat is definately something lacking in MP1.
     

    Jean-Marc

    Portal Pro
    February 28, 2007
    382
    32
    Home Country
    France France
    First of all, big thanks to you, Tony, and to other people taking care of this important nevv feature that I vvas vvaiting for...for years!
    I hope that you'll achieve soon a perfect result, and even if the main team doesn't see the interest (or the possibility?) to include it in a next release, please give us a patch as easiest as possible to include to MP.

    ...a little bit confuse, no more posts since the end of november...?

    I understand that @ajs and you did a lot of changes on the video section, may I give you a suggestion?
    The possibility to play the trailers of the movies is missing.
    I imagine a trailer database, trailer automatically dovvnloaded vvith all other informations of the movie, vvithout the need to use another plugin like it is actually the case. And directly from the video menu, possibility to play it or the movie itself.
    In the past, the "shovvtimes plugin" developed by Nephilim did that but unfortunatly it is broken and Nephilim deseapered...maybe are you able to bring it back to live? (it's really missing) In this case the trailers vvill be dovvloaded in the same data base as video trailers.

    PS: sorry for my v+v=vv...
     

    Anthony Vaughan

    MP Donator
  • Premium Supporter
  • June 25, 2015
    474
    234
    Home Country
    United Kingdom United Kingdom
    When I started to use MP1, I realized that the client databases had some serious design issues; namely that they are fundamentally flat-file in design. It was also apparent that the client databases were locked for the duration of a client session making a multi-user design approach impossible. For several years I left things alone and put up with these deficiencies until I could stand it no longer and decided to role my sleeves up and see if I could do something about it. I did this because I truly believed in MP and wanted to make it work even better for me.
    I started out by, as a proof of concept, trying to see whether it was possible to make the system work in a multi-user environment, still using SQLite as the database architecture. I started this around June last year. By September I had something that worked for me and knew this was a workable solution. But I really had no idea why some things worked the way they did and it took a bit of work to find out why, for example, the video client database was set-up the way it was. AJS pointed out that there was such a thing as stacked files for a video. I had seen references to stacked files in the code but didn't really appreciate what they were and why. I realized that this the cause of some of the issues in the database design - the tail was wagging the dog. I changed the design so that stacked files were stored as parts associated with a movie.
    I am a developer that strongly believes that a system's design hangs principally on the database design and its ability to future-proof the system. In relational databases a flat design is a recipe for disaster because faults are so complex and time-consuming to fix. I couldn't help myself and I had to redesign the music, video, folder and fanart databases so that they would perform optimally and, as a welcome byproduct, be much smaller. So, I took on that task at the end of October and completed it at the end of December.
    The fanart database was 17 MB and is now 1.7 MB , (yes, one tenth of the size), with exactly the same input data. The music database is now half its original size. The video database is about the same size but I have normalized it so that there are at least twice the number of tables - at 10 MB.
    The key thing to understand is that the task of making the database multi-user required a complete re-write of the back-end and many, many changes throughout the system.
    This was not a trivial task and I think it will not be possible to embed these changes into the current system while preserving 100% of the previous functionality.
    I have been maintaining an upgrade path for the refactored system in parallel with the live system. The truth is that, in order to make the system multi-user, it is vital to control ALL database accesses through the Databases project and that all third-party products should call methods in the databases project to perform database tasks. I see this as a workable and feasible path, but it will require total commitment from the team to replace the current system with this solution and I am not convinced that that commitment is there yet.
    Therefore, I will continue to test and upgrade the refactored version for my own use. If anyone would like to use that version then I think it best that they regard it like a whole new product until, and if, it is taken on-board by the team. I have developed this system as a cut-down version of the live product that will work for multiple users in real time covering Video, Music, Pictures and Fanart. The Fanart is now very responsive without the previous delays and I have removed much of the hashtable behaviour - though I haven't been able to remove it completely yet. But, for the most part, the Fanart system is now data-driven; avoiding repeated reprocessing by analyzing existing databases to identify new data for which to retrieve fanart. I did some time-trialing of the scraping procedures and discovered that, with or without multi-threading, the processing took the same time pretty much to the second. So, I have removed multi-threading from the fanart application to minimized database clashes in a multi-user environment.
    All of this is why I have not been doing any posts. I will also not be writing any database upgrade code unless and until there is any demand for the refactored system. I have found that I can install this system against existing data, albeit video, music, pictures or fanart, from scratch very quickly. For me, the multi-user aspect makes it worthwhile. I appreciate that others may not feel the same way.
    If there is any demand for the refactored system then I will post the latest version and it's code. Bare in mind that the way I have implemented the system, you can install the live system and overwrite certain files from my system (batch file provided) - you must ensure that all of the client databases have been deleted first. The databases will build themselves and I recommend that the configuration application is used to import existing videos, music and pictures.
    If there is demand for the refactored system, then I will consider reworking the database upgrade code. However, having said that, I think that the new system approach is so different from the old system that it would be better to think of it as a different product - even though it looks exactly the same. The beauty of MP is that it works as a closed system and can be rebuilt from scratch quite easily - as I have done several times over the last few weeks.

    Tony
     

    Anthony Vaughan

    MP Donator
  • Premium Supporter
  • June 25, 2015
    474
    234
    Home Country
    United Kingdom United Kingdom
    Size doesn't matter, speed matters.
    Exactly. And it is now much faster as a consequence of designing the database to make optimal use the database engine. This is how relational databases are supposed to work - not with fields containing iterations of string data; not to mention joining tables using string columns rather than numbers. The size issue isn't irrelevant though. Index size is critical to optimal performance. When you have string primary keys you force indexes to be far larger than indexes over number fields. All relational database engines are designed to access numeric indexes far more quickly than string indexes. This is really basic stuff.

    I am not trying persuade anyone to do anything they don't want. I am only offering an alternative that provides multi-user capability and optimal database design and performance. I gain nothing either way and I will support what I develop if I am asked to.

    I now have a system that has run error free for weeks. It's a win-win for me and an easy life from now on if no-one else is interested. I will help if asked.

    Tony
     

    ajs

    Development Group
  • Team MediaPortal
  • February 29, 2008
    14,138
    8,690
    Kyiv
    Home Country
    Ukraine Ukraine
    Possibly, but I can't test it yet, all attempts to run the test version result in errors, invalid data, and crashes.
     

    Anthony Vaughan

    MP Donator
  • Premium Supporter
  • June 25, 2015
    474
    234
    Home Country
    United Kingdom United Kingdom
    Possibly, but I can't test it yet, all attempts to run the test version result in errors, invalid data, and crashes.
    That's strange because when I ran the system against your database it upgraded successfully. Something else must be going on.

    This is why I think it would be best to use the refactored system from scratch to prove that it is okay rather than trying to do the most complex part (upgrading the database) first. I don't think anything else would be a fair option because this is such a major upgrade. I think it is only right that people try it using its basic features to ensure that they are working in a multi-user environment - which is the main point after all. Then I can address the more complex stuff; like upgrading the databases.

    BTW, the current version is months out of date and I will only refresh it if there is demand; because I have to feel that people really want it to work before I am going to go to all of that effort.

    Tony
     

    ajs

    Development Group
  • Team MediaPortal
  • February 29, 2008
    14,138
    8,690
    Kyiv
    Home Country
    Ukraine Ukraine
    Many are waiting for the new version, so it's worth publishing it and then people can test it.
     

    Users who are viewing this thread

    Similar threads

    Hi, must be really a while, cuz I can not remember any post from you ;) But might be also that I joined the team too late. Yes, there will be many things. Main reason is simply, that MP2 is not developed based on MP1. I for example never used MP1. I tried it lately to test voice control and was shocked about many inconsistancies in...
    Hi, must be really a while, cuz I can not remember any post from you ;) But might be also that I joined the team too late. Yes...
    It´s been a while... :) Well, simply the reason is cause MP1 was working so well... :) Now I decided to upgrade my old servers and...
    Replies
    1
    Views
    242
    Ok, the importer also remembers the queue of items that are still needed to be processed in settings file. If there is something stuck, it will try again on next server start and probably stuck again. But removing the media source also removes the associated queue. The server's TV plugin automatically creates the media sources for...
    Ok, the importer also remembers the queue of items that are still needed to be processed in settings file. If there is something...
    Hi, I’ve been struggling with MP 2 now for several weeks. Initially all was fine. Then in the client, the listings of recorded TV...
    Replies
    8
    Views
    911
    It is looking and working very well now and I am almost at the point of switching my HTPC to MP2, madvr would definitely push me over the edge for sure.;)
    It is looking and working very well now and I am almost at the point of switching my HTPC to MP2, madvr would definitely push me...
    We are proud to present MediaPortal 2.3 MediaPortal 2.3 is a full-blown media center software that addresses the most common user...
    Replies
    4
    Views
    3K
    MP1 MP2 MP2 DLNA status? DE
    What I meant is that we have a MP2 plugin called UPnP Render that can play media from DLNA servers if I remember correctly. I don't know the current state of the plugin though. That's why MP2 probably never will be certified. ;) Yes, that was also the idea, but it is a big job as all TV's, software etc. seems to not do everything...
    What I meant is that we have a MP2 plugin called UPnP Render that can play media from DLNA servers if I remember correctly. I don't...
    So I'm wondering what the state of play is with MP2 and DLNA functionality? When I started this post, I did see some posts...
    Replies
    10
    Views
    1K
    Thank you joecrow! It seems to resolve the problem I got since 1.22.
    Thank you joecrow! It seems to resolve the problem I got since 1.22.
    Hi Guys, Most times I have had an issue with MP1 it has been a platform/OS/finger-trouble issue that a re-install etc fixes...
    Replies
    9
    Views
    984
    Top Bottom