SQLiteDatabase Plugin for MP2 (1 Viewer)

[MJGraf]

Again just a note so it is not forgotten:

There has somone apparently already written a DDLBuilder class for SQLite - which is the most part of making SQLite compatible with EF4 (i.e. with SlimTV Native):
https://github.com/jcwmoore/athena/blob/master/Athena.SQLite.Entity/DdlBuilder.cs

The code is apparently based on the MS sample provider and that guy ran into the same problem as I did with my first try: SQLite does not support "ALTER TABLE ADD CONSTRAINT ...". You have to specify the constraints already in the "CREATE TABLE" statement. It's not possible to later add them. The solution there is to "fake" the constraints with triggers. OK, that might work (didn't try, yet), but somehow I don't like the idea of faking a functionality, which is actually there but needs to applied just by another syntax.
Nevertheless, this helps a lot and shows that my idea is not completely out of scope...

 

[MJGraf]

Just pushed two cosmetic changes to GitHub:

• Removed unused references from SQLiteDatabase project
• Removed redundant logging of FullUri because the FullUri is contained in the ConnectionString, which is logged as a whole anyway

If we don't find any more bugs in it, we are only one commit away from v0.08:

• Setting plugin version to 0.08 in plugin.xml

 

[morpheus_xx]

The test installer failed to finish. Breese posted his logs, the last line comes from SQLite database. Could it be that it hangs there?

2013-11-09 09:53:22.918020 : PluginManager: Plugin 'Tve3RecordingMetadataExtractor' enabled.
2013-11-09 09:53:22.923020 : PluginManager: Trying to enable plugin 'VideoMetadataExtractor'
2013-11-09 09:53:22.923020 : PluginManager: Plugin 'VideoMetadataExtractor' enabled.
2013-11-09 09:53:22.923020 : PluginManager: Trying to enable plugin 'VideoThumbnailer'
2013-11-09 09:53:22.928020 : PluginManager: Plugin 'VideoThumbnailer' enabled.
2013-11-09 09:53:22.928020 : PluginManager: Trying to enable plugin 'ZipResourceProvider'
2013-11-09 09:53:22.928020 : PluginManager: Plugin 'ZipResourceProvider' enabled.
2013-11-09 09:53:22.928020 : PluginManager: Running in maintenance mode
2013-11-09 09:53:22.933020 : ServiceRegistration: Loading services from plugin manager at location '/Services'
2013-11-09 09:53:22.943020 : PluginManager: Trying to activate plugin 'OnlineLibraries' (id 'b32504f3-4374-4640-94a1-16e1dee84c3c')
2013-11-09 09:53:22.958020 : PluginManager: Plugin 'OnlineLibraries' (id 'b32504f3-4374-4640-94a1-16e1dee84c3c') activated.
2013-11-09 09:53:22.988020 : PluginManager: Error building plugin item 'SeriesTvDbMatcher' at location '/Services'
Exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> MediaPortal.Common.Exceptions.ServiceNotFoundException: Could not find the MediaPortal.Common.Threading.IThreadPool service
at MediaPortal.Common.ServiceRegistration.GetService(Type type, Boolean throwIfNotFound)
at MediaPortal.Common.ServiceRegistration.Get()
at MediaPortal.Extensions.OnlineLibraries.Matches.BaseMatcher`2..ctor()
at MediaPortal.Extensions.OnlineLibraries.SeriesTvDbMatcher..ctor()
--- End of inner exception stack trace ---
at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck)
at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache)
at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean skipVisibilityChecks, Boolean skipCheckThis, Boolean fillCache)
at System.Activator.CreateInstance(Type type, Boolean nonPublic)
at MediaPortal.Common.PluginManager.PluginRuntime.InstantiatePluginObject(String typeName)
at MediaPortal.Common.Services.PluginManager.Builders.ServiceBuilder.BuildItem(PluginItemMetadata itemData, PluginRuntime plugin)
at MediaPortal.Common.Services.PluginManager.PluginManager.RequestItem(PluginItemRegistration itemRegistration, Type type, IPluginItemStateTracker stateTracker)
Message: Exception has been thrown by the target of an invocation.
Site : System.Object CreateInstance(System.RuntimeType, Boolean, Boolean, Boolean ByRef, System.RuntimeMethodHandleInternal ByRef, Boolean ByRef)
Source : mscorlib
Inner Exception(s):
Could not find the MediaPortal.Common.Threading.IThreadPool service
Stack Trace:
at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck)
at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache)
at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean skipVisibilityChecks, Boolean skipCheckThis, Boolean fillCache)
at System.Activator.CreateInstance(Type type, Boolean nonPublic)
at MediaPortal.Common.PluginManager.PluginRuntime.InstantiatePluginObject(String typeName)
at MediaPortal.Common.Services.PluginManager.Builders.ServiceBuilder.BuildItem(PluginItemMetadata itemData, PluginRuntime plugin)
at MediaPortal.Common.Services.PluginManager.PluginManager.RequestItem(PluginItemRegistration itemRegistration, Type type, IPluginItemStateTracker stateTracker)
2013-11-09 09:53:22.998020 : ServiceRegistration: Could not register dynamic service with id 'SeriesTvDbMatcher'
2013-11-09 09:53:22.998020 : PluginManager: Error building plugin item 'MovieTheMovieDbMatcher' at location '/Services'
Exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> MediaPortal.Common.Exceptions.ServiceNotFoundException: Could not find the MediaPortal.Common.Threading.IThreadPool service
at MediaPortal.Common.ServiceRegistration.GetService(Type type, Boolean throwIfNotFound)
at MediaPortal.Common.ServiceRegistration.Get()
at MediaPortal.Extensions.OnlineLibraries.Matches.BaseMatcher`2..ctor()
at MediaPortal.Extensions.OnlineLibraries.MovieTheMovieDbMatcher..ctor()
--- End of inner exception stack trace ---
at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck)
at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache)
at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean skipVisibilityChecks, Boolean skipCheckThis, Boolean fillCache)
at System.Activator.CreateInstance(Type type, Boolean nonPublic)
at MediaPortal.Common.PluginManager.PluginRuntime.InstantiatePluginObject(String typeName)
at MediaPortal.Common.Services.PluginManager.Builders.ServiceBuilder.BuildItem(PluginItemMetadata itemData, PluginRuntime plugin)
at MediaPortal.Common.Services.PluginManager.PluginManager.RequestItem(PluginItemRegistration itemRegistration, Type type, IPluginItemStateTracker stateTracker)
Message: Exception has been thrown by the target of an invocation.
Site : System.Object CreateInstance(System.RuntimeType, Boolean, Boolean, Boolean ByRef, System.RuntimeMethodHandleInternal ByRef, Boolean ByRef)
Source : mscorlib
Inner Exception(s):
Could not find the MediaPortal.Common.Threading.IThreadPool service
Stack Trace:
at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck)
at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache)
at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean skipVisibilityChecks, Boolean skipCheckThis, Boolean fillCache)
at System.Activator.CreateInstance(Type type, Boolean nonPublic)
at MediaPortal.Common.PluginManager.PluginRuntime.InstantiatePluginObject(String typeName)
at MediaPortal.Common.Services.PluginManager.Builders.ServiceBuilder.BuildItem(PluginItemMetadata itemData, PluginRuntime plugin)
at MediaPortal.Common.Services.PluginManager.PluginManager.RequestItem(PluginItemRegistration itemRegistration, Type type, IPluginItemStateTracker stateTracker)
2013-11-09 09:53:23.003020 : ServiceRegistration: Could not register dynamic service with id 'MovieTheMovieDbMatcher'
2013-11-09 09:53:23.008020 : PluginManager: Trying to activate plugin 'ConfigurationManager' (id '1aff4467-64b0-4ca1-af28-9aedf3525bce')
2013-11-09 09:53:23.008020 : PluginManager: Plugin 'ConfigurationManager' (id '1aff4467-64b0-4ca1-af28-9aedf3525bce') activated.
2013-11-09 09:53:23.008020 : PluginManager: Trying to activate plugin 'SQLiteDatabase' (id 'cdfc897c-dbfc-4815-8135-eea1e4628623')
2013-11-09 09:53:23.013020 : PluginManager: Plugin 'SQLiteDatabase' (id 'cdfc897c-dbfc-4815-8135-eea1e4628623') activated.
2013-11-09 09:53:23.038020 : SQLiteDatabase: Database Filename: 'Datastore.s3db' (Default Database Filename: 'Datastore.s3db')
2013-11-09 09:53:23.038020 : SQLiteDatabase: PageSize: 4096 Bytes (Default PageSize: 4096 Bytes)
2013-11-09 09:53:23.043020 : SQLiteDatabase: CacheSize: 65536 pages = 262144KB (RAM: 16062MB, Default CacheSize: 262144KB)
2013-11-09 09:53:23.043020 : SQLiteDatabase: LockTimeout: 30000ms (Default LockTimeout: 30000ms)
2013-11-09 09:53:23.043020 : SQLiteDatabase: Initialization Command: 'PRAGMA locking_mode=EXCLUSIVE;PRAGMA wal_autocheckpoint=32768;PRAGMA temp_store=MEMORY;' (Default Initialization Command: 'PRAGMA locking_mode=EXCLUSIVE;PRAGMA wal_autocheckpoint=32768;PRAGMA temp_store=MEMORY;')
2013-11-09 09:53:23.053020 : SQLiteDatabase: URI used in connection string: 'file:///C:/ProgramData/Team%20MediaPortal/MP2-Server/Datastore.s3db?cache=shared'
2013-11-09 09:53:23.073020 : SQLiteDatabase: Connection String used: 'fulluri="file:///C:/ProgramData/Team%20MediaPortal/MP2-Server/Datastore.s3db?cache=shared";version=3;binaryguid=True;default timeout=30000;cache size=65536;journal mode=Wal;pooling=False;synchronous=Normal;foreign keys=True'

I need to explain the other exceptions before: we had an issue that the "auto attach" CustomAction was not able to use localization, because the plugins were not yet inialized. So I changed the action to init the plugins, although this is not really necessary. Until now I ignored the "hidden" exceptions caused by missing services inside installer log.

 

[MJGraf]

Hm, I never had a look at what happens in the installer. Do we actually startup the plugins separate from the MP2 Server service? I thought the installer just copies all the files and when it is finished with copying, it starts the server service...
Will try the installer, but it may take a while...

 

[MJGraf]

Just a thought:
I haven't managed to dig deep into the installer, but in the CustomActions class I see that we startup the plugin manager in maintenance mode, i.e. plugins should not be auto activated. @morpheus_xx: Is this still true? I'm asking because in the log above it says a few times "PluginManager: Trying to activate plugin ...", which should not be the case in maintenance mode.
This should currently not have any influence on SQLiteDatabase, because it doesn't have a PluginStateTracker (no database plugin has such). SQLiteDatabase is registered as a plugin service and, hence, all the initialization is done in the constructor of SQLiteDatabase, which means it is executed in any case when the plugin is enabled - even if it is not activated.
If we still use maintenance mode (hence the question above), and my initialization code for SQLiteDatabase (currently in the constructor of SQLiteDatabase) is a problem for the installer (although I don't really know why), I could implement a PluginStateTracker for SQLiteDatabase and execute the initialization code in the Activated() method. That way this code would not be executed during install (assuming we use maintenance mode), but only on a real startup of MP2 Server Service.

 

[MJGraf]

Another thought (sorry can currently only read code, don't have my dev computer to test..)
In customactions.gethomeserverdata the messagebroker is never disposed (which is quite bad as messagebroker starts a separate thread which is never stopped without disposing...)

 

[MJGraf]

ok, just did an install on a clean Win7 VM. Can confirm that install hangs - and most likely it is SQLiteDatabase, which makes it hang.
What I suppose is the following:
When the installer hangs, the MP2 Server Service is already started. This means that the database is already opened by the MP2 Server Service process. The installer then instatiates its own pluginmanager in its own process and enables all the plugins. This means that we more or less have all the plugins loaded a second time, just in another process. For SQLiteDatabase this means that a second process tries to open the same database - and this doesn't work because for performance reasons we open the database in exclusive mode in the MP2 Server Service process and so the installer process can't open it a second time.
It should be easy to make a workaround for this, e.g. do not open the database if we are starting up in maintenance mode (that way the installer, which uses maintenance mode, will not try to open the database). But the question is whether it is really a good idea to more or less start up MP2 Server a second time in the installer process. I'm sure that over time there will be more plugins that don't like being executed twice on the same machine in two separate processes.
@morpheus_xx: Any thoughts on this? Do we really need such a big part of MP2 Server started twice? Or is it possible to delay the start of the regular MP2 Server Service process until after we have called the CustomActions to make sure that at least we don't have two MP2 Servers running at the same time?
Michael

 

[MJGraf]

oh and by the way: If you stop the MP2 Server Service via taskmanager while the installer seems to hang, the installer will finish gracefully. You can start the service again immediately after the installer has finished...

 

[MJGraf]

Ok, I take your "like" as a "we will make sure that MP2 Server Service is only started after the whole installation process is finished so that we do not have two instances of PluginManager started at the same time", Morph
If that's not the case and you want me to patch the SQLiteDatabase accordingly, please let me know. I may have some free hours for programming over the next weekend...

 

[morpheus_xx]

The original issue that was fixed is: https://issues.team-mediaportal.com/browse/MP2-155?jql=project = MP2 AND text ~ localization

I also could imagine another way to resolve the SQLite installer issue. Because we don't really need plugins to be enabled, but we need their localization, maybe we could modify the plugin manager startup process to stop after it has built all registration items (language, skin, ...).
@offbyone, do you think this is possible? (as you are just working in this part )