BUG: large (3000+ episodes) NEW install imports FAIL (1 Viewer)

[gamejester]

OK, this has been a difficult one to track down and I still so not have the exact course but quite a few people have seen this and it was driving me mad.

There is a longer thread on the testing and stuff here, but some of the info has now proven to be wrong
https://forum.team-mediaportal.com/my-tvseries-162/not-all-episodes-shown-recognized-38510/index2.html

CONFIG:
MP 1.0 RC1 + SVN
myTVseries 2.0 + SVN 312
NO mediainfo.dll (to speed up testing)

testing has shown that this issue has existed for a long time as I went back through loads of version to try to find the last time it worked reliably; finally coming to the realisation that the issue now exists for me due to the expanded size of my episodes store. Also the problem exists on both XP and Vista.

Problem:
If you do a fresh install, point myTVseries to a single folder with 3000+ episodes, or a series of folders spread accross numerours mapped drives with 3000+ shows things behave oddly.

1. It takes ages to match each local episode in the metadata retrieval section (around 30 seconds per series).
2. At the end of the import a lot of series will only have episode matched.
3. The database will have lots of - (minus) numbers for SeriesID in local_episodes (FYI this usually indicates you have select 'ignore this season', but I have not done this and if I had ALL of the season would be marked like this not every episode except the first one)

Consistent:
This is not 100% consistent, when I was testing all the previosu version I would occasionally get one that scanned the whole dir in one go; only later to return to it to test again and find it no longer worked.

Temp Solution:
Split your shows into folder and subfolders with only around 1000 episodes. As I use DFS to make all my shows appear as though they are in the same folder structure I just used the real folders instead.
Add 1 folder top level folder at a time to MP and do an import.
close the config tool.
Open SQLite Database browser and browse data, table local_episdoes (check there are no - SeriesID). - I found I need to do this step or somtimes the import will fail.
Add the next fodler and import.
This seems to work.
You will know if it is going to work or not as when working correctly it take around 2 second to recognise each show in the metadata retrieval section.

Full Solution:
Ask Inker (although he has said he is taking a break) or other coder to take a look at the memory constraints of the data container (array of something) that is being used as a temp store when initiall dir scan is run to see why this would occur.

or it could have something to do with the interaction db (connection left open or something maybe) as in one test I added each of the smaller DIRs individually and it failed when I added the second one and did an import. When it fails like this the episode count says 1 and when you look in Details only 1 show has a + next to it. If you close config and re-open Details looks OK now and all the shows you imprted the fist time are ok, but all the shows from the second import only have 1 episode (or most of them!)

 

[ltfearme]

Hi Gamejester, I managed to find a Fake Episode Creator. I will modify it to create fake eps out in a root directory so I can try reproduce this issue you are experiencing.

Will let you know how I go.

Damien

 

[gamejester]

Thanks a lot.

I notice you have been doing some dev work on myTVseries lately (no disrespect to Inker, but he said he was taking a break), so if you can turn on any extra logging, create a test harness or have any ideas on the root cause let me know.

I spent 2 days solid doing various test, but to be honest I have not tracked down a reliably re-producible cause that I am happy with; the finding thus far are just 'most likely' to fail but not 100% of the time!

I have resisted the temptation to fire up VS myself, mainly because I think it will be a mine field in there, but maybe if someone was willing to talk me though setting up the project I might give it a whirl! (FYI I do .net c# asp.net web stuff at work, but keep that quiet!) We know when the issue manifests it’s self a negative local_episode number is written, so if you knew where to set break points and watchers in the code I could run that through. I Can also establish secure RDP or VNC SSL tunnelled connections to a troubled box if you want to see the issue first hand. PM me.

This plugin is the only reason I use MP and it is great, but lately I have become very frustrated (even looked at LinuxMCE the other day!)

At present I am working on a new skin (duno why if v2 comes out it will be useless) but t is more of a stabilising project I think for myTVseries plugin (as V1 is likely to be around for a long time if the dev on v2 is anywhere as long as v1!). I have noticed over the year(s) that there are quite a few things that do not work as they should, would be good to lock them down. So my new skin is heavily animated and uses all event triggers to highlight which ones are not correctly firing and when. If we fix up all these bits then I am sure the knock on effect will fix the other screen issues like when you resume the PC, fan art disappearing, etc.....(+ it will look really cool)

 

[Inker]

I think we would all be very happy if you could overcome you're restistance to the temptation to fire up VS yourself If you're insterested I can give you svn write access and talk to you about the code (msn or irc or something).

Let me know if you're interested

 

[ltfearme]

I can reproduce the issue, I created 6000+ episodes using that tool

It seems to only occur when the Database first does not exist and you run the import imediately after.

Steps:
1. Delete *.db file from disk
2. Open Configuration Tool (db is created)
3. Enter in Import Path
4. Start Import

When it tries to Identity unknown series online, it takes roughly 60secs and series are not imported as already described.

It all seems ok if you do the following:

Steps:
1. Delete *.db file from disk
2. Open Configuration tool (db is created)
3. Enter in Import Paths
4. Close Configuration tool and Configuration.exe
5. Start Configuration tool
6. Start Import

Unknown Series are identified immediately and everything is imported OK

Looks like something to do with how the database is initialized.

------------------------------------------------------------------------------------

It appears the problem is that on initial creation of the database there is an exception when it tries to create the indicies:

m_db.Execute("create index if not exists epComp1 ON local_episodes(CompositeID ASC)");
m_db.Execute("create index if not exists epComp2 ON local_episodes(CompositeID2 ASC)");
m_db.Execute("create index if not exists seriesIDLocal on local_series(ID ASC)");
m_db.Execute("create index if not exists seriesIDOnlineEp on online_episodes(SeriesID ASC)");

It fails with an exception the first time because those particular tables do not exist. This is why it takes forever to execute a query with 3000+ rows. Just ends up returns 1 row with data as you see when it doesn't work.

If I set a breakpoint in the code before it tries to match the unknown series/episodes when those indicies don't exist it takes 60+ secs to return when using the SQL Lite Browser but it will return the correct data.

If I step through the code it will return with no error and wont come back with anything (*just gives up I guess*).

Guess the fix to implement is ensure that those indicies are created when the tables are created.

------------------------------------------------------------------------------------

Ok, I committed a new binary to SVN. Hopefully this was the cause of your problem.

Damien

 

[gamejester]

Good work, will give it a go now and report back.

**********
* FIXED *
**********

Thanks a lot, good work. A knock on effect is that as well as the xml info reading being a lot quicker, it also seems that the medi info reading stage seems quicker.

Whilst you are on a roll I will log another couple of bugs, but in new threads.

Inker, yeah I will probably get involved with the code soon as there are new things I want MP to do. Like I said this plugin is what I use 95% of the time so it gets all my focus. The order of stuff I am currently/planning working on is:

1. New skin - heavily animated and uses ALL the visible logic in this plugin. There are a lot of bugs here so want to get them all classified before any effort is spent to fix them up. (this is when I will probably jump in!)

2. Touchscreens. MP is cool and all, but is all TV driven. We need the concept of a GUI remote for MP, like LinuxMCE has orbiters. I propsed xPL for this and it looks like this pluin has added that funcionality.
https://forum.team-mediaportal.com/support-119/cybrdisplay-external-status-plugin-34521/
What I want is my projector to me connected to a pc to run MP and play videos, but an on box running website that also reads the db info and creates dynamic html that can be displayed on web tablets (mokia 770, ipod touch, etc...) this also need real time info, now playing, and control of MP.
Using xPL for this is great as it is a generic transport, so in theory you can conrol every MP in the house over it.
It also means you can incorporate home automation in MP easily, so you culd expand the plugin, or more likely create another pluin that displays xPL info in MP and fires xPL messages on MP events. (ie play video, fires dim lights, etc....)

3. Users - the concept of each user having their own remote or sign in to MP and seeing their own watched flags. This may need more thought, be good if it worked across whole of MP not just TVseries. Perhaps a move towards the MS sql db held on the TVserver would be the way to go, but I suspect speed issues may be a factor, duno.

4. whole house video sync. So at the moment we have a TV database and a yTVseries db. Everything from the TV db can be streamed, so you can watch the same thing on 2 different tvs at the same time in sync. It would be cool if we could do the same for TVseries shows. It would also be cool if the TV you record was dispayed in the same way as that in myTVseries. VLC has the ability to stream any content and is open source, so there may be an option to hook in there for this sort of thing.

If we had all that in MP I would be very happy

 

[coopers]

So, can I confirm that this issue is fixed in the latest SVN 2.0.3296?

I have this issue, and if this fixes it, I'll upgrade!

 

[Guzzi]

I can reproduce the issue, I created 6000+ episodes using that tool

It seems to only occur when the Database first does not exist and you run the import imediately after.

Steps:
1. Delete *.db file from disk
2. Open Configuration Tool (db is created)
3. Enter in Import Path
4. Start Import

When it tries to Identity unknown series online, it takes roughly 60secs and series are not imported as already described.

It all seems ok if you do the following:

Steps:
1. Delete *.db file from disk
2. Open Configuration tool (db is created)
3. Enter in Import Paths
4. Close Configuration tool and Configuration.exe
5. Start Configuration tool
6. Start Import

Unknown Series are identified immediately and everything is imported OK

Looks like something to do with how the database is initialized.

------------------------------------------------------------------------------------

It appears the problem is that on initial creation of the database there is an exception when it tries to create the indicies:

m_db.Execute("create index if not exists epComp1 ON local_episodes(CompositeID ASC)");
m_db.Execute("create index if not exists epComp2 ON local_episodes(CompositeID2 ASC)");
m_db.Execute("create index if not exists seriesIDLocal on local_series(ID ASC)");
m_db.Execute("create index if not exists seriesIDOnlineEp on online_episodes(SeriesID ASC)");

It fails with an exception the first time because those particular tables do not exist. This is why it takes forever to execute a query with 3000+ rows. Just ends up returns 1 row with data as you see when it doesn't work.

If I set a breakpoint in the code before it tries to match the unknown series/episodes when those indicies don't exist it takes 60+ secs to return when using the SQL Lite Browser but it will return the correct data.

If I step through the code it will return with no error and wont come back with anything (*just gives up I guess*).

Guess the fix to implement is ensure that those indicies are created when the tables are created.

------------------------------------------------------------------------------------

Ok, I committed a new binary to SVN. Hopefully this was the cause of your problem.

Damien

Hi Damien,
I made another test with latest SVN; created lots of series to get lots of onlineepisodes and a big db. In my test I had more than 20.000 onlineepisodes registered in the DB. Opening the table in SQlitebrowser takes between 10-30 seconds. When using this db with the plugin, it only shows 1 episode the first time I enter a serie - going back and entering again gives all data. So - is it possible, that the plugin has timeouts that prevent it to properly display the first time db is requested? Other restrictions like memory, etc? Is it possible to loose those restrictions e.g. by increasing internal memory or arrays? Anything else I can do to help?
Thanks for the great plugin and the good work!