Recordings are namend "manuel" when EPG importer is running (1 Viewer)

[disaster123]

Hi!

When an importer like XMLPLugin or TvMovie is running - recordings could be named manuel instead of the correct / real name.

Idea:
XMLPLugin / TvMovie PLugin should make use of table locks or use transactions for deleting AND adding

Stefan

 

[jameson_uk]

When an importer like XMLPLugin or TvMovie is running - recordings could be named manuel instead of the correct / real name.

Have had the same but I think this is a huge amount of work to implement. The problem is that the importers actually sort out the import and the EPG update is done by a different process.

In order to cater for changes most importers do a delete of all data for the period covered by the new import and the re-import.

For me the problems have been because the machine has gone into standby between the importer finishing and the actual update of the EPG data.

https://forum.team-mediaportal.com/xmltv-137/stay-awake-whilst-loading-epg-73426/

might help as it should mean that once fixed that if you update the EPG early in the morning it should be fully populated when recording start later on.

 

[disaster123]

My server runs 24/7 so it doesn't go into standby. I'm using the tvmovie epg importer. And everytime it runs and a recording start in parallel the naming fails. Perhaps another approach would be to use a temp table and then rename the temp table to the final one.

 

[jameson_uk]

My server runs 24/7 so it doesn't go into standby. I'm using the tvmovie epg importer. And everytime it runs and a recording start in parallel the naming fails. Perhaps another approach would be to use a temp table and then rename the temp table to the final one.

I am guessing that big changes are not going to be made to MP1 and your original answer is certainly the way forward for MPII (the delete and insert should be in one transaction so there is no loss of data to the user). The problem (I think) is that the delete and insert are called at very different times so trying to change this would be quite a big thing.

I schedule my xmltv file (90 odd files and two weeks worth of data) to update at 5am as I am never likely to record anything at that time anyway. End to end it takes something like 10-15 minutes to fully reload the EPG data so in the interim could you not just schedule this at a time when you are not going to be recording?

 

[disaster123]

In the tvmovie importer i can only select import every 12 hours, 24 hours, ... no specific time.

 

[jameson_uk]

In the tvmovie importer i can only select import every 12 hours, 24 hours, ... no specific time.

ahhhhhhh ok, ignore me

 

[arion_p]

My server runs 24/7 so it doesn't go into standby. I'm using the tvmovie epg importer. And everytime it runs and a recording start in parallel the naming fails. Perhaps another approach would be to use a temp table and then rename the temp table to the final one.

Using a temp table won't help as at some point new program entries will have to be merged with existing ones (possibly replacing some). Building a temp table and then renaming would only work if you decide to discard all old data and you use only one type of grabber. Also renaming tables is not necessarily a "cheap" operation in all RDBMS's.

I am guessing that big changes are not going to be made to MP1 and your original answer is certainly the way forward for MPII (the delete and insert should be in one transaction so there is no loss of data to the user). The problem (I think) is that the delete and insert are called at very different times so trying to change this would be quite a big thing.

The delete can be moved but the workload of deleting 14 days of programs for 100 channels (~= 20*14*100 = 28000 records) and then inserting the same amount of data in a single transaction would easily bring the system down on its knees for several seconds (if not minutes) at which time practically all queries to the programs table will be blocked (= no recordings will start) and even timeshifting will at least suffer because of heavy disk usage.

The only real solution would be to progressively replace existing programs with new ones using multiple small transactions that leave the programs table in a consistent state in between. There are several cases of mismatches between new and old programs that have to be taken into account so that you end up with the correct programs in the end and not introducing holes in the EPG. I have not yet figured out how to do this.

 

[disaster123]

ok i see not easy

But why deleting and inserting all channels in one transaction?

What about something like:

LOCK TABLE epg;
BEGIN TRANSACTION;
DELETE FROM epg WHERE channelid= X;
INSERT INTO EPG VALUES (x,y,X), (x2,y2,X), (x3,y3,X);
END TRANSACTION;
UNLOCK TABLE epg;
sleep some time
next channel / transaction

20*14 => 280 inserts should be doable without any problem

 

[arion_p]

Well that is pretty much what WebEPG does right now, only it does not do it in a transaction, because the transaction and all database access has to be controlled by TvBusinessLayer (right now only inserts are controlled by TVBusinessLayer) in order to be tight and that in turn means that all other grabbers will need to follow the same pattern (which they don't right now). This is a good first step. I have good knowledge of WebEPG code, some knowledge of XmlTV code but no knowledge for other grabbers (TvMovie and others?), so I cannot do the necessary changes to those (I don't even know if it is possible).

 

[disaster123]

mhm so the tvbusinesslayer needs to be extend first?