No guide info added due to a MYSQL deadlock (1 Viewer)

doskabouter

Development Group
  • Team MediaPortal
  • September 27, 2009
    4,566
    2,938
    Nuenen
    Home Country
    Netherlands Netherlands
    Hi,

    Not sure if this is better put in the WEBEPG section (because that's the plugin that is acting up) but over here it might get somewhat more attention...

    Sometimes I have a lot of info missing in my tv-guide.
    Checking the logs, it is because Mysql runs into a deadlock ( MySql.Data.MySqlClient.MySqlException: Deadlock found when trying to get lock; try restarting transaction)
    This is after quite a number of successfull channels, and after that error no other channels are added.

    I more or less found the cause of it, it happens when the provider (tvgids.tv in this case) suddenly decides to not have programdata for a given channel anymore.
    The combination of nothing new and a bunch of programs still in the epg-table seems to trigger this deadlock.

    Is anyone else having this kind of issues?
    Of course it's better to fix this problem, but I already checked the code and (apart from deadlocks in general to be difficult to fix) it seem to be not trivial to fix this.
    If I'm the only one having these kind of issues I tend to go and delete the programs from the offending channel and hope it doesn't happen too often, but if it would benefit others too I might take a stab at fixing the root-cause.
    Let me know what you think...
     

    doskabouter

    Development Group
  • Team MediaPortal
  • September 27, 2009
    4,566
    2,938
    Nuenen
    Home Country
    Netherlands Netherlands
    hm, didn't got any replies in almost a year. I guess its the wrong area after all...
    perhaps @mm1352000 has any insights as a tv-service guru?
    It's running on MYSQL 5.6 with all the settings that came with it when installing the tvservice way back.
     

    Attachments

    • TVService.zip
      256.5 KB

    sortie

    Portal Member
    April 7, 2021
    6
    3
    Home Country
    Sweden Sweden
    I am trying to resolve a similar problem, described here:

    Epg does not open at the last watched channel + epg grab stuck if not manually deleted first

    Webepg (mysql...) deadlock while grabbing, usually after a few successful channels but not necessarily the same. Stuck at writing in the grabber status window and same error log as you get. In my case, two separate installations (1.24) worked perfectly fine until I reinstalled (after full cleanup) with 1.27. Tried updating to 1.29 without any improvement and finally went the bothersome way to clean everything out again and reinstalled 1.24 in one system two days ago - only to see that the problem now persists.

    What I have discovered is that grabbing works flawlessly in both systems if the epg is manually deleted first (through manual control/refresh dvb epg). Starting with an empty db, the grabber never fails - perfectly repeatable. If not, it deadlocks every time. Also tried dropping/rebuilding mptvdb in the other system, with no improvement.

    Suppose I will have to file a bug report to see what could be done about this really annoying problem as I have been manually deleting/updating my epgs for half a year now.

    Could you or anybody else btw tell in which MP version the manual control section (in tv server configuration) was introduced?
     

    doskabouter

    Development Group
  • Team MediaPortal
  • September 27, 2009
    4,566
    2,938
    Nuenen
    Home Country
    Netherlands Netherlands
    Since switching back to webepg I'm suffering of it too again (and pretty consistent, I used to have it only when a channel stopped having epg info on the website, which was so sporadically that I didn't bother trying to fix it).
    I'll try if I can find anything to finally fix this.

    If memory serves me right, this could also be prevented by using a different storage engine but I never tried that tbh.

    And manual control? That has been there for ages... over 16 years: MediaPortal/MediaPortal-1@aad9ce6
     

    CyberSimian

    Test Group
  • Team MediaPortal
  • June 10, 2013
    2,849
    1,771
    Southampton
    Home Country
    United Kingdom United Kingdom
    finally went the bothersome way to clean everything out again and reinstalled 1.24 in one system two days ago - only to see that the problem now persists.
    When you say that you cleaned everything out, did that include MySQL? If it did not, you might try the following (note: I have no idea whether this will solve your problem):
    • Use the MP Uninstaller to uninstall MP.
    • Use the MySQL Uninstaller to uninstall MySQL.
    • Ensure that Windows Explorer is set to show hidden files and folders.
    • Delete from "Program Files" and "ProgramData" all folders that have "MediaPortal" in the name .
    • Delete from "Program Files" and "ProgramData" all folders that have "MySQL" in the name.
    • Reboot.
    • Install MP (which will also install MySQL). You can decide for yourself which version of MP to try (1.24 or 1.29).
    • Pray that it now works correctly.
    -- from CyberSimian in the UK
     

    sortie

    Portal Member
    April 7, 2021
    6
    3
    Home Country
    Sweden Sweden
    This is the danger of chross-threading - well, I had two separate subjects in my original thread...guess I'll be better off here for now.

    I did describe my systems in the other thread, though. The main setup (server/client + client) was updated with a new system hdd and fresh (no cloning) win10+mp1.27. I suppose it does not get more clean than that but it still had the problem from first startup. This system had a working 1.24 installation on the old drive.

    The other system is on a laptop for use in the cabin etc and what I am experimenting with right now because it is not busy all the time. Followed the mp (full) uninstall/cleanup route which included a login/confirmation for mysql cleaning, iirc. I did not delete any folders/files/reg entries manually, though. Same problem here - the previous 1.24 installation grabbed without any problems, the new 1.27 deadlocks (if epg is not deleted manually).

    A non-educated guess is that this behaviour points against a db delete problem (because everything written is correct - even the channels active when webepg status stalls at writing). Never had part of a period for stalled channels missing or corrupted. It really feels like a problem when deleting or handling something in the switch between just written and upcoming grabs. But as you have already guessed, I'm no db pro.

    Because of this I have now tried a grab after unchecking "delete old programs before import" (in tv server config/webepg/general/global settings) and guess what: it worked! Never tried this because of the recommendation to keep checked. I noticed that the guide became unresponsive during grabbing but hey, I usually only let the systems wake up and do this work at night.

    Will try this setting with the main system as well and report back (before further reinstall work) after a few automated grab cycles (will take a week or two). Doskabouter - would you mind trying this in your system?

    Edit: and thank you (doskabouter) for the info about manual control. Some way I have managed to miss it - since 2006, or so :). Guess I just never looked outside the webepg section.
     
    Last edited:

    doskabouter

    Development Group
  • Team MediaPortal
  • September 27, 2009
    4,566
    2,938
    Nuenen
    Home Country
    Netherlands Netherlands
    I'll try and see if that fixes it (or the error at least)
    Not sure what drawbacks unchecking that option gives (one thing that springs to mind is if epg-data is changed for a given day, but for my channels that is so rare that i can't reliably say if it works out ok or not when unchecked)
     

    doskabouter

    Development Group
  • Team MediaPortal
  • September 27, 2009
    4,566
    2,938
    Nuenen
    Home Country
    Netherlands Netherlands
    hm just checked and no errors for today's import... so if Im going to uncheck it won't give me a conclusive result...
     

    CyberSimian

    Test Group
  • Team MediaPortal
  • June 10, 2013
    2,849
    1,771
    Southampton
    Home Country
    United Kingdom United Kingdom
    Not sure what drawbacks unchecking that option gives (one thing that springs to mind is if epg-data is changed for a given day,
    For the DVB EPG (the EPG that is broadcast embedded within the TV signals), there is an Always try to update existing entries setting. My understanding of this setting is that an EPG entry will be added to the EPG database only if that timeslot is empty. The timeslot is empty when the EPG for the day that is seven days in the future is added to the TV signals at midnight, but thereafter the timeslot is not empty. So any updates to that day's EPG that occur on subsequent days will be ignored.

    For the UK DVB EPG, it is quite common for certain types of programme to have generic descrptions initially, and for these to be changed to more-specific descriptions nearer the day of broadcast. The description in the EPG database will not be updated if Always try to update existing entries is disabled.

    One might speculate that Delete old programs before import for WebEpg has an effect similar to Always try to update existing entries for DVB EPG.

    -- from CyberSimian in the UK
     

    doskabouter

    Development Group
  • Team MediaPortal
  • September 27, 2009
    4,566
    2,938
    Nuenen
    Home Country
    Netherlands Netherlands
    I might have found something...

    Stacktrace of deadlock points to this line: MediaPortal-1/DatabaseEPGDataSink.cs at 9b5dbb1c48d3f6435bf3120fee092971febcad7f · MediaPortal/MediaPortal-1
    but analyzing further I see that when calling MediaPortal-1/DatabaseEPGDataSink.cs at 9b5dbb1c48d3f6435bf3120fee092971febcad7f · MediaPortal/MediaPortal-1 the inserting of programs is done in a separate thread to increase performance.

    So removing programs and inserting programs of different channels from different threads can lead to deadlocks (something rather technical explanation given by mysql but I still think this shouldn't be happening at all)

    Analyzing further I see that after or before inserting of programs this MediaPortal-1/BusinessLayer.cs at 6691182eefdbec7ef1c5dd349cb45377e69946c2 · MediaPortal/MediaPortal-1 is executed (in the same thread as the inserts).

    Summarizing: before inserting programs all old programs are removed for specific channels, and while inserting, old programs are removed for all channels...

    Solution seem to deleting the first removeoldprograms loop (as it is a: running in a different thread, and b: completely superfluous).
    Testing that myself now, dll's (for tv service 1.29) attached so others can test too

    Edit: first day grabbing went without errors!
     

    Attachments

    • webepg.zip
      48 KB
    Last edited:

    Users who are viewing this thread

    Top Bottom