1.30.0 5111: Parallel DVB EPG (1 Viewer)

framug

Super Moderator
  • Team MediaPortal
  • January 31, 2005
    5,884
    1,956
    South of France
    Home Country
    France France
    • Thread starter
    • Moderator
    • #41
    Hello CyberSimian,

    THX for your notice on it.

    I had expected to see a new "TV Server" setting in the idle-grabber section called something like:
    Maximum number of tuners to use concurrently for EPG grabbing
    Well, I will explain more for this after but, for the location, if you talk about this TV Server part :

    zz1.png

    Why should we put this setting here (red rectangle), where grab is also possible when record/timeshift (green rectangle) ?

    If this setting is disabled, TV Server uses only one tuner, starting with the highest priority tuner.
    If this setting is enabled, TV Server uses all tuners, starting with the lowest priority tuner.
    So this setting performs some of the function that I suggested in my previous point.

    Exactly, that's why it performs for :

    The default for this setting would be 1, which means that there should be no change from behaviour in previous releases. But users who live in countries that can benefit from multiple EPG grabbers (and whose HTPCs are capable of sustaining the extra load) could specify a value greater than 1.

    Elsewhere in "TV Server Config", the value 0 is used to denote "all that are available". As there is already an enable/disable setting for the idle grabber, I would suggest that the new setting should accept 0 to mean "use all available tuners".

    When disabled, TV Engine is working like before, consider it as "1" value then, no change in TV Server operation.
    When enabled, consider it as "use all available tuners".

    Why number of tuners to use concurrently for EPG grabbing is not here ?
    Simply because, I want this setting work "alone" and not interfere with others settings, if number of tuners is used, we have to take care about other settings (idle, recording/timeshifting, grab only for channel, etc...).
    Of course, it is possible but, it will complicate much more the code than actually, is it worth it ?

    And finally, for the lowest priority cards, MP use priorities for recording/timeshifting then, same tuners are often used.
    By doing this, it can use tuners with few sollicitation then globally, the load is better balanced between differents tuners.

    For example, before my 8 tuner cards, I had 2 dual tuners cards.
    1/ the same as you, the 2000i
    2/ a Nova-T-500 dual tuner card. (that many user can see a signal strength to "0" value but, this is another story...)

    The Nova card made many discontinuities and video/audio was "unwatchable" that's why, I decided to put these 2 tuners as lowest priorities.
    Even if channel were unwatchable, Nova-T-500 was able to grab EPG without fail, leave 2000i tuners free for record/timeshift.

    CU.
     
    Last edited:

    CyberSimian

    Test Group
  • Team MediaPortal
  • June 10, 2013
    2,849
    1,771
    Southampton
    Home Country
    United Kingdom United Kingdom
    Why should we put this setting here (red rectangle), where grab is also possible when record/timeshift (green rectangle) ?
    Can the timeshift grabber grab from two or more channels at the same time? I had assumed that it could not (so the new setting would apply only to the idle grabber). However, now that I think about it, I don't actually know how the timeshift grabber behaves :confused:. I will try this tomorrow and see what happens. :D

    Why number of tuners to use concurrently for EPG grabbing is not here ?
    Simply because, I want this setting work "alone" and not interfere with others settings
    That's OK (y). Allowing the user to specify the maximum number of tuners to use for grabbing provides more capability for the user, but I am not the one implementing this! :D I understand your desire to keep the new capability separate from the existing capability (so that the one does not adversely affect the other).

    However, I still think that the setting should be named differently. The most important action of this setting is to allow multiple tuners to be used, so the name of the setting should indicate that. I realise that there is not much space available, so I would suggest a name something like this:

    Use all tuners or
    Use all available tuners

    I will do some controlled testing tomorrow and post the results. :)

    -- from CyberSimian in the UK
     

    framug

    Super Moderator
  • Team MediaPortal
  • January 31, 2005
    5,884
    1,956
    South of France
    Home Country
    France France
    • Thread starter
    • Moderator
    • #43
    Can the timeshift grabber grab from two or more channels at the same time? I had assumed that it could not (so the new setting would apply only to the idle grabber). However, now that I think about it, I don't actually know how the timeshift grabber behaves :confused:. I will try this tomorrow and see what happens. :D
    If you want to be sure of that please, take a look at all screenshots in page 3 of this thread. ;)

    However, I still think that the setting should be named differently. The most important action of this setting is to allow multiple tuners to be used, so the name of the setting should indicate that. I realise that there is not much space available, so I would suggest a name something like this:

    Use all tuners or
    Use all available tuners
    Renaming this setting can be done. :)
     

    CyberSimian

    Test Group
  • Team MediaPortal
  • June 10, 2013
    2,849
    1,771
    Southampton
    Home Country
    United Kingdom United Kingdom
    I will do some controlled testing tomorrow and post the results.
    Latest DLLs attached to this post.
    Testing methodology
    (1) I have attempted to run the test cases under identical conditions, but as each run takes around 30 minutes, it was not possible to run the test cases more than once (the HTPC was required for other purposes). Running each test case multiple times would be useful, as it would indicate how consistent the results are.

    (2) I have included the log files in the attached zip, but I took the liberty of using my logclean.rex script to delete all of the power-scheduler lines, as those lines create noise in the log file (but I have retained the original log files if you need them). I have included the screen shots in the zip file too.

    (3) I also wrote an epgstats.rex script to scan a log file and produce some summary statistics for it, to make it easier to compare the different test cases. I have included these statistics files (file extension TXT) in the zip file.

    (4) To compare the test cases, the following criteria can be used:
    • The duration of the grabbing phase
    • The duration of the database-update phase
    • The number of channels updated in the database
    • The number of programmes updated in the database
    • The total duration (grabbing + DB update + other)
    When a single grabber is used, the total duration is only slightly more than the sum of the grabbing phase and the database-update phase. But this does not apply when multiple grabbers are used, as the grabbers run in parallel (so for multiple grabbers, the sum of the individual parts exceeds the total duration).


    Test cases
    (1) Old DLLs, timeshift grabber, 1 channel selected, using these settings:

    1_channel_settings_(1).jpg 1_channel_settings_(2).jpg 1_channel_settings_(3).jpg

    These are the settings that I use on my production system (but this test was performed on my test system). The channel selected can be any 24-hour channel, but in my case it happens to be a radio channel. Note that screen shot 1 shows the settings for test (3) below (new DLLs); test (1) did indeed use the old DLLs.

    (2) Old DLLs, idle grabber, 1 channel selected (settings same as (1)).

    (3) New DLLs, idle grabber, 1 channel selected (settings same as (1)).

    (4) New DLLs, idle grabber, 6 channels selected, using these settings:

    6_channels_settings_(1).jpg 6_channels_settings_(2).jpg 6_channels_settings_(3).jpg 6_channels_grabbing.jpg

    There are 9 MUXes that serve my location in the UK, but 3 of them are too weak to receive reliably, so I have deleted those from the "tuning details" file, leaving me with 6 MUXes. Since I have 6 tuners, I selected one channel in each MUX so that each tuner could tune to a different MUX. The other settings are set appropriately for this. The screen shot shows only 4 channels selected; the other two are later in the list (off screen).

    (5) New DLLs, timeshift grabber, 1 channel selected (settings same as (1)).


    Summary of results
    Code:
                           Grab      DB Update  Total     Channels  Programs
    Test Case              Duration  Duration   Duration  Updated   Updated
    ---------------------  --------  ---------  --------  --------  --------
    old, t/shft, 1 chan     6m 33s    19m 35s    26m  8s    141      25825
    old, idle,   1 chan     8m 26s    19m 10s    27m 37s    141      25746
    new, idle,   1 chan    18m 56s    19m 28s    38m 27s    141      25828
    new, idle,   6 chans   15m 10s    3m - 7m    22m 12s    123      22420
    new, t/shft, 1 chan    18m 27s    19m  3s    37m 31s    141      25852
    ---------------------  --------  ---------  --------  --------  --------



    Observations
    (1) When a single grabber is used, the duration of the database-update phase is re-assuringly consistent, at around 19-20 mins on my system. No conclusions can be drawn from the database update time for multiple grabbers, since each grabber updates only a single MUX (instead of a single grabber updating all MUXes).

    (2) When a single grabber is used, the number of channels updated in the database is always the same (141). When multiple grabbers are used, this value drops to 123. It is not apparent why this value is smaller.

    (3) The number of programmes updated in the database is likely to vary slightly throughout the day. The UK EPG has 7 full days of EPG plus the remainder of the current day. It is probable that as each programme finishes broadcasting, its EPG entry is removed from the broadcast EPG carousel (to minimise datastream). As a result, the EPG received will shrink slightly thoughout the day, until midnight arrives when 24-hours of new EPG is added.

    The tests do show a small variation in the number of programmes updated in the database, although the number does not always get smaller as might be expected.

    However, the multiple-grabbers case shows a value that is around 3400 programmes smaller. Again, it is not apparent why this might be the case.

    (4) When multiple grabbers are used, the duration of grabbing is very similar for each MUX (at around 15 minutes).

    What is more concerning is that for the one-grabber case, the duration of grabbing is considerably longer when using the new DLLs compared to the old DLLs. For the one-grabber case, grabbing takes around 6-8 mins for the old DLLs, but takes 18-19 mins for the new DLLs. It is not apparent why this might be.

    (5) Despite the unexpectedly large grabbing duration for the multiple grabbers case, this case in fact has the smallest total duration :). On the other hand, it seems to have missed 18 channels and around 3400 programmes. :(


    Desiderata
    One test that I have not run is the test that corresponds exactly to the settings shown in post 35. I wll try this if it will provide some useful results, but I am not sure that it is appropriate for the UK. As far as I can see from the screen shots, every channel is selected. In my EPG in MP, I have 73 TV channels and 32 radio channels. If every one of those were selected, I have a suspicion that the grabbing would run for hours :eek: (because TV server would tune to each of the 105 channels in turn, taking 15 minutes to grab each channel, but using 6 tuners, and so taking 105*15/6 = 262 mins, i.e. over 4 hours :eek:).

    Let me know if you want me to try different combinations of settings or repeat some of the tests to see how consistent they are. :)

    -- from CyberSimian in the UK
     

    Attachments

    • epg_test_1.zip
      892.2 KB

    framug

    Super Moderator
  • Team MediaPortal
  • January 31, 2005
    5,884
    1,956
    South of France
    Home Country
    France France
    • Thread starter
    • Moderator
    • #46
    Waouh, this is A TEST ! (y)

    For the difference between duration longer and the 3400 programs skipped, the only thing I see is, at a time, one of your provider didn't diffuse the EPG (not available).
    Or may be at the moment, Windows was busy for other things ? EPG processes are lowest.
    Also, remember that signal is always varying.
    Because the new code doesn't skip some EPG, nor put a wait timer or something like that.
    I think even if some programs are skipped, depending on the refresh timer, programs could be updated at the next caroussel ?
    It seems you have CRC, in England (consuming a bit more).


    For the #35 post, in France, we have some encrypted channel that I don't need EPG. (You couldn't see that in previous screenshots), that's why you saw checked channels :

    zz1.png

    Again, THX for all your tests, at least, we can see there is no crash ! :D

    Regards.
     

    CyberSimian

    Test Group
  • Team MediaPortal
  • June 10, 2013
    2,849
    1,771
    Southampton
    Home Country
    United Kingdom United Kingdom
    When a single grabber is used, the number of channels updated in the database is always the same (141). When multiple grabbers are used, this value drops to 123.
    the 3400 programs skipped
    I have had a thought about this:

    When using the multiple grabbers, I enabled Grab EPG only for channels on same transponder. This was to prevent different grabbers each updating the same programme entries in the database (each programme entry needs to be updated only once). However, EPG entries will also be present for the 3 MUXes that I cannot receive, so this setting will cause TV Server to discard those entries. (y)

    When using the single grabber and grabbing from a single channel, I disabled that setting so that all programme entries are updated in the database. This may have caused TV Server to store those entries in the database, even though those entries were for channels that were not mapped to a card. :( (I don't know what logic TV Server uses in this situation, so this is merely speculation.)

    I have just looked in "TV Server Config", amd I have 21 channels that are not mapped to a card (these are the channels in the 3 weak MUXes). This is close to (but exactly the same as) the "missing" 18 channels, but this may be the explanation (possibly I miscounted somewhere :eek:).

    -- from CyberSimian in the UK
     

    framug

    Super Moderator
  • Team MediaPortal
  • January 31, 2005
    5,884
    1,956
    South of France
    Home Country
    France France
    • Thread starter
    • Moderator
    • #48
    When using the multiple grabbers, I enabled Grab EPG only for channels on same transponder. This was to prevent different grabbers each updating the same programme entries in the database (each programme entry needs to be updated only once). However, EPG entries will also be present for the 3 MUXes that I cannot receive, so this setting will cause TV Server to discard those entries. (y)
    Well, it's up to you but, I don't think it is more efficient.
    If you timeshift,
    If you enable grab only on same transponder, you will have this (at the begining, OFC, when DB is empty) :

    zz1.png

    If you disable it, you have this :

    zz2.png

    And, if you test refresh, you will see that, others times than first, updates DB are faster, that's why, "different grabbers each updating the same programme entries in the database" doesn't happen when programs are already filled in DB.
    Also, the benefit of disabling this setting, if you combine it by enabling "always try to fill holes" (I specify "for EPG" otherwise, others members could talk about it :LOL:) is, if you are lucky, having a more complete EPG if no program at a caroussel but program exist at another time.


    When using the single grabber and grabbing from a single channel, I disabled that setting so that all programme entries are updated in the database. This may have caused TV Server to store those entries in the database, even though those entries were for channels that were not mapped to a card. :( (I don't know what logic TV Server uses in this situation, so this is merely speculation.)

    I have just looked in "TV Server Config", amd I have 21 channels that are not mapped to a card (these are the channels in the 3 weak MUXes). This is close to (but exactly the same as) the "missing" 18 channels, but this may be the explanation (possibly I miscounted somewhere :eek:).
    Hmmm, you explain 21 channels that are not mapped to a card, what about the other card, are channels mapped on ?
     

    CyberSimian

    Test Group
  • Team MediaPortal
  • June 10, 2013
    2,849
    1,771
    Southampton
    Home Country
    United Kingdom United Kingdom
    you explain 21 channels that are not mapped to a card, what about the other card, are channels mapped on ?
    I think that my channel definitions in "TV Server Config" may be suffering from history. :D

    (1) Some channels that are in the 3 MUXes that I cannot receive are flagged in "TV Server Config" as Channel not mapped to a card (meaning that they are not mapped to any of my tuner cards).

    (2) Other channels also reside in the unreceivable MUXes, but they are not flagged in that way.

    I don't know how they ended up in this state, but my guess is that channels flagged as (1) have never been receivable by my system, whereas channels flagged as (2) were receivable in the past, but are no longer receivable. This is possible because previously I kept all 9 MUXes in the tuning details file, and I did attempt to tune those MUXes.

    This used to be successful, but the broadcasting authorities moved those 3 MUXes to frequencies that are more difficult to receive at my location, making reception borderline. The signal strength for them is variable, and so sometimes TV Server can detect them, but at other times it cannot (signal too weak). That is why I eventually decided to omit them completely (those MUXes contain only minor channels and +1 versions of other channels, so no great loss).

    My speculation is that if I delete all of the channels that are in the unreceivable MUXes, and then rescan, all of those channels would then be flagged as "not mapped to a card". Hmm, I might try that after I have created a system image (just in case it all goes horribly wrong :D).

    -- from CyberSimian in the UK
     

    framug

    Super Moderator
  • Team MediaPortal
  • January 31, 2005
    5,884
    1,956
    South of France
    Home Country
    France France
    • Thread starter
    • Moderator
    • #50
    LOL, history ! :)
    Few months ago, I had restarted from scratch, for testing differents things.
    It's time consuming but finally, at the end, a clean MP.xml, TVE settings up to date and, without older things (and more if you test or dev many things)... the gain is appreciable, MP seems to work better. ;)
     

    Users who are viewing this thread

    Top Bottom