[fixed] [MP2-819] MP2 Server crashes on ATSC OTA channel scan (1 Viewer)

morpheus_xx

Lead Dev MP2
  • Team MediaPortal
  • March 24, 2007
    11,600
    7,060
    Germany Germany
    Country flag
    Thanks for your logfiles!

    I think the issue comes from our TsWriter and an unsupported compression/encoding schema:
    [2019-12-30 10:47:40,885] [43a1340] [27c4] - LvctParser: descriptor, tag = 0xa0, length = 79, pointer = 99, end of descriptor = 178
    [2019-12-30 10:47:40,885] [43a1340] [27c4] - LvctParser: unsupported compression type or mode in DecodeString(), compression type = 0x0, mode = 0x20
    [2019-12-30 10:47:40,885] [43a1340] [27c4] - 0: 0x19

    [2019-12-30 10:47:40,885] [43a1340] [27c4] - LvctParser: unsupported compression type or mode in DecodeString(), compression type = 0x0, mode = 0x20
    [2019-12-30 10:47:40,885] [43a1340] [27c4] - 0: 0x19
    [2019-12-30 10:47:40,885] [43a1340] [27c4] - LvctParser: unsupported compression type or mode in DecodeString(), compression type = 0x0, mode = 0x20
    [2019-12-30 10:47:40,885] [43a1340] [27c4] - 0: 0x19
    [2019-12-30 10:47:40,885] [43a1340] [27c4] - LvctParser: unsupported compression type or mode in DecodeString(), compression type = 0x0, mode = 0x20

    After those lines the log ends, so the crash seems related to this.

    Can any of our @Developers who know this part can take a look please?
     

    morpheus_xx

    Lead Dev MP2
  • Team MediaPortal
  • March 24, 2007
    11,600
    7,060
    Germany Germany
    Country flag
    @rback
    I have done a minor modification to the string decoding logic inside the L-VCT parser. Can you please test the attached file?
    1. Stop the MP2-Server service (i.e. right click on Server Monitor in tray)
    2. Go to directory "C:\Program Files (x86)\Team MediaPortal\MP2-Server\Plugins\SlimTv.Service3". Rename the TsWriter.ax (or move it to other directory as backup)
    3. Extract the attached file into this folder
    4. Start the MP2-Server service (i.e. right click on Server Monitor in tray)
    5. Do a channel scan again
    6. Post new logfiles
     

    Attachments

    Last edited:

    mm1352000

    Development Group
  • Team MediaPortal
  • September 1, 2008
    21,571
    8,220
    New Zealand New Zealand
    Country flag
    @morpheus_xx
    I've had a quick look. In my opinion, the issue isn't directly caused by the unsupported compression/encoding scheme. Rather, it is caused by the fact that the channel's extended name has more than one segment.
    1. DecodeMultipleStrings() is not producing the correct output. Instead of producing <numberStrings> strings in <strings>, it is putting all the segments from all the strings directly in <strings>. In other words, for this particular example, it is producing 9 strings (which is the number of segments in string 0) instead of 1.
    2. The loop after DecodeMultipleStrings() is not combining names correctly. Intention is to combine the short name with the first extended name that is not the same as the short name, in the format "<extended name> (<short name>)".
    Fixing point (1) would require fixing DecodeMultipleStrings(). You could try this code:
    Code:
    void CLvctParser::DecodeMultipleStrings(byte* b, int length, vector<char*>* strings)
    {
      if (length < 1)
      {
        LogDebug("LvctParser: invalid multiple strings structure length = %d", length);
        return;
      }
      try
      {
        int numberStrings = b[0];
    
        //LogDebug("LvctParser: parse multiple strings, number of strings = %d", numberStrings);
        int pointer = 1;
        for (int i = 0; i < numberStrings && pointer + 3 < length; i++)
        {
          unsigned int iso639LanguageCode = b[pointer] + (b[pointer + 1] << 8) + (b[pointer + 2] << 16);
          pointer += 3;
          int numberSegments = b[pointer++];
          //LogDebug("LvctParser: string %d, number of segments = %d", i, numberSegments);
    
          vector<char*> segments;
          int segmentCharCount = 0;
          for (int j = 0; j < numberSegments && pointer + 2 < length; j++)
          {
            int compressionType = b[pointer++];
            int mode = b[pointer++];
            int numberBytes = b[pointer++];
            //LogDebug("LvctParser: segment %d, compression type = 0x%x, mode = 0x%x, number of bytes = %d", j, compressionType, mode, numberBytes);
            if (pointer + numberBytes > length)
            {
              LogDebug("LvctParser: invalid string length %d in multiple string structure, pointer = %d, number of bytes = %d, structure length = %d", pointer, numberBytes, length);
              return;
            }
    
            char* segment = NULL;
            DecodeString(&b[pointer], compressionType, mode, numberBytes, &segment);
            if (segment != NULL)
            {
              segments.push_back(segment);
              segmentCharCount += strlen(segment);
            }
    
            pointer += numberBytes;
          }
    
          // combine the segments to produce the final string
          if (segments.size() == 0)
          {
            continue;
          }
          if (segments.size() == 1)
          {
            strings->push_back(segments[0]);
            continue;
          }
    
          char* string = new char[segmentCharCount + 1];
          if (string == NULL)
          {
            LogDebug("LvctParser: failed to allocate %d bytes in DecodeMultipleStrings()", segmentCharCount + 1);
            continue;
          }
    
          string[0] = 0;  // start with an empty string
          for (vector<char*>::iterator it = segments.begin(); it != segments.end(); it++)
          {
            strcat(string, *it);
            delete[] *it;
          }
          strings->push_back(string);
        }
      }
      catch (...)
      {
        LogDebug("LvctParser: unhandled exception in DecodeMultipleStrings()");
      }
    }
    Fixing point (2) would require changing the loop in OnNewSection(). You could try this code:
    Code:
              for (size_t i = 0; i < extendedNames.size(); i++)
              {
                int extendedNameLength = strlen(extendedNames[i]);
                if (extendedNameLength == shortNameLength && strcmp(name, extendedNames[i]) == 0)
                {
                  continue;
                }
                int nameBufferSize = 0;
                if (name != NULL)
                {
                  nameBufferSize += strlen(" ()") + shortNameLength;
                }
                nameBufferSize += extendedNameLength + 1; // + 1 for NULL termination
                char* newName = new char[nameBufferSize];
                if (newName == NULL)
                {
                  LogDebug("LvctParser: failed to allocate %d bytes for the extended name", nameBufferSize);
                  continue;
                }
                strcpy(newName, extendedNames[i]);
                if (name != NULL)
                {
                  strcat(newName, " (");
                  strcat(newName, name);
                  strcat(newName, ")");
                  delete[] name;
                }
                name = newName;
                break;
              }
              for (vector<char*>::iterator it = extendedNames.begin(); it != extendedNames.end(); it++)
              {
                delete[] *it;
              }

    Please note:
    1. The above code excerpts are untested. They may have syntax and/or memory management (ie. memory access violation or leak) errors.
    2. The code for point 1 does not add support for compression type 0 mode 0x20.
    3. My TVE 3.5 TsWriter code is not affected by this issue, and has support for compression type 0 mode 0x20.
     

    morpheus_xx

    Lead Dev MP2
  • Team MediaPortal
  • March 24, 2007
    11,600
    7,060
    Germany Germany
    Country flag
    Thank you very much @mm1352000 :)

    I was on the right way, but you know this topic so much better than me. I will compile a new version based on your changes.

    With debug logging enabled again in last try, I now see the logic in the string segments. The red marked strings are not printable here, so they appear empty. It's should be a printable ' sign (inside "If it's...")
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: iso639LanguageCode: 676e65, string 0, number of segments = 9
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: segment 0, compression type = 0x0, mode = 0x0, number of bytes = 5
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x0, number of bytes = 5
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decoded string = 'If it'
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: segment 1, compression type = 0x0, mode = 0x20, number of bytes = 1
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x20, number of bytes = 1
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decoded string = ''
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: segment 2, compression type = 0x0, mode = 0x0, number of bytes = 6
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x0, number of bytes = 6
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decoded string = 's what'
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: segment 3, compression type = 0x0, mode = 0x20, number of bytes = 1
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x20, number of bytes = 1
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decoded string = ''
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: segment 4, compression type = 0x0, mode = 0x0, number of bytes = 13
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x0, number of bytes = 13
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decoded string = 's new or what'
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: segment 5, compression type = 0x0, mode = 0x20, number of bytes = 1
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x20, number of bytes = 1
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decoded string = ''
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: segment 6, compression type = 0x0, mode = 0x0, number of bytes = 10
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x0, number of bytes = 10
    [2020-01-01 12:03:06,537] [65590b0] [3c5c] - LvctParser: decoded string = 's next, it'
    [2020-01-01 12:03:06,537] [65590b0] [3c5c] - LvctParser: segment 7, compression type = 0x0, mode = 0x20, number of bytes = 1
    [2020-01-01 12:03:06,537] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x20, number of bytes = 1
    [2020-01-01 12:03:06,537] [65590b0] [3c5c] - LvctParser: decoded string = ''
    [2020-01-01 12:03:06,537] [65590b0] [3c5c] - LvctParser: segment 8, compression type = 0x0, mode = 0x0, number of bytes = 9
    [2020-01-01 12:03:06,537] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x0, number of bytes = 9
    [2020-01-01 12:03:06,537] [65590b0] [3c5c] - LvctParser: decoded string = 's on TBD.'

    and has support for compression type 0 mode 0x20.
    Could you point me to the right place? I'd try to port this support back to TVE3 TsWriter. I only found this mode range in an old powerpoint from 2000, but no official documents with exact specs :(

    // 0x00 - 0x3E: 8 - bit Unicode™
    // 0x3F : 16 - bit Unicode™
    // 0x40 - 0xDF : Reserved for future ATSC use
    // 0xE0 - 0xFE : User private
    // 0xFF : Text mode is not applicable

    This seems not to be right, because the 0x19 is no valid printable sign (mode=0x20).
     

    mm1352000

    Development Group
  • Team MediaPortal
  • September 1, 2008
    21,571
    8,220
    New Zealand New Zealand
    Country flag
    Thank you very much
    You're welcome, and happy new year - I hope you're well. :)

    Could you point me to the right place? I'd try to port this support back to TVE3 TsWriter.
    Sure, but I think you'll find it a bit challenging for the following reasons:
    1. The code bases are too different to enable direct back-porting.
    2. ATSC mode 0x20 (and some of the other modes) are actually a simple run-length compression of UTF-16. As you have seen from rback's example, each segment can have a different encoding. Somehow these segments with different encodings must be combined. This isn't so straightforward in the C++ world when you have to work with char*. AFAIK TVE 3's TsWriter has no code for doing this.
    3. TV library cannot reliably receive UTF-16 strings from TsWriter. In fact, the string must be DVB-compatible, which means default processing will be ISO/IEC 6937.
    Anyway, the equivalent functions for text decoding - DVB, ATSC, SCTE, OpenTV, MHW etc. - are all in this class:
    MediaPortal/MediaPortal-1

    Here is the ATSC L-VCT record decoding function:
    MediaPortal/MediaPortal-1

    Here is the function used to decode the multiple string structure for extended channel names (and other text such as EPG), a bit like the old DecodeMultipleStrings():
    MediaPortal/MediaPortal-1

    Here is the function that does the actual string decoding, a bit like the old DecodeString():
    MediaPortal/MediaPortal-1

    I only found this mode range in an old powerpoint from 2000, but no official documents with exact specs :(
    The applicable spec is here:
    https://www.atsc.org/wp-content/upl...tocol-for-Terrestrial-Broadcast-and-Cable.pdf

    Please refer to table 6.41 on page 82.

    In short, for modes 0x00 - 0x06, 0x09 - 0x10, 0x20 - 0x27, and 0x30 - 0x33, the mode value is the most significant byte of the UTF-16 symbol. In rback's example, the provider wishes to encode code point 0x2019. They have chosen to do it using compression_type 0x00 (uncompressed), mode 0x20, and the segment byte(s) [0x19]. Mode 0x20 plus byte 0x19 becomes code point 0x2019. If the segment bytes were [0x19, 0x20, 0x21] then the code points would be 0x2019, 0x2020, 0x2021.

    I hope this is all clear.
     

    rback

    Portal Member
    December 29, 2019
    15
    0
    54
    United States of America United States of America
    Country flag
    @morpheus_xx @mm1352000 The latest fix worked! I scanned all of my OTA TV Tuner cards and they all completed their scans successfully. I've attached the final logs for completeness.

    Thanks a lot for this quick turnaround! You guys are awesome!! Happy New Year!
     

    Attachments

    Last edited:

    Users who are viewing this thread

    Similar threads
    OP Title Forum Replies Date
    M MP2 - V2.3 MP2 & USB Problems Submit: Bug Reports 29
    L MP2 - V2.3 Troubles upgrading from MP2 2.2.3 to 2.3 General 0
    T MP2 - V2.3 MP2 x64 won't run Submit: Bug Reports 5
    J Copying Old VHS tapes Newcomers Forum 8
    W MP2 - V2.3 MP2 2.3 Import von Audioquellen / Datenbankprobleme Allgemeines Support- und Diskussionsforum 26
    G MP2 - V2.3 New install and crashing MP2-2.3 General 6
    S MP2 - V2.3 MP2 No video General 6
    W MP2 - V2.3 MP2 Server stürzt immer nach kurzer Zeit ab Allgemeines Support- und Diskussionsforum 77
    toricred MP2 - V2.3 MP2 2.3 Client won't start General 6
    Grisu002 MP2 - V2.3 MP2 2.3 Client startet nicht - bleibt beim Laden hängen Allgemeines Support- und Diskussionsforum 61
    M [solved] MP2 PROBLEME Standby Allgemeines Support- und Diskussionsforum 2
    Scheibes Was passiert bei MP2? Allgemeines Support- und Diskussionsforum 42
    S Programmierte Aufnahme in MP2 - kein Aufwachen des PC Allgemeines Support- und Diskussionsforum 5
    morpheus_xx [Finished] "What's New" plugin for MP2 upgrades MediaPortal 2 20
    da3dsoul [Request] An MVP Plugin example for custom settings in MP2 Plugin Development 2
    morpheus_xx [Evaluate] Checking MP2 code portability to .Net Core MediaPortal 2 15
    da3dsoul Does MP2 not have any active devs? General 8
    da3dsoul MP2 Plugin - Getting Started (Directory Tree and Player) Plugin Development 27
    R [solved] Kein Player registriert Allgemeines Support- und Diskussionsforum 1
    -Merkur- MP2 Client startet nicht, "libXBMC_addon.dll" fehlt Allgemeines Support- und Diskussionsforum 1
    mp2-ulrich [solved] MP 2.2.3 spielt nur manche Sender Allgemeines Support- und Diskussionsforum 2
    H MP2 database won't keep in synch with recordings General 8
    H [solved] MP2 Wiedergabe von Aufnahmen und Live TV geht nicht Allgemeines Support- und Diskussionsforum 6
    ge2301 [WiP] MP2Web - A platform independent web client for MP2 MediaPortal 2 72
    F Problems with new install of MP2 Newcomers Forum 3

    Similar threads

    Top Bottom