[fixed] [MP2-819] MP2 Server crashes on ATSC OTA channel scan (2 Viewers)

morpheus_xx

Lead Dev MP2
  • Team MediaPortal
  • March 24, 2007
    11,768
    7,121
    Home Country
    Germany Germany
    Country flag
    Thanks for your logfiles!

    I think the issue comes from our TsWriter and an unsupported compression/encoding schema:
    [2019-12-30 10:47:40,885] [43a1340] [27c4] - LvctParser: descriptor, tag = 0xa0, length = 79, pointer = 99, end of descriptor = 178
    [2019-12-30 10:47:40,885] [43a1340] [27c4] - LvctParser: unsupported compression type or mode in DecodeString(), compression type = 0x0, mode = 0x20
    [2019-12-30 10:47:40,885] [43a1340] [27c4] - 0: 0x19

    [2019-12-30 10:47:40,885] [43a1340] [27c4] - LvctParser: unsupported compression type or mode in DecodeString(), compression type = 0x0, mode = 0x20
    [2019-12-30 10:47:40,885] [43a1340] [27c4] - 0: 0x19
    [2019-12-30 10:47:40,885] [43a1340] [27c4] - LvctParser: unsupported compression type or mode in DecodeString(), compression type = 0x0, mode = 0x20
    [2019-12-30 10:47:40,885] [43a1340] [27c4] - 0: 0x19
    [2019-12-30 10:47:40,885] [43a1340] [27c4] - LvctParser: unsupported compression type or mode in DecodeString(), compression type = 0x0, mode = 0x20

    After those lines the log ends, so the crash seems related to this.

    Can any of our @Developers who know this part can take a look please?
     

    morpheus_xx

    Lead Dev MP2
  • Team MediaPortal
  • March 24, 2007
    11,768
    7,121
    Home Country
    Germany Germany
    Country flag
    @rback
    I have done a minor modification to the string decoding logic inside the L-VCT parser. Can you please test the attached file?
    1. Stop the MP2-Server service (i.e. right click on Server Monitor in tray)
    2. Go to directory "C:\Program Files (x86)\Team MediaPortal\MP2-Server\Plugins\SlimTv.Service3". Rename the TsWriter.ax (or move it to other directory as backup)
    3. Extract the attached file into this folder
    4. Start the MP2-Server service (i.e. right click on Server Monitor in tray)
    5. Do a channel scan again
    6. Post new logfiles
     

    Attachments

    • 20191231-TsWriter_Test_LVCT.01.7z
      151.4 KB · Views: 2
    Last edited:

    rback

    Portal Member
    December 29, 2019
    15
    0
    55
    Home Country
    United States of America United States of America
    Country flag
    I assume you mean me to replace the TsWriter.ax file, since that is the file you provided.

    Unfortunately, I still got a crash while scanning channel 24. Attached are the latest logs.
     

    Attachments

    • MediaPortal2-Logs-2019-12-31-11.20.42.zip
      531.5 KB · Views: 5

    morpheus_xx

    Lead Dev MP2
  • Team MediaPortal
  • March 24, 2007
    11,768
    7,121
    Home Country
    Germany Germany
    Country flag
    I assume you mean me to replace the TsWriter.ax file, since that is the file you provided.
    Yes, exactly, was a typo.

    I didn't find the cause yet, but I added more logging that could help. Please try again with attached file and post new logs.
     

    Attachments

    • 20200101-TsWriter_Test_LVCT.02.7z
      151.9 KB · Views: 2

    rback

    Portal Member
    December 29, 2019
    15
    0
    55
    Home Country
    United States of America United States of America
    Country flag
    Attached are the latest logs using the new TsWriter.ax file.
     

    Attachments

    • MediaPortal2-Logs-2020-01-01-12.04.51.zip
      514.2 KB · Views: 6

    mm1352000

    Development Group
  • Team MediaPortal
  • September 1, 2008
    21,577
    8,222
    Home Country
    New Zealand New Zealand
    Country flag
    @morpheus_xx
    I've had a quick look. In my opinion, the issue isn't directly caused by the unsupported compression/encoding scheme. Rather, it is caused by the fact that the channel's extended name has more than one segment.
    1. DecodeMultipleStrings() is not producing the correct output. Instead of producing <numberStrings> strings in <strings>, it is putting all the segments from all the strings directly in <strings>. In other words, for this particular example, it is producing 9 strings (which is the number of segments in string 0) instead of 1.
    2. The loop after DecodeMultipleStrings() is not combining names correctly. Intention is to combine the short name with the first extended name that is not the same as the short name, in the format "<extended name> (<short name>)".
    Fixing point (1) would require fixing DecodeMultipleStrings(). You could try this code:
    Code:
    void CLvctParser::DecodeMultipleStrings(byte* b, int length, vector<char*>* strings)
    {
      if (length < 1)
      {
        LogDebug("LvctParser: invalid multiple strings structure length = %d", length);
        return;
      }
      try
      {
        int numberStrings = b[0];
    
        //LogDebug("LvctParser: parse multiple strings, number of strings = %d", numberStrings);
        int pointer = 1;
        for (int i = 0; i < numberStrings && pointer + 3 < length; i++)
        {
          unsigned int iso639LanguageCode = b[pointer] + (b[pointer + 1] << 8) + (b[pointer + 2] << 16);
          pointer += 3;
          int numberSegments = b[pointer++];
          //LogDebug("LvctParser: string %d, number of segments = %d", i, numberSegments);
    
          vector<char*> segments;
          int segmentCharCount = 0;
          for (int j = 0; j < numberSegments && pointer + 2 < length; j++)
          {
            int compressionType = b[pointer++];
            int mode = b[pointer++];
            int numberBytes = b[pointer++];
            //LogDebug("LvctParser: segment %d, compression type = 0x%x, mode = 0x%x, number of bytes = %d", j, compressionType, mode, numberBytes);
            if (pointer + numberBytes > length)
            {
              LogDebug("LvctParser: invalid string length %d in multiple string structure, pointer = %d, number of bytes = %d, structure length = %d", pointer, numberBytes, length);
              return;
            }
    
            char* segment = NULL;
            DecodeString(&b[pointer], compressionType, mode, numberBytes, &segment);
            if (segment != NULL)
            {
              segments.push_back(segment);
              segmentCharCount += strlen(segment);
            }
    
            pointer += numberBytes;
          }
    
          // combine the segments to produce the final string
          if (segments.size() == 0)
          {
            continue;
          }
          if (segments.size() == 1)
          {
            strings->push_back(segments[0]);
            continue;
          }
    
          char* string = new char[segmentCharCount + 1];
          if (string == NULL)
          {
            LogDebug("LvctParser: failed to allocate %d bytes in DecodeMultipleStrings()", segmentCharCount + 1);
            continue;
          }
    
          string[0] = 0;  // start with an empty string
          for (vector<char*>::iterator it = segments.begin(); it != segments.end(); it++)
          {
            strcat(string, *it);
            delete[] *it;
          }
          strings->push_back(string);
        }
      }
      catch (...)
      {
        LogDebug("LvctParser: unhandled exception in DecodeMultipleStrings()");
      }
    }

    Fixing point (2) would require changing the loop in OnNewSection(). You could try this code:
    Code:
              for (size_t i = 0; i < extendedNames.size(); i++)
              {
                int extendedNameLength = strlen(extendedNames[i]);
                if (extendedNameLength == shortNameLength && strcmp(name, extendedNames[i]) == 0)
                {
                  continue;
                }
                int nameBufferSize = 0;
                if (name != NULL)
                {
                  nameBufferSize += strlen(" ()") + shortNameLength;
                }
                nameBufferSize += extendedNameLength + 1; // + 1 for NULL termination
                char* newName = new char[nameBufferSize];
                if (newName == NULL)
                {
                  LogDebug("LvctParser: failed to allocate %d bytes for the extended name", nameBufferSize);
                  continue;
                }
                strcpy(newName, extendedNames[i]);
                if (name != NULL)
                {
                  strcat(newName, " (");
                  strcat(newName, name);
                  strcat(newName, ")");
                  delete[] name;
                }
                name = newName;
                break;
              }
              for (vector<char*>::iterator it = extendedNames.begin(); it != extendedNames.end(); it++)
              {
                delete[] *it;
              }


    Please note:
    1. The above code excerpts are untested. They may have syntax and/or memory management (ie. memory access violation or leak) errors.
    2. The code for point 1 does not add support for compression type 0 mode 0x20.
    3. My TVE 3.5 TsWriter code is not affected by this issue, and has support for compression type 0 mode 0x20.
     

    morpheus_xx

    Lead Dev MP2
  • Team MediaPortal
  • March 24, 2007
    11,768
    7,121
    Home Country
    Germany Germany
    Country flag
    Thank you very much @mm1352000 :)

    I was on the right way, but you know this topic so much better than me. I will compile a new version based on your changes.

    With debug logging enabled again in last try, I now see the logic in the string segments. The red marked strings are not printable here, so they appear empty. It's should be a printable ' sign (inside "If it's...")
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: iso639LanguageCode: 676e65, string 0, number of segments = 9
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: segment 0, compression type = 0x0, mode = 0x0, number of bytes = 5
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x0, number of bytes = 5
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decoded string = 'If it'
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: segment 1, compression type = 0x0, mode = 0x20, number of bytes = 1
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x20, number of bytes = 1
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decoded string = ''
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: segment 2, compression type = 0x0, mode = 0x0, number of bytes = 6
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x0, number of bytes = 6
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decoded string = 's what'
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: segment 3, compression type = 0x0, mode = 0x20, number of bytes = 1
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x20, number of bytes = 1
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decoded string = ''
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: segment 4, compression type = 0x0, mode = 0x0, number of bytes = 13
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x0, number of bytes = 13
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decoded string = 's new or what'
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: segment 5, compression type = 0x0, mode = 0x20, number of bytes = 1
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x20, number of bytes = 1
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decoded string = ''
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: segment 6, compression type = 0x0, mode = 0x0, number of bytes = 10
    [2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x0, number of bytes = 10
    [2020-01-01 12:03:06,537] [65590b0] [3c5c] - LvctParser: decoded string = 's next, it'
    [2020-01-01 12:03:06,537] [65590b0] [3c5c] - LvctParser: segment 7, compression type = 0x0, mode = 0x20, number of bytes = 1
    [2020-01-01 12:03:06,537] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x20, number of bytes = 1
    [2020-01-01 12:03:06,537] [65590b0] [3c5c] - LvctParser: decoded string = ''
    [2020-01-01 12:03:06,537] [65590b0] [3c5c] - LvctParser: segment 8, compression type = 0x0, mode = 0x0, number of bytes = 9
    [2020-01-01 12:03:06,537] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x0, number of bytes = 9
    [2020-01-01 12:03:06,537] [65590b0] [3c5c] - LvctParser: decoded string = 's on TBD.'

    and has support for compression type 0 mode 0x20.
    Could you point me to the right place? I'd try to port this support back to TVE3 TsWriter. I only found this mode range in an old powerpoint from 2000, but no official documents with exact specs :(

    // 0x00 - 0x3E: 8 - bit Unicode™
    // 0x3F : 16 - bit Unicode™
    // 0x40 - 0xDF : Reserved for future ATSC use
    // 0xE0 - 0xFE : User private
    // 0xFF : Text mode is not applicable

    This seems not to be right, because the 0x19 is no valid printable sign (mode=0x20).
     

    morpheus_xx

    Lead Dev MP2
  • Team MediaPortal
  • March 24, 2007
    11,768
    7,121
    Home Country
    Germany Germany
    Country flag
    @rback please try again with the proposed changes by mm.
     

    Attachments

    • 20200102-TsWriter_Test_LVCT.03.7z
      151.6 KB · Views: 1

    mm1352000

    Development Group
  • Team MediaPortal
  • September 1, 2008
    21,577
    8,222
    Home Country
    New Zealand New Zealand
    Country flag
    Thank you very much
    You're welcome, and happy new year - I hope you're well. :)

    Could you point me to the right place? I'd try to port this support back to TVE3 TsWriter.
    Sure, but I think you'll find it a bit challenging for the following reasons:
    1. The code bases are too different to enable direct back-porting.
    2. ATSC mode 0x20 (and some of the other modes) are actually a simple run-length compression of UTF-16. As you have seen from rback's example, each segment can have a different encoding. Somehow these segments with different encodings must be combined. This isn't so straightforward in the C++ world when you have to work with char*. AFAIK TVE 3's TsWriter has no code for doing this.
    3. TV library cannot reliably receive UTF-16 strings from TsWriter. In fact, the string must be DVB-compatible, which means default processing will be ISO/IEC 6937.
    Anyway, the equivalent functions for text decoding - DVB, ATSC, SCTE, OpenTV, MHW etc. - are all in this class:
    MediaPortal/MediaPortal-1

    Here is the ATSC L-VCT record decoding function:
    MediaPortal/MediaPortal-1

    Here is the function used to decode the multiple string structure for extended channel names (and other text such as EPG), a bit like the old DecodeMultipleStrings():
    MediaPortal/MediaPortal-1

    Here is the function that does the actual string decoding, a bit like the old DecodeString():
    MediaPortal/MediaPortal-1

    I only found this mode range in an old powerpoint from 2000, but no official documents with exact specs :(
    The applicable spec is here:
    https://www.atsc.org/wp-content/upl...tocol-for-Terrestrial-Broadcast-and-Cable.pdf

    Please refer to table 6.41 on page 82.

    In short, for modes 0x00 - 0x06, 0x09 - 0x10, 0x20 - 0x27, and 0x30 - 0x33, the mode value is the most significant byte of the UTF-16 symbol. In rback's example, the provider wishes to encode code point 0x2019. They have chosen to do it using compression_type 0x00 (uncompressed), mode 0x20, and the segment byte(s) [0x19]. Mode 0x20 plus byte 0x19 becomes code point 0x2019. If the segment bytes were [0x19, 0x20, 0x21] then the code points would be 0x2019, 0x2020, 0x2021.

    I hope this is all clear.
     

    rback

    Portal Member
    December 29, 2019
    15
    0
    55
    Home Country
    United States of America United States of America
    Country flag
    @morpheus_xx @mm1352000 The latest fix worked! I scanned all of my OTA TV Tuner cards and they all completed their scans successfully. I've attached the final logs for completeness.

    Thanks a lot for this quick turnaround! You guys are awesome!! Happy New Year!
     

    Attachments

    • MediaPortal2-Logs-2020-01-02-11.42.17.zip
      1,005.8 KB · Views: 4
    Last edited:

    Users who are viewing this thread

    Similar threads

    Thanks. I'll be working through your response and experimenting some more. Will let you know what I find.
    Thanks. I'll be working through your response and experimenting some more. Will let you know what I find.
    MP looks like an attractive replacement for WMC, but unlike Kodi (which I just tried), I'm having a hard time getting my 3-tuner...
    Replies
    2
    Views
    385
    I've added this to Jira MP2-832 so that we remember the fact that LAV is not capable of handling wtv subs.
    I've added this to Jira MP2-832 so that we remember the fact that LAV is not capable of handling wtv subs.
    MediaPortal 2 Version: 2.2.2 Description Hi, I am unable to view subtitles in shows recorded by Windows Media Center (WMC) in WTV...
    Replies
    7
    Views
    818
    for it can record, many channel from the same transponder. but thank you for confirming your asking
    for it can record, many channel from the same transponder. but thank you for confirming your asking
    I'm currently using a TBS6985 DVB-S2 quad tuner, but we have reception issues - seems to be caused internally in the server - I...
    Replies
    2
    Views
    656
    There isn't.
    There isn't.
    I have a system with three SiliconDust HDHomeRun HDHR4-2US over-the-air tuners and one SiliconDust HDHomeRun Prime 3 cable tuner...
    Replies
    1
    Views
    486
    We'll, again I have posted a problem then found the solution myself :whistle: I incorrectly used "Multi seat setup: use UNC paths" when I was setting up initially and having some trouble with paths. I had clicked help for that setting before I tried using it, but many of the help links in the config utility are broken. Today I...
    We'll, again I have posted a problem then found the solution myself :whistle: I incorrectly used "Multi seat setup: use UNC paths"...
    I've been a MP1 user for a very long time. I had been using Argus for OTA ATSC recording and viewing which has been flawless...
    Replies
    1
    Views
    2K
    Top Bottom