[fixed] [MP2-819] MP2 Server crashes on ATSC OTA channel scan (2 Viewers)

[morpheus_xx]

Thanks for your logfiles!

I think the issue comes from our TsWriter and an unsupported compression/encoding schema:
[2019-12-30 10:47:40,885] [43a1340] [27c4] - LvctParser: descriptor, tag = 0xa0, length = 79, pointer = 99, end of descriptor = 178
[2019-12-30 10:47:40,885] [43a1340] [27c4] - LvctParser: unsupported compression type or mode in DecodeString(), compression type = 0x0, mode = 0x20
[2019-12-30 10:47:40,885] [43a1340] [27c4] - 0: 0x19
[2019-12-30 10:47:40,885] [43a1340] [27c4] - LvctParser: unsupported compression type or mode in DecodeString(), compression type = 0x0, mode = 0x20
[2019-12-30 10:47:40,885] [43a1340] [27c4] - 0: 0x19
[2019-12-30 10:47:40,885] [43a1340] [27c4] - LvctParser: unsupported compression type or mode in DecodeString(), compression type = 0x0, mode = 0x20
[2019-12-30 10:47:40,885] [43a1340] [27c4] - 0: 0x19
[2019-12-30 10:47:40,885] [43a1340] [27c4] - LvctParser: unsupported compression type or mode in DecodeString(), compression type = 0x0, mode = 0x20

After those lines the log ends, so the crash seems related to this.

Can any of our @Developers who know this part can take a look please?

 

[morpheus_xx]

@rback
I have done a minor modification to the string decoding logic inside the L-VCT parser. Can you please test the attached file?

1. Stop the MP2-Server service (i.e. right click on Server Monitor in tray)
   
2. Go to directory "C:\Program Files (x86)\Team MediaPortal\MP2-Server\Plugins\SlimTv.Service3". Rename the TsWriter.ax (or move it to other directory as backup)
3. Extract the attached file into this folder
4. Start the MP2-Server service (i.e. right click on Server Monitor in tray)
5. Do a channel scan again
6. Post new logfiles

 

[rback]

I assume you mean me to replace the TsWriter.ax file, since that is the file you provided.

Unfortunately, I still got a crash while scanning channel 24. Attached are the latest logs.

 

[morpheus_xx]

I assume you mean me to replace the TsWriter.ax file, since that is the file you provided.

Yes, exactly, was a typo.

I didn't find the cause yet, but I added more logging that could help. Please try again with attached file and post new logs.

 

[rback]

Attached are the latest logs using the new TsWriter.ax file.

 

[mm1352000]

@morpheus_xx
I've had a quick look. In my opinion, the issue isn't directly caused by the unsupported compression/encoding scheme. Rather, it is caused by the fact that the channel's extended name has more than one segment.

1. DecodeMultipleStrings() is not producing the correct output. Instead of producing <numberStrings> strings in <strings>, it is putting all the segments from all the strings directly in <strings>. In other words, for this particular example, it is producing 9 strings (which is the number of segments in string 0) instead of 1.
2. The loop after DecodeMultipleStrings() is not combining names correctly. Intention is to combine the short name with the first extended name that is not the same as the short name, in the format "<extended name> (<short name>)".

Fixing point (1) would require fixing DecodeMultipleStrings(). You could try this code:

void CLvctParser::DecodeMultipleStrings(byte* b, int length, vector<char*>* strings)
{
  if (length < 1)
  {
    LogDebug("LvctParser: invalid multiple strings structure length = %d", length);
    return;
  }
  try
  {
    int numberStrings = b[0];

    //LogDebug("LvctParser: parse multiple strings, number of strings = %d", numberStrings);
    int pointer = 1;
    for (int i = 0; i < numberStrings && pointer + 3 < length; i++)
    {
      unsigned int iso639LanguageCode = b[pointer] + (b[pointer + 1] << 8) + (b[pointer + 2] << 16);
      pointer += 3;
      int numberSegments = b[pointer++];
      //LogDebug("LvctParser: string %d, number of segments = %d", i, numberSegments);

      vector<char*> segments;
      int segmentCharCount = 0;
      for (int j = 0; j < numberSegments && pointer + 2 < length; j++)
      {
        int compressionType = b[pointer++];
        int mode = b[pointer++];
        int numberBytes = b[pointer++];
        //LogDebug("LvctParser: segment %d, compression type = 0x%x, mode = 0x%x, number of bytes = %d", j, compressionType, mode, numberBytes);
        if (pointer + numberBytes > length)
        {
          LogDebug("LvctParser: invalid string length %d in multiple string structure, pointer = %d, number of bytes = %d, structure length = %d", pointer, numberBytes, length);
          return;
        }

        char* segment = NULL;
        DecodeString(&b[pointer], compressionType, mode, numberBytes, &segment);
        if (segment != NULL)
        {
          segments.push_back(segment);
          segmentCharCount += strlen(segment);
        }

        pointer += numberBytes;
      }

      // combine the segments to produce the final string
      if (segments.size() == 0)
      {
        continue;
      }
      if (segments.size() == 1)
      {
        strings->push_back(segments[0]);
        continue;
      }

      char* string = new char[segmentCharCount + 1];
      if (string == NULL)
      {
        LogDebug("LvctParser: failed to allocate %d bytes in DecodeMultipleStrings()", segmentCharCount + 1);
        continue;
      }

      string[0] = 0;  // start with an empty string
      for (vector<char*>::iterator it = segments.begin(); it != segments.end(); it++)
      {
        strcat(string, *it);
        delete[] *it;
      }
      strings->push_back(string);
    }
  }
  catch (...)
  {
    LogDebug("LvctParser: unhandled exception in DecodeMultipleStrings()");
  }
}

Fixing point (2) would require changing the loop in OnNewSection(). You could try this code:

          for (size_t i = 0; i < extendedNames.size(); i++)
          {
            int extendedNameLength = strlen(extendedNames[i]);
            if (extendedNameLength == shortNameLength && strcmp(name, extendedNames[i]) == 0)
            {
              continue;
            }
            int nameBufferSize = 0;
            if (name != NULL)
            {
              nameBufferSize += strlen(" ()") + shortNameLength;
            }
            nameBufferSize += extendedNameLength + 1; // + 1 for NULL termination
            char* newName = new char[nameBufferSize];
            if (newName == NULL)
            {
              LogDebug("LvctParser: failed to allocate %d bytes for the extended name", nameBufferSize);
              continue;
            }
            strcpy(newName, extendedNames[i]);
            if (name != NULL)
            {
              strcat(newName, " (");
              strcat(newName, name);
              strcat(newName, ")");
              delete[] name;
            }
            name = newName;
            break;
          }
          for (vector<char*>::iterator it = extendedNames.begin(); it != extendedNames.end(); it++)
          {
            delete[] *it;
          }

Please note:

1. The above code excerpts are untested. They may have syntax and/or memory management (ie. memory access violation or leak) errors.
2. The code for point 1 does not add support for compression type 0 mode 0x20.
3. My TVE 3.5 TsWriter code is not affected by this issue, and has support for compression type 0 mode 0x20.

 

[morpheus_xx]

Thank you very much @mm1352000

I was on the right way, but you know this topic so much better than me. I will compile a new version based on your changes.

With debug logging enabled again in last try, I now see the logic in the string segments. The red marked strings are not printable here, so they appear empty. It's should be a printable ' sign (inside "If it's...")
[2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: iso639LanguageCode: 676e65, string 0, number of segments = 9
[2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: segment 0, compression type = 0x0, mode = 0x0, number of bytes = 5
[2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x0, number of bytes = 5
[2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decoded string = 'If it'
[2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: segment 1, compression type = 0x0, mode = 0x20, number of bytes = 1
[2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x20, number of bytes = 1
[2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decoded string = ''
[2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: segment 2, compression type = 0x0, mode = 0x0, number of bytes = 6
[2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x0, number of bytes = 6
[2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decoded string = 's what'
[2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: segment 3, compression type = 0x0, mode = 0x20, number of bytes = 1
[2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x20, number of bytes = 1
[2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decoded string = ''
[2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: segment 4, compression type = 0x0, mode = 0x0, number of bytes = 13
[2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x0, number of bytes = 13
[2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decoded string = 's new or what'
[2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: segment 5, compression type = 0x0, mode = 0x20, number of bytes = 1
[2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x20, number of bytes = 1
[2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decoded string = ''
[2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: segment 6, compression type = 0x0, mode = 0x0, number of bytes = 10
[2020-01-01 12:03:06,536] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x0, number of bytes = 10
[2020-01-01 12:03:06,537] [65590b0] [3c5c] - LvctParser: decoded string = 's next, it'
[2020-01-01 12:03:06,537] [65590b0] [3c5c] - LvctParser: segment 7, compression type = 0x0, mode = 0x20, number of bytes = 1
[2020-01-01 12:03:06,537] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x20, number of bytes = 1
[2020-01-01 12:03:06,537] [65590b0] [3c5c] - LvctParser: decoded string = ''
[2020-01-01 12:03:06,537] [65590b0] [3c5c] - LvctParser: segment 8, compression type = 0x0, mode = 0x0, number of bytes = 9
[2020-01-01 12:03:06,537] [65590b0] [3c5c] - LvctParser: decode string, compression type = 0x0, mode = 0x0, number of bytes = 9
[2020-01-01 12:03:06,537] [65590b0] [3c5c] - LvctParser: decoded string = 's on TBD.'

and has support for compression type 0 mode 0x20.

Could you point me to the right place? I'd try to port this support back to TVE3 TsWriter. I only found this mode range in an old powerpoint from 2000, but no official documents with exact specs

// 0x00 - 0x3E: 8 - bit Unicode™
// 0x3F : 16 - bit Unicode™
// 0x40 - 0xDF : Reserved for future ATSC use
// 0xE0 - 0xFE : User private
// 0xFF : Text mode is not applicable

This seems not to be right, because the 0x19 is no valid printable sign (mode=0x20).

 

[morpheus_xx]

@rback please try again with the proposed changes by mm.

 

[mm1352000]

Thank you very much

You're welcome, and happy new year - I hope you're well.

Could you point me to the right place? I'd try to port this support back to TVE3 TsWriter.

Sure, but I think you'll find it a bit challenging for the following reasons:

1. The code bases are too different to enable direct back-porting.
2. ATSC mode 0x20 (and some of the other modes) are actually a simple run-length compression of UTF-16. As you have seen from rback's example, each segment can have a different encoding. Somehow these segments with different encodings must be combined. This isn't so straightforward in the C++ world when you have to work with char*. AFAIK TVE 3's TsWriter has no code for doing this.
   
3. TV library cannot reliably receive UTF-16 strings from TsWriter. In fact, the string must be DVB-compatible, which means default processing will be ISO/IEC 6937.

Anyway, the equivalent functions for text decoding - DVB, ATSC, SCTE, OpenTV, MHW etc. - are all in this class:
MediaPortal/MediaPortal-1

Here is the ATSC L-VCT record decoding function:
MediaPortal/MediaPortal-1

Here is the function used to decode the multiple string structure for extended channel names (and other text such as EPG), a bit like the old DecodeMultipleStrings():
MediaPortal/MediaPortal-1

Here is the function that does the actual string decoding, a bit like the old DecodeString():
MediaPortal/MediaPortal-1

I only found this mode range in an old powerpoint from 2000, but no official documents with exact specs

The applicable spec is here:
https://www.atsc.org/wp-content/upl...tocol-for-Terrestrial-Broadcast-and-Cable.pdf

Please refer to table 6.41 on page 82.

In short, for modes 0x00 - 0x06, 0x09 - 0x10, 0x20 - 0x27, and 0x30 - 0x33, the mode value is the most significant byte of the UTF-16 symbol. In rback's example, the provider wishes to encode code point 0x2019. They have chosen to do it using compression_type 0x00 (uncompressed), mode 0x20, and the segment byte(s) [0x19]. Mode 0x20 plus byte 0x19 becomes code point 0x2019. If the segment bytes were [0x19, 0x20, 0x21] then the code points would be 0x2019, 0x2020, 0x2021.

I hope this is all clear.

 

[rback]

@morpheus_xx @mm1352000 The latest fix worked! I scanned all of my OTA TV Tuner cards and they all completed their scans successfully. I've attached the final logs for completeness.

Thanks a lot for this quick turnaround! You guys are awesome!! Happy New Year!