Reply to thread

Message

<blockquote data-quote="mm1352000" data-source="post: 1051719" data-attributes="member: 82144">*bump*@[USER=97516]Vasilich[/USER]Have you had a chance to investigate further? I'm interested to know what you find... <img src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" class="smilie smilie--sprite smilie--sprite1" alt=":)" title="Smile&nbsp; &nbsp; :)" loading="lazy" data-shortname=":)" />This afternoon I've been going through the DVB text conversion to tidy it up for TVE 3.5. I don't know if I'm going crazy here, but I noticed a couple of things that seem... well, not right.The main functions/classes that matter for encoding (ignoring decompression - hufman stuff) are:<ul>
<li data-xf-list-type="ul">TsWriter:<ul>
<li data-xf-list-type="ul">DvbUtil.cpp - getString468A()</li>
</ul></li>
<li data-xf-list-type="ul">TvLibrary.Interfaces:<ul>
<li data-xf-list-type="ul">DvbTextConverter.cs</li>
</ul></li>
<li data-xf-list-type="ul">TVLibrary:<ul>
<li data-xf-list-type="ul">Iso6937ToUnicode.cs 
</li>
<li data-xf-list-type="ul">TvCardDvbBase.cs - Epg</li>
</ul></li>
</ul>I have serious concerns about the matching of the TsWriter and TV library code. Read carefully and see what you think happens.With one byte encoding (EN 300 468 annex A table A.3), if you have the incoming bytes...[encoding byte] [content]The way I read it, I think what you would get out is:[encoding byte... if &lt;= 0x05] [content - code points constrained to c &lt;= 0x05, 0x20 &lt;= c &lt; 0x80, c &gt;= 0x9f]With three byte encoding (EN 300 468 annex A table A.4), if you have the incoming bytes:0x10 0x00 [encoding byte] [content]The way I read it, I think what you would get out is:0x10 [encoding byte] [encoding byte... if &lt;= 0x05] [content - code points constrained to c &lt;= 0x05, 0x20 &lt;= c &lt; 0x80, c &gt;= 0x9f]What is the problem here?For one byte encoding, the encoding byte will be thrown away unless it is &lt;= 0x05. That means no support for anything other than ASCII and ISO/IEC 8859-5..9.That is not the end of it. Consider what TV library does with three byte encoding:[code]&nbsp; byte c = Marshal.ReadByte(ptr, 0);&nbsp; if (c &lt; 0x20)...&nbsp; case 0x10:&nbsp; {&nbsp; pos = 3;&nbsp; c = Marshal.ReadByte(ptr, 2);[/code]TV library seems to have forgotten that the second encoding byte had to be removed in order to avoid premature NULL termination. In other words, TsWriter wrote 0x10 [encoding byte] but TV library is still expecting 0x10 0x00 [encoding byte]. This can only work if the encoding byte is &lt;= 0x05. So it looks like ISO/IEC 8859-1..5 would work, but ISO/IEC 8859-6+ would not.If I'm right, these are quite serious flaws.Do you agree?Continuing a bit further...In the EPG text decoding we see this code:[code]&nbsp; if (language.ToUpperInvariant() == &quot;CZE&quot; || language.ToUpperInvariant() == &quot;CES&quot;)&nbsp; {&nbsp; &nbsp; title = Iso6937ToUnicode.Convert(ptrTitle);&nbsp; &nbsp; description = Iso6937ToUnicode.Convert(ptrDesc);&nbsp; }&nbsp; else&nbsp; {&nbsp; &nbsp; title = DvbTextConverter.Convert(ptrTitle);&nbsp; &nbsp; description = DvbTextConverter.Convert(ptrDesc);&nbsp; }[/code]The start of a hack if ever I saw one! <img src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" class="smilie smilie--sprite smilie--sprite3" alt=":(" title="Frown&nbsp; &nbsp; :(" loading="lazy" data-shortname=":(" />In Iso6937ToUnicode we have:[code]&nbsp; if (b &lt; 0x20)&nbsp; {&nbsp; &nbsp; // ISO 6937 encoding must start with character between 0x20 and 0xFF&nbsp; &nbsp; // otherwise it is dfferent encoding table&nbsp; &nbsp; // for example 0x05 means encoding table 8859-9&nbsp; &nbsp; // here is just fallback to system ANSI&nbsp; &nbsp; return Marshal.PtrToStringAnsi(ptr);&nbsp; }[/code]What is going on here?!?It looks like somebody decided all Czech EPG should be decoded as ISO 6937, but then when proper DVB encoding is detected they fall back to ANSI. The hack is bad enough, but surely that PtrToStringAnsi() call should at least be replaced with DvbTextConverter.Convert().Do you agree?Finally, in DvbTextConverter:[code]&nbsp; int encoding = CultureInfo.CurrentCulture.TextInfo.ANSICodePage;&nbsp; try&nbsp; {&nbsp; &nbsp; if (string.IsNullOrEmpty(lang))&nbsp; &nbsp; {&nbsp; &nbsp; &nbsp; lang = CultureInfo.CurrentCulture.ThreeLetterISOLanguageName;&nbsp; &nbsp; }&nbsp; &nbsp; if (lang.Equals(&quot;cze&quot;) || lang.Equals(&quot;ces&quot;))&nbsp; &nbsp; {&nbsp; &nbsp; &nbsp; encoding = 20269; //ISO-6937&nbsp; &nbsp; }&nbsp; &nbsp; else if (lang.Equals(&quot;ukr&quot;) || lang.Equals(&quot;bel&quot;) || lang.Equals(&quot;rus&quot;))&nbsp; &nbsp; {&nbsp; &nbsp; &nbsp; encoding = 28595; //ISO-8859-5&nbsp; }[/code]Another hack. <img src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7" class="smilie smilie--sprite smilie--sprite3" alt=":(" title="Frown&nbsp; &nbsp; :(" loading="lazy" data-shortname=":(" />Since language is always passed as &quot;&quot; (empty string), any person with OS lang/culture set to Ukranian, Belarussian or Russian will have ISO 8859-5 as default encoding. According to my understanding of EN 300 468, the default encoding should be in figure A.1 - ISO/IEC 6937 with a modification for code point 0xa4 (Euro symbol). So Czech may be the only culture that gets the proper default encoding.In summary there seem to be some really nasty bugs and hacks in here. I don't know if any of these directly affect gurabli. Maybe you could confirm?mm</blockquote>

[QUOTE="mm1352000, post: 1051719, member: 82144"] *bump* @[USER=97516]Vasilich[/USER] Have you had a chance to investigate further? I'm interested to know what you find... :) This afternoon I've been going through the DVB text conversion to tidy it up for TVE 3.5. I don't know if I'm going crazy here, but I noticed a couple of things that seem... well, not right. The main functions/classes that matter for encoding (ignoring decompression - hufman stuff) are: [LIST] [*]TsWriter: [LIST] [*]DvbUtil.cpp - getString468A() [/LIST] [*]TvLibrary.Interfaces: [LIST] [*]DvbTextConverter.cs [/LIST] [*]TVLibrary: [LIST] [*]Iso6937ToUnicode.cs [*]TvCardDvbBase.cs - Epg [/LIST] [/LIST] I have serious concerns about the matching of the TsWriter and TV library code. Read carefully and see what you think happens. With one byte encoding (EN 300 468 annex A table A.3), if you have the incoming bytes... [encoding byte] [content] The way I read it, I think what you would get out is: [encoding byte... [B]if <= 0x05[/B]] [content - code points constrained to c <= 0x05, 0x20 <= c < 0x80, c >= 0x9f] With three byte encoding (EN 300 468 annex A table A.4), if you have the incoming bytes: 0x10 0x00 [encoding byte] [content] The way I read it, I think what you would get out is: 0x10 [encoding byte] [encoding byte... [B]if <= 0x05[/B]] [content - code points constrained to c <= 0x05, 0x20 <= c < 0x80, c >= 0x9f] What is the problem here? For one byte encoding, the encoding byte will be thrown away unless it is <= 0x05. That means no support for anything other than ASCII and ISO/IEC 8859-5..9. That is not the end of it. Consider what TV library does with three byte encoding: [code] byte c = Marshal.ReadByte(ptr, 0); if (c < 0x20) ... case 0x10: { pos = 3; c = Marshal.ReadByte(ptr, 2);[/code] TV library seems to have forgotten that the second encoding byte had to be removed in order to avoid premature NULL termination. In other words, TsWriter wrote 0x10 [encoding byte] but TV library is still expecting 0x10 0x00 [encoding byte]. This can only work if the encoding byte is <= 0x05. So it looks like ISO/IEC 8859-1..5 would work, but ISO/IEC 8859-6+ would not. If I'm right, these are quite serious flaws. Do you agree? Continuing a bit further... In the EPG text decoding we see this code: [code] if (language.ToUpperInvariant() == "CZE" || language.ToUpperInvariant() == "CES") { title = Iso6937ToUnicode.Convert(ptrTitle); description = Iso6937ToUnicode.Convert(ptrDesc); } else { title = DvbTextConverter.Convert(ptrTitle); description = DvbTextConverter.Convert(ptrDesc); } [/code] The start of a hack if ever I saw one! :( In Iso6937ToUnicode we have: [code] if (b < 0x20) { // ISO 6937 encoding must start with character between 0x20 and 0xFF // otherwise it is dfferent encoding table // for example 0x05 means encoding table 8859-9 // here is just fallback to system ANSI return Marshal.PtrToStringAnsi(ptr); }[/code] What is going on here?!? It looks like somebody decided all Czech EPG should be decoded as ISO 6937, but then when proper DVB encoding is detected they fall back to ANSI. The hack is bad enough, but surely that PtrToStringAnsi() call should at least be replaced with DvbTextConverter.Convert(). Do you agree? Finally, in DvbTextConverter: [code] int encoding = CultureInfo.CurrentCulture.TextInfo.ANSICodePage; try { if (string.IsNullOrEmpty(lang)) { lang = CultureInfo.CurrentCulture.ThreeLetterISOLanguageName; } if (lang.Equals("cze") || lang.Equals("ces")) { encoding = 20269; //ISO-6937 } else if (lang.Equals("ukr") || lang.Equals("bel") || lang.Equals("rus")) { encoding = 28595; //ISO-8859-5 }[/code] Another hack. :( Since language is always passed as "" (empty string), any person with OS lang/culture set to Ukranian, Belarussian or Russian will have ISO 8859-5 as default encoding. According to my understanding of EN 300 468, the default encoding should be in figure A.1 - ISO/IEC 6937 with a modification for code point 0xa4 (Euro symbol). So Czech may be the only culture that gets the proper default encoding. In summary there seem to be some really nasty bugs and hacks in here. I don't know if any of these directly affect gurabli. Maybe you could confirm? mm [/QUOTE]

Verification