[fixed] Video and sound syncronisation (1 Viewer)

[morpheus_xx]

AW: Re: AW: Video and sound syncronisation

If you ask from me I wouldn't try to do the scheduling in C# side. Is there any specific (mainly technical) reason why it can't be done in C++ side?

I quote myself here as answer:

- The EVR callback does the StretchRect to fill the texture (created from RendertTarget).
- This texture is taken by the VideoBrush, which can be used to paint UI elements with video content.
- Shader effects can be applied when this texture gets presented (sharpen, greyscale...)

This means at least:
my idea to draw the GUI before a video frame arrives is useless, because the VideoBrush is used for UI elements and takes the currently available texture. Indeed it is correct to wait until the frame is there, then starting the rendering of whole GUI, including the VideoBrush'ed controls.

In contrast to MP1, we don't present the video surface/texture directly when it is pushed back from presenter to C#. It's passed to the GUI rendering and it can be used in 1 (or many!!!) controls, with applied transformations (rotate, scale etc.). The video frame is a "simple texture" in this case.

The time we do the callback then should take into account, how much time is need to render the complete GUI scene.

It could but... does the C# side code even try to present at optimal raster offset (v-sync)? Also if you move the sync code to C# side you will generate some uncescessary overhead (DirectX works on native side).

Didn't you quote this?

Windowed mode supports D3DPRESENT_INTERVAL_DEFAULT, D3DPRESENT_INTERVAL_IMMEDIATE, and D3DPRESENT_INTERVAL_ONE. D3DPRESENT_INTERVAL_DEFAULT and the D3DPRESENT_INTERVAL_ONE are nearly equivalent (see the information regarding timer resolution below). They perform similarly to COPY_VSYNC in that there is only one present per frame, and they prevent tearing with beam-following. In contrast, D3DPRESENT_INTERVAL_IMMEDIATE will attempt to provide an unlimited presentation rate.

We use PresentEx with Interval One, shouldn't be the completly rendered frame be correctly presented?

 

[tourettes]

Re: AW: Re: AW: Video and sound syncronisation

If you ask from me I wouldn't try to do the scheduling in C# side. Is there any specific (mainly technical) reason why it can't be done in C++ side?

I quote myself here as answer:

- The EVR callback does the StretchRect to fill the texture (created from RendertTarget).
- This texture is taken by the VideoBrush, which can be used to paint UI elements with video content.
- Shader effects can be applied when this texture gets presented (sharpen, greyscale...)

There seems to be extra overhead with that design - you will copy the texture (with StretchRect) where as you could use the texture directly provided by the EVR presenter (see MP1 code how it could be done - it is not complex, just requires the C++ to gain the access to the texture instead of the surface only).

StretchRect should be avoided as much as possible - it is known to be broken on ATi drivers / GPUs. There is an internal thread about it.

In contrast to MP1, we don't present the video surface/texture directly when it is pushed back from presenter to C#. It's passed to the GUI rendering and it can be used in 1 (or many!!!) controls, with applied transformations (rotate, scale etc.). The video frame is a "simple texture" in this case.

What MP1 does on EVR callback is that it will render the whole GUI - it is not presented directly.

The time we do the callback then should take into account, how much time is need to render the complete GUI scene.

It should be merely few ms at max - MP2 GUI is far less complex than a few years old video games have. If GUI drawing starts to take more than that we are on wrong track - video playback quality must come before the GUI eyecandy But I think it wont be a bottleneck here, MP1's rendering engine is highly inefficient (no batching support etc.) and it is giving already good performance.

Didn't you quote this?

Windowed mode supports D3DPRESENT_INTERVAL_DEFAULT, D3DPRESENT_INTERVAL_IMMEDIATE, and D3DPRESENT_INTERVAL_ONE. D3DPRESENT_INTERVAL_DEFAULT and the D3DPRESENT_INTERVAL_ONE are nearly equivalent (see the information regarding timer resolution below). They perform similarly to COPY_VSYNC in that there is only one present per frame, and they prevent tearing with beam-following. In contrast, D3DPRESENT_INTERVAL_IMMEDIATE will attempt to provide an unlimited presentation rate.

We use PresentEx with Interval One, shouldn't be the completly rendered frame be correctly presented?

If you look at MP1's presenter you will see that there is quite biig amount of logic to make sure that the Present() gets called on optimal v-sync position. Even on 1:1 matches it needs more than just <frame start time> to <frame end time> logic. non 1:1 cases are much more complex (and I have no clue about the current implementation of those - Tony will be able to help ).

This would be doable in C# I think. But the specific v-sync position needs to be sent to the MP Audio Renderer (or not the position, but drift calculation results per frame). In case MP2 wants to use the MP Audio Renderer

 

[morpheus_xx]

AW: Video and sound syncronisation

Some update on the progress and findings: First of all, I started a new branch "EVR_v2", so the code changes can be found there. The older branch I will delete soon.

I've implemented rendering of a tearing test (flowing lines) and rendering some stats as text on screen. To enable stats+tearing test hit "F1", to use maximum FPS hit "F2". In maximum FPS mode the GUI (Rising Skin) renders at ~220 fps (windowed) or ~130 fps (fullscren 1680x1050) on my DevPC. It uses PresentEx(Present.ForceImmediate). Even this causes NO tearing at all.

Another note on rendering performance: on my embedded ATI HD4200 I get only 20(!)fps in fullscreen (1920x1080), but using default Skin I get ~55fps. The GPU load is at 100% then, but CPU is not much used.

The callback from EVR was changed to use Texture instead of Surface (this was already the case in the older branch!).

Findings regarding the frame stuttering:
When the Kate Bush clip is decoded on Cyberlink MPEG decoder, it produces some wrong timed frames. They are scheduled like:

F1(P)-> F2(Q)-> F3(Q) -> F1(E)

 P: just presented
 Q: queued  
 E: to be enqueued again (wrong frame).

I changed the code to detect the correct order and reject those wrong frames. It works on DevPC, but on HTPC using MS decoder I don't get the wrong times, but small stutters in playback. The check I introduced requires an different handling when you are inside DVD-menu. There you will get non-steady frames and they have to be presented. This means to me: keep the check in and add to DVD-menu detection like in MP1 (not sure if this is good), or drop the check as it only cures some bad decoders.

 

[Albert]

AW: Video and sound syncronisation

If the problems are really caused by bad decoders, I'm wondering why the WMP plays the Kate Bush clip and my other test clip "Cylob - Rewind" correctly while MP2 shows those wrong frames in both test clips. Do we use other codecs in MP2? If yes, why? If the problem can be fixed by using correct codecs, we should solve the problem this way.

The handling of those wrong frames is a bad hack. Hacks in MP2 are ok, if they are necessary to cure problems of third-party-components like that codec. But if we decide to use such hacks, they should not be weaved into the actual productive code. They should be placed in separate classes together with an exact description of the bug and the bugfix. The system should only use that code if its usage condition is met (here: if one of the decoders is in use which is known to produce the problems). Else, the system should use the "normal" code without any hacks.

 

[tourettes]

Re: AW: Video and sound syncronisation

If the problems are really caused by bad decoders, I'm wondering why the WMP plays the Kate Bush clip and my other test clip "Cylob - Rewind" correctly while MP2 shows those wrong frames in both test clips. Do we use other codecs in MP2? If yes, why? If the problem can be fixed by using correct codecs, we should solve the problem this way.

WMP doesn't ever use Cyberlink decoder.

What would be good to test is how MP1 behaves. It doesn't have such frame ordering hack and if it plays nicely those clips then there is probably some bug in MP2 EVR presenter code.

What could cause such?

1) Codec tries to be more inteligent than it is nescessary - non maching refresh rate could cause it to generate frames with odd timestamps. MS decoder does this for example when the GPU / CPU power is not enough (it throttles the fps and timestamps!)

2) Deinterlacing fails

3) Something related to the inverse telecine - what type of sample that is?

 

[tourettes]

Re: AW: Video and sound syncronisation

Another note on rendering performance: on my embedded ATI HD4200 I get only 20(!)fps in fullscreen (1920x1080), but using default Skin I get ~55fps. The GPU load is at 100% then, but CPU is not much used.

It would be good to profile what causes the bad performance - from video playback quality point of view it is not good enough. Skin Engine nor skin shouldn't be as hard to GPU or we will have quality issues on video playback area. The default skin is pretty basic so it should be performing much better to allow more complex (read better looking ) skins to be used on integrated GPUs.

 

[Chimen]

Is there any progress in this bug? I follow this thread because I think it's intresting.
But now there hasn't been any updates for a while...

 

[morpheus_xx]

AW: Video and sound syncronisation

Hi,

all changes I've done can be found in the "...EVR_v2" branch. In my opinion this version works better than the "master" version, but there are still issues with wrong, flashing frames and stuttering. This can depend on your hardware though.

The issue with a/v sync I do not have, so I cannot tell if it's better in my branch.

I would really like to get some more testing feedback of the EVR_v2 branch, so I know in what direction I should develop...

Additional note: MP2 build from this branch have 2 diagnostic features: the key "F1" turns on on-screen-statistics and a tearing test; "F2" toggles between vsynced and unlimited framerate ("Max FPS").

 

[morpheus_xx]

AW: Video and sound syncronisation

Here is a very fresh build of the EVR_v2 branch. I did a rebase before, so this version is also up-to-date with the master.

Download you find here: Free uploads without any delay - Start uploading now. Please also unzip the attached file to the main folder (with MP2-client.exe), the dlls seem to be missing...

Please try to run this version, playback different kind of videos: .TS (MPEG2/MPEG4, SD/HD), report if playback is smooth, if a/v is in synch and what codecs you use.

I would also need some information about (please use this as template!):
CPU (type/freq):
RAM size:
Graphics (type, size of memory, screen resolution, refresh rate, fullscreen y/n):
MP2-Skin:
FPS (max. fps enabled, fullscreen) in home screen, idle:

than what type of video you played
Video format/Container: ..., FPS:, ... Resolution:, ... A/V synch: ok/not ok?, Codec: ...

 

[morpheus_xx]

AW: Video and sound syncronisation

My DEV-PC:
CPU (type/freq): Intel Q6600 Quad Core 2.4GHz
RAM size: 4GB
Graphics (type, size of memory, screen resolution, refresh rate, fullscreen y/n): ATI HD 5670, 512 MB GDDR3, 1680x1050@60Hz
MP2-Skin: Rising
FPS (max. fps enabled, fullscreen) in home screen, idle: 204 fps

1) Video format/Container: AVC,mkv FPS: 25, Resolution:, 720x576 A/V synch/playback quality: ok, no stuttering Codec: ffdshow video
2) Video format/Container: AVC, TSFPS: 50, Resolution:, 1280x720 A/V synch/playback quality: ok, no stuttering Codec: ffdshow video

----------------------------------------------------------------------
HTPC:
CPU (type/freq): AMD Athlon x2 215 2.7GHz
RAM size: 4GB
Graphics (type, size of memory, screen resolution, refresh rate, fullscreen y/n): ATI HD 4200, 768 MB HyperMemory (shared), 1920x1080@50Hz
MP2-Skin: Rising
FPS (max. fps enabled, fullscreen) in home screen, idle: 21 (!) fps

3) Video format/Container: MPEG2, TS FPS: 25 (DXVA 50), Resolution:, 720x576 A/V synch/playback quality: ok, bit jerky, Codec: Microsoft DTV-DVD
4) Video format/Container: AVC, TSFPS: 50, Resolution:, 1280x720 A/V synch/playback quality: ok, bit jerky, Codec: ffdshow video

The MP2 version from master-branch is working better on my HTPC, playback is not jerky. Main difference in EVR_v2 is the use of textures in callback from evrpresenter (instead of the surface)...

But on DevPC it's the opposite way...

Detail 1)

Spoiler

Format : Matroska
File size : 967 MiB
Duration : 59mn 58s
Overall bit rate : 2 253 Kbps
Writing application : HandBrake 0.9.4

Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : Main@L3.0
Format settings, CABAC : Yes
Format settings, ReFrames : 2 frames
Codec ID : V_MPEG4/ISO/AVC
Duration : 59mn 58s
Bit rate : 1 800 Kbps
Width : 718 pixels
Height : 576 pixels
Display aspect ratio : 16:9
Frame rate : 25.000 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.174
Stream size : 755 MiB (78%)
Writing library : x264 core 79
Encoding settings : cabac=1 / ref=2 / deblock=1:0:0 / analyse=0x1:0x111 / me=hex / subme=6 / psy=1 / psy_rd=1.0:0.0 / mixed_ref=0 / me_range=16 / chroma_me=1 / trellis=0 / 8x8dct=0 / cqm=0 / deadzone=21,11 / chroma_qp_offset=-2 / threads=3 / nr=0 / decimate=1 / mbaff=0 / constrained_intra=0 / bframes=2 / b_pyramid=0 / b_adapt=1 / b_bias=0 / direct=1 / wpredb=0 / wpredp=2 / keyint=250 / keyint_min=25 / scenecut=40 / rc_lookahead=40 / rc=2pass / mbtree=1 / bitrate=1800 / ratetol=1.0 / qcomp=0.60 / qpmin=10 / qpmax=51 / qpstep=4 / cplxblur=20.0 / qblur=0.5 / ip_ratio=1.40 / aq=1:1.00
Language : English
Color primaries : BT.601-6 525, BT.1358 525, BT.1700 NTSC, SMPTE 170M
Transfer characteristics : BT.709-5, BT.1361
Matrix coefficients : BT.601-6 525, BT.1358 525, BT.1700 NTSC, SMPTE 170M

Audio
ID : 2
Format : AC-3
Format/Info : Audio Coding 3
Mode extension : CM (complete main)
Codec ID : A_AC3
Duration : 59mn 58s
Bit rate mode : Constant
Bit rate : 448 Kbps
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 48.0 KHz
Bit depth : 16 bits
Compression mode : Lossy
Stream size : 192 MiB (20%)

Detail 2)

Spoiler

Format : MPEG-TS
Format/Info : Advanced Video Codec
File size : 9.33 GiB
Duration : 1h 40mn
Overall bit rate : 13.3 Mbps

Video
ID : 48 (0x30)
Menu ID : 137 (0x89)
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L4.0
Format settings, CABAC : Yes
Format settings, ReFrames : 5 frames
Codec ID : 27
Duration : 1h 40mn
Bit rate mode : Variable
Bit rate : 11.7 Mbps
Maximum bit rate : 11.5 Mbps
Width : 1 280 pixels
Height : 720 pixels
Display aspect ratio : 16:9
Frame rate : 50.000 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.254
Stream size : 8.19 GiB (88%)
Color primaries : BT.709-5, BT.1361, IEC 61966-2-4, SMPTE RP177
Transfer characteristics : BT.709-5, BT.1361
Matrix coefficients : BT.709-5, BT.1361, IEC 61966-2-4 709, SMPTE RP177

Audio #1
ID : 64 (0x40)
Menu ID : 137 (0x89)
Format : MPEG Audio
Format version : Version 1
Format profile : Layer 2
Codec ID : 3
Duration : 1h 40mn
Bit rate mode : Constant
Bit rate : 256 Kbps
Channel(s) : 2 channels
Sampling rate : 48.0 KHz
Compression mode : Lossy
Delay relative to video : -1s 69ms
Stream size : 183 MiB (2%)
Language : German

Audio #2
ID : 65 (0x41)
Menu ID : 137 (0x89)
Format : MPEG Audio
Format version : Version 1
Format profile : Layer 2
Codec ID : 3
Duration : 1h 40mn
Bit rate mode : Constant
Bit rate : 256 Kbps
Channel(s) : 2 channels
Sampling rate : 48.0 KHz
Compression mode : Lossy
Delay relative to video : -1s 66ms
Stream size : 183 MiB (2%)

Audio #3
ID : 66 (0x42)
Menu ID : 137 (0x89)
Format : AC-3
Format/Info : Audio Coding 3
Mode extension : CM (complete main)
Codec ID : 6
Duration : 1h 40mn
Bit rate mode : Constant
Bit rate : 448 Kbps
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 48.0 KHz
Bit depth : 16 bits
Compression mode : Lossy
Delay relative to video : -1s 138ms
Stream size : 321 MiB (3%)
Language : German

Text #1
ID : 80 (0x50)
Menu ID : 137 (0x89)
Format : DVB Subtitle
Codec ID : 6
Language : German

Text #2
ID : 142 (0x8E)-100
Menu ID : 137 (0x89)
Format : Teletext
Language : German

Detail 3)

Spoiler

Format : MPEG-TS
File size : 1.66 GiB
Duration : 59mn 50s
Overall bit rate : 3 966 Kbps

Video
ID : 48 (0x30)
Menu ID : 137 (0x89)
Format : MPEG Video
Format version : Version 2
Format profile : Main@Main
Format settings, BVOP : Yes
Format settings, Matrix : Default
Format settings, GOP : M=3, N=12
Codec ID : 2
Duration : 59mn 50s
Bit rate mode : Variable
Bit rate : 3 382 Kbps
Nominal bit rate : 15.0 Mbps
Width : 720 pixels
Height : 576 pixels
Display aspect ratio : 16:9
Frame rate : 25.000 fps
Standard : PAL
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Interlaced
Scan order : Top Field First
Compression mode : Lossy
Bits/(Pixel*Frame) : 0.326
Stream size : 1.41 GiB (85%)

Audio #1
ID : 64 (0x40)
Menu ID : 137 (0x89)
Format : MPEG Audio
Format version : Version 1
Format profile : Layer 2
Codec ID : 3
Duration : 59mn 50s
Bit rate mode : Constant
Bit rate : 192 Kbps
Channel(s) : 2 channels
Sampling rate : 48.0 KHz
Compression mode : Lossy
Delay relative to video : -184ms
Stream size : 82.2 MiB (5%)
Language : German
Language, more info : Clean effects

Audio #2
ID : 65 (0x41)
Menu ID : 137 (0x89)
Format : MPEG Audio
Format version : Version 1
Format profile : Layer 2
Codec ID : 3
Duration : 59mn 50s
Bit rate mode : Constant
Bit rate : 192 Kbps
Channel(s) : 2 channels
Sampling rate : 48.0 KHz
Compression mode : Lossy
Delay relative to video : -184ms
Stream size : 82.2 MiB (5%)
Language : English
Language, more info : Clean effects

Text
ID : 129 (0x81)-100
Menu ID : 137 (0x89)
Format : Teletext
Language : German

Detail 4)

Spoiler

Format : MPEG-TS
Format/Info : Advanced Video Codec
File size : 3.93 GiB
Duration : 40mn 8s
Overall bit rate : 14.0 Mbps

Video
ID : 48 (0x30)
Menu ID : 137 (0x89)
Format : AVC
Format/Info : Advanced Video Codec
Format profile : Main@L4.0
Format settings, CABAC : Yes
Format settings, ReFrames : 5 frames
Codec ID : 27
Duration : 40mn 8s
Bit rate : 12.2 Mbps
Width : 1 280 pixels
Height : 720 pixels
Display aspect ratio : 16:9
Frame rate : 50.000 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.265
Stream size : 3.42 GiB (87%)
Color primaries : BT.709-5, BT.1361, IEC 61966-2-4, SMPTE RP177
Transfer characteristics : BT.709-5, BT.1361
Matrix coefficients : BT.709-5, BT.1361, IEC 61966-2-4 709, SMPTE RP177

Audio #1
ID : 64 (0x40)
Menu ID : 137 (0x89)
Format : MPEG Audio
Format version : Version 1
Format profile : Layer 2
Codec ID : 3
Duration : 40mn 8s
Bit rate mode : Constant
Bit rate : 256 Kbps
Channel(s) : 2 channels
Sampling rate : 48.0 KHz
Compression mode : Lossy
Delay relative to video : -1s 148ms
Stream size : 73.5 MiB (2%)
Language : German

Audio #2
ID : 65 (0x41)
Menu ID : 137 (0x89)
Format : MPEG Audio
Format version : Version 1
Format profile : Layer 2
Codec ID : 3
Duration : 40mn 8s
Bit rate mode : Constant
Bit rate : 192 Kbps
Channel(s) : 2 channels
Sampling rate : 48.0 KHz
Compression mode : Lossy
Delay relative to video : -1s 188ms
Stream size : 55.1 MiB (1%)

Audio #3
ID : 66 (0x42)
Menu ID : 137 (0x89)
Format : MPEG Audio
Format version : Version 1
Format profile : Layer 2
Codec ID : 3
Duration : 40mn 8s
Bit rate mode : Constant
Bit rate : 192 Kbps
Channel(s) : 2 channels
Sampling rate : 48.0 KHz
Compression mode : Lossy
Delay relative to video : -1s 297ms
Stream size : 55.1 MiB (1%)
Language : qaa

Audio #4
ID : 67 (0x43)
Menu ID : 137 (0x89)
Format : AC-3
Format/Info : Audio Coding 3
Mode extension : CM (complete main)
Codec ID : 6
Duration : 40mn 8s
Bit rate mode : Constant
Bit rate : 448 Kbps
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 48.0 KHz
Bit depth : 16 bits
Compression mode : Lossy
Delay relative to video : -1s 299ms
Stream size : 129 MiB (3%)
Language : German

Text
ID : 163 (0xA3)-100
Menu ID : 137 (0x89)
Format : Teletext
Language : German