[fixed] Video and sound syncronisation (1 Viewer)

[tourettes]

Re: AW: Video and sound syncronisation

today I spent some more hours in testing, here are some facts about current situtation:

Nice to see a new directshow developer in the team

1) Scheduling
The evr presenter schedules frames, present them if they are in due time (<1/4 of frame duration).
Here I found one issue: the "Kate Bush..." sample clip has a 29.97 frame rate, but on startup it's detected falsly as 25 fps. I have now fixed this inside scheduler by

hr = pSample->GetSampleDuration(&hnsSampleDuration);

to read the sample's duration. This divided by 4 is the new limit to callback.

At least MS decoder does throttling when CPU / GPU limit is reached and it will change those on the fly - not sure if it affects current code but at least for the audio renderer & A/V sync we need the real fps and not the one that decoder provides (MP1's renderer in audio renderer bnrach has a workaround, it uses the value provided by the source filter).

          using (Surface surf = Surface.FromPointer(new IntPtr(dwSurface)))
          {
            GraphicsDevice.Device.StretchRectangle(surf, cropRect,
                _surface, new Rectangle(Point.Empty, _croppedVideoSize), TextureFilter.None);
          }

StretchRect shouldn't be used - at least in the long run. It has some severe issues with ATI have (latest only) - https://forum.team-mediaportal.com/development-mp-1-1-0-526/possibility-revert-ati-hack-78263/

Here's what to use instead in the future (texture based) - https://forum.team-mediaportal.com/591332-post157.html

          _device.PresentEx(Present.ForceImmediate);

Not sure if we want to have that parameter, since it seems to cause D3DPRESENT_INTERVAL_IMMEDIATE to be used (which causes tearing).

Windowed mode supports D3DPRESENT_INTERVAL_DEFAULT, D3DPRESENT_INTERVAL_IMMEDIATE, and D3DPRESENT_INTERVAL_ONE. D3DPRESENT_INTERVAL_DEFAULT and the D3DPRESENT_INTERVAL_ONE are nearly equivalent (see the information regarding timer resolution below). They perform similarly to COPY_VSYNC in that there is only one present per frame, and they prevent tearing with beam-following. In contrast, D3DPRESENT_INTERVAL_IMMEDIATE will attempt to provide an unlimited presentation rate.

This means, we use the ForceImmediate call that is not bound to vsync (see D3DPRESENT (Windows))

See previous comment

Not sure if it improves the situation, but my idea was to have to GUI already rendered and then we can immediatly PresentEx()...

Idea is good, I hope it will give a bit more time for the video rendering (1 ms maybe ).

3) Available statistics
The frame scheduling part can use collected statistics about PresentCount, PresentRefreshCount, SyncRefreshCount, SyncQPCTime, SyncGPUTime (IDirect3DSwapChain9Ex::GetPresentStatistics Method (Windows), D3DPRESENTSTATS Structure (Windows))

This can be used to improve the scheduling quality more.

Glitch recovery should be also implemented at some point - based on those stats. There is also some sample code how to do it - Direct3D9Ex Flip Mode Present and Present Statistics Sample - Home

 

[morpheus_xx]

AW: Video and sound syncronisation

I think I have to correct some parts of my former description (you see, I'm still learning to understand this process)

- The EVR callback does the StretchRect to fill the texture (created from RendertTarget).
- This texture is taken by the VideoBrush, which can be used to paint UI elements with video content.
- Shader effects can be applied when this texture gets presented (sharpen, greyscale...)

This means at least:
my idea to draw the GUI before a video frame arrives is useless, because the VideoBrush is used for UI elements and takes the currently available texture. Indeed it is correct to wait until the frame is there, then starting the rendering of whole GUI, including the VideoBrush'ed controls.

 

[morpheus_xx]

AW: Video and sound syncronisation

One additional thought about multi-threading:

I started MP2 and set the associated CPU cores only to CPU0 in task manager. I have the feeling that it improves the video playback and shows less frame issues...

 

[tourettes]

Re: AW: Video and sound syncronisation

One additional thought about multi-threading:

I started MP2 and set the associated CPU cores only to CPU0 in task manager. I have the feeling that it improves the video playback and shows less frame issues...

It will just hide some real threading issue

 

[tourettes]

Re: AW: Video and sound syncronisation

- The EVR callback does the StretchRect to fill the texture (created from RendertTarget).

StrectRect shouldn't be used at all. It is something in DirectX that is not workign that well between different manufacturers (ATI being quite bad).

Instead of the extra surface copy (that gives performance hit as well) . Have a look on MP1 code (if you dare ), it is possible to ignore that surface copy - you can access the texture from a render target:

    CComPtr<IDirect3DTexture9> pTexture = NULL;
    pSurface->GetContainer(IID_IDirect3DTexture9, (void**)&pTexture);

And the pass that texture directly to the C# side.

This means at least:
my idea to draw the GUI before a video frame arrives is useless, because the VideoBrush is used for UI elements and takes the currently available texture. Indeed it is correct to wait until the frame is there, then starting the rendering of whole GUI, including the VideoBrush'ed controls.

Ok, then indeed we cannot use the small performance gain with video rendering. In any case it shouldn't be that big loss at least with the full screen video. It might be bigger loss on slow systems when whole MP2 navigation (list for examle) is rendered on top of the video.

 

[morpheus_xx]

AW: Re: AW: Video and sound syncronisation

I think it's possible to pass the texture directly back to c# side, I've already seen this code.

One thing I'm missing is the way how to "crop" the texture. Currently the surface applies the crop settings before stretching it to the target texture. If we don't use the surface, this cropping must be solved in another way.

Regarding the threading, I didn't suggest a limit to one CPU core . It was more meant as an indicator for a threading issue.

I see currently only one potential threading issue:

-> video frame arrives (surface) -> StretchRect -> texture -> GUI.Render (include VideoBrush to render the video frame to our target)

if this Render() process is still active, and the next frame already arrives, the StretchRect could overwrite the texture contents that are currently painted.


I also thought about the frame scheduling:
assume we have 25 fps clip (40 ms frame time), and schedule the frame when it's at 1/4th to present time (this means 10 msec before present). The GUI rendering takes different amount of time, I measured between 1 and 20 msec (the higher number are seldom, i.e. navigating to media subfolders). But anyway, a 20 msec rendering time causes the real video frame to be displayed late, what introduces glitches.

An idea would be to know the last GUI render time (do we know this only in C# or can the renderer stats tell us?) and re-calculate the time offset to present the frame (instead fixed of 1/4th of frame time, we subtract the last rendering time as well).

 

[tourettes]

Re: AW: Re: AW: Video and sound syncronisation

I think it's possible to pass the texture directly back to c# side, I've already seen this code.

One thing I'm missing is the way how to "crop" the texture. Currently the surface applies the crop settings before stretching it to the target texture. If we don't use the surface, this cropping must be solved in another way.

Yes, it is possible since MP1 does it already

About the cropping, maybe it requires the vertex setup to be differently to gain same effect as StrechRect is used.

-> video frame arrives (surface) -> StretchRect -> texture -> GUI.Render (include VideoBrush to render the video frame to our target)

if this Render() process is still active, and the next frame already arrives, the StretchRect could overwrite the texture contents that are currently painted.

I also thought about the frame scheduling:
assume we have 25 fps clip (40 ms frame time), and schedule the frame when it's at 1/4th to present time (this means 10 msec before present). The GUI rendering takes different amount of time, I measured between 1 and 20 msec (the higher number are seldom, i.e. navigating to media subfolders). But anyway, a 20 msec rendering time causes the real video frame to be displayed late, what introduces glitches.

This is a bit problematic now since the rendering logic is done in two places - it might be good idea to move it completely on C++ side.

Could be solved either by giving the full scheduling / presenting responsibility for the C++ side (then frame dropping would be automatic) or by dropping the frame on C# side if the rendering pass is still ongoing.

Of course the best solution would be to take care that Skin Engine (or layouting or anything else) wont ever cause the rendering times to exceed few ms. In some cases this is not possible - for example drivers could be buggy, some non-GPU driver could hog DPC, 3rd party program could hog CPU, AERO can do own quirks and so on. So the code needs to be able to keep the A/V sync in cases where there is Present() call that lasts too long.

An idea would be to know the last GUI render time (do we know this only in C# or can the renderer stats tell us?) and re-calculate the time offset to present the frame (instead fixed of 1/4th of frame time, we subtract the last rendering time as well).

But the frame is already late (Skin Engine's fault this time ) so it should be dropped - also it wont change the presentation time of the future frames at all (if you modify those it will cause A/V sync issues). V-sync on the display device doesn't care if some middle component (Skin Engine) causes some delay. Video frame needs to be presented on the correct v-sync or it needs to be dropped if it cannot be done (no matter what is the reason).

 

[morpheus_xx]

I want to give an update of the situation. I think the presenter and its scheduler are working correctly.

I added some more performance measuring in the SkinEngines rendering process. I got times from 6 to 15 msec to render the whole screen (including the video texture).

The callback from Evr presenter is done when the target present time is less than 3/4 of the frame duration. For 50 fps video these are 3/4 of 20ms=15 ms (maximum!)

I currently have not yet measured the real difference from frame arrival to present time but I guess it's less than 15 ms.

For me this means: the frame gets passed to C# code and it's used in the next render cycle, that probably takes longer than the frames target present time. This can also lead to a small a/v synch issue depending on the gui rendering time.

I will add more time measurement to both parts on sunday.

If this is the root cause, we need to make the gui rendering faster to match the frame present time. At least we must use as much time as we can in Evr callback, meaning to pass every frame back as soon as it arrives.

Sent from my GT-I9000 using Tapatalk

 

[morpheus_xx]

AW: Video and sound syncronisation

I added some logging into the "ScheduleSample" method of the EVR presenter. It's called when a frame arrived and is put into the queue.

If I calculated it correctly, the frames are arriving quite early, so we probably have more than enough time to pass them back and render the GUI.

Here are some measurements, I print the average time from arrival to target display time (in ms) after each 50 frames.

Playback of a 25fps clip, DXVA "accellerated" -> 50 fps:

11-09-2011 20:15:11.186 [206c]EVRCustomPresenter::OnClockStart (offset = 6103566850000)
11-09-2011 20:15:12.153 [2204]avg time to present: 64.320000 ms
11-09-2011 20:15:13.134 [2204]avg time to present: 72.680000 ms
11-09-2011 20:15:14.134 [2204]avg time to present: 73.000000 ms
11-09-2011 20:15:15.132 [2204]avg time to present: 73.060000 ms
11-09-2011 20:15:16.132 [2204]avg time to present: 73.040000 ms
11-09-2011 20:15:17.130 [2204]avg time to present: 73.020000 ms
11-09-2011 20:15:18.130 [2204]avg time to present: 73.000000 ms
11-09-2011 20:15:19.128 [2204]avg time to present: 73.040000 ms
11-09-2011 20:15:20.126 [2204]avg time to present: 73.120000 ms
11-09-2011 20:15:21.122 [2204]avg time to present: 73.080000 ms
11-09-2011 20:15:22.121 [2204]avg time to present: 73.080000 ms
11-09-2011 20:15:23.117 [2204]avg time to present: 72.940000 ms

Playback of a film (23.9.. fps):

11-09-2011 20:15:39.165 [180c]EVRCustomPresenter::OnClockStart (offset = 6103846719999)
11-09-2011 20:15:40.751 [231c]avg time to present: 113.820000 ms
11-09-2011 20:15:42.405 [231c]avg time to present: 123.900000 ms
11-09-2011 20:15:44.070 [231c]avg time to present: 123.940000 ms
11-09-2011 20:15:45.734 [231c]avg time to present: 122.620000 ms
11-09-2011 20:15:47.401 [231c]avg time to present: 124.000000 ms
11-09-2011 20:15:49.063 [231c]avg time to present: 123.920000 ms
11-09-2011 20:15:50.731 [231c]avg time to present: 123.940000 ms
11-09-2011 20:15:52.396 [231c]avg time to present: 123.980000 ms
11-09-2011 20:15:54.060 [231c]avg time to present: 123.980000 ms
11-09-2011 20:15:55.723 [231c]avg time to present: 123.860000 ms

Playback of a 25 fps mkv.

11-09-2011 20:16:10.542 [300]EVRCustomPresenter::OnClockStart (offset = 6104160500000)
11-09-2011 20:16:12.507 [58c]avg time to present: 141.360000 ms
11-09-2011 20:16:14.492 [58c]avg time to present: 148.900000 ms
11-09-2011 20:16:16.487 [58c]avg time to present: 148.980000 ms
11-09-2011 20:16:18.482 [58c]avg time to present: 148.940000 ms
11-09-2011 20:16:20.479 [58c]avg time to present: 149.000000 ms
11-09-2011 20:16:22.477 [58c]avg time to present: 148.980000 ms
11-09-2011 20:16:24.473 [58c]avg time to present: 148.860000 ms
11-09-2011 20:16:26.470 [58c]avg time to present: 148.960000 ms
11-09-2011 20:16:28.464 [58c]avg time to present: 148.900000 ms
11-09-2011 20:16:30.462 [58c]avg time to present: 148.980000 ms

Are these plausible values? Are the frames really decoded so early and then simply hold back in the queue?

My idea is as follows:
We should modify the scheduler to pass frames earlier to C# side to be available inside GUI rendering. The amount of "pre-present-time" can be dynamically adopted by the last (n?) rendering time(s).

Can this work?

 

[tourettes]

Re: AW: Video and sound syncronisation

I added some logging into the "ScheduleSample" method of the EVR presenter. It's called when a frame arrived and is put into the queue.

If I calculated it correctly, the frames are arriving quite early, so we probably have more than enough time to pass them back and render the GUI.

But it wont make any more time for the actual presentation

Are these plausible values? Are the frames really decoded so early and then simply hold back in the queue?[/quote]

I think they are quite valid, looks like the buffer would be around 4 frames?

My idea is as follows:
We should modify the scheduler to pass frames earlier to C# side to be available inside GUI rendering. The amount of "pre-present-time" can be dynamically adopted by the last (n?) rendering time(s).

Can this work?

It could but... does the C# side code even try to present at optimal raster offset (v-sync)? Also if you move the sync code to C# side you will generate some uncescessary overhead (DirectX works on native side).

If you ask from me I wouldn't try to do the scheduling in C# side. Is there any specific (mainly technical) reason why it can't be done in C++ side?

Also I would ask from Olwsroosts' opinion about the threads, C#, etc. as he has quite long experience with fighting different issues (in native only!) presentation environment.

As a 1st step (before asking from Tony ) I would implement a moving bar into the video frame - it helps you to estimate the quality quite quickly.