Audio sync problems when screen set to 24 Hz (2 Viewers)

tourettes

Retired Team Member
  • Premium Supporter
  • January 7, 2005
    17,301
    4,800
    The quick fix seems to work for me. I tested it with a couple of different files. Audio is always perfectly in sync shortly after playback started and stays this way. How many frames are skipped depends on how far it was off in the first place when playback started. Of course you need to adjust for any general processing delay to get audio perfectly spot on.

    Because of the late frames some frames get dropped every now and then. The reason for the late frames is still unknown (machine is nearly idle). But at least I have an acceptable workaround for now and we can investigate the reason for the late frames in more detail. It's a crude patch but it does its job for now.

    I'm still a bit puzzled about the real cause for the late frames. Why does it happen only with 24 fps / 23.976 fps content? Is it the source filter that screws us? Or EVR mixer? Or our EVR custom presenter code?

    ...after all dropped frames are as bad as the audio sync issue.

    If you don't have similar problems I suggest you don't try the patched DLL.

    Actually everyone should test the DLL, just to make sure we aren't breaking something with it.
     

    Scythe42

    Retired Team Member
  • Premium Supporter
  • June 20, 2009
    2,065
    2,703
    51
    Berlin
    Home Country
    Germany Germany
    So, after some testing the patch (proposed in your next post) should be added to SVN. It's relatively safe and shows the dropped frame stats increasing (that is easier to notice than the slowly drifting audio, so it will even help debugging the real issue later on).
    Probably the MAX_WAIT can be increased as well based on the refresh rate. With the current 20 it is dropping frames too early with the quick patch at 24Hz. A delay up to 40ms would be ok for 24fps playback (need to test it) because even though the frames are e.g 25ms late they can still be rendered in time and there's no need to drop them.

    I'll provide a more proper patch suggestions once I tried making most of the constants variable based on the refresh rate/media speed. The current provided patch is just a really crude hot fix helping to investigate the issue further and give other users a chance to use 24p and avoid the 60Hz judder. This currently comes with the price of a few dropped frames now and then (more than actually needed). That's why I suggested only people with very similar issues test this patch for now until I can provide a more general one.

    I need a few more days for a more proper frame dropping patch. But first I want to dig a bit deeper into the code to find out why the frames are late in the first place.

    I'm still a bit puzzled about the real cause for the late frames. Why does it happen only with 24 fps / 23.976 fps content? Is it the source filter that screws us? Or EVR mixer? Or our EVR custom presenter code?
    I'm puzzled, too. So far it only happens when selecting EVR in MP's config and material with a 23,976fps. And I only noticed it with HD so far. With VMR the issues don't come up. But on the other hand VMR just might drop frames and therefore I didn't saw the issue. I am way more sensitive too lip-sync issues than to a dropped frame every minute or so. I need to run some more tests here and see what debug logs tell me.

    Using different filters (CoreAVC, MPC, DivX, ffdshow etc.) showed all the same problem. I tried them all. Only constant was using EVR on Vista x64 so far. And what I observed was the issue of "late but render anyway". One symptom eliminated, on to the next. Now that I know what too look for in the logs I start over again with splitter, filter, audio renderer while keeping MP's EVR presenter.

    Because Win 7 didn't show the problem out of the box with the same config I can't rule out a general Vista/EVR/24Hz issue. But so far I couldn't find anything on MSDN in that regard.

    ...after all dropped frames are as bad as the audio sync issue.
    In this case a constant drift because all frames are always rendered is actually worse because you can't watch a movie at all without going crazy. But I agree with you that dropped frames are no good and should only occur when CPU/GPU are maxed out.

    I'll keep on testing and keep you guys updated if I find anything new why frames are late. And I provide a more general patch suggestion probably on Sunday. Have to code some stuff now at work and I'm out of town Friday and Saturday.
     

    tourettes

    Retired Team Member
  • Premium Supporter
  • January 7, 2005
    17,301
    4,800
    So, after some testing the patch (proposed in your next post) should be added to SVN. It's relatively safe and shows the dropped frame stats increasing (that is easier to notice than the slowly drifting audio, so it will even help debugging the real issue later on).
    Probably the MAX_WAIT can be increased as well based on the refresh rate. With the current 20 it is dropping frames too early with the quick patch at 24Hz.

    I'll provide a more proper patch suggestions once I tried making most of the constants variable based on the refresh rate/media speed. The current provided patch is just a really crude hot fix helping to investigate the issue further and give other users a chance to use 24p and avoid the 60Hz judder. This currently comes with the price of a few dropped frames now and then (more than actually needed). That's why I suggested only people with very similar issues test this patch for now until I can provide a more general one.

    I need a few more days for a more proper frame dropping patch. But first I want to dig a bit deeper into the code to find out why the frames are late in the first place.

    That was already on my todo list when it comes to playback improvements. Good if someone can provide the patch so I can concentrate on some other things and hopefully reveal something pretty interesting to "advanced" / "hifi" crowd of the MP user base (stay tuned :)).

    A delay up to 40ms would be ok for 24fps playback (need to test it) because even though the frames are e.g 25ms late they can still be rendered in time and there's no need to drop them.

    I think maximum allowed wait should be something close to the frame presentation time (there is slight few millisecond paint over head). So, for 25 fps it is 40 ms, for 50 fps 20 ms etc. ( I guess the actual refresh rate needs to be taken into consideration? So for 25fps PAL video on 75 Hz would actually be 13.33...ms. This one needs to be checked. Even if its not that, the deinterlacing has an effect on this).

    About the late frames, MS documentation says that no late frames should be rendered.

    Code:
    Discard the sample because it is late.

    I think it is not a good idea to render a frame that is already late on its presentation time, it would only cause visible judder as timeline of movement in the video won't be continuous. Also it would increase the case that the next frame is also late.

    I'm still a bit puzzled about the real cause for the late frames. Why does it happen only with 24 fps / 23.976 fps content? Is it the source filter that screws us? Or EVR mixer? Or our EVR custom presenter code?
    I'm puzzled, too. So far it only happens when selecting EVR in MP's config and material with a 23,976fps. And I only noticed it with HD so far. With VMR the issues don't come up. But on the other hand VMR just might drop frames and therefore I didn't saw the issue. I am way more sensitive too lip-sync issues than to a dropped frame every minute or so. I need to run some more tests here and see what debug logs tell me.

    Using different filters (CoreAVC, MPC, DivX, ffdshow etc.) showed all the same problem. I tried them all. Only constant was using EVR on Vista x64 so far. And what I observed was the issue of "late but render anyway". One symptom eliminated, on to the next. Now that I know what too look for in the logs I start over again with splitter, filter, audio renderer while keeping MP's EVR presenter.

    That should rule out the bug on codec side. Next step would be try to eliminate the digital audio connection as source of the late samples (audio clock could be drifting away from the video clock, althou the graph clock is sourced currently from the audio clock so it should cause dropped frames or juddering... not sync drifting... but still it is a good idea to rule out that audio renderer / device is not causing the sync drifting).

    Because Win 7 didn't show the problem out of the box with the same config I can't rule out a general Vista/EVR/24Hz issue. But so far I couldn't find anything on MSDN in that regard.

    Few users as well have reported that the issue is not reproducible with Windows 7. Did you install haali splitter with Windows 7 or does it provide some other source filter? Also could you check if MPC-HC source filter is making any difference?


    EC_SAMPLE_LATENCY (Windows) Maybe that could help EVR mixer or any other component to notice that samples are delivered too late?
     
    D

    DMember 49125

    Guest
    Update:

    The quick fix seems to work for me. I tested it with a couple of different files. Audio is always perfectly in sync shortly after playback started and stays this way. How many frames are skipped depends on how far it was off in the first place when playback started. Of course you need to adjust for any general processing delay to get audio perfectly spot on.

    Because of the late frames some frames get dropped every now and then. The reason for the late frames is still unknown (machine is nearly idle). But at least I have an acceptable workaround for now and we can investigate the reason for the late frames in more detail. It's a crude patch but it does its job for now.

    I also tried it with 60Hz. Same happens. Late frames but because of the higher refresh rate they get presented anyway resulting in speed up segments instead of dropped frames and some judder when this happens. IMHO not better because playback is not smooth either way. The frames shouldn't be late in the first place (especially not at the beginning of playback).

    So it looks like a general bug, but this needs to be confirmed by others first.

    PHP:
    Index: EVRCustomPresenter.cpp
    ===================================================================
    --- EVRCustomPresenter.cpp	(revision 23545)
    +++ EVRCustomPresenter.cpp	(working copy)
    @@ -1347,8 +1347,13 @@
           else
           {
             //too late, but present anyway
    -        Log("frame is too late for %I64d ms, last sleep time %d ms.", -*pNextSampleTime/10000, msLastSleepTime );
    -        CHECK_HR(PresentSample(pSample), "PresentSample failed");
    +        //Log("frame is too late for %I64d ms, last sleep time %d ms.", -*pNextSampleTime/10000, msLastSleepTime );
    +        //CHECK_HR(PresentSample(pSample), "PresentSample failed");
    +
    +        //too late, don't present
    +        m_iFramesDropped++;
    +        m_didSkip = true;
    +        Log( "skipping frame, behind %I64d ms, last sleep time %d ms.", -*pNextSampleTime/10000, msLastSleepTime );
           }
         } 
         else

    In case someone wants to try this, I attached the patched dshowhelper.dll I'm using under MP 1.0.10 Beta right now. If you try this let me know if it fixes similar problems on your end or if it breaks anything else. If you don't have similar problems I suggest you don't try the patched DLL.

    You are great Scythe42. Thanks u very much!!! :D
    I just tested the new dll and it works excellent. No audio sync problem anymore!
     

    Scythe42

    Retired Team Member
  • Premium Supporter
  • June 20, 2009
    2,065
    2,703
    51
    Berlin
    Home Country
    Germany Germany
    That was already on my todo list when it comes to playback improvements. Good if someone can provide the patch so I can concentrate on some other things and hopefully reveal something pretty interesting to "advanced" / "hifi" crowd of the MP user base (stay tuned :)).
    I'll do my best. And I am looking forward to your enhancements.

    About the late frames, MS documentation says that no late frames should be rendered.
    I'm currently browsing through the MS documentation to get myself up to speed on EVR presentation....

    That should rule out the bug on codec side. Next step would be try to eliminate the digital audio connection as source of the late samples (audio clock could be drifting away from the video clock, althou the graph clock is sourced currently from the audio clock so it should cause dropped frames or juddering... not sync drifting... but still it is a good idea to rule out that audio renderer / device is not causing the sync drifting).
    I just repeat what I did before to limit this one further down. Once done on Vista I repeat the same on Win 7 again out of curiosity what might be different and to make compare the various configurations. The only thing I know that MS did some enhancements to the automatic quality control in Win 7. According to some paragraph in MSDN it reacts faster than under Vista. Maybe these enhancements are just kicking in here and compensating for this issue. I also look into the EC_SAMPLE_LATENCY as well.
     

    technick

    Portal Pro
    May 6, 2009
    566
    155
    Home Country
    France France
    For your information, I made a test of you patch on many videos. The difference is that I have no more sync problem when I use ffdshow in postprocessing (I don't have without) but I can see on low and long movements that there is some re-adjust jump in the video. Better but not perfect.
     

    tourettes

    Retired Team Member
  • Premium Supporter
  • January 7, 2005
    17,301
    4,800
    For your information, I made a test of you patch on many videos. The difference is that I have no more sync problem when I use ffdshow in postprocessing (I don't have without) but I can see on low and long movements that there is some re-adjust jump in the video. Better but not perfect.

    If ffdshow post processing causes the sync issue you are most likely pushing your CPU too hard. Please check with task manager what CPU usage level you are seeing (50% on a dual core CPU could already be causing stuttering. Unless ffdshow is nowadays fully threaded in all parts))

    If this is the case, we have nothing that could help solving your issue. It is just limitation of your HW.
     

    Scythe42

    Retired Team Member
  • Premium Supporter
  • June 20, 2009
    2,065
    2,703
    51
    Berlin
    Home Country
    Germany Germany
    For your information, I made a test of you patch on many videos. The difference is that I have no more sync problem when I use ffdshow in postprocessing (I don't have without) but I can see on low and long movements that there is some re-adjust jump in the video. Better but not perfect.
    Thanks for testing. What you see are frames that are dropped now because they were late and could not be presented in time.

    Reason for the late frames has not yet been identified (putting hardware limits aside for the moment). I'm working on a more complete workaround that improves 24p playback further and will be available soon.
     

    Scythe42

    Retired Team Member
  • Premium Supporter
  • June 20, 2009
    2,065
    2,703
    51
    Berlin
    Home Country
    Germany Germany
    Update:

    Got it. It's the scheduler. There are multiple things. First we have some checks in the code where under certain circumstances a frame should be skipped and some where it is still rendered when it shouldn't. There are improper workarounds for some issues in the code. Also the sleep timings are preventing a proper playback because they often schedule a frame so that it will be too late for the vsync in the end. When you are relying on the internal vsync function of DirectShow then you must make sure that the frame arrives early enough because the internal clocking is not accurate enough, but not too early. Also there are some other minor issues I need to check in details.

    Right now I have some kind of solution, but it needs further tweaking and testing at various rates and needs some stability testing as well before it makes sense that others test a more complete patch. There's no late frame problem anymore, which reduced the number of dropped frames in comparison to the quick fix. But there will always be a few because Windows is not a real-time OS. This can be further improved by providing an own clocking mechanism that's more accurate than the internal one. But one thing after another.
     

    tourettes

    Retired Team Member
  • Premium Supporter
  • January 7, 2005
    17,301
    4,800
    Got it. It's the scheduler. There are multiple things. First we have some checks in the code where under certain circumstances a frame should be skipped and some where it is still rendered when it shouldn't. There are improper workarounds for some issues in the code. Also the sleep timings are preventing a proper playback because they often schedule a frame so that it will be too late for the vsync in the end. When you are relying on the internal vsync function of DirectShow then you must make sure that the frame arrives early enough because the internal clocking is not accurate enough, but not too early. Also there are some other minor issues I need to check in details.

    Right now I have some kind of solution, but it needs further tweaking and testing at various rates and needs some stability testing as well before it makes sense that others test a more complete patch. There's no late frame problem anymore, which reduced the number of dropped frames in comparison to the quick fix. But there will always be a few because Windows is not a real-time OS. This can be further improved by providing an own clocking mechanism that's more accurate than the internal one. But one thing after another.

    I'm still wondering why it seems to plague only 24fps files. About the clock inaccuracy, we could wake up (say 5 ms early) and then use some busy looping to sleep the rest of the time to reach the target. This is actually used in many game engines to get the exact position. Also with Vista we can use the Multimedia Scheduler to boost the priority (it will quarantee pretty good timing, much better than with XP).

    update: The timer code is at least a little bit buggy on current SVN version (not sure if it has any big enough inaccuraties that would trigger any side effects, but here is something that I have been using while implementing the improved syncing. Maybe it would be worth to test if it reduces the dropped frames you are seeing (I see zero late frames on my own testing with PAL material).

    Code:
    static CCritSec lock; // lock for timer initialization (multiple threads are using the timer during startup)
    
    static LONGLONG GetCurrentTimestamp()
    {
      LONGLONG ticks100ns;
      {
        CAutoLock lock(&lock);
        if( !g_bTimerInitializer ) 
        {
          g_bQPCAvail = QueryPerformanceFrequency( &g_liQPCFreq );
          Log("GetCurrentTimestamp(): Performance timer available: %d", g_bQPCAvail);
          g_bTimerInitializer = true;
        }
      }
    
      if( g_bQPCAvail ) 
      {
        // [url]http://msdn.microsoft.com/en-us/library/ms644904(VS.85).aspx[/url]
        // Use always the same CPU core (no clue if it actually makes any difference...)
        DWORD_PTR oldmask = SetThreadAffinityMask(::GetCurrentThread(), 0);
    
    	  QueryPerformanceCounter ((LARGE_INTEGER*)&ticks100ns);
    	  ticks100ns	= LONGLONG((double(ticks100ns) * 10000000) / double(g_liQPCFreq.QuadPart) + 0.5);
        
        SetThreadAffinityMask(::GetCurrentThread(), oldmask);
      }
      else 
      {
        // to milliseconds to microseconds
        ticks100ns = GetTickCount() * 10000; 
      }
    
      return ticks100ns;
    }
     

    Users who are viewing this thread

    Top Bottom