Germany
Defect Description for Mantis #0002060 and Patch for Review
The patch adjust add the following logic to include proper frame dropping:
For user tests core.dll and dshowhelper.dll needs to be exchange on a MP 1.1.0 Beta 1 installation once the patch gets approved. Fine tuning will be done in a separate patch based on test results.
Known Issues
These issues all exist with the original DLL as well. You might not have seen them that prominently because of the broken frame dropping that didn't drop frames when it should and dropped some when it shouldn't (e.g. DVD menus or FF/REW).
Fast Forward/Rewind
Depending on what MPEG2 codec is used the experience in fast forward or rewind can differ. For example the MS MPEG2 Codec seems to cache two seconds of playback and then providing only every second frame to the mixer in 2x mode. Once the cache is empty it presents 2 frames in a row at normal speed (it forgets to skip the last…) This results in visible judder as the playback speed is not constant. It needs to tested if this ca be compensated by finding a pattern to identify this issue.
With MPC files on rewind you see the first two frames played again in each internal seeking interval the MPC codec or MP does. This results in visible judder, probably related to encoding. It needs to be tested if this can be compensated by finding a pattern to identify these frames and drop them. MPC doesn't seem to support 2x rewind properly for all files. On some it provides the same frame over and over again until higher speeds are selected. This is not the EVR presenter dropping frames. In rewind/fast forward mode it will render anything that's inside the queue.
With using MPC there is a another issue in a 1:1 fps/frame rate application. 2x is not properly supported. When you initiate the 2x playback it sends every second frame to the mixer, so far so good. But it does with the original time stamp. Therefore you will see only every 2nd frame at normal speed. If you have a higher refresh rate than a files fps, it works better. Need to check the ff/rew code of MP a bit more first.
The Nvidia Decoder doesn't seem to support what MP is doing. Fast Forward/Rewind has no effect and may even crash MP on my machine
In general MP needs new ff/rew code completely independent of what a codec offers so that it works basically the same for every file at any rate to eliminate all these small oddities...
MS DVD Navigator
Sometimes the MS DVD Navigator doesn't supply the first frame of a DVD. If this is a still frame for selecting a DVD's menu language the user will still see the MP menu from where they started the movie until they make a new menu selection.
When changing to another menu item while in a menu where video is played in the background, the last frame of this video is presented by the DVD Navigator again with the new overlay instead of providing the new overlay with the next frame. This results in judder because the last frame is displayed two times. DVD Navigator design issue.
1:1 Frame Rate and the internal VIDEO_TS Player
Don't use 25Hz for PAL but use 50Hz. Even though the de-interlaced frames would run at 25fps the mixer still gets the sample times in 20 intervals. This will cause huge frame dropping as the sample time stamps are wrong. Especially when you are in DVD Menus with animation using the MS DVD Navigator you will see an enormous amount of dropped frames. Same applies to NTSC. So for DVDs use a refresh rate that matches the original fps of the DVD and not the progressive one. This can be related to the codecs but also an issue of MP's VIDEO_TS player. Also ff/rew will break here similar to MKV at least 2x and maybe 4x speed.
Playback doesn't start
Sometimes the EVR presenter doesn't get a single frame when playback starts. Users will still see the menu. Audio might already run. This is not EVR presenter related. Probably some error in setting up the DirectShow Graph is not properly handled. More investigation is needed here.
ReClock
Fast Forward and Rewind doesn't work on my machine when ReClock is used as an audio render. Needs to be investigated as ReClock should not be the limiting factor here.
Mantis #0002387
Should be more visual to the user now.
Important Tests
Audio/Video Sync
Test with any combinations of files at any refresh rate with various codecs. The results will be used to adjust the EVR presenter to compensate for reproducible issues if a unique pattern can be identified.
DVD Navigation
Test that all menus work properly with various DVD Navigators and various DVDs. Beside the MS one none worked on my system. Nvidia Decoder crashed MP for example.
Fast Forward/Rewind
Test with any combination of files at any refresh rate with various codecs. The results will be used to adjust the EVR presenter to compensate for reproducible issues if a unique pattern can be identified.
TODO: Fix all the bugs that show up in testing or implement workarounds for issues outside the presenter where appropriate.
PS: As you can see the issues get really complicated because of all the different codecs. But we'll get there. One issue or enhancement at a time. I'll focus on enhancing the player as in my opinion is the most important component.
The current SVN code drops a frame under the following circumstances:
a) next sample is already due (meaning, we are at least one full one frame behind)
b) sample is more then 25ms behind.
If none of the two conditions is fulfilled, the sample will be presented.
If either a) or b) is fulfilled then frames will only be dropped when there is at least another sample scheduled.
If there are no other samples currently scheduled the frame is presented anyway, even it is late. This is the main bug. It's basically the same as only relying on nextDue for dropping fames. Therefore the > 25m delay as a second condition when frames should be dropped is rendered useless. And when frames are coming late from the mixer nextDue is never true. Late samples from the mixer are the most common scenario that introduced the audio/vide de-sync, especially in 1:1 frame rate usage.
This creates a problem when the refresh rate and the file's fps are close to each other. There is nearly no chance to catch up because all late samples will be presented. Each late provided sample that is presented will de-sync audio and video for one frame. This is especially a problem when file fps equals the refresh rate and there is never a chance to catch up on additional vsyncs. Users experience an audio drift (especially when using SPDIF) where the picture falls more and more behind the audio track.
When you running a higher refresh rate than a file's fps, the EVR presenter will have a chance to catch up again because of the higher number of vsyncs. In this case the video playback will be accelerated until the file is back in sync. This is as bad as dropping frames but will only work up to some degree when the refresh rate is significantly larger than the files fps (e.g. 23.976fps played at 60Hz). On 60fps files at 60Hz you'll have the same problem.
It is necessary to drop frames if they cannot presented in time to the renderer. Speeding up playback or dropping frames will always result in juddering. The only correct course of action is to drop frames when they are too late as speeding up is only possible when fps and refresh rate differ significantly.
The proposed patch fixes that problem. A frame will be dropped when it's behind is scheduled time. Any kind of larger threshold value like up to behind 15ms will not work as the frame cannot be displayed anymore at the expected vsync time and then would be displayed at the next causing audio/video already out of sync for full frame. With a 25fps source that would be 40ms and this is really noticeable. There might be a small grace period though depending how long the vertical blank lasts. So fine tuning here is possible.
IMHO it's better to introduce a larger chain of pre-rendered frames to compensate for late frames and avoid frame dropping if just a few single frames are only a few ms late. Proper cause of action needs to be discussed before taking this one further.
What ever the course of action will be, it cannot be part of this patch as it addresses the improper frame dropping causes audio/video de-sync only. But that's not the full monty for smooth playback.
A sample can be currently late under the following scenarios:
a) They are simply late because of difference in fps vs. refresh rate (e.g. 23,976fps played at 24Hz).
b) Samples form the mixer are received too late (scheduling sample from the past in the logs).
For the first one when playing a 23.976fps file at 24Hz a frame needs to be dropped every 41.66 second in theory. With a more precise integer arithmetic I come close too this theoretic value. Before it there was a constant drift of 0,2ms between each frame, resulting a dropped frame every 34 seconds in a 23.976fps at 24Hz scenario.
The second scenario largely depends on the file itself and the used filters when it is not a general CPU/GPU issue. This seems to depend on how a file is exactly encoded. This is mainly related to x264 files. Users might not see high CPU usage because CPU power might only be limited for a single frame. Windows cannot assign CPU power that fast to a thread. This can also happen on a Core i7 machine. As pointed out it largely depends on the encoding of a file.
Windows 7 performs better than Vista on the same testing machine because of various enhancements in regards to media playback and a more reliable timeSetEvent.
The frame dropping is done in CheckForScheduledSample() to stays close to the original code. Doing this in ProcessInputNotify() seems to be a better place though.
Because of the proper dropping of late frames rewind and fast forward were broken. Especially on higher speeds frames come with timestamps way in the past or are even presented in pause mode. This is a result of MP does fast forward and rewind at the moment.
In addition with the MS DVD Navigator the time stamp of a frame in still menus never get updated. Changing user selections are treated as an overlay that don't provide a new timestamp. For DVD menus with animations proper time stamps are provided. If it's a still menu and an audio loop plays in the back a new sample time is provided once the audio loop starts. Therefore for DVD menus every frame needs to rendered and the samples' time stamps need to be ignored.
Because it turned out to be nearly impossible to compensate for this two issues with only having a delta between multiple sample times, the player components also needed an update. They now inform the EVR Presenter when they are in ff/rew and DVD Menu Mode, so that the EVR presenter will not drop any frames.
a) next sample is already due (meaning, we are at least one full one frame behind)
b) sample is more then 25ms behind.
If none of the two conditions is fulfilled, the sample will be presented.
If either a) or b) is fulfilled then frames will only be dropped when there is at least another sample scheduled.
If there are no other samples currently scheduled the frame is presented anyway, even it is late. This is the main bug. It's basically the same as only relying on nextDue for dropping fames. Therefore the > 25m delay as a second condition when frames should be dropped is rendered useless. And when frames are coming late from the mixer nextDue is never true. Late samples from the mixer are the most common scenario that introduced the audio/vide de-sync, especially in 1:1 frame rate usage.
This creates a problem when the refresh rate and the file's fps are close to each other. There is nearly no chance to catch up because all late samples will be presented. Each late provided sample that is presented will de-sync audio and video for one frame. This is especially a problem when file fps equals the refresh rate and there is never a chance to catch up on additional vsyncs. Users experience an audio drift (especially when using SPDIF) where the picture falls more and more behind the audio track.
When you running a higher refresh rate than a file's fps, the EVR presenter will have a chance to catch up again because of the higher number of vsyncs. In this case the video playback will be accelerated until the file is back in sync. This is as bad as dropping frames but will only work up to some degree when the refresh rate is significantly larger than the files fps (e.g. 23.976fps played at 60Hz). On 60fps files at 60Hz you'll have the same problem.
It is necessary to drop frames if they cannot presented in time to the renderer. Speeding up playback or dropping frames will always result in juddering. The only correct course of action is to drop frames when they are too late as speeding up is only possible when fps and refresh rate differ significantly.
The proposed patch fixes that problem. A frame will be dropped when it's behind is scheduled time. Any kind of larger threshold value like up to behind 15ms will not work as the frame cannot be displayed anymore at the expected vsync time and then would be displayed at the next causing audio/video already out of sync for full frame. With a 25fps source that would be 40ms and this is really noticeable. There might be a small grace period though depending how long the vertical blank lasts. So fine tuning here is possible.
IMHO it's better to introduce a larger chain of pre-rendered frames to compensate for late frames and avoid frame dropping if just a few single frames are only a few ms late. Proper cause of action needs to be discussed before taking this one further.
What ever the course of action will be, it cannot be part of this patch as it addresses the improper frame dropping causes audio/video de-sync only. But that's not the full monty for smooth playback.
A sample can be currently late under the following scenarios:
a) They are simply late because of difference in fps vs. refresh rate (e.g. 23,976fps played at 24Hz).
b) Samples form the mixer are received too late (scheduling sample from the past in the logs).
For the first one when playing a 23.976fps file at 24Hz a frame needs to be dropped every 41.66 second in theory. With a more precise integer arithmetic I come close too this theoretic value. Before it there was a constant drift of 0,2ms between each frame, resulting a dropped frame every 34 seconds in a 23.976fps at 24Hz scenario.
The second scenario largely depends on the file itself and the used filters when it is not a general CPU/GPU issue. This seems to depend on how a file is exactly encoded. This is mainly related to x264 files. Users might not see high CPU usage because CPU power might only be limited for a single frame. Windows cannot assign CPU power that fast to a thread. This can also happen on a Core i7 machine. As pointed out it largely depends on the encoding of a file.
Windows 7 performs better than Vista on the same testing machine because of various enhancements in regards to media playback and a more reliable timeSetEvent.
The frame dropping is done in CheckForScheduledSample() to stays close to the original code. Doing this in ProcessInputNotify() seems to be a better place though.
Because of the proper dropping of late frames rewind and fast forward were broken. Especially on higher speeds frames come with timestamps way in the past or are even presented in pause mode. This is a result of MP does fast forward and rewind at the moment.
In addition with the MS DVD Navigator the time stamp of a frame in still menus never get updated. Changing user selections are treated as an overlay that don't provide a new timestamp. For DVD menus with animations proper time stamps are provided. If it's a still menu and an audio loop plays in the back a new sample time is provided once the audio loop starts. Therefore for DVD menus every frame needs to rendered and the samples' time stamps need to be ignored.
Because it turned out to be nearly impossible to compensate for this two issues with only having a delta between multiple sample times, the player components also needed an update. They now inform the EVR Presenter when they are in ff/rew and DVD Menu Mode, so that the EVR presenter will not drop any frames.
The patch adjust add the following logic to include proper frame dropping:
- More precise integer arithmetic for GetCurrentTimeStamp() and code adjustments to 100ns units like DirectShow (supplied by arion_p)
- Frames are dropped when *pNextSampleTime < 0 in CheckForScheduledSample()
- Player Classes were updated to inform the EVR presenter when being in DVD Menus or when fast forward/rewind is in effect to disable frame dropping.
- Magic Numbers were removed as far as possible and are now connected to m_rtTimePerFrame
- timeSetEvent will schedule a thread up to the presentation time or the frame duration whatever is smaller minus 10% of frame duration to compensate for not 100% reliable timeSetEvent
- Frames will also be rendered when in Pause mode for ff/rew and DVD menus.
- Minor code corrections related to this issues
For user tests core.dll and dshowhelper.dll needs to be exchange on a MP 1.1.0 Beta 1 installation once the patch gets approved. Fine tuning will be done in a separate patch based on test results.
Known Issues
These issues all exist with the original DLL as well. You might not have seen them that prominently because of the broken frame dropping that didn't drop frames when it should and dropped some when it shouldn't (e.g. DVD menus or FF/REW).
Fast Forward/Rewind
Depending on what MPEG2 codec is used the experience in fast forward or rewind can differ. For example the MS MPEG2 Codec seems to cache two seconds of playback and then providing only every second frame to the mixer in 2x mode. Once the cache is empty it presents 2 frames in a row at normal speed (it forgets to skip the last…) This results in visible judder as the playback speed is not constant. It needs to tested if this ca be compensated by finding a pattern to identify this issue.
With MPC files on rewind you see the first two frames played again in each internal seeking interval the MPC codec or MP does. This results in visible judder, probably related to encoding. It needs to be tested if this can be compensated by finding a pattern to identify these frames and drop them. MPC doesn't seem to support 2x rewind properly for all files. On some it provides the same frame over and over again until higher speeds are selected. This is not the EVR presenter dropping frames. In rewind/fast forward mode it will render anything that's inside the queue.
With using MPC there is a another issue in a 1:1 fps/frame rate application. 2x is not properly supported. When you initiate the 2x playback it sends every second frame to the mixer, so far so good. But it does with the original time stamp. Therefore you will see only every 2nd frame at normal speed. If you have a higher refresh rate than a files fps, it works better. Need to check the ff/rew code of MP a bit more first.
The Nvidia Decoder doesn't seem to support what MP is doing. Fast Forward/Rewind has no effect and may even crash MP on my machine
In general MP needs new ff/rew code completely independent of what a codec offers so that it works basically the same for every file at any rate to eliminate all these small oddities...
MS DVD Navigator
Sometimes the MS DVD Navigator doesn't supply the first frame of a DVD. If this is a still frame for selecting a DVD's menu language the user will still see the MP menu from where they started the movie until they make a new menu selection.
When changing to another menu item while in a menu where video is played in the background, the last frame of this video is presented by the DVD Navigator again with the new overlay instead of providing the new overlay with the next frame. This results in judder because the last frame is displayed two times. DVD Navigator design issue.
1:1 Frame Rate and the internal VIDEO_TS Player
Don't use 25Hz for PAL but use 50Hz. Even though the de-interlaced frames would run at 25fps the mixer still gets the sample times in 20 intervals. This will cause huge frame dropping as the sample time stamps are wrong. Especially when you are in DVD Menus with animation using the MS DVD Navigator you will see an enormous amount of dropped frames. Same applies to NTSC. So for DVDs use a refresh rate that matches the original fps of the DVD and not the progressive one. This can be related to the codecs but also an issue of MP's VIDEO_TS player. Also ff/rew will break here similar to MKV at least 2x and maybe 4x speed.
Playback doesn't start
Sometimes the EVR presenter doesn't get a single frame when playback starts. Users will still see the menu. Audio might already run. This is not EVR presenter related. Probably some error in setting up the DirectShow Graph is not properly handled. More investigation is needed here.
ReClock
Fast Forward and Rewind doesn't work on my machine when ReClock is used as an audio render. Needs to be investigated as ReClock should not be the limiting factor here.
Mantis #0002387
Should be more visual to the user now.
Important Tests
Audio/Video Sync
Test with any combinations of files at any refresh rate with various codecs. The results will be used to adjust the EVR presenter to compensate for reproducible issues if a unique pattern can be identified.
DVD Navigation
Test that all menus work properly with various DVD Navigators and various DVDs. Beside the MS one none worked on my system. Nvidia Decoder crashed MP for example.
Fast Forward/Rewind
Test with any combination of files at any refresh rate with various codecs. The results will be used to adjust the EVR presenter to compensate for reproducible issues if a unique pattern can be identified.
TODO: Fix all the bugs that show up in testing or implement workarounds for issues outside the presenter where appropriate.
PS: As you can see the issues get really complicated because of all the different codecs. But we'll get there. One issue or enhancement at a time. I'll focus on enhancing the player as in my opinion is the most important component.