AutoCropper plugin (1 Viewer)

[ziphnor]

Ive been running with this a bit now and the status is as follows:

Cyberlink : Works, but some image enhancement features are disabled
Intervideo : Sort of works when DXVA is manually disabled, but a green line is drawn on the right side of screen.
DScaler : No go.
PureVideo : Not tested yet.

My plan is to wrap up the current implementation this weekend. Then i will try to see if i can avoid disabling DXVA in any way. Im currently considering the following approaches:

Screenshot/grab based
Dont particular like this one, but one could grab a few frames as screenshots or maybe via the VMR(anyone know how to do this?) and get bounds based on those. These could then be applied the same way the aspect ratios apply.

Good: Simple, can move analysis code to C#
Bad : Will have to work on the upscaled image.

Seperate filter graph
One simple way way too avoid messing with the DXVA is the following:

1. User requests crop
2. Set up a video only playback graph for the currently playing source, using a software video decoder set to weave deinterlacing for example, instead of a VMR at the end, just place the autocropper filter.
3. Seek to the same position as the currently playing.
4. Get bounds from filter
5. Stop and discard the extra graph
5. Apply bounds from MP side ( in the same way aspects ratios apply ).

Good: Only small changes to the filter
Bad : Cumbersome to implement. There might be some practical issues.

'Injection'
Instead of having my filter in the graph all the time, i will investigate whether or not its feasible to use the following approach:

1. User requests crop
2. Pause graph
3. Insert AutoCrop filter
4. Start graph
5. After X frames, stop graph and get bounding box from autocropper
6. Remove autocropper.
7. Integrate the bounding box as a new aspect ratio
etc.

Good: No changes in filter
Bad: Unlikely that decoderes can handle switching back and forth between hardware and software rendering.

 

[dfbb]

Thanks for the status update.

 

[ziphnor]

Okay, REALLY strange issue.

It seem as long as my filter is used for playback of MPEG-2 files it works perfectly. However, when it is used for timeshifted TV or playback of DVR-MS files (in MP or in graphedit) it fails to work with alot of codecs(it only really works with Cyberlink). If i convert the problematic DVR-MS files to MPEG-2 i have no problems using the same codec (and yes i know that DVR-MS playback uses the TV codec, while MPEG-2 playback uses the movies codec).

I took a screenshot of two playback graphs setup manually in graphedit. One plays the DVR-MS file and the other plays the file after MP has converted it to MPEG-2. The first gives me a black screen, the other works perfectly. Anyone has a clue as to why?

 

[ziphnor]

Today i decided to try another approach, forgetting about the whole filter thing.

I instead created a method that takes a C# bitmap and finds the bounds. I then added an 'Auto' aspect ratio (or 'zoom mode' if you like) which for starters does a screen grab and determines bounds from that. It works very well, and took perhaps 1% of the time required for the DS filter to implement

However, working with a screen grab is a really bad idea because it will also catch any OSD being displayed as well as any added black bars from the current zoom mode. However, i know that MP uses VMR9 renderless, and as far as i can tell that means MP gets a handle to a Direct3D surface with the video rendered in order to render the OSD etc. (as an aside if this approach works out i might crop by updating the 'static' crop settings instead of using a new aspect ratio)

Now, if i can just get a handle on the Direct3D surface, i should be able to use SurfaceLoader to produce a C# bitmap, and everything is good. The biggest issue now is to find out where i can grab a handle on this surface. Seems like it might be PlaneScene.cs ( http://mediaportal.svn.sourceforge....aportal/Core/Player/PlaneScene.cs?view=markup ).

Hopefully i will figure that out tomorrow. But please if you read this and have an understanding of VMR9 Renderless and how it is integrated with MP, please dont hesitate to point me in the right direction!

PS: One could read this as saying i just wasted alot of time on the directshow filter, but thats not entirely correct since i managed to learn quite alot about directshow which is going to be useful when im going to try to add teletext subtitles (the next project in my personal development queue

 

[ziphnor]

Well, only slight progress. I now use the same crop settings as the static cropping system which is very nice. So to summarise : I now have an autocropper that works nicely in fullscreen if you dont bring up the OSD and you are in stretch zoom mode (because then the screenshot provides the info i need

However, i have still failed to get a frame of video 'directly'

It seems the surface im looking for (which a surface of the same size as the original video handed from the VMR9 in renderless mode to MP) is first recieved in the (native) DirectShowHelper in PresentImage which in turn calls Paint which calls back to managed MP using the VMR9Callback interface ending up in PlaneScene.PresentSurface as the uint value of the native pointer.

I thought i could used managed DirectX to get a nice interface to this surface so i tried:

Surface s = new Surface( new IntPtr(pSurface) );

If i had managed that i could have gotten my bitmap using SurfaceLoader.SaveToStream. However if i attempt to use the above managed surface for anything MP crashes (hard) without a decent error message so im guessing that ive somehow abused the Surface constructor? Any help would be appreciated.

The alternative is to go for grabbing the bitmap on the native side in PresentImage/Paint but thats cumbersome compared to the 1-2 lines required with managed DirectX. I really hope someone can give me a pointer here. If i could just get my hand on that ¤#%¤ bitmap it would be a trivial exercise to finish the autocropper.

 

[ziphnor]

I think i managed to make some progress. After much reading of Direct3D documentation i managed to copy the surface upon which the video is drawn using StretchRect to an offscreen surface in ARGB format which is much more friendly and is for example accepted by D3DXSaveSurfaceToFile.

This is a major step forward, i now need to make this new frame grab approach accessible to the C# side autocropper in a performance friendly way (ie we dont want to grab the frame each and every frame!), and then we should have ourselves a nice autocropper mostly implemented in friendly C# ....

And for this approach i have a hard time seeing what could go wrong ( famous last words

 

[dfbb]

Let's hope you don't have too many setbacks in this approach. You already put in a lot of effort and I really appreciate what you're doing.

 

[ziphnor]

Thanks, while it was a bit of a downer to have to give up on the filter approach, this will result in much less code that is also much easier to maintain(and i did learn alot). I do need a slight amount of C++ code though, cant escape the native part completely, but its only:

extern "C" __declspec(dllexport) int __stdcall VideoSurfaceToRGBSurface(IDirect3DSurface9* source, IDirect3DSurface9* dest)
{
IDirect3DDevice9* device = NULL;
HRESULT hr = source->GetDevice(&device);
if(FAILED(hr)){
return hr;
}
else return device->StretchRect(source,NULL,dest,NULL,D3DTEXF_NONE);
}

Anyway, the idea is that when the video surface is passed to managed MP in PlaneScene.PresentSurface as a native pointer, i can create a managed ARGB surface, pass both to the above method, and get my managed ARGB surface filled with the video. This is then easily translated into a Bitmap object which can be used by the cropper for determining bounds. Now that i can do this its only a matter of tying loose ends together, such that the frame is only grabbed when needed (it would have a nasty impact on performance otherwise , something i hope to have time for in the weekend.

Something which came as a positive surprise is that the video surface given by VMR is actually not upscaled yet(the application is responsible for that in VMR9 Renderless), so i am in fact working just on the deinterlaced frame exactly like in my filter!

As for potential problems, i can only imagine that some drivers might not support the necessary color space conversion in StretchRect, but i dont think that this would occur with any DX9 cards (have no idea though). See http://msdn.microsoft.com/library/d.../directx9_c/IDirect3DDevice9__StretchRect.asp for reference.

 

[ziphnor]

Ive got a working version on my development machine now. The cropping works flawlessly and doesnt seem to cause any problems. In the weekend i will modify the old configuration form to match the new implementation. With luck it might even end up in the SVN patch queue in the weekend. Im still debating whether or not to make a quick attempt at dynamic cropping (by just polling say 2 times per second and use some moving average as before for decreasing the bound size) for the first 'release'.

How important is dynamic cropping? Would everybody want to use it or nobody? With manual cropping imagine you would assign the appropriate action to a remote button and simply click it if you wished the bounds to be detected. With dynamic cropping there would be a small delay before the cropper decided to risk reducing the video height, but it would react within 500ms to increases in bound size, such as subtitles appearing for the first time.

Also please remember, that i just promised 'before christmas'

 

[Inker]

Great news, very much looking forward to that.

Personally I would very much like this to work dynamically. Myself I can propably remember and not be bother by having to force a recheck when needed, but other people in the household.... Just don't make it to agressive which could result in changing every other second or so

Anyways, keep us posted!