Are you sure CoreAVC is used to decode your h.264 content? If you don't know how to proceed then start GraphEdit (graphedt.exe, google for it), click on "Render Media File...", select your video file and check if the CoreAVC DirectShow filter is somehow connected on your graph. Report back if you got further info.
For help you should also try to contact CoreCodec after-sales service (and because that doesn't seem to be a MediaPortal issue).