Well, Java could work but I doubt it's multimedia capabilities are great. I haven't tried it though (and I don't have much Java experience so probably won't do it either). HTML5 won't work, as browsers don't convert the stream. It's up to the server to offer one compatible with the browser (and which codecs are compatible differs between browsers, but I found none that are usable until none: H264 is only supported in MP4 containers, which aren't streamable, and the WebM encoder is way too slow on modest hardware).
ffmpeg and VLC actually have hardware decoding support nowadays. I don't think we use it for ffmpeg though, but for VLC it's certainly enabled in MPExtended. That doesn't help much though, as consumer-grade GPUs don't offer integrated hardware encoding, only decoding. Decoding only takes up about 5-10% of the CPU time required for converting the videos, so that won't help much at all. You can use the GPU for encoding through CUDA and the likes, but I haven't found a reliable implementation of that yet and don't intend to write a videocodec myself. Also, a lot of servers (including mine) don't have powerful graphic cards at all, which makes it pretty pointless.
Using DirectShow has some problems too btw. I *hate* the codechell that DirectShow creates, so I really prefer using a single integrated solution. I'm also not sure whether DirectShow supports encoding at all; it's most common use is for decoding. And there is the technical problem that you can't load anything DirectX/DirectShow related when running under a service context.
I'm working with Java every day and it's able to do multimedia...just look at Blu-ray (https://en.wikipedia.org/wiki/BD-J). But with want you have now it's one step back and two forward - it takes time to develop.
DirectShow is capable of encoding and running as an service, but it's not the way to go if you want to avoid the codechell.
Now I have a powerfull graphic card (a leftover from a old gamer pc) in my server, so I think I will try to configure ffmpeg to use GPU...if it's possible.
Thanks for the response and a great product to MediaPortal.
Best regards
Kenneth