IPTV/DVB-IP in MP: Support for http, rtp and udp (4 Viewers)

[velis]

Georgius, Stepko,

I would humbly ask assistance.
I have now done HTTP, but am having serious issues with InternetCrackUrl (MPIptvSourceStream::Load) - I've been debugging for this whole day unsuccessfully
If URL's are plain as in UDP version (IP addresses only), everything works as planned.

When I give it http://tv.x-lan.net.ua:8014/inter (the only HTTP that works for me - in VLC that is), it does a memory overrun and everything crashes. I'm pretty sure the problem is in the ::Load() function because I had to add logging to it to make things at least semi-working and to get even a few log lines before the entire ordeal crashed. I also disabled HTTP grabber, but that doesn't matter because the code never even reaches CMPIptvSourceStream:: DoBufferProcessingLoop.
My logging shows that InternetCrackUrl doesn't retrieve url.lpszUrlPath (it is set to NULL) even though the URL most clearly contains it. I'm assuming at this time something overwrote some of my memory already.
I have compared my Load() code with yours and found that you determine the protocol through dllPreAuth array which contains also pointers to some GetRealURL function. I must admit that I don't understand what this code even does.

Naturally I can't proceed with RTP until I resolve this problem

I'm sure I'm missing something obvious here, but what?

 

[dimka]

Hi guys!

You have done good job by now! Hopefully new version of the fiter will get matured and be added to SVN repository!

Velis - I have compiled your code and it seems InternetCrackLib returns url.lpszUrlPat as it should :

05-04-2011 08:04:56.027 [11fc]Open() Scheme: 4 http://tv.x-lan.net.ua:8014/inter
05-04-2011 08:04:56.027 [11fc]ip: tv.x-lan.net.ua
localip:
get_string: /inter
port: 8014
[/inter]
[(null)]
05-04-2011 08:04:56.029 [1a58]Starting grabber thread

or am I missing something ?

I can try to explain the ideas around dllPreAuth and some extra staff I added to Load function some time ago. Usually UDP and RTP streams are sitting in providers networks and do not require extra authenification. Many HTTP streams are also open but there are many providers which would like to have kind of handshake with user/password/whatever before they provide real url of video stream. That's why it nice to have plugin mecanism where everyone can add it's own pre-auth handshake.

That is how real video URL is calculated for Kartina.TV Http provider:
1) filter predefines 3 major protocols : http, rtp, udp. So urls in format [http|udp|rtp]://xxx doe not require extra job
2) Kartina's playlist is converted into having urls like kartina://someurl/user=xx&pass=yyy
3) during filter load it looks into MPIPTVplugins directory and loads all DLLs from there. Each DLL represents extra "protocol" and should implement two functions : GetProtocol - returns protocol string "kartina" and base protocol(http for kartina for example), second function is GetRealURL - gets url from playlist and returns real URL.
4) so far - at this moment we have 3 major protocols registred plus one fake - kartina.
5) Load function extracts url.lpszScheme and compares with protocols in DllAuth array. as url contains "kartina://xxx" item it calls GetRealURL from kartina's dll. in that function I do some extra steps as getting coockies, do authentfication and so on. After some calculation Kartina's servers rerurns the url of REAL video stream.
Agree - it so specific per provider so nobody could get it into SVN source. From another side there is backdoor to extend such specific functionality.
6) Load function gets url of real video stream and base protocol - see step 3. Function bypasses it to whatever is parsing it, open stream and so on..

That is the idea around plugins. I really hopeyou will keep it as it can be usefull later(and nor for Kartina users )

Anyway - good job is done!

br,
dimka

PS. I do remember I had promised to open SVN repository for filter. So if you still need it I can spend some time to configure it

 

[velis]

Thanks for the explanation Dimka. Unfortunately, for me InternetCrackURL doesn't find that UrlPat
So does that HTTP work for you or does it fail somewhere later? I tried to shorten the algorithm so I'm not sure it even works, especially the chunked processor.
I don't see any problem keeping plugins, but am currently not using them because I didn't understand them. Perhaps you can fix the code after all 3 protocols are done?
I'm pretty much only interested in doing the protocols low CPU / no stutter but after that - be my guest
I can only hope to resolve this problem I have soon before I lose my interest (I only need UDP )
BTW: I have RTP almost finished, maybe in the evening I can post code with it included as well.

 

[dimka]

Hi Velis,

I did not look for more details futher down in log. There were millons of log enties but it seems it did not get a stream. I can test it tonight if you want.

I can add plugin stuff as soon protocols are finished - no problem with that !

br.
dimka

 

[Stepko]

AW: Re: IPTV/DVB-IP in MP: Support for http, rtp and udp

BTW: I have RTP almost finished, maybe in the evening I can post code with it included as well.

Oh, ok. Just wanted to offer to add the rtp part during the next weekend (have no time during the week). So if you don't want to add the rtp stuff, I can do that. I think it would take me just one or two hours as it is quite similar to udp.
BTW: You don't necessarily need to add the whole rtp-buffer stuff with ProcessPacket... because receiving packets in wrong order doesn't happens often.

What you must do is:

• Check the first 2 bits to be 1 and 0
• Check if packet size > 12
• Check for padding
• Check for header extension
• Check that ssrc (source identifier) is the same in all packets
• Extract and check sequence number and set discontinous flag in IMediaSample accordingly
• Extract timestamp and set timestamp in IMediaSample
• Skip header (12 bytes)
• Skip headerExtension (if used)
• Cut Data at the end if padding is used

As already said, I can do this in a few days. It's not a big deal for me and you don't need to dig into that part.

Stepko

 

[velis]

Yes, I know, but still thanks for pointing out the things to look out for.
As for packet reordering, I was just planning for a std::map. I think it will do the job just fine. Still you are correct about order: if my provider can do UDP with NO reordering, I don't suppose your provider should mess things up

Besides I kind of feel obliged because I developed on the old version of the filter. I can't really expect Georgius / you / dimka to do your work all over again... Especially because Georgius' version is actually working fine, just infinitesmally less well than mine...

 

[georgius]

Until velis include RTP support you can use my version of filter. Stepko's code I only transfered, so if it was working before we started changes, then it should work now. I tested it with simulation by VLC and it works.

I'm investigating why my filter is not so efficient like velis ... it seems that the problem is somewhere after my filter (maybe MPEG-2 demultiplexer or TS writter). On my "production" HTPC is my version (just UDP) working well and without any problems.

 

[velis]

Just a minor update. Unfortunately I can't post any code at this moment.

I have found my bug(s). Turns out I didn't have a memory overrun, but rather a ton of bugs that I introduced when I was merging code together and introducing common classes.
Source is now even cleaner and understandable (I hope)

Despite all the rearrangements, UDP remains super stable - so I'm very happy with this part.

I'm still having issues with HTTP. I must figure out a sensible way of dynamically expanding buffers. Ran into this problem because the server returns chunk sizes around 20KB, but my internal buffers were at 16KB.
Or perhaps I should just implement "persistent chunk" meaning that I would just blindly receive data until chunk size was satisfied.
Anyway - when I figure this out, HTTP will be done.
And before you ask: I don't like multi-MB memory usage for no purpose. Right now my filters take 128KB for buffers and 16KB for thread stack. Thread stack can go down to 4KB, but I don't want to push it. I'll just have to fix this "inefficiency" with a bit more code instead of more RAM, I hope.

Ran out of time for RTP completion today

Otherwise progress is good and we should see some results by the end of the week.

 

[velis]

Hm, work and fatigue have caused me to not be able to do the necessary debugging / programming to finish the filters by tonight (today I will also not have enough time...)
Tomorrow I go to Czech republic for a competition and will be out for the weekend. That means no programming

The problem is that I changed class interface quite a lot so I'd hate Stepko to develop on something old writing a gazillion of functions when in fact he only needs to write (better said: debug) one single function in order for RTP to work...

So, here's the question: Do I post unfinished code tonight (HTTP and RTP will not work!) or will you guys wait a few days?

It's not like I wouldn't appreciate if somebody else did my debugging for me

Edit: Oh - one more thing - is there someone willing to do a short "snow" clip? You know - the no signal "snow" as seen on old analog televisions when you unplug the antenna As short as possible, but still a few frames so that at least it animates a bit...

 

[Stepko]

AW: Re: IPTV/DVB-IP in MP: Support for http, rtp and udp

Do I post unfinished code tonight (HTTP and RTP will not work!) or will you guys wait a few days?

I think (and hope ) that I'll have some time on saturday. If you would give you your code, I could try my best
But I really don't want source code thats not working uploaded to the public forum.
I'll send you a PM with my email in a few seconds. Please send me your source via email by staturday.

Thx
Stepko