Really loooong urls (1 Viewer)

[Brownard]

@offbyone @morpheus_xx (sorry to tag you again).

iPlayer urls are longer than the 4000 character limit in OnlineVideoAspect.ATTR_LONGURL so are getting truncated. As far as I can tell there's nothing particularly strange about the url creation. The offending code is here.

Thanks

Example url:

rtmp://vod-rtmp-uk-live.edgesuite.net####AAEAAAD/////AQAAAAAAAAAMAgAAAENPbmxpbmVWaWRlb3MsIFZlcnNpb249Mi4zLjAuMCwgQ3VsdHVyZT1uZXV0cmFsLCBQdWJsaWNLZXlUb2tlbj1udWxsDAMAAABJU3lzdGVtLCBWZXJzaW9uPTQuMC4wLjAsIEN1bHR1cmU9bmV1dHJhbCwgUHVibGljS2V5VG9rZW49Yjc3YTVjNTYxOTM0ZTA4OQUBAAAAJk9ubGluZVZpZGVvcy5NUFVybFNvdXJjZUZpbHRlci5SdG1wVXJsIQAAABVvcGVuQ29ubmVjdGlvblRpbWVvdXQXb3BlbkNvbm5lY3Rpb25TbGVlcFRpbWUcdG90YWxSZW9wZW5Db25uZWN0aW9uVGltZW91dA1hcmJpdHJhcnlEYXRhFDxBcHA+a19fQmFja2luZ0ZpZWxkFjxUY1VybD5rX19CYWNraW5nRmllbGQYPFBhZ2VVcmw+a19fQmFja2luZ0ZpZWxkFzxTd2ZVcmw+a19fQmFja2luZ0ZpZWxkHTxGbGFzaFZlcnNpb24+a19fQmFja2luZ0ZpZWxkFTxBdXRoPmtfX0JhY2tpbmdGaWVsZBk8UGxheVBhdGg+a19fQmFja2luZ0ZpZWxkGTxQbGF5bGlzdD5rX19CYWNraW5nRmllbGQVPExpdmU+a19fQmFja2luZ0ZpZWxkGjxTdWJzY3JpYmU+a19fQmFja2luZ0ZpZWxkFjxTdGFydD5rX19CYWNraW5nRmllbGQVPFN0b3A+a19fQmFja2luZ0ZpZWxkGzxCdWZmZXJUaW1lPmtfX0JhY2tpbmdGaWVsZBY8VG9rZW4+a19fQmFja2luZ0ZpZWxkFDxKdHY+a19fQmFja2luZ0ZpZWxkGjxTd2ZWZXJpZnk+a19fQmFja2luZ0ZpZWxkFzxTd2ZBZ2U+a19fQmFja2luZ0ZpZWxkDVNpbXBsZVVybCt1cmkTU2ltcGxlVXJsK3ZlcmJvc2l0eRpTaW1wbGVVcmwrbmV0d29ya0ludGVyZmFjZRVTaW1wbGVVcmwrY2FjaGVGb2xkZXIYU2ltcGxlVXJsK21heGltdW1Mb2dTaXplGFNpbXBsZVVybCttYXhpbXVtUGx1Z2lucxRTaW1wbGVVcmwrbGl2ZVN0cmVhbTBTaW1wbGVVcmwrPER1bXBQcm90b2NvbElucHV0RGF0YT5rX19CYWNraW5nRmllbGQxU2ltcGxlVXJsKzxEdW1wUHJvdG9jb2xPdXRwdXREYXRhPmtfX0JhY2tpbmdGaWVsZC5TaW1wbGVVcmwrPER1bXBQYXJzZXJJbnB1dERhdGE+a19fQmFja2luZ0ZpZWxkL1NpbXBsZVVybCs8RHVtcFBhcnNlck91dHB1dERhdGE+a19fQmFja2luZ0ZpZWxkLFNpbXBsZVVybCs8RHVtcE91dHB1dFBpbkRhdGE+a19fQmFja2luZ0ZpZWxkAAAABAEBAQEBAQEAAAEAAAABAQAABAQBAQAAAAAAAAAACAgIOk9ubGluZVZpZGVvcy5NUFVybFNvdXJjZUZpbHRlci5SdG1wQXJiaXRyYXJ5RGF0YUNvbGxlY3Rpb24CAAAAAQEPDw8BDwpTeXN0ZW0uVXJpAwAAACtPbmxpbmVWaWRlb3MuTVBVcmxTb3VyY2VGaWx0ZXIuTG9nVmVyYm9zaXR5AgAAAAgIAQEBAQEBAgAAACBOAAAAAAAAYOoAAAkEAAAACgoKBgUAAABbaHR0cDovL3d3dy5iYmMuY28udWsvZW1wL3JlbGVhc2VzL2lwbGF5ZXIvcmV2aXNpb25zLzYxNzQ2M182MTgxMjVfNC82MTc0NjNfNjE4MTI1XzRfZW1wLnN3ZgoKBgYAAACwBG1wNDpzZWN1cmUvMzIwMGticHMvbW9kYXYvYlVua25vd24tNWJjYjVkZWUtNWY2Ni00YjBhLWI0YTAtNmE3YTRmYWJkMjI5X2IwODI5OHpnXzE0NzcxNjk4NDYyMTMubXA0P2F1dGg9ZGFFYlphbWNoYkNkSWJ4YmNiS2FQYmthT2JSYjhiU2RMY1EtYnlmc2xKLWJXRy1Dbm5GSG9FcUpDcUZweEkmYWlmcD12MDAxJnNsaXN0PXNlY3VyZS84MDBrYnBzL21vZGF2L2JVbmtub3duLTViY2I1ZGVlLTVmNjYtNGIwYS1iNGEwLTZhN2E0ZmFiZDIyOV9iMDgyOTh6Z18xNDc3MTY5ODQ4OTQ2Lm1wNDtzZWN1cmUvMzIwMGticHMvbW9kYXYvYlVua25vd24tNWJjYjVkZWUtNWY2Ni00YjBhLWI0YTAtNmE3YTRmYWJkMjI5X2IwODI5OHpnXzE0NzcxNjk4NDYyMTMubXA0O3NlY3VyZS80ODBrYnBzL21vZGF2L2JVbmtub3duLTViY2I1ZGVlLTVmNjYtNGIwYS1iNGEwLTZhN2E0ZmFiZDIyOV9iMDgyOTh6Z18xNDc3MTY5ODQ4MzY2Lm1wNDtzZWN1cmUvMTUwMGticHMvbW9kYXYvYlVua25vd24tNWJjYjVkZWUtNWY2Ni00YjBhLWI0YTAtNmE3YTRmYWJkMjI5X2IwODI5OHpnXzE0NzcxNjk4NDE5NTYubXA0AAAK//////////8wdQAACgoBAAAAAAkHAAAABfj///8rT25saW5lVmlkZW9zLk1QVXJsU291cmNlRmlsdGVyLkxvZ1ZlcmJvc2l0eQEAAAAHdmFsdWVfXwAIAgAAAAQAAAAGCQAAAAAJCQAAAAAAoAAAAQAAAAAAAAAABQQAAAA6T25saW5lVmlkZW9zLk1QVXJsU291cmNlRmlsdGVyLlJ0bXBBcmJpdHJhcnlEYXRhQ29sbGVjdGlvbgEAAAASQ29sbGVjdGlvbmAxK2l0ZW1zA5oBU3lzdGVtLkNvbGxlY3Rpb25zLkdlbmVyaWMuTGlzdGAxW1tPbmxpbmVWaWRlb3MuTVBVcmxTb3VyY2VGaWx0ZXIuUnRtcEFyYml0cmFyeURhdGEsIE9ubGluZVZpZGVvcywgVmVyc2lvbj0yLjMuMC4wLCBDdWx0dXJlPW5ldXRyYWwsIFB1YmxpY0tleVRva2VuPW51bGxdXQIAAAAJCgAAAAUHAAAAClN5c3RlbS5VcmkBAAAAC0Fic29sdXRlVXJpAQMAAAAGCwAAAIEEcnRtcDovL3ZvZC1ydG1wLXVrLWxpdmUuZWRnZXN1aXRlLm5ldDoxOTM1L29uZGVtYW5kP2F1dGg9ZGFFYlphbWNoYkNkSWJ4YmNiS2FQYmthT2JSYjhiU2RMY1EtYnlmc2xKLWJXRy1Dbm5GSG9FcUpDcUZweEkmYWlmcD12MDAxJnNsaXN0PXNlY3VyZS84MDBrYnBzL21vZGF2L2JVbmtub3duLTViY2I1ZGVlLTVmNjYtNGIwYS1iNGEwLTZhN2E0ZmFiZDIyOV9iMDgyOTh6Z18xNDc3MTY5ODQ4OTQ2Lm1wNDtzZWN1cmUvMzIwMGticHMvbW9kYXYvYlVua25vd24tNWJjYjVkZWUtNWY2Ni00YjBhLWI0YTAtNmE3YTRmYWJkMjI5X2IwODI5OHpnXzE0NzcxNjk4NDYyMTMubXA0O3NlY3VyZS80ODBrYnBzL21vZGF2L2JVbmtub3duLTViY2I1ZGVlLTVmNjYtNGIwYS1iNGEwLTZhN2E0ZmFiZDIyOV9iMDgyOTh6Z18xNDc3MTY5ODQ4MzY2Lm1wNDtzZWN1cmUvMTUwMGticHMvbW9kYXYvYlVua25vd24tNWJjYjVkZWUtNWY2Ni00YjBhLWI0YTAtNmE3YTRmYWJkMjI5X2IwODI5OHpnXzE0NzcxNjk4NDE5NTYubXA0BAoAAACaAVN5c3RlbS5Db2xsZWN0aW9ucy5HZW5lcmljLkxpc3RgMVtbT25saW5lVmlkZW9zLk1QVXJsU291cmNlRmlsdGVyLlJ0bXBBcmJpdHJhcnlEYXRhLCBPbmxpbmVWaWRlb3MsIFZlcnNpb249Mi4zLjAuMCwgQ3VsdHVyZT1uZXV0cmFsLCBQdWJsaWNLZXlUb2tlbj1udWxsXV0DAAAABl9pdGVtcwVfc2l6ZQhfdmVyc2lvbgQAADJPbmxpbmVWaWRlb3MuTVBVcmxTb3VyY2VGaWx0ZXIuUnRtcEFyYml0cmFyeURhdGFbXQIAAAAICAkMAAAAAAAAAAAAAAAHDAAAAAABAAAAAAAAAAQwT25saW5lVmlkZW9zLk1QVXJsU291cmNlRmlsdGVyLlJ0bXBBcmJpdHJhcnlEYXRhAgAAAAs=

 

[offbyone]

I'd like to tag @georgius to clear up why those Urls get so long.

On another note: I thought we wanted to add a new attribute for OnlineVideos to get around the cutting of in MP2 completely? Our urls don't go into a database so no need force them to only 4000 chars?

 

[morpheus_xx]

On another note: I thought we wanted to add a new attribute for OnlineVideos to get around the cutting of in MP2 completely?

I've checked this again: even though we don't fill this aspects by our extractors, it must be registered at the MediaLibrary. This involves creation of a database table.

For SQLite it would be no problem to extend to much more length, because it internally handles every character column as "TEXT" type, which is "unrestricted". When we want to stay compatible with MySQL / SQLCE than we need to check at least their limits.

 

[morpheus_xx]

When decoding this url, the result is suprising (at least to me):

��������������COnlineVideos, Version=2.3.0.0, Culture=neutral, PublicKeyToken=null���ISystem, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089���&OnlineVideos.MPUrlSourceFilter.RtmpUrl!���openConnectionTimeoutopenConnectionSleepTimetotalReopenConnectionTimeout
arbitraryData<App>k__BackingField<TcUrl>k__BackingField<PageUrl>k__BackingField<SwfUrl>k__BackingField<FlashVersion>k__BackingField<Auth>k__BackingField<PlayPath>k__BackingField<Playlist>k__BackingField<Live>k__BackingField<Subscribe>k__BackingField<Start>k__BackingField<Stop>k__BackingField<BufferTime>k__BackingField<Token>k__BackingField<Jtv>k__BackingField<SwfVerify>k__BackingField<SwfAge>k__BackingField
SimpleUrl+uriSimpleUrl+verbositySimpleUrl+networkInterfaceSimpleUrl+cacheFolderSimpleUrl+maximumLogSizeSimpleUrl+maximumPluginsSimpleUrl+liveStream0SimpleUrl+<DumpProtocolInputData>k__BackingField1SimpleUrl+<DumpProtocolOutputData>k__BackingField.SimpleUrl+<DumpParserInputData>k__BackingField/SimpleUrl+<DumpParserOutputData>k__BackingField,SimpleUrl+<DumpOutputPinData>k__BackingField������������������:OnlineVideos.MPUrlSourceFilter.RtmpArbitraryDataCollection���
System.Uri���+OnlineVideos.MPUrlSourceFilter.LogVerbosity������ N������`��   ���


���[http://www.bbc.co.uk/emp/releases/iplayer/revisions/617463_618125_4/617463_618125_4_emp.swf

���mp4:secure/3200kbps/modav/bUnknown-5bcb5dee-5f66-4b0a-b4a0-6a7a4fabd229_b08298zg_1477169846213.mp4?auth=daEbZamchbCdIbxbcbKaPbkaObRb8bSdLcQ-byfslJ-bWG-CnnFHoEqJCqFpxI&aifp=v001&slist=secure/800kbps/modav/bUnknown-5bcb5dee-5f66-4b0a-b4a0-6a7a4fabd229_b08298zg_1477169848946.mp4;secure/3200kbps/modav/bUnknown-5bcb5dee-5f66-4b0a-b4a0-6a7a4fabd229_b08298zg_1477169846213.mp4;secure/480kbps/modav/bUnknown-5bcb5dee-5f66-4b0a-b4a0-6a7a4fabd229_b08298zg_1477169848366.mp4;secure/1500kbps/modav/bUnknown-5bcb5dee-5f66-4b0a-b4a0-6a7a4fabd229_b08298zg_1477169841956.mp4��
0u��

����   ���+OnlineVideos.MPUrlSourceFilter.LogVerbosity���value__�������   ����     ������������������:OnlineVideos.MPUrlSourceFilter.RtmpArbitraryDataCollection���Collection`1+itemsSystem.Collections.Generic.List`1[[OnlineVideos.MPUrlSourceFilter.RtmpArbitraryData, OnlineVideos, Version=2.3.0.0, Culture=neutral, PublicKeyToken=null]]���   
������
System.Uri���AbsoluteUri������rtmp://vod-rtmp-uk-live.edgesuite.net:1935/ondemand?auth=daEbZamchbCdIbxbcbKaPbkaObRb8bSdLcQ-byfslJ-bWG-CnnFHoEqJCqFpxI&aifp=v001&slist=secure/800kbps/modav/bUnknown-5bcb5dee-5f66-4b0a-b4a0-6a7a4fabd229_b08298zg_1477169848946.mp4;secure/3200kbps/modav/bUnknown-5bcb5dee-5f66-4b0a-b4a0-6a7a4fabd229_b08298zg_1477169846213.mp4;secure/480kbps/modav/bUnknown-5bcb5dee-5f66-4b0a-b4a0-6a7a4fabd229_b08298zg_1477169848366.mp4;secure/1500kbps/modav/bUnknown-5bcb5dee-5f66-4b0a-b4a0-6a7a4fabd229_b08298zg_1477169841956.mp4
���System.Collections.Generic.List`1[[OnlineVideos.MPUrlSourceFilter.RtmpArbitraryData, OnlineVideos, Version=2.3.0.0, Culture=neutral, PublicKeyToken=null]]���_items_size_version��2OnlineVideos.MPUrlSourceFilter.RtmpArbitraryData[]���   ����������������������0OnlineVideos.MPUrlSourceFilter.RtmpArbitraryData���

Besides some character encoding issue, it looks like whole objects are serialized and passed as url parameters?

 

[Brownard]

it must be registered at the MediaLibrary. This involves creation of a database table.

Is this just for deserializing, could we just log it and skip any aspects that aren't registered? Ideally shouldn't we have the ability to be able to specify an aspect as local only to avoid the creation of an unnecessary table?
As a workaround can the url just be stored in a field in PlaylistItem and the the ResourceLocator populated from that in the GetResourceLocator override?

 

[morpheus_xx]

Ideally shouldn't we have the ability to be able to specify an aspect as local only to avoid the creation of an unnecessary table?

Yes, it could be useful to have such option. So this would be a "transient" aspect which is not persisted to DB, but still a registered one. @henso what do you think about this, as you have done quite much on aspect handlng for MIA rework?

As a workaround can the url just be stored in a field in PlaylistItem and the the ResourceLocator populated from that in the GetResourceLocator override?

Would also be possible. But I'd like to keep the media item handling as uniform as possible.

So before deciding any changes, I'd like to hear @georgius ' opinion first.

 

[henso]

as you have done quite much on aspect handlng for MIA rework?

I think it's doable but I haven't really much experience with aspects and their dependency on the database. Should I try to add it to the MIA rework?

 

[morpheus_xx]

Let's wait for georgius' answer, maybe there is a better solution

 

[henso]

@morpheus_xx
I ended up adding transient aspects anyway, because I needed them for transferring data from the MDE to RSE (RelationshipExtractor). Maybe it can be used to solve the problem.

 

[georgius]

OnlineVideos code for filter uses binary formatter for serialization of filter URL when passing URL from sites to OV player. It can produce long URLs, but this code should be used only in OnlineVideos. Such URL is not playable in filter, it's only internal thing of OnlineVideos. This URL is deserialized in OV player and correct filter URL is created. Anyway, filter URL can be also long, depending on count of parameters and their values.