Tempest EPG Generator (4 Viewers)

PN23 · February 10, 2022

kvanc said:
Hi PN23;

I am developing this app for 2,5 years and actively using for myself for 2 years both on windows(php development server) and debian nginx with my raspberry pis. I have never ever experienced such error as you mentioned. It seems like a wamp server's error but hard to guess without seeing. If you have error logs, Could you please share? Also, how you are calling Tempest ( as url) from wamp and are you able to run any other script? I have used wamp very early stage of development and changed it when I noticed some strange behaviour about its very late working garbage collectors. I can able to save 5000 channels with one click on my rpi0w which is the lowest possible hardware you can get I searched in google and found lots "wamp/url not found on server" of thread which seems all solved with different thing but this one is very strange about keeping an index file in document root

Wamp basic configuration: "The requested URL /my-site/ was not found on this server."

I've just installed Wamp on a new computer with Windows 10 and I don't succeed to access any project in the "www" folder. For example, I've created the folder "C:\wamp64\www\my-site&...

stackoverflow.com

Hi Kvanc,

Apologies, I haven't had much time to troubleshoot as am working most of the time.

It appeared to be an IPv6 binding that was the issue. Have disabled that in Apache and now clicking on save works as expected no matter how many channels are selected. Have never experienced that particular issue myself previously!

Am currently running a 2 day grab against 536 channels as a test.

kvanc · February 10, 2022

PN23 said:
Hi Kvanc,

Apologies, I haven't had much time to troubleshoot as am working most of the time.

It appeared to be an IPv6 binding that was the issue. Have disabled that in Apache and now clicking on save works as expected no matter how many channels are selected. Have never experienced that particular issue myself previously!

Am currently running a 2 day grab against 536 channels as a test.

Perfect to hear it is up and running. Hope you will enjoy your the fastest and the lowest memory consuming grabbing experience

PN23 · February 10, 2022

Now I need to work out how to configure to grab epg from my two other sources!

kvanc · February 10, 2022

PN23 said:
Now I need to work out how to configure to grab epg from my two other sources!

You mean digiguide and schedulesdirect, I think

today have a look digiguide but impossible to see their structure without being member. For schedulesdirect, there is their github page for required request which all and token/session transfer may be done via Tempest.

API 20141201

Code related to download, slice-and-dice and generation of JSON into database. - SchedulesDirect/JSON-Service

github.com

And I have TempestWIKI.pdf which is only 63 pages

Tempest-EPG-Generator/TempestWIKI.pdf at main · K-vanc/Tempest-EPG-Generator

The fastest XMLTV formatted EPG Generator with GUI and the lowest memory usage - K-vanc/Tempest-EPG-Generator

github.com

Just have a try. If you dont manage and dont mind about giving your sd account credentials to me in private, i can try to make a scrapper for you after Monday. Also i am planning to apply official supported add on list of schedulesdirect after earning more reputation with Tempest.

kvanc · February 10, 2022

PN23 said:
Hi Kvanc,

Apologies, I haven't had much time to troubleshoot as am working most of the time.

It appeared to be an IPv6 binding that was the issue. Have disabled that in Apache and now clicking on save works as expected no matter how many channels are selected. Have never experienced that particular issue myself previously!

Am currently running a 2 day grab against 536 channels as a test.

Btw small tip about incrementing mode grabbing. If you want to regularly update same epg file with multiple days, from "Tempest configurator" open your config file and enable fusion mode with fusion level 0 and save back without any other change. This is preserve existing data belongs to grabbing date and more , if there is any failure on a channel on your next run

PN23 · February 11, 2022

kvanc said:
Btw small tip about incrementing mode grabbing. If you want to regularly update same epg file with multiple days, from "Tempest configurator" open your config file and enable fusion mode with fusion level 0 and save back without any other change. This is preserve existing data belongs to grabbing date and more , if there is any failure on a channel on your next run

Hi kvanc,

Thanks again for the help. I was already using fusion level 0 after reading about the other day - that will come in handy.

My current method of creating/updating my tv guide is a little long-winded but the only way really that I've been able to do it.

I use digiguide listings for the UK as these have so far proved to be the best for me, and Schedules Direct for listings from further afield.

Putting it all together involves a shell script which runs every few days, extracts the digiguide listings, scrapes from all 4 of my Schedules Direct lineups, extracts and changes listings for a few channels that I cannot find a +1 listing for, changes the encoding of some of the XML files to UTF-8 where required, change timezones where appropriate and account for UTC/BST/DST, concatenate the resulting XML files in to one tvguide XML file, then move it to the correct location. I grab 7 days at a time (would prefer 14 to be honest like we used to have with WMC) and the entire process can take around 45 minutes on pretty fast machine.

Once MP2 sees this, it can take around another 50 minutes to an hour for it to sort itself out and obviously during that time the MP2 configuration program is unusable.

If I can get all of the UK listings I need from the Sky scraper in Fusion, I can drop digiguide altogether as getting the listings out of the damn thing is a pain (The only way I have found is to use their desktop software which regularly downloads the listings but these are obfuscated and only available to this desktop app, and who wants a separate desktop app to look at listings? I then use the command line version of XMLTV GUI from Alan Birtles which is able to extract the digiguide listings from the app and produce an XML).

I will have a look at the Schedules Direct API documentation as soon as I am able and see if I can get that working with Fusion. With any luck, I will just be able to use Fusion to do the heavy lifting for me and produce my listings.

I do have a question regarding the output format of the XML.

The generated XML file format is channel, programme, programme, programme etc as opposed to channel*, programme* so I cannot validate the files against the standard xmltv.dtd.

Is there an option I have missed that can output the files in this way?

kvanc · February 11, 2022

PN23 said:
Hi kvanc,

Thanks again for the help. I was already using fusion level 0 after reading about the other day - that will come in handy.

My current method of creating/updating my tv guide is a little long-winded but the only way really that I've been able to do it.

I use digiguide listings for the UK as these have so far proved to be the best for me, and Schedules Direct for listings from further afield.

Putting it all together involves a shell script which runs every few days, extracts the digiguide listings, scrapes from all 4 of my Schedules Direct lineups, extracts and changes listings for a few channels that I cannot find a +1 listing for, changes the encoding of some of the XML files to UTF-8 where required, change timezones where appropriate and account for UTC/BST/DST, concatenate the resulting XML files in to one tvguide XML file, then move it to the correct location. I grab 7 days at a time (would prefer 14 to be honest like we used to have with WMC) and the entire process can take around 45 minutes on pretty fast machine.

Once MP2 sees this, it can take around another 50 minutes to an hour for it to sort itself out and obviously during that time the MP2 configuration program is unusable.

If I can get all of the UK listings I need from the Sky scraper in Fusion, I can drop digiguide altogether as getting the listings out of the damn thing is a pain (The only way I have found is to use their desktop software which regularly downloads the listings but these are obfuscated and only available to this desktop app, and who wants a separate desktop app to look at listings? I then use the command line version of XMLTV GUI from Alan Birtles which is able to extract the digiguide listings from the app and produce an XML).

I will have a look at the Schedules Direct API documentation as soon as I am able and see if I can get that working with Fusion. With any luck, I will just be able to use Fusion to do the heavy lifting for me and produce my listings.

I do have a question regarding the output format of the XML.

The generated XML file format is channel, programme, programme, programme etc as opposed to channel*, programme* so I cannot validate the files against the standard xmltv.dtd.

Is there an option I have missed that can output the files in this way?

Wow, it seems you have really heavy job with epg data, PN23

Let's go case by case. Since Tempest is a very newly published and unknown tool, I am currently more focused to its total coverage as a proof that it is fully programmable and may be used in anywhere so I am trying to add minimum +100 channels scrappers from countries that currently not exist in my repo and unfortunately I am Also very busy with my real-life work so trying to do more about it in limited times. Good to hear sky + fusion mode is promising for you. Fusion level will keep previous days as well when increased upto 15 days for catch-up services. If you need some other channels which is not exist in sky but available in any other public sources, I may try to built scrapper for it as well so you can add its channels to your sky config and grab all together. No worry about UTC/DST options because there is a time-converter module which may be set in your configuration file to convert all different time offsets to your requested offset. Example if you set it +03:00 from your config, all grabbed show times of all channels in this config, will be converted
+03:00 (not only offset change) or you can set as "Europe/London" so all will be converted with DST support.

For schedulesdirect, I was just reading their readme file also. Since I am out of their region, I have never personelly tried or used but based on what I read, their json api which contains 20days of data, may be easily done with Tempest. So in my first oppurtunity, I will try make an unlocked scrapper for you to add your credentials and test. Due to my blind try, it may fail on first without seeing exactly what is going there but for sure it will work on second or third.

For the xmltv structure, I know it is not in same order like all others

but that structure is completely valid as well since there is requirement/restriction about it in xmltv_dtd and will be Accepted by all pvrs/players since channel tags and show datas are being parsed individually. I am Also using Tempest made xmtlv files for long time on 5-6 different players, kodi etc without any issue.

xmltv/xmltv.dtd at master · XMLTV/xmltv

Utilities to obtain, generate, and post-process TV listings data in XMLTV format - XMLTV/xmltv

github.com

That small structure change of Tempest, grants huge advantage about cyclone modifications which is alternative channel assignment for a failed channel with same previous xmltv_id/display name but new channels logo(if exists) and reduce overall memory usage of some other operations. So you cannot convert it to tradinitional order with a setting but it is fully legit and safe to use. Also you dont need to worry about xmltv verification for files generated by Tempest because it will fix even malfunctioned bytes and convert to xmltv safe characters with fill UTF-8 output

PN23 · February 11, 2022

kvanc said:
Wow, it seems you have really heavy job with epg data, PN23

Let's go case by case. Since Tempest is a very newly published and unknown tool, I am currently more focused to its total coverage as a proof that it is fully programmable and may be used in anywhere so I am trying to add minimum +100 channels scrappers from countries that currently not exist in my repo and unfortunately I am Also very busy with my real-life work so trying to do more about it in limited times. Good to hear sky + fusion mode is promising for you. Fusion level will keep previous days as well when increased upto 15 days for catch-up services. If you need some other channels which is not exist in sky but available in any other public sources, I may try to built scrapper for it as well so you can add its channels to your sky config and grab all together. No worry about UTC/DST options because there is a time-converter module which may be set in your configuration file to convert all different time offsets to your requested offset. Example if you set it +03:00 from your config, all grabbed show times of all channels in this config, will be converted
+03:00 (not only offset change) or you can set as "Europe/London" so all will be converted with DST support.

For schedulesdirect, I was just reading their readme file also. Since I am out of their region, I have never personelly tried or used but based on what I read, their json api which contains 20days of data, may be easily done with Tempest. So in my first oppurtunity, I will try make an unlocked scrapper for you to add your credentials and test. Due to my blind try, it may fail on first without seeing exactly what is going there but for sure it will work on second or third.

For the xmltv structure, I know it is not in same order like all others but that structure is completely valid as well since there is requirement/restriction about it in xmltv_dtd and will be Accepted by all pvrs/players since channel tags and show datas are being parsed individually. I am Also using Tempest made xmtlv files for long time on 5-6 different players, kodi etc without any issue.

xmltv/xmltv.dtd at master · XMLTV/xmltv

Utilities to obtain, generate, and post-process TV listings data in XMLTV format - XMLTV/xmltv

github.com

That small structure change of Tempest, grants huge advantage about cyclone modifications which is alternative channel assignment for a failed channel with same previous xmltv_id/display name but new channels logo(if exists) and reduce overall memory usage of some other operations. So you cannot convert it to tradinitional order with a setting but it is fully legit and safe to use. Also you dont need to worry about xmltv verification for files generated by Tempest because it will fix even malfunctioned bytes and convert to xmltv safe characters with fill UTF-8 output

Thanks for the detailed reply kvanc,

That has answered a lot of questions. The XML validation question was purely because all of the scripting I have in place so far relies on the files having a valid structure and being UTF-8, but it sounds like all of that is already covered by Tempest so that should no longer be an issue!

I am also busy with work so am dipping into this in between other work when I can, but will look into the SD API as soon as I can.

PN23 · February 15, 2022

@kvanc Ok, so I have made some progress with SD but have not tried to configure anything in Tempest yet as am still working through the API.

SD has two services - SD-XML where you configure your lineups and channels on the website then grab the listings using a scraper for that service, and their SD-JSON service which is configured purely through the JSON API and not from their website.

I have previously been using the XML service with my existing scripting, but have jumped into the JSON one now to get my head around that one.

I am pretty much at the point where I can start to try to grab listings using API calls, but in learning in how the lineups and headends work, have reached the maximum limit of lineups you can change within a 24 hour period so I have to wait for that lockout to expire before I can configure my lineups again and try to grab.

kvanc · February 15, 2022

PN23 said:
@kvanc Ok, so I have made some progress with SD but have not tried to configure anything in Tempest yet as am still working through the API.

SD has two services - SD-XML where you configure your lineups and channels on the website then grab the listings using a scraper for that service, and their SD-JSON service which is configured purely through the JSON API and not from their website.

I have previously been using the XML service with my existing scripting, but have jumped into the JSON one now to get my head around that one.

I am pretty much at the point where I can start to try to grab listings using API calls, but in learning in how the lineups and headends work, have reached the maximum limit of lineups you can change within a 24 hour period so I have to wait for that lockout to expire before I can configure my lineups again and try to grab.

Hi, PN23.

SD-xml is their old traditional service and still being used actively due to easy access support for old end devices/grabbers. To use json service, you need a grabber which supports post/put/delete request methods for different operations. Also json service gives 20days of data. Tempest able to do these sync requests for json service and I more or less have a template in my mind about grabbing and channel creation. But currently I definately dont suggest you to make any try with Tempest since they may notice that request done with an unapproved application and they may terminate your subscription as stated in their user aggrement. I will officially apply them for application approval within this week and if I can get approval (I hope since only criteria is having a non-profit tool like Tempest), I will release an upgrade(rv1.3.2) for sha1 hashing which is needed for json login.

I dont know which countries/regions you need but during waiting you may try some recently added N.America siteconfigs such as USA and Canada which supports all the regions with thousands of channels and more may be generated with explanied zip/postal/region code methode in my github repo.

Edit:
And with my new page;