TV listings, syndication

[Erkki]

I started writing an AJAX* TV guide (or is listing the better word?) that let's you customize which channels you see etc. and make your own personalised listings. I was thinking of doing drag'n'drop like http://brothercake.com/site/resources/scripts/dbx/ , but I realised that there isn't much need for it in reality so currently the personalised listing can be filled by clicking on programmes in the channel listings. Plus, the dragging code looks complex and since I haven't spent much time on this yet, I may take a look at it later.

At first I'm planning to use it for channels broadcasted in Estonia only, but if I can make it flexible enough, I could make it international. That seems like too much of a hassle though.

In the end, if I get that far, it could be so AJAX-ified that it's just a HTML page you download to your computer and the personalisation is stored in cookies.

Right now I have a very early version, running on Apache Tomcat because the "grabbers" that get the listings from the channels' websites are written in Java (I've had enough of PHP for a few years now). Here's a screen:

After doing a bit of searching, I found that there isn't a very easy way to get the listings, though. Some TV channels websites have XML syndication of listings, but it doesn't seem prevalent. A local channel even seems to obfuscate their web site to stop people from grabbing their guides (using seemingly random hashes instead of real dates as URL parameters -- it's possible that this is for another reason, but I have doubts).

I wonder how all the current web TV guides are getting their data? Even if some channels provide XML feeds, they don't seem to advertise it on their sites.

* AJAX - Asynchronous Javascript And XML (I think) -- basically means some stuff is done in the browser (with JavaScript) and XML content can be fetched from the server without refreshing the entire page.

[Dingo Chavez]

* AJAX - Asynchronous Javascript And XML (I think) -- basically means some stuff is done in the browser (with JavaScript) and XML content can be fetched from the server without refreshing the entire page.

That was very helpful because I honestly thought it was the all purpose cleaner Ajax. I honestly did.

Have fun.

I doubt I'll start something like that though, I hardly get any time on the PC (damn that evil big brother of mine). I'm also not very bright.

[Erkki]

However tempting (or not) it is to write screen scrapers, I decided to take a different, even more AJAX-ified route.

There is a tool (and a file format) called XMLTV, which already has some scrapers. I will try to write the TV listings app in JavaScript only, so it doesn't even need to run on a server: you just point it to an XMLTV file and it will import all channels and programmes that are present there.

I already have a prototype working. But I haven't found any online sources that output XMLTV -- so I'm using local files for now.

In other news: When are computers fast enough?

When IDEs are built in JavaScript!

http://developer.tibco.com/

http://weblog.infoworld.com/udell/2005/05/25.html

[Erkki]

You can try the initial version now. It includes a sample XMLTV file,but if you have an URL to a different XMLTV file, you can probably use that.

http://lap.ee/erkki/tv/

before clicking the "load" button, check "load all immediately" or

after the channel list is filled out, click "load all" or a channel name in the channels list to add the listings for that channel for the next 16 hours.

[Chris]

That was very helpful because I honestly thought it was the all purpose cleaner Ajax. I honestly did.

I thought it was the mythological hero.

[Marek]

I thought it was the soccer team.

[miffy495]

I thought it was the little town outside of Toronto. (This is just going to keep going, isn't it?)

[Erkki]

Updated: http://lap.ee/erkki/tv/

Now with drag'n'drop functionality (drag channels to the table header).

Sources:

cache/tv_grab_fi.xml (1,16 MB)

cache/tv_grab_nl.xml (948 KB)

cache/tv_grab_uk_rt.xml (39,9 MB!)

I advise not to try the UK source, ass it will take quite a time to parse (the 39,9 MB file will be fetched only once, but completely parsed every time you add a channel) There's gotta be an easy way in XML-tv to split a file by channels... that'd make it faster.

Oh yeah, and Firefox is recommended. There are more bugs when viewed in IE.

[Huz]

Sorry if you're already aware of this, Erkki, but have you seen the BBC competition to do 'stuff' with the BBC's schedule data?

[Erkki]

No, I wasn't aware of that competition (I was only aware that bbc opened some kind of backstage stuff). Thanx a lot for pointing this out, I'll look into it.

PS. I just uploaded a new version with greatly improved performance, but right now only the Finnish listings are available for TV1,TV2,MTV3 and Nelonen.

Right now my immediate plans with this are:

1) some kind of calendar view to select what date/time to view.

2) rearranging and removing the channels once they have been added.

3) improving the visual style

4) personalising the listings (maybe coloring based on filters)

You could actually download this site and use it locally in Firefox -- IE doesn't seem to handle XMLHttpRequest for file:// urls.

[edit]just for example, I made it available as a .zip file. Removed some more listing data to make it attachable here. You could use your own XMLTV files, if you follow a certain pattern when naming them.

channels_%arbitrary-source-id%.xml for a file that contains only channel listings and %channel-id%_%YYYYMMDD%.xml for listing files split by channel and date.

tv.erkki.lap.ee.zip

[Erkki]

Damn, the competition is only open to UK residents (to win prizes at least). I may still enter though, if I can hack anything together that soon (Sept. 5th)

[Erkki]

Heh, just tried to access a BBC url with an AJAX request:

IE gives a security warning but will be able to load the page if you click yes; interestingly Firefox doesn't produce any errors but just won't do anything. I guess it isn't allowed to use the XmlHttpRequest for a different host than the current page...

[Huz]

You could actually download this site and use it locally in Firefox -- IE doesn't seem to handle XMLHttpRequest for file:// urls.

[edit]just for example, I made it available as a .zip file. Removed some more listing data to make it attachable here. You could use your own XMLTV files, if you follow a certain pattern when naming them.

channels_%arbitrary-source-id%.xml for a file that contains only channel listings and %channel-id%_%YYYYMMDD%.xml for listing files split by channel and date.

Where can I get the XMLTV file for the UK (the huge one)? I tried it on your site earlier - it did indeed take ages, and there seemed to be something strange going on with what channels your app would show. I'd like to try the new version on the same data...

[Erkki]

To anyone who has tried this drag'n'drop thing: do you think it's useful?

I'm starting to get the feeling that although it may look cool at first, it would be easier instead of dragging just to have a link/button to click on to add a channel to the table.

But that would mean adding lots of links/buttons everywhere...

Dragging could still be useful for reordering the table, though.

[Erkki]

Now you can highlight programmes by entering searches/filters (JavaScript regular expressions are supported). If multiple searches have different color highlights, an in between color will be intrepolated. Similarly, existing filters' colors can be modified.

This is all highly experimental, but I hope I'll end up making something usable

next steps:

• reorganize code and make the page work in any state and implement some session saving. this includes:
  
  • you should be able to "reset" the page to it's original state
    
  • you should see the same page you last visited when you come back (some state saved in cookies)
    
  • if you use "save page as" and select "web page, complete" (at least firefox saves dynamically generated HTML when you do that), you should be able to use it offline just as well as online, except the listing data that was not already downloaded will not be available.
    

  [*]select times to display...

  • a calendar view that shows for which days there are listings available etc.
    

  [*]improve searches / filters (possibly search from not-displayed data)

[Erkki]

Where can I get the XMLTV file for the UK (the huge one)? I tried it on your site earlier - it did indeed take ages, and there seemed to be something strange going on with what channels your app would show. I'd like to try the new version on the same data...

oops... I missed this post before... Still interested? I haven't worked on this thing meanwhile, but I'm in the process of putting new data files up (they are now split into smaller sections)

But the data files are not available anywhere as such, but have to be created using XMLTV ( www.xmltv.org I believe) -- As I understand using the listings data scraped from channels' web sites (or whereever the various XMLTV scrapers get it) is ok for personal use, but sharing the files is not ok. So possibly I will make that thing available as a download if I ever finish it (you'd have to provide your own data files), or find other/legal online data sources for it.

[Karimi]

You're just typing all this up to feel smart aren't you ?

[Erkki]

You're just typing all this up to feel smart aren't you ?

And you're just asking this question to feel stupid aren't you?

[Knar]

oops... I missed this post before... Still interested? I haven't worked on this thing meanwhile, but I'm in the process of putting new data files up (they are now split into smaller sections)

But the data files are not available anywhere as such, but have to be created using XMLTV ( www.xmltv.org I believe) -- As I understand using the listings data scraped from channels' web sites (or whereever the various XMLTV scrapers get it) is ok for personal use, but sharing the files is not ok. So possibly I will make that thing available as a download if I ever finish it (you'd have to provide your own data files), or find other/legal online data sources for it.

Yeah, it seems like it. I just read an article on XMLTV - even though the program itself can be used for making your own PC TIVO. Funny how they won't let you modify their data. I mean, the best you can do is work on a personal copy of XMLTV and maybe discuss the interface with us.

This guy seems to a have a solution on how to make the query for his chosen shows, but once again, you still run into the legalities of using XMLTV in the first place.

http://www.hackdiary.com/archives/000038.html

Maybe you can ask the author of the website for any suggestions. Hope this helps!