Cross-domain ajax requests with YQL

Recently, while working on this google chrome extension that let’s you search terms on urban dictionary, I ran into a little problem which I am sure most of you come across from time to time.

I wanted some HTML content from a web page. Now, there are some back end solutions for this problem like curl or functions like file_get_contents() in php. I have used curl before while setting up simple web crawlers and other applications, however functions like these require you to sometimes tamper with the php settings on your server, and enable them in order to get these functionalities. So once you are on a server hosted by some web-hosting company out there, configuration options are limited.

So, there are front end solutions for this. We could make a simple http request in javascript or a get/ajax request when using jquery, but again if you give it a try you will find out that most of the browsers today do NOT allow cross-domain http requests. You can only make those requests from your own server. I think it’s mainly a security concern, which is the whole point of having stuff called API’s which allow you to make “safe” web requests to a server and get a response typically in xml or json format among some options.

YQL(Yahoo Query Language) is an API of API’s like they say. It allows users to query html,rss feeds, tweets, flickr photos from their open data tables. So it’s like making an SQL query, only here you can actually query HTML content and get a result in formatted json or xml and therefore it provides us with the possiblity of making an API out of virtually any website!

I am not going to elaborate more on how to get started with learning YQL, because the YDN has done a great job in putting together a detailed tutorial for it.

The fun part is using the YQL console which looks something like this-

yql console

You can type in any query, in the text box and click on “Test” and you get a nice little rest query in the text box below the formatted output. We are going to use this link to get our result. Now YQL is a very versatile language, it allows you to filter through HTML, limit the number of results you want to get and a ton of other things. Feel free to mess around with the YQL console.

I am going to give you a small jquery template, that I used in my extension for making ajax requests to any server using YQL. You can easily modify it to suit your needs:

Sidenote– Try using ajax instead of get for making web requests in jquery for developing chrome extensions, it just works.