Bills Watcher Malaysia

Recently I got involved in a Open Data Movement in Malaysia, and one of my recent project is called Bill Watcher. It is a webapp that broadcast via twitter and rss on bills that is being debated, and being passed recently.

Main Page

Bills Detail

Basically this page scraped from the malaysian parliament website. And load into a sqlite database now, which I don't really care, because I am doing it via sqlalchemy for now, it make it easy to move to other database. Bottle.py just read from the database via sqlalchemy, and render it. Use 960 gs to make the rendered page look nice. 

The feature of this app, is pretty small, the pdf is iframe, there is no login. Fancy sharing feature is via twitter and facebook button and RSS. Commenting will be provided by disqus, if I figure out where to put it. Javascript is only used on twitter and facebook button. 

I consider this as MVP for this, small basic feature to be extended. So feature will be added as requested, but not all will be added. Also not a lot of information is available on the parliment bills page, so feature will be based on effort needed to extract it from other source, which actually not really easy. But otherwise, we will try our best to get feature to be added inside. 

What next, we going to host it live soon. Then we will add disqus, then finalized twitter notification. To get your hand dirty now. Go to the github link 

https://github.com/sweemeng/Malaysian-Bill-Watcher

I will transfer to the sinar repo soon. Need to do a bit update across repo. 

p.s 

Recently a bill being debated intensely, shows that how many stuff we don't know about the decision process in the country, even though it is there on the parliament side. Which does not make it easy for use nor navigater around. 

A scraper running on the cloud

I have been writing scraper for sometime, as you can see in some of my old post here.

So recently thanks to Kaeru, introduced to me, scraperwiki. This is basically a service for you to run scraper on the cloud, with additional benefits:

• It runs on the cloud
• It provide infrastructure to store the data, in form of sqlite database, which you can download.
• It provide easy way to dump data as excel
• It provide infrastructure to convert the data into API 
• Somebody can fork the scraper and do enhancement on it. 
• A web based IDE, so you just write your scraper on it. 
• Everybody can see the code of the public scraper. 
• Scheduled task

One very cool thing about scraper wiki is, it support a set of third large library that can be used. It support Ruby, PHP, as well as Python. The API for scraper wiki is pretty extensive, it both covers it's own scraper, geocoding function, views for the data hosted on scraper wiki etc. 

My only concern is, let say I want bring my scraper out of the service, I will need to rewrite the saving function. But on the the data can be downloaded anyway, and I use python, so it is not that big of a deal. 

Below is a scraper that I have written, on scraper wiki. While it is mostly a work in progress, it show how it would look like. 

https://scraperwiki.com/scrapers/malaysian_parliament_hansard_url/

adventure with yahoo yql: part 1

Not too long ago, yahoo have release yql. What it does is, it query data from webservice, in a SQL like language. So you can have query like,

select * from tablename
use tablename
desc tablename

There is other query like join, and subquery, etc. But all the yql is read only.

To use the query, you can actually
First, you can use the query console, here. But the query is limited to build in tables from yahoo. Which expose data from yahoo api.

Above is the query console, with build in tables from yahoo, or a more interesting one is here. This actually use table from community made table.
Now you can not just can query yahoo's api, you can query weather data, twitter data, etc

The community made table is hosted at

Which is host at http://datatables.org/ with links to the table hosted on github too.

Another way is to use the REST based interface. For example
http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20search.web%20where%20query%20in%20(select%20title%20from%20rss%20where%20url%3D%22http%3A%2F%2Frss.news.yahoo.com%2Frss%2Ftopstories%22%20%7C%20truncate(count%3D1))%20limit%201&format=xml

Which is a bit of a handful. Which you should write a program to do so.

More can be found in the documentation here.

Ajax Chinese Chess

It's actually not many computer chess game in the market. By then I'm quite ignorant on software written in chinese. It's kinda fascinating to check out this game, since I study a lot of computer chess technique for my thesis. It's also fascinating to see that, chinese chess is not quite solve. Which make it interesting to me.

Either way, here is a chinese chess, on ajax,
http://ajaxchess.pragmaticlogic.com/

Been playing around with mozilla prism

Been playing around with mozilla prism, an interesting project, by mozilla, where a webpage(webapps), is presented like an application on your computer.

The big idea is, once you click an icon, on your desktop you should be able to launch your webpage,but this time, with very minimal interface.

Which to me is an interesting concept, because imagine, a web application, like my google apps, really look like application on the desktop.

Online Chinese Entry Software

There is time when i can't type in chinese, especially when i am using 32 bit software on my laptop. Which happens to be mostly browser. By then sometime there is not chinese entry software installed.

It happens that the guys in cari.com.my have a solution.
http://chinese.cari.com.my/ime/

It is a web based. Chinese entry system, which i suspect is written in javascript.

Blogged with Flock