Python Web Scraping

There is time where there is information in govt website of is very useful, but unfortunately the data is in form of website, it could be worst as it can be in PDF. So it can be a pain if we wanted to use information for programming, but there is no API.

On the other python is a pretty powerful language. It comes with many library, include those that can be use to do HTTP request. Introducing urllib2, it is part of standard library. To use it to download data from a website can be done in 3 line of code

import urllib2 
page = urllib2.urlopen("url")

The problem, then is you get a whole set of HTML, which a bit hard to process. Then python have a few third party library, the one I use is Beautiful Soup. Beautiful Soup is nice that it is very forgiving in processing bad markup in HTML. So you don't need to worry about bad format and focus to get things done. The library itself can also parse XML, among other thing.

To use Beautiful Soup,

from BeautifulSoup
import BeautifulSoup
page = "html goes here"soup = BeautifulSoup(page)
value = soup.findAll('div')
print value[0].text

But you need to get the html first don't you?

import urllib2
from BeautifulSoup import BeautifulSoup
page = urllib2.urlopen("url")
soup = BeautifulSoup(page)
value = soup.findAll('div')
print value[0].text

To use it, just download the data using urllib2 and pass to to beautiful soup. To use it is pretty easy, to me anyway. Though, urllib2 is going to be re organized in python 3. So code need some modification.

To see how the scraper fare, here is a real world example, in github part of a bigger project. But hey it is open source. Just fork and use it, in the this link.

So enjoy go forth and extract some data, and promise to be nice, don't hammer their server.

Usefulness of __dict__

I am writing a validation function to check for objects instance, in this case, it is a django models. The object is mostly the same, except there is a extra field that need to be checked, that field exist only in certain object, but not all object.

So being lazy, I use the __dict__ special attribute. It exist in every python object, and contains symbol tables of the object, which I can use to check for existence of a attribute. For example

https://gist.github.com/1039978

so we can do thing like check a item like 'test_val' in c.__dict__ to check for a existence of a attribute. 

From the same idea, one very cool thing to do is, convert get the __dict__ and convert the attribute into a json object, with example

https://gist.github.com/1040013

One idea I have been playing is, because django model instance is essentially a objects, we can use the same idea, to output the value in a single django model instance into a json string, but we need to be careful on certain data type like double etc. 

The big catch is of using this is, __dict__ only contains attributes of a objects, it does not contain built-in attributes, and it does not contain methods. So if a value is from a method, you need to think of another way. Which actually sucks, as I have a lot of such methods in my classes, well I probably figure out by that time.

So, happy coding

The Great Global Hackerspace Challenge

Not long ago, Me and the Hackerspacekl gang, join The Global Hackerspace Challenge. Basically we build a Arduino Shield that process words.

Actually the whole process is best viewed on the hackerspacekl blog.
http://www.hackerspace.my/category/projects/the-story-box

In summary, here is what we learned.
- K.I.S.S, Keep It Small and Simple
The Atmega168 have only 16KiB of memory which the code itself taken half of it. and it is slow, it runs on such a speed 16 MHz. Modern computer is around 2 GHz. It better be simple, because debugging is hard.

- Serial is your best friend (on arduino anyway)
Unlike programming on PC or Web app. There is no print function to be used to debug, there is no debugger either. Even we have a LCD, it is not reliable. And serial pretty much built into the arduino board. So USE IT.

- Optimization matters
On dynamic language like python, or more modern language like java. There is garbage collector, even c/c++ have OS to help on that. So one don't need to worry about memory issues. But on arduino, removing unused code and code, and save a lot of memory.

- One need to think very low level.
The I2C to eeprom code is about  shifting bits/bytes to write to the EEPROM. For once we are thinking in bytes. And we need to know a bit of hardware, to write the code properly.

Overall, it is fun and a interesting experience. For a programmer that spend time on python, or doing web development. Opening one eyes to embedded programming a bit, a little bit of experience that meant a lot to me.

Let Android read my SMS

Another I tried on SL4A is to play with their SMS function. So, with the resulting code below:

1. As usual import library, and create the Android() object
2. and from the Android object call smsGetMessages, with a required parameter for unread message, True for unread only, false for other wise.
3. and call the build in android Text To Speech software to read it out, by calling ttsSpeak method.

import android

droid = android.Android()

result = droid.smsGetMessages(True)

for i in result.result:

droid.ttsSpeak(i['body'])

the result in the output is a list of such dictionary, in python notnion. Since I only want the message so I call it by i['body']

{u'_id': u'59',
u'address': u'Address of sender aka the phone no',
u'body': u'Message Body',
u'date': u'1300254988000',
u'read': u'1'}

the date is the datetime read and the read is 1 is read, and 0 otherwise. The ttsSpeak method is easy to use too, just pass in a string.

Originally I read all messages, and pass to the tts library. Turn out to be a bad idea, because I have no idea how to stop it from speaking once it started....

My First Day on Android Scripting

I got myself a android phone not that long ago. One reason is, it is pretty amazing piece of hardware. Unlike iphone the SDK is available on major OS, including linux. One of the many stuff I installed is SL4A, Scripting Layer for Android, and the python interpreter for android.

One cool thing SL4A do is, we test the code remotely from a python. The wiki page have a good explanation on how to do this.

http://code.google.com/p/android-scripting/wiki/RemoteControl

One of the first thing I play around via the python interpreter on the laptop

import  android
droid = android.Android()
data = droid.getNetworkOperatorName()
print data.result

The api page have a lot of information, so that is one place that one should look. 
Not much of a program nor a post, but yeah it is a start of something beautiful. I hope