2011 in review: The Year It Get Too Close For Comfort

For a long time geeks like me tend to hide behind the scene when it comes to politics, unlike the US there is no law like the DMCA in Malaysia. We Malaysian don't see often see a law pass that directly affect IT industry here. Until this year, suddenly we have Computing Professional Board bill.

Which you can find a lot more online, but in short it is about regulate programmers that involve in Critical National Information Infrastructure. Which the list provided is too vague for comfort.

So what I hope to do next year, involve more somehow. What this year have shown us, ignorance and apathy can bite. Not all law have an open day that we all know, like the CPB. So law can be passed under our nose, and it will bite, and bite hard.

p.s Not including my own post on the Computing Bill, because it have quite a lot of inaccuracy

2011 in Review: Awesome Book

Books I have read in 2011

1. Makers by Cory Doctorow
2. Little Brother by Cory Doctorow
3. Make Magazine not book, but still
4. Quicksilver by Neal Stephenson, have still a whole set of Baroque Cycle, still try to finish it
5. Cryptonomicon by Neal Stephenson
6. Snow Crash, Neal Stephenson
7. Anathem, Neal Stephenson
8. Javascript the Quick Reference, Douglas Crockford
9. Make Magazine
10. Arduino Bots and Gadgets, that actually belongs to hackerspacekl
11. 
    

2011 in Review: Achievement Unlocked

Just a list of stuff being done in 2011

• Python on android with SL4A
• Programming the arduino for The Great Hackerspace Challenge
• Scraping govt websites for hoping to the Open Data Project
• Writing software for a robot
• Pickup a new python web (micro)framework
• Opening up data with scraping knowledge
• More linux skills on diff and grep
• using android for work
• rooting android and use cyanogen
• know why pdf is evil
• and use elasticsearch(which I didn't blog yet)

Some of the cool one anyway. 

Copy and Paste on GNU Screen

So recently I finally found out how to do copy and paste on gnu screen, previously I just do a drag on a mouse.

So copy and paste on gnu screen,

- ctrl-a [ 

- move to the first character of the string to copy, press enter

- move to the last character of the string to copy, press enter

- now to paste, press ctrl-a ]

p.s kina ashamed that it take me so long to figure out =.=

Rooting HTC Desire HD on 64bit Linux Mint

One reason why I have been resisting rooting my phone is because of the old way of rooting the HTC Desire HD. Best illustrated by the wiki on cyanogen mod. Essentially a many step process, of copying file, downgrade, copy file, reboot, and reboot.

Recently I have discovered Advanced Ace Hack Kit, basically these guys already automate everything as a 1 step process, if the firmware need to be downgrade it, will do it. But here is the big catch, it is a 32bit binary, and I installed a 64bit linux.

Thankfully mint, which essentially built on top of ubuntu(which in turn debian) have ia32-lib, which have 32bit version of library for 64bit OS. Just install it.

And read the manual, it is inside the package along with all the tools needed, unzip, and read the manual inside it. It covers everything, you need. There will be a quiz in the end ;-)

A little adventure on the bill watcher on the pdf part. One I am hoping to able to attach comment to a section on the pdf, then have a script that compile the comments, and email blast to the MP's

So a few thing I am trying

Loading pdf into iframe, 

It is actually a pretty standard approach of handling it. Except I am linux now, and i am lazy to install a plugin. 

Also iframe height is a problem, it is hard to get it work well across browser.

I don't know ways to have javascript to access the text inside the pdf. So I don't know how to attach jquery on it. 

Use pdf.js

Work well on firefox, but not anything else. Also this is also a very new library. 

While there is a text layer for text, it don't line up well in text in the bill document . I suspect it is because of badly form pdf. 

Javascript should work well on it though, as it have a  text layer div.

Convert pdf into html

Have jquery to use .load(). It work across browser, until the layout gone wrong, could be because of bootstrap(the css library). Pdftotext work very well, the html generated can be processed pretty easily. The layout on the other hand....

javascript should work well too.

load converted pdf's html into iframe

It does not solve the height problem on iframe, but javascript should able to use it. It is a bit of a pain though. Because the pdf generated is pretty well formed. Solved the issue of layout gone wrong because of css library though.

New Toy for christmas

Before Unbox

After Unbox

It is an arduino based board for android accessory development FYI. Expect new stuff coming this coming soon.

Hackerspacekl Meeting 2011

Hackerspacekl will have a meeting to decide on the future direction of our hackerspace. Detail of the meeting can be found in

http://www.hackerspace.my/2011/11/25/hskl-members-meeting-saturday-17th-dec-2011-10am

Opinion after #CPB2011 II

From Amanz
http://amanz.my/2011/12/lebih-info-berkaitan-akta-badan-perkomputeran-professional-diperlihatkan/

Thanks to amanz, they have capture some pictures from the Open Day.

For Item 4. Why not just take existing certification and use it? Are Malaysia that unique?
(This is actually question asked by many during the open day)
For Item 5. I did not receive a straight answer on this.
For Item 6. How does creating an extra barrier enhance supply for manpower?
For Item 7. This is disturbing. Does it mean, that refusal to register make me untrusted?


I think this is troublesome

p.s Anyone in India want to hire python programmer? I'm jobless starting on 23rd dec.

Opinion After #CPB2011 Open Day

While from their mouth, the Computing Professional will be limited to govt, I still waiting for supporting document. And from their mouth again, it is a work in progress. Interestingly no one from mosti is attended.

Paraphrase from the discussion from representative, it is to i.e paraphrase from what I understand the word they say.

1. Raise a standard of computing professional, both in education 
2. and b) work 
3. Because other country have a requirement in ISO for staff with certain requirement anyway. 
4. It is before the requirement before the govt can sign the Seoul Accord, disclaimer I do not know what is the requirement for the accord, so somebody enlighten me please. 
5. Add accountability
6. And it will be limited to govt CNII project only.
7. Everybody can apply to it
8. For IT professional to be recognized globally?

For 1, I do not believe there is a need for a board to fix it. There is an accreditation agency for university already,  and they are not doing a very good job. What make them think they can do a better job? Chances is, if a university/college have a good curriculum, there is no need to create a board for the first place, market will grab them first. SO FIX CURRICULUM FIRST!!!!!

For 2. I am not even sure what professional means. 

For 3. I believe that , it should be in the SLA. Again not a board should decide it. 

For 4, I won't comment, because I don't know enough. 

For 5, to add accountability. To be honest, I don't know understand how forming a board would help in that. There is contract, ain't that more than enough?

For 6, during the meeting, there is no clear definition of CNII for the first place. Even though the representative is assuring us that, it will be limited to govt. Unfortunately in IT, it is interrelated. A big chunk of malaysian IT, is involve in govt project either in contract or subcontract. Even though it says in govt, if tools get adopted, does it mean it falls under their jurisdiction? How about stuff that is regulated, such as system that interface with bank negara? Does it count as CNII?

For 7, if everybody can apply, then what is the point then?

For 8, that is done by, create good product, open source or not. Again not by a board.

Extra stuff

What does the cert do, that normal cisco, microsoft and other cert doesn't do? What make them less qualified that this

If everybody can apply a certificate, then why even bother? 

IT change fast, what the examination for it will based on. 

Personally, I believe IT is doing pretty well without any govt involvement. And from what I understand the justification during the open day. I still believe a board is not needed. And consider that many IT company aim for govt contract one time or another, it could spell trouble for industry in general. 

To be honest, after the open day, there is still more question than answers, many is unanswered. 

p.s I am jobless starting on 24rd Dec, anyone outside malaysia want to hire me?

#CPB2011 redux

I have been in the discussion. It is my fault in spreading some of the old fud. I here by apologized for the mistake I have done

Here is a few thing I have gathers, and I just pick up words as said by the representative.

1) CNII is limited to the govt only, for private sector it is business as usual!!!!!
2) IT IS A WORK IN PROGRESS!!!!!
3) Everybody can certified.
4) If you are a startup that are not involved CNII, it ok not to be certified!!!!

More to that later.

Let dig out a Localtunnel!!!

So as I creating a very early version of the webapp, I want to get opinion from friends. A few thing happen, I'm not quite ready to commit to a full blown installation. And I am too lazy to setup a ssh tunnel, and I didn't attempt it before.

Of course, there is option number 3. Localtunnel, basically localtunnel is a ruby script that make tunneling very easy, without you having a ssh box remotely. They provide a linux box remotely, which have no shell and only authorized_keys file in home. So you don't need a dedicated ssh box somewhere.

On linux, make sure you have ruby install, and rubygems installed. On ubuntu, on fedora just replace with yum. The rest of the instruction is on the link below.

sudo apt-get install ruby1.8 rubygems

http://progrium.com/localtunnel/

Bills Watcher Malaysia

Recently I got involved in a Open Data Movement in Malaysia, and one of my recent project is called Bill Watcher. It is a webapp that broadcast via twitter and rss on bills that is being debated, and being passed recently.

Main Page

Bills Detail

Basically this page scraped from the malaysian parliament website. And load into a sqlite database now, which I don't really care, because I am doing it via sqlalchemy for now, it make it easy to move to other database. Bottle.py just read from the database via sqlalchemy, and render it. Use 960 gs to make the rendered page look nice. 

The feature of this app, is pretty small, the pdf is iframe, there is no login. Fancy sharing feature is via twitter and facebook button and RSS. Commenting will be provided by disqus, if I figure out where to put it. Javascript is only used on twitter and facebook button. 

I consider this as MVP for this, small basic feature to be extended. So feature will be added as requested, but not all will be added. Also not a lot of information is available on the parliment bills page, so feature will be based on effort needed to extract it from other source, which actually not really easy. But otherwise, we will try our best to get feature to be added inside. 

What next, we going to host it live soon. Then we will add disqus, then finalized twitter notification. To get your hand dirty now. Go to the github link 

https://github.com/sweemeng/Malaysian-Bill-Watcher

I will transfer to the sinar repo soon. Need to do a bit update across repo. 

p.s 

Recently a bill being debated intensely, shows that how many stuff we don't know about the decision process in the country, even though it is there on the parliament side. Which does not make it easy for use nor navigater around. 

What Can't One Do in #CPB2011

From this link

http://www.scribd.com/doc/75107593/CPB2011-Draft

If this bill is pass you are not allowed to

1. Fix your friends computer, that is considered practice of computing. And you need to pass an examination for that anyway
2. Write your own mobile app, iOS or Android, because you need to proof that you can
3. On top of that, you probably don't have a service provider cert anyway. You can't sell it on the app store and android marketplace
4. So if there is new code being written on the internet, you probably cannot use it for service. Because there is no certification for it.
5. And you can't use anything that is new, to provide a service because you didn't register it. 
6. And the term of service is wide, you now can't sell your software
7. Write instruction on your blog on the software
8. You can't publish an web app, that uses your new found knowledge. 
9. Tweet about problem
10. Rant about the problem
11. Commit code to github, because you are not certified
12. Report bug, because, you didn't register that you know about other software that you use
13. You can't even learn new technology and share your knowledge
14. You can't share your idea, because you probably not qualified for design work
15. I am not allowed to setup my own server at work, and have to wait for other to do it, and worst if I am the business owner, I have to pay more. 
16. You can't even publish code unrelated to your service on github, because you are not a registered expert on that subject
17. You can't give input to forum on solving problem
18. You can't be a generalist. Now you need 1 sysadmin, 1 database people, 1 server people, 1 javascript people, 1 css people, 1 artist, and 1 backend guy(not sure how to split it up), 1 tester
19. You can't write your own unit test. Because you are not a certified tester
20. You can't do mockup
21. You can't do prototyping on your own
22. You can't do javascript, so no ajax or what not, because there is no cert
23. You can't do html, because there is no cert
24. I can't do python as a service, because there is no cert
25. I can't even use sql, because I have no cert

Just among those I remember. Could be more

The Malaysian Computing Profession Act

One of the hottest topic today is the Computer Professional Act 2011. As a software developer, this affect me quite a lot. I for one is against such a act, for both professional, and personal reason.

http://www.scribd.com/doc/75107593/CPB2011-Draft

A draft have been out not very long ago.


Definition


A few definition on the document

“Computing” is a goal-oriented activity to plan, architect, design, create, develop, implement, use and manage information technology or information technology systems.

So here I assume computing in general, from creation to usage.

“Computing Practitioner” means a person who has a job function in computing or qualification in computing

Here I assume this mean people that uses computer, from architect, software developer, system administrator, end user. You get the idea.


Registration
Finish a few definition. Now lets jump ahead to part 3, on registration. Lets continue

There shall be indicated against the name of each Registered Computing Practitioner and Registered Computing Professional kept in the Register a record of disciplines or specializations on computing obtained or acquired by such personnel through academic qualification or training including on-the-job training or skill or specialist or professional certifications.

So it means here, I need to declare what I know, what does it mean? Does it mean that I need to produce a cert? Does it mean my boss need to write a letter? Thing get a bit uncomfortable here.

• If we just declare, then it means the whole process is useless? We can just claim what we know.
• If we need to have a cert then here is an interesting part, a lot of new technology don't provide cert. Open Source Project also don't always have cert. Thing change fast enough the training will need to be updated often, and it will be expensive. 
• On job training? My experience on job is, it tend to go safe, and rarely able to use the latest in technology
• School? That usually is not helpful at all
• Then if not above, that how to judge this?

Let go next 

For certifications mentioned in subsection (2), the Board may maintain a list of certifications provided by associations and bodies in Computing recognised by the Board and will keep the list updated from time to time.

What is the criteria to go into a list? Does it mean it will monitor training provider, via accreditation? Consider the countries, reputation. How not to abuse it.

Qualification

Now on to qualification

14.(1) (a) Subject to this Act, a person who holds –  

(i) the qualifications required for Graduate Membership of a professional body or organisation recognized by the Board, and the qualifications are recognised by the Board; or 

(ii) any qualification in Information Technology or Computing which is recognised by the     Board; or 

(iii) any other qualifications, certifications or relevant experiences recognised by the Board,

      shall be entitled on application to be registered as a
      Registered Computing Practitioner.

So remember the definition on top, Computing Practitioner is someone that just use a computer. From my understanding, does it mean that, everybody need to register to use a software?  For i) what kind of qualification, for iii) what kind of relevant experience, and certification?

2) Subject to this Act, the following persons shall be entitled on application to be registered as a Registered Computing Professional:

(a) any person who is a Computing Graduate or any person who has other qualifications recognized by the board 

    (i) who has obtained the practical experience as prescribed under subsection (1)(b); and 

    (ii) who has passed a professional assessment examination conducted by the Board,    

Corporate Member of or is a a professional body or organisation recognized by the Board; and

    (iii) who has paid the prescribed fee and 

    (iv) who has complied with all the requirements of the Board;

On top of experience we now have examination.

• What examination will be conducted? How many type
• What platform? or how many
•  Or practical knowledge on only stuff from one vendor? 
• Does it cover just computing, then in the definition above, it can be anything, even using spreadsheet is computing. 
• If not what does it cover?
• If so, does it mean everybody can claim to be professional?

An extra note on, examination is that useless, there is too many tool, too many option. Limiting to one will be ridiculous and unfair to other.

Scope

Lets go to the scope of what I can or cannot do

(a) a Registered Computing Practitioner employment may take up which  requires him to perform Computing

Services subject to the 

i. work is carried out under the supervision or instruction by a Registered Computing Professional,

or

ii. similar work scope has been carried out by the Registered Computing Practitioner before.

and

A Registered Computing Professional may only provide

Computing Services in the disciplines or specialisations of Computing

 he is qualified to practise and as is shown in the Register under

subsection 12(2).

Does it mean that, I register myself as a software developer, does it mean that

• that I can only write program. 
• I cannot administer my own server? 
• I cannot fix my own computer? 
• Because I only declare that I can only do webapp, does it mean I cannot write other type of software(say web server). 
• Now if I self taught new technology, does it mean I cannot use it at work when opportunity is given. 

And because it is based on similar work carried out before, it means that;

• I can only do stuff that I only do before,
• so if I started with writing webapp, does it mean I cannot be administer my own server at work? 
• it mean I cannot write mobile application professionally? does it mean, if i ever sell my own mobile application, because I did not declare this, outside of my work?
• Or I can only work in one industry, once in banking forever in banking

I graduated in AI in UM(don't ask), does it mean I cannot do mobile application? etc.

One more thing is how do we describe a similar scope? Does it go very granular, i.e you can only do php, because you have done php. Or it can be high level, i.e you have to do asp.net because you do web development.

Software tend to require multiple skillset, in reality a person involve in software need to have multiple skill, more obviously seen in software developer. And we have to learn on our own, the world just move fast, if we ignore the problem, the train just gone. Setting up rule like that, just make thing worst.

Conclusion
I believe that this act do not benefit professional like me for many reason.

• For one, I don't see the point, what kind of qualification for one, everybody use computer now? 
• It limit my opportunity, I need to declare what I know, somehow. 
• Worst I can only work on stuff I declared and no more, 
• Opportunity tend to be based on past experience. 
• Self learning, is not covered, and will not be recognized. 
• Worst I am not even allowed to use stuff that is new, because I am did not declare that I am know it. But I need to use to learn, to use a new technology. So does it mean I can only learn in secret. 
• Technically I am not even allowed to sell mobile app, which nowadays is very easy to distribute

In the end of the day, I believe this bring us 10 step backward for the IT industry. And only make thing worst. 

.

Realized New Behavior in Virtualenv

Not too long ago, I covered virtualenv. One of the behavior is the --no-site-package option which will totally isolate the python environment. Starting from version 1.7, --no-site-package will be the default option, so you don't need this flag anymore. If you invoke virtualenv with this flag, it will display an error.

Converting PDF to Text with pdftohtml

Previously I have tried to extract pdf information by converting PDF to text, as described here.

Problem is,  a big wall of text is very hard to process.
Here come pdftohtml it is part of the poppler package on linux. But gnuwin do not have it for windows. Which is one reason I use pdftotext.

pdftohtml convert pdf to html. simple usage is

pdftohtml yourpdffile.pdf

You will get your html file. But it is a bit plain as they just extract text from it. It there is image inside pdf, or you pdf is pretty complicated, like Malaysian Hansard. You can use the -c

pdftohtml -c yourpdffile.pdf

Here is the catch, it will generate 1 html per page in the pdf, with images. But the layout is maintained. For document like Malaysian Hansard, it would be hundreds of page.

Then there is way to produce xml

pdftohtml -xml yourpdffile.pdf

You will get an xml file which the position information.

p.s I'm using this for http://opendataday.org/ Whether there will be result today.