I got a feeling this will be a interesting year. For everyone.
Friday, December 31, 2010
Wednesday, December 29, 2010
A little misadventure with django and database
** this could be long**
It is almost the end of my project now, along the way, found a misadventure with quite number of stuff. Which included Django ORM. What is not to love for a ORM, it make a nicer abstraction to wring program that need to talk to db. Making us, to worry less about sql. Using django ORM with south, it make refactoring database a lot easier.
It is almost the end of my project now, along the way, found a misadventure with quite number of stuff. Which included Django ORM. What is not to love for a ORM, it make a nicer abstraction to wring program that need to talk to db. Making us, to worry less about sql. Using django ORM with south, it make refactoring database a lot easier.
Well in my current job. The concept of using ORM is pretty new in the company(from what I see). So the projects they done is done like pre-ORM days. Writing SQL, doing select etc. For a person that is used to ORM, I am thinking like "Really?". So the way they do things is obviously very different, and this with Django's ORM, not a pretty sight.
For example, when integrating with other project. The expectation is, from a temp table that stores information from the other DB then write a INSERT statement, and push it to the target table. I was like again thinking "Really? They really did that?", especially the target table is huge.
And of course, in Django, I pretty used to have foreign key to reference to another models(table), which I have a few. And because I use django south, so that I can refactor the database from time to time. The order of the column is weird. Not to mention that the idea of a ORM is to make it look like a object, in my case python object, so I used to use boolean type, which from other table, uses char. So I imagine create a INSERT statement with a few SELECT, and map the variable properly. Pretty scary to me. I ends up writing a python script that uses the ORM to do it.
For the temp table, that contains info from the program i need to integrate with. It tend to be created by hand, because it make the existing script to export the table to sane. So have to create a unmanaged table. So django don't create the same table. Which I found, that for that particular case, I can't create a django fixtures for that models without losing data. Which I still investigating. In the end of the day, I just do a sql dump, just in case.
Also, as we build the application, the team is expecting a totally different behavior for Django queryset API. When in SQL, when you can't just delete a table that have another table reference to it. Django queryset don't seems to care, and will delete it anyway. So that is causing issue in a way that, we have to add checking before the user can delete anything in the application.
Talk about deletion, django ORM added constraints on column that refer to another table as they create the table. So manually delete and insert via SQL is a pain. But it is always faster to do that!!! Because the queryset API can be slow. And sometime they have quite a number of magic that scares me sometime and sometime bites.(or maybe I just need to understand it more).
After all these misadventure. I just realized, you can't just drop a database guy can get them to use ORM. because the workflow is totally different. The technology is different. Everybody need agree that approach, or properly planned it. Actually I am not sure. But what is sure, mix of ORM and non-ORM approach can be a mess.
For example, when integrating with other project. The expectation is, from a temp table that stores information from the other DB then write a INSERT statement, and push it to the target table. I was like again thinking "Really? They really did that?", especially the target table is huge.
And of course, in Django, I pretty used to have foreign key to reference to another models(table), which I have a few. And because I use django south, so that I can refactor the database from time to time. The order of the column is weird. Not to mention that the idea of a ORM is to make it look like a object, in my case python object, so I used to use boolean type, which from other table, uses char. So I imagine create a INSERT statement with a few SELECT, and map the variable properly. Pretty scary to me. I ends up writing a python script that uses the ORM to do it.
For the temp table, that contains info from the program i need to integrate with. It tend to be created by hand, because it make the existing script to export the table to sane. So have to create a unmanaged table. So django don't create the same table. Which I found, that for that particular case, I can't create a django fixtures for that models without losing data. Which I still investigating. In the end of the day, I just do a sql dump, just in case.
Also, as we build the application, the team is expecting a totally different behavior for Django queryset API. When in SQL, when you can't just delete a table that have another table reference to it. Django queryset don't seems to care, and will delete it anyway. So that is causing issue in a way that, we have to add checking before the user can delete anything in the application.
Talk about deletion, django ORM added constraints on column that refer to another table as they create the table. So manually delete and insert via SQL is a pain. But it is always faster to do that!!! Because the queryset API can be slow. And sometime they have quite a number of magic that scares me sometime and sometime bites.(or maybe I just need to understand it more).
After all these misadventure. I just realized, you can't just drop a database guy can get them to use ORM. because the workflow is totally different. The technology is different. Everybody need agree that approach, or properly planned it. Actually I am not sure. But what is sure, mix of ORM and non-ORM approach can be a mess.
Wednesday, November 03, 2010
Load Testing with Python Multi Mechanize : The setup
Few week back, I was demonstrating a script to open a webpage with python using mechanize. Extend upon that, I use this along with multi mechanize. A Python based tools to do load testing on a webpage. What it does differently with mechanize is, this actually run a set of test script in parallel. And you can use any library to access the page, it can be mechanize, or urllib or httplib.
To setup, you will need python 2.6 above. Because it uses multiprocessing module that is available for python 2.6 and above. You will also need matplotlib which also require scipy.
Then you just download multi mechanize and extract the file to any accessible directory. Go to the project folder there will be a default folder, that is a example project. A project will need 2 folders and a config file, create a new project folder, within it create results directory, and a test_scripts directory, and copy config.cfg from default folder.
Next round we going to write a script
To setup, you will need python 2.6 above. Because it uses multiprocessing module that is available for python 2.6 and above. You will also need matplotlib which also require scipy.
Then you just download multi mechanize and extract the file to any accessible directory. Go to the project folder there will be a default folder, that is a example project. A project will need 2 folders and a config file, create a new project folder, within it create results directory, and a test_scripts directory, and copy config.cfg from default folder.
Next round we going to write a script
Monday, October 25, 2010
Startup in 3 days aka #startupweekendkl2010
I got a chance to join Startup Weekend KL 2010 last weekend, thanks to Daniel Cerventus. The idea of Startup Weekend is to launch a startup within 54 hours. It is a place where aspiring entrepreneurs share and grow their idea, and hopefully launch a startup by the end of the program.
We start by sharing our idea on Friday and form a team on the same day. On Saturday we begin to work on our idea, with whatever we have, some work on presentation, some doing business plan, some codes. We also have visits from Mentors to guide us our way to the business . This continue until the end of Sunday where we will pitch to Mentors which already launch their own startups before.
I join a team called, findmiddle.com. We produce a webapp that recommend place for meetup based on a center point from 2 location. Our team is awesome in a way that we are one of the very few team that manage to produce a product in the end of the day, we also have a balance team, where we have techie and business guy.
What I learn on the day on startups.
Learn more about startup weekend here
http://startupweekend.org/
You can look at our slide here
https://docs.google.com/present/view?id=0ARiCz9gqQzLNZGM1d3RxYnFfNmZwODVxd2Ny&hl=en
Photo is here, thanks to vivian
http://picasaweb.google.com/GEWMY2010/StartupWeekendKL02
We have a facebook group in
http://on.fb.me/startupweekendkl
Our product is in
findmiddle.com
We start by sharing our idea on Friday and form a team on the same day. On Saturday we begin to work on our idea, with whatever we have, some work on presentation, some doing business plan, some codes. We also have visits from Mentors to guide us our way to the business . This continue until the end of Sunday where we will pitch to Mentors which already launch their own startups before.
I join a team called, findmiddle.com. We produce a webapp that recommend place for meetup based on a center point from 2 location. Our team is awesome in a way that we are one of the very few team that manage to produce a product in the end of the day, we also have a balance team, where we have techie and business guy.
What I learn on the day on startups.
- Get a balance team, we got a combination of tech and business, builder and speaker.
- Keep it small, so that we can focus, so that we can build it fast
- Web api is free and useful, use it. It would be impossible to build findmiddle without google map, 4sq api and bit.ly api.
- Cloud based collaboration tools is free and useful. We use gdoc, dropbox, and google analytics.
- Open Source Software is easily available, and is full of libraries. We use a php, jquery and php mailer.
- Tech doesn't matter, who can use the tools fast, rules. I prefer python, but we got 2 php guy in the team, so findmiddle.com is on php.
- Web hosting is cheap, few hundreds Ringgits. Nuff said
- Human attention span is short, make a pitch short and interesting.
- Talk is cheap. Implementing findmiddle is not without bumps, but in the end it worth it.
- Murphy's Law is everywhere.
In the end, I would like to thank, all the organizer of startup weekend in kl. I have learn a lot, and it is a interesting experience. Sorry for the wordy story, I forget to bring my camera.
Learn more about startup weekend here
http://startupweekend.org/
You can look at our slide here
https://docs.google.com/present/view?id=0ARiCz9gqQzLNZGM1d3RxYnFfNmZwODVxd2Ny&hl=en
Photo is here, thanks to vivian
http://picasaweb.google.com/GEWMY2010/StartupWeekendKL02
We have a facebook group in
http://on.fb.me/startupweekendkl
Our product is in
findmiddle.com
Thursday, October 14, 2010
Meetup with Graham Dumpleton
Hi
Everyone is invited for a meetup with Graham Dumpleton, he is the developer of modwsgi, a implementation of WSGI standard for as a apache module . WSGI is a interface for python web application in accordance with PEP333
Details
Date: 16th October 2010
Time: 2.00 PM
Location: Hackerspacekl Damansara Uptown
Anyone that is interested, join us.
Subscribe to:
Posts (Atom)