Django, solr and a few interesting stuff Part 1 : Solr power!!!!

****Long story alert, no code, but tech related, jump to lesson learned in the end*****

The Beginning

One of the thing I am working on now involving solr, django and a few thing. Along the way we discovered a haystack. 

So the story started when we found a project, that generate cash. After talking to the client, and looking at their system. We decided to try, and we decide build it using django, and solr, among a few component. Though we did not confirm that we will offer search for the first place...

To keep it short. Thus we begin to build it. 

To Serve Just java -jar start.jar

We started with solr, because it is easy to install anyway. Just grab solr from http://lucene.apache.org/solr/. 

What really cool about solr, within the solr folder, is example folder. That is a fully functional solr project. 

just cd into example folder. and run java -jar start.jar. Follow the tutorial to use the example folder

Since we are a lazy bunch, so we just copy the example folder into our project folder. And modify schema.xml in example/solr/conf/. Since schema.xml in the folder is usable, we just modify the fields. 

Roadblock

Since this project original database have quite a lot of table. So FIRST STEP is denormalize the data. Flatten the whole thing. 

And we decided to use solrpy for our script to load the data in database into solr. Which look ok until we have a situation of 1 to many relation in the database. In solrpyr, each repetitive field is put into a list and added to solr. 

This is not the worst issue. When we pull the data from solr. The field is not in order. Thus, it doesn't maintain the structure.

Thus we learn, JUST store the ID or primary key in SOLR, so that it can be refered to the database, just use the KEY to pull data from database. 

Then after many trial of error, to detect empty field etc. 

We give up, we discovered haystack!!!

Lesson Learned in Using Solr

1) Denormalized all fields to be stored in Solr.

2) Sometime, solr output does not reflect to the structure of the database(ok, we are very new in solr)

3) So if it is from a database, indexed everything  and store the key(probably a bad idea, because we are not to hit the db a lot, i don't know really)

4) Don't reinvent the wheel. Turn out that haystack have solved our problem and more.

5) Partially number 4, the client just dump us a file outputed in database, we tried to be hero, but turn out better to store in db first....because it is a mess processing the data..

listing open file with lsof

So the story goes that I need to process (quite) a number of files..
Which I didn't write  a output to say what is being process..

lsof save that day. lsof can monitor file being open, which includes sockets.
For my case, the usage is easy

lsof +r -p  

The -p is for opening process, +r is to repeat until no file is opened by the process.

enjoy

End of the year and End of the decades

A short personal summary of what happen in the last 10 years

- Survived through school and university
- which I have learned c, c++, python, java, php, javascript etc
- and change os, from windows from linux, thanks to internet in university
- thus develop slangs and TWAs
- and run a few programs in university, involve junior bullying
- then outside uni got involve in a few barcamps and foss events
- also developer events
- and got a first job, in a bank
- and quit it barely after a year
- and realized that the computer is smaller and smaller, from thinkpad to eeepc
- and still the old nokia phone is still the best, how ofter you get 7 days of battery life in one charge
- this among the few of many stuff in the last decade....

switching to opensuse

opensuse is ok, it is linux, anyway so most is understandable..

With all localization build in, with many program. built in virtualization support(who don't provide that anyway in linux). Filesystem encryption support, with apparmor. plus with LVM. Plus all the enterpricey stuff like LDAP client etc.

wireless don't just works without tinkering. YES THANKS FOR THE TROUBLE BROADCOM. ATI will sucks so i don't bother using it. Python version is not the latest compared to ubuntu. but I use django mostly anyway. LVM is not as easy to setup, but I get the hang of it, you just need to make the the unused space as LVM, and resize it later.

Now please just let me do use my computer peacefully, I am just sick of have to be in between the stupid war between ubuntu and fedora....... So this is a small price to pay to avoid the annoyance in the community.

Though compatibility will sucks, but that is the job for xen(or kvm depends when I want to switch)

End of The Year Post: Community Attack!!!!!

** Long random Story Alert **
As usual, I just continue with a series of community events around me. Which is really interesting. Because some of the community is really a new one.

So some of the usual one is barcamp. Which is really the same. Though some of the geek among the barcamp community will give a more active role in the future.

As usual I am involved ubuntu-my, they are a official loco. And I have involved a little in mscosconf with them. But probably this is a last year I am involve with them. Or not let see...Since I change my os anyway. And I get cozy with fedora my...Which is a very technical community.

And python.my have started. We are a community of python programmer in malaysia. Which is an interesting community. One reason is, we are a programming language community. Consisted of user of both proprietary and open source software. We just join an learn the language. We also have our first big project with student. Which is pretty cool actually, me thinks.

So for some of the open source group in malaysia, many have happens. And many have get involve in universities. Which is very interesting to see. Though I really rather to see more synergy within all the groups.

So the tech community is getting active. Though, the community is growing. But the thing is, we don't just need to grow. I really think that some of the community really need to grow up. Attitude is a big problem for some. But generally it is interesting to see the community exist.

And we have a few that just started so stay tuned.