Blogs

We attended a Faculty Development Program(FDP) held at Thiagarajar College of Engineering. It covered various FOSS topics for one of the elective subjects. During our stay we met the HOD of Computer Science Department Dr.S.Mercy Shalinie. She is a great teacher, she told us (Puneeth and me) about their various activities related to campus wide network management, website development and computing related infrastructure, which were very apt for creating active student community.

She asked her students to show us the features of tcenet and other initiatives for student participation and development. First of all, they declined to take services from their hardware (servers, routers etc.) providers, so that students can setup things on their own and get a chance to work on live projects. Faculty members also play a very minimal role in these activities. Students are given the freedom to try, test and fix. Then they told us about their hierarchy and work-flow. Their website, intranet, and mailing lists, are all managed by students (passed-out and present). They have an svn repository for all these projects. A few fourth year students have the commit rights and some alumni who were involved previously have commit rights from outside the campus. They have a trac based system for project management, and active mailing lists for discussions related to all these projects. Students from second year are encouraged to participate, initially they are required to submit patches for fixing bugs, and then just like all other open source projects they also have to prove themselves to get commit rights (AMAZING).

Not only that, they also provide an svn repository for all the final year students for their Major projects, even for non CS/IT students. For beginners, they had a poster system with sticky notes to paste the tickets and tasks so that they can get hang of version control systems and trac. All students, current and passed out, are provided shell access (feature which I find really essential and handy to try out some things which are not possible cause of work restrictions). Continuation of a previously done major project by new final year students is encouraged, so that code keeps on building and improving, which makes sense to me. Student sitting around in the lab, during the hands-on sessions were all into Python and Emacs whose limitless powers were unknown to me until recently.

To add to this, Shalinie madam was visibly taking pride in the achievements of her students when she was mentioning all this. She never forgot to mention that all this was new even for her and she hung around with her students to learn all of this. For each tech-fest and event, they try to invite passed out students to interact with current students and work together to fix issues, which is really ideal thing to create efficient learning atmosphere.

All this perfectly fits a Chinese proverb used in kurose-ross computer network book which goes like this:
Tell me and I'll forget; show me and I may remember; involve me and I'll understand.

Hello,
I must first warn you that this is a long post!
In the midst of a slightly misguided discussion at BangPypers, there was a comment made on the nature of what SciPy is all about. It occurred to me that folks outside the scientific computing community may be unaware of the nature of issues that concern the scientific community and their significant "mainstream" Python contributions. Let me add that I mean "Scientific computing" and "Python" in the larger sense, not just the "scipy" package.
So, what has the Sci brought to Py?
Scientific computing has an extremely large audience -- at least in terms of the sheer number of students. There are probably 10 times as many people who do not do IT/CSE than those who do. The problems they have to deal with are typically quite complex! Many of them have to perform serious numeric computation, symbolic computation, make plots of complex data in 2D and 3D, perform effective visualization, build user interfaces, be web aware and most importantly work with other languages in the interest of performance. They also have some of the largest computational needs anywhere.
Secondly, it is a fact that many of the mainstream scientific computing tools today that are written in Python are actually really sophisticated beasts. I will illustrate several of these with specific examples. I am detailing several packages whose internals embody very interesting tools and techniques and often these have crossed over to mainstream Python.
Probably the most popular Python interpreter used world over is IPython , which was started by a good friend of mine (Fernando Perez) who is actually a physicist by training and currently working on neuro-imaging at Berkeley today. Guess what, his motivation was to write the tool we all love today? -- the need for a powerful interactive interpreter like Matlab/IDEAL and Mathematica. Technically there is a ton of interesting stuff going on under the hood -- working with a curses-based terminal and getting that working in a cross platform fashion is a big task. IPython happens to interact really well with the underlying shell and provides several threaded environments (-wthread, -q4thread, -gthread, -pylab). IPython is also breaking interesting ground in the area of distributed/parallel computing with their newer versions. In addition to this they also broke somewhat new ground in the "Python for scientific computing" community by using DVCS very early. It is very likely that Fernando will be here with us for SciPy.in 2009.
Sage is a huge package that brings together a very large collection of other tools written in a variety of languages under one Python based platform along with a very cool web based interactive notebook. The target audience being mathematicians, scientists and engineers. Sage also has a very large developer base, and follow the principle of "release very early and very often". They have some great documentation and a really solid development policy. They have over 75% test coverage (and growing) -- for a tool that large, that is really quite something. The sage project also forked a branch from Pyrex that has led to a very important piece of the scientific computing puzzle, namely high performance Python via Cython. This is one project that will probably make best use of any type annotation additions to Python.
Keep your eyes open for a potential "Sage days" (a Sage conference) in August in India next year that FOSSEE will be organizing with the Sage lead, Prof. William Stein.
The NumPy package is the backbone of most Python based numerical computing packages and has actually motivated the need for enhancing the buffer protocol in Python3. Incidentally, Travis Oliphant, the architect of numpy and the buffer protocol for Python 3 is the keynote speaker at SciPy.in this year.
The SciPy package itself builds atop numpy and a suite of well established algorithms written both in C/C++ and FORTRAN to provide Pythonistas a mechanism for serious number crunching.
matplotlib library is the leading 2D plotting Python package in business today. Almost everyone uses it, all over the place.
The tools that I've been personally working on, for the last several years also deserve mention for their sheer technical qualities. The Enthought Tool Suite (ETS) boasts a huge array of useful packages along with very interesting technology (all of which is BSD licensed BTW). Traits forms the cornerstone of this tool stack. Traits allows us to write code with a strong emphasis on focusing on your model, strongly encourages MVC and allows for inversion of control. I like to think of traits as Python attributes on steroids (without side effects)! It makes it extremely easy to create simple and fairly complex UI's with little or no UI code (save a declarative specification of what elements you want to see). Traits also provides two UI backends for wxPython and Qt4. Gael Varoquaux has written an excellent tutorial on the topic.
ETS features Envisage which is a Python-based framework for building extensible applications. The applications can be extended by using plugins. Think of it as an Eclipse-like application framework for Python.
Sitting atop this is Mayavi which features several interesting features. It provides powerful traits based 3D visualization that is at accessible as an application, a pure library, a one-line (matlab-like) convenience wrapper usable like matplotlib's pylab from an IPython session (or pure Python scripts), envisage plugin and a pluggable widget for traits based apps. Mayavi also features some really neat features like automatic script recording.
ETS also features a powerful interactive 2D and flexible 2D plotting via Chaco.
Clearly, there is a lot of very interesting stuff in the few tools that I've listed above that have a significant impact to both folks in the scientific computing community and outside.
Incidentally, did you know that GvR was the keynote at SciPy07 and Alex Martelli was the keynote last year at SciPy08? And do you know who the keynote was at SciPy09? Peter Norvig! These conferences were held at Caltech.
From this, it should be evident to those who are in the community that really the Scientific computing community is not a fringe community in relation to the mainstream Python community.
Since Python is such a powerful programming toolkit for scientists and engineers, we wish to spread the adoption of Python in the college curriculum and thereby improve the quality of education at the college level.

There has been a bit of a lively discussion at BangPypers over our choice of drupal and also on the skill levels of our team member etc. Of course it is the inalienable right of anyone to make such comments, express their opinions and to tell us what we should be doing and take us to task when we say or do something that is incorrect. The feedback part is welcome.
Of course there is some confusion there as people have assumed that the site scipy.in is in drupal and on that mistaken assumption have given kind advice about how we may lose our credibility in that conference.
scipy.in is developed in django
fossee.in a site that represents a larger project namely python + scilab + .... is in drupal.
Let me try to explain what and why we did.
We are a group formed with the goal of aiding and enabling the use of python in the teaching of Engineering subjects, maths and science. It is part of a larger project to enable the use of open source tools (of which python is only one) in education.
All the python group members have been recruited with this goal and in our communications with them we have stressed that they will be contributing to this end, by

• designing and developing courses for students and faculty to learn python
• helping faculty modify existing courses by replacing examples that use commercial software with python based tools
• contributing to the documentation of open source python projects that we use in the tasks above
• contributing code to the python projects

Of course in the early days of the project there were and are many, many things to do. And these things related to the larger fossee projects not just to its python part. And all the team members have been 'volunteered' to do those.
Building a website was one such task. We had expected to hire a good web developer in the first batch but it did not happen. So some members volunteered to do that. I asked that we do a website in Plone as I have some plans for using Plone to build an app around the dissemination, marshalling and publishing of the content we will be producing.
Unfortunately in the time available to us, remember we had to get the courses designed, developed etc, we could not develop the site using plone. Of course it has nothing to do with the tool. Since PyCon was looming large I decided to let the site be done in Drupal. And we essentially did a hurried job without even clearly separating the python.fossee.in and the main fossee.in
Now to sum up:

• Was Drupal our first choice? No
• Are we committed to drupal? No
• Do we plan to change? Yes
• When? As soon as we get the python framework person in
• What tool? It will be what our resident expert to be hired is good in

Before I sign off

• We are hiring! So if you are good in django/zope/plone/tg/pylons/yopf do get in touch. pasokan { AT } fossee.in will reach me
• There was a comment of how Py is SciPy. Prabhu Ramachandran will be writing a detailed blog on the use of Python in Scientfic Computing

We have official FOSSEE t-shirts now! So, yay!

We are about 2 months into the project - baby steps leading hopefully to humongous leaps soon. And what better way to celebrate, spread joy, and share our enthusiasm, than through ... t-shirts? :)

We are serious. We are here to stay. We are a fun team. We like geek humor. The t-shirt embodies all these. Have a look for yourself (image is attached to this post) and tell us what you think.

For now, the t-shirts are designed in two colors - black, and white. The image is that of the white sample (pardon the stating of the obvious :)

They will soon be up for grabs in conferences and workshops near you, apart from any project tasks you contribute to. We distributed about 70 t-shirts at PyCon India (which was held in IISc Bengaluru, recently) and we ran out of them in a matter of minutes, like water in a picnic (so to speak :)!

So, cheers and let thy cup runneth over!

This all started with training programme for teachers of Amravati University, PR(aka Prabhu) asked, "Can we have a LiveCD kind of setup with all Python goodies packaged together?". The motivation was to have a handy DVD for conducting workshops in better way. Rather then fixing systems with all packages for all participants, lets have this, which can be booted directly. And once they are comfortable, they can install also.
I had already done similar kind of work previously at Sarai. I was very much impressed by working and ease of Ubuntu customization. Documentation available is really well written. Following it, one can roll out one's own Ubuntu release in no time. Though there are GUI tools available, I was more comfortable with this approach. After following basics I wrote a simple script with all commands to make things very fast.

The biggest challenge was to satisfy PR, once I presented him the first "in-house" DVD on Virtual Box, he got really excited. Then started deep hacking of the DVD to fit in various tools and latest release of packages. It was really extensive and feed back and requirements from PR were very frequent. We used to sit together and hack this media for hours.
Some of the beasts we put in this DVD were sage binary(at present 32 bit) 4.1.1, latest Mayavi-3.3.0, entire ets(Enthought Tool Suit), documentations for respective packages and lot more. Complete list is available here.
After this we had really long bike-shedding on how to package this, how to name different versions. And Madhu, PR and Vattam came up with awesome idea of using "zen of python" quotes per release. Then we had punchagan who created this great DVD cover:

But still we were missing something. It needed more hacks. We had all packages but they were all hidden behind scary but awesome linux terminal. Hence we thought of introducing a new menu entry. We created a new /usr/share/desktop-directories/Python.directory file, and added entries to /usr/share/applications/ to come up with:

One more addition was adding videos to DVD itself, so that it is a complete bundle. One wants to get started with Python, there are 8 hours videos bundled inside DVD itself. For that we tweaked /etc/skel/examples.desktop :)

PS: people were asking me to write a blog, and here I ended up with a "ramayan."

We are excited to announce that the first "Scientific Computing with Python" conference in India (http://scipy.in) will be held from December 12th to 17th, 2009 at the Technopark in Trivandrum, Kerala, India.
The theme of the conference will be "Scientific Python in Action" with respect to application and teaching. We are pleased to have Travis Oliphant, the creator and lead developer of numpy as the keynote speaker.
Here is a rough schedule of the conference:
Sat. Dec. 12 (conference)
Sun. Dec. 13 (conference)
Mon. Dec. 14 (tutorials)
Tues. Dec. 15 (tutorials)
Wed. Dec. 16 (sprint)
Thu. Dec. 17 (sprint)
The tutorial sessions will have two tracks, one specifically for teachers and one for the general public.
There are no registration fees.
Please register at:
http://scipy.in
The call for papers will be announced soon.
This conference is organized by the FOSSEE project funded by the Ministry of Human Resources and Development's National Mission on Education (NME) through Information and Communication Technology (ICT) jointly with SPACE-Kerala.

You might be wondering why we are using drupal when we speak so much about Python. I shall enunciate the reasons for the same in this post.

When we started off with the website we had the option of choosing among the various available CMSs namely, Plone, Drupal, Joomla and so on. There was also the option of going with Django, which we discarded because most of our site would be static and we thought it would be too much work to do for a static site. We chose Plone, the reasons being, its written in Python and we had seen quite a few websites that ran plone like plone.org and Connexions. It took us a couple of days to figure out stuff and get the basic site up and running using Plone. Thats when the problems started. Maintaining a plone site a great pain. Firstly, Plone is a beast! Its a package of 34MiB which comes with its own Python2.4 and Zope server. Installing Plone is not a difficult task I must say. But the problem starts only after the installation. Every time a new module has to be installed, the module tarball has to be downloaded and extracted to the Plone src folder, then the buildout.cfg has to be edited, which in itself is a great pain and then the buildout script has to be run. Then the zope server has to be restarted. All this just to make the module appear on the Plone administration set up page. The installation is complete only after the module is selected for installation on the set up page. The plone docs say that the module should get installed by just specifying the url to the module tarball if they are under pypy or plone itself. But this procedure doesn't work most of the times. The documentation for most of the modules is sometimes misleading and completely non-existent most of the times. Also most modules that might come close to being useful are usually in the alpha or the beta releases and hence are buggy and lack support. If some of you have visited this site before, then you would've experienced the messy and the cluttered interface. Also adding content on to the site was non-intuitive to say the least and a huge pain in the backside to say the most. Hence we decided to do something about it.

We decided to work on improving the plone site and if it was completely impossible to do it then ditch plone and take up alternatives. So Shantanu, PR and me started looking for alternatives. In the meanwhile I did try to tweak the plone site a little but I was more inclined on trying something better. We looked at Pinax and Django-cms. Pinax is too social-networking-site-ish and we felt that the django-cms installation procedure was too cumbersome. So we reviewed our options and considered drupal as well as wordpress this time. Since
I had some experience with drupal, I suggested that we go with drupal, since I had some experience with its working. I worked for a couple of hours to set up a basic site on my local machine, just as a demo and PR was quite impressed with it. Shantanu and Puneeth provided valuable input on the necessities of the website and its basic design. Once this was set we started out with the actual website. PR and Asokan gave their reviews with inputs about what they want from the site. Thus we went on to develop the website

Working with drupal is very simple and instinctive. Drupal provides an interface that is uncluttered and quite uncomplicated to navigate through. Also drupal has a whole suite of modules that provide scale the functionality of a website to great levels. Also it has an amazing set of beautiful, eye-catchy themes. And the best part is that configuring all these modules and themes is straight forward and doesn't involve complex stuff like editing configuration files remotely or running complex scripts. Its all done on the browser. Also drupal has amazing support in the form of simple, easy-to-understand documentation as well as an amazing community with people always ready to help on #drupal-support. All these factors led us to choose drupal.

Don't tell me I didn't warn you, if you walk into our lab sometime and mistake it for a game zone in your neighbourhood. You are indeed in "the" lab. If you came looking for Prof. Prabhu you'll have to return. You'll only find PR vowing to see the end of baali, for having a nice time pawning him. You won't find me around - I'm cloaked and busy creating havoc. Unless, of course, you are the seer KD. Watch your step extremely carefully or you won't know when and how fuchmed's GM hit you. And don't dare venture anywhere near madrazr, jumping all around with a shockwave, unless you are capable of enjoying some real nasty shocks. Well, the team is busy in one of it's two SB sessions per day. SB (short for Stress Buster) sessions are intense 15 minute rounds of bzflag - 3D first person tank battle game.
Hope you didn't start thinking, I'm just goofing off here, doing nothing at all. If it's not already clear, SB sessions are exactly what they are called - Stress Busters. Just to refresh ourselves and get back to work, with greater force. What then, am I doing for the rest of my time?
Mainly I've been developing course content that we have been working on, called Software Tools, Techniques and Practices. I have been working on a session for elementary LaTeX and basic Linux tools. I've also been attending classes of Digital Control and trying to Python-ize the Matlab/Scilab code that the course uses.
But, that's not all. I've helped a guy restore a Joomla site on Day-1. I screwed up a brand new installation of a server with some real skill; then expect to be screwed but get some encouragement with the words, "It was a good learning experience for us." Seen a Debian server upgrade from Etch to Lenny in under 10 minutes. Installed an instance of twiki and struggled to configure it. Helped a Humanities Scholar with LaTeX. Attended a workshop on Instruction Design. Restored Qmail on an old Fedora 3 server. Goofed around in gimp, for a CD cover and a T-shirt design. Chipped in with bits, for mutating Ubuntu into "LivePython". Did a bit of css and javascript tweaking... The list goes on!
Now, don't you think we deserve the SB's? ;)

In order to enable Clean URLs module on Drupal, an Apache server module named mod_rewrite has to be enabled. The following is the procedure to do it on Ubuntu which has LAMP installed using tasksel.

The mod_rewrite on Ubuntu LAMP is not enabled by default. It can be enabled by running the following command on the terminal.

sudo a2enmod rewrite

This will enable the mod_rewrite module on Apache. Now restart Apache server by running.

sudo /etc/init.d/apache2 restart

Now navigate to the /etc/apache2/mods-enabled directory and list the files. This directory will contain a file named rewrite.load. This means that the module has been installed properly.

Now open the file /etc/apache2/sites-enabled/000-default using your favorite editor.

To the file add the following lines:

Options Indexes FollowSymLinks MultiViews
AllowOverride All
Order allow,deny
allow from all

Now restart apache as stated earlier. This should enable the Clean URLs on Drupal. You can check by going to the Administration-> Site configuration-> Clean URLs page.

At the end of a meeting, I was only a contact point/person (the guy responsible for X, the guy to be questioned/queried regarding X) for setting up a mail system and mailing lists on the server. But slowly, I ended up being the guy who should set it up. And for a guy, who has only used mailman from the user end and didn't even know what Postfix meant, setting up Postfix and mailman for two different domains, is a challenge (if not anything, more). I struggled quite a bit with it, but finally managed a bare-bones system.
A simple Goo(gle)-search does throw up a hazaar blog posts and how-tos on how to get it done. And that probably is the trouble ;). Hundreds of posts on doing it for a single domain and a handful for doing it for Multiple domains. But, I didn't really find anything comprehensive, if I can say so. This post is a pretty comprehensive on on how to setup Postfix for a single domain (on Debian). I first followed this and got a working Postfix MTA. Then, I set-up a virtual domain on Postfix. The README has it all.
I only had to make a few changes, to add the virtual domain. You just need to add the virtual_alias_domains and add virtual_alias_maps. Here is a glimpse of how our main.cf looks:

myhostname = mail.abc.in
mydomain = abc.in
mydestination = $myhostname, $mydomain, localhost.$mydomain
myorigin = /etc/mailname
virtual_alias_maps =
hash:/etc/postfix/virtual-xyz,
hash:/etc/postfix/virtual-abc,
hash:/var/lib/mailman/data/virtual-mailman
virtual_alias_domains = xyz.in, mail.xyz.in

virtual-xyz and virtual-abc contain the virtual aliases for each domain. Note that Postfix expects hash files and postmap /etc/postfix/virtual-abc needs to be run, each time the file is changed, to keep the hash file in sync with the text file. Also, postfix reload needs to be run, each time changes are made either to main.cf or master.cf.
And now, Mailman. Well the Installation Manual of Mailman, literally has it all. Here is a look at how our mm_cfg.py looks:

DEFAULT_EMAIL_HOST = 'abc.in'
DEFAULT_URL_HOST = 'mail.abc.in'
add_virtualhost(DEFAULT_URL_HOST, DEFAULT_EMAIL_HOST)
POSTFIX_STYLE_VIRTUAL_DOMAINS=['xyz.in']
add_virtualhost('mail.xyz.in', 'xyz.in')
MTA='Postfix'

Sweet and simple. But, we wanted to run lists with the same names on both the domains. This apparently isn't easy (and I didn't get this working). I experimented with multiple mailman instances but that didn't work, because Postfix (the MTA) can't differentiate between the two instances of mailman. The alternative is "probably" to have two instances of Postfix running too. But, that I felt, was like going too far. I settled with one mailman instance and different list names on each of the domains.
All this is pretty straight forward, right? But since I knew nought about MTAs and mailing lists, it took me quite sometime to get all this in place.
[Well, thanks to Shantanu and Vattam for the inputs; and all the people who cared enough to document their experiences with Mailman and Postfix]