May 17, 2012

Django on Rimuhosting

thumbs up Django on RimuhostingThis is just a quick post to confirm that Django does indeed work perfectly on Rimuhosting with very little effort required to set up. It is fast, it connected to my Mysql db without a peep of a complaint, and it is making me a happy developer.

Setup details

I’m using Rimu’s “MiroVPS3″ plan. It is $39 a month, less 10% for being an open-source developer. It provides 4G hard drive space, 60G bandwidth (in and out combined), and 160 MB ram.

I’m running the following “stack”:

  • Debian: I know and love Debian much more than any other Linux distro. Almost everything is installable with a simple “aptitude install whatever” command.
  • APF: firewall, set up according (mostly) to the instructions on Webhostgear. I had to tweak it for Debian, by setting “monokern=1″ and by symlinking /etc/rc.d/init.d to /etc/init.d
  • Imap/Smtp set up according to the instructions on The Perfect Rails/Debian/LightHttpd Stack article.
  • Mysql: Actually not my favorite db, but I didn’t want to convert to PostgreSQL at the moment, at least not until I’d proven my existing dev setup worked. Rimu set this up for me, since I’d requested a Django setup.
  • PIL: I need the Python imaging library, so “aptitude install python-imaging” did the trick.
  • python_mysqldb: Another simple “aptitude install python-mysqldb”
  • Apache2 with Mod_Python: Apache2 is more flexible about vhosts, and I’m going to use the heck out of them. Mod_python is the preferred deployment environment. I am not using any other mod_plugins. Rimu set this up, and I made vhosts entries based on what I read in the django documentation. The only “tricky” bit was setting up a different vhost for the media URLs. I did that one without mod_python support, to make for lighter server threads.

It is a very responsive environment. I can hardly wait to deploy some real apps on it very soon.

[tags]django,rimuhosting,debian,vps setup,python webhosting[/tags]

Django and authenticated smtp

handshake Django and authenticated smtpI just sent off my first patch for Django. It adds the ability to send mail via SMTP servers which require authentication. I need this for Dreamhost, which has authentication on its mail servers.

The modifications were simple, and the code I was modifying was refreshingly clean. Better yet, the Django project site gives easy and clear instructions about how to contribute. So, I’m feeling very good about the project, the ease of contributing, and the maturity of the code.

Update (next day): They accepted the patch and merged it into the trunk of the dev version. I updated SVN and it is all there. That is the fastest response ever for one of my patches to be accepted into a public open-source project!

[tags]django,dreamhost[/tags]

XML Is Java's Scripting Language

troll XML Is Java's Scripting LanguageGreat article at Lesscode.org, responding to Gosling’s dismissal of scripting languages.

Usually, I ignore attacks like Gosling’s as not worth my limited time and attention. I love Python, and it really doesn’t bother me when the ignorant attack my language-of-choice. This article is better than the original by orders of magnitude, due to its heavy linking and its excellent reader commentary.

My favorite quote from the article is: "this is why XML is used so heavily in Java for things like O/R mapping, dependency injection, configuration, build processes, etc. — XML is Java’s scripting language". It is actually from a reader comment, which is worth quoting in full.

But Ruby, Python, Perl, Smalltalk, etc. are not specialized scripting languages. They are general purpose programming languages. I believe one of the reasons the term “scripting language” has stuck with them is partially because it’s become so common to create mini-languages (i.e. DSLs) in dynamic languages that are specialized for certain tasks. So it’s sometimes hard to separate the specialized pieces from the general purpose pieces. Contrast that with static languages, where you are forced to create a completely separate “scripting language” when you need to provide a constrained/specialized language (this is why XML is used so heavily in Java for things like O/R mapping, dependency injection, configuration, build processes, etc. — XML is Java’s scripting language). In dynamic languages, you are generally not required to leave the core language to get that specialization: you use techniques like meta-programming and DSLs or just include/require/use some designated file at a designated point and let it do it’s thing with the API.

First use of the Django Magic Removal

guitar First use of the Django Magic RemovalLast night I took a break from my most recent Django app, a mailing-list manager, to explore the new "Magic Removal Branch".

I diligently followed the instructions on the Django Wiki to change my existing code. There were more changes needed than I’d anticipated, but they all made sense, so I didn’t mind. Better, I agreed philosophically with all of them.

Basically, the idea is to make Django apps act more like normal Python modules. That means ceasing all the “magic”, where it used to automatically provide entire new classes to manage objects, and used to magically move around the modules, making them available in namespaces you hadn’t explicitly named. I found that confusing, to be honest. I always had to look at example code to find out where the magic stuff was going to be. The magic, which was supposed to be helpful, actually ended up costing me time and thought.

Roadblock 1 – Database Init

Unfortunately, I’m not all the way there yet in migrating my code. First, I ran into a significant problem with database initialization. That is, it didn’t work.

$ bin/admin.py init

File "/sw/lib/python2.4/site-packages/django/core/management.py", line 498, in init
contenttypes_app = models.get_app('contenttypes')

File "/sw/lib/python2.4/site-packages/django/db/models/loading.py", line 30, in get_app
raise ImproperlyConfigured, "App with label %s could not be found" % app_label

ImproperlyConfigured: App with label contenttypes could not be found

OK, I restored the old database. It was a good thing I backed up before trying this! I then executed the SQL recommended on the Wiki to convert an existing install to the new branch. That mostly worked, but complained that a few tables were missing. “django_flatpages”? Never used that one, I assume it was a contrib module.

Roadblock 2 – code changes

I think I’ve converted all the existing code correctly, but the server dies on start. Apparently, I did not convert my old "”Class META" to the new "Class Admin" correctly. I’m sure I’ll knock that problem down the next time I have a moment to tackle the conversion. Unfortunately, I ran out of time, exhausting the one hour I had left before “The Daily Show” last night.

Conclusions

Absolutely worth the pain. It will work with a little more tweaking. These are normal pains to experience when using a pre-release, actively developed branch.

Update: The contenttypes error is corrected by a patch available from bug #1280.
[tags]django,python,magic removal[/tags]

IPython brings joy

jump joy IPython brings joyAfter my rant yesterday, it is time to share another programming tool I find absolutely indispensable. IPython is an enhanced command-line shell for Python.

By itself, that may not sound like much, but you really get a lot. You get command history, tab-completion, and automatic introspection of classes. That means you can look up the specific code any function will call, at any time. It has all sorts of automatic and “magic” things it can do, all of which make it a must-try for the working Python programmer.

Yes, but what has it done for me lately?

Since installing IPython last year, I find myself using it for all sorts of tiny tasks which I would have done in a more fragile, cumbersome, or laborious way in the past. Last week, I needed to insert 50,000 records into a database. They were very simple insertions, so I could have just run an Emacs macro to do make all my Sql. But now with IPython, I just pop into a shell, load the unformatted records from a CSV file, and build the SQL in three lines or less of code. Sweet!.

Today, I needed to find the missing item in the haystack. I had two lists, one text and one from a database. The database was missing one of the items from the list. But which one? That was a trivial thing to conquer using a little 1 minute IPython session. Like so:

In [1]: f = open('cc_tables','r')
In [2]: tables = f.readlines()
In [3]: f.close()
In [4]: f = open('cc_list','r')
In [5]: dump = f.readlines()
In [6]: f.close()
In [7]: len(tables)
Out[7]: 104
In [8]: len(dump)
Out[8]: 103
In [9]: [x for x in tables if x not in dump]
Out[9]: ['DataExtractionStatus\n']

The answer, in 9 lines, 4 of which were unneeded, was “DataExtractionStatus”. I could’ve done that in OpenOffice Calc, but the thing would have still been loading in the time it took me to bang out the answer with IPython. Thank you IPython, for making my job that much easier and faster.

[tags]Ipython,python[/tags]

Another win for Django

thumbs up Another win for DjangoOne nice thing about Django is that unlike many frameworks, you do not have to “grok” the whole thing to be productive. In my case, I hadn’t read a lot of the documentation other than the tutorial and the details of a few of the features I need for my first app.

Surprise!

Imagine my happy surprise when I found the very nicely implemented “documentation” link in the automatically generated admin site. I’d assumed it was a link to the static documentation or some such thing, like it is in most systems. Nope. It is much more.

It has details and documentation on everything I’ve done on the site. I habitually put in docstrings, so they are formatted for me and presented on the detail pages for my model objects. Wow. This is automatic, and a big incentive to keep doing that documentation! Every view is similarly detailed. In fact, there are auto-generated pages for Tags, Filters, Models, and Views. In each case, the documentation only shows things valid for the application. If you’ve added custom tags, they are there, along with all the built-in ones.

I hate to rave, but this level of detail and usefulness is unprecedented in my professional experience. Especially in a free and open programming framework.

[tags]Django,documentation,raves,python[/tags]

Top ten reasons to use Django for your web framework

guitar Top ten reasons to use Django for your web frameworkDjango is pulling ahead of every other web framework I’ve tried. For reference, that includes such recent heavy-hitters as: Jakarta Struts/Tiles, Spring, PHP Smarty, Zope, and Plone.

I’d been thinking about why that was so, why I’ve become so fond of Django so quickly, when I ran across a post at Jacobian.org, titled Why you should use Django. I’ll take his top ten points and add my own observations:

1. Django works — right now

It really does. The example tutorial works correctly the first time. That is something hard to find in the online world. Better, there is no outstanding wart which is holding up adoption, and which will be fixed “real soon now”.

2. Deployment is a breeze

I know that I will be able to deploy on DreamHost, which is the most important part to me as a user at this time. As a developer, it was amazingly simple to set up and run my development system. This took literally hours less time than setting up an equivalent Java system.

3. Your site won’t go down

That’s one of Django’s big claims, and is part of the reason I was initially attracted to the framework. It speaks of quality to me.

4. Django is fast

It is faster by far than Kid, and Cheetah. If it beats those, then it literally blows Zope out of the water. PHP should eat its dust as well. But that is all beside the point to me at this point. I care about “fast to develop for”, and that is where it truly shines for me.

5. Django scales

Not qualified to judge.

6. Django’s admin will blow your mind

I hate writing admin interfaces. Django removes almost all the pain for me. In fact, they are almost free, which is a wonderful and deeply appreciated thing. How many times have I released a half-ass admin interface “for trained users only” to justify the fact that using it might cripple the site if the wrong button were clicked? More times than I like to think. Django won’t prevent all of that, but it gives me back so much time that I can then use to put in better logic-checking and validation.

7. You don’t need to go whole-hog

I am preparing to port Invisible Castle to Django, and I’ll report back about my progress. I expect it to be fairly easy, actually.

8. Django plays well with designers

I’m not qualified to judge, I think he’s probably right

9. Django comes from the Real World

I love the fact that they don’t add functionality just because it looks fun. The best projects/software in the world were built to solve real problems, and more importantly had real users right from the get-go. Django was built exactly that way, with immediate feedback and guidance from their sponsor, and it has clearly benefited from that.

10. We’re here to help

My post about logging in frameworks which referenced Django was picked-up and responded to within a day. That’s amazing. What’s more, the result is that they may very well adopt what I am suggesting. It wasn’t a flame-fest, despite my initial harsh tone. The professionalism and openness both surprised and pleased me.

[tags]python,django,webframeworks,web frameworks[/tags]

How to install Cheetah on Dreamhost

cheetah How to install Cheetah on DreamhostInstalling Cheetah on Dreamhost’s shared servers is actually a snap, if you have access to a Linux development system. All you have to realize is that the servers are Intel-based Linux systems, Python setup utilities know how to build “dumb” distribution, and that python doesn’t care if you use symlinks to trick it.

Building Cheetah

  1. Skip this step, if you like, by downloading my precompiled Dreamhost-friendly version.
    Download: Cheetah-1.0.linux-i686.tar.gz
    (MD5 Signature: 5e64240ae6dd7eb0ffe92fdde61fe5ea)
  2. To build, go to a Linux computer, preferably Debian, with Python 2.3 installed.
  3. Download Cheetah. I’ve tested this with 1.0. I don’t know about 2.0, probably the same steps would work.
  4. Unpack Cheetah in a temp dir.
    tar xzf Cheetah-1.0.tar.gz
  5. Compile a “dumb distribution”
    cd Cheetah-1.0
    python setup.py bdist_dumb --relative
  6. Copy the compiled distribution to Dreamhost. You’ll find it at “dist/Cheetah-1.0.linux-i686.tar.gz”

Installing Cheetah

  1. Unpack the compiled Cheetah package in some directory. I’ll assume you are going to unpack it in your home directory.
    tar xzf Cheetah-1.0.linux-i686.tar.gz
  2. It will unpack into two directories.
    • lib/python2.3/site-packages/Cheetah — the compiled site packages
    • bin — Cheetah compiler binary
  3. Now, you need to edit your ~/.bash_profile and add a couple lines:
    PATH=$PATH:$HOME/bin
    PYTHONPATH=$PYTHONPATH:$HOME/lib/python2.3/site-packages
    export PATH PYTHONPATH
  4. Now, add the environment variables to your current session.
    source /.bash_profile
  5. At this point, Cheetah should work! Congratulations!
  6. If it compiles all right, but doesn’t work via the web, then you need to make sure your Cheetah dir is available from your web-context pythonpath. The easiest, hackiest, fastest way to do this is to simply make a symlink to Cheetah. This fools Python into seeing the “Cheetah” module in the same place it sees your project files. My domain has its Python cgi files right in the root, so I just did it like this:
    cd ~/mydomain.com
    ln -s ~/lib/python2.3/site-packages/Cheetah .

Please let me know how this works for you, I’ll attempt to help you out if you run into problems.

[tags]python,dreamhost,cheetah,cheetahtemplates,howto[/tags]

Logging is good for frameworks too

log Logging is good for frameworks tooOne thing I just don’t understand is why more people, especially framework designers, omit or skip logging. Django appears to, which is incredibly annoying to me. It is a complex framework, which makes all kinds of assumptions and relies on convention to infer a lot of functionality. That’s great, but being able to “turn up” the log-level and see some discussion about what decisions it is making, based on what criteria it is using would be incredibly valuable to me. Not only to me, I wager, but to everyone who is learning the framework, or who is teasing out a difficult bug. Yet going in and adding framework-level logging is almost impossible, even on an open-source project, since your changes will not be accepted and will quickly become a merge nightmare.

Don’t believe me about changes “not being accepted”? Here’s proof. (That’s not my bug, by the way, just proof). The developers closed the ticket saying they “fail to see what this gains Django”. Foolish and shortsighted.

Logging is preferred by most senior programmers over line-by-line code stepping. It is both more efficient, and allows for post-mortem analysis of problems in a way that debuggers never will. Yet many act as if it is a nice-to-have, or worse an irrelevant distraction.

Shortsightedness is deadly

I once worked on a team with a programmer who refused to add any logging to a super-awesome-framework he’d written for Java. Evidently the framework was the solution to everything we needed. I challenged him in a team-meeting about his refusal, talking about the numerous benefits of good logging throughout a system. He replied “I don’t write bugs, so I don’t need to waste time logging to find them”. I had him fired, and good riddance, within a week. Attitudes like that have no place on a software team. It was his foolish assertion that he didn’t write bugs that was the final nail, but his refusal to see the benefits of logging certainly didn’t help his case any.

Logging is built-in to Python, and is a commonly available module for Java. It is high time to make this practice part of generally accepted programming standards for frameworks. Don’t want a log? Turn it off. But you should always be able to get one. Always.
[tags]logging,java,python,django,rants[/tags]

On Closures

Recently, I’ve been something of a language evangelist at work. I’ve been promoting the power and utility of Python and Javascript to both coworkers and management. One thing I’ve found difficult to get across to them is the concept of "Closures", and why the lack of Closures in many mainstream languages (such as Java) is an irritating, slowing, nuisance.

I use them all the time, yet it is not so simple to explain why they are, to phrase it the way my son would, "So Sweet". I would go so far as to say that if you aren’t using closures all over the place in Javascript you probably dislike Javascript and think it is a toy suitable only for web designers. Seriously, closures are what turned me into a Javascript fan, that and the incredible Prototype library. In Python, closures are natural, yet frightening for so many who come from a Java or C++ background.

This article won’t teach you everything about Closures, and it is written using Ruby as an example, but it may show you why you would want to try them. Thanks, Tom Moertel, and Reddit.

link: Closures and the Professional Programmer