December 25, 2014

Discovered the killer combo of Pysqlite and SQLObject

light bulb yellow Discovered the killer combo of Pysqlite and SQLObjectAt work, I’m doing a bunch of data acquisition on various machines outside the firewall. I think I’ve hit on an absolutely great combination of Python technologies to do this in a really manageable and scalable manner.

The design constraints:

  • No long-running processes. These are heavily loaded boxes. The data acquisition will run via scheduled cron jobs.
  • No new ports open. I can’t run Mysql on these boxes.
  • No drilling back from these servers into the corporate network. So I can’t run mysql inside and connect from the outside.

The plan

  • Each remote box will run a small python app on a scheduled basis, writing its data to a single file.
  • A "gatherer" system inside the firewall will retrieve the files from all remote machines on an hourly basis.
  • On acquisition, the remote files will be removed.
  • The gatherer process will then merge the remote data into a master Mysql database for reporting.

The technologies

  • PySqlite: will provide access to an on-the-fly generated database file on remote machines.
  • SQLObject: can talk to both PySqlite and Mysql. I simply define the object model using its scheme, and it will happily generate the tables for me in both Sqlite and Mysql. Writing the merge process was the trickiest bit, but it only took a few hours to write and test.


This was an extremely fast development path. I was all the way from concept to deployment in just three days, and I didn’t even know anything about SQLObject or Sqlite at the start.

Contrast that productivity to Java, where a similar project would take at the minimum a couple weeks, and you start to see why I think that Python has become the secret weapon in my developer toolbox.


Why am I writing an ecommerce engine from scratch?

shopping cart Why am I writing an ecommerce engine from scratch?Sometimes I have to remind myself why I am writing a brand-new shopping cart system. It seems on the face of it to be an incredibly foolish thing to do.

I know.

The thing is, I am determined to use modern, systematic approaches to selling, as described in “Call to Action: Secret Formulas to Improve Online Results” (Bryan Eisenberg, Jeffery Eisenberg). I need the flexibility to experiment with the systems, I need the explicit knowledge about how my system works and why.

I cannot be held-up, waiting for the “next new release”, available “real soon now” of this critical component of my store. Also, during this experiment, I can’t sit there and grit my teeth poking at horrible, ugly PHP code, guessing and hoping that I am changing the right thing.

I was going to use Zen Cart, but its latest release has been stuck for far too long, and it can’t even do the very first experiment I want to try, which is to try not requiring registration before purchase.

I actually don’t think it will be so hard.

I’m using Django, so I’m not writing any admin pages at all. What I do write is working very quickly, and I trust it. That’s key for me. I can sit down in an evening and hammer out a new functional app.

I’ve broken down the functionality I want into manageable chunks, small apps that work together. With focus, I am usually able to put a working app together in a remarkably short time, with non-trivial functionality.

For example, I wrote my email auto-responder subscription system in just two evenings. It’s full-featured, too. It has multi-level fallback templates, and generic templates as well. With this system, I can manage an unlimited number of simultaneous email campaigns, with customized signup templates, across an equally unlimited number of domain names. It figures out everything it needs to keep the campaigns separate, and tracks all the click sources as well as abandonment rate.

Now I’m working on the catalog. I got all the model work done in just a couple hours. The templates will be built using my fallback mechanism, so any product will have a default template, but can be customized independently by product line, by product, or by hosting site. See, the apps are additive, and the system is so clean that they work together nearly seamlessly.

Next will be the cart itself, then the gateway. Add some shell templates and some design work and I’ve got a cart system!

Yes, I am insane, but I am an insane guy who knows exactly what he wants and who cannot find it anywhere else.

[tags]django,shopping cart[/tags]

Fresh design examples

searching magnify Fresh design examplesWeb Design From Scratch has a nice, well illustrated article on Current Style in Web Design.

Of course, I don’t want to advocate always running with the herd, but in this case I’d argue that it isn’t so much the herd, but more of a pack. OK, so that isn’t such a great metaphor. What I’m trying to say is that I think the sites they mention exemplify some of the best, freshest sites on the net, and that I personally take a lot of inspiration from them.

I think the first thing I’ll do from this list is to lighten up my various sites templates a bit, making the page less “weighty”:

The great sites above share the following design features:

  • Simple layout
  • 3D effects, used sparingly
  • Soft, neutral background colours
  • Strong colour, used sparingly
  • Cute icons, used sparingly
  • Plenty of whitespace
  • Nice big text

[tags]design,layout,web design[/tags]

Website Usability Guidelines

Smart Money Daily points out a great resource for website usability guidelines. Created by the US national cancer institute, it gives more than simple “give img tags ‘alt’ attributes’. Instead, it goes into why you should do so, and even more importantly, what evidence there is that this is an important guideline for you to follow.
[tags]usability, accessibility, section 508[/tags]