How not to program WSGI
12 May 2008
or how not to serve robots.txt with PyBlosxom
So as you may have noticed, I moved this site from PyBlosxom to Django, which depending on your perspective is a fabulous thing to do or is tantamount to treason on the high seas. I will explain more about that later.
Old links to the site should, in the main, still work hopefully as I have done some regular expressions jujitsu which should hopefully send everyone to where they were supposed to be going.
However, some posts and comments will have their formatting up the creek. So I want the old version of the site to be available (at archive.commandline.org.uk) for a while longer.
Because the archived version is deprecated and on the way out, I do not want the search engines to index it. Therefore I needed to make a robots.txt file for that subdomain excluding them from indexing it.
The last version of this site, like many dynamic sites, is composed of a number of layers, part of which was a lot of my own nonsense code doing various things. Ignoring that, when a request for a packet came in it would go to WSGI which would then pass the request on to PyBlosxom which was at the bottom of it all doing the hard work.
To deploy it properly, one would normally put Apache at the front as well, but I never got around to that. In theory this is a bad thing to do. But in practice it worked really well without the huge and complicated server that is Apache in the mix. It actually ran fine for a year without stopping, and blazing fast too; if it also confused a few comment spam bots then all the merrier.
So I tried putting Apache into the mix so I could use a Location directive to direct /robots.txt to somewhere with the robots.txt file, but no joy, this would have required doing a lot of what I never got around to before.
So I then looked into how the test server was deploying the site, thinking that I could do some kind of smart regular expressions type matching like in Django or Pylons. But nope.
Hack for the win
So the next step down is PyBlosxom, so I looked out of chance in Pyblosxom/pyblosxom.py and saw the following:
def __call__(self, env, start_response):
"""
Runs the WSGI app.
"""
# ensure that PATH_INFO exists. a few plugins break if this is
# missing.
if "PATH_INFO" not in env:
env["PATH_INFO"] = ""
p = PyBlosxom(self.config, env)
p.run()
pyresponse = p.getResponse()
start_response(pyresponse.status, list(pyresponse.headers.items()))
pyresponse.seek(0)
return [pyresponse.read()]
Bingo! As soon as I saw it, I just somehow, on auto pilot, typed in the following lines before the line p = PyBlosxom(self.config, env):
if env["PATH_INFO"] == "/robots.txt":
start_response('200 OK', [('Content-type','text/plain')])
return ["""User-agent: * \nDisallow: /"""]
And unbelievably it worked. What I had subconsciously done was to see that we have some kind of string referred to by env["PATH_INFO"]. Then further on we have an object called start_response which is being passed a status and some headers. Then we are returning the response.
I was kidding around more than anything so I just replaced everything I didn't know about with reasonable looking constants (you will know these well if you have ever done Python CGI programming).
I am sure there are millions of far better ways to serve robots.txt with PyBlosxom. But this hack works for me until I no longer need the old site anymore.
1 John Reese says...
The new site looks good. Just thought I'd stop by long enough to say that. Cheers
Posted at 3:59 a.m. on May 13, 2008
2 Ryan says...
Yeah, good layout too. Very clear. :) Better than the last, in fact! I'm another python/django nerd, so I'll be listening even more now. I guess one of the things that's inspiring about Django is they're concerned pretty hardcore with security fixes. Just this week, an email came out and they released new sub-versions for each major Django release to include the fix. Very awesome.
For your blog post model, what did you do for entering posts? Do you still use the default admin interface, or did you make your own views for posting and whatnot? I haven't looked into it much, but does django automatically include much in the way of wysiwyg text editors for text fields?
Posted at 5:28 p.m. on May 15, 2008
3 dbr says...
I concur with the other two comments - this is one of the nicer blog'y site layouts I've seen. The comment system is also actually pleasant to use, unlike every single other one I've (not)-used \o/ One slight bug, you need to enter two backslashes to make it visible.
Anyway.. This hack seems an extremely convoluted and bad way to serve a simple text file, does it not? I've not used Pyblosxom, but it seems insane that it doesn't allow you to serve up static files (specifically /robots.txt and /favicon.ico)..?
As much as I now dislike PHP compared to Python, this does reaffirm my decision to stick with PHP for web-applications - no web-framework has gotten near the simplicity of shoving an index.php file in htdocs/
Posted at 2:53 p.m. on May 16, 2008
4 Zeth says...
Hi guys, thanks for your comments. I deal with them in more depth in a post that I will publish shortly.
However, one small thing; dbr, I really advice against inferring anything from this post. You are completely right that the hack is extremely convoluted and a crazy way to serve a text file.
It is crazy because of me, not because of the software, but because I am bulldog and I won't let go until it is dead!:
This blog has gone from Blogger to Wordpress to my owned hacked Pyblosxom to Django. No other formatted text has gone this way so it is not really something you can learn positive lessons from. Most other people would have used Apache so you get that to serve your robots file.
When moving from Wordpress to Pyblosxom, I made it easy on myself by using a Wordpress style of pseudo-HTML as my markup format in Pyblosxom.
If I had a time machine then I would have used something XML-compatible (such as ReStructuredText) instead.
I still like Pyblosxom, it is a really nice way to get a simple blog up, and I still prefer it to any other pre-made blog application.
But the way I want to develop the site going forward involves a bit more freedom, and for that I want a real web framework. Django is by far my favourite so it is natural that I use that.
Posted at 4:37 p.m. on May 16, 2008