A blog about one guys quest to learn Python, Django and Google App Engine.
Feed: http://appengineguy.com/atom.xml

söndag 28 december 2008

HTTP response was too large

In case you run into this error:
HTTP response was too large: 3410251. The limit is: 1048576.
(The number 3410251 above might vary completely arbitrarily)
Solutions like this won't help you, because you know for a fact that you have no file that large or anything that could reasonably cause it, then it might be because you have a maximum recursion depth reached error. This causes an enourmous stack trace which will exceed Google App Engines maxiumum response length (1MB). Try sweeping your code for any place where infinite recursion might occur.

måndag 18 augusti 2008

Serving static files with Django and AWS - going fast on a budget

Found this little gem linked from TechCrunch today.

Serving static files with Django and AWS - going fast on a budget

Speed matters. When Google tried adding 20 additional results to their search pages, traffic dropped by 20%. The reason? Page generation took an extra .5 seconds. This article will show how Eventseer utilizes an often overlooked way of improving the responsiveness of a web application: Pre-generating and serving static files instead of dynamic pages.

måndag 11 augusti 2008

New Django Helper

I've been way too hard at work on SlurpBOX to notice, but the fine App Engine Helper for Django guys have made a new release! See the top item in the release notes? I found that bug! :P Go download.

Release notes
Wed 6 August 2008

This is the last version of the Google App Engine Helper for Django that will support Django 0.96. Future development of the helper will be targetted for the upcoming 1.0 release of Django.

  • Improved SDK detection on Windows by looking at both the PATH variable that may be set by the installer and using the win32api module (if available) to look for the SDK in the default Program Files location.
  • Replaced the startapp command with a version that installs an App Engine Compatible application skeleton. Patch contributed by Andi Albrecht.
  • Changed the default runserver port to 8000 to match standard Django behaviour. Path contributed by Waldemar Kornewald
  • Email server settings from the Django settings file are provided to the App Engine Mail API. Patch contributed by Waldemar Kornewald
  • Added support for the Django memcache cache backend. Patch contributed by Jonca Rafal.
  • Added support for the Django session middle with db and cache backends for Django 1.0alpha only. Patches contributed by Jonca Rafal and Waldemar Kornewald
  • Moved the Django compatible login_required decorator to the standard Django location. Patch contributed by Andi Albrecht
  • Replaced the Django ModelForm class with the App Engine ModelForm class
  • Added a repr implementation for the BaseModel class
  • Many minor improvements to increase robustness and avoid errors if portions of Django are not present.

fredag 8 augusti 2008

My first Real App Engine App

Hey folks! Check out my first real App Engine application, SlurpBOX! Please check it out, and offer suggestions on how to improve it!

fredag 1 augusti 2008

Encoding/unicode reminders to myself and others

Encoding is, along with Regular Expressions, one of those subjects I never seem to properly learn. Here are a few pointers to my future self whenever I run into a pesky UnicodeEncodeError.
  1. Read this article for the umtillionth time. This might be the time it "clicks" for you.
  2. Don't get fancy with ISO-8859-1. Just use UTF-8. App Engine hates everything that is UTF-8.
  3. Make sure your text editor saves the files using UTF-8 encoding.
  4. Make sure that the
  5. Put # coding=UTF-8 at the top of every .py file
  6. Put a u in front of every string that contains non-ascii chars. Like this u"Jag är en liten hatt och är bög"
  7. When handling incoming requests, make sure you set the encoding correctly using request.encoding = "UTF-8" in your view
  8. Use ugettext as your alias for _
  9. If a method (such as quote() or hashlib.sha224() requires bytestrings as input, encode your unicode strings first - like this: theunicodestring.encode("utf-8")

tisdag 29 juli 2008

Internationalization problems

Just ran into a few issues when trying to get internationalization working - these are tips I really could have used.

First, you need to create a directory called locale in your django application dir (i.e. the directory where you keep the views.py file).

To create the language files, you need to run django-admin.py makemessages from your django app dir. When I ran this, I got this:

Error message: makemessages is an unknown command on django-admin.py
Took me a while to figure this one out - this can occur if django is not installed - i.e if you just have it in your app engine dir, it won't cut it. You need to run setup.py install to sort it out. Bastard.

You need to have gettext on your system for the localization to work. Unix systems have this per default, but Windows does not, and getting it to work involves a frickin' moon process. This is what you need to do:

  1. Download and unzip gettext into some directory.
  2. Add the /bin subdirectory of that directory to your path.
  3. Download iconv.dll and slap it into the bin dir.
  4. Download intl.dll (I have no idea what this is) and slap it into the bin dir.

Sorry for this jumbled up post - it's written at 2AM after fighting with this for way too long.

tisdag 22 juli 2008

Gist: Hosted version control of code snippets

Okay, this is just seriously cool. GitHub (hosting provider for the open source version control system git) has just launched Gist - a version control system for little snippets of code. I'll be trying it out on my code examples in the blog, henceforth.

onsdag 16 juli 2008

How to do AVG and SUM in Google App Engine Data Store

People who are used to relational databases, which is pretty much every gosh-darned web developer out there, will run into pretty much the same obstacles with the app engine datastore - one of them is How the heck do I do SUM or AVG?. Yeah, due to how the Data Store works - you cannot do any kind of aggregate query. Instead, you have to re-calculate the totals at write time and keep them in a Counter instead. Like this:


And you use it like this...


Performance note: It is a slightly bad idea to do counters like I have done above, but in my defence, I have done it for simplicity. In practice, it's important that you shard your counters.

As always, comments and questions are always welcome!

måndag 30 juni 2008

Proper Unit Testing of App Engine/Django

While there are a couple of things written on unit testing with App Engine, I unfortunately found them all lacking in one way or another. With this post, I'd like to describe how to unit test Django views in App Engine, using as little mocking as possible.

More specifically, I did not want to mock the Data Store Model objects, since that very quickly turns into mocking hell - you just end up spending three times as much time thinking about mocking than you are about code. Fortunately, after quite a bit of digging, I discovered that Google provides a temporary, in-memory datastore stub, specifically made for unit testing! They also provide stubs for mail, urlfetch and the user service. MAN that is nice!

For this example, I am using:

We will be using the Test Runner provided by Django. Django looks for tests in a number of places, one is tests.py - which it expects to be in same folder as models.py and views.py. To have django run your tests you go:

manage.py test nameofYourApp

The code




address.html is basically just "{{ form }}" and some HTML.

There ya go. Post any questions as comments, and I will try to answer them!

lördag 28 juni 2008

Google App Engine: The Book

For some people, this is old news, but I didn't catch it until now. It turns out that Noah Gift (co-author of Python for Unix and Linux) and Mike Orr (an editor of Linux Gazette), are already writing a bloody book on Google App Engine!

It is going to get published by Manning. One really nice thing about Manning is that they gradually release their books in electronic form as they are getting written through their early access program. Anyway, they have a blog where you can follow their progress (and get chapters) - it doesn't look much for the world yet, but it's most likely worth keeping an eye on!

torsdag 26 juni 2008

Unit Tests for Google App Engine

I am a firm believer that if you don't use unit tests, you are being a bad developer.

Read this!

onsdag 25 juni 2008

How to check out the latest version of Django using TortoiseSVN

This is a few pointers on how to get Django 0.97 and the App Engine Django Helper running.

I notice that a lot of people are using the included 0.96 version of Django included with the App Engine. Don't do this, it's stupid, and not recommended by anyone. The 0.96 just doesn't work all that well, it doesn't properly work with the App Engine Django Helper, and lacks a LOT of really nice stuff.

I think that the main reason people don't use the 0.97 version is that it's an unreleased version, which means it cannot be downloaded in a nice zip - instead, you need to check it out from the Django Subversion repository. New developers don't realize how incredibly easy it is to do this - it's almost easier than downloading a zip file.

Anyway, here is how you check out the current development version of Django (as of writing, v0.97), on windows:

  1. Download TortoiseSVN
  2. Install it and restart your computer.
  3. Create a folder called django or something, right click it, and click CheckOut.

  4. Enter http://code.djangoproject.com/svn/django/trunk/ in the URL field and click OK.

  5. Grab a cup of coffee REALLY FUCKING QUICKLY.

  6. All done! You now have Django 0.97!

Installning the Django Helper

As for installing the Django Helper, the introduction is pretty straightforward. There are three issues that you need to be aware of though:

1. You must remove a few unnecessary files from django to get below the 1000 file limit of App Engine. Waste these directories:

  • django/bin
  • django/contrib/admin
  • django/contrib/databrowse

(DON'T delete django/contrib/auth like the Django Helper inroduction says. It's actually needed.)

2. VERY IMPORTANT if you have a non-english operating system:
There is a very annoying bug in the Django helper r30 that causes the error "The Google App Engine SDK could not be found" when you try to run the Django Server if you run a non-english operating system. Workaround and patch is availiable.

3. You need to have the Python win32 extensions installed if you are on windows.

Stay curious,

tisdag 24 juni 2008

Uploading files to App Engine via Django ModelForms

The FileField of the Django forms can be tied up just fine to the BlobProperty of a Data Store entity. This is how you do it.

The model

Displaying the form

The albumentry.html file is rather uninteresting, just stick {{ form }} in there and remember to put the enctype="multipart/form-data" attribute on the <form> tag. The above example will work just fine, and save the image as blob data to your BlobProperty.

The code used in this blog entry requires: App Engine SDK 1.1 Django Helper r30 Django 0.97


Found this today: http://code.google.com/p/gmemsess/

gmemsess is a secure lightweight memcache-backed session class for Google appengine. Currently it is only suitable for short-term sessions, for providing your own authentication system or shopping cart, for instance. The session cookie expires when the browser is closed, and the reliability of memcache for longer-term storage has yet to be demonstrated.
Pretty nice. I guess it should have a datastore backup, though, since we don't know how the MemCache API evicts data. Still cool that the little open source projects plop up so fast!

måndag 23 juni 2008

Getting random entities from the DataStore

Sometimes it amazes me how easy it is to do some things using App Engine/Django, and how some simple things are a goddamn science. This one is somewhere in the middle. Getting a random entity from the Datastore. This is how you do it!

See what happens here? We assign a random float to the entity on it's creation - they will look something like this in the datastore:


Then, when doing a query, we simply create a new random value and query the Datastore for a single entity that is larger than that random value. In the off chance that this query returns no results, we'll have a backup query that runs the whole thing in reverse. This can happen if you have very few entities, and none of them have a random value in the upper ranges:


If the random value you send into the GqlQuery above is 0.898912291991, it will return no rows. Therefore, the backup query is needed. The system will, of course, rely less and less on the backup query as more entities are added.

söndag 22 juni 2008

FileField not working?

I was getting very frustrated with my FileField not working in Django, but then I ran across this solution after some intense googling:


Turns out that you have to go:

instead of just

Makes sense, but had me totally stumped!

torsdag 19 juni 2008

Mutual Model relations

Got this baby today. DuplicatePropertyError ModelName already has property ModelName_set. Caused by this code: The solution is to define the underlying collection name via the collection_name argument to the ReferenceProperty constructor, as it will default to "modelname_set" otherwise, causing the DuplicatePropertyError.

    giver = db.ReferenceProperty(Address, collection_name='address_giver_set')
    receiver = db.ReferenceProperty(Address, collection_name='address_receiver_set')

måndag 16 juni 2008

Even faster websites

I ran across this just awesome presentation by Steve Sounders whom Google stole from Yahoo. :P

You can get the slides here.

fredag 13 juni 2008

Google I/O videos about app engine

In case you did not know, the Google IO videos are now online! The App Engine specific talks are:

torsdag 12 juni 2008

How-to: Full-text search in Google App Engine

It turns out that Google App Engine DOES have support for full-text search, it's just not documented, because the feature is still in development.

When App Engine first arrived, a lot of people, including myself, was baffled at the lack of full-text search in the DataStore API. What the fudge - Google is THE full-text search company, and their database solution does not have support for Full-text indexing!

The DataStore is built on top of Googles BigTable, which is a huge-arse database that powers a lot of projects at Google, including Search Indexing. Yes, the insanely limited, strange Data Storage is what Google is using to power their blazingly fast search engine.

The Google App Engine API has a very primitive implementation of a full-text index for the datastore, hidden away in google.appengine.ext.search. (There is basically no documentation of it, so you have to read the source, lazy boy) You use it by creating your models from search.SearchableModel, instead of the usual db.Model.

Like this:


This is basically just "find entries that contains these words" - it has no exact phrase match, substring match, boolean operators, stemming, or other common full-text features.

The nitty gritty

Save latency The philophy behind the Data Store is to make use of the fact that disk space is cheap, and perform and store calculations when a piece of data is stored. This applies to SearchableModels as well - they create the index for the entity when Save() is called. This means that instances created from SearchableModels take slightly longer to save than standard models. Keep this in mind.

Index of the index As you might now, The Google App Engine SDK generates indexes in index.yaml for all queries that you run while you are developing the app. However, since you might not be running all the imaginable cases of queries while you are developing, the index.yaml might be inadequate, and need to be manually appended with indexes. In these cases, you need to know that the full-text index is placed in a propertly called __searchable_text_index. To add indexes for it, the full-text index property:

  - kind: Article
    - name: __searchable_text_index
    - name: publishDate
      direction: desc 
There you go! Full-text indexing on App Engine. Not perfect at all, but it works for a lot of scenarios!

web2py: A first look from me

Web2Py (formerly called Gluon) is a very cool concept. It's a framework very, very similiar to Django, except it has almost zero configuration, and allows you to create and develop your applications entirely online. It does a LOT for you, and kind of runs on Google App Engine.
web2py on the Google appengine from mdipierro on Vimeo. I'm always sceptical of frameworks that does lots of stuff for you in enourmously elegant ways, because they normally stop being elegant the minute you want to do something a little more custom. That said, the idea of getting this entire shabang running on Google App Engine, allowing you to do your entire development for App Engine in a browser, is a very funky idea. Web2py is worth keeping an eye on.

onsdag 11 juni 2008


This is my first entry for this blog. It's gonna be about me learning App Engine, since I learn the best by teaching others. But first, a little insight about it, to get be going with the writing. I work at a consulting company, and is currently rented, full time, to a local company that does internal development of it's CRM system. The last few days, I've been trying to deploy a very, very simple web application. It's basically a simple one-page form, tied to a database and a web service. It also does some mailing stuff. Problem is, deploying this app is insanely complicated, even in this organization that has a pretty small IT department. I have to order a bloody server, along with a functional specification. I have to specifically order the SQL server, and the mail server. I also have to request IP access to the web service. And I have to go through two guys which are pretty hard to get a hold of. The whole thing has taken like 3 days. To upload a web app. This is something that would have taken me approximately one hour TOPS on my own server. On app engine? 10 minutes. I wonder what would have happened if I had developed the application in Python and uploaded it onto app engine. I would probably had gotten into trouble, but I imagine this happening in a lot of companies. Google very much considers App Engine to be a part of it's Google Apps suite - you can tie an app engine to your Google Apps for your Domain account, and it's listed right there, next to your Docs, Mail, Calendar and whatnot: This made me realize - Google is REALLY going after the internal IT departments of companies, in a very sneaky way. Lets just say that I was a little insane, and got just one wiki (Sites) and one app engine going in the organization, and got a subdomain of the company tied up to them (theapp.thecompanyiworkfor.com) through some minor but clever politics and bribery by candy. Lets then say that the organization started to rely a bit on these two things. Boom. Suddendly, google is in your organization, and you are not getting rid of it. You might as well start to use docs. Or mail. EVERYONE uses skype in the organization, by the way, even though absolutely no company policy dicates it's use. It's just there, and has grown into the organization. I imagine the same happening with Google Apps. Really, really frickin' sneaky, google.