A blog about one guys quest to learn Python, Django and Google App Engine.
Feed: http://appengineguy.com/atom.xml

måndag 30 juni 2008

Proper Unit Testing of App Engine/Django

While there are a couple of things written on unit testing with App Engine, I unfortunately found them all lacking in one way or another. With this post, I'd like to describe how to unit test Django views in App Engine, using as little mocking as possible.

More specifically, I did not want to mock the Data Store Model objects, since that very quickly turns into mocking hell - you just end up spending three times as much time thinking about mocking than you are about code. Fortunately, after quite a bit of digging, I discovered that Google provides a temporary, in-memory datastore stub, specifically made for unit testing! They also provide stubs for mail, urlfetch and the user service. MAN that is nice!

For this example, I am using:

We will be using the Test Runner provided by Django. Django looks for tests in a number of places, one is tests.py - which it expects to be in same folder as models.py and views.py. To have django run your tests you go:

manage.py test nameofYourApp

The code




address.html is basically just "{{ form }}" and some HTML.

There ya go. Post any questions as comments, and I will try to answer them!

lördag 28 juni 2008

Google App Engine: The Book

For some people, this is old news, but I didn't catch it until now. It turns out that Noah Gift (co-author of Python for Unix and Linux) and Mike Orr (an editor of Linux Gazette), are already writing a bloody book on Google App Engine!

It is going to get published by Manning. One really nice thing about Manning is that they gradually release their books in electronic form as they are getting written through their early access program. Anyway, they have a blog where you can follow their progress (and get chapters) - it doesn't look much for the world yet, but it's most likely worth keeping an eye on!

torsdag 26 juni 2008

Unit Tests for Google App Engine

I am a firm believer that if you don't use unit tests, you are being a bad developer.

Read this!

onsdag 25 juni 2008

How to check out the latest version of Django using TortoiseSVN

This is a few pointers on how to get Django 0.97 and the App Engine Django Helper running.

I notice that a lot of people are using the included 0.96 version of Django included with the App Engine. Don't do this, it's stupid, and not recommended by anyone. The 0.96 just doesn't work all that well, it doesn't properly work with the App Engine Django Helper, and lacks a LOT of really nice stuff.

I think that the main reason people don't use the 0.97 version is that it's an unreleased version, which means it cannot be downloaded in a nice zip - instead, you need to check it out from the Django Subversion repository. New developers don't realize how incredibly easy it is to do this - it's almost easier than downloading a zip file.

Anyway, here is how you check out the current development version of Django (as of writing, v0.97), on windows:

  1. Download TortoiseSVN
  2. Install it and restart your computer.
  3. Create a folder called django or something, right click it, and click CheckOut.

  4. Enter http://code.djangoproject.com/svn/django/trunk/ in the URL field and click OK.

  5. Grab a cup of coffee REALLY FUCKING QUICKLY.

  6. All done! You now have Django 0.97!

Installning the Django Helper

As for installing the Django Helper, the introduction is pretty straightforward. There are three issues that you need to be aware of though:

1. You must remove a few unnecessary files from django to get below the 1000 file limit of App Engine. Waste these directories:

  • django/bin
  • django/contrib/admin
  • django/contrib/databrowse

(DON'T delete django/contrib/auth like the Django Helper inroduction says. It's actually needed.)

2. VERY IMPORTANT if you have a non-english operating system:
There is a very annoying bug in the Django helper r30 that causes the error "The Google App Engine SDK could not be found" when you try to run the Django Server if you run a non-english operating system. Workaround and patch is availiable.

3. You need to have the Python win32 extensions installed if you are on windows.

Stay curious,

tisdag 24 juni 2008

Uploading files to App Engine via Django ModelForms

The FileField of the Django forms can be tied up just fine to the BlobProperty of a Data Store entity. This is how you do it.

The model

Displaying the form

The albumentry.html file is rather uninteresting, just stick {{ form }} in there and remember to put the enctype="multipart/form-data" attribute on the <form> tag. The above example will work just fine, and save the image as blob data to your BlobProperty.

The code used in this blog entry requires: App Engine SDK 1.1 Django Helper r30 Django 0.97


Found this today: http://code.google.com/p/gmemsess/

gmemsess is a secure lightweight memcache-backed session class for Google appengine. Currently it is only suitable for short-term sessions, for providing your own authentication system or shopping cart, for instance. The session cookie expires when the browser is closed, and the reliability of memcache for longer-term storage has yet to be demonstrated.
Pretty nice. I guess it should have a datastore backup, though, since we don't know how the MemCache API evicts data. Still cool that the little open source projects plop up so fast!

måndag 23 juni 2008

Getting random entities from the DataStore

Sometimes it amazes me how easy it is to do some things using App Engine/Django, and how some simple things are a goddamn science. This one is somewhere in the middle. Getting a random entity from the Datastore. This is how you do it!

See what happens here? We assign a random float to the entity on it's creation - they will look something like this in the datastore:


Then, when doing a query, we simply create a new random value and query the Datastore for a single entity that is larger than that random value. In the off chance that this query returns no results, we'll have a backup query that runs the whole thing in reverse. This can happen if you have very few entities, and none of them have a random value in the upper ranges:


If the random value you send into the GqlQuery above is 0.898912291991, it will return no rows. Therefore, the backup query is needed. The system will, of course, rely less and less on the backup query as more entities are added.

söndag 22 juni 2008

FileField not working?

I was getting very frustrated with my FileField not working in Django, but then I ran across this solution after some intense googling:


Turns out that you have to go:

instead of just

Makes sense, but had me totally stumped!

torsdag 19 juni 2008

Mutual Model relations

Got this baby today. DuplicatePropertyError ModelName already has property ModelName_set. Caused by this code: The solution is to define the underlying collection name via the collection_name argument to the ReferenceProperty constructor, as it will default to "modelname_set" otherwise, causing the DuplicatePropertyError.

    giver = db.ReferenceProperty(Address, collection_name='address_giver_set')
    receiver = db.ReferenceProperty(Address, collection_name='address_receiver_set')

måndag 16 juni 2008

Even faster websites

I ran across this just awesome presentation by Steve Sounders whom Google stole from Yahoo. :P

You can get the slides here.

fredag 13 juni 2008

Google I/O videos about app engine

In case you did not know, the Google IO videos are now online! The App Engine specific talks are:

torsdag 12 juni 2008

How-to: Full-text search in Google App Engine

It turns out that Google App Engine DOES have support for full-text search, it's just not documented, because the feature is still in development.

When App Engine first arrived, a lot of people, including myself, was baffled at the lack of full-text search in the DataStore API. What the fudge - Google is THE full-text search company, and their database solution does not have support for Full-text indexing!

The DataStore is built on top of Googles BigTable, which is a huge-arse database that powers a lot of projects at Google, including Search Indexing. Yes, the insanely limited, strange Data Storage is what Google is using to power their blazingly fast search engine.

The Google App Engine API has a very primitive implementation of a full-text index for the datastore, hidden away in google.appengine.ext.search. (There is basically no documentation of it, so you have to read the source, lazy boy) You use it by creating your models from search.SearchableModel, instead of the usual db.Model.

Like this:


This is basically just "find entries that contains these words" - it has no exact phrase match, substring match, boolean operators, stemming, or other common full-text features.

The nitty gritty

Save latency The philophy behind the Data Store is to make use of the fact that disk space is cheap, and perform and store calculations when a piece of data is stored. This applies to SearchableModels as well - they create the index for the entity when Save() is called. This means that instances created from SearchableModels take slightly longer to save than standard models. Keep this in mind.

Index of the index As you might now, The Google App Engine SDK generates indexes in index.yaml for all queries that you run while you are developing the app. However, since you might not be running all the imaginable cases of queries while you are developing, the index.yaml might be inadequate, and need to be manually appended with indexes. In these cases, you need to know that the full-text index is placed in a propertly called __searchable_text_index. To add indexes for it, the full-text index property:

  - kind: Article
    - name: __searchable_text_index
    - name: publishDate
      direction: desc 
There you go! Full-text indexing on App Engine. Not perfect at all, but it works for a lot of scenarios!

web2py: A first look from me

Web2Py (formerly called Gluon) is a very cool concept. It's a framework very, very similiar to Django, except it has almost zero configuration, and allows you to create and develop your applications entirely online. It does a LOT for you, and kind of runs on Google App Engine.
web2py on the Google appengine from mdipierro on Vimeo. I'm always sceptical of frameworks that does lots of stuff for you in enourmously elegant ways, because they normally stop being elegant the minute you want to do something a little more custom. That said, the idea of getting this entire shabang running on Google App Engine, allowing you to do your entire development for App Engine in a browser, is a very funky idea. Web2py is worth keeping an eye on.

onsdag 11 juni 2008


This is my first entry for this blog. It's gonna be about me learning App Engine, since I learn the best by teaching others. But first, a little insight about it, to get be going with the writing. I work at a consulting company, and is currently rented, full time, to a local company that does internal development of it's CRM system. The last few days, I've been trying to deploy a very, very simple web application. It's basically a simple one-page form, tied to a database and a web service. It also does some mailing stuff. Problem is, deploying this app is insanely complicated, even in this organization that has a pretty small IT department. I have to order a bloody server, along with a functional specification. I have to specifically order the SQL server, and the mail server. I also have to request IP access to the web service. And I have to go through two guys which are pretty hard to get a hold of. The whole thing has taken like 3 days. To upload a web app. This is something that would have taken me approximately one hour TOPS on my own server. On app engine? 10 minutes. I wonder what would have happened if I had developed the application in Python and uploaded it onto app engine. I would probably had gotten into trouble, but I imagine this happening in a lot of companies. Google very much considers App Engine to be a part of it's Google Apps suite - you can tie an app engine to your Google Apps for your Domain account, and it's listed right there, next to your Docs, Mail, Calendar and whatnot: This made me realize - Google is REALLY going after the internal IT departments of companies, in a very sneaky way. Lets just say that I was a little insane, and got just one wiki (Sites) and one app engine going in the organization, and got a subdomain of the company tied up to them (theapp.thecompanyiworkfor.com) through some minor but clever politics and bribery by candy. Lets then say that the organization started to rely a bit on these two things. Boom. Suddendly, google is in your organization, and you are not getting rid of it. You might as well start to use docs. Or mail. EVERYONE uses skype in the organization, by the way, even though absolutely no company policy dicates it's use. It's just there, and has grown into the organization. I imagine the same happening with Google Apps. Really, really frickin' sneaky, google.