Proper Unit Testing of App Engine/Django

While there are a couple of things written on unit testing with App Engine, I unfortunately found them all lacking in one way or another. With this post, I'd like to describe how to unit test Django views in App Engine, using as little mocking as possible.

More specifically, I did not want to mock the Data Store Model objects, since that very quickly turns into mocking hell - you just end up spending three times as much time thinking about mocking than you are about code. Fortunately, after quite a bit of digging, I discovered that Google provides a temporary, in-memory datastore stub, specifically made for unit testing! They also provide stubs for mail, urlfetch and the user service. MAN that is nice!

For this example, I am using:

• App Engine SDK 1.1
• Django Helper r30
• Django 0.97

We will be using the Test Runner provided by Django. Django looks for tests in a number of places, one is tests.py - which it expects to be in same folder as models.py and views.py. To have django run your tests you go:

manage.py test nameofYourApp

The code

models.py
from appengine_django.models import BaseModel from google.appengine.ext import db class Address(BaseModel): user = db.UserProperty(required=True) name = db.StringProperty(verbose_name="Full name",required=True) addressLine1 = db.StringProperty(verbose_name="Address Line 1",required=True) addressLine2 = db.StringProperty(verbose_name="Address Line 2",required=True) zip = db.StringProperty(verbose_name="Zip code",required=True) city = db.StringProperty(required=True) country = db.StringProperty(required=True)

tests.py
import os #import the stubs, i.e. the fake datastore, user and mail service and urlfetch from google.appengine.api import apiproxy_stub_map from google.appengine.api import datastore_file_stub from google.appengine.api import mail_stub from google.appengine.api import urlfetch_stub from google.appengine.api import user_service_stub from google.appengine.api import users # import the Django Forms replacement provided by the App Engine SDK, # so that Django Forms get Data Store model support. from google.appengine.ext.db.djangoforms import ModelForm # I use Python Mocker to create mock objects: http://labix.org/mocker import mocker from mocker import MockerTestCase from views import address # importing our view from santastormsite.models import Address # importing our Model class TestStart(MockerTestCase): def setUp(self): # Start with a fresh api proxy. apiproxy_stub_map.apiproxy = apiproxy_stub_map.APIProxyStubMap() # Use a fresh stub datastore. # From this point on in the tests, all calls to the Data Store, such as get and put, # will be to the temporary, in-memory datastore stub. stub = datastore_file_stub.DatastoreFileStub(u'myTemporaryDataStorage', '/dev/null', '/dev/null') apiproxy_stub_map.apiproxy.RegisterStub('datastore_v3', stub) # Use a fresh stub UserService. apiproxy_stub_map.apiproxy.RegisterStub('user', user_service_stub.UserServiceStub()) os.environ['AUTH_DOMAIN'] = 'gmail.com' os.environ['USER_EMAIL'] = 'myself@appengineguy.com' # set to '' for no logged in user os.environ['SERVER_NAME'] = 'fakeserver.com' os.environ['SERVER_PORT'] = '9999' # Use a fresh urlfetch stub. apiproxy_stub_map.apiproxy.RegisterStub('urlfetch', urlfetch_stub.URLFetchServiceStub()) # Use a fresh mail stub. apiproxy_stub_map.apiproxy.RegisterStub('mail', mail_stub.MailServiceStub()) # Mock all calls to HttpResponseRedirect self.HttpResponseRedirect = self.mocker.replace("django.http.HttpResponseRedirect") # Mock all calls to the render_to_response shortcut self.render_to_response = self.mocker.replace("django.shortcuts.render_to_response") def testAddress(self): # setting the mocker to a shorthand variable. m = self.mocker # Making sure the current user has no address. # This is not necessary for a good unit test, but it makes the test more descriptive. addressOfCurrentUser = Address.all().filter("user = ", users.get_current_user()).get() self.assertTrue(addressOfCurrentUser == None) # create a fake httprequest. mockRequest = self.mocker.mock() # create a fake collection that we will use to fake the POST variables. mockPostCollection = self.mocker.mock() mockRequest.POST # expect the tested code to call the POST collection m.count(1, None) # expect it to call it one or more times m.result(mockPostCollection) #make it return the fake collection # expect the tested code to get some variables from the faked collection mockPostCollection.get("name", None) m.result("Mattias Johansson") mockPostCollection.get("addressLine1", None) m.result("Phat Cool Street 5") mockPostCollection.get("addressLine2", None) m.result("Park Lane") mockPostCollection.get("zip", None) m.result("361.12-1") mockPostCollection.get("city", None) m.result("New York") mockPostCollection.get("country", None) m.result("Hatland") #expect the tested code to do a redirect. self.HttpResponseRedirect("/profile-was-saved") # okay, we are done setting up our expectations! self.mocker.replay() # call the view! address(mockRequest) #make sure that an Address entity was created and properly saved. retrAddress = Address.all().filter("user = ", users.get_current_user()).get() self.assertEquals("Mattias Johansson", retrAddress.name) self.assertEquals("Phat Cool Street 5", retrAddress.addressLine1) self.assertEquals("Park Lane", retrAddress.addressLine2) self.assertEquals("361.12-1", retrAddress.zip) self.assertEquals("New York", retrAddress.city) self.assertEquals("Hatland", retrAddress.country) def testAddressAddressExists(self): # If a logged in user navigates directly to /address, # his existing address should be displayed in the form. m = self.mocker # Create an address for the user. sampleAddress = Address(user = users.get_current_user(), name = "John Wayne", addressLine1 = "Banana Street 13", addressLine2 = "Wanana",zip = "55555", city = "Woogaboo City", country = "Oompaloompaland" ) sampleAddress.put() # fake the request and POST, as in the test above, but this time, return None. mockRequest = self.mocker.mock() mockRequest.POST m.result(None) # This is the function that gets called instead of the real, actual render_to_response. def checkArgumentsOfRenderToResponse(path, params): containsCorrectName = (params['form']['name'].as_text().count("John Wayne") == 1) self.assert_(containsCorrectName, "name field had incorrect data.") containsCorrectAddressLine1 = (params['form']['addressLine1'].as_text().count("Banana Street 13") == 1) self.assert_(containsCorrectAddressLine1, "address line 1 field had incorrect data.") containsCorrectAddressLine2 = (params['form']['addressLine2'].as_text().count("Wanana") == 1) self.assert_(containsCorrectAddressLine2, "address line 2 field had incorrect data.") containsCorrectZip = (params['form']['zip'].as_text().count("55555") == 1) self.assert_(containsCorrectZip, "zip field had incorrect data.") containsCorrectCity = (params['form']['city'].as_text().count("Woogaboo City") == 1) self.assert_(containsCorrectCity, "city field had incorrect data.") containsCorrectCountry = (params['form']['country'].as_text().count("Oompaloompaland") == 1) self.assert_(containsCorrectCountry, "country field had incorrect data.") return '<html/>' # expect a call to render_to_response. We expect ANY parameters here, # since checkArgumentsOfRenderToResponse will be testing the input. self.render_to_response(mocker.ANY, mocker.ANY) # forward the render_to_response call to checkArgumentsOfRenderToResponse m.call(checkArgumentsOfRenderToResponse) #okay, we are done recording our expectations. m.replay() # call the view! address(mockRequest) retrAddress = Address.all().filter("user = ", users.get_current_user()).get() self.assertTrue("Mattias Johansson", retrAddress.name) def testAddressNotLoggedInRedirect(self): # fake the request mockRequest = self.mocker.mock() os.environ['USER_EMAIL'] = '' self.assertTrue(users.get_current_user() == None) # The user is now None # Expect a redirect to login. self.HttpResponseRedirect(users.create_login_url("/address")) #okay, we are done recording our expectations. self.mocker.replay() # call the view! address(mockRequest)

views.py
from django.http import HttpResponse from django.http import HttpResponseRedirect from django.shortcuts import render_to_response from google.appengine.api import users from models import * from google.appengine.ext.db.djangoforms import ModelForm class AddressForm(ModelForm): class Meta: model = Address exclude = ("user") def address(request): if not users.get_current_user(): return HttpResponseRedirect(users.create_login_url("/address")); currentUser = users.get_current_user() addressOfCurrentUser = Address.all().filter("user = ", currentUser).get() if request.POST: if not addressOfCurrentUser: # assign dummy values to required fields addressOfCurrentUser = Address(user = users.get_current_user(), name = "#", addressLine1 = "#", addressLine2 = "#",zip = "#", city = "#", country ="#" ) form = AddressForm(request.POST, instance=addressOfCurrentUser) else: if addressOfCurrentUser: form = AddressForm(instance=addressOfCurrentUser) else: form = AddressForm(user = users.get_current_user()) if form.is_valid(): address = form.save() return HttpResponseRedirect('/profile-was-saved') return render_to_response('address.html', locals())

address.html is basically just "{{ form }}" and some HTML.

There ya go. Post any questions as comments, and I will try to answer them!

Google App Engine: The Book

For some people, this is old news, but I didn't catch it until now. It turns out that Noah Gift (co-author of Python for Unix and Linux) and Mike Orr (an editor of Linux Gazette), are already writing a bloody book on Google App Engine!

It is going to get published by Manning. One really nice thing about Manning is that they gradually release their books in electronic form as they are getting written through their early access program. Anyway, they have a blog where you can follow their progress (and get chapters) - it doesn't look much for the world yet, but it's most likely worth keeping an eye on!

Unit Tests for Google App Engine

I am a firm believer that if you don't use unit tests, you are being a bad developer.

Read this!
http://blog.appenginefan.com/2008/06/unit-tests-for-google-app-engine-apps.html

How to check out the latest version of Django using TortoiseSVN

This is a few pointers on how to get Django 0.97 and the App Engine Django Helper running.

I notice that a lot of people are using the included 0.96 version of Django included with the App Engine. Don't do this, it's stupid, and not recommended by anyone. The 0.96 just doesn't work all that well, it doesn't properly work with the App Engine Django Helper, and lacks a LOT of really nice stuff.

I think that the main reason people don't use the 0.97 version is that it's an unreleased version, which means it cannot be downloaded in a nice zip - instead, you need to check it out from the Django Subversion repository. New developers don't realize how incredibly easy it is to do this - it's almost easier than downloading a zip file.

Anyway, here is how you check out the current development version of Django (as of writing, v0.97), on windows:

1. Download TortoiseSVN
2. Install it and restart your computer.
   
3. Create a folder called django or something, right click it, and click CheckOut.
   
   

4. Enter http://code.djangoproject.com/svn/django/trunk/ in the URL field and click OK.
   
   

5. Grab a cup of coffee REALLY FUCKING QUICKLY.
   
   

6. All done! You now have Django 0.97!
   
   

Installning the Django Helper

As for installing the Django Helper, the introduction is pretty straightforward. There are three issues that you need to be aware of though:

1. You must remove a few unnecessary files from django to get below the 1000 file limit of App Engine. Waste these directories:

• django/bin
• django/contrib/admin
• django/contrib/databrowse

(DON'T delete django/contrib/auth like the Django Helper inroduction says. It's actually needed.)

2. VERY IMPORTANT if you have a non-english operating system:
There is a very annoying bug in the Django helper r30 that causes the error "The Google App Engine SDK could not be found" when you try to run the Django Server if you run a non-english operating system. Workaround and patch is availiable.

3. You need to have the Python win32 extensions installed if you are on windows.

Stay curious,
Mattias

Uploading files to App Engine via Django ModelForms

The FileField of the Django forms can be tied up just fine to the BlobProperty of a Data Store entity. This is how you do it.

The model

from appengine_django.models import BaseModel from google.appengine.ext import db class AlbumItem(BaseModel): creationDate = db.DateTimeProperty(auto_now_add=True) image = db.BlobProperty(required=True) note = db.StringProperty()
Displaying the form

class AlbumItemForm(ModelForm): image = FileField() note = CharField(widget=Textarea) class Meta: model = AlbumItem exclude = ("creationDate") def albumentry(request): if request.POST: form = AlbumItemForm(request.POST, request.FILES) if form.is_valid(): albumEntry = form.save() return HttpResponseRedirect('/showalbumentry/%s' % albumEntry.key()) else: return render_to_response('albumentry.html', locals()) else: form = AlbumItemForm() return render_to_response('albumentry.html', locals())

The albumentry.html file is rather uninteresting, just stick {{ form }} in there and remember to put the enctype="multipart/form-data" attribute on the <form> tag. The above example will work just fine, and save the image as blob data to your BlobProperty.

The code used in this blog entry requires: App Engine SDK 1.1 Django Helper r30 Django 0.97

GMemsess

Found this today: http://code.google.com/p/gmemsess/

gmemsess is a secure lightweight memcache-backed session class for Google appengine. Currently it is only suitable for short-term sessions, for providing your own authentication system or shopping cart, for instance. The session cookie expires when the browser is closed, and the reliability of memcache for longer-term storage has yet to be demonstrated.

Pretty nice. I guess it should have a datastore backup, though, since we don't know how the MemCache API evicts data. Still cool that the little open source projects plop up so fast!

Getting random entities from the DataStore

Sometimes it amazes me how easy it is to do some things using App Engine/Django, and how some simple things are a goddamn science. This one is somewhere in the middle. Getting a random entity from the Datastore. This is how you do it!

# Setting up the model class myModel(BaseModel): random = db.FloatProperty(required=True) # Creating an entity myModelInstance = myModel(random = random.random()) myModelInstance.save() # Querying the datastore gql = "SELECT * FROM myModel WHERE random >= :random ORDER BY random ASC LIMIT 1" randomSantaRelation = GqlQuery(gql, random=random.random()).get() if randomSantaRelation is None: gql = "SELECT * FROM myModel WHERE random <= :random ORDER BY random DESC LIMIT 1" randomSantaRelation = GqlQuery(gql, random=random.random()).get()

See what happens here? We assign a random float to the entity on it's creation - they will look something like this in the datastore:

0.216565955485

Then, when doing a query, we simply create a new random value and query the Datastore for a single entity that is larger than that random value. In the off chance that this query returns no results, we'll have a backup query that runs the whole thing in reverse. This can happen if you have very few entities, and none of them have a random value in the upper ranges:

0.628912291991
0.416565323566
0.216565955485
0.118278328322
0.013212121212

If the random value you send into the GqlQuery above is 0.898912291991, it will return no rows. Therefore, the backup query is needed. The system will, of course, rely less and less on the backup query as more entities are added.

FileField not working?

I was getting very frustrated with my FileField not working in Django, but then I ran across this solution after some intense googling:

http://hurley.wordpress.com/2007/10/06/django-binding-uploaded-files-to-a-form/

Turns out that you have to go:
form = NoteForm(request.POST, request.FILES)

instead of just

form = NoteForm(request.POST)

Makes sense, but had me totally stumped!

Mutual Model relations

Got this baby today. DuplicatePropertyError ModelName already has property ModelName_set. Caused by this code: class Address(BaseModel): name = db.StringProperty() addressLine1 = db.StringProperty() addressLine2 = db.StringProperty() zip = db.StringProperty() city = db.StringProperty() country = db.StringProperty() class SantaRelation(BaseModel): code = db.StringProperty() giver = db.ReferenceProperty(Address) receiver = db.ReferenceProperty(Address) isReceived = db.BooleanProperty() creationDate = db.DateTimeProperty(auto_now_add=True) image = db.Blob() note = db.StringProperty() The solution is to define the underlying collection name via the collection_name argument to the ReferenceProperty constructor, as it will default to "modelname_set" otherwise, causing the DuplicatePropertyError.

    giver = db.ReferenceProperty(Address, collection_name='address_giver_set')
    receiver = db.ReferenceProperty(Address, collection_name='address_receiver_set')

Presto!

Even faster websites

I ran across this just awesome presentation by Steve Sounders whom Google stole from Yahoo. :P

You can get the slides here.

Google I/O videos about app engine

In case you did not know, the Google IO videos are now online! The App Engine specific talks are:

• Rapid Development with Python, Django, and Google App Engine
• Building Scalable Web Applications with Google App Engine
• Engaging User Experiences with Google App Engine
• Under the Covers of the Google App Engine Datastore
• Best Practices - Building a Production Quality Application on Google App Engine
• Working with Google App Engine Models

How-to: Full-text search in Google App Engine

It turns out that Google App Engine DOES have support for full-text search, it's just not documented, because the feature is still in development.

When App Engine first arrived, a lot of people, including myself, was baffled at the lack of full-text search in the DataStore API. What the fudge - Google is THE full-text search company, and their database solution does not have support for Full-text indexing!

The DataStore is built on top of Googles BigTable, which is a huge-arse database that powers a lot of projects at Google, including Search Indexing. Yes, the insanely limited, strange Data Storage is what Google is using to power their blazingly fast search engine.

The Google App Engine API has a very primitive implementation of a full-text index for the datastore, hidden away in google.appengine.ext.search. (There is basically no documentation of it, so you have to read the source, lazy boy) You use it by creating your models from search.SearchableModel, instead of the usual db.Model.

Like this: from google.appengine.ext import db from google.appengine.ext import search class Article(search.SearchableModel): title = db.TextProperty() publishDate = db.DateTimeProperty(auto_now_add=True) text = db.TextProperty() # should NOT be returned article = Article(text ='''This is the totally secret article text which talks about sausages and cheese in the middle of itself.''') article.title = "Fine cuisine" article.save() # should BE returned article = Article() article.title = "What I feed my dogs" article.text = '''This is the totally secret article text which talks about sausages and cheese in the middle of itself.''' article.save() print "Results" query = Article.all().search("sausages cheese dogs").order("-publishDate") for a in query: print "%s | %s" % (a.title, a.publishDate) print "Done printing"

Limitations

This is basically just "find entries that contains these words" - it has no exact phrase match, substring match, boolean operators, stemming, or other common full-text features.

The nitty gritty

Save latency The philophy behind the Data Store is to make use of the fact that disk space is cheap, and perform and store calculations when a piece of data is stored. This applies to SearchableModels as well - they create the index for the entity when Save() is called. This means that instances created from SearchableModels take slightly longer to save than standard models. Keep this in mind.

Index of the index As you might now, The Google App Engine SDK generates indexes in index.yaml for all queries that you run while you are developing the app. However, since you might not be running all the imaginable cases of queries while you are developing, the index.yaml might be inadequate, and need to be manually appended with indexes. In these cases, you need to know that the full-text index is placed in a propertly called __searchable_text_index. To add indexes for it, the full-text index property:

  - kind: Article
    properties:
    - name: __searchable_text_index
    - name: publishDate
      direction: desc 

There you go! Full-text indexing on App Engine. Not perfect at all, but it works for a lot of scenarios!

web2py: A first look from me

Web2Py (formerly called Gluon) is a very cool concept. It's a framework very, very similiar to Django, except it has almost zero configuration, and allows you to create and develop your applications entirely online. It does a LOT for you, and kind of runs on Google App Engine.
web2py on the Google appengine from mdipierro on Vimeo. I'm always sceptical of frameworks that does lots of stuff for you in enourmously elegant ways, because they normally stop being elegant the minute you want to do something a little more custom. That said, the idea of getting this entire shabang running on Google App Engine, allowing you to do your entire development for App Engine in a browser, is a very funky idea. Web2py is worth keeping an eye on.

Right

This is my first entry for this blog. It's gonna be about me learning App Engine, since I learn the best by teaching others. But first, a little insight about it, to get be going with the writing. I work at a consulting company, and is currently rented, full time, to a local company that does internal development of it's CRM system. The last few days, I've been trying to deploy a very, very simple web application. It's basically a simple one-page form, tied to a database and a web service. It also does some mailing stuff. Problem is, deploying this app is insanely complicated, even in this organization that has a pretty small IT department. I have to order a bloody server, along with a functional specification. I have to specifically order the SQL server, and the mail server. I also have to request IP access to the web service. And I have to go through two guys which are pretty hard to get a hold of. The whole thing has taken like 3 days. To upload a web app. This is something that would have taken me approximately one hour TOPS on my own server. On app engine? 10 minutes. I wonder what would have happened if I had developed the application in Python and uploaded it onto app engine. I would probably had gotten into trouble, but I imagine this happening in a lot of companies. Google very much considers App Engine to be a part of it's Google Apps suite - you can tie an app engine to your Google Apps for your Domain account, and it's listed right there, next to your Docs, Mail, Calendar and whatnot: This made me realize - Google is REALLY going after the internal IT departments of companies, in a very sneaky way. Lets just say that I was a little insane, and got just one wiki (Sites) and one app engine going in the organization, and got a subdomain of the company tied up to them (theapp.thecompanyiworkfor.com) through some minor but clever politics and bribery by candy. Lets then say that the organization started to rely a bit on these two things. Boom. Suddendly, google is in your organization, and you are not getting rid of it. You might as well start to use docs. Or mail. EVERYONE uses skype in the organization, by the way, even though absolutely no company policy dicates it's use. It's just there, and has grown into the organization. I imagine the same happening with Google Apps. Really, really frickin' sneaky, google.