May 14th, 2010

Speed Up Django’s Unit Test with SQLite and Database Transaction by Gilang Chandrasa

Building test cases for your Django project is a good practice that can catch a lot of bugs during development but unfortunately, running a large number of test cases can take quite some time.

One of our Django projects which has 215 tests takes 460.117s to complete with MySQL’s MyISAM storage engine on a relatively modern Core 2 Duo box with 4GB of RAM. But we later found out that we can run the tests much faster if we run the tests using SQLite database. To illustrate this, the test that took 460.117s to complete in MySQL only took 6.919s in SQLite. That is very huge improvement (roughly 65x) and will save your time. According to Django’s documentation, the reason for the huge speed boost is because when used to run tests, SQLite will create databases in memory, bypassing the filesystem entirely so SQLite is definitely a huge time saver when used to run tests.

Switching to SQLite engine is easy, all you need to do is change your database settings (in settings.py) from:

DATABASE_ENGINE = 'mysql'    
DATABASE_NAME = 'db_name' 
DATABASE_USER = 'username'         
DATABASE_PASSWORD = 'password' 

to

DATABASE_ENGINE = 'sqlite3'    
DATABASE_NAME = 'test.db'
DATABASE_USER = ''         
DATABASE_PASSWORD = '' 

But it’s changing your database settings every time you want to run tests is very annoying. One solution to avoid this hassle is to append http://seanhayes.name/2010/01/09/test-database-django/a few lines of code to the end of settings.py (you have to add this below the main database configuration so it properly overwrites your default configuration).

...
if 'test' in sys.argv:
    DATABASE_ENGINE = 'sqlite3'          
    DATABASE_NAME = 'test.db'
...

Now, every time we call manage.py test in our project, it will automatically use SQLite engine to run.

We can even improve this further by modifying Django’s setup and tear down methods to use database transaction. Instead of tearing down the tables every time a test is made, we use SQLite’s rollback function to revert our test table’s state to when we first set it up. The code below shows how to do exactly this (taken from Yummy Apple Pie’s blog):

from django.db import transaction
from django.core import mail
from django.test import TestCase as DjangoTestCase
from django.core.management import call_command
from django.conf import settings
from django.core.urlresolvers import clear_url_caches

class TestCase(DjangoTestCase):

    def _pre_setup(self):
        transaction.enter_transaction_management()
        transaction.managed(True)
        if hasattr(self, 'fixtures'):
            # We have to use this slightly awkward syntax due to the fact
            # that we're using *args and **kwargs together.
            call_command('loaddata', *self.fixtures, **{'verbosity': 0})
        if hasattr(self, 'urls'):
            self._old_root_urlconf = settings.ROOT_URLCONF
            settings.ROOT_URLCONF = self.urls
            clear_url_caches()
        mail.outbox = []

    def _post_teardown(self):
        transaction.rollback()
        super(TestCase, self)._post_teardown()

Happy coding and hope this post encourages you to use Django’s excellent testing framework more extensively!

Credits:

  1. http://yummyapplepie.wordpress.com/2008/07/14/speed-up-your-functional-django-tests/
  2. http://seanhayes.name/2010/01/09/test-database-django/

Copyright © 2017 User Inspired Technology Services.