Tuesday, 5 July 2016

unittest quirks - Python 3.5 vs 3.4

This is something I discovered when a CI server wouldn't find any of the tests in a Django project.

Consider the following directory structure. (app here is a Django app).
├── app
│   ├── __init__.py
│   ├── admin.py
│   ├── migrations
│   │   └── __init__.py
│   ├── models.py
│   ├── tests
│   │   ├── __init__.py
│   │   └── stuff.py
│   └── views.py
├── manage.py
stuff.py looks like this.
from django.test import TestCase

class SimpleTest(TestCase):
    def test_thing(self):
        self.assertEqual('foo', 'foo')
And here's __init__.py
from .stuff import SimpleTest  # noqa
By default, a Django app has an empty tests.py.
Replacing it with a module, and importing all tests in tests/__init__.py should work, right?
Creating test database for alias 'default'...

Ran 1 tests in 0.000s

Destroying test database for alias 'default'...
That's using Python 3.5.

What happens when you're using 3.4?
Creating test database for alias 'default'...

Ran 0 tests in 0.000s

Destroying test database for alias 'default'...
Wait, what?

The same version of Django is used in both cases, on the same codebase. What happened here?

Let's take a look at the Python 3.5 changelog
The TestLoader.loadTestsFromModule() method now accepts a keyword-only argument pattern which is passed toload_tests as the third argument. Found packages are now checked for load_tests regardless of whether their path matches pattern, because it is impossible for a package name to match the default pattern. (Contributed by Robert Collins and Barry A. Warsaw in issue 16662.)

Since the pattern Django uses is “test*.py”, a module wouldn't match, and the tests won't be found.

This can be made to work in 3.4 by changing stuff.py inside tests/ to test_stuff.py, (and removing the import in __init__.py, which isn't needed).

Lessons learnt:
  • Use the same version of Python in dev and production. Even minor versions matter.
  • Test against multiple versions of Python (using something like tox).