Tuesday, 5 July 2016

unittest quirks - Python 3.5 vs 3.4

This is something I discovered when a CI server wouldn't find any of the tests in a Django project.

Consider the following directory structure. (app here is a Django app).
.
├── app
│   ├── __init__.py
│   ├── admin.py
│   ├── migrations
│   │   └── __init__.py
│   ├── models.py
│   ├── tests
│   │   ├── __init__.py
│   │   └── stuff.py
│   └── views.py
├── manage.py
stuff.py looks like this.
from django.test import TestCase


class SimpleTest(TestCase):
    def test_thing(self):
        self.assertEqual('foo', 'foo')
And here's __init__.py
from .stuff import SimpleTest  # noqa
By default, a Django app has an empty tests.py.
Replacing it with a module, and importing all tests in tests/__init__.py should work, right?
Creating test database for alias 'default'...

----------------------------------------------------------------------
Ran 1 tests in 0.000s

OK
Destroying test database for alias 'default'...
That's using Python 3.5.

What happens when you're using 3.4?
Creating test database for alias 'default'...

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
Destroying test database for alias 'default'...
Wait, what?

The same version of Django is used in both cases, on the same codebase. What happened here?

Let's take a look at the Python 3.5 changelog
unittest
The TestLoader.loadTestsFromModule() method now accepts a keyword-only argument pattern which is passed toload_tests as the third argument. Found packages are now checked for load_tests regardless of whether their path matches pattern, because it is impossible for a package name to match the default pattern. (Contributed by Robert Collins and Barry A. Warsaw in issue 16662.)

Since the pattern Django uses is “test*.py”, a module wouldn't match, and the tests won't be found.

This can be made to work in 3.4 by changing stuff.py inside tests/ to test_stuff.py, (and removing the import in __init__.py, which isn't needed).

Lessons learnt:
  • Use the same version of Python in dev and production. Even minor versions matter.
  • Test against multiple versions of Python (using something like tox).

Thursday, 31 March 2016

Better Django choices

A typical Django model looks something like this:

from django.db import models


class MyModel(models.Model):
    STATUS_CHOICES = (
        (1, "Active"),
        (2, "In-Progress"),
        (-1, "Inactive"),
    )
    status = models.SmallIntegerField(choices=STATUS_CHOICES, default=1)

    def __str__(self):
        return '{}'.format(self.pk)

To filter on "status", one would do this

MyModel.objects.filter(status=1)

Not only is hardcoding non-informative in this case, anyone reading or writing a filter has to refer to the Model every time. In the two fairly large Django codebases that I have worked on, there was a mapping system that would let you do this.

MyModel.objects.filter(status=STATUS_MAP["Active"])

This gives better context to someone reading the code. Authors, on the other hand, still need to hardcode a string.

"There must be a better way"

Sure there is, a quick search will lead you to this.

One can pip install django-choices and call it a day. To me, this problem seems too trivial to require a dependency.

More searching will lead you to this. What I suggest is a mix of both, with some restructuring.

Create a directory named "choices" in the app's folder. For each model in your app, add a file in this directory (for "MyModel", you can add "mymodel.py").

For each field with choices, you define a class (with the same name capitalised).

class Status:
    ACTIVE = 1
    IN_PROGRESS = 2
    INACTIVE = -1

    choices = (
        (ACTIVE, "Active"),
        (IN_PROGRESS, "In-Progress"),
        (INACTIVE, "Inactive"),
    )

MyModel will be updated likewise. An extra property of the form <model_name><capitalised_field_name> is added. You can, and probably should, experiment with the naming conventions here.


from django.db import models

from .choices import mymodel


class MyModel(models.Model):
    MyModelStatus = mymodel.Status
    status = models.SmallIntegerField(choices=MyModelStatus.choices,
                                      default=MyModelStatus.ACTIVE)

    def __str__(self):
        return '{}'.format(self.pk)

Filtering will be thus:

MyModel.objects.filter(status=MyModel.MyModelStatus.ACTIVE)

There's several enhancements you can make to the "Status" class. Such as creating a base that auto-defines "choices", gives default values, etc. This does the job without adding much overhead. It is slightly verbose, but I'd take that any day for improved readability. Authors will also benefit from autocomplete in supporting editors and IDEs.

For those who might consider refactoring an existing codebase, you can run "makemigrations" to test it out. If you did everything correctly, no changes will be detected when you run "python manage.py makemigrations".

Making changes to production systems however, based on the latest blog posts that you read, is a risk that is yours to take :)

Tuesday, 29 March 2016

Stateful Modules in Python

TIL that random.seed persists across imports, aka modules with state. (aka unintentional MonkeyPatching?)

Let's say we have "moduleA.py" (WARNING:  Do not rely on "random" for cryptographic purposes, use "os.urandom" instead)

import random


def get_random_numbers():
    s = random.sample(range(1000), 10)
    return s

And another file named "main.py"

from moduleA import get_random_numbers

import random


if __name__ == '__main__':
    for i in range(3):
        random.seed(1234)
        # do things with random
        
        # code from another module that uses random
        print(get_random_numbers())

The side effect may be obvious here (prints the same random numbers on each iteration).

Perhaps .seed should be treated as a runtime context? (instead of associating state with the module)

with random.seed(1337):
    _ = random.random()

To avoid this, one can seed random with os.urandom before each use.

import random
import os


def get_random_numbers():
    random.seed(os.urandom(256))
    s = random.sample(range(1000), 10)
    return s


Friday, 11 December 2015

MediaWiki, VisualEditor & Parsoid

If you're trying to get rid of the (horrid) default editor that ships with the latest version of MediaWiki (in 2015) and use shiny stuff like VisualEditor, here's a tip that might save two days of your time.

VisualEditor needs to communicate with a backend that does actual parsing, which in this case, is Parsoid (nodejs package), but for reasons unknown the latest version of Parsoid (v0.4.x) does not work with the latest version of VisualEditor (for MediaWiki 1.25.3, which was released in October 2015 and is already deemed legacy… just PHP things).

One must check out (using git, for example) v0.3.0 of Parsoid, for VisualEditor to be able to communicate with it (when it comes to actual parsing of pages).

Tuesday, 31 March 2015

Nginx forbidden

Today I learned that to serve a file with nginx, you need to satisfy (at least) two conditions.

Nginx must have read access to the file you want to serve.


Even if nginx workers are running as root, if the file is marked 000, then nginx cannot serve the file!

In most cases this can be as easy as doing

chmod o+r filename

Or finer group-level permissions depending on access control restrictions for that file.


And the second condition which had me searching the web for hours is

Every directory in the path of the file must be set as executable.


If you want to serve files in /var/www/static/css

location /css/ {
    root /var/www/static;
}

Then var, www, static and css directories must be executable by the nginx process.

Most web server's master process runs as root spawning worker processes as www-data or whichever user you specify. www-data or the user must have appropriate access to all the files and directories you want to serve.


Wednesday, 4 February 2015

Fun with Python

Came across an interesting, albeit easy-rated problem on /r/dailyprogrammer.

Having done something similar on codingame before, I decided to overcomplicate the solution and scramble my brains in the process, for a challenge.

I had a quick glance at the highest rated solution on the reddit page and it was in inspiration when writing in Python but this is not a direct translation.

Here's the result:

from __future__ import print_function

p, q = 0x6f095e5b39, 0x7377497f7b


def g(num):
    return ((p if num < 5 else q) % (16 ** (10 - (num % 5) * 2 or 2))) // \
        (16 ** ((10 - (num % 5) * 2 or 2) - 2))


def r(number):
    b, m = [2 ** x for x in range(6, -1, -1)], [g(int(x)) for x in number]
    r = ''.join(' _ ' if y else '   ' for y in (b[0] & x for x in m))
    print(r, '\n' + '\n'.join(''.join([''.join(
        (z[1] if z[0] else ' ' for z in x)) for x in
        (zip([b[3 * i + j] & x for j in range(1, 4)], '|_|') for x in m)
    ]) for i in range(2)) + '\n')

r("0123456789")


Runs with both python2 and python3, and is also PEP8 compliant. For bonus points!

Figuring out how it works is left as an exercise for the reader. Let me know if you need help.

Sunday, 25 January 2015

Django tips #2

More Django tips. Check out the previous post in the series here: http://mixedquantum.blogspot.com/2015/01/python-django-tips.html

Exclude Inlines when creating an object.


This is actually well-documented but including here nonetheless.

(This is a method of admin.ModelAdmin, override in your custom admin class)

inlines = [MyInline]

def get_formsets_with_inlines(self, request, obj=None):
    for inline in self.get_inline_instances(request, obj):
        if isinstance(inline, MyInline) and obj is None:
            continue
        yield inline.get_formset(request, obj), inline

Original docs: https://docs.djangoproject.com/en/1.7/ref/contrib/admin/#django.contrib.admin.ModelAdmin.get_formsets_with_inlines


More control in Django admin.


When you want to limit the model instances that are seen by users depending on some property in the request object (or always). Override get_queryset. We have used this before but that was to annotate the QuerySet.

class MyModelAdmin(admin.ModelAdmin):

    def get_queryset(self, request):
        if not request.user.is_superuser:
            return MyModel.objects.filter(some_property=10)
        return super(MyModelAdmin, self).get_queryset(request)

Modifying Fieldsets


When you need to change fieldsets (choose which fields to display / how to display), override get_fieldsets.

class MyModelAdmin(admin.ModelAdmin):
    fieldsets = [
        (None, {
            'fields': ['a', 'b', 'c', 'd']
        })
    ]
    
    def _get_add_fieldsets(self):
        return [(None,{'fields':['a', 'b', 'c']})]

    def get_fieldsets(self, request, obj=None):
        if not request.user.is_superuser:
            self.exclude = ('a')
        if not obj:
            return self._get_add_fieldsets()
        return super(MyModelAdmin, self).get_fieldsets(request, obj=obj)

Modifying QuerySet for individual fields (foreign keys)


We override get_form

def get_form(self, request, obj=None, **kwargs):
    f = super(MyModelAdmin, self).get_form(request, obj=obj, **kwargs)
    if not request.user.is_superuser:
        # a is a foreign key here, whose model is A
        f.base_fields['a'].queryset = A.objects.filter(some_property=1)
    return f

For changing the QuerySet of a field in an Inline admin class


We will override formfield_for_foreignkey and / or formfield_for_manytomany:

class InlineAdmin(admin.StackedInline):
    model = MyModel

    def formfield_for_manytomany(self, db_field, request, **kwargs):
        if db_field.name == 'a':
            kwargs['queryset'] = A.objects.filter(some_property=2)
        return super(InlineAdmin, self).formfield_for_manytomany(
            db_field, request, **kwargs)

Trickery to hide models in Admin 


(When you have to register the model but not show it - for example, in case of django-polymorphic, you have to register child models, but not necessarily show them in admin).

def get_model_perms(self, request):
    return {}



Model functions:


Do last minute changes when a model is saved: save_model (this is also available in admin)


def save_model(self, request, obj, form, change):
    if not request.user.is_superuser:
        obj.some_property = 1337
    obj.save()

The advantage of doing this in admin is that you get access to the request / form / change objects, based on which you may do modifications before saving the model instance.