Friday, 11 December 2015

MediaWiki, VisualEditor & Parsoid

If you're trying to get rid of the (horrid) default editor that ships with the latest version of MediaWiki (in 2015) and use shiny stuff like VisualEditor, here's a tip that might save two days of your time.

VisualEditor needs to communicate with a backend that does actual parsing, which in this case, is Parsoid (nodejs package), but for reasons unknown the latest version of Parsoid (v0.4.x) does not work with the latest version of VisualEditor (for MediaWiki 1.25.3, which was released in October 2015 and is already deemed legacy… just PHP things).

One must check out (using git, for example) v0.3.0 of Parsoid, for VisualEditor to be able to communicate with it (when it comes to actual parsing of pages).

Tuesday, 31 March 2015

Nginx forbidden

Today I learned that to serve a file with nginx, you need to satisfy (at least) two conditions.

Nginx must have read access to the file you want to serve.

Even if nginx workers are running as root, if the file is marked 000, then nginx cannot serve the file!

In most cases this can be as easy as doing

chmod o+r filename

Or finer group-level permissions depending on access control restrictions for that file.

And the second condition which had me searching the web for hours is

Every directory in the path of the file must be set as executable.

If you want to serve files in /var/www/static/css

location /css/ {
    root /var/www/static;

Then var, www, static and css directories must be executable by the nginx process.

Most web server's master process runs as root spawning worker processes as www-data or whichever user you specify. www-data or the user must have appropriate access to all the files and directories you want to serve.

Wednesday, 4 February 2015

Fun with Python

Came across an interesting, albeit easy-rated problem on /r/dailyprogrammer.

Having done something similar on codingame before, I decided to overcomplicate the solution and scramble my brains in the process, for a challenge.

I had a quick glance at the highest rated solution on the reddit page and it was in inspiration when writing in Python but this is not a direct translation.

Here's the result:

from __future__ import print_function

p, q = 0x6f095e5b39, 0x7377497f7b

def g(num):
    return ((p if num < 5 else q) % (16 ** (10 - (num % 5) * 2 or 2))) // \
        (16 ** ((10 - (num % 5) * 2 or 2) - 2))

def r(number):
    b, m = [2 ** x for x in range(6, -1, -1)], [g(int(x)) for x in number]
    r = ''.join(' _ ' if y else '   ' for y in (b[0] & x for x in m))
    print(r, '\n' + '\n'.join(''.join([''.join(
        (z[1] if z[0] else ' ' for z in x)) for x in
        (zip([b[3 * i + j] & x for j in range(1, 4)], '|_|') for x in m)
    ]) for i in range(2)) + '\n')


Runs with both python2 and python3, and is also PEP8 compliant. For bonus points!

Figuring out how it works is left as an exercise for the reader. Let me know if you need help.

Sunday, 25 January 2015

Django tips #2

More Django tips. Check out the previous post in the series here:

Exclude Inlines when creating an object.

This is actually well-documented but including here nonetheless.

(This is a method of admin.ModelAdmin, override in your custom admin class)

inlines = [MyInline]

def get_formsets_with_inlines(self, request, obj=None):
    for inline in self.get_inline_instances(request, obj):
        if isinstance(inline, MyInline) and obj is None:
        yield inline.get_formset(request, obj), inline

Original docs:

More control in Django admin.

When you want to limit the model instances that are seen by users depending on some property in the request object (or always). Override get_queryset. We have used this before but that was to annotate the QuerySet.

class MyModelAdmin(admin.ModelAdmin):

    def get_queryset(self, request):
        if not request.user.is_superuser:
            return MyModel.objects.filter(some_property=10)
        return super(MyModelAdmin, self).get_queryset(request)

Modifying Fieldsets

When you need to change fieldsets (choose which fields to display / how to display), override get_fieldsets.

class MyModelAdmin(admin.ModelAdmin):
    fieldsets = [
        (None, {
            'fields': ['a', 'b', 'c', 'd']
    def _get_add_fieldsets(self):
        return [(None,{'fields':['a', 'b', 'c']})]

    def get_fieldsets(self, request, obj=None):
        if not request.user.is_superuser:
            self.exclude = ('a')
        if not obj:
            return self._get_add_fieldsets()
        return super(MyModelAdmin, self).get_fieldsets(request, obj=obj)

Modifying QuerySet for individual fields (foreign keys)

We override get_form

def get_form(self, request, obj=None, **kwargs):
    f = super(MyModelAdmin, self).get_form(request, obj=obj, **kwargs)
    if not request.user.is_superuser:
        # a is a foreign key here, whose model is A
        f.base_fields['a'].queryset = A.objects.filter(some_property=1)
    return f

For changing the QuerySet of a field in an Inline admin class

We will override formfield_for_foreignkey and / or formfield_for_manytomany:

class InlineAdmin(admin.StackedInline):
    model = MyModel

    def formfield_for_manytomany(self, db_field, request, **kwargs):
        if == 'a':
            kwargs['queryset'] = A.objects.filter(some_property=2)
        return super(InlineAdmin, self).formfield_for_manytomany(
            db_field, request, **kwargs)

Trickery to hide models in Admin 

(When you have to register the model but not show it - for example, in case of django-polymorphic, you have to register child models, but not necessarily show them in admin).

def get_model_perms(self, request):
    return {}

Model functions:

Do last minute changes when a model is saved: save_model (this is also available in admin)

def save_model(self, request, obj, form, change):
    if not request.user.is_superuser:
        obj.some_property = 1337

The advantage of doing this in admin is that you get access to the request / form / change objects, based on which you may do modifications before saving the model instance.

Thursday, 8 January 2015

Trello iCalendar feed - Information Leakage (fixed)

+Trello recently changed the protocol of the iCalendar feed URI that they provide in their Calendar power-up.

Previous feed format:



This changes how a Calendar clients retrieves the calendar feed. If using webcal, the client will query the host over HTTP with the full feed URI (Just changing webcal to http). Trello responds with a 302 pointing to the same URI, but with the protocol replaced with https.

Observing network traffic in Wireshark:

It is trivial for anyone in control of the network or any passive attacker on a wireless network to retrieve the same and get full read-only access to all cards on a board that has the power-up enabled and is marked with a date. With the new feed (using https), no communication takes place in the clear.

This is more of a limitation of the webcal protocol that doesn't have any way to specify secure communication. I suggested replacing webcal with https as it worked for the client I tested with ( on OSX), but this would probably need testing with other Calendar clients before being pushed as a fix.

Response timeline.

I reported this on 2015/5/1 at 06:30 EST.

First (human) response: 2015/5/1 14:27 EST.

Acknowledging issue: 2015/5/1 14:43 EST - Saying issue was previously reported and fix was expected sometime this week.

Fixed: 2015/8/1 09:07 EST - webcal replaced with https.

+1 for Trello security.

Thursday, 1 January 2015

Python & Django tips

Some code snippets that I've found useful. (Mostly from, sorry for not pointing to the actual answers, my bookmarks are a mess)

Ordering by custom column in Django admin:

You have to annotate your QuerySet, add a method for your custom column and enable ordering on it.

Here we want to display how many books each Publisher has. And display the Publisher's name and the book count in the admin interface.

We override get_queryset and annotate the QuerySet.

class PublisherAdmin(admin.ModelAdmin):
    def get_queryset(self, request):
        return Publisher.objects.annotate(book_count=Count('book'))

    # Return the annotated column's name
    def total_books(self, publisher_object):
        return publisher_object.book_count
    # Enable ordering
    total_books.admin_order_field = 'book_count'

    list_display = ('name', 'total_books')

Serializing datetime (and possibly other) objects when passed to json.dumps

def _unix_time(dt):
    epoch =
    delta = dt - epoch
    return delta.total_seconds()

def serialize_datetime(obj):
    if hasattr(obj, 'isoformat'):
        return obj.isoformat()
        # Or if you want the date in another format
        # return obj.strftime('%d-%m-%Y')
        # To get ms since epoch
        # return _unix_time(obj)
    return None

j = json.dumps(list_of_objects_with_datetime, default=serialize_datetime)

Ordering by multiple fields, aggregating and ordering QuerySet.

This seems difficult to achieve at first, but it couldn't be easier.

MyModel.objects.values('A', 'B') \
    .annotate(count=Sum('C')) \

In the above example, you group by columns A and B. Then you can Count or Sum another column (the aggregation), and finally we order by one of the columns we grouped by (A or B) or the annotated column (count).