In Django you cannot add or remove through= on M2M fields

Using native migrations in Django has been a very positive experience. Now and then however, some thing appear difficult or impossible. In this post, I will demonstrate how migrations can be used to move from an automatic ManyToMany relation in Django to an explicit relation. Lets start by creating a small app with a ManyToMany relation. Suppose we want some way to order the list of Bar instances in the relation. In setting up the ManyToMany relation, Django has created a table for us, named through_foo_bars. We can see what it looks like in python using the inspectdb command

$ ./manage.py inspectdb through_foo_bars

We can make explicit the automatic relation that Django has created in the form of an extra table, by adding the model to our application. Note the ordering we specified in the Meta definition. We told Django to use our new model for the relation by specifying it as the through argument in the ManyToManyField. Django is happy to create a migration for us, incorporating the change.

$ ./manage.py makemigrations

The problem starts when we try to execute the migrations

$ ./manage.py migrate
...
ValueError: Cannot alter field through.Foo.bars into through.Foo.bars - they are not compatible types (you cannot alter to or from M2M fields, or add or remove through= on M2M fields)

So according to Django this kind of migration is not possible. Let's analyse what the problem is. The below migration tries to create a new model called ThroughFooBars and then change the ManyToManyField on Foo. The problem is ofcourse that no new model needs to be created because the table allready exists. In this case we DO want to change the migration state, altering the relation, but we do NOT want to change the database schema. We are lucky that django offers such migrations by providing us with migrations.SeparateDatabaseAndState. We will first modify the state to make the relation explicit, after which we can do the modifications to the schema to add the ordering. In the migration below, you can see we first modify the state, making the relation explicit in the state only, without modifying the database schema. Next we add the order field and the ordering Meta definition. We can check if the migration was succesful with inspectdb:

$ ./manage.py inspectdb through_foo_bars

Seems to look good to me?

Plone conf 2012 buildout bash completion

Quickly navigate your buildout with this virtualenvwrapper postactive hook:
https://github.com/specialunderwear/buildout-bash-completion

How I managed to edit /etc/hosts without becoming root and still got rejected from the app store.

Back in may, I was on holiday in Spain for a month. Finally without a contract, I decided to try and pick up cocoa. The new Mac OS X app store was still rather empty and especially the developer tools section was filled with useless icon resize tools. Opportunity called upon me to add something useful.

The concept, a system preference pane for /etc/hosts.

OS X has a lot of nice GUI tools for it's BSD core, for example network settings can be managed from the system preferences. One thing that does not have a GUI, is /etc/hosts. To do something to your hosts file you have to pop up emacs and manually edit the file. That's fine for techies, but sometimes managers need to check a website, before the DNS is switched and they can't be bothered to fire up a terminal.

I decided to add a preference pane to system preferences that allows you to edit the entries in your hosts file, with input validation and keeping the first few standard entries safe. One more feature I wanted for myself was a checkbox to toggle entries on and off. Sometimes I need to skip a load balancer or varnish and it's nice to be able to switch between those environments quickly.

The rules.

I had a hunch that root access, which is required for editing /etc/hosts, could pose a problem in being accepted in the app store. Indeed, apples review guidelines state the following:

2.27 Apps that request escalation to root privileges or use setuid attributes will be rejected

Ok fine, I took that as noted and still started development on my preference pane. I figured apple might not object after they noticed how awesome and totally invaluable my new preference pane is (yeah LOL!). I first started out on the parser for the host file entries. Cocoa does not offer anything useful for parsing so I used bison and flex to make the parser. As it turned out, ipv4 and ipv6 regex patterns are among the examples of the flex manual, so that started off well. Soon I had a GUI that read and displayed the entries of my /etc/hosts file.

Writing to /etc/hosts in cocoa.

For writing to the hosts file I knew I somehow needed the user to enter his password to authenticate as an admin, just like the network preference pane. Cocoa features a system called Authorization Services. Authorization services can be used to ask permission from the operating system to perform a privileged action. In my case I want to edit /etc/hosts. Since OS X is BSD based that means the process that does this must be root. Apple Authorization Services offers several options. The first one is to use AuthorizationExecuteWithPrivileges, something like this should work:

After obtaining permission from Authorization Services which opens the password dialog, any process can be ran as root using AuthorizationExecuteWithPrivileges. Apple heavily advices against this approach, because an attacker can modify the parameters in the call -which is just a C string- to run any process as root, if your application get's compromised. This is why AuthorizationExecuteWithPrivileges is deprecated in OS X 10.7 and the documentation argues heavily against using it.

Security goes first.

If i was Apple I would make sure nothing that gets in the app store could ever be the cause of a major security risk. They sell the applications, so that makes them responsible. It's like selling viruses to customers, for money! That is why I never even tried using AuthorizationExecuteWithPrivileges, I figured using it would mean sure rejection.

Apple offers 2 more ways to make an application do stuff as root. But before we go into these there is one more thing you need to understand. I wanted my preference pane to work just like the network preferences. With a lock icon at the bottom. You click the lock, enter your password and then you can make changes to the hosts file all you want. Authorization Services offers a mechanism called pre-authorization. You authenticate once, get authorization for a specific operation, and while the authorization lasts, you aren't asked for your password anymore. That is what the lock does, everywhere.

Helper application.

For running operations as root, Apple recommends using a helper application. Your main application can pre-authorize the root operation and pass the authentication handle to the helper script. The script should then validate the handle and proceed if valid. Sounds good? There is one catch, instead of using AuthorizationExecuteWithPrivileges to start the helper application -which leaves your application open to exploits-, it should have it's setuid bit set to root ...

2.27 Apps that request escalation to root privileges or use setuid attributes will be rejected

I'm running out of options. I can't ship a helper application which uses setuid attributes and neither can I use AuthorizationExecuteWithPrivileges. Turns out apple has one more way you can run an operation as root. According to Apple's documentation:

There are a couple of functions you might be able to use to avoid forking off a privileged helper application: The authopen function lets you obtain temporary rights to create, read, or update a file.

Yay! I only need to read and write, so I'll just use authopen. That way I never include any code with setuid attributes, because authopen is shipped with OS X.

It turns out there is a catch. Remember the lock? And pre-authorization? A quick google will let you know you can't do that with authopen.

This is where google and I went separate ways. I couldn't believe you couldn't preauthorize authopen. What is the point of such a tool then? If I couldn't pre-authorize the edits, my preference pane would ask for your password every single time you made a change (in fact it did that when i shipped a broken build to my testers once). So I started reading everything I could find about Authorization Services.

The Authorization Database.

When you ask the Authorization Server for a specific privilege, it has a name. If you want to change parental controls in Safari, Safari acquires 'com.apple.Safari.parental-controls' on your behalf. When authopen wants to let you edit /etc/hosts it tries to obtain 'sys.openfile.readwrite./etc/hosts'. The names of the privileges are in the authorization database, which is an xml file located at /private/etc/authorization. This is just an xml file, but Authorization Services provides an api for adding or changing privileges in the authorization database. The only privileges you can't touch start with 'system'. You have to manually edit the xml file if you really want that. Ok now read back a couple of lines:

sys.openfile.readwrite./etc/hosts

That says 'sys' and not 'system'. No! It couldn't be? Could I change the properties of the authopen authorization? Could I change it so I'd be able to pre-authorize it? Yes I can, but in a way which is far less hackish as this story seems to turn to.

What your authorization database has as a key for authopen is in fact not 'sys.openfile.readwrite./etc/hosts' but ''sys.openfile.' (note the dot at the end). The dot at the end means that this is a wildcard privilege. You can not even obtain it, you can only obtain privileges that have something after the dot. Also, you can not modify it using the Authorization Services api.

BUT what you CAN do is ADD a concrete privilege to the authorization database!

So I can ADD 'sys.openfile.readwrite./etc/hosts' and set it's properties so I can pre-authorize it. That's far more elegant than I'd hoped. In fact it's even secure. Think about your sudo prompt. You only enter your password once every five minutes. That is what I set 'sys.openfile.readwrite./etc/hosts' to as well. And now you can edit your hosts file in the preference pane and only enter your password once every five minutes. And did I mention that changing the authorization database only requires admin privileges, not root? Bingo!

What did apple have to say about this.

I tried to submit the preference pane to the app store. I even had a signed installer, signed with the correct certificate and all. Turns out you can't submit preference panes or installers of any kind, only application bundles. I saw that one coming, so I was developing a regular app parallel to the preference pane. I submitted the app and after a couple of days I got an email stating my app was 'In review'. One minute later I got an email stating my app was 'Rejected':

2.27 The app requests root privileges from the user during operation when doing when the lock icon is clicked. See attached screen shot.

Hmm, my evil plot failed. I managed to never become root and still edit /etc/hosts, but I got rejected, how is that fair? Ofcourse I sent in an appeal to the review board. As they stated, it does not matter that none of the code I wrote ever runs as root, thereby neutralizing any security risk. Rule 2.27 also means you can not authenticate as an admin either!

I added a last comment, asking Apple to update their review guidelines to include that admin is forbidden as well and then open sourced the whole damn thing. But not before getting really pissed off.

You can download Hosts.prefpane from github if you're curious. Bonus points for whoever removes the auth database hack and replaces it with a proper helper script.

Tweet