Thursday, October 11, 2012

Wednesday, September 5, 2012

Autoscaling with puppet

Puppet is great. Having a repeatable installation script of a server is the best documentation there is. Configuring an entire server with all kinds of services is easy! Even when the configuration of one service updates, dependent services can be made to reload automatically.

On a slightly different scale, where dependencies don't exist between services on the same machine, but on different machines, the options are limited

Puppet supports eventual consistency by running puppet agent on each machine on a fixed interval, which will update configuration. For auto scaling, this means your interval determines the time it takes for a machine to be visible by the load balancer. Also, to have this functionality, it is required to store all information about each server in the so called stored config database. Needless to say, with a large server park, having a low interval puts more stress on the database. Another drawback is that down scaling requires cleaning the database of servers that have been removed.

Meet mcollective

mcollective's philosophy can be summarized as 'the network is the database'. With a message queue based architecture, all kinds of information about your server park can be retrieved at mind boggling speed.

"There is no central asset database to go out of sync."

Queries are executed in parallel on all machines. Let's use mcollective instead of a database to retrieve facts about servers in the server park. And while we're at it let's use mcollective to update services on seperate machines. Having that, we can do autoscaling both up and down without stressing out a database server or risk being out of sync with reality.

puppet-kicker: cross server puppet notifications

puppet-kicker requires mcollective with the puppetd plugin for querying and notifying servers. Be sure to export all your facter/puppet facts to mcollective. (FactsFacterYAML for example)

Puppet-kicker triggers runs of puppet agent on dependent servers.

Suppose we've got a loadbalancer and some nodes that need to be balanced. With puppet-kicker that pattern looks like this;

On the node, you've got to make sure a fact is available with the name role. After that all it takes is

Immediately after a new node is installed, all haproxy servers will be kicked and puppet will update their configuration, adding the new server immediately.

On the loadbalancer, the role fact should also be defined and set to 'haproxy'

and the configfile would look like this

That should be enough to be able to add new nodes and have the loadbalancer be updated immediately. You can find puppet kicker on github

Sunday, June 10, 2012

Xcode workspaces and external libraries.

A lot has been said about Xcode workspaces. Still, when using the xcode workspace feature to add dependencies on external (static) libraries, you might still stumble upon the following error:

clang: error: linker command failed with exit code 1 (use -v to see invocation)

ld: warning: ignoring file /Users/michaeljackson/looma/libs/build/Debug-iphoneos/libMapView.a, missing required architecture i386 in file
Undefined symbols for architecture i386:


Let's think about the steps that where taken before this error emerged.

  1. Some external static library project was added to the workspace.
  2. The static library target from that project was added to 'link Binary with Libraries' in 'Build phases'.
    'link Binary with Libraries' in 'Choose frameworks and libraries to add' pane in 'Build phases'



Very strange, because xcode promised it would put all build products in a shared 'build' directory, so why can't it find the libraries?

It did in fact find the libraries, because they where there when you selected them from the picker pane in 'Build phases' right??!!

It turns out that xcode does not put all build products in a shared build directory. It ONLY does this if all your xcode projects are in the same folder. So if your project looks like this:

├── libs
│   ├── MapView.xcodeproj
│   ├── hoolahoop.xcodeproj
│   └── nostrils.xcodeproj
├── looma.xcodeproj
└── looma.xworkspace

You're in trouble. There will be a looma/libs/build and a looma/build directory.

In that case, when you select the library from 'link Binary with Libraries', you are selecting just a path to one of the platform specific builds in the looma/libs/build directory, which will probably be 'looma/libs/build/Release-iphoneos'.

As soon as you switch to the ios simulator, you will see the linker error, because your build is still being linked against the 'looma/libs/build/Release-iphoneos' build, which is an arm7 build and not i386.

The solution is to manually edit your library search path
library search path and add the following:


That will add the build folder in 'libs' to your search path for the current build scheme.

The same trick will work if you are having a lot of git submodules, where each project has a different structure. You can put them all in a folder and add a line to the library search path that adds the build folder for each project.

Tuesday, November 22, 2011

How I managed to edit /etc/hosts without becoming root and still got rejected from the app store.

Back in may, I was on holiday in Spain for a month. Finally without a contract, I decided to try and pick up cocoa. The new Mac OS X app store was still rather empty and especially the developer tools section was filled with useless icon resize tools. Opportunity called upon me to add something useful.

The concept, a system preference pane for /etc/hosts.

OS X has a lot of nice GUI tools for it's BSD core, for example network settings can be managed from the system preferences. One thing that does not have a GUI, is /etc/hosts. To do something to your hosts file you have to pop up emacs and manually edit the file. That's fine for techies, but sometimes managers need to check a website, before the DNS is switched and they can't be bothered to fire up a terminal.

I decided to add a preference pane to system preferences that allows you to edit the entries in your hosts file, with input validation and keeping the first few standard entries safe. One more feature I wanted for myself was a checkbox to toggle entries on and off. Sometimes I need to skip a load balancer or varnish and it's nice to be able to switch between those environments quickly.

the hosts prefpane

The rules.

I had a hunch that root access, which is required for editing /etc/hosts, could pose a problem in being accepted in the app store. Indeed, apples review guidelines state the following:

2.27 Apps that request escalation to root privileges or use setuid attributes will be rejected

Ok fine, I took that as noted and still started development on my preference pane. I figured apple might not object after they noticed how awesome and totally invaluable my new preference pane is (yeah LOL!). I first started out on the parser for the host file entries. Cocoa does not offer anything useful for parsing so I used bison and flex to make the parser. As it turned out, ipv4 and ipv6 regex patterns are among the examples of the flex manual, so that started off well. Soon I had a GUI that read and displayed the entries of my /etc/hosts file.

Writing to /etc/hosts in cocoa.

For writing to the hosts file I knew I somehow needed the user to enter his password to authenticate as an admin, just like the network preference pane. Cocoa features a system called Authorization Services. Authorization services can be used to ask permission from the operating system to perform a privileged action. In my case I want to edit /etc/hosts. Since OS X is BSD based that means the process that does this must be root. Apple Authorization Services offers several options. The first one is to use AuthorizationExecuteWithPrivileges, something like this should work:

After obtaining permission from Authorization Services which opens the password dialog, any process can be ran as root using AuthorizationExecuteWithPrivileges. Apple heavily advices against this approach, because an attacker can modify the parameters in the call -which is just a C string- to run any process as root, if your application get's compromised. This is why AuthorizationExecuteWithPrivileges is deprecated in OS X 10.7 and the documentation argues heavily against using it.

Security goes first.

If i was Apple I would make sure nothing that gets in the app store could ever be the cause of a major security risk. They sell the applications, so that makes them responsible. It's like selling viruses to customers, for money! That is why I never even tried using AuthorizationExecuteWithPrivileges, I figured using it would mean sure rejection.

Apple offers 2 more ways to make an application do stuff as root. But before we go into these there is one more thing you need to understand. I wanted my preference pane to work just like the network preferences. With a lock icon at the bottom. You click the lock, enter your password and then you can make changes to the hosts file all you want. Authorization Services offers a mechanism called pre-authorization. You authenticate once, get authorization for a specific operation, and while the authorization lasts, you aren't asked for your password anymore. That is what the lock does, everywhere.

Helper application.

For running operations as root, Apple recommends using a helper application. Your main application can pre-authorize the root operation and pass the authentication handle to the helper script. The script should then validate the handle and proceed if valid. Sounds good? There is one catch, instead of using AuthorizationExecuteWithPrivileges to start the helper application -which leaves your application open to exploits-, it should have it's setuid bit set to root ...

2.27 Apps that request escalation to root privileges or use setuid attributes will be rejected

I'm running out of options. I can't ship a helper application which uses setuid attributes and neither can I use AuthorizationExecuteWithPrivileges. Turns out apple has one more way you can run an operation as root. According to Apple's documentation:

There are a couple of functions you might be able to use to avoid forking off a privileged helper application: The authopen function lets you obtain temporary rights to create, read, or update a file.

Yay! I only need to read and write, so I'll just use authopen. That way I never include any code with setuid attributes, because authopen is shipped with OS X.

It turns out there is a catch. Remember the lock? And pre-authorization? A quick google will let you know you can't do that with authopen.

This is where google and I went separate ways. I couldn't believe you couldn't preauthorize authopen. What is the point of such a tool then? If I couldn't pre-authorize the edits, my preference pane would ask for your password every single time you made a change (in fact it did that when i shipped a broken build to my testers once). So I started reading everything I could find about Authorization Services.

The Authorization Database.

When you ask the Authorization Server for a specific privilege, it has a name. If you want to change parental controls in Safari, Safari acquires '' on your behalf. When authopen wants to let you edit /etc/hosts it tries to obtain 'sys.openfile.readwrite./etc/hosts'. The names of the privileges are in the authorization database, which is an xml file located at /private/etc/authorization. This is just an xml file, but Authorization Services provides an api for adding or changing privileges in the authorization database. The only privileges you can't touch start with 'system'. You have to manually edit the xml file if you really want that. Ok now read back a couple of lines:


That says 'sys' and not 'system'. No! It couldn't be? Could I change the properties of the authopen authorization? Could I change it so I'd be able to pre-authorize it? Yes I can, but in a way which is far less hackish as this story seems to turn to.

What your authorization database has as a key for authopen is in fact not 'sys.openfile.readwrite./etc/hosts' but ''sys.openfile.' (note the dot at the end). The dot at the end means that this is a wildcard privilege. You can not even obtain it, you can only obtain privileges that have something after the dot. Also, you can not modify it using the Authorization Services api.

BUT what you CAN do is ADD a concrete privilege to the authorization database!

So I can ADD 'sys.openfile.readwrite./etc/hosts' and set it's properties so I can pre-authorize it. That's far more elegant than I'd hoped. In fact it's even secure. Think about your sudo prompt. You only enter your password once every five minutes. That is what I set 'sys.openfile.readwrite./etc/hosts' to as well. And now you can edit your hosts file in the preference pane and only enter your password once every five minutes. And did I mention that changing the authorization database only requires admin privileges, not root? Bingo!

What did apple have to say about this.

I tried to submit the preference pane to the app store. I even had a signed installer, signed with the correct certificate and all. Turns out you can't submit preference panes or installers of any kind, only application bundles. I saw that one coming, so I was developing a regular app parallel to the preference pane. I submitted the app and after a couple of days I got an email stating my app was 'In review'. One minute later I got an email stating my app was 'Rejected':

2.27 The app requests root privileges from the user during operation when doing when the lock icon is clicked. See attached screen shot.

Hmm, my evil plot failed. I managed to never become root and still edit /etc/hosts, but I got rejected, how is that fair? Ofcourse I sent in an appeal to the review board. As they stated, it does not matter that none of the code I wrote ever runs as root, thereby neutralizing any security risk. Rule 2.27 also means you can not authenticate as an admin either!

I added a last comment, asking Apple to update their review guidelines to include that admin is forbidden as well and then open sourced the whole damn thing. But not before getting really pissed off.

You can download Hosts.prefpane from github if you're curious. Bonus points for whoever removes the auth database hack and replaces it with a proper helper script.