Some time ago, I have explored the mobile deployment space domain a bit, by implementing a function capable of building Android Apps with the Nix package manager. However, Android is definitely not the only platform used in the mobile space. Another major one is Apple's iOS, used by mobile devices, such as the iPhone, iPod, and iPad. In this blog post, I describe how I have implemented a Nix function capable of building Apps for these platforms.

Although it was quite some work to package the Android SDK, it took me less work for iOS. However, all prerequisites for this platform cannot be entirely packaged in Nix, due to the fact that they require Xcode, that must be installed through the App store, and it has all kinds of implicit assumptions on the system (such as location of certain tools and binaries), which prevents all required components to be packaged purely. Therefore, I had to use similar "tricks" that I have used to allow .NET applications to be deployed through the Nix package manager.

Installing Xcode

The first dependency that requires manual installation is Apple's Xcode, providing a set of development tools and an IDE. It must be obtained from the App store through the following URL: https://developer.apple.com/xcode. The App store takes care of the complete installation process.

After installing Xcode, we must also manually install the command-line utilities. This can be done by starting Xcode, picking the preferences option from the 'Xcode' menu bar, selecting the 'Downloads' tab in the window and then by installing the command-line utilities, as shown in the screenshot below:


An Xcode installation includes support for the most recent iOS versions by default. If desired, SDKs for older versions can be installed through the same dialog.

Installing MacPorts

To use the Nix package manager, we can either download a collection of bootstrap binaries or we can compile it from source. I took the latter approach for which it is required to install a number of prerequisites through MacPorts. Installation of MacPorts is straight forward -- it's just downloading the installer from their website (for the right Mac OS X version) and running it.

Installing the Nix package manager

To install Nix from source code, I first had to install a bunch of dependencies through MacPorts. I opened a terminal and I've executed the following commands:

$ sudo port install sqlite3
$ sudo port install pkgconfig
$ sudo port install curl
$ sudo port install xz

I have obtained the source tarball of Nix from the Nix homepage, and I have installed it from source by running the following command-line instructions:

$ tar xfvj nix-1.1.tar.bz2
$ cd nix-1.1
$ ./configure
$ make
$ sudo make install

Furthermore, I was a bit lazy and I only care about using a single-user installation. To use Nix as a non-root user I ran the following command to change the permissions to my user account:

sudo chown -R sander /nix

For development, it's also useful to have a copy of Nixpkgs containing expressions for many common packages. I cloned the git repository, by running:

$ git clone https://github.com/NixOS/nixpkgs.git

To conveniently access stuff from the Nix packages repository, it is useful to add the following line to the ~/.profile file so that they are in Nix's include path:

$ export NIX_PATH=nixpkgs=/path/to/nixpkgs

I also added channels that provides substitutes for builds performed at our build infrastructure, so that I don't have to compile everything from source:

$ nix-channel --add http://nixos.org/releases/nixpkgs/channels/nixpkgs-unstable
$ nix-channel --update

"Packaging Xcode" in Nix

In order to package iOS applications, we have to make the Xcode build infrastructure available to Nix package builds. However, as explained earlier, we cannot package Xcode in Nix, because it must be installed through other means (through Apple's App store) and it has a number of assumptions on locations of utilities on the host system.

Instead we use the same trick that we have used earlier for the .NET framework -- we create a proxy component wrapper that contains symlinks to essential binaries on the host system. Our xcodewrapper package expression looks as follows:

{stdenv}:

let
  version = "4.5.2";
in
stdenv.mkDerivation {
  name = "xcode-wrapper-"+version;
  buildCommand = ''
    ensureDir $out/bin
    cd $out/bin
    ln -s /usr/bin/xcode-select
    ln -s /usr/bin/xcodebuild
    ln -s /usr/bin/xcrun
    ln -s /usr/bin/security
    ln -s "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform\
/Developer/Applications/iPhone Simulator.app/Contents/MacOS/iPhone Simulator"

    # Check if we have the xcodebuild version that we want
    if [ -z "$($out/bin/xcodebuild -version | grep ${version})" ]
    then
        echo "We require xcodebuild version: ${version}"
        exit 1
    fi
  '';
}

Basically, the following expression contains a number of symlinks to essential Xcode binaries:

• xcode-select is a utility to determine filesystem location of Xcode, or to switch between default versions.
• xcodebuild is a command-line utility to build Xcode projects.
• xcrun is a utility to execute specific Xcode-specific operations, such as packaging a build result.
• security is a command-line interface to Mac OS X's keychain facility that manages security certificates and identities.
• iPhone Simulator. A program to simulate iOS applications on the host system.

Furthermore, the expression contains a check that verifies whether the installed Xcode matches the version that we want to use, ensuring that -- if we upgrade Xcode -- we use the version that we want, and that everything that has Xcode as build-time dependency gets rebuilt, if necessary.

By adding the xcodewrapper to the buildInputs parameter of a package expression, the Xcode build utilities can be found and used.

Building iOS apps for the simulator

Getting Xcode working in Nix is a bit ugly and does not provide full reproducibility as Xcode must be installed manually, but it works. Another important question to answer is, how do we build iOS apps with Nix?

Building iOS apps is pretty straight forward, by running xcodebuild inside an Xcode project directory. The xcodebuild reference served as a good reference for me. The following instruction suffices to build an app for the iPhone simulator:

$ xcodebuild -target HelloWorld -configuration Debug \
  -sdk iphonesimulator6.0 -arch i386 ONLY_ACTIVE_ARCH=NO \
  CONFIGURATION_TEMP_DIR=$TMPDIR CONFIGURATION_BUILD_DIR=$out

The above command-line instruction builds an app for the i386 architecture (Apps in the simulator have the same architecture as the host system) using the iPhone 6.0 simulator SDK. The last two parameters ensure that intermediate files generated by the compiler are stored in our temp directory and that the resulting build is stored in our desired prefix, instead of the Xcode project directory.

Building iOS apps for release

Building actual releases for Apple devices is a bit trickier though. In order to deploy an App to a Apple device, a distribution certificate is required as well as an provisioning profile. To test on an Apple device, an ad-hoc provisioning profile is required. For the App store, you need a distribution profile. These files must be obtained from Apple. More information on this can be found here.

Furthermore, in order to use a certificate and provisioning profile, we need to import these into the keychain of the user that executes a build. I use the following instructions to automatically create and unlock a new keychain:

$ security create-keychain -p "" mykeychain
$ security default-keychain -s mykeychain
$ security unlock-keychain -p "" mykeychain

The above instructions generate a keychain with the name mykeychain in the user's home directory: ~/Library/Keychains.

After creating the keychain, we must import our certificate into it. The following instruction imports a key with the name mykey.p12 into the keychain with name: mykeychain and uses the password: secret for the certificate (the -A parameter grants all applications access to the certificate):

$ security import mykey.p12 -k mykeychain -P "secret" -A 

We must also make the provisioning profile available. This can be done by extracting the UUID from the mobile provisioning file and to copy the file into ~/Library/MobileDevice/Provisioning Profiles/<UUID>.mobileprovision:

PROVISIONING_PROFILE=$(grep UUID -A1 -a myprovisioningprofile.mobileprovision | \
  grep -o "[-A-Z0-9]\{36\}")
mkdir -p "$HOME/Library/MobileDevice/Provisioning Profiles"
cp myprovisioningprofile.mobileprovision \
  "$HOME/Library/MobileDevice/Provisioning Profiles/$PROVISIONING_PROFILE.mobileprovision"

After importing the certificate and provisioning files, we can invoke xcodebuild to build a release App using the earlier certificate and provisioning profile (I have based this invocation on a Stack overflow post about setting a provisioning profile):

$ xcodebuild -target HelloWorld -configuration Release \
  -sdk iphoneos6.0 -arch armv7 ONLY_ACTIVE_ARCH=NO \
  CONFIGURATION_TEMP_DIR=$TMPDIR CONFIGURATION_BUILD_DIR=$out \
  CODE_SIGN_IDENTITY="My Cool Company" \
  PROVISIONING_PROFILE=$PROVISIONING_PROFILE \
  OTHER_CODE_SIGN_FLAGS="--keychain $HOME/Library/Keychains/mykeychain"

The above command-line invocation is nearly identical to the one that builds an App for the simulator, except that we have a different SDK parameter: iphoneos6.0, and the armv7 architecture. Moreover, we have to specify the identity name of the key, and the locations of the provisioning profiles and keychains.

To test an app on an Apple device, we must generate an IPA file, a Zip archive containing all files belonging to an App. This can be done by running:

$ xcrun -sdk iphoneos PackageApplication \
  -v HelloWorld.app -o HelloWorld.ipa

By using the Xcode organiser the given App can be uploaded to an Apple device.

In order to release an App through the Apple App store, we must generate an xcarchive, a zip archive that also contains debugging symbols and plist files. An xcarchive can be generated by adding the archive parameter to xcodebuild. The resulting xcarchive is automatically added to the Xcode organiser.

If the build process has been completed, we can eventually remove our keychain:

$ security delete-keychain mykeychain

Simulating iOS applications

We can also script the iPhone simulator to automatically start an App produced by xcodebuild, for a particular iOS device:

"iPhone Simulator" -SimulateApplication build/HelloWorld \
  -SimulateDevice 'iPhone'

The above command launches an iPhone instance running our simulator build performed earlier. The former command-line option refers to the executable that we have built. The latter specifies the type of device we want to simulate. Other device options that worked for me are: 'iPad', 'iPad (Retina)', 'iPhone (Retina 3.5-inch)', 'iPhone (Retina 4-inch)'.

Implementing a test case

To demonstrate how the iOS deployment functions work, I have implemented the trivial 'Hello world' example case described in Apple's iOS tutorial. It was also a nice exercise to get myself familiar with Xcode and Objective C development.

Apart from implementing the example case, I have made two extensions:

• The example only implements the iPhone storyboard. On the iPad, it shows me an empty screen. I also implemented the same stuff in the iPad storyboard to make it work smoothly on the iPad.
• In order to pass the validation process for releases, we need to provide icons. I added two files: Icon.png (a 59x59 image) and Icon-72.png (a 72x72 image) and I added them to the project's info.plist file.

Deploying iOS applications with Nix

I have encapsulated the earlier described build steps in a Nix function called: xcodeenv.buildApp. To build the Hello World example for the simulator, one could write the following expression:

{xcodeenv}:

xcodeenv.buildApp {
  name = "HelloWorld";
  src = ../../src/HelloWorld;
  release = false;
}

The function is very simple -- it invokes the xcodeenv.buidApp function and takes three parameters: name is the name of the xcode project, src is a path referring to the source code and release indicates whether we want to build for release or not. If release is false, then it will build the app for the simulator. It also assumes that the architecture (arch) is i386 and the SDK is the iphonesimulator, which is usually the case. These parameters can be overridden by adding them to the function parameters of the function invocation.

To make a build for release, the following expression can be written:

{xcodeenv}:

xcodeenv.buildApp {
  name = "HelloWorld";
  src = ../../src/HelloWorld;
  release = true;

  certificateFile = /Users/sander/mycertificate.p12;
  certificatePassword = "secret";
  codeSignIdentity = "iPhone Distribution: My Cool Company";  
  provisioningProfile = /Users/sander/provisioningprofile.mobileprovision;
  generateIPA = false;
  generateXCArchive = false;
}

The above expression sets the release parameter to true. In this case, the architecture is assumed to be armv7 and the SDK iphoneos, which ensures that the App is build for the right architecture for iOS devices. For release builds we also need to specify the certificate, provisioning profile, and their relevant properties. Furthermore, we can also specify whether we want to generate an IPA (an archive to deploy an App to a device) or xcarchive (an archive to deploy an App to Apple's App store).

The function invocation above conveniently takes care of the build, key signing and packaging process, although there were two nasty caveats:

• To ensure purity, Nix restricts access to the user's home directory by setting the HOME environment variable to /homeless-shelter. For some operations that require a home directory, it suffices to set HOME to a temp directory. This seems not to work for key signing on Mac OS X. Generating a keychain and importing a certificate seems to work fine, but signing does not. I suspect this has something to do with the fact that the keychain is a system service.
  
  I have circumvented this issue by generating a keystore in the real user's home directory, with exactly the same name as the Nix component name that we are currently building, including the unique hash derived from the build inputs. The hash should ensure that the name never interferes with other components. Furthermore, the keychain is always removed after the build, whether it succeeds or not. This solution is a bit nasty and tricky, but it works fine for me and should not affect an existing build, running simultaneously.
• Another caveat is the generation of xcarchives. It works fine, but the result is always stored in a global location: ~/Library/Developer/Xcode/Archives that allows the Xcode organiser to find it. It looks like there is no way to change this. Therefore, I don't know how to make the generation of xcarchives pure, although this is a task that only needs to be performed when releasing an App through Apple's App store. This task is typically performed only once in a while.

As with ordinary Nix expressions, we cannot use the previous two expressions to build an App directly, but we also need to compose them by calling them with the right xcodeenv parameter. This is done in the following composition expression:

rec {
  xcodeenv = import ...

  helloworld = import ./pkgs/helloworld {
    inherit xcodeenv;
  };

  simulate_helloworld_iphone = import ./pkgs/simulate-helloworld {
    inherit xcodeenv helloworld;
    device = "iPhone";
  };

  simulate_helloworld_ipad = import ./pkgs/simulate-helloworld {
    inherit xcodeenv helloworld;
    device = "iPad";
  };
  ...
}

By using the expression above and by running nix-build, we can automatically build an iOS app:

$ nix-build default.nix -A hello
/nix/store/0nlz31xb1q219qrlmimxssqyallvqdyx-HelloWorld
$ cd result
$ ls
HelloWorld.app  HelloWorld.app.dSYM

Besides build an App, we can also write an expression to simulate an iOS app:

{xcodeenv, helloworld, device}:

xcodeenv.simulateApp {
  name = "HelloWorld";
  app = helloworld;
  inherit device;
}

The above expression calls the xcodeenv.simulateApp function that generates a script launching the iPhone Simulator. The name parameter corresponds to the name of the App, the app parameter refers to a simulator build of an App, an the device parameter specifies the device to simulate, such as an iPhone or iPad.

If we take the composition expression (shown earlier), we also see a simulator attribute: simulate_helloworld_iphone, which we can evaluate to generate a script launching a simulator for the iPhone. By executing the generated script we launch the simulator:

$ nix-build -A simulate_helloworld_iphone
$ ./result/bin/run-test-simulator

Which shows me the following:


In addition to the iPhone, I have also developed a function invocation simulating the iPad (as can be seen in the composition expression, shown earlier):

$ nix-build -A simulate_helloworld_ipad
$ ./result/bin/run-test-simulator

And this is the result:

Conclusion

In this blog post, I have described two Nix functions that allow users to automatically build, sign and package iOS applications and to run these in an iPhone simulator instance.

Although the Nix functions seems to work, they are still highly experimental, have some workarounds (such as the key signing) and are not entirely pure, due to the fact that it requires Xcode which must be installed through Apple's App store.

I have conducted these experiments on a Mac OS X 10.7.5 (Lion) machine. Mac OS X 10.8 Mountain Lion, is also reported to work.

I'd like to thank my colleagues Mathieu Gerard (for showing me the keysigning command-line instructions) and Alberto Gonzalez Sanchez (for testing this on Mac OS X Mountain Lion) from Conference Compass.

This function is part of Nixpkgs. The example case can be obtained from my github page.

Today, it's my blog's second anniversary, so it looks like this is a good time for some reflection.

2012 in a nutshell

Although I have fewer blog posts this year compared to last year (although the difference is not that big), 2012 was in fact an extremely busy/stressful year. First, I had to write my PhD thesis, which was quite a project. Actually, from February up to August nearly all my time was consumed writing it, when I was not sleeping or eating. Besides my thesis, also some publication work had to be done, such as the HotSWUp publication about deploying mutable software components and a journal version of the WASDeTT paper about Disnix.

Although I was very busy, I've tried to keep blogging as much as possible. In the beginning this worked out well, but later this year I had some difficulties posting anything relevant (as may be observed from April 2012 until October 2012).

I also had some depressing moments. One of them was about science in general. One of the things that made me really mad/frustrated were reviewers "complimenting" our papers with: "this is a remarkable piece of engineering" (which implies that it's not science, so the paper will be rejected). I wrote quite a lengthy blog post about this issue, and dived into literature to find an explanation, which I was able to find.

Another thing that bothered me was the fact that so little of the work that we are doing in the research world are picked up by the other fractions of the software engineering community, such as industry. Similarly, I also see that people in industry have their struggles and needs (in particular: practical application of new concepts), but refrain from finding solution elsewhere or collaborating, such as with the research world. Today, all software engineering fractions more or less all live in our separate world, while it is (obviously) better to collaborate. Although it makes sense to do that, only a few are actually doing it.

Although I had some depressing/stressful moments, I also published two blog posts at the beginning of this year, about fun projects involving my good old Amiga, with a bit of overleap to my research. Most of the work for these blog posts was already done last year, but I still had to write them down.

After finishing my thesis draft and a short break, my employment contract with the university expired and I had to switch jobs. The process of finding a new job was quite interesting. A few things that I've noticed is that only a few companies have real interest in who you are and what background you have. A lot of them use a standard application procedures, such as capacity tests, and a lot of "standard" interview questions. Most of them also treated me as an undergrad and offered me salaries way lower than my PhD salary.

Furthermore, before I started talking to these companies I considered myself still as somebody who's more like an engineer as a scientist. But when I spoke to them, I've noticed then I'm more "scientific" than I thought. Although these observations, nearly all the applications I did went fine and I have received many job offers. I was rejected one time, probably because that company was looking for somebody "who just wants to program in Java", instead of a critical person like me.

Finally, I was hired by Conference Compass as a Software Architect. Here, I still use some stuff I was working on during research, such as deployment with Nix and related tools. I've produced two blog posts related to my deployment work there -- deployment of mobile apps on Android and iOS devices.

Blog posts

Another interesting observation is the "popularity" of my blog. This year, I think I have attracted 10 times as many visitors compared to last year. Especially in the last two months, I've attracted a lot of readers, thanks to Reddit. At this moment, the top 10 of the most frequently read blog posts is:

1. On Nix and GNU Guix. Not so long ago, the GNU project had announced GNU Guix, a package manager based on Nix, implementing an internal DSL using GNU Guile, instead of using the Nix expression language, an external DSL. I wanted to give my opinion about it and the blog post has been picked up by the Reddit community, attracting a huge amount of visitors.
2. An alternative explanation of the Nix package manager. I wrote this blog post, because I had to explain Nix to my new employer. I've used a different explanation style that is language-oriented and I actually find this version a bit better that the explanation recipe I normally use. I've decided to blog about it and it has been picked up by the Reddit community, attracting a lot of readers.
3. Second computer. This is the blog post about my good ol' Commodore Amiga. It was also in last year's top 10, and it still seems to be very popular. This shows me that there are still a lot of people out there with nostalgic feelings :-).
4. An evaluation and comparison of GoboLinux. This blog post was also in last year's top 10 in which I compare the deployment properties of NixOS with GoboLinux. This blog has also been reddited some time ago.
5. Porting software to AmigaOS (unconventional style). This was a random fun experiment in which I combine some nostalgic Amiga feelings with my current research. I did not expect it to be that interesting, but for some reason it has attracted a lot of readers and even a few people who wanted to try the function I have developed.
6. NixOS: A purely functional Linux distribution. Another blog post that was in last year's top 10 and still seems to attract readers. This proves that NixOS remains interesting!
7. Software deployment complexity. One year ago, this was my most popular blog post, that's highly related to my research. It still seems to be relevant, fortunately :-).
8. Software engineering fractions, collaborations and "The System". This was a blog post I wrote this year in a depressing period, but nonetheless, I still think it's very relevant for anyone involved in the software engineering community to read. Some people say that I'm in a good shape when I'm ranting about something.
9. The Nix package manager. This was my original explanation of the Nix package manager to use the standard explanation recipe. This blog post was also in last year's top 10.
10. First blog post. Yet another blog post, that was ranked third in my top 10 last year. People still want to know who I am :-).

Some thoughts

As with my last's year reflection about my blog, I have noticed that my blog attracts many more readers than all my scientific papers combined. Another thing that I observed is that writing for my blog is more convenient than writing papers, because I'm not bound to all kinds of restrictions, such as the page limit, topics that are interesting (and are uninteresting), the amount of "greek symbols" that I should use, whether something is engineering or science etc. It feels like a big relief, it's much more fun, attracts more readers and gives me more feedback.

Actually, it seems that this observation is not unique to me and has been noticed quite some time ago already by Edsger Dijkstra, a very famous computer scientist who published most of his contributions through EWDs, manuscripts which he wrote about anything he wanted. In EWD1000 titled: 'Twenty-eight years', he reflected over his previous 28 research years and one of the interesting things he wrote in that manuscript is:

"If there is one "scientific" discovery I am proud of, it is the discovery of the habit of writing without publication in mind. I experience it as a liberating habit: without it, doing the work becomes one thing and writing it down becomes another one, which is often viewed as an unpleasant burden. When working and writing have merged, that burden has been taken away."

Most manuscripts of him after 1974 were hand-written and he never owned a computer. Because of that people consider him old fashioned. But on the other hand, he was way ahead of his time. In fact, Dijkstra's way of working could be considered an early form of blogging using a pen+paper+photocopier instead of a computer+internet.

I also think that writing more frequently and distributing your findings more often offers much more value. Quite frequently people give the expression that publishing a paper goes like this:
You reach your goal without any obstacles. But in reality it goes more like this:
You will reach many obstacles and you may not even fully reach your goal, or only under special circumstances. These obstacles are also interesting to know about, as well as how to practically implement certain aspects. It gives you better feedback, helps to "structure your mind" and to better reach your audience. Most researchers refrain from doing that.

And another thing is, if you really want any research work to succeed, you much also reach non-researchers. For this practical aspects as well as usable software is important. That's probably a big reason why this blog attract quite some readers.

Concluding remarks

In this blog post I've reflect over 2012 and it seems that it's still worth having this blog. Next year, there is more interesting stuff to come. The final thing I'd like to say is:

HAPPY NEW YEAR!!!!!

Lately, I have been doing some JavaScript programming in environments such as Node.js.

For some reason, I need to perform deployment activities from JavaScript programs. To do that, I could (re)implement the deployment mechanisms that I need from scratch, such a feature that builds packages and a feature that fetches dependencies from external repositories.

What happens in practice is that people exactly do this -- they create programming language specific package managers, implementing features that are already well supported by generic tools. Nowadays, almost every modern programming language environment has one, such as the Node.js Package Manager, CPAN, HackageDB, Eclipse plugin manager etc.

Each language-specific package manager implements deployment activities in their own way. Some of these package managers have good deployment properties, others have annoying drawbacks. Some of them can easily integrate or co-exist with the host's systems package manager, others cannot. Most importantly, they don't offer the non-functional properties I care about, such as reliable and reproducible deployment.

For me, Nix offers the stuff I want. Therefore, I want to integrate it with my programs implemented in a general purpose programming language.

A common integration solution is to generate Nix expressions through string manipulation and to invoke the nix-build (or nix-env) processes to build it. For every deployment operation or configuration step, a developer is required to generate an expression that builds or configures something (by string manipulation, that is unparsed and unchecked) and pass that to Nix, which is inelegant, laborious, tedious, error-prone, and results in more code that needs to be maintained.

As a solution for this, I have created NiJS: An internal DSL for Nix in JavaScript.

Calling Nix functions from JavaScript

As explained ealier, manually generating Nix expressions as strings have various drawbacks. In earlier blog posts, I have explained that in Nix (which is a package manager borrowing concepts from purely functional programming languages) every build action is modeled as a function and the expressions that we typically want to evaluate (or generate) are function invocations.

To me, it looks like a better and more elegant idea to be able to call these Nix functions through function calls from the implementation language, rather than generating these function invocations as strings. This is how the idea of NiJS was born.

For example, it would be nice to have the following JavaScript function invocation that calls the stdenv.mkDerivation function of Nixpkgs, which is used to build a particular package from source code and its given build-time dependencies:

stdenv().mkDerivation ({
  name : "hello-2.8",

  src : args.fetchurl({
    url : "mirror://gnu/hello/hello-2.8.tar.gz",
    sha256 : "0wqd8sjmxfskrflaxywc7gqw7sfawrfvdxd9skxawzfgyy0pzdz6"
  }),

  doCheck : true,

  meta : {
    description : "A program that produces a familiar, friendly greeting",
    homepage : {
      _type : "url",
      value : "http://www.gnu.org/software/hello/manual"
    },
    license : "GPLv3+"
  }
});

The above JavaScript example, defines how to build GNU Hello -- a trivial example package -- from source code. We can easily translate that function call to the following Nix expression containing a function call to stdenv.mkDerivation:

let
  pkgs = import <nixpkgs> {};
in
pkgs.stdenv.mkDerivation {
  name = "hello-2.8";

  src = pkgs.fetchurl {
    url = "mirror://gnu/hello/hello-2.8.tar.gz";
    sha256 = "0wqd8sjmxfskrflaxywc7gqw7sfawrfvdxd9skxawzfgyy0pzdz6";
  };

  doCheck = true;

  meta = {
    description = "A program that produces a familiar, friendly greeting";
    homepage = http://www.gnu.org/software/hello/manual;
    license = "GPLv3+";
  }
}

The above expression can be passed to nix-build to actually build GNU Hello.

To actually make the JavaScript function call work, I had to define the mkDerivation() JavaScript function as follows:

var mkDerivation = function(args) {
    return {
      _type : "nix",
      value : "pkgs.stdenv.mkDerivation "+nijs.jsToNix(args)
    };
}

The function takes an object as a parameter and returns an object with the type property set to nix (more details on this later), and a value property that is a string containing the generated Nix expression.

As you may notice from the code example, only little string manipulation is done. The Nix expression is generated almost automatically. We compose the Nix expression that needs to be evaluated, by putting the name of the Nix function that we want to call in a string and we append the output of the jsToNix() function invocation to args, the argument of the JavaScript function.

The jsToNix() function is a very powerful one -- it takes objects from the JavaScript language and translates them to Nix language objects, in a generic and straight forward manner. For example, it translates a JavaScript object into a Nix attribute set (with the same properties), a JavaScript array into an expression having a list, and so on.

The resulting object (with the nix type) can be passed to the callNixBuild() function, which adds the import <nixpkgs> {}; statement to the beginning of the expression and invokes nix-build to evaluate it, which as a side-effect, builds GNU Hello.

Defining packages in NiJS

We have just shown how to invoke a Nix function from JavaScript, by creating a trivial proxy that translates the JavaScript function call to a string with a Nix function call.

I have intentionally used stdenv.mkDerivation as an example. While I could have picked another example, such as writeTextFile that writes a string to a text file, I did not do this.

The stdenv.mkDerivation function is a very important function in Nix -- it's directly and indirectly called from almost every Nix expression building a package. By creating a proxy to this function from JavaScript, we have created a new range of possibilities -- we can also use this proxy to write package build recipes and their compositions inside JavaScript instead of the Nix expression language.

As explained in earlier blog posts, a Nix package description is a file that defines a function taking its required build-inputs as function arguments. The body of the function describes how to build the package from source code and its build-time dependencies.

In JavaScript, we can also do something like this. We can define each package as a separate CommonJS module that exports a pkg property. The pkg property refers to a function declaration, in which we describe how to build a package from source code and its dependencies provided as function arguments:

exports.pkg = function(args) {
  return args.stdenv().mkDerivation ({
    name : "hello-2.8",

    src : args.fetchurl({
      url : "mirror://gnu/hello/hello-2.8.tar.gz",
      sha256 : "0wqd8sjmxfskrflaxywc7gqw7sfawrfvdxd9skxawzfgyy0pzdz6"
    }),

    doCheck : true,

    meta : {
      description : "A program that produces a familiar, friendly greeting",
      homepage : {
        _type : "url",
        value : "http://www.gnu.org/software/hello/manual"
      },
      license : "GPLv3+"
    }
  });
};

The result of the function call is a string that contains our generated Nix expression.

As explained in earlier blog posts about Nix, we cannot use these function declarations to build a package directly, but we have to compose them by calling the function with it arguments. These function arguments provide a particular version of a dependency. In NiJS, we can compose packages in a CommonJS module that looks like this:

var pkgs = {

  stdenv : function() {
    return require('./pkgs/stdenv.js').pkg;
  },

  fetchurl : function(args) {
    return require('./pkgs/fetchurl.js').pkg(args);
  },

  hello : function() {
    return require('./pkgs/hello.js').pkg({
      stdenv : pkgs.stdenv,
      fetchurl : pkgs.fetchurl
    });
  },

  zlib : function() {
    return require('./pkgs/zlib.js').pkg;
  },

  file : function() {
    return require('./pkgs/file.js').pkg({
      stdenv : pkgs.stdenv,
      fetchurl : pkgs.fetchurl,
      zlib : pkgs.zlib
    });
  },

  ...
};

exports.pkgs = pkgs;

The above module defines a pkgs property that contains an object in which each member refers to a function. Each function invokes a CommonJS module that builds a package, such as the GNU Hello example that we have shown earlier, and passes its required dependencies as function arguments. The dependencies are defined in the same composition object, such as stdenv and fetchurl.

Apart from the language, the major difference between this composition module and the top-level composition expression in Nix, is that we have added an extra function indirection for each object member. In Nix, the composition expression is an attribute set with function calls. Because Nix is a lazy-language, these function calls only get evaluated when they are needed.

JavaScript is an eager language and will evaluate all function invocations when the composition object is generated. To prevent this, we have wrapped these invocations in functions that need to be called. This also explains why we refer to stdenv (and any other package) as a function call in the GNU Hello example.

By importing the composition expression shown earlier and by passing the result of the application of one of its function members to callNixBuild(), a package such as GNU Hello can be built:

var nijs = require('nijs');
var pkgs = require('pkgs.js').pkgs;

nijs.callNixBuild({
  nixObject : pkgs.hello(),
  onSuccess : function(result) {
    process.stdout.write(result + "\n");
  },
  onFailure : function(code) {
    process.exit(code);
  }
});

The above fragment asynchronously builds GNU Hello and writes its resulting Nix store path to the standard output (done by the onSuccess() callback function). As building individual packages from a composition specification is a common use case, I have created a command-line utility: nijs-build that can automatically do this, which is convenient for testing:

$ nijs-build pkgs.js -A hello
/nix/store/xkbqlb0w5snmrxqi6ysixfszx1wc7mqd-hello-2.8

The above command-line instruction builds GNU Hello defined in our composition JavaScript specification.

Translating JavaScript objects to Nix expression language objects

We have seen that the jsToNix() function performs most of the "magic". Most of the mappings from JavaScript to Nix are straight forward, mostly by using JavaScript's typeof operator:

• Boolean and number values can be converted verbatim
• Strings and XML objects can be almost taken verbatim, but quotes must be placed around them and they must be properly escaped.
• Objects in JavaScript are a bit trickier, as arrays are also objects. We must use the Array.isArray() method to check for this. We can recursively convert objects into either Nix lists or Nix attribute sets.
• null values can't be determined by type. We must check for the null reference explicitly.
• Objects that have an undefined type will throw an exception.

Nix provides a number of types that are not in JavaScript. Therefore, we had to artificially create them:

• Recursive attribute sets can be generated by adding the: _recursive = true member to an object.
• URLs can be defined by creating an object with the: _type = "url" member and a value member containing the URL in a string.
• Files can be defined by creating an object with the: _type = "file" member and a value containing the file path. File names with spaces are also allowed and require a special trick in the Nix expression language to allow it to work properly. Another tricky part is that file names can be absolute or relative. In order to make the latter case work, we have to know the path of to the CommonJS module that is referencing a file. Fortunately the module.filename property inside a CommonJS module will exactly tell us that. By passing the module parameter to the file object, we can make this work.

We also need to distinguish between parts that have already been converted. The proxies, such as the one to stdenv.mkDerivation are just pieces of Nix expression code that must be used without conversion. For that, I have introduced objects with the nix type, as shown earlier that are just placed into the generated expression verbatim.

Then there is still one open question. JavaScript also has objects that have the function type. What to do with these? Should we disallow them, or is there a way in which we can allow them to be used in our internal DSL?

Converting JavaScript functions to Nix expressions

I'm not aware of any Nix expression language construct that is semantically equivalent (or similar) to a JavaScript function. The Nix expression language has functions, but these are executed lazily and can only use primops in the Nix expression language. Therefore, we cannot really "compile" JavaScript functions to Nix functions.

However, I do know a way to call arbitrary processes from Nix expressions (through a derivation) and to expose them as functions that return Nix language objects, by converting the process' output to Nix expressions that are imported. I have used this trick earlier in our SEAMS paper to integrate deployment planning algorithms. I can also use the same trick to allow JavaScript functions to be called from Nix expressions:

{stdenv, nodejs}:
{function, args}:

let
  nixToJS = ...
in
import (stdenv.mkDerivation {
  name = "function-proxy";
  buildInputs = [ nodejs ];
  buildCommand = ''
    (
    cat <<EOF
    var nijs = require('${./nijs.js}');
    var fun = ${function};

    var args = [
      ${stdenv.lib.concatMapStrings (arg: nixToJS arg+",\n") args}
    ];

    var result = fun.apply(this, args);
    process.stdout.write(nijs.jsToNix(result));
    EOF
    ) | node > $out
  '';
})

The above code fragment shows how the nijsFunProxy function is defined that can be used to create a proxy to a JavaScript function and allows somebody to call it as if it was a Nix function.

The proxy takes a JavaScript function definition in a string and a list of function arguments that can be of any Nix expression language type. Then the parameters are converted to JavaScript objects (through our inverse nixToJS() function) and a JavaScript file is generated that performs the function invocation with the converted parameters. Finally, the resulting JavaScript object returned by the function is converted to a Nix expression and written to the Nix store. By importing the generated Nix expression we can return an equivalent Nix language object.

In JavaScript, every function definition is in fact an object that is an instance of the Function prototype. The length property will tell us the number of command-line arguments, the toString() method will dump a string representation of the function definition, and apply() can be used to evaluate the function object with an array of arguments.

By using these properties, we can generate a Nix function wrapper with an invocation to nijsFunProxy that looks like this:

fun =
  arg0: arg1:

  nijsFunProxy {
    function = ''
      function sumTest(a, b) {                                                                                                          
        return a + b;                                                                                                                                              
      }
    '';
    args = [ arg0 arg1 ];
  };

By creating such wrappers calling the nijsFunProxy, we can "translate" JavaScript functions to Nix and call them from Nix expressions. However, there are a number of caveats:

• We cannot use variables outside the scope of the function, e.g. global variables.
• We must always return something. If nothing is returned, we will have an undefined object, which cannot be converted. Nix 'void functions' don't exist.
• Functions with a variable number of positional arguments are not supported, as Nix functions don't support this.

It may be a bit inconvenient to use self-contained JavaScript functions, as we cannot access anything outside the scope of JavaScript function. Fortunately, the proxy can also include specified CommonJS modules, that provide standard functionality. See the package's documentation for more details on this.

Calling JavaScript functions from Nix expressions

Of course, the nijsFunProxy can also be used directly from ordinary Nix expressions, if desired. The following expression uses a JavaScript function that adds two integers. It writes the result of an addition to a file in the Nix store:

{stdenv, nijsFunProxy}:

let
  sum = a: b: nijsFunProxy {
    function = ''
      function sum(a, b) {
        return a + b;
      }
    '';
    args = [ a b ];
  };
in
stdenv.mkDerivation {
  name = "sum";

  buildCommand = ''
    echo ${toString (sum 1 2)} > $out
  '';
}

Conclusion

In this blog post, I have described NiJS: an internal DSL for Nix in JavaScript. It offers the following features:

• A way to easily generate function invocations to Nix functions from JavaScript.
• A translation function that maps JavaScript language objects to Nix expression language objects.
• A way to define and compose Nix packages in JavaScript.
• A way to invoke JavaScript functions from Nix expressions

I'd also like to point out that NiJS is not supposed to be a Nix alternative. It's rather a convenient means to use Nix from a general purpose language (in this case: JavaScript). Some of the additional use cases were implications of having a proxy and interesting to experiment with. :-)

Finally, I think NiJS may also give people that are unfamiliar with the Nix expression language the feeling that they don't have to learn it, because packages can be directly created from JavaScript in an environment that they are used to. I think this is a false sense of security. Although we can build packages from JavaScript, under the hood still Nix expressions are being generated that may yield errors. Errors from NiJS objects are much harder to debug. In such cases it's still required to know what happens.

Moreover, I find JavaScript and the asynchronous programming model of Node.js ugly and very error prone, as the language does not restrict you of doing all kinds of harmful things. But that's a different discussion. People may also find this blog post about Node.js interesting to read.

Related work

Maybe this internal DSL approach on top of an external DSL approach sounds crazy, but it's not really unique to NiJS, nor is NiJS the only internal DSL approach for Nix:

• GNU Guix is an internal DSL for Nix in Scheme (through GNU Guile). Guix is a more sophisticated approach with the intention of deploying a complete GNU distribution. It also generates lower-level Nix store derivation files, instead of Nix expressions. NiJS has a much simpler implementation, fewer use-cases and a different goal.
• ORM systems such as Hibernate can also be considered an internal DSL (Java) on-top-of an external DSL (SQL) approach. They allow developers to treat records in database tables as (collections of) objects in a general purpose programming language. They offer various advantages, such as less boilerplate code, but also disadvantages, such as the ORM mismatch, resulting in issues, such as performance penalties. NiJS has also issues related due to mismatches between the source and target language, such as debugging issues.

Availability

NiJS can be obtained from the NiJS GitHub page and used under the MIT license. The package also includes a number of example packages in JavaScript and a number of example JavaScript function invocations from Nix expressions.

As mentioned in my previous blog post, I'm doing some extensive JavaScript programming lately. It almost looks like I have become a religious JavaScript fanatical these days, but I can ensure you that I'm not. :-)

Moreover, I used to be a teaching assistant for TU Delft's concepts of programming languages course a few years ago. In that course, we taught several programming languages that are conceptually different from the first programming language students have to learn, which is Java. Java was mainly used to give students some basic programming knowledge and to teach them the basics of structured programming and object oriented programming with classes.

One of the programming languages covered in the 'concepts of programming languages' course was JavaScript. We wanted to show students that it's also possible to do object-oriented programming in a language without classes, but with prototypes. Prototypes can be used to simulate classes and class inheritance.

When I was still a student I had to do a similar exercise, in which I had to implement an example case with prototypes to simulate the behaviour of classes. It was an easy exercise, and the explanation that was given to me looked easy. I also did the exercise quite well.

Today, I have to embarrassingly admit that there are a few bits (a.k.a. nasty details) that I did not completely understand and it was driving me nuts. These are the bits that almost nobody tells you and are omitted in most explanations that you will find on the web or given by teachers. Because the "trick" that allows you to simulate class inheritance by means of prototypes is often already given, we don't really think about what truly happens. One day you may have to think about all the details and then you will probably run into the same frustrating moments as I did, because the way prototypes are used in JavaScript is not logical.

Furthermore, because this subject is not as trivial as people may think, there are dozensofarticlesandblogposts on the web, in which authors are trying the clarify the concepts. However, I have seen that a lot of these explanations are very poor, do not cover all the relevant details (and are sometimes even confusing) and implement solutions that are suboptimal. Many blog posts also copy the same "mistakes" from each other.

Therefore, I'm writing yet another blog post in which I will explain what I think about this and what I did to solve this problem. Another benefit is that this blog article is going to prevent me to keep telling others the same stuff over and over again. Moreover, considering my past career, I feel that it's my duty to do this.

Object Oriented programming

The title of this blog post contains: 'Object Oriented' (which we can abbreviate with OO). But what is OO anyway? Interestingly enough, there is no single (formal) definition of OO and not everyone shares the same view on this.

A possible (somewhat idealized) explanation is that OO programs can be considered a collection of objects that interact with each other. Objects encapsulate state (or data) and behaviour. Furthermore, they often have an analogy to objects that exist in the real world, such as a car, a desk, a person, or a shape, but this is not always the case.

For example, a person can have a first name and last name (data) and can walk and talk (behaviour). Shapes could have the following properties (state): a color, a width, height or a radius (depending on the type of shape, if they are rectangular or a circle). From these properties, we can calculate the area and the perimeter (behaviour).

When designing OO programs, we can often derive the kind of objects that we need from the nouns and the way they behave and interact from the verbs from a specification that describes what a program should do.

For example, consider the following specification:

We have four shapes. The first one is a red colored rectangle that has a width of 2 and an height of 4. The second shape is a green square having a width and height of 2. The third shape is a blue circle with a radius of 3. The fourth shape is a yellow colored circle with a radius of 4. Calculate the areas and perimeters of the shapes.

This specification may result in a design that looks as follows:



The figure above shows a UMLobject diagram, containing four rectangled shapes. Each of these shapes represent an object. The top section in a shape contains the name of the object, the middle section contains their properties (called attributes in UML) and state, and the bottom section contains its behaviour (called operations in UML).

To implement an OO program we can use an OO programming language, such as C++, Java, C#, JavaScript, Objective C, Smalltalk and many others, which is what most developers often do. However, it's not required to use an OO programming language to be Object Oriented. You can also keep track of the representation of the objects yourself. In C for example (which is not an OO language), you can use structs and pointers to functions to achieve roughly the same thing, although the compiler is not able to statically check whether you're doing the right thing.

Class based Object Oriented programming

In our example case, we have designed only four objects, but in larger programs we may have many circles, rectangles and squares. As we may observe from the object diagram shown earlier, our objects have much in common. We have designed multiple circles (we have two of them) and they have exactly the same attributes (a color and a radius) and behaviour. The only difference between each circle object is their state, e.g. they have a different color and radius value, but the way we calculate the area and perimeter remains the same.

From that observation, we could say that every circle object belongs to the same class. The same thing holds for the rectangles and squares. As it's very common to have multiple objects that have the same properties and behaviour, most OO programming languages require developers to define classes that capture these common properties. Objects in class based OO languages are (almost) never created directly, but by instantiation of a particular class.

For example, consider the following example implemented in the Java programming language that defines Person class (every person has a first and a last name from which a full name can be generated):

public class Person
{
    public String firstName, lastName;

    public Person(String firstName, String lastName)
    {
        this.firstName = firstName;
        this.lastName = lastName;
    }

    public String generateFullName()
    {
        return firstName + " " + lastName;
    }
}

Basically, the above class definition consists of three blocks. The upper block defines the attributes (first and last name), the middle block defines a constructor that is used to create a person object -- Every person is supposed to have a first and a last name. The bottom block is an operation (called method in Java) that generates the person's full name from its first and last name.

By using the new operator in Java in combination with the constructor, we can create objects that are instances of a Person class, for example:

Person sander = new Person("Sander", "van der Burg");
Person john = new Person("John", "Doe");

And we can use the generateFullName() method to display the full names of the persons. The way this is done is common for all persons:

System.out.println(sander.generateFullName()); /* Sander van der Burg */
System.out.println(john.generateFullName()); /* John Doe */

Instantiating classes have a number of advantages over directly creating objects with their state and behaviour:

• The class definition ensures that every object instance have their required properties and behaviour.
• The class definition often serves the role as a contract. We cannot give an object a property or method that is not defined in a class. Furthermore, the constructor function forces users to provide a first and last name. As a result, we cannot create person object with no first and last name.
• As objects are always instances of a class, we can easily determine whether an object belongs to a particular class. In Java this can be done using the instanceof operator:

  sander instanceof Person; /* true */
  sander instanceof Rectangle; /* false */
  

• The behaviour, i.e. the methods can be shared over all object instances, as they are the same for every object. For example, we don't have to implement the function that generates the full name for every person object that we create.

Returning to our previous example with the shapes: Apart from defining classes that capture commonalities of every object instance (i.e. we could define classes for Rectangles, Squares and Circles), we can also observe that each of these shape classes have commonalities. For example, every shape (regardless of whether it's a rectangle, square or circle) has a color, and for each shape we can calculate an area and perimeter (although the way that's done differs for each type).

We can capture common properties and behaviour of classes in so called super classes, allowing us to treat all shapes the same way. By adapting the earlier design of the shapes using classes and super classes, we could end-up with a new design that will look something like this:



In the above figure a UML class diagram is shown:

• The class diagram looks similar to the previous UML object diagram. The main difference is that the rectangled shapes are classes (not objects). Their sections still have the same meaning.
• The arrows denote extends (or inheritance) relationships between classes. Inheriting from a super class (parent) creates a sub class (child) that extends the parent's attributes and behaviour.
• On top, we can see the Shape class capturing common properties (color) and behaviour (calculate area and perimeter operations) of all shapes.
• A Rectangle is a Shape extended to have a width and height. A Circle is a shape having a radius. Moreover, both use different formulas to calculate their areas and perimeters. Therefore, the calculateArea() and calculatePerimeter() operations from the Shape class are overridden. In fact: they are abstract in the Shape class (because there is no general way to do it for all shapes) enforcing us to do so.
• The Square class is a child class of Rectangle, because we can define a squares as rectangles with equal widths and heights.

When we invoke a method of an object, first its class is consulted. If it's not provided by the class and the class has a parent class, then the parent class is consulted, then its parent's parent and so on. For example, if we run the following Java code fragment:

Square square = new Square(2);
System.out.println(square.calculateArea()); /* 4 */

The calculateArea() method call is delegated to the calculateArea() method provided by the Rectangle class (since the Square class does not provide one), which calculates the square object's area.

The instanceof() operator in Java also takes inheritance into account. For example, the following statements all yield true:

square instanceof Rectangle; /* true, Square class inherits from Rectangle */
square instanceof Shape; /* true, Square class indirectly inherits from Shape */

However the following statement yields false (since the square object is not an instance of Circle or any of its sub classes):

square instanceof Circle; /* false */

Prototype based Object Oriented programming

Apart from the length, the attractive parts described in this blog post about OO programming is that OO programs often (ideally?) draw a good analogy to what happens in the real world. Class based OO languages enable reuse and sharing. Moreover, they offer some means of enforcing that objects constructed the right away, i.e. that they have their required properties and intended behaviour.

Most books and articles explaining OO programming immediately use the term classes. This is probably due to the fact that most commonly used OO languages are class based. I have intentionally omitted the term classes for a while. Moreover, not everyone thinks that classes are needed in OO programming.

Some people don't like classes -- because they often serve as contracts, they can also be too strict sometimes. To deviate from a contract, either inheritance must be used that extends (or restricts?) a class or wrapper classes must be created around them (e.g. through the adapter design pattern). This could result in many layers of glue code, significantly complicating programs and growing it unnecessarily big, which is not uncommon for large systems implemented in Java.

There is also a different (and perhaps a much simpler) way to look at OO programming. In OO languages such as JavaScript (and Self, which greatly influenced JavaScript) there are no classes. Instead, we create objects, their properties and behaviour directly.

In JavaScript, most language constructs are significantly simpler than Java:

• Objects are associative arrays in which each member refers to other objects.
• Arrays are objects with numerical indexes.
• Functions are also objects and can be assigned to variables and object members. This his how behaviour can be implemented in JavaScript. Functions also have attributes and methods. For example, toString() returns the function's implementation code and length() returns the number of parameters the function requires.

"Simpler" languages such as JavaScript have a number of benefits. It's easier to learn as fewer concepts have to be understood/remembered and easier to implement by language implementers. However, as we create objects, their state and behaviour directly, you may wonder how we can share common properties and behaviour among objects or how we can determine whether an object belongs to a particular class? There is a different mechanism to achieve such goals: delegation to prototypes.

In prototype based OO languages, such as Self and JavaScript, every object has a prototype. Prototypes are also objects (having their own prototype). The only exception in JavaScript is the null object, which prototype is a null reference. When requesting an attribute or invoking a method of an object, first the object itself is consulted. If the object does not implement it, it consults its prototype, then the prototype's prototype and so on.

Although prototype based OO languages do not provide classes, we can simulate class behaviour (including inheritance) through prototypes. For example, we can simulate the two particular person objects that are an instance of a Person class (as shown earlier) as follows:



The above figure shows an UML object diagram (not a Class diagram, since we don't have classes ;) ) in which we simulate class instantation:

• As explained earlier in this blog post, objects belonging to a particular class have common behaviour, but their state differs. Therefore, the attributes and their state have to be defined and stored in every object instance (as can be observed from the object diagram from the fact that every person has its own first and last name property).
• We can capture common behaviour among persons in a common object, that will serve as the prototype of every person. This object serves the equivalent role of a class. Moreover, since each person instance can refer to exactly the same prototype object, we also have sharing and reuse.
• When we invoke a method on a person object, such as generateFullName() then the method invocation is delegated to the Person prototype object, which will give us the person's full name. This exactly offers us the same behaviour as we have in a class based OO language.

By using multiple layers of indirection through prototypes we can simulate class inheritance. For example, to define a super class of a class, we set the prototype's prototype to an object capturing the behaviour of the super class. The following UML object diagram shows how we can simulate our earlier example with shapes:



As can be observed from the picture: if we would invoke the calculateArea() method on the square object (shown at the bottom), then the invocation is delegated to the square prototype's prototype (which is an object representing the Rectangle class). That method will calculate the area of the square for us.

We can also use prototypes to determine whether a particular object is an instance of a (simulated) class. In JavaScript this is done by checking the prototype chain to see whether there is an object that has exactly the same properties as the simulated class.

Simulating classes in JavaScript

So far I think the concepts of simulating classes and class inheritance through prototypes are clear. In short, there are three things we must remember:

• A class can be simulated by creating a singleton object capturing its behaviour (and state that is common to all object instances).
• Instantiation of class can be simulated by creating an object having a prototype that refers to the simulated class object and by calling the constructor that sets its state.
• Class inheritance can be simulated by setting a simulated class object's prototype to the simulated parent class object.

The remaining thing I have to explain is how to implement our examples in JavaScript. And this is where the pain/misery starts. The main reason of my frustration comes from the fact that every object in JavaScript has a prototype, but we cannot (officially) see or touch them directly.

You may probably wonder why have I used the word 'officially'? In fact, there is a way to see or touch prototypes in Mozilla-based browsers, through the hidden __proto__ object member, but this is a non-standard feature, does not officially exist, should not be used and does not work in many other JavaScript implementations. So in practice, there is no way to access an object's prototype directly.

So how do we 'properly' work with prototypes in JavaScript? We must use the new operator, that looks very much like Java's new operator, but don't let it fool you! In Java, new is called in combination with the constructor defined in a class to create an object instance. Since we don't have classes in JavaScript, nor language constructs that allow us to define constructors, it achieves its goal in a different way:

function Person(firstName, lastName) {
    this.firstName = firstName;
    this.lastName = lastName;
}

var sander = new Person("Sander", "van der Burg");
var john = new Person("John", "Doe");

In the above JavaScript code fragment, we define the constructor of the Person class as a (ordinary) function, that sets a person's first and last name. We have to use the new operator in combination with this function to create a person object.

What does JavaScript's new operator do? It creates an empty object, calls the constructor function that we have provided with the given parameters, and sets this to the empty object that it has just created. The result of the new invocation is an object having a first and last name property, and a prototype containing a constructor property that refers to our constructor function.

So how do we create a person object that has the Person class object as its prototype, so that we can share its behaviour and determine to which class an object belongs? In JavaScript, we can set the prototype of an object to be constructed as follows:

function Person(firstName, lastName) {
    this.firstName = firstName;
    this.lastName = lastName;
}

Person.prototype.generateFullName = function() {
    return this.firstName + " " + this.lastName;
};

/* Shows: Sander van der Burg */
var sander = new Person("Sander", "van der Burg");

/* Shows: John Doe */
var john = new Person("John", "Doe");

document.write(sander.generateFullName() + "<br>\n");
document.write(john.generateFullName() + "<br>\n");

To me the above code sample looks a bit weird. In the code block after the function definition, we adapt the prototype object member of the constructor function? What the heck is this and how does this work?

In fact, as I have explained earlier, functions are also objects in Java and we can also assign properties to them. The prototype property is actually not the prototype of the function (as I have said: prototypes are invisible in JavaScript). In our case, it's just an ordinary object member.

However, the prototype member of an object is used by the new operator that creates objects. If we call new in combination with a function that has a prototype property, then the resulting object's real prototype refers to that prototype object. We can use this to allow an object instance's prototype to refer to a class object. Moreover, the resulting prototype always refers to the same prototype object (namely Person.prototype), that allows us to share common class properties and behaviour. Got it? :P

If you didn't get it (for which I can't blame you): The result of the new invocations in our last code fragment exactly yields the object diagram that I have shown in the previous section containing person objects that refer to a Person prototype.

Simulating class inheritance in JavaScript

Now that we "know" how to create objects that are instances of a class in JavaScript, there is even a bigger pain. How to simulate class inheritance in JavaScript? This was something that really depressed me.

As explained earlier, to create a class that has a super class, we must set the prototype of a class object to point to the super class object. However, since we cannot access or change object's prototypes directly, it's a bit annoying to do this. Let's say that we want to create an object that is an instance of a Rectangle (which inherits from Shape) and calculate its area. A solution that I have seen in quite some articles on the web is:

/* Shape constructor */
function Shape(color) {
    this.color = color;
}

/* Shape behaviour (does nothing) */
Shape.prototype.calculateArea = function() {
    return null;
};

Shape.prototype.calculatePerimeter = function() {
    return null;
};

/* Rectangle constructor */
function Rectangle(color, width, height) {
    Shape.call(this, color); /* Call the superclass constructor */
    this.width = width;
    this.height = height;
}

/* Rectangle inherits from Shape */
Rectangle.prototype = new Shape();
Rectangle.prototype.constructor = Rectangle;

/* Rectangle behaviour */
Rectangle.prototype.calculateArea = function() {
    return this.width * this.height;
};

Rectangle.prototype.calculatePerimeter = function() {
    return 2 * this.width + 2 * this.height;
};

/* Create a rectangle instance and calculate its area */
var rectangle = new Rectangle("red", 2, 4);
document.write(rectangle.calculateArea() + "<br>\n");

The "trick" to simulate class inheritance they describe is to instantiate the parent class (without any parameters), setting that object as the child class prototype object and then adding the child class' properties to it.

The above solution works in most cases, but it's not very elegant and a bit inefficient. Indeed, the resulting object has a prototype that refers to the parent's prototype, but we have a number of undesired side effects too. By running the base class' constructor we will do some obsolete work - it now assigns undefined properties to a newly created object that we don't need. Because of this, the Rectangle prototype now looks like this:



It also stores undefined class properties in the Rectangle class object, which is unnecessary. We only need an object that has a prototype referring to the parent class object, nothing more. Furthermore, a lot of people may also build checks in constructors that may throw exceptions if certain parameters are unspecified.

A better solution would be to create an empty object which prototype refers to the parent's class object, which we can extend with our child class properties. To do that we can use a dummy constructor function that just returns an empty object:

function F() {};

Then we set the prototype property of the dummy function to the Shape constructor function's prototype object member (the object representing the Shape class):

F.prototype = Shape.prototype;

Then we call the new operator in combination with F (our dummy constructor function). We'll get an empty object having the Rectangle class object as its prototype. We can use this object as a basis for the prototype that defines the Rectangle class:

Rectangle.prototype = new F();

Then we must fix the Rectangle class object's constructor property to point to the Rectangle constructor, because now it has been set to the parent's constructor function (due to calling the new operation previously):

Rectangle.prototype.constructor = Rectangle;

Finally, we can add our own class methods and properties to the Rectangle prototype, such as calculateArea() and calculatePerimeter(). Still got it? :P

Since the earlier procedure is so weird and complicated (I don't blame you if you don't get it :P), we can also encapsulate the weirdness in a function called inherit() that will do this for any class:

function inherit(parent, child) {
    function F() {}; 
    F.prototype = parent.prototype; 
    child.prototype = new F();
    child.prototype.constructor = child;
}

The above function takes a parent constructor function and child constructor as function parameters. It points the child constructor function's prototype property to an empty object which prototype points to the super class object. After calling this function, we can extend the subclass prototype object with class members of the child class. By using the inherit() function, we can rewrite our earlier code fragment as follows:

/* Shape constructor */
function Shape(color) {
    this.color = color;
}

/* Shape behaviour (does nothing) */
Shape.prototype.calculateArea = function() {
    return null;
};

Shape.prototype.calculatePerimeter = function() {
    return null;
};

/* Rectangle constructor */
function Rectangle(color, width, height) {
    Shape.call(this, color); /* Call the superclass constructor */
    this.width = width;
    this.height = height;
}

/* Rectangle inherits from Shape */
inherit(Shape, Rectangle);

/* Rectangle behaviour */
Rectangle.prototype.calculateArea = function() {
    return this.width * this.height;
};

Rectangle.prototype.calculatePerimeter = function() {
    return 2 * this.width + 2 * this.height;
};

/* Create a rectangle instance and calculate its area */
var rectangle = new Rectangle("red", 2, 4);
document.write(rectangle.calculateArea() + "<br>\n");

The above example is the best solution I can recommend to properly implement simulated class inheritance.

Discussion

In this lengthy blog post I have explained two ways of doing object programming: with classes and with prototypes. Both approaches makes sense to me. Each of them have advantages and disadvantages. It's up to developers to make a choice.

However, what bothers me is the way prototypes are implemented in JavaScript and how they should be used. From my explanation, you will probably conclude that it completely sucks and makes things extremely complicated. This is probably the main reason why there is so much stuff on this subject on the web.

Some people argue that classes should be added to JavaScript. For me personally, there is nothing wrong with using prototypes. The only problem is that JavaScript does not allow people to use them properly. Instead, JavaScript exposes itself as a class based OO language, while it's prototype based. The Self language (which influenced JavaScript) for instance, does not hide its true nature.

Another minor annoyance is that if you want to properly simulate class inheritance in JavaScript, the best solution is probably to steal my inherit() function described here. You can probably find dozens of similar functions in many other places on the web.

You may probably wonder why JavaScript sucks when it comes to OO programming? JavaScript was originally developed by Netscape as part of their web browser, in a rough period known as the browser wars, in which there was heavy competition between Netscape and Microsoft.

At some point Netscape bundled the Java platform with its web browser allowing developers to embed Java Applets in web pages, which are basically advanced computer programs. They also wanted to offer a light weight alternative for less technical users that had to look like Java, which had to be implemented in a short time span.

They didn't want to design and implement a new language completely from scratch. Instead, they took an existing language (probably Self) and adapted it to have curly braces and Java keywords, to make it look a bit more like Java. Although Java and JavaScript have some syntactic similarities, they are in fact fundamentally different languages. JavaScript hides its true nature, such as the fact that it's prototype based. Nowadays, its popularity has significantly increased and we use it for many other purposes in addition to the web browser.

Fortunately, the latest ECMAScript standard and recent JavaScript implementations ease the pain a little. New implementations have the Object.create() method allowing you to directly create an object with a given prototype. There is also a Object.getPrototypeOf() which gives you read-only access to a prototype of any object. However, to use these functions you need a modern browser or JavaScript runtime (which a lot of people don't have). It will probably take several years before everybody has updated their browsers to versions that support these.

References

In the beginning of this blog post, I gave a somewhat idealized explanation of Object Oriented programming. There is no uniform definition of Object-Oriented programming and its definition seems to be still in motion. On the web I found an interesting blog post written by William Cook, titled: 'A Proposal for Simplified, Modern Definitions of "Object" and "Object Oriented', which may be interesting to read.

Moreover, this is not the only blog post about programming languages concepts that I wrote. A few months ago I also wrote an unconventional blog post about (purely) functional programming languages. It's a bit unorthodox, because I draw an analogy to package management.

Finally, Self is a relatively unknown language for the majority of developers in the field. Although it started a long time ago and it's considered an ancient language, to me it looks very simple and powerful. It's a good lesson for people who want to design and implement an OO language. I could recommend everybody to have a look at the following video lecture from Stanford University about Self. Moreover, Self is still maintained and can be obtained from the Self language website.

After my tiny JavaScript depression, I have made some interesting improvements to NiJS which I'd like to describe in this blog post.

Defining non-native types in JavaScript

As explained in my previous blog post about NiJS, we can "translate" most JavaScript language objects to equivalent/similar Nix expression language constructs. However, the Nix expression language has certain value types, such as files and a URLs, that JavaScript does not have. In the old NiJS implementation, objects of these types can be artificially created by adding a _type object member that refers to a string containing url or file.

For various reasons, I have never really liked that approach very much. Therefore, I have adapted NiJS to use prototypes to achieve the same goal, as I now have discovered how to properly use them in JavaScript (which is not logical if you'd ask me). To create a file, we must create an object that is an instance of the NixFile prototype. URLs can be created by instantiating NixURL and recursive attribute sets by instantiating NixRecursiveAttrSet.

By adapting the GNU Hello NiJS package from the previous NiJS blog post using prototypes, we end up with the following CommonJS module:

var nijs = require('nijs');

exports.pkg = function(args) {
  return args.stdenv().mkDerivation ({
    name : "hello-2.8",

    src : args.fetchurl({
      url : new nijs.NixURL("mirror://gnu/hello/hello-2.8.tar.gz"),
      sha256 : "0wqd8sjmxfskrflaxywc7gqw7sfawrfvdxd9skxawzfgyy0pzdz6"
    }),

    doCheck : true,

    meta : {
      description : "A program that produces a familiar, friendly greeting",
      homepage : new nijs.NixURL("http://www.gnu.org/software/hello/manual"),
      license : "GPLv3+"
    }
  });
};

In my opinion, this looks a bit better and more logical.

Using CommonJS modules in embedded JavaScript functions

In the previous NiJS blog post, I have also shown that we can wrap JavaScript functions in a function proxy and call them from Nix expressions as if they were Nix functions.

One of the things that is a bit inconvenient is that these functions must be self-contained -- they cannot refer to anything outside the function's scope and we have to provide everything the function needs ourselves.

For me it's also useful to use third party libraries, such as the Underscore library. I have adapted the nijsFunProxy with two extra optional parameters. The modules parameter can be used to refer to external Node.JS packages (provided by Nixpkgs), that are added to the buildInputs of the proxy. The requires parameter can be used to generate require() invocations that import the CommonJS modules that we want to use.

The following example expression invokes a JavaScript function that utilises and imports the Underscore library to convert an array of integers to an array of strings:

{stdenv, nodejs, underscore, nijsFunProxy}:

let
  underscoreTestFun = numbers: nijsFunProxy {
    function = ''
      function underscoreTestFun(numbers) {
        var words = [ "one", "two", "three", "four", "five" ];
        var result = [];

        _.each(numbers, function(elem) {
          result.push(words[elem - 1]);
        });

        return result;
      }
    '';
    args = [ numbers ];
    modules = [ underscore ];
    requires = [
      { var = "_"; module = "underscore"; }
    ];
  };
in
stdenv.mkDerivation {
  name = "underscoreTest";

  buildCommand = ''
    echo ${toString (underscoreTestFun [ 5 4 3 2 1 ])} > $out
  '';
}

In the above expression, the requires parameter to the proxy generates the following line before the function definition, allowing us to properly use the functions provided by the Underscore library:

var _ = require('underscore');

Calling asynchronous JavaScript functions from Nix expressions

This nijsFunProxy is interesting, but most of the functions in the NodeJS API and external NodeJS libraries are executed asynchronously, meaning that they will return immediately and invoke a callback function when the work is done. We cannot use these functions properly from a proxy that is designed to support synchronous functions.

I have extended the nijsFunProxy to take the async parameter, which defaults to false. When the async parameter is set to true, it does not take the return value of the function, but waits until a callback function is called (nijsCallbacks.onSuccess) with a JavaScript object as parameter serving the equivalent of a return value. This can be used to invoke asynchronous JavaScript functions from a Nix function.

{stdenv, nijsFunProxy}:

let
  timerTest = message: nijsFunProxy {
    function = ''
      function timerTest(message) {
        setTimeout(function() {
          nijsCallbacks.onSuccess(message);
        }, 3000);
      }
    '';
    args = [ message ];
    async = true;
  };
in
stdenv.mkDerivation {
  name = "timerTest";

  buildCommand = ''
    echo ${timerTest "Hello world! The timer test works!"} > $out
  '';
}

The above Nix expression shows a simple example, in which we create a timeout event after three seconds. The timer callback calls the nijsCallbacks.onSuccess() callback function to provide a return value containing the message that the user has given as a parameter to the Nix function invocation.

Writing inline JavaScript code in Nix expressions

NiJS makes it possible to use JavaScript instead of the Nix expression language to describe package definitions and their compositions, although I have no reasons to recommend the former over the latter.

However, the examples that I have shown so far in the blog posts use generic build procedures that basically execute the standard GNU Autotools build procedure: ./configure; make; make install. We may also have to implement custom build steps for packages. Usually this is done by specifying custom build steps in Bash shell code embedded in strings.

Not everyone likes to write shell scripts and to embed them in strings. Instead, it may also be desirable to use JavaScript for the same purpose. I have made this possible by creating a nijsInlineProxy function, that generates a string with shell code executing NodeJS to execute a piece of JavaScript code within the same build process.

The following Nix expression uses the nijsInlineProxy to implement the buildCommand in JavaScript instead of shell code:

{stdenv, nijsInlineProxy}:

stdenv.mkDerivation {
  name = "createFileWithMessage";
  buildCommand = nijsInlineProxy {
    requires = [
      { var = "fs"; module = "fs"; }
      { var = "path"; module = "path"; }
    ];
    code = ''
      fs.mkdirSync(process.env['out']);
      var message = "Hello world written through inline JavaScript!";
      fs.writeFileSync(path.join(process.env['out'], "message.txt"), message);
    '';
  };
}

The above Nix expression creates a Nix store output directory and writes a message.txt file into the corresponding output directory.

As with ordinary Nix expressions, we can refer to the parameters passed to stdenv.mkDerivation as well as the output folder by using environment variables. Inline JavaScript code has the same limitations as embedded JavaScript functions, such as the fact that we can't refer to global variables.

Writing inline JavaScript code in NiJS package modules

If we create a NiJS package module, we also have to use shell code embedded in strings to implement custom build steps. We can also use inline JavaScript code by creating an object that is an instance of the NixInlineJS prototype. The following code fragment is the NiJS equivalent of the previous Nix expression:

var nijs = require('nijs');

exports.pkg = function(args) {
  return args.stdenv().mkDerivation ({
    name : "createFileWithMessageTest",
    buildCommand : new nijs.NixInlineJS({
      requires : [
        { "var" : "fs", "module" : "fs" },
        { "var" : "path", "module" : "path" }
      ],
      code : function() {
        fs.mkdirSync(process.env['out']);
        var message = "Hello world written through inline JavaScript!";
        fs.writeFileSync(path.join(process.env['out'], "message.txt"), message);
      }
    })
  });
};

The constructor of the NixInlineJS prototype can take two types of parameters. It can take a string containing JavaScript code or a JavaScript function (that takes no parameters). The latter case has the advantage that its syntax can be checked by an interpreter/compiler and that we can use syntax highlighting in editors.

By using inline JavaScript code in NiJS, we can create Nix packages by only using JavaScript. Isn't that awesome? :P

Conclusion

In this blog post, I have described some interesting improvements to NiJS, such as the fact that we can create Nix packages by only using JavaScript. NiJS seems to be fairly complete in terms of features now.

If you want to try it, the source can be obtained from the NiJS GitHub page, or installed from Nixpkgs or by NPM.

Another thing that's in my mind now is whether I can do the same stuff for a different programming language. Maybe when I'm bored or I have a use case for it, I'll give it a try.

Recently, I have set up a continuous integration facility at work, because we think that there is a need for such a facility.

While developing software systems, there are all kinds of reasons that may (potentially) break something. For example, by committing a broken piece of source code to a master repository or by upgrading one its dependencies, such as a library, to an incompatible version.

All these potentially breaking changes may result in a product that behaves incorrectly, which may not be delivered in time, because there is usually a very large time window in which it's not known if the system still behaves correctly. In such cases, it is a quite an effort to get all bugs fixed (because the effort of fixing every bug is cumulative) and to predict how long it will take to deliver a new version.

A solution for this is continuous integration (CI) or daily builds -- a practice in which all components of a software system are integrated at least once every day (preferably more often) and automatically verified by an automated build solution reporting about the impact of these changes as quickly as possible. There are many automated integration solutions available, such as Buildbot, Jenkins, Tinderbox, CruiseControl, and Hydra.

Hydra: the Nix-based continuous integration server

As some of you may know, I have written many articles about Nix and their applications on this blog. One of the applications that I have mentioned a few times, but didn't cover in detail is Hydra, a Nix-based continuous integration facility.

For people that know me, it's obvious that I have picked Hydra, but apart from the fact that I'm familiar with it, there are a few other reasons to pick Hydra over other continuous integration solutions:

• In Hydra, the build environments are managed, which is typically not the case with other CI solutions. In other CI solutions, dependencies happen to be already there and it's up to the system administrator and the tools that are invoked to ensure that they are present and correct. Since Hydra builds on top of Nix, it uses the Nix package manager to install all its required dependencies automatically and on-demand, such as compilers, linkers etc. Moreover, because of Nix's unique facilities, all dependencies are ensured to be present and package installations do not affect other packages, as they are stored safely in isolation from each other.
  
  Apart from building components, this is also extremely useful for system integration tests. The Nix package manager ensures that all the required prerequisites of a test are present and that they exactly conform to the versions that we need.
  
  Similarly, because in Nix every dependency is known, we can also safely garbage collect components that are no longer in use without having to worry that the components that are still in use are removed.
• Hydra has strong support for specifying and managing variability. For us it's important to to be able to build a certain package for many kinds of platforms and various kinds of versions of the same platform. We don't want to accidentally use a wrong version.
  
  Since Hydra uses Nix to perform builds, dependencies can only be found when they are specified. For example, if we insist on using an older version of the GCC compiler, then Nix ensures that only the older compiler version can be found. This can be done for any build-time dependency, such as the linker, the Java runtime environment, libraries etc.
  
  With other CI solutions it's much harder to ensure that an already installed version of e.g. the GCC compiler does not conflict with a different version, as it's harder to safely install two versions next to each other and to ensure that the right version is used. For example, if the default compiler in the PATH is GCC 4.7, how can we be absolutely sure that we use an older version without invoking any of the newer compiler's components?
• Hydra also has good facilities to perform builds and tests on multiple operating systems. When the Nix packager is requested to build something it always takes the requested architecture identifier into account. If the Nix package manager is unable to perform a build for the requested architecture on a given machine, it can delegate a build to another machine capable of building it, giving the caller the impression that the build is performed locally.
• Another interesting feature is scalability. Since for every build the Nix package manager is used, that invokes functions that are referentially transparent (under normal circumstances) due to its underlying purely functional nature, we can easily divide the build of a package and their dependencies among machines in a network.

Hydra components

Hydra is composed of several components. Obviously, Nix is one of them and does the majority of the work -- it's used to take job definitions specified in the Nix expression language and to execute them in a reliable and reproducible manner (which is analogous to building packages through Nix). Moreover, it can delegate job executions to machines in a network.

Besides Nix, Hydra consists of three other major components:

• The scheduler regularly checks out configured source code repositories (e.g. managed by a VCS, such as Git or Subversion) and evaluates the Nix expressions that define jobsets. Each derivation in a Nix expression (in many cases a direct or indirect function invocation to stdenv.mkDerivation) gets appended to the build queue. Every derivation corresponds to a job.
• The queue runner builds the derivations that are queued by the evaluator through the Nix package manager.
• The server component is a web application providing end-users a pretty interface which they can use to configure projects and jobsets and inspect the status of the builds.

Usage: Projects

The following screenshot shows the entry page of the Hydra web front-end showing an overview of projects. On the highest level, every job that Hydra executes belongs to a certain project:

Administrators are free to choose any name and description of a project. Projects can be created, edited and deleted through the Hydra web interface. In this example, I have defined four projects: Disnix (a tool part of the Nix project which I have developed), the Nix android test case, the Nix xcode test case, and KitchenSink -- a showcase example for the Titanium SDK, a cross-platform mobile development kit.

Clicking on a project will show you its jobsets.

Usage: Jobsets

Every project contains zero or more jobsets. A jobset contains one or more jobs that execute something, typically a build or a test case. The following screenshot shows you the jobsets that I have defined for the nix-androidenvtests project:


As we can see, we have defined only one jobset, which regularly checks out the master Git repository of the example case and builds it. Jobsets can be defined to build from any source code repository and can have any name. Quite frequently, I use jobsets to define sub projects or to make a distinction between branches of the same repository, e.g. it may be desirable to execute the same jobs on a specific experimental branch of a source code repository.

In the Disnix project, for example, I define jobsets for every sub project:


Besides a listing of jobsets, the screen also shows you their statuses. The green colored icons indicate how many jobs have succeeded, and the red colored icons show you the amount of jobs that have failed. Grey icons show the amount of jobs that are queued and still have to be executed.

To create jobsets, we have to perform two steps. First, we must define a jobset Nix expression and store it in a source code repository, such as a Git repository or on the local filesystem. The nix-androidenvtests-master jobset expression looks as follows:

{nixpkgs, system}:

let
  pkgs = import nixpkgs { inherit system; };
in
rec {
  myfirstapp_debug = import ./myfirstapp {
    inherit (pkgs) androidenv;
    release = false;
  };

  myfirstapp_release = import ./myfirstapp {
    inherit (pkgs) androidenv;
    release = true;
  };

  emulate_myfirstapp_debug = import ./emulate-myfirstapp {
    inherit (pkgs) androidenv;
    myfirstapp = myfirstapp_debug;
  };

  emulate_myfirstapp_release = import ./emulate-myfirstapp {
    inherit (pkgs) androidenv;
    myfirstapp = myfirstapp_release;
  };
}

The above Nix expression defines the nix-androidenvtests-master jobset, shown in the first jobsets screenshot. A jobset Nix expression is a function returning an attribute set, in which each attribute defines a job. Every job can be either a direct function invocation to a function that builds/executes something, or a function taking dependencies of the given job returning a function invocation that builds/execute something.

Functions in a jobset expression are used to define its variation points. For example, in our above expression we have only two of them -- the nixpkgs parameter refers to a checkout of the Nixpkgs repository, that contains a collection of common packages, such as compilers, linkers, libraries, end-user software etc. The system parameter refers to the system architecture that we want to build for, which can be (for example) 32-bit Linux machines, 64-bit Linux machines, 64-bit Mac OS X and so on.

Nix expressions defining packages and Nix expressions defining jobsets are quite similar. Both define a function (or functions) that build something from given function arguments. They both cannot be used to build something directly, but we must also compose the builds by calling them with the right function arguments. Function arguments specify (for example) which versions of packages we want to use and for what kind of system architectures we want to build. For ordinary Nix packages, this is done in the all-packages.nix composition expression. For jobsets, we use the Hydra web interface to configure them.

In the following screenshot, we configure the nix-androidenvtests jobset (done by picking the 'Create jobset' function from the Projects menu) and their parameters through the Hydra web interface:

As can be observed from the screenshot, we configure the nix-androidenvtests jobset as follows:

• The identifier is a unique name identifying the jobset, and a description can be anything we want.
• The Nix expression field defines what jobset Nix expression file we must use and in which input it is stored. The Nix expression that we have just shown, is stored inside the Nix android tests GitHub repository. Its relative path is deployment/default.nix. The name of the input that refers to the Git checkout named nixandroidenvtests, which is later declared in the build inputs section.
• We can configure e-mail notifications so that you will be informed if a certain builds succeeds or fails.
• We can also specify how many successful builds we want to keep. Older builds or failed builds will be automatically garbage collected once in a while.
• In the remaining section of the screen, we can configure the jobset's build inputs, which basically provide parameters to the functions that we have seen in the jobset Nix expression.
  
  The first build input is a Git checkout of the master branch of the Android testcase. As we have seen earlier, this build input provides the Nix jobset expression. The second parameter provides a checkout of the Nixpkgs repository. In our case it's configured to take the last checkout of the master branch, but we can also configure it to take e.g. a specific branch. The third parameter specifies for which platforms we want to build. We have specified three values: 32-bit Linux (i686-linux), 64-bit Linux (x86_64-linux), and 64-bit Mac OS X (x86-64-darwin). If multiple values are specified for a specific input, Hydra will evaluate the Cartesian product of all inputs, meaning that (in this case) we build the same jobs for 32-bit + 64-bit Linux + 64-bit Mac OS X simultaneously.

The Android testcase is a relatively simple project with simple parameters. Hydra has many more available options. The following screenshot shows the jobset configuration of disnix-trunk:


In the above configuration, apart from strings and VCS checkouts, we use a number of other interesting parameter types:

• The disnix jobset contains a job named tarball that checks out the source code and packages it in a compressed tarball, which is useful for source releases. To build the binary package, we use the tarball instead of the checkout directly.
  
  We can reuse the output of an existing Hydra job (named tarball) by setting a build input type to: 'Build output'. The build output is allowed to be of any platform type, which is not a problem as the tarball should always be the same.
• We can also refer to a build of a different jobset, which we have done for disnix_activation_scripts, which a dependency of Disnix. Since we have to run that package, it must be built for the same system architecture, which can be forced by setting the type to: 'Build output (same system)'.

The queue

If we have defined a jobset and if its enabled, then the evaluator should regularly check it, and their corresponding jobs should appear in the queue view (which can be accessed from the menu by picking Status -> Queue):

Inspecting job statuses

To inspect the build results, we can pick a jobset and open the 'Job status' tab. The following image shows the results of the nix-androidenvtests-master jobset:


As can observed from the screenshot, we have executed its jobs on three platforms and all the builds seem to have succeeded.

Obtaining build products

In addition to building jobs and performing testcases, it is also desired to obtain the build artifacts produced by Hydra so that we can run or inspect them locally. There are various ways to do this. The most obvious way is to click on a build result, that will redirect you its status page.

The following page shows you the result for the emulate_myfirstapp_release, a job that produces a script starting the Android emulator running the release version of the demo app:

By taking the url to which an one click install link points and by running the following instruction (Nix is required to be present on that machine):

$ nix-install-package --url http://nixos/build/169/nix/pkg/MyFirstApp-x86_64-linux.nixpkg

The package gets downloaded and imported in the Nix store of the machine and can be accessed under the same Nix store path as the status page shows (which is /nix/store/z8zn57...-MyFirstApp. By running the following command-line instruction:

$ /nix/store/z8zn57...-MyFirstApp/bin/run-test-emulator

We can automatically launch an emulator instance running the recently built app.

For mobile apps (that are typically packaged inside an APK bundle for Android or an IPA bundle for iOS), or documents (such as PDF files) it may be a bit inconvenient to fetch and import the remote builds into the Nix store of the local machine to view or to test them. Therefore, it is also possible to declare build products inside a build job.

For example, I have adapted the buildApp {} functions for Android and iOS apps, shown in my earlier blog posts, to declare the resulting bundle as build product, by appending the following instructions to the build procedure:

mkdir -p $out/nix-support
echo "file binary-dist \"$(echo $out/*.apk)\"" \
> $out/nix-support/hydra-build-products

The above shell commands adds a nix-support/hydra-build-products file to the resulting package, which is a file that Hydra can parse. The first two columns define the type and subtype for the build product, the third column specifies the path to the build product.

As a result, building an App with Hydra produces the following status page:

As can be observed, the page provides a convenient download link that allows us to download the APK file. Another interesting benefit is that I can use my Android phone to browse to this result page and to install it automatically, saving me a lot of effort. ;)

The most powerful mechanism to obtain builds is probably the Nix channel mechanism. Every project and jobset have a channel, which info page can be accessed by selecting the 'Channel' option from the project or jobset menu. The following screenshot shows the Nix channel page for the nix-androidenvtests-master jobset:


The above page displays some instructions that users having the Nix package manager installed must follow. By adding the Nix channel and updating it, the user receives a collection of Nix expressions and a manifest with a list of binaries. By installing a package that's defined in the channel, the Nix package gets downloaded automatically and added to the user's Nix profile.

Experience

We are using Hydra for a short time now at Conference Compass, primarily for producing mobile apps for the Android and iPhone (which is obvious to people that know me). It's already showing us very promising results. To make this possible I'm using the Nix functions that I have described in earlier blog posts.

I have also worked with the Hydra developers to make some small changes to make my life more convenient. I have contributed a patch that allows build products with spaces in them (as this is common for Apps), and I have adapted/fixed the included NixOS configuration module so that Hydra can be conveniently installed in NixOS using a reusable module. I'd like to thank the Hydra developers: Eelco Dolstra, Rob Vermaas, Ludovic Courtès, and Shea Levy for their support.

Conclusion

In this blog post, I have given an incomplete tour of Hydra's features with some examples I care about. Apart from the features that I have described, Hydra has a few more facilities, such as creating views and releases and various management facilities. The main reason for me to write this blog post is to keep the explanation that I have given to my colleagues as a reference.

This blog post describes Hydra from a user/developer perspective, but I also had to deploy a cluster of build machines, which is also interesting to report about. In the next blog post, I will describe what I did to set up a Hydra cluster.

References

For people that are eager to try Hydra: Hydra is part of the Nix project and available as free and open-source software under the GPLv3 license.

There are also a number of publications available about Nix buildfarms. The earliest prototype is described in Eelco Dolstra's PhD thesis. Furthermore, on Eelco's publications page a few papers about the Nix buildfarm can be found.

Another thing that may be interesting to read about is an old blog of me titled: 'Using NixOS for declarative deployment and testing' describing how we can execute distributed system integration tests using NixOS. Hydra can be used to call NixOS' system integration test facilities. In fact, we already do this quite extensively to test the NixOS distribution itself. Moreover, Disnix also has a very comprehensive distributed integration test suite.

In the previous blog post, I have described Hydra -- a Nix-based continuous integration server and I have given an incomplete tour of its features.

In order to be able to use Hydra, we have to set up a build cluster consisting of one or more build machines. In this blog post, I will describe how I have set up a Hydra cluster of three machines.

Prerequisites

To set up a cluster, we need two kinds of machines:

• We need a build coordinator, with the Nix package manager installed running the three Hydra components: evaluator, queue runner and server. Hydra can be installed on any Linux distribution, but it's more convenient to use NixOS as it provides all the configuration steps as a NixOS module. Other distributions require more manual installation and configuration steps.
• We need one or more build slaves, or we have to use the coordinator machine as build slave. Various types of build slaves can be used, such as machines running Linux, Mac OS X, FreeBSD and Cygwin.

I have set up 3 machines, consisting of a Linux coordinator, Linux build slave and a Mac OS X build slave, which I will describe in the next sections.

Installing a NixOS build slave

To set up a build slave (regardless of the operating system that we want to use), we need have to install two system services -- we need the Nix package manager and the OpenSSH server so that it can be remotely accessed from the build coordinator.

Setting up a Linux build slave running NixOS is straightforward. I have used the NixOS Live CD to install NixOS. After booting from the Live CD, I first had to configure my harddrive partitions:

$ fdisk /dev/sda

I have created a swap partition (/dev/sda1) and root partition (/dev/sda2). Then I had to initialize the filesystems:

$ mkswap -L nixosswap /dev/sda1
$ mke2fs -j -L nixos /dev/sda2

Then I had to mount the root partition on /mnt:

$ mount LABEL=nixos /mnt

And I have created a NixOS configuration.nix file and stored it in /mnt/etc/nixos/configuration.nix. My NixOS configuration looks roughly as follows:

{pkgs, ...}:

{
  boot.initrd.kernelModules = [ "uhci_hcd" "ehci_hcd" "ata_piix" ];

  nix.maxJobs = 2;

  boot.loader.grub.enable = true;
  boot.loader.grub.version = 2;
  boot.loader.grub.device = "/dev/sda";

  networking.hostName = "i686linux";

  fileSystems."/".device = "/dev/disk/by-label/nixos";

  swapDevices =
    [ { device = "/dev/disk/by-label/nixosswap"; } ];

  services.openssh.enable = true;
}

By running the following instruction NixOS gets installed, taking care of downloading/installing all required packages and composing the entire system configuration:

$ nixos-install

After the installation has succeeded, we can reboot the machine and boot into our freshly installed NixOS installation. The new installation has a root user account without any password. Therefore, it's smart to change the root password to something that's slightly more difficult to guess:

$ passwd

And then the installation is done :-). However, apart from the steps that I have described, it may also be convenient to add a non-privileged user account and install some administration tools.

Upgrading the NixOS installation can be done by running:

$ nixos-rebuild --upgrade switch

The above command-line instruction fetches the latest channel expressions and manifests containing the latest releases of the packages, and then rebuilds the entire system configuration and finally activates it.

As a sidenote: we can also use ordinary Linux distributions as build slaves, but this requires more manual installation and configuration, especially if you want to use it's more advanced features, such as multi-user builds. Moreover, since NixOS is almost a "pure system", it reduces the chances on side effects, which is a bit harder to guarantee with conventional Linux distributions.

Installing a Mac OS X build slave

Mac OS X was already pre-installed on our Mac machine, so I only had to set up a user account and perform some basic settings.

On the Mac OS X machine, I have to install the Nix package manager manually. To do this, I have obtained the Nix bootstrap binaries for x86_64-darwin (the system architecture identifier for a 64-bit Mac OS X machine) and installed it by running the following commands on the terminal:

$ sudo -i
# cd /
# tar xfvj /Users/sander/Downloads/nix-1.5.1-x86_64-darwin.tar.bz2
# chown -R sander /nix
# exit
$ nix-finish-install

By running the above command-line instructions, the /nix directory has been set up containing a Nix store with the Nix package manager and all its dependencies. The last command: nix-finish-install takes care of initializing the Nix database. We run this build as ordinary user, since we don't want to use the superuser for Nix installations.

If we add a Nix package to the user's profile, we also want it to be in the user's PATH, so that we can start a program without specifying its full path. I have appended the following line to both the user profile ~/.profile as well as ~/.bashrc:

$ cat >> ~/.profile <<EOF
source $HOME/.nix-profile/etc/profile.d/nix.sh
EOF

By adding the above code fragment to the user's shell profile, the Nix profile is appended to PATH allowing you to conveniently launch packages without specifying their full path including their hash-codes.

You may probably wonder why this line needs to be added to both .profile and .bashrc? The former case allows you to start packages when a login shell is used, e.g. by launching a terminal from the Mac OS X desktop. The latter case is needed for non-login shells. If the Hydra coordinator remotely executes a command-line instruction through SSH, then the shell is a non-login shell. If we don't add this line to .bashrc then we're unable to run the Nix package manager, because it's not in the PATH.

After performing the above steps, we can run a simple sanity check to see if the Nix package manager works as expected. The following instructions add a Nixpkgs channel and fetches the latest expressions and manifests:

$ nix-channel --add http://nixos.org/channels/nixpkgs-unstable
$ nix-channel --update

After updating the Nix channel, we should be able to install a Nix package into our profile, such as GNU Hello:

$ nix-env -i hello

And we should be able to run it from the command-line as the user's profile is supposed to be in our PATH:

$ hello
Hello, world!

After installing the Nix package manager, there may be some other desirable steps that must be performed. In order to build iOS apps or Mac OS X applications, Apple's Xcode needs to be installed, which must be done through Apple's App Store. We cannot use Nix for this purpose unfortunately. I have given some instructions in a previous blog posts about building iOS apps with the Nix package manager.

Finally, to be able use the Mac OS X machine as build slave we need to configure two other things. First, we must enable the SSH server so that the build machine can be remotely invoked from the coordinator machine. We need to open Mac OS X's system preferences for this, which can be found by clicking on the Apple logo and by picking 'System preferences':


The System preferences screen looks as follows:


By picking the 'Sharing' icon we can configure various services that makes the machine remotely accessible:

As can be observed from the above screenshot, we can enable remote SSH access by enabling the 'Remote login' option. Furthermore, we must configure the hostname to set it to something that we can remember.

Another issue is that we need to turn some power management settings off, because otherwise the Mac machine will turn standby after a while and cannot be used to perform builds. Power management settings can be adapted by picking 'Energy saver' from the System preferences screen, which will show you the following:

I have set the 'Computer sleep' time to 'Never' and I've disabled putting the harddisks to sleep.

Setting up the NixOS build coordinator machine

Then comes the most complicated part -- setting up the build coordinator machine. First, I performed a basic NixOS installation, which installation procedure is exactly the same as the NixOS build slave described earlier. After performing the basic installation, I have adapted its configuration, to turn it into a build coordinator.

Since Hydra is not part of the standard NixOS distribution, we have to obtain it ourselves from Git and store the code in a directory on the filesystem (such as the /root folder):

$ git clone https://github.com/NixOS/hydra.git

Then I have extended the machine's configuration, by adding a number of settings to the attribute set body of /etc/nixos/configuration.nix:

• To be able to use Hydra's NixOS configuration properties, we must include the Hydra NixOS module:
  

  
  require = [ /root/hydra/hydra-module.nix ];
  

• We must enable the Hydra server and configure some of its mandatory and optional properties:
  

  
  services.hydra = {
    enable = true;
    hydra = (import /root/hydra/release.nix {}).build {
      system = pkgs.stdenv.system;
    };
    logo = ./logo.png;
    dbi = "dbi:Pg:dbname=hydra;host=localhost;user=hydra;";
    hydraURL = "http://nixos";
    notificationSender = "yes@itsme.com";
  };
  

  
  In the above code fragment, hydra refers to the actual Hydra package, the logo is an optional parameter that can be used to show a logo in the web front-end's header, dbi is a Perl DBI database connection string, configured to make a localhost connection to a PostgreSQL database named: hydra using the hydra user, hydraURL contains the URL to the web front-end, and notificationSender contains the administrator's e-mail address.
  
• In order to be able to delegate builds to build slaves for scalability and portability, we have to enable Nix's distributed builds feature:
  

  
  nix.distributedBuilds = true;
  nix.buildMachines = [
    { hostName = "i686linux";
      maxJobs = 2;
      sshKey = "/root/.ssh/id_buildfarm";
      sshUser = "root";
      system = "i686-linux";
    }
  
    { hostName = "macosx";
      maxJobs = 2;
      sshKey = "/root/.ssh/id_buildfarm";
      sshUser = "sander";
      system = "x86_64-darwin";
    }
  ];
  

  The above code fragment allows us to delegate 32-bit Linux builds to the NixOS build slave and 64-bit Mac OS X builds to the Mac OS X machine.
  
  
• Hydra needs to store its data, such as projects, jobsets and builds into a database. For production use it's recommended to use PostgreSQL, which can be enabled by adding the following line to the configuration:
  

  
  services.postgresql.enable = true;
  

• The Hydra server runs its own small webserver on TCP port 3000. In production environments, it's better to add a proxy in front of it. We can do this by adding the following Apache HTTP server configuration settings:
  

  
  httpd = {
    enable = true;
    adminAddr = "yes@itsme.com";
  
    extraConfig = ''
  <Proxy *>
      Order deny,allow
      Allow from all
  </Proxy>
  
      ProxyRequests     Off
      ProxyPreserveHost On
      ProxyPass         /    http://localhost:3000/ retry=5 disablereuse=on
      ProxyPassReverse  /    http://localhost:3000/
    '';
  };
  

• To allow e-mail notifications to be sent, we must configure a default mail-server. For example, the following does direct delivery through sendmail:
  

  
  networking.defaultMailServer = {
    directDelivery = true;
    hostName = "nixos";
    domain = "nixos.local";
  };
  

• As you may probably know from earlier blog posts, Nix always stores versions of components next to each other, and components never get overwritten or removed automatically. At some point we may run out of diskspace. Therefore, it's a good idea to enable garbage collection:

  
  nix.gc = {
    automatic = true;
    dates = "15 03 * * *";
  };
  
  services.cron = {
    enable = true;
  
    systemCronJobs =
      let
        gcRemote = { machine, gbFree ? 4, df ? "df" }:
          "15 03 * * *  root  ssh -x -i /root/.ssh/id_buildfarm ${machine} " +
          ''nix-store --gc --max-freed '$((${toString gbFree} * 1024**3 - 1024 * ''+
          ''$(${df} -P -k /nix/store | tail -n 1 | awk "{ print \$4 }")))' ''+
          ''> "/var/log/gc-${machine}.log" 2>&1'';
      in
      [ (gcRemote { machine = "root@i686linux"; gbFree = 50; })
        (gcRemote { machine = "sander@macosx"; gbFree = 50; })
      ];
  };
  


The nix.gc config attribute generates a cron job that runs the garbage collector service at 3:15 AM every night. The services.cron configuration also remotely connects to the build slave machines and runs the garbage collector if a certain threshold has been reached.


• It may also be worth enabling some advanced features of Nix. For example, in our situation we have many large components that are very similar to each other consuming a lot of diskspace. It may be helpful to enable hard-link sharing, so that identical files are stored only once.
  
  Moreover, in our current configuration we also download substitutes from the NixOS' Hydra build instance, so that we don't have to build the complete Nixpkgs collection ourselves. It may also be disable this and take full control.
  
  Another interesting option is to enable chroot builds, reducing the chances on side effects even more:
  

  
  nix.extraOptions = ''
    auto-optimise-store = true
    build-use-substitutes = false
    build-use-chroot = true
  '';
  

  The nix.conf manual page has more information about these extra options.
  
  

After adapting the coordinator's configuration.nix, we must activate it by running:

$ nixos-rebuild switch

The above command-line instruction downloads/installs all the required packages and generates all configuration files, such as the webserver and cron jobs.

After rebuilding, we still don't have a working Hydra instance yet. We must still set up its storage, by creating a PostgreSQL database and Hydra user. To do this, we must perform the following instructions as root user:

# createuser -S -D -R -P hydra
# createdb -O hydra hydra

By running the hydra-init job, we can setup its schema or migrate it to a new version:

# start hydra-init

Then we must create a configuration file that allows the unprivileged Hydra user to connect to it:

# su hydra
$ echo "localhost:*:hydra:hydra:password" > ~/.pgpass
$ chmod 600 ~/.pgpass

The .pgpass file contains the hostname, database, username and password which must be replaced by the user's real password, of course.

We also need to set up a user, as the user database is completely empty. The following will create an administration user named 'root' with password 'foobar':

echo "INSERT INTO Users(userName, emailAddress, password) VALUES ('root', 'yes@itsme.com', \
'$(echo -n foobar | sha1sum | cut -c1-40)');" | psql hydra
echo "INSERT INTO UserRoles(userName, role) values('root', 'admin');" | psql hydra

And finally we can activate the three Hydra processes, which allows us to use it and to access the web front-end:

$ exit
# start hydra-{evaluator,queue-runner,server}

Setting up connections

We now have a running Hydra instance, but there is still one detail missing. In order to allow the coordinator to connect to the build slaves, we need SSH keys without passphrases allowing us to connect automatically. Generating a SSH keypair can be done as follows:

$ ssh-keygen -t rsa

The above command asks you a couple of questions. You have to keep in mind that we should not specify a passphrase.

Assuming that we have called the file: id_buildfarm, then we have to two files, a private key called: id_buildfarm and a public key called id_buildfarm.pub. We must copy the public key to all build slaves and run the following instruction on each client machine, under the user which performs the build (which is root on the Linux machine and sander on the Mac OS X machine):

$ cat id_buildfarm.pub >> ~/.ssh/authorized_keys

The above command adds the public key to a list of authorized keys, allowing the coordinator to connect to it with the private key.

After installing the public keys, we can try connecting to the build slaves from the coordinator through the private key, by running the following command as root user:

$ ssh -i ~/.ssh/id_buildfarm root@i686linux

If we run the above command, we should be able to connect to the machine without being asked for any credentials. Moreover, the first time that you connect a machine, the host key is added to the known_hosts list.

Another issue that I have encountered with the Mac OS X machine is that it may stall connections after no input has been received from the coordinator for while. To remedy this, I added the following lines to the SSH config of the root user on the coordinator machine:

$ cat > /root/.ssh/config <<EOF
ServerAliveCountMax 3
ServerAliveInterval 60
EOF

Conclusion

In this blog post, I have described the steps that I have performed to set up a cluster of three Hydra machines consisting of two Linux machines and one Mac OS X machine. To get an impression on how Hydra can be used, I recommend users the read my previous blog post.

In the past, I have also set up some more "exotic" build slaves, such as the three BSDs: FreeBSD, OpenBSD, NetBSD and an OpenSolaris machine. To install these platforms, we can roughly repeat the procedure that I have done to install the Mac OS X build slave. First install the host OS itself, then the Nix package manager (there are bootstrap binaries for several platforms available or you must do a source install), and then set up SSH.

Today, I'm happy to announce the title and contents of my PhD dissertation on this blog!

Background

I have written many software deployment related topics on my blog. Earlier, I have explained that certain developments in software engineering, such as component-based software engineering (CBSE), have increased the quality of software systems, but also have significantly increased the complexity of deployment processes of software systems. Moreover, I have written about solutions, typically involving tools from the Nix project.

Before I started my research as PhD (and master's) student, we already had the following deployment tools in the Nix project:

• The Nix package manager for deployment of individual packages from declarative specifications.
• NixOS, a Linux distribution completely built around the Nix package manager. NixOS deploys entire system configurations from a single declarative specification.
• Hydra, formerly known as just the "Nix build farm". Hydra builds jobsets from declarative specifications.

Distributed software deployment

As you may notice, these tools all have something in common: to fully automate software deployment processes from declarative specifications, in a reliable, reproducible, generic and efficient way. Their difference is the level in which they operate.

However, to support modern generation systems, typically exposed as services through the Internet, these tools are not sufficient:

• We have to deploy multiple machines in a network, instead of a single machine.
• Components may have dependencies on other components residing on different machines in the network, as well as system configurations (called inter-dependencies in this thesis).
• Components need underlying infrastructure components, e.g. an application server or a DBMS.
• Upgrading these systems may give a large time window in which a system is inconsistent.
• We have to meet important non-functional requirements, e.g. performance, security, privacy and the license under which components are governed.

To support these new generation of systems, we have developed a collection of new tools in the Nix project:

• Disnix, to deploy services (such as web services, web applications, UNIX processes etc) inside networks of machines from declarative specifications.
• Dynamic Disnix, to automatically redeploy services through Disnix based on technical and non-functional constraints if an event occurs in the network, e.g. machine crash.
• NixOps (formerly known as Charon and nixos-deploy-network) and the NixOS test driver, to deploy and test networks of NixOS configurations from declarative specifications.
• Dysnomia, a prototype-tool to deploy mutable software components.
• grok-trace, a prototype-tool to dynamically analyze build processes to discover license constraints.

These tools solve various distributed deployment aspects having similar quality attributes as the older Nix tools and make it possible to support some important non-functional requirements.

A Reference Architecture for Distributed Software Deployment

However, to fully automate deployments of service-oriented systems, we must combine these tools implementing generic deployment aspects with components supporting important non-functional requirements of a domain, such as performance, security and privacy. However, there is no silver bullet that solves all these problems for any domain, as some non-functional requirements are more important in one domain than another, and some may conflict with each other.


Instead, we have designed a reference architecture that can be used to construct a concrete architecture for a domain-specific deployment tool, capable of fully automating a deployment process with its desired quality attributes.

My thesis gives some background information about software deployment in general and Nix, it proposes a reference architecture for distributed software deployment, it describes most of the newer Nix components, and shows how these can be applied. Finally, we reflect on how well we have achieved our desired quality attributes.

Availability

My PhD thesis can be obtained from the PhD thesis page of my homepage. Moreover, the defence will be at June 3, 2013 14:30 in the Aula Congresscentre of Delft University of Technology. If you're interested in attending my defence, please let me know!

Apart from the contents of my PhD thesis, theses at our university are usually accompanied by a list of propositions. According to our university's Doctorate regulations, they must be defendable and opposable, at least six of the propositions are not supposed to be directly related to the research subject and two of them may be slightly playful. Besides the contents of my thesis, committee members are also allowed to ask questions about the propositions.

A colleague of mine: Felienne Hermans, has covered her propositions on her blog to elaborate about them. I have decided to do the same thing, although I'm not planning to create separate blog posts for each individual proposition. Instead, I cover all of them in a single blog post.

Propositions

1. Many of the non-functional requirements of a deployed service-oriented system can be realized by selecting an appropriate software deployment system.
   
   As I have explained earlier in a blog post about software deployment complexity, systems are rarely self-contained but composed of components. An important property of a component is that it can be deployed independently, significantly complicating a software deployment process.
   
   Components of service-oriented systems are called "services". It's a bit of an open debate to exactly tell what they are, since people from industry often think in terms of web services (things that use SOAP, WSDL, UDDI), while I have also seen the description "autonomous platform independent entities that can be loosely coupled" in the academic literature.
   
   Although web services are some sort of platform independent entities, they still have implementations behind their interfaces depending on certain technology and can be deployed to various machines in a network. We have seen that deployment on a single machine is hard and that deploying components into networks of machines is even more complicated, time consuming and error prone.
   
   Besides deployment activities, there are many important non-functional requirements a system has to meet. Many of them can be achieved by designing an architecture, e.g. components, connectors and the way they interact through architectural patterns/styles. Architectural patterns (e.g. layers, pipes and filters, blackboard etc.) implement certain quality attributes.
   
   For service-oriented systems, it's required to deploy components properly into a network of machines to be able to compose systems. In other words: we have to design and implement a proper deployment architecture. This has several technical challenges, such as the fact that we have to deploy components in such a way that they exactly match the deployment architecture, the deployment activities themselves, and the fact that we have to determine whether a system is capable of running a certain component (i.e. a service using Windows technology cannot run on a Linux machine or vice-versa).
   
   In addition to technical constraints, there are also many non-functional issues related to deployment that require attention, i.e. where to place components and how to combine them to achieve certain non-functional requirements? For example, privacy could be achieved by placing services providing access to privacy-sensitive data in a restricted zone and robustness by deploying multiple redundant instances of the same service.
   
   It can also be hard to manually find a deployment architecture that satisfies all non-functional requirements. In such cases, deployment planning algorithms are very helpful. In some cases it's even too hard or impossible to find an optimal solution.
   
   Because of all these cases, an automated deployment solution taking all relevant issues into account is very helpful in achieving many non-functional requirements of a deployed service-oriented system. This is what I have been trying to do in my PhD thesis.
   
   
2. The intention of the Filesystem Hierarchy Standard (FHS) is to provide portability among Linux systems. However, due to ambiguity, over-specification, and legacy support, this standard limits adoption and innovation. This phenomenon applies to several other software standards as well.
   
   There are many standards in the software engineering domain. In fact, it's dominated by them. One standard that is particularly important in my research is the Filesystem Hierarchy Standard (FHS) defining the overall filesystem structure of Linux systems. I have written a blog post on the FHS some time ago.
   
   In short: the FHS defines the purposes of directories, it makes a distinction between static and variable parts of a system, it defines hierarchies (e.g. / is for boot/recovery, /usr is for user software and for /usr/local nobody really knows (ambiguity)). Moreover, it also defines the contents of certain directories, e.g. /bin should contain /bin/sh (over-specification).
   
   I have problems with the latter two aspects -- the hierarchies do not provide support for isolation, allowing side-effects to easily manifest themselves while deploying and enabling destructive upgrades. I also have a minor problem with strict requirements of the contents of directories, as they easily allow builds to trigger side-effects while assuming that certain tools are always present.
   
   For all these reasons, we deviate on some aspects of the FHS in NixOS. Some people consider this unacceptable, and therefore they will not be able to incorporate most of our techniques to improve the quality of deployment processes.
   
   Moreover, as the FHS itself has issues, we observe that although the filesystem structure is standardized, file system layouts in many Linux distributions are still slightly different and portability issues still arise.
   
   In other domains, I have also observed various issues with standards:
   
   
   • Operating systems: nearly every operating system is more or less forced to implement POSIX and/or the Single UNIX Specification, taking a lot of effort. Furthermore, by implementing these standards they UNIX is basically reimplemented. These standards have many strict requirements on how certain library calls should be implemented, although it also specifies undefined behaviour at the same time. Apart from the fact that it's difficult and time consuming to implement these standards, there is little room to implement an operating system that is conceptually different from UNIX as it conflicts with portability.
   • The Web (e.g. HTML, CSS, DOM etc.): First, a draft is written in a natural language (which is inherently ambiguous) and sometimes underspecified. Then vendors start implementing these drafts. As initially these standards are ambiguous and underspecified, implementations behave very differently. Slowly these implementations converge into something that is uniform by collaborating with other vendors and the W3C to improve the draft version of a standard. Some vendors intentionally or accidentally implement conformance bugs, which don't get fixed for quite some time.
     
     These buggy implementations may become the de-facto standard, which has happened in the past, e.g. with Internet Explorer 6, requiring web developers to implement quirks code. Since the release of Internet Explorer 6 in 2001, Microsoft had 95% market share and did not release a new version until 2006. This was seriously hindering innovation in web technology. It also took many years before other implementations with better web standards conformance and more features gained acceptance.

So are standards bad? Not necessarily, but I think we have to critically evaluate them and not consider them as holy books. Moreover, standards need to be developed with some formality and elegance in mind. If junk gets standardized, it will remain junk and requires everybody to cope with junk for quite some time.

One of the things that may help is using good formalisms. For example, a good one I can think of is BNF that was used in the ALGOL 60 specification.


3. To move the software engineering community as a whole forward, industry and academia should collaborate. Unfortunately, their Key Performance Indicators (KPIs) drive them apart, resulting in a prisoner's dilemma.
   
   This proposition is related to an earlier rant blog post about software engineering fractions and collaborations. If I would use stereotypes, then the primary KPI of academia are the amount of (refereed) publications and the primary KPI of industry is how much they sell.
   
   It's obvious that, if both fractions would let themselves go a bit from their KPIs, i.e. academia does a bit more in engineering tools and transferring knowledge, while industry spends some of their effort in experimenting and paying attention to "secondary tasks", that both parties benefits. However, in practice often the opposite happens (although there are exceptions of course).
   
   This is analogous to a prisoner's dilemma, which is a peculiar phenomenon. Visualize the following situation: two prisoners have jointly committed a crime and got busted. If both confess their crime then the amount of time they have to spend in prison are five years. If one prisoner confesses while the other does not, then the prisoner that confesses goes ten years into jail and the other remains free. If none of them confess, they both have to spent twenty years in prison.
   
   In this kind of situation the (obvious) win-win situation for both criminals is that they both confess. However, because they both give priority to their self-interests, none of them confesses as they assume that they remain free. But instead, the situation has the worst outcome: both have to spent twenty years in prison.
   
   
4. In software engineering, the use of social media, such as blogging and twitter, are an effective and efficient way to strengthen collaboration between industry and academia.
   
   This proposition is related to the previous one. How can we create a win-win situation? Often I hear people saying: "Well collaboration is interesting, but it costs time and money, which we don't have right now and we have other stuff to do".
   
   I don't think the barrier has to be that high. Social media, such as blogging and twitter, can be used for free and allows one to easily share stories, thoughts, results and so on. Moreover, recipients can also share these with people they know.
   
   My blog for example, has attracted many more readers and has given me much more feedback then all my research papers combined. Moreover, I'm not limited by all kinds of contraints that program committee members impose on me.
   
   However these observations are not unique to me. Many years ago, a famous Dutch computer scientist named Edsger Dijsktra wrote many manuscripts that he sent to his peers directly. He wrote about subjects that he found relevant. His peers spread these manuscripts through their colleagues allowing him to reach a wide range of people, eventually reaching thousands of people.
   
   
5. While the vision behind the definition of free software as described by the Free Software Foundation to promote freedom is compelling, the actual definition is ambiguous and inadequately promoted.
   
   The free software definition defines four freedoms. I can rephrase them in one sentence: "Software that can be used, studied, adapted and shared for any purpose". An essential precondition for this is the availability of the source code. I think this definition is clear and makes sense.
   
   However, there is a minor issue with the definition. The word 'free' is ambiguous in English. In the definition, it refers to free as in freedom not free in price (gratis). In Dutch or French it's not a problem. In these languages free software translates to 'vrije software' and 'libre software'.
   
   Moreover, although (almost) all free software is gratis, it's also allowed to sell free software for any price, which is often misunderstood.
   
   I have seen that the ambiguity of the word free is often used as an argument why the definition is not attracting a general audience.
   
   I think there is a bigger problem: the way free software is advertised. Most advertisements are not about what's good about free software, but about what's bad about proprietary software and how evil certain practices of certain companies are.
   
   Although I don't want to say that they are not right and we should tolerate such bad practices, I think it would also help to pay more attention to the good aspects of free software. The open source definition has much more care for this. For example, being able to improve quality of software. That's something I think that would attract people from the other side, whereas negative campaigning does not.
   
   
6. Compared to the definition of free software provided by the Free Software Foundation, the definition of Open Source as provided by the Open Source Initiative, fails to improve on freedom. While it has been more effectively promoted, it lacks a vision and does not solve ambiguity.
   
   The open source definition lists ten pragmatic points with the intention of having software that is free (as in freedom), e.g. availability of source code, means to share modified versions, and so on. However, it does not explain why it's desired for others to respect these ten pragmatic points and what their rationale is (although there is an annotated definition that does).
   
   Because of these reasons, I have seen that sometimes software is incorrectly advertised as being open-source, while in reality they are not. For example, there is also software available with source code, for which it is not allowed to do commercial redistributions, such as the LCC compiler. That's not open source (nor free software). Another prominent example is Microsoft's Shared Source initiative, only allowing someone to look at code, but not to modify or redistribute it.
   
   A very useful aspect of open source is the way it's advertised. It pays a lot of attention in selling its good points. For example, that everyone is able to improve its quality, and allowed to collaborate etc. Companies (even those that sell proprietary software) acknowledge these positive aspects and are sometimes willing to work with open-source people on certain aspects or to "open-source" pieces of their proprietary products. Examples of this are the Eclipse platform and the Quake 3 arena video game.
   
   
7. Just like making music is more than translating notes and rests into tones and pauses with specific durations, developing software systems is more than implementing functional requirements. In both domains, details, collaboration and listening to others are most important.
   
   I have observed that in both domains we make estimations. In software development, we try to estimate how much effort it takes to develop something and in music we try to estimate how much effort it takes to practice and master a composition.
   
   In software development, we often look at functional requirements (describing what a system should do) to estimate. I have seen that sometimes functional requirements may look ridiculously simple, such as displaying tabular data on a web page. Nearly every software developer would say: "that's easy".
   
   But even if functional requirements are simple, certain non-functional requirements (describing how and where) could make it very difficult. For example, properly implementing security facilities, a certain quality standard (such as ISO 9126) or to provide scalability. These kind of aspects may be much more complicated than the features of a system itself.
   
   Moreover, software is often developed in a team. Good communication and being able to divide work properly is important. In practice, you will almost always see that something goes wrong with that, because people have assumptions that all details are known by others or there is no clear architecture of a system so that work can be properly divided among team members.
   
   These are all reasons that may result in development times that are significantly longer than expected and failure to properly deliver what clients have asked.
   
   In my spare time I'm a musician and in music I have observed similar things. People make effort estimations by looking at the notes written on paper. Essentially, you could see those as functional requirements as they tell you what to play.
   
   However, besides playing notes and pausing, there are many more aspects that are important, such as tempo, dynamics (sudden and gradual loudness) and articulation. You could compare these aspects to non-functional requirements in software, as they tell somebody how to play (series of) notes.
   
   Moreover, making music can also be a group effort, such as a band or an orchestra, requiring people to properly interact with each other. If others make mistakes they may confuse you as well.
   
   I vividly remember a classical composition of 15 years ago. I just joined an orchestra and we were practising: "Land of the Long White Cloud" by Philip Sparke. In the middle of the composition, there is a snare drum passage consisting of only sixteenth notes. I already learned playing patterns like these in my first drum lesson, so I thought that it would be easy.
   
   However, I had to play these notes in a very fast tempo and very quietly, which are usually conflicting constraints for percussionists. Furthermore, I had to keep up the right tempo and don't let the other members distract me. Unfortunately, I couldn't cope with all these additional constraints, and that particular passage had to be performed by somebody else. I felt like an idiot and I was very disappointed. However, we did win the contest in which we had to perform that particular composition.
   
   As a side note: "The land of the long white cloud" is also known as Aotearoa or New Zealand.
   
   
8. Multilingualism is a good quality for a software engineer as it raises awareness that in natural languages as well as in software languages and techniques, there are things that cannot be translated literally.
   
   It's well known that some words or sentences cannot be literally translated from one natural language to another. In such cases, we have to reformulate a sentence into something that has an equivalent meaning, which is not always trivial.
   
   For example, non-native English speakers, like Dutch people, tend to make (subtle) mistakes now and then, which sometimes have hilarious outcomes. Make that the cat wise is a famous website that elaborates on this, calling the Dutch variant of English: Dunglish.
   
   Although we are aware of the fact that we cannot always translate things literally in natural languages, I have observed that in the software domain the same phenomenon occurs. One particular programming language may be more useful for a certain goal, than another programming language. Eventually, code written in a programming language gets compiled into machine code or another programming language (having an equivalent reformulation in a different language) or interpreted by an interpreter.
   
   However, I have also observed that in the software engineering domain there is a lot of programming language conservatism. Most conventional programming languages used nowadays (Python, C++, Java, C# etc.) use structured and imperative programming concepts in combination with class-based OO techniques. Unconventional languages such as purely functional programming languages (Haskell) or declarative languages (Prolog, Erlang) only get little mainstream acceptance, although they have very powerful features. For example, programs implemented in a purely functional language easily scale across multiple cores/processors.
   
   Instead, many developers use conventional languages to achieve the same, imposing many additional problems that need to be solved and more chances on errors. Our research based on the purely functional deployment model also suffers from conservatism. Therefore, I think multilingualism is very powerful asset for an engineer, as he is not limited by a solution set that is too narrow.
   
   
9. Stubbornness is both a positive as well as a negative trait of a researcher.
   
   I think that when you do research and if you discover something that is uncommon, others may reject it or tell you to do something that they consider more relevant. Some researchers choose to comply and give up stuff that they think is relevant. If every scientist would do that, then I think certain things would have never been discovered. I think it's a scientist's duty to properly defend important discoveries.
   
   In the middle ages it was even worse. For example, a famous scientist named Galileo Galilei revealed that not the Earth but the Sun was the centre of our solar system. He was sentenced to house arrest for the rest of his life by the catholic church.
   
   However, stubbornness also has a negative aspect. It often comes with ignorance and sometimes that's bad. For example, I have "ignored" some advice about properly studying related work and taking evaluation seriously, resulting in a paper that was badly rejected.
   
   

Conclusion

In this blog post, I have described some thoughts on my PhD propositions. The main reason of writing this down is to prepare myself for my defence. I know this blog post is lengthy, but that's good. This will probably prevent my committee members to read all the details, so that they cannot use everything I have just written against me :-) (I'm very curious to see if anyone has notice that I just said this :P).

As a follow up story on the previous two blog posts, I can tell you that I have defended my PhD thesis last Monday, which went successfully. Now I can officially call myself a Doctor.

It was quite a tense, busy and interesting day. I'm not really used to days in which you are the centre of attention for most of the time. In the morning, our foreign guests: Sam Malek and Roberto di Cosmo (who are members of of my committee) arrived to tell a bit about their ongoing research. We had some sort of a small software deployment workshop with quite some interesting ideas and discussions. Having them visiting us, gave me some renewed excitement about ongoing research and some interesting future ideas, since software deployment (as I have explained) is typically a misunderstood and under appreciated research subject.

After our small workshop, I had to quickly pick up our suits, and then return to the university. In the early afternoon, I had to give a laymen's talk to my family and friends in which I tried to explain the background and the goal of my PhD thesis. Then the actual defence started lasting for exactly one hour (no more, no less) in which the committee members asked me questions about my thesis and the accompanied propositions.

At Delft University of Technology and any other Dutch university, there is a lot of ceremonial stuff involved with the PhD defence. Professors have to be dressed up in gowns. Me and my paranimphs (the two people sitting in front of me who help me and take over my defence if I pass out) had to be dressed up in suits. I have to address to committee members formally depending on their roles, such as: "hooggeleerde opponent" and the committee members had to formally address me with "waarde promovendus".

I received some interesting questions during my defence round. To have an idea what these questions were, Vadim Zaytsev has made 140 character transcriptions of each question and answer on Twitter.

Although I was experiencing some nervousness at the beginning of the day, as I didn't know what exactly was going to happen and what kind of questions I would receive, in the end it was fun and I liked it very much. After the defence I received my PhD diploma:


For people that want to know what my thesis is exactly about:

• My PhD thesis can be obtained from my homepage.
• I have also written a blog post about my PhD thesis for people familiar with Nix that is significantly shorter.
• For people who think that even the blog post too long, there is also 140 character version of my PhD thesis.

In two earlier blog posts, I have given a general description about Hydra and described how Hydra can be set up.

Another important aspect is to know that most of the facilities of Hydra are publicly accessible by anyone, except for the administration tasks, such as maintaining projects and jobsets for which it is required to have a user account.

A way to prevent Hydra from having public user access is to secure the reverse proxy that is in front of it. In this blog post, I will describe how to set up an HTTPS virtual domain with password authentication for this purpose. Moreover, the approach is not restricted to Hydra -- it can be used to secure any web application by means of a reverse proxy.

Generating a self-signed SSL certificate

To be able to set up HTTPS connections we need an SSL certificate. The easiest way to get one is to create a self-signed SSL certificate. The following command-line instruction suffices for me to generate a private key:

$ openssl genrsa -des3 -out hydra.example.com.key 1024

The above command asks you to provide a passphrase. Then we need to generate a certificate signing request (CSR) file to sign the certificate with our own identity:

$ openssl req -new -key hydra.example.com.key \
    -out hydra.example.com.csr

The above command asks you some details about your identity, such as the company name, state or province and country. After the CSR has been created, we can generate a certificate by running:

$ openssl x509 -req -days 365 -in hydra.example.com.csr \
    -signkey hydra.example.com.key -out hydra.example.com.orig.crt

To prevent the web server from asking me for the passphrase on every start, I ran the following to adapt the certificate:

$ openssl rsa -in hydra.example.com.orig.crt \
    -out hydra.example.com.crt

Now we have successfully generated a self-signed certificate allowing us to encrypt the remote connection to our Hydra instance.

Obviously, if you really care about security, it's better to obtain a cross-signed SSL certificate, since that provides you (some sort of) guarantee that a remote host can be trusted, whereas our self-signed SSL certificate forces users to create a security exception in their browsers.

Creating user accounts

A simple way to secure an Apache HTTP server with user authentication is by using the htpasswd facility. I did the following to create a user account for myself:

$ htpasswd -c /etc/nixos/htpasswd sander

The above command creates a user account named: sander, asks me for a password and stores the resulting htpasswd file in /etc/nixos. The above command can be repeated to add more user accounts.

Adapting the NixOS system configuration

As a final step, the reverse proxy must be reconfigured to accept HTTPS connections. The following fragment shows how the Apache HTTPD NixOS service can be configured to act as a reverse proxy having an unsecured HTTP virtual domain, and a secured HTTPS virtual domain requiring users to authenticate with a username and password:

services.httpd = {
  enable = true;
  adminAddr = "sander@example.com";
  hostName = "hydra.example.com";
  sslServerCert = "/etc/nixos/hydra.example.com.crt";
  sslServerKey = "/etc/nixos/hydra.example.com.key";

  virtualHosts = [
    { hostName = "localhost";
      extraConfig = ''
<Proxy *>
          Order deny,allow
          Allow from all
</Proxy>

        ProxyRequests     Off
        ProxyPreserveHost On
        ProxyPass         /    http://localhost:3000/ retry=5 disablereuse=on
        ProxyPassReverse  /    http://localhost:3000/
      '';
    }

    { hostName = "localhost";
      extraConfig = ''
<Proxy *>
          Order deny,allow
          Allow from all
          AuthType basic
          AuthName "Hydra"
          AuthBasicProvider file
          AuthUserFile /etc/nixos/htpasswd
          Require user sander
</Proxy>

        ProxyRequests     Off
        ProxyPreserveHost On
        ProxyPass         /    http://localhost:3000/ retry=5 disablereuse=on
        ProxyPassReverse  /    http://localhost:3000/
        RequestHeader set X-Forwarded-Proto https
        RequestHeader set X-Forwarded-Port 443
      '';
      enableSSL = true;
    }
  ];
};

There is one important aspect that I had to take into account in the configuration of the HTTPS virtual host. Without providing the RequestHeader properties, links in Hydra are not properly generated, as Catalyst (the MVC framework used by Hydra) does not know that links should use the https:// scheme instead of http://.

Restricting the unsecured HTTP virtual domain

It may also desired to prevent others from having access to the insecure HTTP virtual host. In a NixOS system configuration, you can set the firewall configuration to only accept HTTPS connections by adding the following lines:

networking.firewall.enabled = true;
networking.firewall.allowedTCPPorts = [ 443 ];

Conclusion

The tricks described in this blog post allowed me to secure Hydra with an HTTPS connection and password authentication, so that I can open a web browser and use: https://hydra.example.com to access the secured virtual host.

The major disadvantage of this approach is that you cannot use Nix's channel mechanism to automatically obtain substitutes from Hydra. Perhaps, in the future Nix can be adapted to also support connections with user authentication.

I have written quite some Nix-related articles on this blog. Nix is typically advertised as the core component of the NixOS Linux distribution. However, it can also be used separately on conventional Linux distributions and other UNIX-like systems, such as FreeBSD and Mac OS X.

Using Nix on conventional systems makes it possible to use the interesting features of Nix and its applications, such as Hydra and Disnix, while still being able to use your favorite distribution.

Single-user Nix installations

I have noticed that on non-NixOS systems, the Nix package manager is often installed for only one single user, as performing single-user installations is relatively simple. For example, in my blog post describing how to build apps for iOS with Nix, I perform a Nix installation that can only be used by my personal user account.

For most of the simple use cases, single user installations are sufficient. However, they have a number of issues besides the fact that only one user of a machine (in addition to the super-user) is able to use it:

• Although Nix creates builds environments that remove several important side-effects, e.g. by clearing environment variables and storing all packages in isolation in the Nix store, builds can still refer to executables and other files in global directories by having hardcoded references, such as /usr/bin or /var, which may influence the result of the build.
  
  On NixOS these references are typically not an issue since these directories do not exist, but on conventional distributions these do exist, which may cause (purity) problems.
• Many packages have hard-coded references to the default Bourne-compatible shell in: /bin/sh. Some of these packages assume that this shell is bash and use bash-specific features.
  
  However, some Linux distributions, such as Debian and Ubuntu, use dash as default /bin/sh causing some builds to break. Moreover, BSDs such as FreeBSD also use a simpler Bourne shell implementation by default.
• Some build processes may unknowingly try to download stuff from the Internet causing impurities.
• Package builds have the same privileges as the calling user, allowing external processes run by the user to interfere with the build process. As a consequence, impurities may sneak in when executing multiple package builds in parallel.

Multi-user Nix installations

As a remedy for the issues just described, Nix is also capable of executing each build with separate user privileges in a nearly clean chroot environment in which we bind mount the Nix store. However, in order to use these features, a multi-user Nix installation is required, as these operations require super-user privileges. In NixOS, a multi-user Nix installation comes for free.

In multi-user Nix installations, builds are not executed directly by each individual user, since they cannot be trusted. Instead we run a server, called the nix-daemon that builds packages on behalf of a user. This daemon also takes care of running processes as a unique unprivileged user and setting up chroot environments.

Although the Nix manual provides some pointers to set up a multi-user installation, it turned out to be a bit trickier than I thought. Moreover, I have noticed that a few practical bits were missing in the manual and the Nix distribution.

In this blog post, I have investigated these issues and implemented a few improvements that provide a solution for these missing parts. I have performed these steps on a Ubuntu 12.04 LTS machine.

Installing Nix from source

As a first step, I installed Nix from source by running the following commands:

$ ./configure --prefix=/usr --sysconfdir=/etc
$ make
$ sudo make install

Adding the Nix build group and users

Since every concurrent build must be run as a separate user, we have to define a common group of which these users should be a member:

$ sudo groupadd -g 20000 nixbld

Then we need to add user accounts for each build that gets executed simultaneously:

$ for i in `seq 1 10`
do
    sudo useradd -u `expr 20000 + $i` -g nixbld \
      -c "Nix build user $i" -d /var/empty -s /noshell
done

In the code fragment above, we assume that 10 users are sufficient, but if you want/need to utilise more processors/cores this number needs to be raised.

Finally, we have to specify the build users group in the Nix configuration:

$ sudo echo "build-users-group = nixbld" >> /etc/nix/nix.conf

Changing permissions of the Nix store

In ordinary installations the user is made owner of /nix/store. In multi-user installations, it must be owned by root and group owned by the nixbld user. The following shell commands grant the Nix store the right permissions:

$ sudo chgrp nixbld /nix/store
$ sudo chmod 1775 /nix/store

Creating per-user profile and garbage collection root folders

In single user installations, we only have one system-wide default profile (/nix/var/nix/profiles/default) owned by the user. In multi-user installations, each user should be capable of creating their own profiles and garbage collector roots. The following shell commands ensure that it can be done:

$ sudo mkdir -p -m 1777 /nix/var/nix/profiles/per-user
$ sudo mkdir -p -m 1777 /nix/var/nix/gcroots/per-user

Setting up the Nix daemon

We must also run the nix-daemon that executes builds on behalf of a user. To be able to start and stop it, I have created an init.d script for the Nix daemon. The interesting part of this script is the start operation:

DAEMON=/usr/bin/nix-daemon
NAME=nix-daemon

if test -f /etc/default/nix-daemon; then
    . /etc/default/nix-daemon
fi

...

case "$1" in

start)
    if test "$NIX_DISTRIBUTED_BUILDS" = "1"; then
        NIX_BUILD_HOOK=$(dirname $DAEMON)/../libexec/nix/build-remote.pl

        if test "$NIX_REMOTE_SYSTEMS" = "" ; then
            NIX_REMOTE_SYSTEMS=/etc/nix/remote-systems.conf
        fi

        # Set the current load facilities
        NIX_CURRENT_LOAD=/var/run/nix/current-load

        if test ! -d $NIX_CURRENT_LOAD; then
            mkdir -p $NIX_CURRENT_LOAD
        fi
    fi

    start-stop-daemon -b --start --quiet \
        --exec /usr/bin/env \
        NIX_REMOTE_SYSTEMS=$NIX_REMOTE_SYSTEMS \
        NIX_BUILD_HOOK=$NIX_BUILD_HOOK \
        NIX_CURRENT_LOAD=$NIX_CURRENT_LOAD \
        $DAEMON -- $DAEMON_OPTS
    echo "$NAME."
    ;;

...

esac

For the start operation, we have to spawn the nix-daemon in background mode. Moreover, to allow Nix to perform distributed builds, we must set a number of environment variables that provide the locations of the build hook script, the configuration file containing the properties of the external machines and a directory containing files keeping track of the load of the machines. Moreover, some directories may also have to be created if they don't exist.

To ensure that it's automatically launched on startup, we must add the following symlinks for the relevant runlevels:

$ sudo -i
# cd /etc/rc2.d
# ln -s ../init.d/nix-daemon S60nix-daemon
# cd ../rc3.d
# ln -s ../init.d/nix-daemon S60nix-daemon
# cd ../rc4.d
# ln -s ../init.d/nix-daemon S60nix-daemon
# cd ../rc5.d
# ln -s ../init.d/nix-daemon S60nix-daemon
# exit

I think the above init.d script can be trivially ported to other distributions.

Setting up user profiles

To allow users to install software through Nix and allow them to refer to their installed programs from a simple command-line invocation, we need to add some stuff to the user's shell profile, such as setting the PATH environment variable pointing to certain Nix profiles.

However, the nix.shprofile.d script in the Nix distribution only performs the necessary steps for single user installations. For example, it only adds to system-wide Nix profile and assumes that the user has all the rights to configure a channel.

I have ported all the relevant features from NixOS to create /etc/profile.d/nix-multiuser.sh, supporting all required features to set up a shell profile for multi-user installations:

First, we have to set up a user's profile directory in the per-user profile directory, if it doesn't exist:

export NIX_USER_PROFILE_DIR=/nix/var/nix/profiles/per-user/$USER

mkdir -m 0755 -p $NIX_USER_PROFILE_DIR
if test "$(stat --printf '%u' $NIX_USER_PROFILE_DIR)" != "$(id -u)"; then
    echo "WARNING: bad ownership on $NIX_USER_PROFILE_DIR" >&2
fi

In single user installations, a ~/.nix-profile symlink is created pointing to the system-wide default Nix profile. For multi-user installations, we must create a ~/.nix-profile symlink pointing to the per-user profile. For the root user, we can still use the system wide Nix profile providing software for all users of the system:

if ! test -L $HOME/.nix-profile; then
    echo "creating $HOME/.nix-profile" >&2
    if test "$USER" != root; then
        ln -s $NIX_USER_PROFILE_DIR/profile $HOME/.nix-profile
    else
        # Root installs in the system-wide profile by default.
        ln -s /nix/var/nix/profiles/default $HOME/.nix-profile
    fi
fi

In single user installations, we add the bin directory of the system-wide Nix profile to PATH. In multi-user installations, we have to do this both for the system-wide and the user profile:

export NIX_PROFILES="/nix/var/nix/profiles/default $HOME/.nix-profile"

for i in $NIX_PROFILES; do
    export PATH=$i/bin:$PATH
done

In single user installations, the user can subscribe itself to the Nixpkgs unstable channel providing pre-built substitutes for packages. In multi-user installations only the super-user can do this (as ordinary users cannot be trusted). Although root can only subscribe to a channel, ordinary users can still install from the subscribed channels:

if [ "$USER" = root -a ! -e $HOME/.nix-channels ]; then
    echo "http://nixos.org/channels/nixpkgs-unstable nixpkgs" \
> $HOME/.nix-channels
fi

We have to create a garbage collector root folder for the user, if it does not exists:

NIX_USER_GCROOTS_DIR=/nix/var/nix/gcroots/per-user/$USER
mkdir -m 0755 -p $NIX_USER_GCROOTS_DIR
if test "$(stat --printf '%u' $NIX_USER_GCROOTS_DIR)" != "$(id -u)"; then
    echo "WARNING: bad ownership on $NIX_USER_GCROOTS_DIR" >&2
fi

We must also set the default Nix expression, so that we can conveniently install packages from Nix channels:

if [ ! -e $HOME/.nix-defexpr -o -L $HOME/.nix-defexpr ]; then
    echo "creating $HOME/.nix-defexpr" >&2
    rm -f $HOME/.nix-defexpr
    mkdir $HOME/.nix-defexpr
    if [ "$USER" != root ]; then
        ln -s /nix/var/nix/profiles/per-user/root/channels \
          $HOME/.nix-defexpr/channels_root
    fi
fi

Unprivileged users do not have the rights to build package directly, since they cannot be trusted. Instead the daemon must do that on behalf of the user. The following shell code fragment ensures that:

if test "$USER" != root; then
    export NIX_REMOTE=daemon
else
    export NIX_REMOTE=
fi

Using multi-user Nix

After having performed the previous steps, we can start the Nix daemon by running the init.d script as root user:

$ sudo /etc/init.d/nix-daemon start

Then if we login as root, we can update the Nix channels and install packages that are supposed to be available system-wide:

$ nix-channel --update
$ nix-env -i hello
$ hello
Hello, world!

We should also be able to log in as an unprivileged user and capable of installing software:

$ nix-env -i wget
$ wget # Only available to the user that installed it
$ hello # Also works because it's in the system-wide profile

Enabling parallel builds

With a multi-user installation, we should also be able to safely run multiple builds concurrently. The following change can be made to allow 4 builds to be run in parallel:

$ sudo echo "build-max-jobs = 4" >> /etc/nix/nix.conf

Enabling distributed builds

To enable distributed builds (for example to delegate a build to a system with a different architecture) we can run the following:

$ sudo echo "NIX_DISTRIBUTED_BUILDS=1" > /etc/defaults/nix-daemon

$ sudo cat > /etc/nix/remote-systems.conf << "EOF"
sander@macosx.local x86_64-darwin /root/.ssh/id_buildfarm 2
EOF

The above allows us to delegate builds for Mac OS X to a Mac OS X machine.

Enabling chroot builds

On Linux, we can also enable chroot builds allowing us to remove many undesired side-effects that single-user Nix installations have. Chroot environments require some directories of the host system to be bind mounted, such as /dev, /dev/pts, /proc and /sys.

Moreover, we need a default Bourne shell in /bin/sh that must be bash, as other more primitive Bourne compatible shells may give us trouble. Unfortunately, we cannot bind mount the host's system /bin folder, as it's filled with all kinds of executables causing impurities. Moreover, these executables have requirements on shared libraries residing in /lib, which we do not want to expose in the chroot environment.

I know two ways to have bash as /bin/sh in our chroot environment:

• We can install bash through Nix and expose that in the chroot environment:
  
  

  
  $ nix-env -i bash
  $ sudo mkdir -p /nix-bin
  $ sudo ln -s $(readlink -f $(which bash)) /nix-bin/sh
  

  The approach should work, as bash's dependencies all reside in the Nix store which is available in the chroot environment.
  
  
• We could also create a static bash that does not depend on anything. The following can be run to compile a static bash manually:
  
  

  
  $ ./configure --prefix=~/bash-static --enable-static-link \
      --without-bash-malloc --disable-nls
  $ make
  $ make install
  $ sudo mkdir -p /nix-bin
  $ sudo cp bash /nix-bin/sh
  

  
  Or by using the following Nix expression:
  
  

  
  with import <nixpkgs> {};
  
  stdenv.mkDerivation {
    name = "bash-static-4.2";
    src = fetchurl {
      url = mirror://gnu/bash/bash-4.2.tar.gz;
      sha256 = "1n5kbblp5ykbz5q8aq88lsif2z0gnvddg9babk33024wxiwi2ym2";
    };
    patches = [ ./bash-4.2-fixes-11.patch ];
    buildInputs = [ bison ];
    configureFlags = [
      "--enable-static-link"
      "--without-bash-malloc"
      "--disable-nls"
    ];
  }
  

Finally, we have to add a few properties to Nix's configuration to enable chroot builds:

$ sudo echo "build-use-chroot = true" >> /etc/nix/nix.conf
$ sudo echo "build-chroot-dirs = /dev /dev/pts /bin=/nix-bin" \
>> /etc/nix/nix.conf

The last line exposes /dev, /dev/pts and /nix-bin (mounted on /bin, containing only sh) of the host system to the chroot environment allowing us to build stuff purely.

Conclusion

In this blog post, I have described everything I did to set up a multi-user Nix installation supporting distributed and chroot builds on a conventional Linux distribution (Ubuntu) which was a bit tricky.

I'm planning to push some of things I did upstream, so that others can benefit from it. This is a good thing, because I have the feeling that most non-NixOS Nix users will lose their interest if they have figure out the same stuff I just did.

I've written a number of blog posts about JavaScript. Earlier, I had my frustrations with inheritance and prototypes, and I have developed NiJS: An internal DSL for Nix, to conveniently use the Nix package manager to deploy packages from JavaScript programs.

Besides the issues previously described, there is even more pain/frustration to cope with, which I'd like to discuss in this blog post. The main motivation to write this down is because I'm a control freak that wants to know what happens, it took me some effort discovering these details, and I have found myself repeating the same stuff to others.

Executing tasks concurrently

In addition to organizing data in a program, the ability to execute tasks concurrently is also important for many types of applications, such as:

• Applications with graphical user interfaces (GUIs). For example, if a user clicks on a button executing a task, the GUI must remain responsive while the task is being processed.
• Server applications. A server application has to serve multiple connected clients simultaneously, while each task remains responsive. For example, if a task is being processed of some client, it should not be necessary for a new client to wait until the requests of the others are done.
• Applications communicating with hardware peripherals. Many peripherals, such as (hard/DVD/Blu-Ray) disk drives or printers are significantly slower than the CPU of the computer. For example, it's quite inconvenient that execution of a program temporarily blocks the entire system when a file is being read from the hard drive. Instead, it's (obviously) better to keep the programming running (and respond on user events) while a file is being read, and to notify the user when the task is done.
  

To allow tasks to be executed concurrently, many operating systems/execution environments have facilities, such as interrupts, processes and/or threads allowing a developer to do that in a (slightly) convenient way. The execution environment ensures that these processes or threads are executed simultaneously (well not really, in fact it divides CPU execution time over processes and threads, but in order to keep this blog post short, I won't go into detail on that).

Executing tasks concurrently in JavaScript

Although most operating systems, execution environments, and programming languages have facilities for threads and/or processes, JavaScript -- a programming language originally used inside a web browser with increasing popularity outside the browser -- does not have such facilities. In JavaScript, there is only one single execution thread.

Consider the following example HTML page:

<!DOCTYPE html>

<html>
<head><title>Stretching button</title></head>

<body>
<h1>Stretching button!</h1>

<button id="button" onclick="stretchButton();">Stretch</button>

<script type="text/javascript">
        enabled = false;

        function updateButtonSizes() {
            var grow = true;
            var width = 100;
            var height = 20;
            var button = document.getElementById('button');

            while(enabled) {
                /* Update the button sizes */
                button.style.width = width + "px";
                button.style.height = height + "px";

                /* Adjust the button sizes */
                if(grow) { width++; height++; }
                else { width--; height--; }

                /* If the limits are reached, stretch in
                 * the opposite direction
                 */
                if(width >= 150) grow = false; else if(width <= 100) grow = true;
            }
        }

        function stretchButton() {
            if(enabled) {
                enabled = false;
            } else {
                enabled = true;
                updateButtonSizes();
            }
        }
</script>
</body>
</html>

The purpose of the HTML code shown above is to display a button that animates by increasing and decreasing its width and height once the user has clicked on it. The animation keeps repeating itself and stops if the user clicks on the same button again.

By just looking at the the code, it may give you the impression that it should work fine. However, once we have clicked on the button, the browser blocks completely and prevents us clicking on the button again. Furthermore, because we cannot click on the button again, we are also incapable of stopping the animation sequence. As a consequence, we have run into an infinite loop.

The reason that we run into trouble, is because if JavaScript code is being executed -- that runs in a single execution thread -- the browser cannot do anything else, such as responding to user events or updating the screen. After a minute or so, the browser will notice that some script is blocking it and asks you if it should terminate the script for you.

So you may probably wonder how to deal with this blocking issue? To allow the browser to respond to user events and update the screen, we must stop the execution of the program (by reaching the end of its control flow). Then at a later stage, we must resume execution by generating an event. Besides generating events from page elements, such as buttons as can be seen in the code fragment above, we can also generate timeout events:

setTimeout(function() {
    document.write("Hello world!<br>");
}, 1000);

In the snippet above, the setTimeout() function invocation tells the browser to execute the function displaying "Hello World" after 1000 milliseconds. Like events generated by page elements, timed events can also only be processed if nothing is being executed at that moment. So the program must reach the end of its execution flow first before the timeout event can be processed.

An adapted version of the earlier example with the button that correctly handles events and screen updates, may look as follows:

<!DOCTYPE html>

<html>
<head>
<title>Stretching button</title>
</head>

<body>
<h1>Stretching button!</h1>

<button id="button" onclick="stretchButton();">Stretch</button>

<script type="text/javascript">
            enabled = false;
            button = document.getElementById('button');

            function updateButtonSizes() {
                /* Update the button sizes */
                button.style.width = width + "px";
                button.style.height = height + "px";

                /* Adjust the button sizes */
                if(grow) { width++; height++; }
                else { width--; height--; }

                /* If the limits are reached then stretch in
                 * the opposite direction
                 */
                if(width >= 150) grow = false; else if(width <= 100) grow = true;

                /* Keep repeating this until the user disables it */
                if(enabled) setTimeout(updateButtonSizes, 0);
            }

            function stretchButton() {
                grow = true;
                width = 100;
                height = 20;

                if(enabled) {
                    enabled = false;
                } else {
                    enabled = true;
                    updateButtonSizes();
                }
            }
</script>
</body>
</html>

The above example code is quite similar to the previous one, with the following differences:

• The updateButtonSizes() function does not contain a while loop. Instead, it implements a single iteration step and calls setTimeout() to generate a timeout event that calls itself.
• After setting the timeout event, the program reaches the end of the execution flow.
• The browser updates the screen, handles user input and resumes execution by processing the timeout event that invokes updateButtonSizes() again.
• I also made several variables global so that their values are remembered after each execution of updateButtonSizes()

To prove that the above example works, I have included an embedded version in this blog post:

Cooperative multi-tasking

In short, to execute tasks in a browser while keeping the user interface responsive, we must do the following:

• Generate timeout events when it's desired to give other tasks the opportunity to do something.
• Reach the end of the control flow so that events can be processed by the execution environment. The generated timeout event calling a function ensures that execution is resumed at a later stage.

This approach looks very similar to cooperative multi-tasking using by classic Mac OS 9 and earlier and Windows operating systems prior to 3.x.

The fact that a function needs to be called every time you want to allow the browser/runtime environment to process events, reminds of the way I was developing programs many years ago on the AmigaOS. I still remember quite vividly that when using AMOS BASIC, if I wanted Workbench applications running in the background to do something while my program was running, I had to add the 'Multi Wait' instruction to the main loop of the program:

While busy
    ' Perform some tasks

    Multi Wait
Wend

By using the 'Multi Wait' instruction, the operating system gets notified, interrupts the program's execution and the scheduler decides which other process is allowed to use the CPU. Then at some point, the OS scheduler decides that my program is allowed to continue and the program's execution continues as if nothing happended.

However, compared the JavaScript approach, the AmigaOS approach is a bit more powerful since it also takes care of interrupting and resuming a program at the right location within the right context.

Cooperative multitasking works as long as tasks properly cooperate, putting great responsibility by developers of a program. It's not uncommon however, that people make mistakes and the system as a whole blocks forever.

Asynchronous programming

Apart from the browser, many other environments embedding a JavaScript runtime require developers to use cooperative multitasking techniques.

Node.js, a server-side software system designed for writing scalable Internet applications, uses almost the same approach as the browser just described earlier. One minor difference is that instead of using setTimeout(), a more efficient alternative is used:

process.nextTick(function() {
    process.stdout.write("Hello world!\n");
});

The process.nextTick() function executes the given callback function once the end of the execution flow of the program has been reached. Because it does not use the timer, it's more efficient.

To save users the burden of calling process.nextTick() and ending the execution flow each time something may block the system, most library functions in Node.js are asynchronous by default (with some having a synchronous alternative). Asynchronous functions have the following properties:

• They return (almost) immediately.
• A callback function (provided as function parameter) gets invoked when the task is done, which can be used to retrieve its result or status.
• They automatically generate tick events and reach the end of their control flow to prevent the system being blocked.

For example, to read a file and store its contents in a string without blocking the system, one could do:

var fs = require('fs');

fs.readFile("/etc/hosts", function(err, data) {
    if(err)
        process.stderr.write("Cannot read /etc/hosts\n");
    else
        process.stdout.write("/etc/hosts contains:\n" + data);
});

process.stderr.write("Hello!\n");

The above example opens /etc/hosts asynchronously. The fs.readFile() function returns immediately, then the string "Hello" is printed and the program reaches the end of its control flow. Once the file has been read, the callback function is invoked displaying either an error message (if the file cannot be opened) or the contents of the file.

Although asynchronous functions prevent the system being blocked and allow multitasking, one of its big drawbacks is that the callback mechanism makes it much more difficult to structure a program properly. For example, if I want to create a folder called "out", then a subfolder called "test" in the previous folder, inside the "test" subfolder a file called "hello.txt" containing "Hello world!", and finally verify whether the written text file contains "Hello world!", we may end up writing:

var fs = require('fs');
var path = require('path');

fs.mkdir("out", 0755, function(err) {
    if(err) throw err;

    fs.mkdir(path.join("out, "test"), 0755, function(err) {
        if (err) throw err;        
        var filename = path.join("out", "test", "hello.txt");

        fs.writeFile(filename, "Hello world!", function(err) {
            if(err) throw err;

            fs.readFile(filename, function(err, data) {
                if(err) throw err;

                if(data == "Hello world!")
                    process.stderr.write("File is correct!\n");
                else
                    process.stderr.write("File is incorrect!\n");
            });
        });
    });
});

For each step in the above sequential program structure, we end up one indentation level deeper. For the above example it's probably not so bad, but what if we have to execute 10 asynchronous functions sequentially?

Moreover, it is also non-trivial to modify the above piece of code. What, for example, if we want to write another text file, before the function that creates the test folder? It requires us to weave another function callback in the middle of the program and to refactor the entire indentation structure.

Fortunately, there are also solutions to cope with that. For example, the async library contains a collection of functions supporting control flow patterns and functions supporting data collections that can be used to deal with complexities of asynchronous functions.

The waterfall pattern, for example, can be used to execute a collection of asynchronous function sequentially and stops if any of the callbacks returns an error. This can be used to flatten the structure of our previous code example and also makes it better maintainable:

var fs = require('fs');
var path = require('path');

filename = path.join("out", "test", "hello.txt");

async.waterfall([
    function(callback) {
        fs.mkdir("out", 0755, callback);
    },

    function(callback) {
        fs.mkdir(path.join("out, "test"), 0755, callback);
    },

    function(callback) {
        fs.writeFile(filename, "Hello world!", callback);
    },

    function(callback) {
        fs.readFile(filename, callback);
    },

    function(data, callback) {
        if(data == "Hello world!")
            process.stderr.write("File is correct!\n");
        else
            process.stderr.write("File is incorrect!\n");
    }

], function(err, result) {
    if(err) throw err;
});

Besides the waterfall pattern, the async library supports many other control flow and collection patterns, such as executing functions in parallel or asynchronous loops over items in an array.

Mutable state

There is another important aspect that we have to keep in mind while executing stuff concurrently. Shared mutable state may influence the result of an execution of a task. For example, in our last example using async.waterfall(), I have declared the filename variable as a global variable, because I (naively) thought that it was a good way to share its value among the callback functions.

However, since it's possible to run multiple tasks concurrently, another task may change this global variable to something else. As a consequence, when we have reached the callback using it, it may has been changed to a different value and thus our expected result may differ. Moreover, it's also extremely hard to debug these kind of problems.

To protect ourselves from these kind of problems, it's better to scope a variable, which can be done with the var keyword. However, although most programming languages, such as Java (which influenced JavaScript) use block-level scoping, in JavaScript variables are function-level scoped.

For example, the following Java example shows the effect of block-level scoping:

public class Test {
    public static void main(String[] args) {
        int i = 1;
        int j = 0;

        if(j == 0) {
            int i = 2;
        }

        return i; // 1;
    }
}

In the example above we have two i variables. The first one is declared inside the main() body, the second inside the if-statement block. Because the variables are scoped within the nearest block boundary, the value of i inside the if-block is discarded after it terminates.

However, a direct port of the above code to JavaScript yields a different result, because i is scoped to the nearest function boundary:

function test() {
    var i = 1;
    var j = 0;

    if(j == 0) {
        var i = 2;
    }

    return i; // 2;
}

So in our JavaScript code fragment, the second var i; declaration inside the if-block is also bound to the scope of the test() function, therefore yielding a different result. Fortunately, block level scoping can be easily simulated by declaring a function without parameters and calling it without parameters (function() { var x; ... } ();):

function test() {
    var i = 1;
    var j = 0;

    if(j == 0) {
        function() {
            var i = 2; /* i is scoped within the surrounding function */
        }();
    }

    return i; // 1;
}

Similarly this trick also allows us to scope the filename variable shown in our earlier example with async.waterfall():

function() {
    var filename; /* filename is scoped within the surrounding function */

    async.waterfall(...);
}();

In the above example, the filename variable is not global anymore. The callback functions inside async.waterfall() will not refer to the global variable named filename any longer, but to the scoped variable inside the function block. As a result, other tasks running concurrently adapting global variables cannot influence this task any longer.

Conclusion

In this blog post, I have described some of the pains I have experienced with multitasking inside a JavaScript environment. From this, I have learned the following lessons:

• If a script is being executed, then nothing else can be done, such as screen updates, processing user events, handling client connections etc.
• To allow multitasking (and prevent the system being blocked entirely) we have to cooperate by generating timeout or tick events and ending the control flow of the program.
• Asynchronous functions use callbacks to retrieve their results and/or statuses making it extremely hard to structure your program properly. In other words, it easily becomes a mess. To cope with this, you need to think about patterns to combine them. Libraries, such as async, may help you with this.
• Shared state is evil, error prone and makes a program hard/impossible to debug. Therefore, only use global variables that are immutable or refer to singleton instances (such as CommonJS modules). Otherwise, scope them.
• JavaScript does not support block level scoping, but it can be simulated by encapsulating a block inside a function that gets called without parameters. Fortunately, the ECMAScript 6 standard defines the let keyword supporting block-level scoping natively, but it's not supported in most implementations, such as in Node.js.

Although I'm a little happy about the fact that I know how to handle these problems, I'm very sad at the same time because of the following reasons:

• JavaScript was originally advertised as "a lightweight and simpler variant of Java". The fact that every JavaScript developer must cope with all these concurrency issues, which Java mostly solves with a threading API, definitely makes JavaScript NOT simpler than Java in this respect IMHO.
• Cooperative multitasking is a technique that was very common/useful in the early 1980s (and probably invented years before it), but has been replaced by preemptive multitasking in modern operating systems. One of the reasons to take responsibility away from the processes (besides the fact that it makes the lives of programmers easier) is because they cannot be trusted. However, we still seem to "trust" JavaScript code embedded in malicious web pages.
• We're using a programming language for all kinds of purposes for which it has not been designed. Moreover, we require solutions for these issues that are not part of the language, API or runtime, such as (ab)using the timer and relying on external libraries implementing patterns to combine asynchronous functions.
• Sometimes these new use cases of JavaScript are called innovation, but the fact that we use techniques that are over 30 years old (and intentionally not using better modern alternatives), does not really look like innovation to me IMHO.

So are solutions using JavaScript that bad? I see that there is practical use and value in it (e.g. the V8 runtime compiles to efficient native code and there is a big eco-system of JavaScript libraries), but dealing with concurrency like this is definitely something I consider a big drawback and a huge step backwards.

Some time ago, I have packaged the Android SDK and some of its plugins in Nix, which was quite a challenge. Moreover, I have developed two Nix functions that can be used to automatically build Android apps and spawn emulator instances.

Meanwhile, I have implemented some major updates since the last blog post. Besides upgrading to a newer Android SDK version, we now also support a bunch of new interesting features.

Android SDK packaging updates

Currently, I have packaged the 22.05 version of the Android SDK (which is the latest version at the time writing this blog post). To package it, I have followed the same recipe as described in the previous blog post, with some additions:

• Newer versions of the Android SDK include x86-64 versions of the emulator executables. I had to patch/wrap these executables to allow them to find the x86-64 libraries they depend on.
• The newer Android SDK has a new sub package called build-tools. Apparently, this sub package contains some tools that were formerly part of platform-tools, such as debugging libraries.
• The GUI front-end of the android utility did not work in the last blog post, which I have fixed by wrapping it, so that it can find its dependencies, such as GTK+.
• There are also x86 and MIPS application binary interface (ABI) system images, but they reside in external package repositories. I have adapted the Nix expression generator to include these as well.
• I have adapted the build and emulation Nix functions to take both API-level and ABI versions into account
• I have implemented some facilities to improve the performance of the emulator instances

Packaging build-tools

The build-tools package contains various tools and libraries that used to belong to platform-tools. It turns out that some of these files, such as the debugging libraries, are API-level specific. The purpose of the build-tools package is to provide API-level specific build tools. Currently, the Android SDK supports two versions: 17 and 18.01 that can be installed next to each other.

I have packaged the 18.01 version in a straight forward manner. The only thing I needed to do is inspecting the dependencies of the executables to the paths to these libraries to the executables' RPATH.

For example, I did this to patch aapt:

patchelf --set-rpath ${stdenv.gcc.libc}/lib:${stdenv.gcc.gcc}/lib:${stdenv.zlib}/lib aapt

Every version of the build-tools package is symlinked into the Android SDK basedir as follows: build-tools/android-<version>

Fixing the Android GUI front-end

If the android tool is called without any parameters, it should show a GUI that allows somebody to download plugins or to configure AVDs. However, the GUI did not work so far. Invoking 'android' gave me the following error message:

$ android
Exception in thread "main" java.lang.UnsatisfiedLinkError:
no swt-pi-gtk-3550 or swt-pi-gtk in swt.library.path,
java.library.path or the jar file
        at org.eclipse.swt.internal.Library.loadLibrary(Unknown Source)
        at org.eclipse.swt.internal.Library.loadLibrary(Unknown Source)
        at org.eclipse.swt.internal.gtk.OS.(Unknown Source)
        at org.eclipse.swt.internal.Converter.wcsToMbcs(Unknown Source)
        at org.eclipse.swt.internal.Converter.wcsToMbcs(Unknown Source)
        at org.eclipse.swt.widgets.Display.(Unknown Source)
        at com.android.sdkmanager.Main.showSdkManagerWindow(Main.java:346)
        at com.android.sdkmanager.Main.doAction(Main.java:334)
        at com.android.sdkmanager.Main.run(Main.java:120)
        at com.android.sdkmanager.Main.main(Main.java:103)

First, I thought that some file belonging to SWT was missing, but after manually digging through all Android files I couldn't find it and I gave up. However, after using strace, I discovered that it's actually trying to load the GTK+ library and some other libraries:

$ strace -f android
...
[pid  8169] stat("/tmp/swtlib-64/libswt-pi-gtk.so", 0x7fe4e0cffb20) = -1 ENOENT (No such file or directory)
[pid  8169] stat("/tmp/swtlib-64/libswt-pi-gtk-3550.so", {st_mode=S_IFREG|0755, st_size=452752, ...}) = 0
[pid  8169] stat("/tmp/swtlib-64/libswt-pi-gtk-3550.so", {st_mode=S_IFREG|0755, st_size=452752, ...}) = 0
[pid  8169] open("/tmp/swtlib-64/libswt-pi-gtk-3550.so", O_RDONLY|O_CLOEXEC) = 26
[pid  8169] read(26, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\234\3\0\0\0\0\0"..., 832) = 832
[pid  8169] fstat(26, {st_mode=S_IFREG|0755, st_size=452752, ...}) = 0
[pid  8169] mmap(NULL, 1503944, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 26, 0) = 0x7fe4b5d7e000
[pid  8169] mprotect(0x7fe4b5dea000, 1044480, PROT_NONE) = 0
[pid  8169] mmap(0x7fe4b5ee9000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 26, 0x6b000) = 0x7fe4b5ee9000
[pid  8169] mmap(0x7fe4b5eec000, 4808, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fe4b5eec000
[pid  8169] close(26)                   = 0
[pid  8169] open("/run/opengl-driver/lib/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  8169] open("/run/opengl-driver-32/lib/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  8169] open("/nix/store/zm4bhsm8lprkzvrjgqr0klfkvr21als4-glibc-2.17/lib/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
...

To allow GTK+ and the other missing libraries to be found, I have wrapped the android utility:

wrapProgram `pwd`/android \
   --prefix PATH : ${jdk}/bin \
   --prefix LD_LIBRARY_PATH : ${glib}/lib:${gtk}/lib:${libXtst}/lib

Besides the libraries, I also had to wrap android to use the right JDK version. Apparently, loading these libraries only works with OpenJDK in Nix.

After wrapping the android utility and running it without parameters, it shows me the following:


As you may see in the above image, the GUI works. It also shows us all the plugins that are installed through Nix. Although the GUI seems to work fine, we have to keep in mind that we cannot use it to install additional Android SDK plugins. All plugins must be packaged and installed with Nix instead.

Supporting external system images

The Android distribution provides a collection of system images that can be used with an emulator to test an app during development. However, the default repository only contains ARM-based system images, which are quite slow to emulate.

There also seem to be x86-based system images available (provided by Intel) and MIPS-based images (provided by MIPS Technologies). These system images reside in external repositories. I was able to discover the locations of these repositories, by running the following command:

$ android list sdk
Refresh Sources:
  Fetching https://dl-ssl.google.com/android/repository/addons_list-2.xml
  Validate XML
  Parse XML
  Fetched Add-ons List successfully
  Refresh Sources
  Fetching URL: https://dl-ssl.google.com/android/repository/repository-8.xml
  Validate XML: https://dl-ssl.google.com/android/repository/repository-8.xml
  Parse XML:    https://dl-ssl.google.com/android/repository/repository-8.xml
  Fetching URL: https://dl-ssl.google.com/android/repository/addon.xml
  Validate XML: https://dl-ssl.google.com/android/repository/addon.xml
  Parse XML:    https://dl-ssl.google.com/android/repository/addon.xml
  Fetching URL: https://dl-ssl.google.com/android/repository/extras/intel/addon.xml
  Validate XML: https://dl-ssl.google.com/android/repository/extras/intel/addon.xml
  Parse XML:    https://dl-ssl.google.com/android/repository/extras/intel/addon.xml
  Fetching URL: https://dl-ssl.google.com/android/repository/sys-img.xml
  Validate XML: https://dl-ssl.google.com/android/repository/sys-img.xml
  Parse XML:    https://dl-ssl.google.com/android/repository/sys-img.xml
  Fetching URL: https://dl-ssl.google.com/android/repository/sys-img/mips/sys-img.xml
  Validate XML: https://dl-ssl.google.com/android/repository/sys-img/mips/sys-img.xml
  Parse XML:    https://dl-ssl.google.com/android/repository/sys-img/mips/sys-img.xml
  Fetching URL: https://dl-ssl.google.com/android/repository/sys-img/x86/sys-img.xml
  Validate XML: https://dl-ssl.google.com/android/repository/sys-img/x86/sys-img.xml
  Parse XML:    https://dl-ssl.google.com/android/repository/sys-img/x86/sys-img.xml

The latter two URLs referring to a file called sys-img.xml provide x86 and MIPS images. The structure of these XML files are similar to the main repository. Both files contain XML elements that look like this:

<sdk:system-image>
<sdk:description>Android SDK Platform 4.1.1</sdk:description>
<sdk:revision>1</sdk:revision>
<sdk:api-level>16</sdk:api-level>
<sdk:abi>x86</sdk:abi>
<sdk:uses-license ref="intel-android-sysimage-license">
<sdk:archives>
<sdk:archive arch="any" os="any">
<sdk:size>131840348</sdk:size>
<sdk:checksum type="sha1">9d35bcaa4f9b40443941f32b8a50337f413c021a</sdk:checksum>
<sdk:url>sysimg_x86-16_r01.zip</sdk:url>
</sdk:archive>
</sdk:archives>
</sdk:uses-license></sdk:system-image>

These elements can be converted to a Nix expression through XSL in a straight forward manner:

{stdenv, fetchurl, unzip}:

let
  buildSystemImage = args:
    stdenv.mkDerivation (args // {
      buildInputs = [ unzip ];
      buildCommand = ''
        mkdir -p $out
        cd $out
        unzip $src
    '';
  });
in
{
  sysimg_x86_16 = buildSystemImage {
    name = "sysimg-x86-16";
    src = fetchurl {
      url = https://dl-ssl.google.com/android/repository/sys-img/x86/sysimg_x86-16_r01.zip;
      sha1 = "9d35bcaa4f9b40443941f32b8a50337f413c021a";
    };
  };

  ...
}

The generated expression shown above is nearly identical to the one shown in the previous blog post. The only difference is that we also use the ABI identifier in the attribute names. For example: sysimg_x86_16 is used to refer to a x86-based system image for Android API-level 16. Likewise, we can refer to the ARM-based variant with: sysimg_armeabi-v7a_16.

Apart from giving each attribute a proper name, the resulting system image package must be symlinked into the Android SDK basedir taking the ABI identifier into account. We symlink every architecture-dependent system image package into: system-images/android-<api-level>/<abi>.

I have adapted the androidenv.emulateApp {} Nix function to also take ABI versions into account. By default, the emulator function uses an ARM-based system image (armeabi-v7a), since this is the most common hardware architecture used by Android OSes. By setting the abiVersion parameter to x86, we can use an x86-based Android system-image:

{androidenv, myfirstapp}:

androidenv.emulateApp {
  name = "MyFirstApp";
  app = myfirstapp;
  platformVersion = "16";
  abiVersion = "x86";
  package = "com.example.my.first.app";
  activity = "MainActivity";
}

An interesting benefit of using an x86 system image with the emulator (which is based on QEMU) is that KVM can be used, allowing programs inside the emulator to run at nearly native speed with only little emulation.

And of course, setting the abiVersion parameter to mips makes it possible to use MIPS-based system images.

Enabling GPU acceleration

The x86 system images in the emulator may give testing a significant boost. Emulation performance can even be improved a bit further by enabling GPU acceleration.

To make GPU acceleration possible, I had to adapt the emulator's wrapper script to include Mesa in the linker path. Moreover, in the AVD configuration I have to add the following boolean value to $ANDROID_SDK_HOME/.android/avd/device.avd/config.ini:

hw.gpu.enabled = yes

The androidenv.emulateApp {} automatically enables GPU acceleration from API-levels 15 and onwards. It can be disabled by setting the enableGPU parameter to false.

Supporting Google APIs in non-ARM-based system images

Although supporting x86 system images have various benefits, they also have a major drawback -- the Google APIs are not supported preventing some applications from working. One of the solutions I have seen (described in this Stack Overflow post) is to create a custom system image by starting an ARM-based emulator instance with Google APIs, fetch the libraries, start an x86-based emulator instance without Google APIs and manually install the fetched libraries.

I was able to automate the steps in the Stackflow article in a Nix expression:

{ platformVersion ? "16"
, abiVersion ? "x86"
, androidenv
, jdk
}:

let
  androidsdk = androidenv.androidsdk {
    platformVersions = [];
    abiVersions = [];
    useGoogleAPIs = false;
  };

  emulateAndroidARMWithGoogleAPIs = androidenv.emulateApp {
    name = "emulate-arm";
    inherit platformVersion;
    abiVersion = "armeabi-v7a";
    useGoogleAPIs = true;
  };

  emulateAndroidX86WithoutGoogleAPIs = androidenv.emulateApp {
    name = "emulate-x86";
    inherit platformVersion abiVersion;
    useGoogleAPIs = false;
  };

  mkfsYaffs2X86 = fetchurl {
    url = http://android-group-korea.googlecode.com/files/mkfs.yaffs2.x86;
    sha1 = "9362064c10a87ca68de688f09b162b171c55c66f";
  };
in
stdenv.mkDerivation {
  name = "sysimg_x86_${platformVersion}-with-google-apis";
  buildInputs = [ jdk androidsdk ];
  NIX_ANDROID_EMULATOR_FLAGS = "-no-window";
  buildCommand = ''
    source ${emulateAndroidARMWithGoogleAPIs}/bin/run-test-emulator
    portARM=$port
    adb -s emulator-$portARM pull /system/etc/permissions/com.google.android.maps.xml
    adb -s emulator-$portARM pull /system/framework/com.google.android.maps.jar
    adb -s emulator-$portARM emu kill

    export NIX_ANDROID_EMULATOR_FLAGS="$NIX_ANDROID_EMULATOR_FLAGS -partition-size 1024 -no-snapshot-save"
    source ${emulateAndroidX86WithoutGoogleAPIs}/bin/run-test-emulator
    portX86=$port
    adb -s emulator-$portX86 remount rw
    adb -s emulator-$portX86 push com.google.android.maps.jar /system/framework
    adb -s emulator-$portX86 push com.google.android.maps.xml /system/etc/permissions

    cp ${mkfsYaffs2X86} mkfs.yaffs2.x86
    adb -s emulator-$portX86 push mkfs.yaffs2.x86 /data
    adb -s emulator-$portX86 shell chmod 777 /data/mkfs.yaffs2.x86
    adb -s emulator-$portX86 shell /data/mkfs.yaffs2.x86 /system /data/system.img
    adb -s emulator-$portX86 pull /data/system.img
    adb -s emulator-$portX86 emu kill

    mkdir -p $out
    cp system.img $out
  '';
}

The above Nix function creates an ARM emulator instance with Google APIs and an x86 emulator instance without Google APIs. Then it starts the ARM emulator instance, fetches the Google API files from it and then stops it. Then the x86 emulator instance is started and the Google APIs are pushed to it. Finally we generate a system image from the system folder that is stored in the Nix store.

By using the Nix function shown above and setting the extraAVDFiles parameter of the emulateApp {} function, we can use our custom system image in a script that automatically spawns the emulator instance:

{androidenv, jdk}:

let
  platformVersion = "16";
  systemImg = import ./generateGoogleAPISystemImage.nix {
    inherit androidenv jdk platformVersion;
    abiVersion = "x86";
  };
in
androidenv.emulateApp {
  name = "MyFirstApp";
  extraAVDFiles = [ "${systemImg}/system.img" ];
  inherit platformVersion;
  abiVersion = "x86";
}

The above expression invokes the function shown previously and starts an emulator instance with the system image containing Google APIs. Of course, we can also use the same trick for MIPS-based system images, which also don't include Google APIs.

Building Android apps for arbitrary API-level revisions

I have also made a small change to the androidenv.buildApp {} Nix function to take API-levels into account. By default, it will use the version that is inside the Ant configuration. However, by setting the target property through the command-line, we can override this:

$ ant -Dtarget=android-17

The above command-line instruction ensures that we build the app against the API-level 17 platform. I have added an antFlags parameter to the Nix function to make it provide arbitrary flags to Ant:

{androidenv}:

androidenv.buildApp {
  name = "MyFirstApp";
  src = ../../src/myfirstapp;
  platformVersions = [ "17" ];
  useGoogleAPIs = true;
  antFlags = "-Dtarget=android-17";
}

Improving the test suite

In the previous blog post, I have implemented a testcase based on the Android introduction tutorial (MyFirstApp) that I have used throughout this article as well. To test all these new features, I have extended its composition expression to take three parameters: buildPlatformVersions (defaults to 16), emulatePlatformVersions (defaults to 16), and abiVersions (defaults to armeabi-v7a). The expression automatically generates all possible combinations with the values inside these three lists:

For example, to build a debug version of the test app APK for API-level 16, we can run:

$ nix-build -A myfirstapp_debug.build_16

We can also build a release version of the same APK that is signed with a key:

$ nix-build -A myfirstapp_release.build_16

Furthermore, we can automatically generate a script starting an emulator instance running the app. The following instruction builds an App for Android API-level 16, generates a script launching an emulator with a Android API-level 16 system-image that uses the armeabi-v7a ABI:

$ nix-build -A emulate_myfirstapp_debug.build_16.emulate_16.armeabi-v7a
$ ./result/run-test-emulator

To support more API-levels and ABIs, the parameters of the composition function must be altered. For example, by running the following command-line instruction:

$ nix-build --arg buildPlatformVersions '[ "16" "17" ]' \
  --arg emulatePlatformVersions '[ "15" "16 " 17" ]' \
  --arg abiVersions '[ "armeabi-v7a" "x86" ]' \
  -A emulate_myfirstapp_release.build_16.emulate_17.x86

The cartesian product of build and emulator instances is created taking the three dimensions into account. This allows us to (for example) build the App against the API-level 16 platform API, emulate on an API-level 17 system image using the x86 ABI, which emulates much more efficiently on x86 machines, because hardware visualization features (such as KVM) can be used.:

Apart from specifying these dimensions through the command line, the composition expression can also be used together with Hydra, the Nix-based continuous integration server, allowing it to pass these parameters through its web interface. Hydra will build all the combinations generated by the composition expression automatically:



As can be observed from the above screenshot, quite a few jobs are being generated.

Conclusion

In this blog post, I have described some new changes and features that I have implemented in the Nix Android SDK package since the last blog post, such as the fact that x86 and MIPS system-images can be used. All these changes are part of the Nixpkgs project.

Moreover, the Nix Android example testcase can be obtained from my GitHub page.

In the previous blog post, I have described some new features of the Nix Android build environment. Apart from Android, the Nix iOS build environment has a few improvements as well, which are mostly related to testing.

Simulating multiple iOS versions

In the previous blog post about the Nix iOS build function, I have described how to install various iPhone simulator SDK versions by picking the preferences option from the 'Xcode' menu bar and selecting the 'Downloads' tab in the window, as shown in the screenshot below:


However, I did not implement any facilities in the simulator function that take these SDK versions into account yet.

Picking a specific SDK version can be done quite easily, by providing the -currentSDKRoot parameter to the iPhone simulator, such as:

$ "iPhone Simulator" -SimulateApplication build/HelloWorld \
  -SimulateDevice 'iPhone' -currentSDKRoot \
  "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform\
/Developer/SDKs/iPhoneSimulator6.1.sdk"

In the above case, we have configured the simulator to use the iPhone simulator 6.1 SDK. By changing this parameter, we can use a different version, such as version 5.1.

I have encapsulated the above command-line invocation in the xccodeenv.simulateApp {} Nix function and the paths to the iPhone simulator SDKs in the xcodewrapper. The SDK version can be configured by providing the sdkVersion parameter (which defaults to 6.1):

{xcodeenv, helloworld, device}:

xcodeenv.simulateApp {
  name = "HelloWorld";
  app = helloworld;
  inherit device;
  sdkVersion = "6.1";
}

This parameter makes it possible to spawn a simulator instance running a specific iOS version.

The described facilities earlier are only for simulating apps. To build an app for a specific iOS revision, the iOS target version must be changed inside the Xcode project settings.

Building signed Apps for release

Apart from the simulator, we may also want to deploy apps to real devices, such as an iPhone, iPad or iPod, either directly or through the App store. In order to do that, we require permission from Apple by signing the apps that we have just built.

As described in the previous blog post, we require a certificate that contains a developer's or company's identity and a mobile provisioning file describing which (groups of) apps of a certain company/developer can be run of which (groups of) devices. These files must be obtained through Apple's Dev Center.

Because of these restrictions, it's hard to test the trivial "Hello World" example case I have developed in the previous blog post on a real device, since it contains a dummy app and company name.

To alleviate these problems, I have created a script that "renames" the example app into a different app, so that an existing developer or company certificate and a mobile provisioning profile of a different app can be used.

By setting the rename parameter to true when calling the composition expression of the example case, we can automatically generate jobs building IPAs and xcarchives for the "renamed" app:

import ./nix-xcodeenvtests/deployment {
  rename = true;
  newName = "MyVeryCoolApp";
  newId = "coolapp";
  newCompanyName = "My Cool Company";
  newDomain = "com.coolcompany";
  ipaCertificateFile = ./adhoc.p12;
  ipaCodeSignIdentity = "iPhone Distribution: My Cool Company";
  ipaCertificatePassword = "";
  ipaProvisioningProfile = ./adhoc.mobileprovision;
  xcArchiveCertificateFile = ./distribution.p12;
  xcArchiveCodeSignIdentity = "iPhone Distribution: My Cool Company";
  xcArchiveCertificatePassword = "";
  xcArchiveProvisioningProfile = ./distribution.mobileprovision;
}

In the above expression, we changed the name of the app into: "MyVeryCoolApp" and the company name into: "My Cool Company". Moreover, we provide a certificate and mobile provisioning file for the IPA job (which can be deployed to a device directly) and the xcarchive job (which can be used to deploy to the App store).

By running the following command-line instruction:

$ nix-build -A renamedPkgs.testapp_ipa

We can build an IPA allowing us to deploy the test app on a real device. Of course, to do this yourself, the company name and app names should correspond to the ones defined in your certificate and mobile provisioning profile.

Moreover, as with the Android environment in the previous blog post, we can also use Hydra to provide parameters to the composition expression and to retrieve all the resulting artifacts:

Conclusion

In this blog post, I have described some improvements to the Nix iOS facilities. We are now also capable of configuring the SDK versions for the iPhone simulator, and we can build the trivial testcase to run on a real device in addition to the simulator. This is useful for testing the Nix iOS build function.

For some reason, I'm having quite some discussions about Nix's user environment mechanism lately. For example, in my last two talks I have spent a considerable amount of time answering questions that were related to it.

The user environment mechanism is an important key feature of Nix, which primary purpose is to allow someone to conveniently launch programs installed by the Nix package manager, but it has a few additional use cases as well, such as determining which packages are in use (and which are not) in order to safely garbage collect obsolete packages.

I have only briefly described them in some of my blog posts, but I have never covered in detail how they work on my blog. Because of the amount of questions I have received lately, I think it would be a good idea to write about it.

The Nix store

As described in my exhaustive explanation blog post on Nix (and the original explanation recipe), every Nix package is stored in an isolated directory in a so called "Nix store". In the Nix store every package has a special filename that may look like this:

/nix/store/spy13ngvs1fyj82jw2w3nwczmdgcp3ck-firefox-23.0.1

The first part of the basename (spy13ngvs1fyj82jw2w3nwczmdgcp3ck) is a cryptographic hash derived from all build inputs used to build Firefox. For example, if we would build Firefox with a different compiler or for a different architecture, the generated hash code will be different, which makes it possible to safely store multiple variants of packages next to each other, because none of them will have the same name.

Although this storing convention looks a bit odd, it provides a number of interesting benefits. Since Nix executes builds inside environments in which as many side effects are removed, we can use the hash codes to uniquely identify package variants regardless on what machine they have been built.

They also provide better reproducibility, since these paths to Nix packages including their hash codes must always be explicitly specified, in order to allow a package build to find its dependencies. The advantage of this is that unspecified dependencies cannot accidentally allow a build to succeed, limiting reproducibility guarantees.

More details on the benefits can be found in my two explanation recipes. However, storing packages this way also causes a major inconvenience for end-users. To launch a particular program, the user must always provide the full path to the executable that resides inside the Nix store, which is quite impractical due to the fact that every Nix package has a hash code in it.

Nix profiles

As a means for users to conveniently launch software, the Nix package manager provides the concept of Nix profiles, which are user environments exposing a generation of installed packages to a user. Software packages can be automatically installed in Nix profiles by using the nix-env command-line utility.

To install software, such as Info ZIP's zip and unzip utilities, we can run:

$ nix-env -i zip unzip
installing `zip-3.0'
installing `unzip-6.0'
building path(s) `/nix/store/1zhgva9cmfz0wam9m7ibgaih2m3cic6l-user-environment'
created 33 symlinks in user environment

Allowing us to run any of the installed tools without specifying any paths with hash codes, e.g.:

$ zip --version
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
Currently maintained by E. Gordon.  Please send bug reports to
the authors using the web page at www.info-zip.org; see README for details.

One of the things the above nix-env command-line invocation does is automatically building the requested packages (or downloading its substitutes) and composing a user environment, which is a symlink tree that synthesizes the contents of the currently installed packages (zip and unzip) into one single component. A user environment is also a package residing in the Nix store, as shown in the following figure:


Besides creating a user environment that exposes all installed packages from a single location, a few additional symlink indirections are used. Each time nix-env performs a deployment activity, a generation symlink is generated (called profile-1 in the figure above). There is also a default symlink (profile) pointing to the generation symlink that is currently active. That symlink is also referenced from the user's home directory through ~/.nix-profile. By adding the latter symlink to the user's PATH, e.g.:

export PATH=~/.nix-profile/bin:$PATH

Packages installed through nix-env can be found automatically through a few layers of symlink indirections.

When we perform another deployment activity with nix-env, such as removing zip:

$ nix-env -e zip
uninstalling `zip-3.0'

Then a new symlink tree is composed containing the then currently installed packages (in this case only unzip), a new generation symlink is generated (profile-2), and the default symlink is updated to point to the new generation. Flipping symlinks is actually an atomic operation on UNIX. As a consequence, we can perform upgrades atomically, because there is never a situation in which we refer to packages belonging to the old and new user environment at the same time.


As may be observed in the picture above, the old generation user environment is still present allowing us to do rollbacks as well. For example, if we regret uninstalling zip, we can switch back to the previous generation by running:

$ nix-env --rollback

Then the default symlink's reference is updated to point to the previous generation symlink that provides the zip utility. Again, this is an atomic operation.

Apart from exposing packages installed by users, the directory containing profile generations also serves a second goal; it is used to determine which packages are in use and which are not.

The generations of the Nix profiles of all users of the system are treated as so called garbage collector roots -- any package that is installed in a profile generation and any of its dependencies are considered packages that are live (a.k.a. in use) and should not be garbage collected. Furthermore, Nix also treats open files and running processes as garbage collector roots. Therefore, it is safe to run the Nix garbage collector at any time.

We can also remove older Nix profile generations if we don't need them anymore:

$ nix-env --delete-generations old

The above command only removes the older generation symlinks, but not the packages that have become obsolete. To actually remove packages from the system that are no longer in use, we must explicitly run the garbage collector:

$ nix-collect-garbage

So far, I have shown examples that work on a per-user profile. In multi-user Nix installations, such as in NixOS, every user has a personal Nix profile (residing in /nix/var/nix/profiles/per-user) and there is also a system-wide default Nix profile (residing in /nix/var/nix/profiles/default) containing packages that are exposed to all users of a system.

NixOS system profiles

Apart from individual packages, NixOS -- the Linux distribution built around the Nix package manager, also uses Nix profiles to manage generations of system configurations.

As described in my earlier blog post about NixOS, a complete system configuration is deployed from a single declarative specification, including all system services and their configurations, the desktop, and user packages. Each time we deploy a system configuration, e.g. by running:

$ nixos-rebuild switch

a Nix profile is built which consists of (mostly) a symlink tree containing all packages and configuration files that constitutes all static parts of a system, including the Linux kernel and the kernel modules. This profile is called a system profile and offers the same benefits as ordinary package profiles, such as the fact that we can store multiple generations next to each other and perform atomic upgrades and rollbacks.

There is a subtle difference with NixOS deployment compared to ordinary package deployment. Merely flipping symlinks is not enough to get a new NixOS configuration deployed. We also have to perform a collection of imperative deployment steps, such as setting up dynamic directories, such as /var, starting and stopping system services, and updating the GRUB bootloader configuration. This is done by an activation script (/run/current-system/bin/switch-to-configuration) part of the NixOS system profile, that is executed each time a new version of a NixOS configuration is deployed.


As can be seen in the screenshot above, once a new NixOS system configuration has been deployed, the GRUB bootloader can be used to boot into any previous NixOS configuration that has not been garbage collected yet.

Generating user environments from Nix expressions

In addition to using nix-env, user environments can also be generated from Nix expression by using the buildEnv {} function. The following Nix expression produces the same user environment (shown earlier) containing zip and unzip:

with import <nixpkgs> {};

buildEnv {
  name = "zipprofile";
  paths = [ zip unzip ];
}

This expression can be built by running:

$ nix-build zipprofile.nix

Conclusion

In this blog post, I have explained how the Nix user environment mechanism works. Essentially user environments are symlink trees that blend all currently installed packages into a single component so that they can be referenced from a single location. Another symlink layer manages generations of user environments. We can atomically switch between these generations by flipping symlinks. By adding the latter symlink to a user's PATH, packages can be automatically found without providing any absolute paths with hash codes in them that are difficult to remember.

Besides exposing packages in a user's PATH, Nix profiles are also used in NixOS to automatically find other resources, such as manual pages (through MANPATH) and KDE desktop menu items (through XDG_DATA_DIRS).

More information on Nix profiles can be found in Eelco Dolstra's PhD thesis which can be obtained from his publications page. Additionally, my PhD thesis also covers Nix profiles in its background chapter. Furthermore, there is a chapter that extends their concepts to distributed systems. This chapter is loosely based on our HotSWUp 2008 paper titled: "Atomic Upgrading of Distributed Systems" which can be obtained from my publications page.

As described in a blog post from a while ago, NixOS deviates on certain aspects from the Filesystem Hierarchy Standard (FHS) for very good reasons. For example, NixOS lacks standard directories such as /usr/bin and /usr/lib that conventional Linux distributions have. The most important reason that Nix/NixOS deviates from the FHS is to provide reliable and reproducible deployment of packages.

The filesystem organization that Nix uses, in which every package is stored in a separate folder in a so called "Nix store", is typically not a problem to get software packaged and deployed in source form.

Packages in Nix are deployed from build recipes called Nix expressions that typically invoke functions implementing abstractions ensuring that the specified dependencies can be found in a convenient manner.

For example, when providing Java or Perl dependencies, the generic builder may automatically set some required environment variables, such CLASSPATH or PERL5LIB containing Java or Perl dependencies. Moreover, a number of build utilities have been patched (such as ld) to automatically add shared libraries to an ELF executable's RPATH allowing it to find the libraries that it needs (you could see this as as static linking of shared libraries to an executable).

However, deployment of third party binary packages is a bit complicated and inconvenient in NixOS for the following reasons:

• Nearly all software must be deployed by the Nix package manager, since nearly all of the system's components reside in the Nix store.
• Third party binaries frequently require shared libraries and other kinds dependencies to run. However, in most cases these dependencies cannot be found, because executables assume that they reside in standard FHS folders such as /usr/lib, which do not exist in NixOS.
• Executables using dynamic libraries cannot find the dynamic linker (e.g. /lib/ld-linux.so.2), because the corresponding file does not exist in NixOS.

As a solution, we often use PatchELF to change the path to the dynamic linker and modify the RPATH of a binary allowing it to find the libraries that it needs. We may also have to wrap executables in shell scripts to set certain environment variables, such as LD_LIBRARY_PATH.

Some applications that are packaged in Nixpkgs this way are Adobe Reader, Belastingaangifte (Dutch Tax administration) and the Android SDK, as I have described in twoearlier blog posts.

However, I have recently developed something that can solve this problem in a different way.

Composing FHS-compatible chroot environments

In my previous blog post, I have described how Nix user environments work, which are symlink trees that blend the contents of a collection of packages into a single symlink tree. User environments are primarily used to allow end-users to conveniently access packages installed by Nix from a single location.

We can also use the user environment mechanism in combination with setting up a chroot environment to simulate an FHS-compatible filesystem. This allows us to run an unpatched binary that works out of the box, if all required dependencies of the package are (correctly) specified.

I have created a Nix function called buildFHSChrootEnv {} which makes it possible to compose such environments. This function roughly does the following:

• It invokes buildEnv {} to compose a so-called "/usr user environment". In this user environment, we synthesize the contents of all our specified packages into a single Nix profile. This profile contains all static parts of all packages and is supposed to simulate both the /usr directory as well as the static folders in the root folder of the chroot environment, such as /bin and /lib.
• Besides the dependencies of the binary that we want to run, we also need a collection of base packages, such as the bash shell, GNU Coreutils, glibc, and the shadow password suite, since we also have to be able to run a basic Linux system and do basic shell activities inside the chroot environment. These packages are always part of the /usr profile.
• It generates a global bash profile setting the PS1 environment variable to configure the shell prompt and other miscellaneous settings that a program may need in order to run.
• It generates a collection of scripts taking care of setting up (or destroying) the dynamic parts of the chroot environment, since Nix packages and user environment are static and can never change after they have been built.

Setting up the dynamic parts of the chroot environment

Merely composing a /usr profile is not enough to get a working chroot environment because they only contain the static parts of a system, such as executables, libraries, and data files. We also have to set up a few dynamic parts. A collection of configuration scripts bundled with the generated /usr profile can be used for this purpose:

• init-*-chrootenv: Initializes most dynamic parts of the chroot environment inside a directory with write permissions (i.e. /run/chrootenv/<name>).
  
  First, it creates symlinks to the static folders of the /usr profile and the /usr symlink itself. Then the script ensures that the Name Service Switch (NSS) facilities work by symlinking /etc/{passwd,group,shadow,hosts,resolv.conf,nsswitch.conf) of the host system into the chroot environment. Then some other files from the host are symlinked, such as PAM configuration files and font configuration stuff. Finally, a few empty state folders are created, such as: /var, /run, /tmp.
  
  The initialization script has to be executed only once.
• mount-*-chrootenv: Bind mounts a number of directories of the host system into the chroot folder, so that files from the host system can be used while we have entered the chroot environment.
  
  Currently, we bind mount the Nix store (/nix/store), directories used by the kernel: /dev, /dev/pts, /dev/shm, /proc, /sys, users' home directories: /home, a few state directories: /var, /run, and the host's /etc directory containing shared configuration files, such as the ones related to NSS and PAM.
  
  This script has to be executed every time we have booted into NixOS.
• load-*-chrootenv: Enters the chroot environment and launches a shell. The shell inherits a few environment variables from the host environment, such as DISPLAY allowing someone to run GUI applications and TERM to make certain fancy console applications to work properly.
• umount-*-chrootenv: Unmounts the bind mounts.
• destroy-*-chrootenv: Removes the writable chrootenv directory.

tl;dr - Getting Steam to work under NixOS

If you're still awake and still interested in reading this blog post, I will reveal why I have done all this work, which is probably much more exciting to read. In fact, this function's primary use case is to get Steam working in NixOS. Steam is a very special piece of software compared to other binary-only packages, because:

• Steam is also a deployment tool (like Nix) capable of automatically downloading, installing and upgrading software packages. In Steam's case, these software packages are mostly games.
• Software packages installed by Steam have the same problems as the deployment of "ordinary" binary packages in NixOS -- they don't work out of the box, because they cannot find the dynamic linker and their dependencies, such as shared libraries.
• Of course, we can patch the executables installed by Steam, but Steam wants control over them and does not like patched versions of software. Furthermore, patching the packages makes it impossible for Steam to update them.
• It is also a bit impractical to patch the executables every time they are updated, because I have seen that the games I have are updated quite frequently.

By using the buildFHSChrootEnv {} Nix function getting Steam (including the games installed by it) to run in NixOS is a piece of cake. First I had to package Steam itself, which was quite easy:

{stdenv, fetchurl, dpkg}:

stdenv.mkDerivation {
  name = "steam-1.0.0.42";
  src = fetchurl {
    url = http://repo.steampowered.com/steam/archive/precise/steam-launcher_1.0.0.42_all.deb;
    sha256 = "1jyvk0h1z78sdpvl4hs1kdvr6z2kwamf09vjgjx1f6j04kgqrfbw";
  };
  buildInputs = [ dpkg ];
  unpackPhase = "true";
  installPhase = ''
    mkdir -p $out
    dpkg -x $src $out
    cp -av $out/usr/* $out
    rm -Rf $out/usr
  '';
}

I just had to unpack the Steam Debian package and move its contents into the Nix store. I intentionally did not patch or wrap anything.

Then I had to create a Nix expression that composes a FHS-compatible chroot environment with Steam's dependencies and runtime settings (such as OpenGL driver settings):

{ buildFHSChrootEnv, steam
, xterm, libX11, zenity, python, mesa, xdg_utils, dbus_tools, alsaLib
}:

buildFHSChrootEnv {
  name = "steam";
  pkgs = [ steam xterm libX11 zenity python mesa xdg_utils dbus_tools alsaLib ];
  profile = ''
    export LD_LIBRARY_PATH=/run/opengl-driver/lib:/run/opengl-driver-32/lib:/lib
    export FONTCONFIG_FILE=/etc/fonts/fonts.conf
  '';
}

After becoming the super user, I can install the above Nix expression providing the /usr environment profile and configuration scripts, by running:

$ su
# nix-env -f '<nixpkgs>' -iA steamChrootEnv

Now the chroot environment configuration scripts have been added to the system-wide Nix profile. I can initialize the chroot environment folder by running:

# init-steam-chrootenv

I can do the bind mounts by running:

# mount-steam-chrootenv

And then enter the chroot environment by running:

# load-steam-chrootenv

Now I have entered the chroot environment in which I can run the unpatched Steam executable. However, I should not run Steam as a root user. Therefore, I have to become "myself" again:

# su -s /bin/bash sander

And now I can finally run Steam, install my favourite games and play them:

$ steam

As you may see in the screenshot above, it works quite well on my machine. Moreover, to give you some proof that I'm not lying, I have also opened a terminal in the background showing you NixOS' root folder that does not contain most of the standard FHS directories.

So far, I have tried Half Life (and extensions), Half Life 2 (and extensions), Portal and Counter Strike and they all seem to work without any trouble. Isn't that AWESOME??? :P

Finally, if you need to discard the chroot environment (which should not be necessary IMHO :P), you could run:

# umount-steam-chrootenv
# destroy-steam-chrootenv

Discussion

In this blog post, I have described the buildFHSChrootEnv {} Nix function that composes FHS-compatible chroot environments in NixOS, allowing me to run Steam and other third party binary packages without patching them.

Moreover, we still have control over the dependencies in the chroot environment, because apart from a number of common base packages, these chroot environments are composed from Nix expressions that provide exactly the dependencies that are specified. Undeclared dependencies cannot influence a binary package running in the chroot environment.

However, this approach also has a big drawback. In order to chroot, we need super-user privileges making it hard to properly support unprivileged user installations of such packages. Most "ordinary" Nix packages can be deployed by unprivileged users without any trouble. Therefore, I would not recommend this approach over patching executables with PatchELF, unless there is no other way, such as in Steam's case.

Acknowledgements

This is not the first attempt to get Steam packaged in Nixpkgs. In my attempt, I have tried to use the chroot approach which turns out to work. I'd like to thank Evgeny Egorochkin, Aszlig Neusepoff and Carles Pagès for their earlier efforts.

Availability

The buildFHSChrootEnv {} as well as the Steam components are part of Nixpkgs.

A while ago, I have written a blog post about my IFF file format experiments in which I have developed a collection of libraries and tools capable of parsing and displaying two IFF file format applications. Some time before that, I have developed a Nix function capable of building software for AmigaOS allowing me to easily backport these experimental software packages to AmigaOS.

The development of these experimental packages were dormant for quite some time, but recently I was able to find some time to do some improvements. This time I have updated the Amiga video emulation library to support a few new features and to more accurately emulate an Amiga display.

Differences between Amiga and PC displays

So why is emulation of Amiga video modes necessary? Emulation of Amiga graphics hardware is required because pixels are encoded differently than "modern" PC hardware, resolutions have different semantics, and Amiga hardware has a few special screen modes to "squeeze" more colors out of the amount of available color registers.

Pixels

Encoding pixels on modern hardware is not so complicated these days. Modern hardware allows us to address each individual pixel's color value using 4 bytes per pixel in which one byte represents the red color intensity, one byte the blue color intensity and one byte the green color intensity. The fourth byte is often unused, or serves as the alpha component setting the transparency of a pixel in multi-layered images.

However, older PC hardware, such as those used during the DOS era, were often not powerful enough to address every pixel's color value individually, mainly due to memory constraints. Many classic DOS games used a 320x200 video resolution with a color palette consisting of 256 color values. Each pixel was encoded as a byte referring to an index value of a palette. This encoding is also known as chunky graphics.

System resources in the Amiga era were even more scarce, especially during its launch in 1985. The original graphics chipset was only capable of storing 32 color values in a palette, although there were a few special screen modes capable of squeezing more colors out of the 32 configurable ones. It also used a different way of encoding pixels, probably to make storage of graphics surfaces as memory efficient as possible.

In Amiga chipsets pixels are encoded as bitplanes rather than bytes. When using bitplane encoding, an image is stored multiple times in memory. In every image occurence, a bit represents a pixel. In the first occurence, a bit is the least significant bit of an index a value. In the last occurence a bit is the most significant bit of an index value. By adding all the bits together we can determine the index value of the palette to determine a pixel's color value. For example, if we would encode an image using 16 colors, then we need 4 bitplane surfaces.

To be able to display an Amiga image on a PC we have to convert the bitplane format to either chunky graphics or RGB graphics. Likewise, to be able to display a PC image on an image we have to convert the pixel surface to bitplane format.

Palette

As explained earlier, due to memory constraints, Amiga graphics use a palette of 32 configurable color values (or 256 colors when using the newer AGA chipset is used). Each pixel refers to an index of a palette instead of a color value of its own.

The original chipset color values are stored in 16-bit color registers in which every color component consists of 4 bits (4 bytes are unused). The newer AGA chipset as well as VGA graphics use 8 bits per color component.


Furthermore, the Amiga video chips have special screen modes to squeeze more colors out of the amount of color registers available. The Extra Half Brite (EHB) screen mode is capable of displaying 64 colors out of a predefined 32, in which the last 32 colors values have half of the intensity of the first 32 colors. The above screenshot is an example picture included with Deluxe Paint V using the EHB screenmode. A closer look at the floor in the picture look may reveal to you that an EHB palette is used.

The Hold-and-Modify (HAM) screen mode is used to modify a color component of the previous adjacent pixel or to provide a new color from the given palette. This screen mode makes it possible to use all possible color values (4096) in one screen with some loss of image quality.


The above screenshot is an example image included with Deluxe Paint V using HAM screenmode to utilise many more colors than the amount available color registers. A closer look at the image very may reveal that it has some quality loss because of the limitations of HAM compression.

To be able to properly display an image on a PC we need to convert 12-bit color values to 32-bit color values. Moreover, we also have to calculate each pixel's color value when HAM screen modes are used and convert the image to true color graphics, since a picture using HAM mode may use more than 256 colors.

Resolutions

Another difference between Amiga displays and "modern" displays are the display resolutions. On PCs, resolutions refer to the amount of pixels per scanline and the amount of scanlines per screen.

On the Amiga, resolutions only refer to the amount of pixels per scanline and this amount is often fixed. For example most displays support 320 lowres pixels per scanline, although this value can be slightly increased by utilising the overscan region. A high resolution screen has twice the amount of pixels per scanline compared to a low resolution screen. A super hires screen has double the amount of pixels per scanline compared to a high resolution screen. Moreover, a low resolution pixel is twice a wide as a high resolution pixel and so on.

Vertically, there are only two options. In principle, there are a fixed amount of scanlines on a display. Typically, NTSC displays support 200 scanlines and PAL displays support 256 scanlines. This amount can be slightly increased by utilising the overscan region of a display.

The amount of scanlines can be doubled, using a so-called interlace screen mode. However, interlace screen modes have a big drawback -- they draw the odd scanlines in one frame and the even ones in another. On displays with a short after glow, flickering may be observed.

Because of these differences, we may observe odd results in some cases when we convert an Amiga pixel surface to a chunky/RGB pixel surface. For example, a non-interlaced high resolution image looks twice as wide on a PC display than on an Amiga display, as can be seen in the left picture above, which contains a pixel-by-pixel conversion of a the Workbench screenshot.

To give it the same look as an Amiga display, we must correct its aspect ratio by doubling the amount of scanlines on the PC display, which is done in the right screenshot. People who have used the Amiga Workbench will immediately notice that this image is much closer to what can be seen on a real Amiga.

A library for performing conversions of Amiga display surfaces

While writing my previous blog post, I have developed a library (libamivideo) capable of automatically converting Amiga bitplane surfaces to chunky/RGB pixel surface and vice versa. In the latest release, I have added the following features and improvements:

• Support for super hires video modes
• Correct aspect ratio support
• A better defined API

The library can be obtained from my GitHub page. I have also updated the SDL-based ILBM viewer (SDL_ILBM) as well as the native AmigaOS ILBM viewer (amiilbm) to use the new library implementation.

A while ago, I wrote a blog post about my asynchronous programming experiences/frustrations with JavaScript. The main message of this blog post is that if we want to run stuff concurrently, we have to implement cooperative multitasking by generating events (such as timeouts or ticks) and then stop the execution so that the events get processed that invoke callbacks to resume execution.

As a consequence of using callbacks, it has become much harder to structure code properly. Without any proper software abstractions, we may end up writing pyramid code that is hard to write, read, adapt and maintain. One of the solutions to alleviate such problems is by using software abstractions implemented by libraries such as async.

Recently, I've been experimenting with several JavaScript libraries/frameworks (such as AngularJS, Selenium JavaScript webdriver, and jQuery) and I went to a Titanium SDK (a cross-platform JavaScript framework for developing apps) developer meetup. I have noticed that yet another software abstraction has become quite popular in these worlds.

Promises

The software abstraction I am referring to is called "promises". To get some understanding on what they are supposed to mean, I will cite the Promises/A proposal, that is generally regarded as its official specification:

A promise represents the eventual value returned from the single completion of an operation.

To me this description looks quite abstract. Fortunately, the proposal also outlines a few practical bits. It turns out that promises are supposed to be encoded as objects having the following characteristics:

• A promise can be in one of the following three states: unfulfilled, fulfilled, and failed. A promise in an unfulfilled state can either transition to a fullfilled state or failed state. The opposite transitions are not possible -- once a promise is fulfilled or failed it should remain in that state.
• A promise object has a then member referring to a function with the following signature:

  
  .then(onFullfilled, onError, onProgress);
  

• The then function parameters are all optional and refer to functions with the following purposes:
  
  
  • onFullfilled: A function that gets invoked once the promise reaches its fulfilled state.
  • onError: A function that gets invoked once the promise reaches its failed state.
  • onProgress: A function that may get invoked while the promise is still unfulfilled. Promises are actually not required to support this function at all.
  
  If any of these parameters is not a function, then they are ignored.
  
• Each then function is supposed to return another promise (which also have a then property), allowing users to "chain" multiple then function invocations together. This is how promises allow someone to structure code more properly and prevent people from writing pyramid code, because chaining can be done at the same indentation level.

Semantic issues

Although some of the promise characteristics listed above may look useful, it has turned out that the Promises/A proposal has a number of weaknesses as well.

First of all, it is just a proposal containing some random notes and not an official specification. Furthermore, it has been written in an informal way and it is incomplete and underspecified. For example, it leaves a few semantic issues open, such as what should happen if a promise enters a failed state, when a particular then function invocation in a chain does not implement an onError callback? As a consequence, this has lead to several implementations having different and incompatible behaviour.

To address some of these issues, a second specification has been developed called the Promises/A+ specification. This specification is not intended to completely replace the Promises/A proposal, but instead it is concentrated mostly on the semantics of the then function (not including the onProgress parameter). Moreover, this specification is also written in a much more formal and concise way in my opinion.

For example, the Promises/A+ specification adds the following details:

• It specifies the ability to invoke a promise's then function multiple times.
• Propagation of results and errors in a then function invocation chain, if any of their callback function parameters are not set.

When using Promises/A+ compliant promise objects, error propagation works in a similar way compared to a synchronous try { } catch { } block allowing one to execute a group of instructions and use a separate block at the end to catch any exceptions that may be thrown while executing these.

An example: Using the Selenium webdriver

To demonstrate how promises can be used, I have developed an example case with the JavaScript Selenium webdriver, a web browser automation framework used frequently for testing web applications. This package uses promises to encode query operations.

Our testcase works on a webpage that contains a form allowing someone to enter its first and last name. By default, they contain my first and last name:


The corresponding HTML code looks as follows:

<!DOCTYPE html>

<html>
<head><title>Selenium test</title></head>

<body>
<h1>Selenium test</h1>

<p>Please enter your names:</p>

<form action="test.html" method="post">
<p>
<label>First name</label>
<input type="text" name="firstname" value="Sander">
</p>
<p>
<label>Last name</label>
<input type="text" name="lastname" value="van der Burg">
</p>
<button type="submit" id="submit">Submit</button>
</form>
</body>
</html>

To automatically retrieve the values of both input fields and to click on the submit button, I have written the following JavaScript code:

var webdriver = require('selenium-webdriver');
var remote = require('selenium-webdriver/remote');
var webdrvr = require('webdrvr');

/* Create Selenium server */
var server = new remote.SeleniumServer(webdrvr.selenium.path, {
    args: webdrvr.args
});

/* Start the server */
server.start();

/* Set up a Chrome driver */
var driver = new webdriver.Builder()
    .usingServer(server.address())
    .withCapabilities(webdriver.Capabilities.chrome())
    .build();

/* Execute some tests */

driver.get('file://'+process.cwd()+'/test.html')
.then(function() {
    return driver.findElement(webdriver.By.name('firstname'));
})
.then(function(firstNameInput) {
    return firstNameInput.getAttribute('value');
})
.then(function(firstName) {
    console.log("First name: "+firstName);
    return driver.findElement(webdriver.By.name('lastname'));
})
.then(function(lastNameInput) {
    return lastNameInput.getAttribute('value');
})
.then(function(lastName) {
    console.log("Last name: "+lastName);
    return driver.findElement(webdriver.By.id('submit'));
})
.then(function(submitButton) {
    return submitButton.click();
}, function(err) {
    console.log("Some error occured: "+err);
});

/* Stop the server */
server.stop();

As may be observed from the code fragment above, besides setting up a test driver that uses Google Chrome, all the test operations are encoded as a chain of then function invocations each returning a promise:

• First, we instruct the test driver to get our example HTML page containing the form by creating a promise.
• After the page has been opened, we create and return a promise that queries the HTML input element that is used for filling in a first name.
• Then we retrieve the value of the first name input field, by creating and returning a promise that fetches the value attribute of the HTML input element.
• We repeat the previous two steps in a similar way to fetch the last name as well.
• Finally, we create and return promises that queries the submit button and clicks on it.
• In the last then function invocation, we define a onError callback that catches any errors that might occur when executing any of the previous steps.

As can be observed, we do not have to write any nested callback blocks that result in horrible pyramid code.

Implementing promises

The Selenium example shows us how asynchronous test code can be better structured using promises. The next thing I was thinking about is how can I use promises to fix structuring issues in my own code?

It turns out that there are many ways to do this. In fact, the Promise/A and Promise/A+ specifications are just merely interface specifications and leave it up to developers to properly implement them. For example, it can be done completely manually, but there are also several libraries available providing abstractions for that, such as Q, RSVP, and when.

As an experiment, I have restructured the following pyramid example code fragment from my previous blog post (that I originally used to restructure with async) into a better structured code fragment with promises:

var fs = require('fs');
var path = require('path');

fs.mkdir("out", 0755, function(err) {
    if(err) throw err;

    fs.mkdir(path.join("out, "test"), 0755, function(err) {
        if (err) throw err;        
        var filename = path.join("out", "test", "hello.txt");

        fs.writeFile(filename, "Hello world!", function(err) {
            if(err) throw err;

            fs.readFile(filename, function(err, data) {
                if(err) throw err;

                if(data == "Hello world!")
                    process.stderr.write("File is correct!\n");
                else
                    process.stderr.write("File is incorrect!\n");
            });
        });
    });
});

I have picked the RSVP library to implement promises, since it claims to be lightweight and fully Promises/A+ compliant. The restructured code looks as follows:

var fs = require('fs');
var path = require('path');
var rsvp = require('rsvp');

/* Promise object definitions */

var mkdir = function(dirname) {
    return new rsvp.Promise(function(resolve, reject) {
        fs.mkdir(dirname, 0755, function(err) {
            if(err) reject(err);
            else resolve();
        });
    });
};

var writeHelloTxt = function(filename) {
    return new rsvp.Promise(function(resolve, reject) {
        fs.writeFile(filename, "Hello world!", function(err) {
            if(err) reject(err);
            else resolve();
        });
    });
};

var readHelloTxt = function(filename) {
    return new rsvp.Promise(function(resolve, reject) {
        fs.readFile(filename, function(err, data) {
            if(err) reject(err);
            else resolve(data);
        });
    });
};

/* Promise execution chain */

var filename = path.join("out", "test", "hello.txt");

mkdir(path.join("out"))
.then(function() {
    return mkdir(path.join("out", "test"));
})
.then(function() {
    return writeHelloTxt(filename);
})
.then(function() {
    return readHelloTxt(filename);
})
.then(function(data) {
    if(data == "Hello world!")
        process.stderr.write("File is correct!\n");
    else
        process.stderr.write("File is incorrect!\n");
}, function(err) {
    console.log("An error occured: "+err);
});

The revised code fragment above is structured as follows:

• The overall structure consists of two parts: First we define a number of promises and then we execute them by composing a then function invocation chain.
• I have encapsulated most of the operations from the previous code fragment into reusable promises: one makes a directory, one writes a text file, one reads the text file.
• A promise is created by instantiating the rsvp.Promise prototype. The constructor function always takes the same function as a parameter taking resolve and reject as parameters. resolve is a callback that can be used to notify a caller that it has reached its fulfilled state. The reject parameter notifies a caller that it has reached its failed state. Both of these functions can take a value as parameter that can be passed to a then function invocation.
• To be able to parametrize these promises, I have encapsulated them in functions
• In the remainder of the code, I instantiate the promises and execute them in a then function invocation chain. As can be observed, we have no pyramid code.
• However, I observed that our refactored code has a big drawback as well -- it has grown significantly in size which can be considered a deterioration compared to the pyramid code fragment.

Discussion

So far, I have given an explanation of what promises are supposed to mean, how they can be used and how they can be created. We have seen that code structuring issues are "solved" by the fact that then function invocations can be chained.

In my previous blog post, I have explored the async library as a means to solve the same kind of issues, which make me wonder which approach is best? To me, the approach of chaining then function invocations looks quite similar to async's waterfall pattern. However, I find the promise approach slightly better, because it uses return values instead of callbacks preventing us to accidentally reach limbo state if we forget to call them.

However, I think async's waterfall pattern has a lower usage barrier because we do not have to capture all the steps we want into promise objects resulting in less overhead of code.

I think that using promises to structure code works best for things that can be encoded as reusable objects, such as those used to implement a system having a Model-View-Controller (MVC) pattern, that e.g. pass reusable objects from the model to the controller and view. In situations in which we just have to execute a collection of ad-hoc operations (such as those in my example) it simply produces a lot of extra overhead.

Finally, async's waterfall pattern as well as chaining then function invocations work well for things that consist of a fixed number of steps that have to be executed sequentially. However, it may also be desirable to execute steps in parallel, or to use asynchronous for and while loops. The promises specifications do not provide any solutions to cope with these kind of code structuring issues.

Some other thoughts

After doing these experiments and writing this blog post, I still have some remaining thoughts:

• I have noticed that promises are considered the "the next great paradigm" by some people. This actually makes me wonder what the "previous great paradigm" was. Moreover, I think promises are only useful in certain cases, but definitely not for all JavaScript's asynchronous programming problems.
• Another interesting observation I did is not related to JavaScript or asynchronous programming in general. I have observed that writing specifications, which purpose is to specify what kind of requirements a solution should fulfill, is a skill that is not present with quite a few people doing software engineering. It's not really surprising to me that the Promise/A proposal leads to so much issues. Fortunately, the Promises/A+ authors demonstrate how to properly write a specification (plus that they are aware of the fact that specifications always have limitations and issues).
• I'm still surprised how many solutions are being developed for a problem that is non-existent in other programming environments. Actually, I consider these software abstractions dealing with asynchronous programming "workarounds", not solutions. Because none of them is ideal there are many of them (the joyent Node wiki currently lists 79 libraries and I know there are more of them), I expect many more to come.
• Personally, I consider the best solution a runtime environment and/or programming language that has facilities to cope with concurrency. It is really surprising to me that nobody is working on getting that done. UPDATE: There is a bit of light at the end of the tunnel according to Zef's blog titled: "Callback-Free Harmonious Node.js" post that I just read.

I have noticed that while doing development work, many outsiders experience the way I work as quite odd and consider it to be inefficient.

The first reason is (probably) because I like the command-line for many tasks and I frequently use a unconventional text editor, which some people don't understand. Second, I also often use Nix (and related Nix utilities) during development.

In this blog post, I'd like to elaborate a bit about the second aspect and discuss why it is actually quite useful to do this.

Obtaining and installing the dependencies of a software project

In all software projects I have been involved with so far, the first thing I typically had to do is installing all its required dependencies. Examples of such dependencies are a compiler/interpreter + runtime environment for a specific programming language (such as GCC, the Java Development Kit, Perl, Python, Node.js etc.), a development IDE (such as Eclipse), and many library packages.

I have experienced that this step is often quite time consuming, since many dependencies have to be downloaded and installed. Moreover, I have frequently encountered situations in which a bunch of special settings have to be configured to make everything work right, which can be quite cumbersome and tedious.

Many developers just simply download and install everything on their machine in an ad-hoc manner and manually perform all the steps they have to do.

Automating the deployment of dependencies of a software project

What I often do instead is to immediately use Nix to automate the process of installing all software project dependencies and configuring all their settings.

To fully automate deployment with Nix, all dependencies have to be provided as Nix packages. First, I investigate if everything I need is in the Nix packages collection. Fortunately, the Nix package collection provides packages for many kinds of programming languages and associated libraries. If anything is missing, I package it myself. Although some things are very obscure to package (such as the Android SDK), most things can be packaged in a straight forward manner.

After packaging all the missing dependencies, I must install them and generate an environment in which all the dependencies can be found. For example, in order to let Java projects find their library dependencies, we must set the CLASSPATH environment variable to refer to directories or JAR files containing our required compiled classes, PERL5LIB to let Perl find its Perl modules, NODE_PATH to let Node.js find its CommonJS modules etc.

The Nix packages collection has a nice utility function called myEnvFun that can be used to automatically do this. This function can be used to compose development environments that automatically set nearly all of its required development settings. The following example Nix expression (disnixenv.nix) can be used to deploy a development environment for Disnix:

with import <nixpkgs> {};

myEnvFun {
  name = "disnix";
  buildInputs = [
    pkgconfig dbus dbus_glib libxml2 libxslt getopt
    autoconf automake libtool dysnomia
  ];
}

The development environment can be deployed by installing it with the Nix package manager:

$ nix-env -f disnixenv.nix -i env-disnix

The above command-line invocation deploys all Disnix's dependencies and generates a shell script (named: load-env-disnix) that launches a shell session in an environment in which all build settings are configured. We can enter this development environment by running:

$ load-env-disnix
env-disnix loaded

disnix:[sander@nixos:~]$

As can be seen in the code fragment above, we have entered an environment in which all its development dependencies are present and configured. For example, the following command-line instruction should work, since libxml2 is declared as a development dependency:

$ xmllint --version
xmllint: using libxml version 20901

We can also unpack the Disnix source tarball and run the following build instructions, which should work without any trouble:

$ ./bootstrap
$ ./configure
$ make

Besides deploying a development environment, we can also discard it if we don't need it anymore by running:

$ nix-env -e env-disnix

Automating common deployment tasks

After deploying a development environment, I'm usually doing all kinds of development tasks until the project reaches a more stable state. Once this is the case, I immediately automate most of its deployment operations in a Nix expression (that is often called: release.nix).

This Nix expression is typically an attribute set following Hydra's (the Nix-based continuous integration server) convention in which each attribute represents a deployment job. Jobs that I typically implement are:

• Source package. This is a job that helps me to easily redistribute the source code of a project to others. For GNU Autotools based projects this typically involves the: make dist command-line instruction. Besides GNU Autotools, I also generate source packages for other kinds of projects. For example, in one of my Node.js projects (such a NiJS) I produce a TGZ file that can be deployed with the NPM package manager.
• Binary package jobs that actually build the software project for every target system that I want to support, such as i686-linux, x86_64-linux, and x86_64-darwin.
• A job that automatically compiles the program manual from e.g. Docbook, if this applicable.
• A program documentation catalog job. Many programming allow developers to write code-level documentation in comments from which a documentation catalog can be generated, e.g. through javadoc, doxygen or JSDuck. I also create a job that takes care of doing that.
• Unit tests. If my project has a unit test suite, I also create a job executing those, since it is required to deploy most of its development dependencies to run the test suite as well.
• System integration tests. If I have any system integration tests that can run be on Linux, I try to implement a job using the NixOS test driver to run those. The NixOS test driver also automatically deploys all the required environmental dependencies in a VM, such as a DBMS, web server etc. in such a way that you can run them as a unprivileged user without affecting the host system's configuration.

After writing a release Nix expression, I can use the Nix package manager to perform all the deployment tasks for my software project. For example, I can run the following job that comes with Disnix to build a source tarball:

$ nix-build release.nix -A tarball

Another advantage is that I can use the same expression in combination with Hydra to so that I can continuously build and test it.

Spawning development environments from Nix jobs

Apart from being able to deploy a software project and its dependencies, another nice advantage of having its deployment automated through Nix is that I can use the same Nix jobs to reproduce its build environment. By replacing nix-build with nix-shell all its dependencies are deployed, but instead of building the package itself, a shell session is started in which all its dependencies are configured:

$ nix-shell release.nix -A tarball

The above command-line invocation produces a similar build environment as shown earlier with myEnvFun by just using a Nix expression for an existing job.

Discussion

So far I have explained what I typically do with the Nix package manager during development. So why is this actually useful?

• Besides deploying systems into production environments, setting up a development environment is a similar problem with similar complexities. For me it is logical that I solve these kind of problems the same way as I do in production environments.
• Packaging all software project dependencies in Nix ensures reliable and reproducible deployment in production environments as well, since Nix ensures dependency completeness and removes as many side effects as possible. In Nix, we have strong guarantees that a system deployed through Nix in production behaves exactly the same in development and vice-versa.
• We can also share development environments, which is useful for a variety of reasons. For example, we can reproduce the same development environment on a second machine, or give it to another developer to prevent him spending a lot of time in setting up his development environment.
• Supporting continuous integration and testing with Hydra comes (almost) for free.
• With relatively little effort you can also enable distributed deployment of your software project with Disnix and/or NixOps.

Although I have encountered people telling me that it's consuming too much time, you also get a lot for it in return. Moreover, in most common scenarios I don't think the effort of automating the deployment of a development environment is that big if you have experience with it.

As a concluding remark, I know that things like Agile software development and continuous delivery of new versions is something that (nearly) everybody wants these days. To implement continuous delivery/deployment, the first step is to automate the deployment of a software project from the very beginning. That is exactly what I'm doing by executing the steps described in this blog post.

People may also want to read an old blog post that contains recommendations to improve software deployment processes.