New features

Visualization tool

Dysnomia

type=mysql-database
mysqlUsername=root
mysqlPassword=verysecret

create table author
( AUTHOR_ID  INTEGER       NOT NULL,
  FirstName  VARCHAR(255)  NOT NULL,
  LastName   VARCHAR(255)  NOT NULL,
  PRIMARY KEY(AUTHOR_ID)
);

create table books
( ISBN       VARCHAR(255)  NOT NULL,
  Title      VARCHAR(255)  NOT NULL,
  AUTHOR_ID  VARCHAR(255)  NOT NULL,
  PRIMARY KEY(ISBN),
  FOREIGN KEY(AUTHOR_ID) references author(AUTHOR_ID)
    on update cascade on delete cascade
);

$ dysnomia --operation activate --component ~/test-database \
  --container ~/mysql-production

Concurrent closure transfers

$ disnix-env -s services.nix -i infrastructure.nix \
  -d distribution.nix --max-concurrent-transfers 4

Concurrent service activation and deactivation

{
  test1 = {
    hostname = "test1";
    numOfCores = 2;
  };
}

Multi connection protocol support

{
  test1 = {
    hostname = "test1";
    targetProperty = "hostname";
    clientInterface = "disnix-ssh-client";
  };

  test2 = {
    hostname = "test2";
    targetEPR = http://test2:8080/DisnixWebService/services/DisnixWebService;
    targetProperty = "targetEPR";
    clientInterface = "disnix-soap-client";
  };
}

• To connect to machine: test1, the hostname attribute contains the address and the disnix-ssh-client tool should be invoked to connect it.
• To connect to machine: test2, the targetEPR attribute contains the address and the disnix-soap-client tool should be invoked to connect it.

NixOps integration

$ nixops create ./network.nix ./network-ec2.nix -d ec2
$ nixops deploy -d ec2

$ export NIXOPS_DEPLOYMENT=ec2
$ disnixos-env -s services.nix -n network.nix -d distribution.nix \
    --use-nixops

Omitted features

Dynamic Disnix

Snapshot/restore features of Dysnomia

Conclusion

What is Snappy Ubuntu?

The snappy approach is faster, more reliable, and lets us provide stronger security guarantees for apps and users -- that's why we call them "snappy" applications.

Snappy apps and Ubuntu Core itself can be upgraded atomically and rolled back if needed -- a bulletproof approach that is perfect for deployments where predictability and reliability are paramount. It's called "transactional" or "image-based" systems management, and we’re delighted to make it available on every Ubuntu certified cloud.

Package organization

/app/<name>/<version>[.<developer>]

$ ls -l /apps
drwxr-xr-x 2 root ubuntu 4096 Apr 25 20:38 bin
drwxr-xr-x 3 root root   4096 Apr 25 15:56 docker
drwxr-xr-x 3 root root   4096 Apr 25 20:34 go-example-webserver.canonical
drwxr-xr-x 3 root root   4096 Apr 25 20:31 hello-world.canonical
drwxr-xr-x 3 root root   4096 Apr 25 20:38 webcam-demo.canonical
drwxr-xr-x 3 root ubuntu 4096 Apr 23 05:24 webdm.sideload

$ cd /apps/webcam-demo.canonical/1.0.1
$ find . -type f ! -iname ".*"
./bin/x86_64-linux-gnu/golang-static-http
./bin/x86_64-linux-gnu/fswebcam
./bin/golang-static-http
./bin/runner
./bin/arm-linux-gnueabihf/golang-static-http
./bin/arm-linux-gnueabihf/fswebcam
./bin/fswebcam
./bin/webcam-webui
./meta/readme.md
./meta/package.yaml
./lib/x86_64-linux-gnu/libz.so.1
./lib/x86_64-linux-gnu/libc.so.6
./lib/x86_64-linux-gnu/libX11.so.6
./lib/x86_64-linux-gnu/libpng12.so.0
./lib/x86_64-linux-gnu/libvpx.so.1
...
./lib/arm-linux-gnueabihf/libz.so.1
./lib/arm-linux-gnueabihf/libc.so.6
./lib/arm-linux-gnueabihf/libX11.so.6
./lib/arm-linux-gnueabihf/libpng12.so.0
./lib/arm-linux-gnueabihf/libvpx.so.1
...

$ ls -l /apps/webcam-demo.canonical
drwxr-xr-x 7 clickpkg clickpkg 4096 Apr 24 19:38 1.0.0
drwxr-xr-x 7 clickpkg clickpkg 4096 Apr 25 20:38 1.0.1
lrwxrwxrwx 1 root     root        5 Apr 25 20:38 current -> 1.0.1

# for debugging convience we also make the binary available as a command
binaries:
 - name: bin/webcam-webui

$ webcam-demo.webcam-webui

Dependency management

./lib/x86_64-linux-gnu/libz.so.1
./lib/x86_64-linux-gnu/libc.so.6
./lib/x86_64-linux-gnu/libX11.so.6
./lib/x86_64-linux-gnu/libpng12.so.0
./lib/x86_64-linux-gnu/libvpx.so.1
...
./lib/arm-linux-gnueabihf/libz.so.1
./lib/arm-linux-gnueabihf/libc.so.6
./lib/arm-linux-gnueabihf/libX11.so.6
./lib/arm-linux-gnueabihf/libpng12.so.0
./lib/arm-linux-gnueabihf/libvpx.so.1
...

frameworks: docker, foo, bar # list of frameworks required

System organization

• boot partition. This is a very tiny partition used for booting and should be big enough to contain a few kernels.
• system-a partition. This partition contains a minimal working base system. This partition is mounted read-only.
• system-b partition. An alternative partition containing a minimal working base system. This partition is mounted read-only as well.
• writable partition. A writable partition that stores everything else including the applications and frameworks.

Creating packages

$ snappy build .

Isolation

Overview

Discussion

Frameworks exist primarily to provide mediation of shared resources (eg, device files, sensors, cameras, etc)

References

Differences between Amiga and PC displays

Pixels

Palette

Resolutions

A library for performing conversions of Amiga display surfaces

• Support for super hires video modes
• Correct aspect ratio support
• A better defined API

Using the command-line

• It's a habit. This is obviously not a very strong argument, but I'm quite used to typing commands when executing system administration and development tasks, and they are quite common in a Linux/Unix world -- much of the documentation that you will find online explain how to do things on the command-line.
  
  Moreover, I have been using a command-line interface since the very first moment I was introduced to a computer. For example, my first computer's operating system shell was a BASIC programming language interpreter, which you even had to use to perform simple tasks, such as loading a program from disk/tape and running it.
  
  
• Speed. This argument may sound counter-intuitive to some people, but I can accomplish many tasks quite quickly on the command-line and probably even faster than using a graphical user interface.
  
  For example, when I need to do a file management task, such as removing all backup files (files having a ~ suffix), I can simply run the following shell command:
  
  

  $ rm *~
  

  
  If I would avoid the shell and use a graphical file manager (e.g. Dolphin, GNOME Files, Windows Explorer, Apple Finder), picking these files and moving them to the trashcan would certainly take much more time and effort.
  
  Moreover, shells (such as bash) have many more useful goodies to speed things up, such as TAB-completion, allowing someone to only partially type a command or filename and let the shell complete it by pressing the TAB-key.
  
  
• Convenient integration. Besides running a single command-line instruction to do a specific task, I also often combine various instructions to accomplish something more difficult. For example, the following chain of commands:
  
  

  $ wc -l $(find . -name \*.h -or -name \*.c) | head -n -1 | sort -n -r
  

  
  comes in handy when I want to generate an overview of lines of code per source file implemented in the C programming language and sort them in reverse order (that is the biggest source file first).
  
  As you may notice, I integrate four command-line instructions: the find instruction seeks for all relevant C source files in the current working directory, the wc -l command calculates the lines of code per file, the head command chops off the last line displaying the total values and the sort command does a numeric sort of the output in reverse order.
  
  Integration is done by pipes (using the | operator) which make the output of one process the input of another and command substitution (using the $(...) construct) that substitutes the invocation by its output.
  
  
• Automation. I often find myself repeating common sets of shell instructions to accomplish certain tasks. I can conveniently turn them into shell scripts to make my life easier.

Recommendations for developing command-line utilities

Interface conventions

• On Linux and most other UNIX-like systems (e.g. Mac OS X, FreeBSD) I follow GNU's convention for command line parameters.
  
  For each parameter, I always define a long option (e.g. --help) and for the most common parameters also a short option (e.g. -h). I only implement a purely short option interface if the target platform does not support long options, such as classic UNIX-like operating systems.
• On DOS/Windows, I follow the DOS convention. That is: command line options are single character only and prefixed by a slash, e.g. /h.

$ export DISNIX_PROFILE=testenv
$ manifest=$(disnix-manifest -s services.nix -i infrastructure.nix \
    -d distribution.nix)
$ disnix-distribute $manifest
$ disnix-activate $manifest
$ disnix-set $manifest

• If a tool takes one single file as input or produces one single output or a combination of both, I also allow it to read from the standard input or write to the standard output so that it can be used as a component in a pipe. In most cases supporting these special file descriptors is almost as straight forward as opening arbitrary files.
  
  For example, the ILBM image viewer's primary purpose is just to view an ILBM image file stored on disk. However, because I also allow it to read from the standard input I can also do something like this:
  
  

  
  $ iffjoin picture1.ILBM picture2.ILBM | ilbmviewer
  

  
  In the above example, I concatenate two ILBM files into a single IFF container file and invoke the viewer to view both of them without storing the immediate result on disk first. This can be useful for a variety of purposes.
  
  
• A tool that produces output writes to the standard output data that can be parsed/processed by another process in a pipeline. All other output, e.g. errors, debug messages, notifications go the the standard error.

Command-line option parsing

• We must know which parameters are command-line option flags (e.g. starting with -, -- or /) and which are non-option parameters.
• Some command-line options have a required argument, some take an optional argument and some take none.

• For UNIX programs implemented in C/C++: getopt_long(), getopt().
• For shell scripts: the getopt command-line utility
• Java command-line utilities: jargs
• JavaScript/Node.js: optparse.js

Validation and error reporting

Documentation

• A usage line describing briefly in what ways the command-line tool can be invoked and what their mandatory parameters are.
• A brief description explaining what the tool does.
• The command-line options. For each option, I document the following properties:
  
  
  • The short and long option identifiers.
  • Whether the option requires no parameter, an optional parameter, or a required parameter.
  • A brief description of the option
  • What the default value is, if appropriate
  
  An example of an option that I have documented for the disnix-distribute utility is:
  
  

  
  -m, --max-concurrent-transfers=NUM  Maximum amount of concurrent closure
                                      transfers. Defauls to: 2
  

  
  that specifies that the amount of concurrent transfers can be specified through the -m short or --max-concurrent-transfers long option, requires a numeric argument and defaults to 2 if the option is unspecified.
  
  
• An environment section describing which environment variables can be configured and their meanings.
• I only document the exit statuses if any of them has a special meaning. Zero in case of success and non-zero in case of a failure is too obvious.

$ help2man --output=disnix-distribute.1 --no-info --name \
'Distributes intra-dependency closures of services to target machines' \
  --libtool ./disnix-distribute

$ doclifter -x disnix-distribute.1

Modularization

• Interface conventions
• Command-line option parsing
• Validation and error reporting
• Documentation

Reuse

Discussion

echo "hello"
false | cat
echo "world"
false
echo "!!!"

$ bash script.bash
Hello
world
!!!

$ bash -e script.bash
Hello
world

$ bash -e -o pipefail script.bash
Hello

Executing state management activities

type=mysql-database
mysqlUsername=root
mysqlPassword=verysecret

create table author
( AUTHOR_ID  INTEGER       NOT NULL,
  FirstName  VARCHAR(255)  NOT NULL,
  LastName   VARCHAR(255)  NOT NULL,
  PRIMARY KEY(AUTHOR_ID)
);

create table books
( ISBN       VARCHAR(255)  NOT NULL,
  Title      VARCHAR(255)  NOT NULL,
  AUTHOR_ID  VARCHAR(255)  NOT NULL,
  PRIMARY KEY(ISBN),
  FOREIGN KEY(AUTHOR_ID) references author(AUTHOR_ID)
    on update cascade on delete cascade
);

$ dysnomia --operation activate \
  --component ~/testdb \
  --container ~/mysql-production

$ dysnomia --operation snapshot \
  --component ~/testdb \
  --container ~/mysql-production

$ dysnomia-snapshots --query-all --container mysql-database --component testdb
mysql-production/testdb/9b0c3562b57dafd00e480c6b3a67d29146179775b67dfff5aa7a138b2699b241
mysql-production/testdb/1df326254d596dd31d9d9db30ea178d05eb220ae51d093a2cbffeaa13f45b21c
mysql-production/testdb/330232eda02b77c3629a4623b498855c168986e0a214ec44f38e7e0447a3f7ef

$ dysnomia-snapshots --resolve mysql-database/testdb/330232eda02b77c3629a4623b498855c...
/var/state/dysnomia/snapshots/mysql-production/testdb/330232eda02b77c3629a4623b498855...

$ dysnomia --operation restore \
  --component ~/testdb \
  --container ~/mysql-production

$ dysnomia --operation deactivate \
  --component ~/testdb \
  --container ~/mysql-production

$ dysnomia --operation collect-garbage \
  --component ~/testdb \
  --container ~/mysql-production

$ dysnomia-snapshots --gc --keep 3

Managing state of service-oriented systems

staff = {
  name = "staff";
  pkg = customPkgs.staff;
  dependsOn = {};
  type = "mysql-database";
  deployState = true;
};

$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix

$ disnix-snapshot

$ disnix-restore

$ disnix-restore --no-upgrade

$ disnix-clean-snapshots --keep 3 infrastructure.nix

Discussion

• The prototype implementation stores snapshots in the /dysnomia folder (analogous to the Nix store that resides in /nix/store), which is a non-FHS compliant directory. Nix has a number of very good reasons to deviate from the FHS and requires packages to be addressed by their absolute paths across machines so that they can be uniformly accessed by a dynamic linker.
  
  However, such a level of strictness is not required for addressing snapshots. In the current implementation, snapshots are stored in /var/state/dysnomia which is FHS-compliant. Furthermore, snapshots are identified by their relative paths to the store folder. The snapshot store's location can be changed by setting the DYSNOMIA_STATEDIR environment variable, allowing someone to have multiple snapshot stores.
• In the prototype, the semantics of the deactivate operation also imply deleting the state of mutable component in a container. As this is a dangerous and destructive operation, the current implementation separates the actual delete operation into a garbage collect operation that must be invoked explicitly.
• In both the prototype and the current implementation, a Dysnomia plugin can choose its own naming convention to identify snapshots. In the prototype, the naming must reflect both the contents and the order in which the snapshots have been taken. As a general fallback, I proposed using timestamps.
  
  However, timestamps are unreliable in a distributed setting because the machines' clocks may not be in sync. In the current implementation, I use output hashes as a general fallback. As hashes cannot reflect the order in their names, Dysnomia provides a generations folder containing symlinks to snapshots which names reflect the order in which they have been taken.
  

• The incremental snapshot operation is unimplemented. Although this feature may sound attractive, I could only properly do this with Subversion repositories and MySQL databases with binary logging enabled.
• To upgrade a service-oriented system (that includes moving state) atomically, access to the system in the transition and data migration phases must be blocked/queued. However, when moving large data sets, this time window could be incredibly big.
  
  As an optimization, I proposed an improved upgrade process in which incremental snapshots are transferred inside the locking time window, while full snapshots are transferred before the locking starts. Although it may sound conceptually nice, it is difficult to properly apply it in practice. I may still integrate it some day, but currently I don't need it. :)

• Filesystem-level snapshotting is typically faster since files only need to be copied. However, its biggest drawback is that physical state may be inconsistent (because of unfinished write operations) and non-portable. Moreover, it may be difficult to manage individual chunks of state. NixOps, for example, supports partition-level state management of EBS volumes.
• Database replication engines can also typically capture and transfer state much more efficiently.

Availability

• We must write an infrastructure model reflecting all relevant properties of all target machines. Although writing such a configuration file for a new environment is doable, it is quite tedious and error prone to keep it up to date and in sync with their actual configurations -- whenever a machine's property or the network changes, the infrastructure model must be updated accordingly.
  
  (As a sidenote: when using the DisnixOS extension, a NixOS network model is used instead of an infrastructure model from which the machine's configurations can be automatically deployed making the consistency problem obsolete. However, the problem remains to persist if we need to deploy to a network of non-NixOS machines)
• We must manually specify the distribution of services to machines. This problem typically becomes complicated if services have specific technical requirements on the host that they need to run (e.g. operating system, CPU architecture, infrastructure components such as an application server).
  
  Moreover, a distribution could also be subject to non-functional requirements. For example, a service providing access to privacy-sensitive data should not be deployed to a machine that is publicly accessible from the internet.
  
  Because requirements may be complicated, it is typically costly to repeat the deployment planning process whenever the network configuration changes, especially if the process is not automated.

• A discovery service that captures the properties of the machines in the network from which an infrastructure model is generated.
• A framework allowing someone to automate deployment planning processes using a couple of algorithms described in the literature.

Microservices

is a unit of independent deployment

Deploying Microservices with Disnix

The port assignment problem

• Each service must listen on a port that is unique to the machine the service runs on. In some cases, it may also be desirable to assign a port that is unique to the entire network (instead of a single machine) so that it can be uniformly accessed regardless of its location.
• The assigned ports must be within a certain range so that (for example) they do not collide with system services.
• Once a port number has been assigned to a service, it must remain reserved until it gets undeployed.
  
  The alternative would be to reassign all port numbers to all services for each change in the network, but that can be quite costly in case of an upgrade. For example, if we upgrade a network running 100 microservices, all 100 of them may need to be deactivated and activated to make them listen on their newly assigned ports.

Dynamically configuring ports in Disnix models

{distribution, system, pkgs}:

let
  portsConfiguration = if builtins.pathExists ./ports.nix
    then import ./ports.nix else {};
  ...
in
rec {
  roomservice = rec {
    name = "roomservice";
    pkg = customPkgs.roomservicewrapper { inherit port; };
    dependsOn = {
      inherit rooms;
    };
    type = "process";
    portAssign = "private";
    port = portsConfiguration.ports.roomservice or 0;
  };

  ...

  stafftracker = rec {
    name = "stafftracker";
    pkg = customPkgs.stafftrackerwrapper { inherit port; };
    dependsOn = {
      inherit roomservice staffservice zipcodeservice;
    };
    type = "process";
    portAssign = "shared";
    port = portsConfiguration.ports.stafftracker or 0;
    baseURL = "/";
  };
}

$ dydisnix-port-assign -s services.nix -i infrastructure.nix \
    -d distribution.nix > ports.nix

{
  ports = {
    roomservice = 8001;
    ...
    zipcodeservice = 3003;
  };
  portConfiguration = {
    globalConfig = {
      lastPort = 3003;
      minPort = 3000;
      maxPort = 4000;
      servicesToPorts = {
        stafftracker = 3002;
      };
    };
    targetConfigs = {
      test2 = {
        lastPort = 8001;
        minPort = 8000;
        maxPort = 9000;
        servicesToPorts = {
          roomservice = 8001;
        };
      };
    };
  };
}

• The ports attribute contains the actual port numbers that have been assigned to each service. The services defined in the services model (shown earlier) refer to the port values defined here.
• The portConfiguration attribute contains port configuration settings for the network and each target machine. The globalConfig attribute defines a TCP port range with ports that must be unique to the network. Besides the port range it also stores the last assigned TCP port number and all global port reservations.
• The targetConfigs attribute contains port configuration settings and reservations for each target machine.

$ dydisnix-port-assign -s services.nix -i infrastructure.nix \
    -d distribution.nix -p ports.nix > ports2.nix

Conclusion

Availability

Obtaining and installing the dependencies of a software project

Automating the deployment of dependencies of a software project

with import <nixpkgs> {};

myEnvFun {
  name = "disnix";
  buildInputs = [
    pkgconfig dbus dbus_glib libxml2 libxslt getopt
    autoconf automake libtool dysnomia
  ];
}

$ nix-env -f disnixenv.nix -i env-disnix

$ load-env-disnix
env-disnix loaded

disnix:[sander@nixos:~]$

$ xmllint --version
xmllint: using libxml version 20901

$ ./bootstrap
$ ./configure
$ make

$ nix-env -e env-disnix

Automating common deployment tasks

• Source package. This is a job that helps me to easily redistribute the source code of a project to others. For GNU Autotools based projects this typically involves the: make dist command-line instruction. Besides GNU Autotools, I also generate source packages for other kinds of projects. For example, in one of my Node.js projects (such a NiJS) I produce a TGZ file that can be deployed with the NPM package manager.
• Binary package jobs that actually build the software project for every target system that I want to support, such as i686-linux, x86_64-linux, and x86_64-darwin.
• A job that automatically compiles the program manual from e.g. Docbook, if this applicable.
• A program documentation catalog job. Many programming languages allow developers to write code-level documentation in comments from which a documentation catalog can be generated, e.g. through javadoc, doxygen or JSDuck. I also create a job that takes care of doing that.
• Unit tests. If my project has a unit test suite, I also create a job executing those, since it is required to deploy most of its development dependencies to run the test suite as well.
• System integration tests. If I have any system integration tests that can run be on Linux, I try to implement a job using the NixOS test driver to run those. The NixOS test driver also automatically deploys all the required environmental dependencies in a VM, such as a DBMS, web server etc. in such a way that you can run them as a unprivileged user without affecting the host system's configuration.

$ nix-build release.nix -A tarball

Spawning development environments from Nix jobs

$ nix-shell release.nix -A tarball

Discussion

• Besides deploying systems into production environments, setting up a development environment is a similar problem with similar complexities. For me it is logical that I solve these kind of problems the same way as I do in production environments.
• Packaging all software project dependencies in Nix ensures reliable and reproducible deployment in production environments as well, since Nix ensures dependency completeness and removes as many side effects as possible. In Nix, we have strong guarantees that a system deployed through Nix in production behaves exactly the same in development and vice-versa.
• We can also share development environments, which is useful for a variety of reasons. For example, we can reproduce the same development environment on a second machine, or give it to another developer to prevent him spending a lot of time in setting up his development environment.
• Supporting continuous integration and testing with Hydra comes (almost) for free.
• With relatively little effort you can also enable distributed deployment of your software project with Disnix and/or NixOps.

Obtaining my PhD degree

Programming in JavaScript

Continuous integration and testing

Nix/NixOS development

Fun programming

Research

Blog posts

1. On Nix and GNU Guix. This is a critical blog post that also ended up first in last year's top 10. I think this blog posts will remain at the top position for the time being, since it attracted an insane amount of visitors.
2. An alternative explanation of the Nix package manager. My alternative explanation of Nix, which I wrote to clarify things. It was also second in last year's top 10.
3. Setting up a Hydra build cluster for continuous integration and testing (part 1). Apparently, Hydra and some general principles about continuous integration have attracted quite some vistors. However, the follow up blog posts I wrote about Hydra don't seem to be that interesting to outsiders.
4. Using Nix while doing development. I wrote this blog post 2 days ago, and it attracted quite some visitors. I have noticed that setting up development environments is an attractive feature for Nix users.
5. Second computer. This is an old blog post about my good ol' Amiga. It was also in all previous top 10s and I think it will remain like that for the time being. The Amiga rocks!
6. An evaluation and comparison of GoboLinux. Another blog article that remains popular from the beginning. It's still a pity that GoboLinux has not been updated and sticks to their 014.01 release, which dates from 2008.
7. Composing FHS-compatible chroot environments with Nix (or deploying Steam in NixOS). This is something I have developed to be able to run Steam in NixOS. It seems to have attracted quite some users, which does not come as a surprise. NixOS users want to play Half-Life!
8. Software deployment complexity. An old blog post about software deployment complexity in general. Still remains popular.
9. Deploying iOS applications with the Nix package manager. A blog post that I wrote last year describing how we can use the Nix package manager to build apps for the iPhone/iPad. For a long time the Android variant of this blog post was more popular, but recently this blog article surpassed it. I have no clue why.
10. Porting software to AmigaOS (unconventional style). People still seem to like one of my craziest experiments.

Conclusion

What is Titanium SDK?

Packaging the Titanium CLI

Packaging the Titanium SDK

• The SDK distribution contains a collection of Python scripts that execute build and debugging tasks. However, to be able to run them in NixOS, the shebangs must be changed so that the Python interpreter can be found:
  
  

  
  find . -name \*.py | while read i
  do
      sed -i -e "s|#!/usr/bin/env python|#!${python}/bin/python|" $i
  done
  

• The SDK contains a subdirectory (mobilesdk/3.2.1.v20140206170116) with a version number and timestamp in it. However, the timestamp is a bit inconvenient, because the Titanium CLI explicitly checks for SDK folders that correspond to a Titanium SDK version number in a Titanium project file (tiapp.xml). Therefore, I strip it out of the directory name to make my life easier:
  
  

  
  $ cd mobilesdk/*
  $ mv 3.2.1.v20140206170116 3.2.1.GA
  

• The Android builder script (mobilesdk/*/android/builder.py) packages certain files into an APK bundle (which is technically a ZIP file).
  
  However, the script throws an exception if it encounters files with timestamps below January 1, 1980, which are not supported by the ZIP file format. This is a problem, because Nix automatically resets timestamps of deployed packages to one second after January 1, 1970 (a.k.a. UNIX-time: 1) to make builds more deterministic. To remedy the issue, I had to modify several pieces of the builder script.
  
  What I basically did to fix this is searching for invocations to ZipFile.write() that adds a file from the filesystem to a zip archive, such as:
  
  

  
  apk_zip.write(os.path.join(lib_source_dir, 'libtiverify.so'), lib_dest_dir + 'libtiverify.so')
  

  
  I refactored such invocations into a code fragment using a file stream:
  
  

  
  info = zipfile.ZipInfo(lib_dest_dir + 'libtiverify.so')
  info.compress_type = zipfile.ZIP_DEFLATED
  info.create_system = 3
  tf = open(os.path.join(lib_source_dir, 'libtiverify.so'))
  apk_zip.writestr(info, f.read())
  tf.close()
  

  
  The above code fragment ignores the timestamp of the files to be packaged and uses the current time instead, thus fixing the issue with files that reside in the Nix store.
  
• There were two ELF executables (titanium_prep.{linux32,linux64}) in the distribution. To be able to run them under NixOS, I had to patch them so that the dynamic linker can be found:
  
  

  
  $ patchelf --set-interpreter ${stdenv.gcc.libc}/lib/ld-linux-x86-64.so.2 \
      titanium_prep.linux64
  

• The Android builder script (mobilesdk/*/android/builder.py) requires the sqlite3 python module and the Java Development Kit. Since dependencies do not reside in standard locations in Nix, I had to wrap the builder script to allow it to find them:
  
  

  
  mv builder.py .builder.py
  cat > builder.py <<EOF
  #!${python}/bin/python
  
  import os, sys
  
  os.environ['PYTHONPATH'] = '$(echo ${python.modules.sqlite3}/lib/python*/site-packages)'
  os.environ['JAVA_HOME'] = '${jdk}/lib/openjdk'
  
  os.execv('$(pwd)/.builder.py', sys.argv)
  EOF
  

  
  Although the Nixpkgs collection has a standard function (wrapProgram) to easily wrap executables, I could not use it, because this function turns any executable into a shell script. The Titanium CLI expects that this builder script is a Python script and will fail if there is a shell code around it.
  
• The iOS builder script (mobilesdk/osx/*/iphone/builder.py) invokes ditto to do a recursive copy of a directory hierarchy. However, this executable cannot be found in a Nix builder environment, since the PATH environment variable is set to only the dependencies that are specified. The following command fixes it:
  
  

  
  $ sed -i -e "s|ditto|/usr/bin/ditto|g" \
      $out/mobilesdk/osx/*/iphone/builder.py
  

• When building IPA files for iOS devices, the Titanium CLI invokes xcodebuild, that in turn invokes the Titanium CLI again. However, it does not seem to propagate all parameters properly, such as the path to the CLI's configuration file. The following modification allows me to set an environment variable called: NIX_TITANIUM_WORKAROUND providing additional parameters to work around it:
  
  

  
  $ sed -i -e "s|--xcode|--xcode '+process.env['NIX_TITANIUM_WORKAROUND']+'|" \
      $out/mobilesdk/osx/*/iphone/cli/commands/_build.js
  

Building Titanium Apps

export HOME=$TMPDIR

mkdir -p $HOME/.titanium
cat > $HOME/.titanium/auth_session.json <<EOF
{ "loggedIn": true }
EOF

$ echo "{}"> $TMPDIR/config.json
$ titanium --config-file $TMPDIR/config.json --no-colors \
    config sdk.defaultInstallLocation ${titaniumsdk}

$ titanium --config-file $TMPDIR/config.json --no-colors \
    config paths.modules ${titaniumsdk}

$ sed -i -e "s|<sdk-version>[0-9a-zA-Z\.]*</sdk-version>|<sdk-version>${tiVersion}</sdk-version>|" tiapp.xml

Building Android apps from Titanium projects

$ titanium config --config-file $TMPDIR/config.json --no-colors \
    android.sdkPath ${androidsdkComposition}/libexec/android-sdk-*

$ titanium build --config-file $TMPDIR/config.json --no-colors --force \
    --platform android --target emulator --build-only --output $out

$ titanium build --config-file $TMPDIR/config.json --no-colors --force \
    --platform android --target dist-playstore --keystore ${androidKeyStore} \
    --alias ${androidKeyAlias} --password ${androidKeyStorePassword} \
    --output-dir $out

Building iOS apps from Titanium projects

export NIX_TITANIUM_WORKAROUND="--config-file $TMPDIR/config.json"

$ cp -av * $out
$ cd $out

$ titanium build --config-file $TMPDIR/config.json --force --no-colors \
    --platform ios --target simulator --build-only \
    --device-family universal --output-dir $out

export HOME=/Users/$(whoami)
export keychainName=$(basename $out)

security create-keychain -p "" $keychainName
security default-keychain -s $keychainName
security unlock-keychain -p "" $keychainName
security import ${iosCertificate} -k $keychainName -P "${iosCertificatePassword}" -A

provisioningId=$(grep UUID -A1 -a ${iosMobileProvisioningProfile} | grep -o "[-A-Z0-9]\{36\}")

if [ ! -f "$HOME/Library/MobileDevice/Provisioning Profiles/$provisioningId.mobileprovision" ]
then
    mkdir -p "$HOME/Library/MobileDevice/Provisioning Profiles"
    cp ${iosMobileProvisioningProfile} \
"$HOME/Library/MobileDevice/Provisioning Profiles/$provisioningId.mobileprovision"
fi

cp -av ${titaniumsdk} $TMPDIR/titaniumsdk

find $TMPDIR/titaniumsdk | while read i
do
    chmod 755 "$i"
done

titanium --config-file $TMPDIR/config.json --no-colors \
    config sdk.defaultInstallLocation $TMPDIR/titaniumsdk

$ mkdir -p $HOME/.titanium
$ cat > $HOME/.titanium/auth_session.json <<EOF
{ "loggedIn": true }
EOF

$ titanium build --config-file $TMPDIR/config.json --force --no-colors \
    --platform ios --target dist-adhoc --pp-uuid $provisioningId \
    --distribution-name "${iosCertificateName}" \
    --keychain $HOME/Library/Keychains/$keychainName \
    --device-family universal --output-dir $out

Example: KitchenSink

{ titaniumenv, fetchgit
, target, androidPlatformVersions ? [ "11" ], release ? false
}:

titaniumenv.buildApp {
  name = "KitchenSink-${target}-${if release then "release" else "debug"}";
  src = fetchgit {
    url = https://github.com/appcelerator/KitchenSink.git;
    rev = "d9f39950c0137a1dd67c925ef9e8046a9f0644ff";
    sha256 = "0aj42ac262hw9n9blzhfibg61kkbp3wky69rp2yhd11vwjlcq1qc";
  };
  tiVersion = "3.2.1.GA";

  inherit target androidPlatformVersions release;

  androidKeyStore = ./keystore;
  androidKeyAlias = "myfirstapp";
  androidKeyStorePassword = "mykeystore";
}

import ./kitchensink {
  inherit (pkgs) fetchgit titaniumenv;
  target = "android";
  release = false;
}

import ./kitchensink {
  inherit (pkgs) fetchgit titaniumenv;
  target = "iphone";
  release = false;
}

sed -i -e "s|com.appcelerator.kitchensink|${newBundleId}|" tiapp.xml
sed -i -e "s|com.appcelerator.kitchensink|${newBundleId}|" manifest

import ./kitchensink {
  inherit (pkgs) stdenv fetchgit titaniumenv;
  target = "iphone";
  release = true;
  rename = true;
  newBundleId = "com.example.kitchensink";
  iosMobileProvisioningProfile = ./profile.mobileprovision;
  iosCertificate = ./certificate.p12;
  iosCertificateName = "Cool Company";
  iosCertificatePassword = "secret";
}

Discussion

$ sudo chmod u+s /usr/sbin/dtrace

Availability

$ nix-build -A kitchensink_android_debug

$ nix-build -A kitchensink_android_release

$ nix-build -A kitchensink_ios_development

$ nix-build --arg rename true \
    --argstr newBundleId com.example.kitchensink \
    --arg iosMobileProvisionProfile ./profile.mobileprovision \
    --arg iosCertificate ./certificate.p12 \
    --argstr iosCertificateName "Cool Company" \
    --argstr iosCertificatePassword secret \
    -A kitchensink_ipa

Target-agnostic service deployment

StaffTracker = [ infrastructure.test2 ];

StaffTracker = [ infrastructure.test1 ];

Deploying target-specific services

nginx = [ infrastructure.test1 infrastructure.test2 ];

{pkgs, system, distribution}:

let
  customPkgs = ...
in
rec {
  ...

  nginx-wrapper-test1 = rec {
    name = "nginx-wrapper-test1";
    pkg = customPkgs.nginx-wrapper;
    dependsOn = {
      inherit webapp1 webapp2;
    };
    type = "wrapper";
  };

  nginx-wrapper-test2 = rec {
    name = "nginx-wrapper-test2";
    pkg = customPkgs.nginx-wrapper;
    dependsOn = {
      inherit webapp3 webapp4;
    };
    type = "wrapper";
  };
}

{infrastructure}:

{
  ...
  nginx-wrapper-test1 = [ infrastructure.test1 ];
  nginx-wrapper-test2 = [ infrastructure.test2 ];
}

{infrastructure}:

let
  inherit (builtins) listToAttrs attrNames getAttr;
in
{
  webapp1 = [ infrastructure.test1 ];
  webapp2 = [ infrastructure.test1 ];
  webapp3 = [ infrastructure.test2 ];
  webapp4 = [ infrastructure.test2 ];
} //

# To each target, distribute a reverse proxy

listToAttrs (map (targetName: {
  name = "nginx-wrapper-${targetName}";
  value = [ (getAttr targetName infrastructure) ];
}) (attrNames infrastructure))

{system, pkgs, distribution, invDistribution}:

let
  customPkgs = import ../top-level/all-packages.nix {
    inherit pkgs system;
  };
in
{
  webapp1 = ...

  webapp2 = ...

  webapp3 = ...

  webapp4 = ...
} //

# Generate nginx proxy per target host

builtins.listToAttrs (map (targetName:
  let
    serviceName = "nginx-wrapper-${targetName}";
    servicesToTarget = (builtins.getAttr targetName invDistribution).services;
  in
  { name = serviceName;
    value = {
      name = serviceName;
      pkg = customPkgs.nginx-wrapper;
      # The reverse proxy depends on all services distributed to the same
      # machine, except itself (of course)
      dependsOn = builtins.removeAttrs servicesToTarget [ serviceName ];
      type = "wrapper";
    };
  }
) (builtins.attrNames invDistribution))

{
  test1 = {
    services = {
      nginx-wrapper-test1 = ...
      webapp1 = ...
      webapp2 = ...
    };
    hostname = "test1";
  };

  test2 = {
    services = {
      nginx-wrapper-test2 = ...
      webapp3 = ...
      webapp4 = ...
    };
    hostname = "test2";
  };
}

Conclusion

• Delays. There are many factors that will unnecessarily increase the time it will take to implement a change. Many of my big delays were caused by the fact that I always have to search for all the relevant installation artifacts, such as documentation, installation discs, and so on. I have also encountered many times that artifacts were missing requiring me to obtain copies elsewhere.
• Errors. Any change could potentially lead to errors for many kinds of reasons. For example, implementing a set of changes in the wrong order could break a system. Also, components of which a system consist may have complex dependencies on other components that have to be met. Quite often, it is not fully clear what the dependencies of a component or system are, especially when documentation is incomplete or lacking.
  
  Moreover, after having solved an error, you need to remember many arbitrary things, such as workarounds, that tend to become forgotten knowledge over time.
  
• Disruptions. When implementing changes, a system may be partially or fully unavailable until all changes have been implemented. Preferably this time window should be as short as possible. Unfortunately, the inconsistency time window most likely becomes quite big when the configuration management process is not optimal or subject to errors.

The IEEE Standard for Configuration Management

• CM identification, which concerns identifying, naming, describing, versioning, storing and retrieving configuration items (CIs) and their baselines.
• CM change control is about planning, requesting, approval and verification of change activities.
• CM status accounting is about identifying the status of CIs and change requests.
• CM auditing concerns identifying, tracing and reporting discrepancies with regards to the CIs and their baselines.
• CM release management is about planning, defining a format for distribution, delivery, and archival of configuration items.

Implementing a configuration management process

CM identification

<environment>/<machine | container>/<application>

• The top-levelREADME.md describes which environments are available and what their purposes are.
• The environment-levelREADME.md describes which machines/containers are part of it, a brief description of their purpose, and a picture showing how they are connected. I typically use Dia to draw them, because the tool is simple and free.
• The machine-levelREADME.md describes the purpose of the machine and the activities that must be carried out to reproduce its configuration from scratch.
• The application-levelREADME.md captures the steps that must be executed to reproduce an application's configuration.

• README.md files are not supposed to be tool manuals. I mention the steps that must be executed and what their purposes are, but I avoid explaining how a tool works. That is the purpose of the tool's manual.
• When there are common files used among machines, applications or environments, I do not duplicate them. Instead, I use a _common/ folder that I put one directory level higher. For example, the _common/ folder in the internal/ directory contains shared artifacts that are supposed to reused among all machines belonging to our internal IT infrastructure.
• I also capture common configuration steps in a separate README.md and refer to it, instead of duplicating the same steps in multiple README.md files.

CM change control

• Some configuration changes require validation including review and testing before they can be actually implemented in production. I typically keep the master Git branch in a state releasable state, meaning that it is ready to be deployed into production. Any changes that require explicit validation go into a different branch first.
  
  Moreover, when using tools from the Nix project it is relatively easy to reliably test changes first by deploying a system into a test environment, or by spawning virtual NixOS machines in which integration tests can be executed.
• Sometimes you need to make an analysis of the impact and costs that a change would bring. Up-to-date and consistent documentation of the CIs including their dependencies makes this process more reliable.
  
  Furthermore, with tools from the Nix project you can also make better estimations by executing a dry-run deployment process -- the dry run shows what activities will be carried out without actually executing them or bringing the system down.
• After a change has been deployed, we also need to validate whether the new configuration is correct. Typically, this requires testing.
  
  Tools from the Nix project support congruent deployment, meaning that if the deployment has succeeded, the actual configuration is guaranteed to match the deployment specification for the static parts of a system, giving better guarantees about its validity.
  
• Also you have to pick the right moment to implement potentially disrupting changes. For example, it is typically a bad idea to do this while your service is under peak load.

CM status accounting

• Automating deployment with tools from the Nix project ensure that the static parts of a deployed system are congruent with the deployment configuration specification and atomic -- either a deployment is in the old configuration or the new configuration but never in an inconsistent mix of the two. As a result, we have fewer broken configurations as a result of (re)deploying a system.
• We must also observe a system's runtime behavior and take action if things will grow out of hand. For example, when a machine runs out of system resources.
  
  Using a monitoring service, such as Zabbix or Datadog, helps me a lot in accomplishing this. They can also be used to configure alarms that warn you when things become critical.

CM auditing

CM release management

Discussion

Why prebuilt binaries will typically not work

with import <nixpkgs> {};

stdenv.mkDerivation {
  name = "pngout-20130221";

  src = fetchurl {
    url = http://static.jonof.id.au/dl/kenutils/pngout-20130221-linux.tar.gz;
    sha256 = "1qdzmgx7si9zr7wjdj8fgf5dqmmqw4zg19ypg0pdz7521ns5xbvi";
  };

  installPhase = ''
    mkdir -p $out/bin
    cp x86_64/pngout $out/bin
'';
}

$ nix-build pngout.nix

$ ./result/bin/pngout
bash: ./result/bin/pngout: No such file or directory

$ readelf -l ./result/bin/pngout 

Elf file type is EXEC (Executable file)
Entry point 0x401160
There are 8 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000000400040 0x0000000000400040
                 0x00000000000001c0 0x00000000000001c0  R E    8
  INTERP         0x0000000000000200 0x0000000000400200 0x0000000000400200
                 0x000000000000001c 0x000000000000001c  R      1
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
  LOAD           0x0000000000000000 0x0000000000400000 0x0000000000400000
                 0x000000000001593c 0x000000000001593c  R E    200000
  LOAD           0x0000000000015940 0x0000000000615940 0x0000000000615940
                 0x00000000000005b4 0x00000000014f9018  RW     200000
  DYNAMIC        0x0000000000015968 0x0000000000615968 0x0000000000615968
                 0x00000000000001b0 0x00000000000001b0  RW     8
  NOTE           0x000000000000021c 0x000000000040021c 0x000000000040021c
                 0x0000000000000044 0x0000000000000044  R      4
  GNU_EH_FRAME   0x0000000000014e5c 0x0000000000414e5c 0x0000000000414e5c
                 0x00000000000001fc 0x00000000000001fc  R      4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     8

Why complication from source works

$ ld test.o -lz -o test

$ ld test.o -lz \
  -rpath /nix/store/31w31mc8i...-zlib-1.2.8/lib \
  -dynamic-linker \
    /nix/store/hd6km3hscb...-glibc-2.21/lib/ld-linux-x86-64.so.2 \
  ...
  -o test

#include <stdio.h>
#include <zlib.h>

int main()
{
    printf("zlib version is: %s\n", ZLIB_VERSION);
    return 0;
}

with import <nixpkgs> {};

stdenv.mkDerivation {
  name = "test";
  buildInputs = [ zlib ];
  buildCommand = ''
    gcc ${./test.c} -lz -o test
    mkdir -p $out/bin
    cp test $out/bin
'';
}

nix-build test.nix

$ readelf -l ./result/bin/test 

Elf file type is EXEC (Executable file)
Entry point 0x400680
There are 9 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000000400040 0x0000000000400040
                 0x00000000000001f8 0x00000000000001f8  R E    8
  INTERP         0x0000000000000238 0x0000000000400238 0x0000000000400238
                 0x0000000000000050 0x0000000000000050  R      1
      [Requesting program interpreter: /nix/store/hd6km3hs...-glibc-2.21/lib/ld-linux-x86-64.so.2]
  LOAD           0x0000000000000000 0x0000000000400000 0x0000000000400000
                 0x000000000000096c 0x000000000000096c  R E    200000
  LOAD           0x0000000000000970 0x0000000000600970 0x0000000000600970
                 0x0000000000000260 0x0000000000000268  RW     200000
  DYNAMIC        0x0000000000000988 0x0000000000600988 0x0000000000600988
                 0x0000000000000200 0x0000000000000200  RW     8
  NOTE           0x0000000000000288 0x0000000000400288 0x0000000000400288
                 0x0000000000000020 0x0000000000000020  R      4
  GNU_EH_FRAME   0x0000000000000840 0x0000000000400840 0x0000000000400840
                 0x0000000000000034 0x0000000000000034  R      4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     8
  PAX_FLAGS      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000         8

$ readelf -d ./result/bin/test 

Dynamic section at offset 0x988 contains 27 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000f (RPATH)              Library rpath: [
/nix/store/8w39iz6sp...-test/lib64:
/nix/store/8w39iz6sp...-test/lib:
/nix/store/i9nn1fkcy...-gcc-4.9.3/libexec/gcc/x86_64-unknown-linux-gnu/4.9.3:
/nix/store/31w31mc8i...-zlib-1.2.8/lib:
/nix/store/hd6km3hsc...-glibc-2.21/lib:
/nix/store/i9nn1fkcy...-gcc-4.9.3/lib]
 0x000000000000001d (RUNPATH)            Library runpath: [
/nix/store/8w39iz6sp...-test/lib64:
/nix/store/8w39iz6sp...-test/lib:
/nix/store/i9nn1fkcy...-gcc-4.9.3/libexec/gcc/x86_64-unknown-linux-gnu/4.9.3:
/nix/store/31w31mc8i...-zlib-1.2.8/lib:
/nix/store/hd6km3hsc...-glibc-2.21/lib:
/nix/store/i9nn1fkcy...-gcc-4.9.3/lib]
 0x000000000000000c (INIT)               0x400620
 0x000000000000000d (FINI)               0x400814
 0x0000000000000019 (INIT_ARRAY)         0x600970
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x600978
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x0000000000000004 (HASH)               0x4002a8
 0x0000000000000005 (STRTAB)             0x400380
 0x0000000000000006 (SYMTAB)             0x4002d8
 0x000000000000000a (STRSZ)              528 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x600b90
 0x0000000000000002 (PLTRELSZ)           72 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x4005d8
 0x0000000000000007 (RELA)               0x4005c0
 0x0000000000000008 (RELASZ)             24 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x4005a0
 0x000000006fffffff (VERNEEDNUM)         1
 0x000000006ffffff0 (VERSYM)             0x400590
 0x0000000000000000 (NULL)               0x0

$ ./result/bin/test 
zlib version is: 1.2.8

Patching existing ELF binaries

$ patchelf --set-interpreter \
    ${stdenv.glibc}/lib/ld-linux-x86-64.so.2 $out/bin/pngout

$ readelf -d ./result/bin/pngout

Dynamic section at offset 0x15968 contains 22 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x400ea8
 0x000000000000000d (FINI)               0x413a78
 0x0000000000000004 (HASH)               0x400260
 0x000000006ffffef5 (GNU_HASH)           0x4003b8
 0x0000000000000005 (STRTAB)             0x400850
 0x0000000000000006 (SYMTAB)             0x4003e8
 0x000000000000000a (STRSZ)              379 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x615b20
 0x0000000000000002 (PLTRELSZ)           984 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x400ad0
 0x0000000000000007 (RELA)               0x400a70
 0x0000000000000008 (RELASZ)             96 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x400a30
 0x000000006fffffff (VERNEEDNUM)         2
 0x000000006ffffff0 (VERSYM)             0x4009cc
 0x0000000000000000 (NULL)               0x0

$ patchelf --set-rpath ${stdenv.glibc}/lib $out/bin/pngout

with import <nixpkgs> {};

stdenv.mkDerivation {
  name = "pngout-20130221";

  src = fetchurl {
    url = http://static.jonof.id.au/dl/kenutils/pngout-20130221-linux.tar.gz;
    sha256 = "1qdzmgx7si9zr7wjdj8fgf5dqmmqw4zg19ypg0pdz7521ns5xbvi";
  };

  installPhase = ''
    mkdir -p $out/bin
    cp x86_64/pngout $out/bin
    patchelf --set-interpreter \
        ${stdenv.glibc}/lib/ld-linux-x86-64.so.2 $out/bin/pngout
    patchelf --set-rpath ${stdenv.glibc}/lib $out/bin/pngout
'';
}

$ nix-build pngout.nix

$ ./result/bin/pngout 
PNGOUT [In:{PNG,JPG,GIF,TGA,PCX,BMP}] (Out:PNG) (options...)
by Ken Silverman (http://advsys.net/ken)
Linux port by Jonathon Fowler (http://www.jonof.id.au/pngout)

A more complex example: Quake 4 demo

$ bash ./quake4-linux-1.0-demo.x86.run --noexec --keep

$ cd quake4-linux-1.0-demo
$ ls
bin
Docs
License.txt
openurl.sh
q4base
q4icon.bmp
README
setup.data
setup.sh
version.info

$ file $(find . -type f)         
./Docs/QUAKE4_demo_readme.txt:     Little-endian UTF-16 Unicode text, with CRLF line terminators
./bin/Linux/x86/libstdc++.so.5:    ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped
./bin/Linux/x86/quake4-demo:       POSIX shell script, ASCII text executable
./bin/Linux/x86/quake4.x86:        ELF 32-bit LSB executable, Intel 80386, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.0.30, stripped
./bin/Linux/x86/quake4-demoded:    POSIX shell script, ASCII text executable
./bin/Linux/x86/libgcc_s.so.1:     ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped
./bin/Linux/x86/q4ded.x86:         ELF 32-bit LSB executable, Intel 80386, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.0.30, stripped
./README:                          ASCII text
./version.info:                    ASCII text
./q4base/game100.pk4:              Zip archive data, at least v2.0 to extract
./q4base/mapcycle.scriptcfg:       ASCII text, with CRLF line terminators
./q4base/game000.pk4:              Zip archive data, at least v1.0 to extract
./License.txt:                     ISO-8859 text, with very long lines
./openurl.sh:                      POSIX shell script, ASCII text executable
./q4icon.bmp:                      PC bitmap, Windows 3.x format, 48 x 48 x 24
...

$ cd ./bin/Linux/x86
$ readelf -d q4ded.x86 

Dynamic section at offset 0x366220 contains 25 entries:
  Tag        Type                         Name/Value
 0x00000001 (NEEDED)                     Shared library: [libpthread.so.0]
 0x00000001 (NEEDED)                     Shared library: [libdl.so.2]
 0x00000001 (NEEDED)                     Shared library: [libstdc++.so.5]
 0x00000001 (NEEDED)                     Shared library: [libm.so.6]
 0x00000001 (NEEDED)                     Shared library: [libgcc_s.so.1]
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
...

$ readelf -d ./quake4.x86 

Dynamic section at offset 0x3779ec contains 29 entries:
  Tag        Type                         Name/Value
 0x00000001 (NEEDED)                     Shared library: [libSDL-1.2.so.0]
 0x00000001 (NEEDED)                     Shared library: [libpthread.so.0]
 0x00000001 (NEEDED)                     Shared library: [libdl.so.2]
 0x00000001 (NEEDED)                     Shared library: [libstdc++.so.5]
 0x00000001 (NEEDED)                     Shared library: [libm.so.6]
 0x00000001 (NEEDED)                     Shared library: [libgcc_s.so.1]
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
 0x00000001 (NEEDED)                     Shared library: [libX11.so.6]
 0x00000001 (NEEDED)                     Shared library: [libXext.so.6]
...

$ readelf -d ./libgcc_s.so.1 

Dynamic section at offset 0x7190 contains 23 entries:
  Tag        Type                         Name/Value
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
...

$ readelf -d ./libstdc++.so.5 

Dynamic section at offset 0xadd8c contains 25 entries:
  Tag        Type                         Name/Value
 0x00000001 (NEEDED)                     Shared library: [libm.so.6]
 0x00000001 (NEEDED)                     Shared library: [libgcc_s.so.1]
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
...

with import <nixpkgs> { system = "i686-linux"; };

stdenv.mkDerivation {
  name = "quake4-demo-1.0";
  src = fetchurl {
    url = ftp://ftp.idsoftware.com/idstuff/quake4/demo/quake4-linux-1.0-demo.x86.run;
    sha256 = "0wxw2iw84x92qxjbl2kp5rn52p6k8kr67p4qrimlkl9dna69xrk9";
  };
  buildCommand = ''
    # Extract files from the installer
    cp $src quake4-linux-1.0-demo.x86.run
    bash ./quake4-linux-1.0-demo.x86.run --noexec --keep

    # Move extracted files into the Nix store
    mkdir -p $out/libexec
    mv quake4-linux-1.0-demo $out/libexec
    cd $out/libexec/quake4-linux-1.0-demo

    # Remove obsolete setup files
    rm -rf setup.data

    # Patch ELF binaries
    cd bin/Linux/x86
    patchelf --set-interpreter ${stdenv.cc.libc}/lib/ld-linux.so.2 ./quake4.x86
    patchelf --set-rpath $(pwd):${stdenv.cc.libc}/lib:${SDL}/lib:${xlibs.libX11}/lib:${xlibs.libXext}/lib ./quake4.x86
    chmod +x ./quake4.x86

    patchelf --set-interpreter ${stdenv.cc.libc}/lib/ld-linux.so.2 ./q4ded.x86
    patchelf --set-rpath $(pwd):${stdenv.cc.libc}/lib ./q4ded.x86
    chmod +x ./q4ded.x86

    patchelf --set-rpath ${stdenv.cc.libc}/lib ./libgcc_s.so.1
    patchelf --set-rpath $(pwd):${stdenv.cc.libc}/lib ./libstdc++.so.5
'';
}

• We import the Nixpkgs collection so that we can provide the external dependencies that the package needs. Because the executables are 32-bit x86 binaries, we need to refer to packages built for the i686-linux architecture.
• We download the Quake 4 demo installer from Id software's FTP server.
• We automate the steps we have done earlier -- we extract the files from the installer, move them into Nix store, prune the obsolete setup files, and finally we patch the ELF executables and libraries with the paths to the dependencies that we have discovered in our investigation.

$ nix-build quake4demo.nix

./result/libexec/quake4-linux-1.0-demo/bin/Linux/x86/quake4.x86

...
no 'q4base' directory in executable path /nix/store/0kfgsjryycsk5kfv97phj8ypv66n6caz-quake4-demo-1.0/libexec/quake4-linux-1.0-demo/bin/Linux/x86, skipping
no 'q4base' directory in current durectory /home/sander/quake4, skipping

$ strace -f ./result/libexec/quake4-linux-1.0-demo/bin/Linux/x86/quake4.x86
...
stat64("/nix/store/0kfgsjryycsk5kfv97phj8ypv66n6caz-quake4-demo-1.0/libexec/quake4-linux-1.0-demo/bin/Linux/x86/q4base", 0xffd7b230) = -1 ENOENT (No such file or directory)
write(1, "no 'q4base' directory in executa"..., 155no 'q4base' directory in executable path /nix/store/0kfgsjryycsk5kfv97phj8ypv66n6caz-quake4-demo-1.0/libexec/quake4-linux-1.0-demo/bin/Linux/x86, skipping
) = 155
...

$ cd result/libexec/quake4-linux-1.0-demo
$ ./bin/Linux/x86/quake4.x86
...
--------------- R_InitOpenGL ----------------
Initializing SDL subsystem
Loading GL driver 'libGL.so.1' through SDL
libGL error: unable to load driver: i965_dri.so
libGL error: driver pointer missing
libGL error: failed to load driver: i965
libGL error: unable to load driver: swrast_dri.so
libGL error: failed to load driver: swrast
X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  154 (GLX)
  Minor opcode of failed request:  3 (X_GLXCreateContext)
  Value in failed request:  0x0
  Serial number of failed request:  33
  Current serial number in output stream:  34

$ export LIBGL_DEBUG=verbose

--------------- R_InitOpenGL ----------------
Initializing SDL subsystem
Loading GL driver 'libGL.so.1' through SDL
libGL: OpenDriver: trying /run/opengl-driver-32/lib/dri/tls/i965_dri.so
libGL: OpenDriver: trying /run/opengl-driver-32/lib/dri/i965_dri.so
libGL: dlopen /run/opengl-driver-32/lib/dri/i965_dri.so failed (/nix/store/0kfgsjryycsk5kfv97phj8ypv66n6caz-quake4-demo-1.0/libexec/quake4-linux-1.0-demo/bin/Linux/x86/libgcc_s.so.1: version `GCC_3.4' not found (required by /run/opengl-driver-32/lib/dri/i965_dri.so))
libGL error: unable to load driver: i965_dri.so
libGL error: driver pointer missing
libGL error: failed to load driver: i965
libGL: OpenDriver: trying /run/opengl-driver-32/lib/dri/tls/swrast_dri.so
libGL: OpenDriver: trying /run/opengl-driver-32/lib/dri/swrast_dri.so
libGL: dlopen /run/opengl-driver-32/lib/dri/swrast_dri.so failed (/nix/store/0kfgsjryycsk5kfv97phj8ypv66n6caz-quake4-demo-1.0/libexec/quake4-linux-1.0-demo/bin/Linux/x86/libgcc_s.so.1: version `GCC_3.4' not found (required by /run/opengl-driver-32/lib/dri/swrast_dri.so))
libGL error: unable to load driver: swrast_dri.so
libGL error: failed to load driver: swrast
X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  154 (GLX)
  Minor opcode of failed request:  3 (X_GLXCreateContext)
  Value in failed request:  0x0
  Serial number of failed request:  33
  Current serial number in output stream:  34

with import <nixpkgs> { system = "i686-linux"; };

stdenv.mkDerivation {
  name = "quake4-demo-1.0";
  src = fetchurl {
    url = ftp://ftp.idsoftware.com/idstuff/quake4/demo/quake4-linux-1.0-demo.x86.run;
    sha256 = "0wxw2iw84x92qxjbl2kp5rn52p6k8kr67p4qrimlkl9dna69xrk9";
  };
  buildCommand = ''
    # Extract files from the installer
    cp $src quake4-linux-1.0-demo.x86.run
    bash ./quake4-linux-1.0-demo.x86.run --noexec --keep

    # Move extracted files into the Nix store
    mkdir -p $out/libexec
    mv quake4-linux-1.0-demo $out/libexec
    cd $out/libexec/quake4-linux-1.0-demo

    # Remove obsolete setup files
    rm -rf setup.data

    # Patch ELF binaries
    cd bin/Linux/x86
    patchelf --set-interpreter ${stdenv.cc.libc}/lib/ld-linux.so.2 ./quake4.x86
    patchelf --set-rpath $(pwd):${stdenv.cc.cc}/lib:${stdenv.cc.libc}/lib:${SDL}/lib:${xlibs.libX11}/lib:${xlibs.libXext}/lib ./quake4.x86
    chmod +x ./quake4.x86

    patchelf --set-interpreter ${stdenv.cc.libc}/lib/ld-linux.so.2 ./q4ded.x86
    patchelf --set-rpath $(pwd):${stdenv.cc.cc}/lib:${stdenv.cc.libc}/lib ./q4ded.x86
    chmod +x ./q4ded.x86

    patchelf --set-rpath $(pwd):${stdenv.cc.libc}/lib ./libstdc++.so.5

    # Remove libgcc_s.so.1 that conflicts with Mesa3D's libGL.so
    rm ./libgcc_s.so.1

    # Create wrappers for the executables
    mkdir -p $out/bin
    cat > $out/bin/q4ded <&ltEOF
    #! ${stdenv.shell} -e
    cd $out/libexec/quake4-linux-1.0-demo
    ./bin/Linux/x86/q4ded.x86 "\$@"
    EOF
    chmod +x $out/bin/q4ded

    cat > $out/bin/quake4 <<EOF
    #! ${stdenv.shell} -e
    cd $out/libexec/quake4-linux-1.0-demo
    ./bin/Linux/x86/quake4.x86 "\$@"
    EOF
    chmod +x $out/bin/quake4
'';
}

$ nix-env -f quake4demo.nix -i quake4-demo

$ quake4

Conclusion

• To enable hardware accelerated OpenGL for 32-bit applications in a 64-bit NixOS, add the following property to /etc/nixos/configuration.nix:
  
  

  
  hardware.opengl.driSupport32Bit = true;
  

• Id sofware's FTP server seems to be quite slow to download from. You can also obtain the demo from a different download site (e.g. Fileplanet) and run the following command to get it imported into the Nix store:
  
  

  
  $ nix-prefetch-url file:///home/sander/quake4-linux-1.0-demo.x86.run
  

Configuring the target machines

Instantiating the VMs

Installing the operating systems

{ pkgs, ... }:

{
  boot.loader.grub.device = "/dev/sda";
  fileSystems = {
"/" = { label = "root"; };
  };
  networking.firewall.enable = false;
  services.openssh.enable = true;
  services.tomcat.enable = true;
  services.disnix.enable = true;
  services.disnix.useWebServiceInterface = true;

  environment.systemPackages = [ pkgs.mc ];
}

Configuring SSH

$ sudo apt-get install openssh-server

$ ssh-host-config

Setting up user accounts

$ ssh-keygen

$ mkdir -p ~/.ssh
$ chmod 700 ~/.ssh
$ cat id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod 600 ~/.ssh/authorized_keys

Installing Nix, Dysnomia and Disnix

$ curl https://nixos.org/nix/install | sh

$ nix-env -i $(nix-build -E 'with import <nixpkgs> {}; dysnomia.override { jobTemplate = "direct"; }')

$ nix-env -f '<nixpkgs>' -iA disnix

$ sudo cp /nix/var/nix/profiles/default/etc/dbus-1/system.d/disnix.conf \
    /etc/dbus-1/system.d

<policy user="root">

<policy user="sander">

$ sudo cp /nix/var/nix/profiles/default/share/doc/disnix/disnix-service.initd /etc/init.d/disnix-service
$ sudo ln -s ../init.d/disnix-service /etc/rc2.d/S06disnix-service
$ sudo ln -s ../init.d/disnix-service /etc/rc3.d/S06disnix-service
$ sudo ln -s ../init.d/disnix-service /etc/rc4.d/S06disnix-service
$ sudo ln -s ../init.d/disnix-service /etc/rc5.d/S06disnix-service

DAEMONUSER=sander

curl
patch
perl
libbz2-devel
sqlite3
make
gcc-g++
pkg-config
libsqlite3-devel
libcurl-devel
openssl-devel
libcrypt-devel
libdbus1-devel
libdbus-glib-1-devel
libxml2-devel
libxslt-devel
dbus
openssh

$ perl -MCPAN -e shell
install DBD::SQLite
install WWW::Curl

tar xfv nix-1.10.tar.xz
cd nix-1.10
./configure
make
make install

tar xfv dysnomia-0.5pre1234.tar.gz
cd dysnomia-0.5pre1234
./configure --with-job-template=direct
make
make install

tar xfv disnix-0.5pre1234.tar.gz
cd disnix-0.5pre1234
./configure
make
make install

$ cp /nix/var/nix/profiles/default/etc/dbus-1/system.d/disnix.conf \
    /etc/dbus-1/system.d

<policy user="root">

<policy user="sander">

$ cygrunsrv -I dbus -p /usr/bin/dbus-daemon.exe \
    -a '--system --nofork'

$ cygrunsrv -I disnix -p /usr/local/bin/disnix-service.exe \
  -e 'PATH=/bin:/usr/bin:/usr/local/bin' \
  -y dbus -u sander

$ editrights -u sander -l

$ editrights -u sander -a SeServiceLogonRight

Deploying the example scenarios

$ cd deployment/DistributedDeployment

{
  test1 = { # x86 Linux machine (Kubuntu) reachable with SSH
    hostname = "192.168.56.101";
    system = "i686-linux";
    targetProperty = "hostname";
    clientInterface = "disnix-ssh-client";
  };

  test2 = { # x86-64 Linux machine (NixOS) reachable with SOAP/HTTP
    hostname = "192.168.56.102";
    system = "x86_64-linux";
    targetEPR = http://192.168.56.102:8080/DisnixWebService/services/DisnixWebService;
    targetProperty = "targetEPR";
    clientInterface = "disnix-soap-client";
  };

  test3 = { # x86-64 Windows machine (Windows 7) reachable with SSH
    hostname = "192.168.56.103";
    system = "x86_64-cygwin";
    targetProperty = "hostname";
    clientInterface = "disnix-ssh-client";
  };
}

{infrastructure}:

{
  hello_world_server = [ infrastructure.test2 ];
  hello_world_client = [ infrastructure.test1 ];
}

$ disnix-env -s services-without-proxy.nix \
  -i infrastructure.nix -d distribution.nix

$ /nix/var/nix/profiles/disnix/default/bin/hello-world-client

{infrastructure}:

{
  hello_world_server = [ infrastructure.test2 ];
  hello_world_client = [ infrastructure.test1 infrastructure.test3 ];
}

$ disnix-env -s services-without-proxy.nix \
  -i infrastructure.nix -d distribution.nix

{infrastructure}:

{
  hello_world_server = [ infrastructure.test3 ];
  hello_world_client = [ infrastructure.test1 infrastructure.test3 ];
}

$ disnix-env -s services-without-proxy.nix \
  -i infrastructure.nix -d distribution.nix

$ disnix-env -s services-without-proxy.nix \
  -i infrastructure.nix -d distribution.nix \
  --build-on-targets

Discussion

References

• The most obvious solution is buying bigger or additional hardware resources, but this typically increases the costs of maintaining a production environment. Moreover, it does not take the source of some of the problems away.
• Another solution would be to fix and optimize problematic services, but this could be a time consuming and costly process, in particular when there is a high technical debt.
  
• A third solution would be to support on-demand service activation and self termination -- a service gets activated the first time it is consulted and terminates itself after a period of idleness.

Retrieving a socket from the host system's service manager

static int create_server_socket(int source_port)
{
    int sockfd, on = 1;
    struct sockaddr_in client_addr;

    /* Create socket */
    sockfd = socket(AF_INET, SOCK_STREAM, 0);
    if(sockfd < 0)
    {
        fprintf(stderr, "Error creating server socket!\n");
        return -1;
    }    

    /* Create address struct */
    memset(&client_addr, '\0', sizeof(client_addr));
    client_addr.sin_family = AF_INET;
    client_addr.sin_addr.s_addr = htonl(INADDR_ANY);
    client_addr.sin_port = htons(source_port);

    /* Set socket options to reuse the address */
    setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &on, 4);

    /* Bind the name (ip address) to the socket */
    if(bind(sockfd, (struct sockaddr *)&client_addr, sizeof(client_addr)) < 0)
        fprintf(stderr, "Error binding on port: %d, %s\n", source_port, strerror(errno));

    /* Listen for connections on the socket */
    if(listen(sockfd, 5) < 0)
        fprintf(stderr, "Error listening on port %d\n", source_port);

    /* Return the socket file descriptor */
    return sockfd;
}

static int create_server_socket(int source_port)
{
    int sockfd, on = 1;

#ifdef SYSTEMD_SOCKET_ACTIVATION
    int n = sd_listen_fds(0);

    if(n > 1)
    {
        fprintf(stderr, "Too many file descriptors received!\n");
        return -1;
    }
    else if(n == 1)
        sockfd = SD_LISTEN_FDS_START + 0;
    else
    {
#endif
        struct sockaddr_in client_addr;

        /* Create socket */
        sockfd = socket(AF_INET, SOCK_STREAM, 0);
        if(sockfd < 0)
        {
            fprintf(stderr, "Error creating server socket!\n");
            return -1;
        }

        /* Create address struct */
        memset(&client_addr, '\0', sizeof(client_addr));
        client_addr.sin_family = AF_INET;
        client_addr.sin_addr.s_addr = htonl(INADDR_ANY);
        client_addr.sin_port = htons(source_port);

        /* Set socket options to reuse the address */
        setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &on, 4);

        /* Bind the name (ip address) to the socket */
        if(bind(sockfd, (struct sockaddr *)&client_addr, sizeof(client_addr)) < 0)
            fprintf(stderr, "Error binding on port: %d, %s\n", source_port, strerror(errno));

        /* Listen for connections on the socket */
        if(listen(sockfd, 5) < 0)
            fprintf(stderr, "Error listening on port %d\n", source_port);

#ifdef SYSTEMD_SOCKET_ACTIVATION
    }
#endif

    /* Return the socket file descriptor */
    return sockfd;
}

CC = gcc

ifeq ($(SYSTEMD_SOCKET_ACTIVATION),1)
    EXTRA_BUILDFLAGS=-DSYSTEMD_SOCKET_ACTIVATION=1 $(shell pkg-config --cflags --libs libsystemd)
endif

all:
 $(CC) $(EXTRA_BUILDFLAGS) hello-world-server.c -o hello-world-server

...

$ make SYSTEMD_SOCKET_ACTIVATION=1

Implementing self termination

volatile unsigned int num_of_connections = 0;

while(TRUE)
{
    /* Create client socket if there is an incoming connection */
    if((client_sockfd = wait_for_connection(server_sockfd)) >= 0)
    {
        /* Fork a new process for each incoming client */
        pid_t pid = fork();

        if(pid == 0)
        {
            /* Handle the client's request and terminate
             * when it disconnects */
        }
        else if(pid == -1)
            fprintf(stderr, "Cannot fork connection handling process!\n");
#ifdef SELF_TERMINATION
        else
            num_of_connections++;
#endif
    }

    close(client_sockfd);
    client_sockfd = -1;
}

#ifdef SELF_TERMINATION
    signal(SIGCHLD, sigreap);
#endif

#ifdef SELF_TERMINATION

void sigreap(int sig)
{
    pid_t pid;
    int status;
    num_of_connections--;

    /* Event handler when a child terminates */
    signal(SIGCHLD, sigreap);

    /* Wait until all child processes terminate */
    while((pid = waitpid(-1, &status, WNOHANG)) > 0);

    if(num_of_connections == 0)
        _exit(0);
}
#endif

Deploying services with on demand activation and self termination enabled

{stdenv, pkgconfig, systemd}:
{port, enableSystemdSocketActivation ? false}:

let
  makeFlags = "PREFIX=$out port=${toString port}${stdenv.lib.optionalString enableSystemdSocketActivation " SYSTEMD_SOCKET_ACTIVATION=1"}";
in
stdenv.mkDerivation {
  name = "hello-world-server";
  src = ../../../services/hello-world-server;
  buildInputs = if enableSystemdSocketActivation then [ pkgconfig systemd ] else [];
  buildPhase = "make ${makeFlags}";
  installPhase = ''
    make ${makeFlags} install

    mkdir -p $out/etc
    cat > $out/etc/process_config <<EOF
    container_process=$out/bin/process
    EOF

    ${stdenv.lib.optionalString enableSystemdSocketActivation ''
      mkdir -p $out/etc
      cat > $out/etc/socket <<EOF
      [Unit]
      Description=Hello world server socket

      [Socket]
      ListenStream=${toString port}
      EOF
''}
'';
}

• We make the socket activation and self termination features configurable by exposing it as a function parameter (that defaults to false disabling it).
• If the socket activation parameter has been enabled, we pass the SYSTEMD_SOCKET_ACTIVATION=1 flag to 'make' so that these facilities are enabled in the build system.
• We must also provide two extra dependencies: pkgconfig and systemd to allow the program to find the required library functions to retrieve the socket from systemd.
• We also compose a systemd socket unit file that configures systemd on the target system to allocate a server socket that activates the process when a client connects to it.

Modifying Dysnomia modules to support socket activation

$ systemctl start disnix-53bb1pl...-hello-world-server.socket

$ systemctl stop disnix-53bb1pl...-hello-world-server.socket
$ systemctl stop disnix-53bb1pl...-hello-world-server.service

Discussion

References

Disnix

The Nix project

Methodology

Development

Side projects and research

Blog posts

1. On Nix and GNU Guix. As with the previous three blog reflections, this blog post remains on top. However, its popularity finally seems to be challenged by the number two!
2. An evaluation and comparison of Snappy Ubuntu. This is the only blog post I have written this year that ended up in the overall top 10. It attracted a record number of visitors in one month and now rivals the number one in popularity.
3. An alternative explanation of the Nix package manager. This was last year's number two and dropped to the third place, because of the Snappy Ubuntu blog post.
4. Setting up a multi-user Nix installation on non-NixOS systems. This blog post was also in last year's top 10 but it seems to have become even more popular. I think this is probably caused by the fact that it is still hard to set up a multi-user installation.
5. Managing private Nix packages outside the Nixpkgs tree. I wrote this blog for newcomers and observed that people keep frequently consulting it. As a consequence, it has entered the overall top 10.
6. Asynchronous programming with JavaScript. This blog post was also in last year's top 10 and became slightly more popular. As a result, it moved to the 6th position.
7. Yet another blog post about Object Oriented Programming and JavaScript. Another JavaScript related blog post that was in last year's top 10. It became slightly more popular and moved to the 7th place.
8. Composing FHS-compatible chroot environments with Nix (or deploying Steam in NixOS). This blog post was the third most popular last year, but now seems to be not that interesting anymore.
9. Setting up a Hydra build cluster for continuous integration and testing (part 1). Remains a popular blog post, but also considerably dropped in popularity compared to last year.
10. Using Nix while doing development. A very popular blog post last year, but considerably dropped in popularity.

Conclusion

Two example invocations

fs.readFile("hello.txt", function(err, data) {
    if(err) {
        console.log("Error while opening file: "+err);
    } else {
        console.log("File contents is: "+data);
    }
});

Task.findAll().then(function(tasks) {
    for(var i = 0; i < tasks.length; i++) {
        var task = tasks[i];
        console.log(task.title + ": "+ task.description);
    }
}, function(err) {
    console.log("An error occured: "+err);
});

Comparing the invocation styles

var tasks = Task.findAll();

displayTasks(tasks);

var fs = require('fs');
var path = require('path');

fs.mkdir("out", 0755, function(err) {
    if(err) throw err;

    fs.mkdir(path.join("out, "test"), 0755, function(err) {
        if (err) throw err;        
        var filename = path.join("out", "test", "hello.txt");

        fs.writeFile(filename, "Hello world!", function(err) {
            if(err) throw err;

            fs.readFile(filename, function(err, data) {
                if(err) throw err;

                if(data == "Hello world!")
                    process.stderr.write("File is correct!\n");
                else
                    process.stderr.write("File is incorrect!\n");
            });
        });
    });
});

var fs = require('fs');
var path = require('path');

filename = path.join("out", "test", "hello.txt");

async.waterfall([
    function(callback) {
        fs.mkdir("out", 0755, callback);
    },

    function(callback) {
        fs.mkdir(path.join("out, "test"), 0755, callback);
    },

    function(callback) {
        fs.writeFile(filename, "Hello world!", callback);
    },

    function(callback) {
        fs.readFile(filename, callback);
    },

    function(data, callback) {
        if(data == "Hello world!")
            process.stderr.write("File is correct!\n");
        else
            process.stderr.write("File is incorrect!\n");
    }

], function(err, result) {
    if(err) throw err;
});

var fs = require('fs');
var path = require('path');
var Promise = require('rsvp').Promise;

/* Promise object definitions */

var mkdir = function(dirname) {
    return new Promise(function(resolve, reject) {
        fs.mkdir(dirname, 0755, function(err) {
            if(err) reject(err);
            else resolve();
        });
    });
};

var writeHelloTxt = function(filename) {
    return new Promise(function(resolve, reject) {
        fs.writeFile(filename, "Hello world!", function(err) {
            if(err) reject(err);
            else resolve();
        });
    });
};

var readHelloTxt = function(filename) {
    return new Promise(function(resolve, reject) {
        fs.readFile(filename, function(err, data) {
            if(err) reject(err);
            else resolve(data);
        });
    });
};

/* Promise execution chain */

var filename = path.join("out", "test", "hello.txt");

mkdir(path.join("out"))
.then(function() {
    return mkdir(path.join("out", "test"));
})
.then(function() {
    return writeHelloTxt(filename);
})
.then(function() {
    return readHelloTxt(filename);
})
.then(function(data) {
    if(data == "Hello world!")
        process.stderr.write("File is correct!\n");
    else
        process.stderr.write("File is incorrect!\n");
}, function(err) {
    console.log("An error occured: "+err);
});

Mixing function invocations from both styles

async.waterfall([
    function(callback) {
        Task.sync().then(function() {
            callback();
        }, function(err) {
            callback(err);
        });
    },

    function(callback) {
        Task.create({
            title: "Get some coffee",
            description: "Get some coffee ASAP"
        }).then(function() {
            callback();
        }, function(err) {
            callback(err);
        });
    },

    function(callback) {
        Task.create({
            title: "Drink coffee",
            description: "Because I need caffeine"
        }).then(function() {
            callback();
        }, function(err) {
            callback(err);
        });
    },

    function(callback) {
        Task.findAll().then(function(tasks) {
            callback(null, tasks);
        }, function(err) {
            callback(err);
        });
    },

    function(tasks, callback) {
        for(var i = 0; i < tasks.length; i++) {
            var task = tasks[i];
            console.log(task.title + ": "+ task.description);
        }
    }
], function(err) {
    if(err) {
        console.log("An error occurred: "+err);
        process.exit(1);
    } else {
        process.exit(0);
    }
});

function chainCallback(promise, callback) {
    promise.then(function() {
        var args = Array.prototype.slice.call(arguments, 0);

        args.unshift(null);
        callback.apply(null, args);
    }, function() {
        var args = Array.prototype.slice.call(arguments, 0);

        if(args.length == 0) {
            callback("Promise error");
        } else if(args.length == 1) {
            callback(args[0]);
        } else {
            callback(args);
        }
    });
}

• We define a function takes a promise and a Node.js-style callback function as parameters and invokes the then() method of the promise.
• When the promise has been fulfilled, it sets the error parameter of the callback to null (to indicate that there is no error) and propagates all resulting objects as remaining parameters to the callback.
• When the promise has been rejected, we propagate the resulting error object. Because the Node.js-style-callback requires a single defined object, we compose one ourselves if no error object was returned, and we return an array as an error object, if multiple error objects were returned.

async.waterfall([
    function(callback) {
        prom2cb.chainCallback(Task.sync(), callback);
    },

    function(callback) {
        prom2cb.chainCallback(Task.create({
            title: "Get some coffee",
            description: "Get some coffee ASAP"
        }), callback);
    },

    function(callback) {
        prom2cb.chainCallback(Task.create({
            title: "Drink coffee",
            description: "Because I need caffeine"
        }), callback);
    },

    function(callback) {
        prom2cb.chainCallback(Task.findAll(), callback);
    },

    function(tasks, callback) {
        for(var i = 0; i < tasks.length; i++) {
            var task = tasks[i];
            console.log(task.title + ": "+ task.description);
        }
    }
], function(err) {
    if(err) {
        console.log("An error occurred: "+err);
        process.exit(1);
    } else {
        process.exit(0);
    }
});

Task.findAll().then(function(tasks) {
    fs.writeFile("tasks.txt", JSON.stringify(tasks), function(err) {
        if(err) {
            console.log("error: "+err);
        } else {
            console.log("everything is OK");
        }
    });
}, function(err) {
    console.log("error: "+err);
});

function promisify(Promise, fun) {
    return function() {
       var args = Array.prototype.slice.call(arguments, 0);

       return new Promise(function(resolve, reject) {
            function callback() {
                var args = Array.prototype.slice.call(arguments, 0);
                var err = args[0];
                args.shift();

                if(err) {
                    reject(err);
                } else {
                    resolve(args);
                }
            }

            args.push(callback);

            fun.apply(null, args);
        });
    };
}

• We define a function that takes two parameters: a Promise prototype that can be used to construct promises and a function representing any Node.js-style function (which the last parameter is a Node.js-style callback).
• In the function, we construct (and return) a wrapper function that returns a promise.
• We construct an adapter callback function, that invokes the Promise toolkit's reject() function in case of an error (with the corresponding error object provided by the callback), and resolve() in case of success. In case of success, it simply propagates any result object provided by the Node.js-style callback.
• Finally, we invoke the Node.js-function with the given function parameters and our adapter callback.

Task.findAll().then(function(tasks) {
    return prom2cb.promisify(Promise, fs.writeFile)("tasks.txt", JSON.stringify(tasks));
})
.then(function() {
    console.log("everything is OK");
}, function(err) {
    console.log("error: "+err);
});

Conclusions

Related work

References

Some background information

Prototype 1

Prototype 2

Disnix 0.1

Disnix 0.2

Disnix 0.3

Disnix 0.4

Disnix 0.5

Conclusion

The first attempt

{
"name" : "nijs",
"version" : "0.0.23",
"dependencies" : {
"optparse" : ">= 1.0.3",
"slasp": "0.0.4"
  }
  ...
}

nijs/
  ...
  package.json
  node_modules/
    optparse/
      package.json
      ...
    slasp/
      package.json
      ...

/nix/store/ab12pq...-nijs-0.0.24/
  ...
  package.json
  node_modules/
    optparse -> /nix/store/4pq1db...-optparse-1.0.5
    slasp -> /nix/store/8j12qp...-slasp-0.0.4
/nix/store/4pq1db...-optparse-1.0.5/
  ...
  package.json
/nix/store/8j12qp...-slasp-0.0.4/
  ...
  package.json

{
"name": "myproject",
"version": "0.0.1",
"dependencies": {
"mongoose": "3.8.5",
"ironhorse": "0.0.11"
  }
}

{
"name": "ironhorse",
"version": "0.0.11",
"license" : "MIT",
"dependencies" : {
"underscore": "~1.5.2",
"mongoose": "*",
"temp": "*",
    ...
  },
  ...
}

myproject/
  ...
  package.json
  node_modules/
    mongoose/
      ...
    ironhorse/
      ...
      package.json
      node_modules/
        underscore/
        temp/

/nix/store/67ab07...-myproject-0.0.1
  ...
  package.json
  node_modules/
    mongoose -> /nix/store/ec704c...-mongoose-3.8.5
    ironhorse -> /nix/store/3ee85e...-ironhorse-0.0.11
/nix/store/3ee85e...-ironhorse-0.0.11
  ...
  package.json
  node_modules/
    underscore -> /nix/store/10af96...-underscore-1.5.2
    mongoose -> /nix/store/a37f75...-mongoose-4.4.5
    temp -> /nix/store/fae379...-temp-0.8.3
/nix/store/ec704c...-mongoose-3.8.5
  package.json
  ...
/nix/store/a37f75...-mongoose-4.4.5
  package.json
  ...
/nix/store/10af96...-underscore-1.5.2
/nix/store/fae379...-temp-0.8.3

Second attempt: simulating shared dependency behaviour

Third attempt: computing the dependency graph ahead of time

webapp/
  ...
  package.json
  node_modules/
    express/
      ...
      package.json
      node_modules/
        accepts/
        array-flatten/
        content-disposition/
        ...
    ejs/
      ...
      package.json

webapp/
  ...
  package.json
  node_modules/
    accepts/
    array-flatten/
    content-disposition/
    express/
      ...
      package.json
    ejs/
      ...
      package.json

• When a package requires a dependency: I check whether the parent directory has a conflicting dependency. This means: it either has a dependency bundled with the same name and a different version or indirectly binds to another parent that provides a conflicting version.
• If the dependency conflicts: bundle the dependency in the current package.
• If the dependency does not conflict: bind the package to the dependency (but do not include it) and consult the parent package one level higher.

$ npm2nix

$ npm2nix -5

Discussion

• The habit of "reusing" modules that have been bundled with any of the includers, makes it IMO difficult and counter-intuitive to predict which version will actually be used in a certain context. In some cases, packagers might expect that the latest version of a version range will be used, but this is not guaranteed to be the case. This could, for example, reintroduce security and stability issues without end users noticing (or expecting) it.
• Flat module installations are less deterministic and make it really difficult to predict what a dependency graph looks like -- the dependencies that appear at a certain level in the directory structure depend on the order in which dependencies are installed. Therefore, I do not consider this an improvement over npm 2.x.

• All involved packages should be stored in a singlenode_modules/ folder instead of multiple nested hierarchies of node_modules/ folders.
• When a module requests another module, the module loader should consult the package.json configuration file of the package where the includer module belongs to. It should take the latest conforming version in the central node_modules/ folder. I consider taking the last version of a version range to be less counter-intuitive than taking any conforming version.
• To be able to store multiple versions of packages in a single node_modules/ folder, a better directory naming convention should be adopted. Currently, NPM only identifies modules by name in a node_modules/ folder, making it impossible to store two versions next to each other in one directory.
  
  If they would, for example, use both the name and version numbers in the directory names, more things are possible. Adding more properties in the path names makes sharing even better -- for example, a package with a name and version number could originate from various sources, e.g. the NPM registry or a Git repository -- reflecting this in the path makes it possible to store more variants next to each other in a reliable way.
  
  Naming things to improve shareability is not really rocket science -- Nix uses hash codes (that are derived from all build-time dependencies) to uniquely identify packages and the .NET Global Assembly Cache uses so-called strong names that include various naming attributes, such as cryptographic keys to ensure that no library conflicts. I am convinced that adopting a better naming convention for storing NPM packages would be quite beneficial as well.
• To cope with cyclic dependencies: I would simply say that it suffices to disallow them. Packages are supposed to be units of reuse, and if two packages mutually depend on each other, then they should be combined into one package.

Availability

On being declarative

A declarative sentence makes a statement. It is punctuated by a period.

The dog in the neighbor's yard is barking.

An imperative sentence is a command or polite request. It ends in a period or exclamation mark.

On being declarative in programming

In computer science, declarative programming is a programming paradigm -- a style of building the structure and elements of computer programs -- that expresses the logic of a computation without describing its control flow.

Declarative: describing "what" is to be computed rather than "how" to compute the result/behavior

Imperative: a description of a computation that involves implicit effects, usually mutable state and input/output.

I agree with those who say that "declarative" is a spectrum. For example, some people say that Haskell is a declarative language, but I my view Haskell programs are very much about *how* to compute a result.

int a = 5;
a += 3;

int sum(int a, int b)
{
    return a + b;
}

int result = sum(5, 3);

<!DOCTYPE html>

<html>
<head>
<title>Test</title>
<link rel="stylesheet" href="style.css" type="text/css">
</head>
<body>
<div id="outer">
<div id="inner">
<p>HTML and CSS are declarative and so cool!</p>
</div>
</div>
</body>
</html>

#outer {
    margin-left: auto;
    margin-right: auto;
    width: 20%;
    border-style: solid;
}

#inner {
    width: 500px;
}

• Because it does not matter (much) how a page is rendered, we can fully utilize a system's resources (e.g. a GPU) to render a page in a faster and more fancy way, and optionally degrade a page's appearance if a system's resources are limited.
• We can also interpret the page in many ways. For example, we can pass the text in paragraphs to a text to speech engine, for people that are visually impaired.

On being declarative in deployment

• Building (if a compiled language is used, such as Java).
• Packaging (e.g. into a JAR file).
• Distributing (transferring artifacts to the production machines).
• Activating (e.g. a Java web application in a Servlet container).
• In case of an upgrade: deactivating obsolete components.

• Many kinds of steps need to be executed, in particular when the technology used is diverse. Without any automation, it becomes extra complicated and time consuming.
• Deployment in production must be typically done on a large scale. In development, a web application/web service typically serves one user only (the developer), while in production it may need to serve thousands or millions of users. In order to serve many users, you need to manage a cluster of machines having complex constraints in terms of system resources and connectivity.
• There are non-functional requirements that must be met. For example, while upgrading you want to minimize a system's downtime as much possible. You probably also want to roll back to a previous version if an upgrade went wrong. Accomplishing these properties is often much more complicated than expected (sometimes even impossible!).

Mainstream solutions: convergent deployment

...

wordpress_latest = Chef::Config[:file_cache_path] + "/wordpress-latest.tar.gz"

remote_file wordpress_latest do
  source "http://wordpress.org/latest.tar.gz"
  mode "0644"
end

directory node["phpapp"]["path"] do
  owner "root"
  group "root"
  mode "0755"
  action :create
  recursive true
end

execute "untar-wordpress" do
  cwd node['phpapp']['path']
  command "tar --strip-components 1 -xzf " + wordpress_latest
  creates node['phpapp']['path'] + "/wp-settings.php"
end

The NixOS approach

{pkgs, ...}:

{
  boot.loader.grub.device = "/dev/sda";

  fileSystems = [ { mountPoint = "/"; device = "/dev/sda2"; } ];
  swapDevices = [ { device = "/dev/sda1"; } ];

  services = {
    openssh.enable = true;

    xserver = {
      enable = true;
      desktopManager.kde4.enable = true;
    };
  };

  environment.systemPackages = [ pkgs.mc pkgs.firefox ];
}

• The GRUB bootloader should be installed on the MBR of partition: /dev/sda.
• The /dev/sda2 partition should be mounted as a root partition, /dev/sda1 should be mounted as a swap partition.
• We want Mozilla Firefox and Midnight Commander as end user packages.
• We want to use the KDE 4.x desktop.
• We want to run OpenSSH as a system service.

$ nixos-rebuild switch

• Reliability. Nix (the underlying package manager) ensures that all dependencies are present. It stores new versions of packages next to old versions, without overwriting them. As a result, you can always switch back to older versions if needed.
• Reproducibility. Undeclared dependencies do not influence builds -- if a build works on one machine, then it works on others as well.
• Efficiency. Nix only deploys packages and configuration files that are needed.

How declarative are these deployment solutions?

WebDSL

Conference compass

Conclusion

References

• I did a more elaborate comparison of infrastructure deployment solutions in Chapter 6. I also cover convergent deployment and used CFEngine as an example.
• I have covered webdsldeploy in Chapter 11, including some background information about WebDSL and its deployment aspects.
• The overall objective of my PhD thesis is constructing deployment tools for specific domains. Most of the chapter cover the ingredients to do so, but Chapter 3 explains a reference architecture for deployment tools, having similar (or comparable) properties to tools in the Nix project.

• Building services and their intra-dependencies from source code. By default, Disnix performs the builds on the coordinator machine, but can also optionally delegate them to target machines in the network.
• Distributing services and their intra-dependency closures to the appropriate target machines in the network.
• Activating newly deployed services, and deactivating obsolete services.
• Optionally snapshotting, transferring and restoring the state of services (or a subset of services) that have moved from a target machine to another.

• A MySQL database schema component and a MySQL DBMS container.
• An Java web application component (WAR file) and an Apache Tomcat container.
• A UNIX process component and a systemd container.
• Even NixOS configurations can be considered mutable components.

• Some systems are monolithic and cannot be be decomposed into services (i.e. distributable units) of deployment.
• Some NixOS modules have scripts to initialize the state of a system service on first startup, such as a database, but do it in their own ad-hoc way, e.g. there is no real formalism behind it.
• You may also want to use Dysnomia's (primitive) snapshotting facilities for backup purposes.

Deploying NixOS configurations

{pkgs, ...}:

{
  boot.loader.grub = {
    enable = true;
    device = "/dev/sda";
  };

  fileSystems."/" = {
    device = "/dev/disk/by-label/nixos";
    fsType = "ext4";  
  };

  services = {
    openssh.enable = true;

    mysql = {
      enable = true;
      package = pkgs.mysql;
      rootPassword = ../configurations/mysqlpw;
    };
  };
}

$ nixos-rebuild switch

...

mysql = {
  enable = true;
  package = pkgs.mysql;
  rootPassword = ../configurations/mysqlpw;
};

tomcat = {
  enable = true;
  commonLibs = [ "${pkgs.mysql_jdbc}/share/java/mysql-connector-java.jar" ];
  catalinaOpts = "-Xms64m -Xmx256m";
};

...

$ nixos-rebuild switch

Managing collections of mutable components

$ dysnomia-containers --query-containers
mysql-database
process
tomcat-webapplication
wrapper

$ dysnomia-containers --query-available-components
mysql-database/rooms
mysql-database/staff
mysql-database/zipcodes
tomcat-webapplication/GeolocationService
tomcat-webapplication/RoomService
tomcat-webapplication/StaffService
tomcat-webapplication/StaffTracker
tomcat-webapplication/ZipcodeService

$ dysnomia-containers --deploy
Activating component: rooms in container: mysql-database
Activating component: staff in container: mysql-database
Activating component: zipcodes in container: mysql-database
Activating component: GeolocationService in container: tomcat-webapplication
Activating component: RoomService in container: tomcat-webapplication
Activating component: StaffService in container: tomcat-webapplication
Activating component: StaffTracker in container: tomcat-webapplication
Activating component: ZipcodeService in container: tomcat-webapplication

$ dysnomia-containers --query-activated-components
mysql-database/rooms
mysql-database/staff
mysql-database/zipcodes
tomcat-webapplication/GeolocationService
tomcat-webapplication/RoomServiceWrapper
tomcat-webapplication/StaffService
tomcat-webapplication/StaffTracker
tomcat-webapplication/ZipcodeService

$ dysnomia-containers --snapshot

$ dysnomia-snapshots --query-all
mysql-database/rooms/faede34f3bf658884020a31ca98f16503da9a90bf3313cc96adc5c2358c0b054
mysql-database/staff/e9af7042064c33379ba9fe9272f61986b5a85de63c57732f067695e499a3a18f
mysql-database/zipcodes/637faa3e79ec6c2db71ac4023e86f29890e54233ea6592680fd88481725d44a3

$ dysnomia-containers --restore

$ dysnomia-containers --collect-garbage

NixOS integration

dysnomia.enable = true;

dysnomia.containers = {
  mysql-database = {
    mysqlUsername = "root";
    mysqlPassword = "secret";
    mysqlPort = 3306;
  };
  tomcat-webapplication = {
    tomcatPort = 8080;
  };
  ...
};

services.mysql = {
  enable = true;
  package = pkgs.mysql;
  rootPassword = ../configurations/mysqlpw;
};

dysnomia.components = {
  mysql-database = {
    rooms = pkgs.writeTextFile {
      name = "rooms";
      text = ''
        create table room
        ( Room     VARCHAR(10)    NOT NULL,
          Zipcode  VARCHAR(6)     NOT NULL,
          PRIMARY KEY(Room)
        );
'';
    };
    staff = ...
    zipcodes = ...
  };

  tomcat-webapplication = {
    ...
  };
};

Distributed infrastructure state management

{
  test1 = import ./configurations/mysql-tomcat.nix;
  test2 = import ./configurations/empty.nix;
}

$ disnixos-deploy-network network.nix

$ dysnomia-containers --snapshot

$ dysnomia-containers --restore

$ dysnomia-containers --collect-garbage

$ disnixos-snapshot-network network.nix

$ dysnomia-snapshots --query-latest
nixos-configuration/nixos-system-test1-16.03pre-git/4c4751f10648dfbbf8e25c924391e80913c8a6a600f7b481d73cd88ff3d32730

$ cd /var/state/dysnomia/snapshots/$(dysnomia-snapshots --query-latest)
$ find -maxdepth 3 -mindepth 3 -type d
./mysql-database/rooms/faede34f3bf658884020a31ca98f16503da9a90bf3313cc96adc5c2358c0b054
./mysql-database/staff/e9af7042064c33379ba9fe9272f61986b5a85de63c57732f067695e499a3a18f
./mysql-database/zipcodes/637faa3e79ec6c2db71ac4023e86f29890e54233ea6592680fd88481725d44a3

$ disnixos-snapshot-network network.nix

$ disnixos-delete-network-state network.nix

TL;DR: Discussion

dysnomia.enable = true;

services.mysql.enable = true;

dysnomia.components = {
  mysql-database = {
    rooms = pkgs.writeTextFile {
      name = "rooms";
      text = ''
        create table room
        ( Room     VARCHAR(10)    NOT NULL,
          Zipcode  VARCHAR(6)     NOT NULL,
          PRIMARY KEY(Room)
        );
'';
    };
  };
}

$ nixos-rebuild switch

$ dysnomia-containers --deploy

References

imports = [ /home/sander/dysnomia/dysnomia-module.nix ];