In the last couple of months, I have made a number of major changes to the internals of Disnix. As described in a couple of older blog posts, deployment with Disnix is driven by three models each capturing a specific concern:

• The services model specifies the available distributable components, how to construct them (from source code, intra-dependencies and inter-dependencies), and their types so that they can be properly activated or deactivated on the target machines.
• The infrastructure model specifies the available target machines and their relevant deployment properties.
• The distribution model maps services in the service model to target machines in the infrastructure model.

By running the following command-line instruction with the three models as parameters:

$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix

Disnix executes all required activities to get the system deployed, including building, distributing, activating and deactivating services.

I have always described the final step, the activation phase, as deactivating obsolete and activating new services on the target machines. However, this is an over simplification of what really happens.

In reality, Disnix does more than just carrying out an activation step on a target machine -- to get a service activated or deactivated, Disnix invokes Dysnomia that modifies the state of a so-called container hosting a collection of components. As with components, the definition of a container in Dysnomia is deliberately left abstract and represent anything, such as a Java Servlet container (e.g. Apache Tomcat), a DBMS (e.g. MySQL) or the operating system's service manager (e.g. systemd).

So far, these details were always hidden in Disnix and the container mapping was an implicit operation, which I never really liked. Furthermore, there are situations in which you may want to have more control over this mapping.

In this blog post, I will describe my recent modifications and a new notational convention that can be used to treat containers as first-class citizens.

A modified infrastructure model formalism

Previously, a Disnix infrastructure model had the following structure:

{
  test1 = {
    hostname = "test1.example.org";
    tomcatPort = 8080;
    system = "i686-linux";
  };

  test2 = {
    hostname = "test2.example.org";
    tomcatPort = 8080;
    mysqlPort = 3306;
    mysqlUsername = "root";
    mysqlPassword = "admin";
    system = "x86_64-linux";
    numOfCores = 1;
    targetProperty = "hostname";
    clientInterface = "disnix-ssh-client";
  }; 
}

The above Nix expression is an attribute set in which each key corresponds to a target machine in the network and each value is an attribute set containing arbitrary machine properties.

These properties are used for a variety of deployment activities. Disnix made no hard distinction between them -- some properties have a special meaning, but most of them could be freely chosen, yet this does not become clear from the model.

In the new notational convention, the target machine properties have been categorized:

{
  test1 = {
    properties = {
      hostname = "test1.example.org";
    };

    containers = {
      tomcat-webapplication = {
        tomcatPort = 8080;
      };
    };

    system = "i686-linux";
  };

  test2 = {
    properties = {
      hostname = "test2.example.org";
    };

    containers = {
      tomcat-webapplication = {
        tomcatPort = 8080;
      };

      mysql-database = {
        mysqlPort = 3306;
        mysqlUsername = "root";
        mysqlPassword = "admin";
      };
    };

    system = "x86_64-linux";
    numOfCores = 1;
    targetProperty = "hostname";
    clientInterface = "disnix-ssh-client";
  }; 
}

The above expression has a more structured notation:

• The properties attribute refers to arbitrary machine-level properties that are used at build-time and to connect from the coordinator to the target machine.
• The containers attribute set defines the available container services on a target machine and their relevant deployment properties. The container properties are used at build-time and activation time. At activation time, they are passed as parameters to the Dysnomia module that activates a service in the corresponding container.
• The remainder of the target attributes are optional system properties. For example, targetProperty defines which attribute in properties contains the address to connect to the target machine. clientInterface refers to the executable that establishes a remote connection, system defines the system architecture of the target machine (so that services will be correctly built for it), and numOfCores defines how many concurrent activation operations can be executed on the target machine.

As may have become obvious, in the new notation it becomes clear what container services the target machine provides, whereas in the old notation they were hidden.

An alternative distribution model notation

I have also introduced an alternative notation for mappings in the distribution model. A traditional Disnix distribution model typically looks as follows:

{infrastructure}:

{
  ...
  StaffService = [ infrastructure.test2 ];
  StaffTracker = [ infrastructure.test1 infrastructure.test2 ];
}

In the above expression, each attribute name refers to a service in the service model and each value to a list of machines in the infrastructure model.

As explained earlier, besides deploying a service to a machine, a service also gets deployed to a container hosted on the machine, which is not reflected in the distribution model.

When using the above notation, Disnix executes a so-called auto mapping strategy to containers. It simply takes the type attribute from the services model (which is used to determine which Dysnomia module to use that carries out the activation and deactivation steps):

StaffTracker = {
  name = "StaffTracker";
  pkg = customPkgs.StaffTracker;
  dependsOn = {
    inherit GeolocationService RoomService;
    inherit StaffService ZipcodeService;
  };
  type = "tomcat-webapplication";
};

and deploys the service to the container with the same name as the type. For example, all services of type: tomcat-webapplication will be deployed to a container named: tomcat-webapplication (and uses the Dysnomia module named: tomcat-webapplication to activate or deactivate it).

In most cases auto-mapping suffices -- we typically only run one container service on a machine, e.g. one MySQL DBMS, one Apache Tomcat application server. That is why the traditional notation remains the default in Disnix.

However, sometimes it may also be desired to have more control over the container mappings. The new Disnix also supports an alternative and more verbose notation. For example, the following mapping of the StaffTracker service is equivalent to the traditional mapping shown in the previous distribution model:

StaffTracker = {
  targets = [ { target = infrastructure.test1; } ];
};

We can use the alternative notation to control the container mapping, for example:

{infrastructure}:

{
  ...

  StaffService = {
    targets = [
      { target = infrastructure.test1;
        container = "tomcat-production";
      }
    ];
  };
  StaffTracker = {
    targets = [
      { target = infrastructure.test1;
        container = "tomcat-test";
      }
    ];
  };
};

By adding the container attribute to a mapping, we can override the auto mapping strategy and specify the name of the container that we want to deploy to. This alternative notation allows us to deploy to a container whose name does not match the type or to manage networks of machines having multiple instances of the same container deployed.

For example, in the above distribution model, both services are Apache Tomcat web applications. We map StaffService to a container called: tomcat-production and StaffTracker to a container called: tomcat-test. Both containers are hosted on the same machine: test1.

A modified formalism to refer to inter-dependency parameters

As a consequence of modifying the infrastructure and distribution model notations, referring to inter-dependency parameters in Disnix expressions also slightly changed:

{stdenv, StaffService}:
{staff}:

let
  contextXML = ''
<Context>
<Resource name="jdbc/StaffDB" auth="Container"
        type="javax.sql.DataSource"
        maxActivate="100" maxIdle="30" maxWait="10000"
        username="${staff.target.container.mysqlUsername}"
        password="${staff.target.container.mysqlPassword}"
        driverClassName="com.mysql.jdbc.Driver"
        url="jdbc:mysql://${staff.target.properties.hostname}:${toString (staff.target.container.mysqlPort)}/${staff.name}?autoReconnect=true" />
</Context>
'';
in
stdenv.mkDerivation {
  name = "StaffService";
  buildCommand = ''
    mkdir -p $out/conf/Catalina
    cat > $out/conf/Catalina/StaffService.xml <<EOF
    ${contextXML}
    EOF
    ln -sf ${StaffService}/webapps $out/webapps
'';
}

The above example is a Disnix expression that configures the StaffService service. The StaffService connects to a remote MySQL database (named: staff) which is provided as an inter-dependency parameter. The Disnix expression uses the properties of the inter-dependency parameter to configure a so called context XML file which Apache Tomcat uses to establish a (remote) JDBC connection so that the web service can connect to it.

Previously, each inter-dependency parameter provided a targets sub attribute referring to targets in the infrastructure model to which the inter-dependency has been mapped in the distribution model. Because it is quite common to map to a single target only, there is also a target sub attribute that refers to the first element for convenience.

In the new Disnix, the targets now refer to container mappings instead of machine mappings and implement a new formalism to reflect this:

• The properties sub attribute refers to the machine level properties in the infrastructure model
• The container sub attribute refers to the container properties to which the inter-dependency has been deployed.

As can be observed in the expression shown above, both sub attributes are used in the above expression to allow the service to connect to the remote MySQL database.

Visualizing containers

Besides modifying the notational conventions and the underlying deployment mechanisms, I have also modified disnix-visualize to display containers. The following picture shows an example:



In the above picture, the light grey boxes denote machines, the dark grey boxes denote containers, the ovals services and the arrows inter-dependency relationships. In my opinion, these new visualizations are much more intuitive -- I still remember that in an old blog post that summarizes my PhD thesis I used a hand-drawn diagram to illustrate why deployments of service-oriented systems were complicated. In this diagram I already showed containers, yet in the visualizations generated by disnix-visualize they were missing. Now finally, this mismatch has been removed from the tooling.

(As a sidenote: it is still possible to generate the classic non-containerized visualizations by providing the: --no-containers command-line option).

Capturing the infrastructure model from the machines' Dysnomia container configuration files

The new notational conventions also makes it possible to more easily implement yet another use case. As explained in an earlier blog post, when it is desired to deploy services with Disnix, we need predeployed machines running Nix, Dysnomia and Disnix installed and a number of container services (such as MySQL and Apache Tomcat) first.

After deploying the machines, we must hand-write an infrastructure model reflecting their properties. Hand writing infrastructure models is sometimes tedious and error prone. In my previous blog post, I have shown that it is possible to automatically generate Dysnomia container configuration files from NixOS configurations that capture properties of entire machine configurations.

We can now also do the opposite: generating an expression of a machines' Dysnomia container configuration files and compose an infrastructure model from it. This takes away the majority of the burden of handwriting infrastructure models.

For example, we can write a Dysnomia-enabled NixOS configuration:

{config, pkgs, ...}:

{
  services = {
    openssh.enable = true;

    mysql = {
      enable = true;
      package = pkgs.mysql;
      rootPassword = ../configurations/mysqlpw;
    };

    tomcat = {
      enable = true;
      commonLibs = [ "${pkgs.mysql_jdbc}/share/java/mysql-connector-java.jar" ];
      catalinaOpts = "-Xms64m -Xmx256m";
    };
  };

  dysnomia = {
    enable = true;
    enableAuthentication = true;
    properties = {
      hostname = config.networking.hostName;
      mem = "$(grep 'MemTotal:' /proc/meminfo | sed -e 's/kB//' -e 's/MemTotal://' -e 's/ //g')";
    };
  };
}

The above NixOS configuration deploys two container services: MySQL and Apache Tomcat. Furthermore, it defines some non-functional machine-level properties, such as the hostname and the amount of RAM (mem) the machine has (which is composed dynamically by consulting the kernel's /proc filesystem).

As shown in the previous blog post, when deploying the above configuration with:

$ nixos-rebuild switch

The Dysnomia NixOS module automatically composes the /etc/dysnomia/properties and /etc/dysnomia/containers configuration files. When running the following command:

$ dysnomia-containers --generate-expr
{
  properties = {
"hostname" = "test1";
"mem" = "1023096";
"supportedTypes" = [
"mysql-database"
"process"
"tomcat-webapplication"
    ];
"system" = "x86_64-linux";
  };
  containers = {
    mysql-database = {
"mysqlPassword" = "admin";
"mysqlPort" = "3306";
"mysqlUsername" = "root";
    };
    tomcat-webapplication = {
"tomcatPort" = "8080";
    };
  };
}

Dysnomia generates a Nix expression of the general properties and container configuration files.

We can do the same operation in a network of machines by running the disnix-capture-infra tool. First, we need to write a very minimal infrastructure model that only captures the connectivity attributes:

{
  test1.properties.hostname = "test1";
  test2.properties.hostname = "test2";
}

When running:

$ disnix-capture-infra infrastructure-basic.nix
{
  test1 = {
    properties = {
"hostname" = "test1";
"mem" = "1023096";
"supportedTypes" = [
"mysql-database"
"process"
"tomcat-webapplication"
      ];
"system" = "x86_64-linux";
    };
    containers = {
      mysql-database = {
"mysqlPassword" = "admin";
"mysqlPort" = "3306";
"mysqlUsername" = "root";
      };
      tomcat-webapplication = {
"tomcatPort" = "8080";
      };
    };
  };
  test2 = ...
}

Disnix captures the configurations of all machines in the basic infrastructure model and returns an augmented infrastructure model containing all its properties.

(As a sidenote: disnix-capture-infra is not the only infrastructure model generator I have developed. In the self-adaptive adaptive deployment framework built on top of Disnix, I have developed an Avahi-based discovery service that can also generate infrastructure models. It is also more powerful (but quite hacky and immature) because it dynamically discovers the machines in the network, so it does not require a basic infrastructure model to be written. Moreover, it automatically responds to events when a machine's configuration changes.

I have modified the Avahi-based discovery tool to use Dysnomia's expression generator as well.

Also, the DisnixOS toolset can generated infrastructure models from networked NixOS configurations).

Discussion

In this blog post, I have described the result of a number of major internal changes to Disnix that make the containers concept a first class citizen. Fortunately, from an external perspective the changes are minor, but still backwards incompatible -- we must follow a new convention for the infrastructure model and refer to the target properties of inter-dependency parameters in a slightly different way.

In return you will get:

• A more intuitive notation. As explained, we do not only deploy to a machine, but also to a container hosted on the machine. Now the deployment models and corresponding visualizations reflect this concept.
• More control and power. We can deploy to multiple containers of the same type on the same machine, e.g. we can have two MySQL DBMSes on the same machine.
• More correctness. When activating or deactivating a service all infrastructure properties were propagated as parameters to the corresponding Dysnomia module. Why does mysql-database module needs to know about a postgresql-database and vice versa? Now Dysnomia modules only get to know what they need to know.
• Discovery. We can generate the infrastructure model from Dysnomia container configuration hosted on the target machines with relative ease.

A major caveat is that deployment planning (implemented in the Dynamic Disnix framework) can also potentially be extended from machine-level to container-level.

At the moment, I did make these modifications. This means that Dynamic Disnix can still generate distribution models, but only on machine level. As a consequence, Dynamic Disnix only allows a user to refer to a target's machine-level properties (i.e. the properties attribute in the infrastructure model) for deployment planning purposes, and not to any container-specific properties.

Container-level deployment planning is also something I intend to support at some point in the future.

Availability

The new notational conventions and containers concepts are part of the development version of Disnix and will become available in the next release. Moreover, I have modified the Disnix examples to use the new notations.

As explained in an earlier blog post, Disnix is a service deployment tool that can only be used after a collection of machines have been predeployed providing a number of container services, such as a service manager (e.g. systemd), a DBMS (e.g. MySQL) or an application server (e.g. Apache Tomcat).

To deploy these machines, we need an external solution. Some solutions are:

• Manual installations requiring somebody to obtain a few machines, manually installing operating systems (e.g. a Linux distribution), and finally installing all required software packages, such as Nix, Dysnomia, Disnix and any additional container services. Manually configuring a machine is typically tedious, time consuming and error prone.
• NixOps. NixOps is capable of automatically instantiating networks of virtual machines in the cloud (such as Amazon EC2) and deploying entire NixOS system configurations to them. These NixOS configurations can be used to automatically deploy Dysnomia, Disnix and any container service that we need. A drawback is that NixOps is NixOS-based and not really useful if you want to deploy services to machines running different kinds of operating systems.
• disnixos-deploy-network. In a Disnix-context, services are basically undefined units of deployment, and we can also automatically deploy entire NixOS configurations to target machines as services. A major drawback of this approach is that we require predeployed machines running Disnix first.

Although there are several ways to manage the underlying infrastructure of services, they are basically all or nothing solutions with regards to automation -- we either have to manually deploy entire machine configurations ourselves or we are stuck to a NixOS-based solution that completely automates it.

In some scenarios (e.g. when it is desired to deploy services to non-Linux operating systems), the initial deployment phase becomes quite tedious. For example, it took me quite a bit of effort to set up the heterogeneous network deployment demo I have given at NixCon2015.

In this blog post, I will describe an approach that serves as an in-between solution -- since services in a Disnix-context can be (almost) any kind of deployment unit, we can also use Disnix to deploy container configurations as services. These container services can also be deployed to non-NixOS systems, which means that we can alleviate the effort in setting up the initial target system configurations where Disnix can deploy services to.

Deploying containers as services with Disnix

As with services, containers in a Disnix-context could take any form. For example, in addition to MySQL databases (that we can deploy as services with Disnix), we can also deploy the corresponding container: the MySQL DBMS server, as a Disnix service:

{ stdenv, mysql, dysnomia
, name ? "mysql-database"
, mysqlUsername ? "root", mysqlPassword ? "secret"
, user ? "mysql-database", group ? "mysql-database"
}:

stdenv.mkDerivation {
  inherit name;

  buildCommand = ''
    mkdir -p $out/bin

    # Create wrapper script
    cat > $out/bin/wrapper <<EOF
    #! ${stdenv.shell} -e

    case "\$1" in
        activate)
            # Create group, user and the initial database if it does not exists
            # ...

            # Run the MySQL server
            ${mysql}/bin/mysqld_safe --user=${user} --datadir=${dataDir} --basedir=${mysql} --pid-file=${pidDir}/mysqld.pid &

            # Change root password
            # ...
            ;;
        deactivate)
            ${mysql}/bin/mysqladmin -u ${mysqlUsername} -p "${mysqlPassword}" -p shutdown

            # Delete the user and group
            # ...
            ;;
    esac
    EOF

    chmod +x $out/bin/wrapper

    # Add Dysnomia container configuration file for the MySQL DBMS
    mkdir -p $out/etc/dysnomia/containers

    cat > $out/etc/dysnomia/containers/${name} <<EOF
    mysqlUsername="${mysqlUsername}"
    mysqlPassword="${mysqlPassword}"
    EOF

    # Copy the Dysnomia module that manages MySQL databases
    mkdir -p $out/etc/dysnomia/modules
    cp ${dysnomia}/libexec/dysnomia/mysql-database $out/etc/dysnomia/modules
'';
}

The above code fragment is a simplified Disnix expression that can be used to deploy a MySQL server. The above expression produces a wrapper script, which carries out a set of deployment activities invoked by Disnix:

• On activation, the wrapper script starts the MySQL server by spawning the mysqld_safe daemon process in background mode. Before starting the daemon, it also intitializes some of the server's state, such as creating user accounts under which the daemon runs and setting up the system database if it does not exists (these steps are left out of the example for simplicity reasons).
• On deactivation it shuts down the MySQL server and removes some of the attached state, such as the user accounts.

Besides composing a wrapper script, we must allow Dysnomia (and Disnix) to deploy databases as Disnix services to the MySQL server that we have just deployed:

• We generate a Dysnomia container configuration file with the MySQL server settings to allow a database (that gets deployed as a service) to know what credentials it should use to connect to the database.
• We bundle a Dysnomia plugin module that implements the deployment activities for MySQL databases, such as activation and deactivation. Because Dysnomia offers this plugin as part of its software distribution, we make a copy of it, but we could also compose our own plugin from scratch.

With the earlier shown Disnix expression, we can define the MySQL server as a service in a Disnix services model:

mysql-database = {
  name = "mysql-database";
  pkg = customPkgs.mysql-database;
  dependsOn = {};
  type = "wrapper";
};

and distribute it to a target machine in the network by adding an entry to the distribution model:

mysql-database = [ infrastructure.test2 ];

Configuring Disnix and Dysnomia

Once we have deployed containers as Disnix services, Disnix (and Dysnomia) must know about their availability so that we can deploy services to these recently deployed containers.

Each time Disnix has successfully deployed a configuration, it generates Nix profiles on the target machines in which the contents of all services can be accessed from a single location. This means that we can simply extend Dysnomia's module and container search paths:

export DYSNOMIA_MODULES_PATH=$DYSNOMIA_MODULES_PATH:/nix/var/nix/profiles/disnix/containers/etc/dysnomia/modules
export DYSNOMIA_CONTAINERS_PATH=$DYSNOMIA_CONTAINERS_PATH:/nix/var/nix/profiles/disnix/containers/etc/dysnomia/containers

with the paths to the Disnix profiles that have containers deployed.

A simple example scenario

I have modified the Java variant of the ridiculous Disnix StaffTracker example to support a deployment scenario with containers as Disnix services.

First, we need to start with a collection of machines having a very basic configuration without any additional containers. The StaffTracker package contains a bare network configuration that we can deploy with NixOps, as follows:

$ nixops create ./network-bare.nix ./network-virtualbox.nix -d vbox
$ nixops deploy -d vbox

By configuring the following environment variables, we can connect Disnix to the machines in the network that we have just deployed with NixOps:

$ export NIXOPS_DEPLOYMENT=vbox
$ export DISNIX_CLIENT_INTERFACE=disnix-nixops-client

We can write a very simple bootstrap infrastructure model (infrastructure-bootstrap.nix), to dynamically capture the configuration of the target machines:

{
  test1.properties.hostname = "test1";
  test2.properties.hostname = "test2";
}

Running the following command:

$ disnix-capture-infra infrastructure-bootstrap.nix > infrastructure-bare.nix

yields an infrastructure model (infrastructure-containers.nix) that may have the following structure:

{
"test1" = {
    properties = {
"hostname" = "test1";
"system" = "x86_64-linux";
    };
    containers = {
      process = {
      };
      wrapper = {
      };
    };
"system" = "x86_64-linux";
  };
"test2" = {
    properties = {
"hostname" = "test2";
"system" = "x86_64-linux";
    };
    containers = {
      process = {
      };
      wrapper = {
      };
    };
"system" = "x86_64-linux";
  };
}

As may be observed in the captured infrastructure model shown above, we have a very minimal configuration only hosting the process and wrapper containers, that integrate with host system's service manager, such as systemd.

We can deploy a Disnix configuration having Apache Tomcat and the MySQL DBMS as services, by running:

$ disnix-env -s services-containers.nix \
  -i infrastructure-bare.nix \
  -d distribution-containers.nix \
  --profile containers

Note that we have provided an extra parameter to Disnix: --profile to isolate the containers from the default deployment environment. If the above command succeeds, we have a deployment architecture that looks as follows:


Both machines have Apache Tomcat deployed as a service and machine test2 also runs a MySQL server.

When capturing the target machines' configurations again:

$ disnix-capture-infra infrastructure-bare.nix > infrastructure-containers.nix

we will receive an infrastructure model (infrastructure-containers.nix) that may have the following structure:

{
"test1" = {
    properties = {
"hostname" = "test1";
"system" = "x86_64-linux";
    };
    containers = {
      tomcat-webapplication = {
"tomcatPort" = "8080";
      };
      process = {
      };
      wrapper = {
      };
    };
"system" = "x86_64-linux";
  };
"test2" = {
    properties = {
"hostname" = "test2";
"system" = "x86_64-linux";
    };
    containers = {
      mysql-database = {
"mysqlUsername" = "root";
"mysqlPassword" = "secret";
"mysqlPort" = "3306";
      };
      tomcat-webapplication = {
"tomcatPort" = "8080";
      };
      process = {
      };
      wrapper = {
      };
    };
"system" = "x86_64-linux";
  };
}

As may be observed in the above infrastructure model, both machines provide a tomcat-webapplication container exposing the TCP port number that the Apache Tomcat server has been bound to. Machine test2 exposes the mysql-database container with its connectivity settings.

We can now deploy the StaffTracker system (that consists of multiple MySQL databases and Apache Tomcat web applications) by running:

$ disnix-env -s services.nix \
  -i infrastructure-containers.nix \
  -d distribution.nix \
  --profile services

Note that I use a different --profile parameter, to tell Disnix that the StaffTracker components belong to a different environment than the containers. If I would use --profile containers again, Disnix will undeploy the previously shown containers environment with the MySQL DBMS and Apache Tomcat and deploy the databases and web applications, which will lead to a failure.

If the above command succeeds, we have the following deployment architecture:


The result is that we have all the service components of the StaffTracker example deployed to containers that are also deployed by Disnix.

An advanced example scenario: multi-containers

We could go even one step beyond the example I have shown in the previous section. In the first example, we deploy no more than one instance of each container to a machine in the network -- this is quite common, as it rarely happens that you want to run two MySQL or Apache Tomcat servers on a single machine. Most Linux distributions (including NixOS) do not support deploying multiple instances of system services out of the box.

However, with a few relatively simple modifications to the Disnix expressions of the MySQL DBMS and Apache Tomcat services, it becomes possible to allow multiple instances to co-exist on the same machine. What we basically have to do is identifying the conflicting runtime resources, making them configurable and changing their values in such a way that they no longer conflict.

{ stdenv, mysql, dysnomia
, name ? "mysql-database"
, mysqlUsername ? "root", mysqlPassword ? "secret"
, user ? "mysql-database", group ? "mysql-database"
, dataDir ? "/var/db/mysql", pidDir ? "/run/mysqld"
, port ? 3306
}:

stdenv.mkDerivation {
  inherit name;

  buildCommand = ''
    mkdir -p $out/bin

    # Create wrapper script
    cat > $out/bin/wrapper <<EOF
    #! ${stdenv.shell} -e

    case "\$1" in
        activate)
            # Create group, user and the initial database if it does not exists
            # ...

            # Run the MySQL server
            ${mysql}/bin/mysqld_safe --port=${toString port} --user=${user} --datadir=${dataDir} --basedir=${mysql} --pid-file=${pidDir}/mysqld.pid --socket=${pidDir}/mysqld.sock &

            # Change root password
            # ...
            ;;
        deactivate)
            ${mysql}/bin/mysqladmin --socket=${pidDir}/mysqld.sock -u ${mysqlUsername} -p "${mysqlPassword}" -p shutdown

            # Delete the user and group
            # ...
            ;;
    esac
    EOF

    chmod +x $out/bin/wrapper

    # Add Dysnomia container configuration file for the MySQL DBMS
    mkdir -p $out/etc/dysnomia/containers

    cat > $out/etc/dysnomia/containers/${name} <<EOF
    mysqlUsername="${mysqlUsername}"
    mysqlPassword="${mysqlPassword}"
    mysqlPort=${toString port}
    mysqlSocket=${pidDir}/mysqld.sock
    EOF

    # Copy the Dysnomia module that manages MySQL databases
    mkdir -p $out/etc/dysnomia/modules
    cp ${dysnomia}/libexec/dysnomia/mysql-database $out/etc/dysnomia/modules
'';
}

For example, I have revised the MySQL server Disnix expression with additional parameters that change the TCP port the service binds to, the UNIX domain socket that is used by the administration utilities and the filesystem location where the databases are stored. Moreover, these additional configuration properties are also exposed by the Dysnomia container configuration file.

These additional parameters make it possible to define multiple variants of container services in the services model:

{distribution, invDistribution, system, pkgs}:

let
  customPkgs = import ../top-level/all-packages.nix {
    inherit system pkgs;
  };
in
rec {
  mysql-production = {
    name = "mysql-production";
    pkg = customPkgs.mysql-production;
    dependsOn = {};
    type = "wrapper";
  };

  mysql-test = {
    name = "mysql-test";
    pkg = customPkgs.mysql-test;
    dependsOn = {};
    type = "wrapper";
  };

  tomcat-production = {
    name = "tomcat-production";
    pkg = customPkgs.tomcat-production;
    dependsOn = {};
    type = "wrapper";
  };

  tomcat-test = {
    name = "tomcat-test";
    pkg = customPkgs.tomcat-test;
    dependsOn = {};
    type = "wrapper";
  };
}

I can, for example, map two MySQL DBMS instances and the two Apache Tomcat servers to the same machines in the distribution model:

{infrastructure}:

{
  mysql-production = [ infrastructure.test1 ];
  mysql-test = [ infrastructure.test1 ];
  tomcat-production = [ infrastructure.test2 ];
  tomcat-test = [ infrastructure.test2 ];
}

Deploying the above configuration:

$ disnix-env -s services-multicontainers.nix \
  -i infrastructure-bare.nix \
  -d distribution-multicontainers.nix \
  --profile containers

yields the following deployment architecture:


As can be observed, we have two instances of the same container hosted on the same machine. When capturing the configuration:

$ disnix-capture-infra infrastructure-bare.nix > infrastructure-multicontainers.nix

we will receive a Nix expression that may look as follows:

{
"test1" = {
    properties = {
"hostname" = "test1";
"system" = "x86_64-linux";
    };
    containers = {
      mysql-production = {
"mysqlUsername" = "root";
"mysqlPassword" = "secret";
"mysqlPort" = "3306";
"mysqlSocket" = "/run/mysqld-production/mysqld.sock";
      };
      mysql-test = {
"mysqlUsername" = "root";
"mysqlPassword" = "secret";
"mysqlPort" = "3307";
"mysqlSocket" = "/run/mysqld-test/mysqld.sock";
      };
      process = {
      };
      wrapper = {
      };
    };
"system" = "x86_64-linux";
  };
"test2" = {
    properties = {
"hostname" = "test2";
"system" = "x86_64-linux";
    };
    containers = {
      tomcat-production = {
"tomcatPort" = "8080";
"catalinaBaseDir" = "/var/tomcat-production";
      };
      tomcat-test = {
"tomcatPort" = "8081";
"catalinaBaseDir" = "/var/tomcat-test";
      };
      process = {
      };
      wrapper = {
      };
    };
"system" = "x86_64-linux";
  };
}

In the above expression, there are two instances of MySQL and Apache Tomcat deployed to the same machine. These containers have their resources configured in such a way that they do not conflict. For example, both MySQL instances bind to a different TCP ports (3306 and 3307) and different UNIX domain sockets (/run/mysqld-production/mysqld.sock and /run/mysqld-test/mysqld.sock).

After deploying the containers, we can also deploy the StaffTracker components (databases and web applications) to them. As described in my previous blog post, we can use an alternative (and more verbose) notation in the distribution model to directly map services to containers:

{infrastructure}:

{
  GeolocationService = {
    targets = [
      { target = infrastructure.test2; container = "tomcat-test"; }
    ];
  };
  RoomService = {
    targets = [
      { target = infrastructure.test2; container = "tomcat-production"; }
    ];
  };
  StaffService = {
    targets = [
      { target = infrastructure.test2; container = "tomcat-test"; }
    ];
  };
  StaffTracker = {
    targets = [
      { target = infrastructure.test2; container = "tomcat-production"; }
    ];
  };
  ZipcodeService = {
    targets = [
      { target = infrastructure.test2; container = "tomcat-test"; }
    ];
  };
  rooms = {
    targets = [
      { target = infrastructure.test1; container = "mysql-production"; }
    ];
  };
  staff = {
    targets = [
      { target = infrastructure.test1; container = "mysql-test"; }
    ];
  };
  zipcodes = {
    targets = [
      { target = infrastructure.test1; container = "mysql-production"; }
    ];
  };
}

As may be observed in the distribution model above, we deploy databases and web application to both instances that are hosted on the same machine.

We can deploy the services of which the StaffTracker consists, as follows:

$ disnix-env -s services.nix \
  -i infrastructure-multicontainers.nix \
  -d distribution-advanced.nix \
  --profile services

and the result is the following deployment architecture:


As may be observed in the picture above, we now have a running StaffTracker system that uses two MySQL and two Apache Tomcat servers on one machine. Isn't it awesome? :-)

Conclusion

In this blog post, I have demonstrated an approach in which we deploy containers as services with Disnix. Containers serve as potential deployment targets for other Disnix services.

Previously, we only had NixOS-based solutions to manage the configuration of containers, which makes using Disnix on other platforms than NixOS painful, as the containers had to be deployed manually. The approach described in this blog post serves as an in-between solution.

In theory, the process in which we deploy containers as services first followed by the "actual" services, could be generalized and extended into a layered service deployment model, with a new tool automating the process and declarative specifications capturing the properties of the layers.

However, I have decided not to implement this new model any time soon for practical reasons -- in nearly all of my experiences with service deployment, I have almost never encountered the need to have more than two layers supported. The only exception I can think of is the deployment of Axis2 web services to an Axis2 container -- the Axis2 container is a Java web application that must be deployed to Apache Tomcat first, which in turn requires the presence of the Apache Tomcat server.

Availability

I have integrated the two container deployment examples into the Java variant of the StaffTracker example.

The new concepts described in this blog post are part of the development version of Disnix and will become available in the next release.

Recently, I was asked whether it is possible to use Disnix as a tool for remote package deployment.

As described in a number of earlier blog posts, Disnix's primary purpose is not remote (or distributed) package management, but deploying systems that can be decomposed into services to networks of machines. To deploy these kinds of systems, Disnix executes all required deployment activities, including building services from source code, distributing them to target machines in the network and activating or deactivating them.

However, a service deployment process is basically a superset of an "ordinary" package deployment process. In this blog post, I will describe how we can do remote package deployment by instructing Disnix to only use a relevant subset of features.

Specifying packages as services

In the Nix packages collection, it is a common habit to write each package specification as a function in which the parameters denote the (local) build and runtime dependencies (something that Disnix's manual refers to as intra-dependencies) that the package needs. The remainder of the function describes how to build the package from source code and its provided dependencies.

Disnix has adopted this habit and extended this convention to services. The main difference between Nix package expressions and Disnix service expressions is that the latter also take inter-dependencies into account that refer to run-time dependencies on services that may have been deployed to other machines in the network. For services that have no inter-dependencies, a Disnix expression is identical to an ordinary package expression.

This means that, for example, an expression for a package such as the Midnight Commander is also a valid Disnix service with no inter-dependencies:

{ stdenv, fetchurl, pkgconfig, glib, gpm, file, e2fsprogs
, libX11, libICE, perl, zip, unzip, gettext, slang
}:

stdenv.mkDerivation {
  name = "mc-4.8.12";

  src = fetchurl {
    url = http://www.midnight-commander.org/downloads/mc-4.8.12.tar.bz2;
    sha256 = "15lkwcis0labshq9k8c2fqdwv8az2c87qpdqwp5p31s8gb1gqm0h";
  };

  buildInputs = [ pkgconfig perl glib gpm slang zip unzip file gettext
      libX11 libICE e2fsprogs ];

  meta = {
    description = "File Manager and User Shell for the GNU Project";
    homepage = http://www.midnight-commander.org;
    license = "GPLv2+";
    maintainers = [ stdenv.lib.maintainers.sander ];
  };
}

Composing packages locally

Package and service expressions are functions that do not specify the versions or variants of the dependencies that should be used. To allow services to be deployed, we must compose them by providing the desired versions or variants of the dependencies as function parameters.

As with ordinary Nix packages, Disnix has also adopted this convention for services. In addition, we have to compose a Disnix service twice -- first its intra-dependencies and later its inter-dependencies.

Intra-dependency composition in Disnix is done in a similar way as in the Nix packages collection:

{pkgs, system}:

let
  callPackage = pkgs.lib.callPackageWith (pkgs // self);

  self = {
    pkgconfig = callPackage ./pkgs/pkgconfig { };

    gpm = callPackage ./pkgs/gpm { };

    mc = callPackage ./pkgs/mc { };
  };
in
self

The above expression (custom-packages.nix) composes the Midnight Commander package by providing its intra-dependencies as function parameters. The third attribute (mc) invokes a function named: callPackage {} that imports the previous package expression and automatically provides the parameters having the same names as the function parameters.

The callPackage { } function first consults the self attribute set (that composes some of Midnight Commander's dependencies as well, such as gpm and pkgconfig) and then any package from the Nixpkgs repository.

Writing a minimal services model

Previously, we have shown how to build packages from source code and its dependencies, and how to compose packages locally. For the deployment of services, more information is needed. For example, we need to compose their inter-dependencies so that services know how to reach them.

Furthermore, Disnix's end objective is to get a running service-oriented system and carries out extra deployment activities for services to accomplish this, such as activation and deactivation. The latter two steps are executed by a Dysnomia plugin that is determined by annotating a service with a type attribute.

For package deployment, specifying these extra attributes and executing these remaining activities are in principle not required. Nonetheless, we still need to provide a minimal services model so that Disnix knows which units can be deployed.

Exposing the Midnight Commander package as a service, can be done as follows:

{pkgs, system, distribution, invDistribution}:

let
  customPkgs = import ./custom-packages.nix {
    inherit pkgs system;
  };
in
{
  mc = {
    name = "mc";
    pkg = customPkgs.mc;
    type = "package";
  };
}

In the above expression, we import our intra-dependency composition expression (custom-packages.nix), and we use the pkg sub attribute to refer to the intra-dependency composition of the Midnight Commander. We annotate the Midnight Commander service with a package type to instruct Disnix that no additional deployment steps need to be performed beyond the installation of the package, such activation or deactivation.

Since the above pattern is common to all packages, we can also automatically generate services for any package in the composition expression:

{pkgs, system, distribution, invDistribution}:

let
  customPkgs = import ./custom-packages.nix {
    inherit pkgs system;
  };
in
pkgs.lib.mapAttrs (name: pkg: {
  inherit name pkg;
  type = "package";
}) customPkgs

The above services model exposes all packages in our composition expression as a service.

Configuring the remote machine's search paths

With the services models shown in the previous section, we have all ingredients available to deploy packages with Disnix. To allow users on the remote machines to conveniently access their packages, we must add Disnix' Nix profile to the PATH of a user on the remote machines:

$ export PATH=/nix/var/nix/profiles/disnix/default/bin:$PATH

When using NixOS, this variable can be extended by adding the following line to /etc/nixos/configuration.nix:

environment.variables.PATH = [ "/nix/var/nix/profiles/disnix/default/bin" ];

Deploying packages with Disnix

In addition to a services model, Disnix needs an infrastructure and distribution model to deploy packages. For example, we can define an infrastructure model that may look as follows:

{
  test1.properties.hostname = "test1";
  test2 = {
    properties.hostname = "test2";
    system = "x86_64-darwin";
  };
}

The above infrastructure model describes two machines that have hostname test1 and test2. Furthermore, machine test2 has a specific system architecture: x86_64-darwin that corresponds to a 64-bit Intel-based Mac OS X.

We can distribute package to these two machines with the following distribution model:

{infrastructure}:

{
  gpm = [ infrastructure.test1 ];
  pkgconfig = [ infrastructure.test2 ];
  mc = [ infrastructure.test1 infrastructure.test2 ];
}

In the above distribution model, we distribute package gpm to machine test1, pkgconfig to machine test2 and mc to both machines.

When running the following command-line instruction:

$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix

Disnix executes all activities to get the packages in the distribution model deployed to the machines, such as building them from source code (including its dependencies), and distributing their dependency closures to the target machines.

Because machine test2 may have a different system architecture as the coordinator machine responsible for carrying out the deployment, Disnix can use Nix's delegation mechanism to forward a build to a machine that is capable of doing it.

Alternatively, packages can also be built on the target machines through Disnix:

$ disnix-env --build-on-targets \
  -s services.nix -i infrastructure.nix -d distribution.nix

After the deployment above command-line instructions have succeeded, we should be able to start the Midnight Commander on any of the target machines, by running:

$ mc

Deploying any package from the Nixpkgs repository

Besides deploying a custom set of packages, it is also possible to use Disnix to remotely deploy any package in the Nixpkgs repository, but doing so is a bit tricky.

The main challenge lies in the fact that the Nix packages set is a nested set of attributes, whereas Disnix expects services to be addressed in one attribute set only. Fortunately, the Nix expression language and Disnix models are flexible enough to implement a solution. For example, we can define a distribution model as follows:

{infrastructure}:

{
  mc = [ infrastructure.test1 ];
  git = [ infrastructure.test1 ];
  wget = [ infrastructure.test1 ];
"xlibs.libX11" = [ infrastructure.test1 ];
}

Note that we use a dot notation: xlibs.libX11 as an attribute name to refer to libX11 that can only be referenced as a sub attribute in Nixpkgs.

We can write a services model that uses the attribute names in the distribution model to refer to the corresponding package in Nixpkgs:

{pkgs, system, distribution, invDistribution}:

pkgs.lib.mapAttrs (name: targets:
  let
    attrPath = pkgs.lib.splitString "." name;
  in
  { inherit name;
    pkg = pkgs.lib.attrByPath attrPath
      (throw "package: ${name} cannot be referenced in the package set")
      pkgs;
    type = "package";
  }
) distribution

With the above service model we can deploy any Nix package to any remote machine with Disnix.

Multi-user package management

Besides supporting single user installations, Nix also supports multi-user installations in which every user has its own private Nix profile with its own set of packages. With Disnix we can also manage multiple profiles. For example, by adding the --profile parameter, we can deploy another Nix profile that, for example, contains a set of packages for the user: sander:

$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix \
  --profile sander

The user: sander can access its own set of packages by setting the PATH environment variable to:

$ export PATH=/nix/var/nix/profiles/disnix/sander:$PATH

Conclusion

Although Disnix has not been strictly designed for this purpose, I have described in this blog post how Disnix can be used as a remote package deployer by using a relevant subset of Disnix features.

Moreover, I now consider the underlying Disnix primitives to be mature enough. As such, I am announcing the release of Disnix 0.6!

Acknowledgements

I gained the inspiration for writing this blog post from discussions with Matthias Beyer on the #nixos IRC channel.

As may have become obvious by reading some of my previous blog posts, I am frequently using JavaScript for a variety of programming purposes. Although JavaScript was originally conceived as a programming language for use in web browsers (mainly to make web pages more interactive), it is also becoming increasingly more popular in environments outside the browser, such as Node.js, a runtime environment for developing scalable network applications and Appcelerator Titanium, an environment to develop cross-platform mobile applications.

Apart from the fact these environments share a common programming language -- JavaScript -- and a number of basic APIs that come with the language, they all have their own platform-specific APIs to implement most of an application's basic functionality.

Moreover, they have their own ecosystem of third-party software packages. For example, in Node.js the NPM package manager is the ubiquitous way of publishing and obtaining software. For web browsers, bower can be used, although its adoption is not as widespread as NPM.

Because of these differences, reuse between JavaScript environments is not optimal, in particular for packages that have dependencies on functionality that is not part of JavaScript's core API.

In this blog post, I will describe our experiences with porting the simple-xmpp from the Node.js ecosystem to Titanium. This library has dependencies on Node.js-specific APIs, but with our porting strategy we were able to integrate it in our Titanium app without making any modifications to the original package.

Motivation: Adding chat functionality

Not so long ago, me and my colleagues have been looking into extending our mobile app product-line with chat functionality. As described in an earlier blog post, we use Titanium for developing our mobile apps and one of my responsibilities is automating their builds with Nix (and Hydra, the Nix-based continuous integration server).

Developing chat functionality is quite complex, and requires one to think about many concerns, such as the user-experience, security, reliability and scalability. Instead of developing a chat infrastructure from scratch (which would be much too costly for us), we have decided to adopt the XMPP protocol, for the following reasons:

• Open standard. Everyone is allowed to make software implementing aspects of the XMPP standard. There are multiple server implementations and many client libraries available, in many programming languages including JavaScript.
• Decentralized. Users do not have to connect to a single server -- a server can relay messages to users connected to another server. A decentralized approach is good for reliability and scalability.
• Web-based. The XMPP protocol is built on technologies that empower the web (XML and HTTP). The fact that HTTP is used as a transport protocol, means that we can also support clients that are behind a proxy server.
• Mature. XMPP has been in use for a quite some time and has some very prominent users. Currently, Sony uses it to enrich the PlayStation platform with chat functionality. In the past, it was also used as the basis for Google and Facebook's chat infrastructure.

Picking a server implementation was not too hard, as ejabberd was something I had experience with in my previous job (and as an intern at Philips Research) -- it supports all the XMPP features we need, and has proven to be very mature.

Unfortunately, for Titanium, there was no module available that implements XMPP client functionality, except for an abandoned project named titanium-xmpp that is no longer supported, and no longer seems to work with any recent versions of Titanium.

Finding a suitable library

As there was no working XMPP client module available for Titanium and we consider developing such a client for Titanium from scratch too costly, we first tried to fix titanium-xmpp, but it turned out that too many things were broken. Moreover, it used all kinds of practices (such as an old fashioned way of module loading through Ti.include()) that have been deprecated a long time ago.

Then we tried porting other JavaScript-based libraries to Titanium. The first candidate was strophe.js which is mainly browser-oriented (and can be used in Node.js through phantomjs, an environment providing a non-interactive web technology stack), but had too many areas that had to be modified and browser-specific APIs that require substitutes.

Finally, I discovered node-xmpp, an XMPP framework for Node.js that has a very modular architecture. For example, the client and server aspects are very-well separated as well as the XML parsing infrastructure. Moreover, we discovered simple-xmpp, a library built on top of it to make a number of common tasks easier to implement. Moreover, the infrastructure has been ported to web browsers using browserify.

Browserify is an interesting porting tool -- its main purpose is to provide a replacement for the CommonJS module system, which is a first-class citizen in Node.js, but unavailable in the browser. Browserify statically analyzes closures of CommonJS modules, and packs them into a single JavaScript file so that the module system is no longer needed.

Furthermore, browserify provides browser-equivalent substitutes for many core Node.js APIs, such as events, stream and path, making it considerably easier to migrate software from Node.js to the browser.

Porting simple-xmpp to Titanium

In addition to browserify, there also exists a similar approach for Titanium: titaniumifier, that has been built on top of the browserify architecture.

Similar to browserify, titaniumifier also packs a collection of CommonJS modules into a single JavaScript file. Moreover, it constructs a Titanium module from it, packs it into a Zip file that can be distributed to any Titanium developer so that it can be used by simply placing it into the root folder of the app project and adding the following module requirement to tiapp.xml:

<module platform="commonjs">ti-simple-xmpp</module>

Furthermore, it provides Titanium-equivalent substitute APIs for Node.js core APIs, but its library is considerably more slim and incomplete than browserify.

We can easily apply titatiumifier to simple-xmpp, by creating a new NPM project (package.json file) that has a dependency on simple-xmpp:

{
"name": "ti-simple-xmpp",
"version": "0.0.1",
"dependencies": {
"simple-xmpp": "1.3.x"
  }
}

and a proxy CommonJS module (index.js) that simply exposes the Simple XMPP module:

exports.SimpleXMPP = require('simple-xmpp');

After installing the project dependencies (simple-xmpp only) with:

$ npm install

We can attempt to migrate it to Titanium, by running the following command-line instruction:

$ titaniumifier

In my first titaniumify attempt, the tool says that some mandatory Titanium specific properties, such as a unique GUID identifier, are missing that need to be added to package.json:

"titaniumManifest": {
"guid": "76cb731c-5abf-3b79-6cde-f04202e9ea6d"
},

After adding the missing GUID property, a CommonJS titanium module gets produced that we can integrate in any Titanium project we want:

$ titaniumifier
$ ls
ti-simple-xmpp-commonjs-0.0.1.zip

Fixing API mismatches

With our generated CommonJS package, we can start experimenting by creating a simple app that only connects to a remote XMPP server, by adding the following lines to a Titanium app's entry module (app.js):

var xmpp = require('ti-simple-xmpp').SimpleXMPP;

xmpp.connect({
    websocket: { url: 'ws://myserver.com:5280/websocket/' },
    jid : 'username@myserver.com',
    password : 'password',
    reconnect: true,
    preferred: 'PLAIN',
    skipPresence: false
});

In our first trial run, the app crashed with the following error message:

Object prototype may only be an Object or null

This problem seemed to be caused by the following line that constructs an object with a prototype:

ctor.prototype = Object.create(superCtor.prototype, {

After adding a couple of debugging statements in front of the Object.create() line that print the constructor and the prototype's properties, I noticed that it tries to instantiate a stream object (not a constructor function) without a prototype member. Referring to a prototype that is undefined, is apparently not allowed.

Deeper inspection revealed the following code block:

/*<replacement>*/
var Stream;
(function() {
  try {
    Stream = require('st' + 'ream');
  } catch (_) {} finally {
    if(!Stream) Stream = require('events').EventEmitter;
  }
})();
/*</replacement>*/

The above code block attempts to load the stream module, and provides the event emitter as a fallback if it cannot be loaded. The stream string has been scrambled to prevent browserify to statically bundle the module. It appears that the titaniumifier tool provides a very simple substitute that is an object. As a result, it does not use the event emitter as a fallback.

We can easily fix the stream object's prototype problem, by supplying it with an empty prototype property by creating a module (overrides.js) that modifies it:

try {
    var stream = require('st' + 'ream');
    stream.prototype = {};
} catch(ex) {
    // Just ignore if it didn't work
}

and by importing the overrides module in the index module (index.js) before including simple-xmpp:

exports.overrides = require('./overrides');
exports.SimpleXMPP = require('simple-xmpp');

After fixing the prototype problem, the next trial run crashed the app with the following error message:

undefined is not an object (evaluation process.version.slice)

which seemed to be caused by the following line:

var asyncWrite = !process.browser && [ 'v0.10', 'v0.9.'].indexOf(process.version.slice(0, 5)) > -1 ? setImmediate : processNextTick;

Apparently, titaniumifier does not provide any substitute for process.version and as a result invoking the slice member throws an exception. Luckily, we can circumvent this by making sure that process.browser yields true, by adding the following line to the overrides module (overrides.js):

process.browser = true;

The third trial run crashed the app with the following message:

Can't find variable: XMLHttpRequest at ti-simple.xmpp.js (line 1943)

This error is caused by the fact that there is no XMLHttpRequest object -- an API that a web browser would normally provide. I have found a Titanium-based XHR implementation on GitHub that provides an identical API.

By copying the xhr.js file into our project, wrapping it in a module (XMLHttpRequest.js), we can provide a constructor that is identical to the browser API:

exports.XMLHttpRequest = require('./xhr');

global.XMLHttpRequest = module.exports;

By adding it to our index module:

exports.overrides = require('./overrides');
exports.XMLHttpRequest = require('./XMLHttpRequest');
exports.SimpleXMPP = require('simple-xmpp');

we have provided a substitute for the XMLHttpRequest API that is identical to a browser.

In the fourth run, the app crashed with the following error message:

Can't find variable: window at ti-simple-xmpp.js (line 1789)

which seemed to be caused by the following line:

var WebSocket = require('faye-websocket') && require('faye-websocket').Client ? require('faye-websocket').Client : window.WebSocket

Apparently, there is no window object nor a WebSocket constructor, as they are browser-specific and not substituted by titaniumifier.

Fortunately, there seems to be a Websocket module for Titanium that works both on iOS and Android. The only inconvenience is that its API is similar, but not exactly identical to the browser's WebSocket API. For example, creating a WebSocket in the browser is done as follows:

var ws = new WebSocket("ws://localhost/websocket");

whereas with the TiWS module, it must be done as follows:

var tiws = require("net.iamyellow.tiws");

var ws = tiws.open("ws://localhost/websocket");

These differences make it very tempting to manually fix the converted simple XMPP library, but fortunately we can create an adapter that has an identical interface to the browser's WebSocket API, translating calls to the Titanium WebSockets module:

var tiws = require('net.iamyellow.tiws');

function WebSocket() {
    this.ws = tiws.createWS();
    var url = arguments[0];
    this.ws.open(url);

    var self = this;

    this.ws.addEventListener('open', function(ev) {
        self.onopen(ev);
    });

    this.ws.addEventListener('close', function() {
        self.onclose();
    });

    this.ws.addEventListener('error', function(err) {
        self.onerror(err);
    });

    this.ws.addEventListener('message', function(ev) {
        self.onmessage(ev);
    });
}

WebSocket.prototype.send = function(message) {
    return this.ws.send(message);
};

WebSocket.prototype.close = function() {
    return this.ws.close();
};

if(global.window === undefined) {
    global.window = {};
}

global.window.WebSocket = module.exports = WebSocket;

Adding the above module to the index module (index.js):

exports.overrides = require('./overrides');
exports.XMLHttpRequest = require('./XMLHttpRequest');
exports.WebSocket = require('./WebSocket');
exports.SimpleXMPP = require('simple-xmpp');

seems to be the last missing piece in the puzzle. In the fifth attempt, the app seems to properly establish an XMPP connection. Coincidentally, all the other chat functions also seem to work like a charm! Yay! :-)

Conclusion

In this blog post, I have described a process in which I have ported simple-xmpp from the Node.js ecosystem to Titanium. The process was mostly automated, followed by a number of trial, error and fix runs.

The fixes we have applied are substitutes (identical APIs for Titanium), adapters (modules that translate calls to a particular API into a calls to a Titanium-specific API) and overrides (imperative modifications to existing modules). These changes did not require me to modify the original package (the original package is simply a dependency of the ti-simple-xmpp project). As a result, we do not have to maintain a fork and we have only little maintenance on our side.

Limitations

Although the porting approach seems to fit our needs, there are a number of things missing. Currently, only XMPP over WebSocket connections are supported. Ordinary XMPP connections require a Titanium-equivalent replacement for Node.js'net.Socket API, which is completely missing.

Moreover, the Titanium WebSockets library has some minor issues. The first time we tested a secure web socket wss:// connection, the app crashed on iOS. Fortunately, this problem has been fixed now.

References

The ti-simple-xmpp package can be obtained from GitHub. Moreover, I have created a bare bones Alloy/Titanium-based example chat app (XMPPTestApp) exposing most of the library's functionality. The app can be used on both iOS and Android:

Acknowledgements

The work described in this blog post is a team effort -- Yiannis Tsirikoglou first attempted to port strophe.js and manually ported simple-xmpp to Titanium before I managed to complete the automated approach described in this blog post. Carlos Henrique Lustosa Zinato provided us Titanium-related advice and helped us diagnosing the TiWS module problems.

Five years ago, while I was still in academia, I built an extension framework around Disnix (named: Dynamic Disnix) that enables self-adaptive redeployment of service-oriented systems. It was an interesting application as it demonstrated the full potential of service-oriented systems having their deployment processes automated with Disnix.

Moreover, the corresponding research paper was accepted for presentation at the SEAMS 2011 symposium (co-located with ICSE 2011) in Honolulu (Hawaii), which was (obviously!) a nice place to visit. :-)

Disnix's development was progressing at a very low pace for a while after I left academia, but since the end of 2014 I made some significant improvements. In contrast to the basic toolset, I did not improve Dynamic Disnix -- apart from the addition of a port assigner tool, I only kept the implementation in sync with Disnix's API changes to prevent it from breaking.

Recently, I have used Dynamic Disnix to give a couple of demos. As a result, I have improved some of its aspects a bit. For example, some basic documentation has been added. Furthermore, I have extended the framework's architecture to take a couple of new deployment planning aspects into account.

Disnix

For readers unfamiliar with Disnix: the primary purpose of the basic Disnix toolset is executing deployment processes of service-oriented systems. Deployments are driven by three kinds of declarative specifications:

• The services model captures the services (distributed units of deployments) of which a system consists, their build/configuration properties and their inter-dependencies (dependencies on other services that may have to be reached through a network link).
• The infrastructure model describes the target machines where services can be deployed to and their characteristics.
• The distribution model maps services in the services model to machines in the infrastructure model.

By writing instances of the above specifications and running disnix-env:

$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix

Disnix executes all activities to get the system deployed, such as building their services from source code, distributing them to the target machines in the network and activating them. Changing any of these models and running disnix-env again causes the system to be upgraded. In case of an upgrade, Disnix will only execute the required activities making the process more efficient than deploying a system from scratch.

"Static" Disnix

So, what makes Disnix's deployment approach static? When looking at software systems from a very abstract point of view, they are supposed to meet a collection of functional and non-functional requirements. A change in a network of machines affects the ability for a service-oriented system to meet them, as the services of which these systems consist are typically distributed.

If a system relies on a critical component that has only one instance deployed and the machine that hosts it crashes, the functional requirements can no longer be met. However, even if we have multiple instances of the same components giving better guarantees that no functional requirements will be broken, important non-functional requirements may be affected, such as the responsiveness of a system.

We may also want to optimize a system's non-functional properties, such as its responsiveness, by adding more machines to the network that offer more system resources, or by changing the configuration of existing machine, e.g. upgrading the amount available RAM.

The basic Disnix toolset is considered static, because all these events events require manual modifications to the Disnix models for redeployment, so that a system can meet its requirements under the changed conditions.

For simple systems, manual reconfiguration is still doable, but with one hundred services, one hundred machines or a high frequency of events (or a combination of the three), it becomes too complex and time consuming.

For example, when a machine has been added or removed, we must rewrite the distribution model in such a way that all services are deployed to at least one machine and that none of them are mapped to machines that are not capable or allowed to host them. Furthermore, with microservices (one of their traits is that they typically embed HTTP servers), we must typically bind them to unique TCP ports that do not conflict with system services or other services deployed by Disnix. None of these configuration aspects are trivial for large service-oriented systems.

Dynamic Disnix

Dynamic Disnix extends Disnix's architecture with additional models and tools to cope with the dynamism of service oriented-systems. In the latest version, I have extended its architecture (which has been based on the old architecture described in the SEAMS 2011 paper and corresponding blog post):


The above diagram shows the structure of the dydisnix-self-adapt tool. The ovals denote command-line utilities, the rectangles denote files and the arrows denote files as inputs or outputs. As with the basic Disnix toolset, dydisnix-self-adapt is composed of command-line utilities each being responsible for executing an individual deployment activity:

• On the top right, the infrastructure generator is shown that captures the configurations of the machines in the network and generates an infrastructure model from it. Currently, two different kinds of generators can be used: disnix-capture-infra (included with the basic toolset) that uses a bootstrap infrastructure model with connectivity settings, or dydisnix-geninfra-avahi that uses multicast DNS (through Avahi) to retrieve the machines' properties.
• dydisnix-augment-infra is responsible for augmenting the generated infrastructure model with additional settings, such as passwords. It is typically undesired to automatically publish privacy-sensitive settings over a network using insecure connection protocols.
• disnix-snapshot can be optionally used to preemptively capture the state of all stateful services (services with property: deployState = true; in the services model) so that the state of these services can be restored if a machine crashes or disappears. This tool is new in the extended architecture.
• dydisnix-gendist generates a mapping of services to machines based on technical and non-functional-properties defined in the services and infrastructure models.
• dydisnix-port-assignassigns unique TCP port numbers to previously undeployed services and retains assigned TCP ports in a previous deployment for optimization purposes. This tool is new in the extended architecture.
• disnix-envredeploys the system with the (statically) provided services model and the dynamically generated infrastructure and distribution models.

An example usage scenario

When a system has been configured to be (statically) deployed with Disnix (such as the infamous StaffTracker example cases that come in several variants), we need to add a few additional deployment specifications to make it dynamically deployable.

Auto discovering the infrastructure model

First, we must configure the machines in such a way that they publish their own configurations. The basic toolset comes with a primitive solution called: disnix-capture-infra that does not require any additional configuration -- it consults the Disnix service that is installed on every target machine.

By providing a simple bootstrap infrastructure model (e.g. infrastructure-bootstrap.nix) that only provides connectivity settings:

{
  test1.properties.hostname = "test1";
  test2.properties.hostname = "test2";
}

and running disnix-capture-infra, we can obtain the machines' configuration properties:

$ disnix-capture-infra infrastructure-bootstrap.nix

By setting the following environment variable, we can configure Dynamic Disnix to use the above command to capture the machines' infrastructure properties:

$ export DYDISNIX_GENINFRA="disnix-capture-infra infrastructure-bootstrap.nix"

Alternatively, there is the Dynamic Disnix Avahi publisher that is more powerful, but at the same time much more experimental and unstable than disnix-capture-infra.

When using Avahi, each machine uses multicast DNS (mDNS) to publish their own configuration properties. As a result, no bootstrap infrastructure model is needed. Simply gathering the data published by the machines on the same subnet suffices.

When using NixOS on a target machine, the Avahi publisher can be enabled by cloning the dydisnix-avahi Git repository and adding the following lines to /etc/nixos/configuration.nix:

imports = [ /home/sander/dydisnix/dydisnix-module.nix ];
services.dydisnixAvahiTest.enable = true;

To allow the coordinator machine to capture the configurations that the target machines publish, we must enable the Avahi system service. In NixOS, this can be done by adding the following lines to /etc/nixos/configuration.nix:

services.avahi.enable = true;

When running the following command-line instruction, the machines' configurations can be captured:

$ dydisnix-geninfra-avahi

Likewise, when setting the following environment variable:

$ export DYDISNIX_GENINFRA=dydisnix-geninfra-avahi

Dynamic Disnix uses the Avahi-discovery service to obtain an infrastructure model.

Writing an augmentation model

The Java version of StaffTracker for example uses MySQL to store data. Typically, it is undesired to publish the authentication credentials over the network (in particular with mDNS, which is quite insecure). We can augment these properties to the captured infrastructure model with the following augmentation model (augment.nix):

{infrastructure, lib}:

lib.mapAttrs (targetName: target:
  target // (if target ? containers && target.containers ? mysql-database then {
    containers = target.containers // {
      mysql-database = target.containers.mysql-database //
        { mysqlUsername = "root";
          mysqlPassword = "secret";
        };
    };
  } else {})
) infrastructure

The above model implements a very simple password policy, by iterating over each target machine in the discovered infrastructure model and adding the same mysqlUsername and mysqlPassword property when it encounters a MySQL container service.

Mapping services to machines

In addition to a services model and a dynamically generated (and optionally augmented) infrastructure model, we must map each service to machine in the network using a configured strategy. A strategy can be programmed in a QoS model, such as:

{ services
, infrastructure
, initialDistribution
, previousDistribution
, filters
, lib
}:

let
  distribution1 = filters.mapAttrOnList {
    inherit services infrastructure;
    distribution = initialDistribution;
    serviceProperty = "type";
    targetPropertyList = "supportedTypes";
  };

  distribution2 = filters.divideRoundRobin {
    distribution = distribution1;
  };
in
distribution2

The above QoS model implements the following policy:

• First, it takes the initialDistribution model that is a cartesian product of all services and machines. It filters the machines on the relationship between the type attribute and the list of supportedTypes. This ensures that services will only be mapped to machines that can host them. For example, a MySQL database should only be deployed to a machine that has a MySQL DBMS installed.
• Second, it divides the services over the candidate machines using the round robin strategy. That is, it divides services over the candidate target machines in equal proportions and in circular order.

Dynamically deploying a system

With the services model, augmentation model and QoS model, we can dynamically deploy the StaffTracker system (without manually specifying the target machines and their properties, and how to map the services to machines):

$ dydisnix-env -s services.nix -a augment.nix -q qos.nix

The Node.js variant of the StaffTracker example requires unique TCP ports for each web service and web application. By providing the --ports parameter we can include a port assignment specification that is internally managed by dydisnix-port-assign:

$ dydisnix-env -s services.nix -a augment.nix -q qos.nix --ports ports.nix

When providing the --ports parameter, the specification gets automatically updated when ports need to be reassigned.

Making a system self-adaptable from a deployment perspective

With dydisnix-self-adapt we can make a service-oriented system self-adaptable from a deployment perspective -- this tool continuously monitors the network for changes, and runs a redeployment when a change has been detected:

$ dydisnix-self-adapt -s services.nix -a augment.nix -q qos.nix

For example, when shutting down a machine in the network, you will notice that Dynamic Disnix automatically generates a new distribution and redeploys the system to get the missing services back.

Likewise, by adding the ports parameter, you can include port assignments as part of the deployment process:

$ dydisnix-self-adapt -s services.nix -a augment.nix -q qos.nix --ports ports.nix

By adding the --snapshots parameter, we can preemptively capture the state of all stateful services (services annotated with deployState = true; in the services model), such as the databases in which the records are stored. If a machine hosting databases disappears, Disnix can restore the state of the databases elsewhere.

$ dydisnix-self-adapt -s services.nix -a augment.nix -q qos.nix --snapshot

Keep in mind that this feature uses Disnix's snapshotting facilities, which may not be the best solution to manage state, in particular with large databases.

Conclusion

In this blog post, I have described an extended architecture of Dynamic Disnix. In comparison to the previous version, a port assigner has been added that automatically provides unique port numbers to services, and the disnix-snapshot utility that can preemptively capture the state of services, so that they can be restored if a machine disappears from the network.

Despite the fact that Dynamic Disnix has some basic documentation and other improvements from a usability perspective, Dynamic Disnix remains a very experimental prototype that should not be used for any production purposes. In contrast to the basic toolset, I have only used it for testing/demo purposes and I still have no real-life production experience with it. :-)

Moreover, I still have no plans to officially release it yet as many aspects still need to be improved/optimized. For now, you have to obtain the Dynamic Disnix source code from Github and use the included release.nix expression to install it. Furthermore, you probably need to a lot of courage. :-)

Finally, I have extended the Java and Node.js versions of the StaffTracker example as well as the virtual hosts example with simple augmentation and QoS models.

A while ago, I "rebranded"my second re-engineered version of npm2nix into node2nix and officially released it as such. My main two reasons for giving the tool a different name is that node2nix is neither a fork nor a continuation of npm2nix, but a tool that is written from scratch (though it incorporates some of npm2nix's ideas and concepts including most of its dependencies).

Furthermore, it approaches the expression generation problem in a fundamentally different way -- whereas npm2nix generates derivations for each package in a dependency tree and composes symlinks to their Nix store paths to allow a package to find its dependencies, node2nix deploys an entire dependency tree in one derivation so that it can more accurately mimic NPM's behaviour including flat-module installations (at the expense of losing the ability to share dependencies among packages and projects).

Because node2nix is conceptually different, I have decided to rename the project so that it can be used alongside the original npm2nix tool that still implements the old generation concepts.

Besides officially releasing node2nix, I have recently extended its feature set with a new concept for a recurring class of NPM development projects.

Global NPM development dependencies

As described in earlier blog posts, node2nix (as well as npm2nix) generate Nix expressions from a set of third party NPM packages (obtained from external sources, such as the NPM registry) or a development project's package.json file.

Although node2nix works fine for most of my development projects, I have noticed that for a recurring class of projects, the auto generation approach is too limited -- some NPM projects may require the presence of globally installed packages and must run additional build steps in order to be deployed properly. A prominent example of such a category of projects are Grunt projects.

Grunt advertises itself as a "The JavaScript Task Runner" and can be used to run all kinds of things, such as code generation, linting, minification etc. The tasks that Grunt carries out are implemented as plugins that must be deployed as a project's development dependencies with the NPM package manager.

(As a sidenote: it is debatable whether Grunt is a tool that NPM developers should use, as NPM itself can also carry out build steps through its script directive, but that discussion is beyond the scope of this blog post).

A Grunt workflow typically looks as follows. Consider an example project, with the following Gruntfile.js:

module.exports = function(grunt) {

  grunt.initConfig({
    jshint: {
      files: ['Gruntfile.js', 'src/**/*.js'],
      options: {
        globals: {
          jQuery: true
        }
      }
    },
    watch: {
      files: ['<%= jshint.files %>'],
      tasks: ['jshint']
    }
  });

  grunt.loadNpmTasks('grunt-contrib-jshint');
  grunt.loadNpmTasks('grunt-contrib-watch');

  grunt.registerTask('default', ['jshint']);

};

The above Gruntfile defines a configuration that iterates over all JavaScript files (*.js files) in the src/ directory and invokes jshint to check for potential errors and code smells.

To deploy the development project, we first have to globally install the grunt-cli command-line utility:

$ npm install -g grunt-cli
$ which grunt
/usr/local/bin/grunt

To be able to carry out the steps, we must update a project's package.json file to have grunt and all its required Grunt plugins as development dependencies:

{
"name": "grunt-test",
"version": "0.0.1",
"private": "true",
"devDependencies": {
"grunt": "*",
"grunt-contrib-jshint": "*",
"grunt-contrib-watch": "*"
  }
}

Then we must install the development dependencies with the NPM package manager:

$ npm install

And finally, we can run the grunt command-line utility to execute our tasks:

$ grunt

"Global dependencies" in Nix

Contrary to NPM, Nix does not support "global dependencies". As a matter of fact, it takes all kinds of precautions to prevent global dependencies to influence the builds that it performs, such as storing all packages in isolation in a so-called Nix store (e.g. /nix/store--grunt-1.0.1 as opposed to storing files in global directories, such as /usr/lib), initially clearing all environment variables (e.g. PATH) and setting these to the Nix store paths of the provided dependencies to allow packages to find them, running builds in chroot environments, etc.

These precautions are taken for a very good reason: purity -- each Nix package stored in the Nix store has a hash prefix that is computed from all involved build-time dependencies.

With pure builds, we know that (for example) if we encounter a build performed on one machine with a specific hash code and a build with an identical hash code on another machine, their build results are identical as well (with some caveats, but in general there are no observable side effects). Pure package builds are a crucial ingredient to make deployments of systems reliable and reproducible.

In Nix, we must always be explicit about the dependencies of a build. When a dependency is unspecified (something that commonly happens with global dependencies), a build will typically fail because it cannot be (implicitly) found. Similarly, when a build has dependencies on packages that would normally have to be installed globally (e.g. non-NPM dependencies), we must now explicitly provide them as a build inputs.

The problem with node2nix is that it automatically generates Nix expressions and that global dependencies cannot be detected automatically, because they are not specified anywhere in a package.json configuration file.

To cope with this limitation, the generated Nix expressions are made overridable, so that any missing dependency can be provided manually. For example, we may want to deploy an NPM package named floomatic from the following JSON file (node-packages.json):

[
"floomatic"
]

We can generate Nix expressions from the above specification, by running:

$ node2nix -i node-packages.json

One of floomatic's dependencies is an NPM package named: native-diff-match-patch that requires the Qt 4.x library and pkgconfig. These two packages are non-NPM package dependencies left undetected by the node2nix generator. In conventional Linux distributions, these packages typically reside in global directories, such as /usr/lib, and can still be implicitly found.

By creating an override expression (named: override.nix), we can inject these missing (global) dependencies ourselves:

{pkgs ? import <nixpkgs> {
    inherit system;
}, system ? builtins.currentSystem}:

let
  nodePackages = import ./default.nix {
    inherit pkgs system;
  };
in
nodePackages // {
  floomatic = nodePackages.floomatic.override (oldAttrs: {
    buildInputs = oldAttrs.buildInputs ++ [
      pkgs.pkgconfig
      pkgs.qt4
    ];
  });
}

With the override expression shown above, we can correctly deploy the floomatic package, by running:

$ nix-build override.nix -A floomatic

Providing supplemental NPM packages to an NPM development project

Similar to non-NPM dependencies, we also need to supply the grunt-cli as an additional dependency to allow a Grunt project build to succeed in a Nix build environment. What makes this process difficult is that grunt-cli is also an NPM package. As a consequence, we need to generate a second set of Nix expressions and propagate their generated package configurations as parameters to the former expression. Although it was already possible to do this, because the Nix language is flexible enough, the process is quite complex, hacky and inconvenient.

In the latest node2nix version, I have automated this workflow -- when generating expressions for a development project, it is now also possible to provide a supplemental package specification. For example, for our trivial Grunt project, we can create the following supplemental JSON file (supplement.json) that provides the grunt-cli:

[
"grunt-cli"
]

We can generate Nix expressions for the development project and supplemental package set, by running:

$ node2nix -d -i package.json --supplement-input supplement.json

Besides providing the grunt-cli as an additional dependency, we also need to run grunt after obtaining all NPM dependencies. With the following wrapper expression (override.nix), we can run the Grunt task runner after all NPM packages have been successfully deployed:

{ pkgs ? import <nixpkgs> {}
, system ? builtins.currentSystem
}:

let
  nodePackages = import ./default.nix {
    inherit pkgs system;
  };
in
nodePackages // {
  package = nodePackages.package.override {
    postInstall = "grunt";
  };
}

As may be observed in the expression shown above, the postInstall hook is responsible for invoking the grunt command.

With the following command-line instruction, we can use the Nix package manager to deploy our Grunt project:

$ nix-build override.nix -A package

Conclusion

In this blog post, I have explained a recurring limitation of node2nix that makes it difficult to deploy projects having dependencies on NPM packages that (in conventional Linux distributions) are typically installed in global file system locations, such as grunt-cli. Furthermore, I have described a new node2nix feature that provides a solution to this problem.

In addition to Grunt projects, the solution described in this blog post is relevant for other tools as well, such as ESLint.

All features described in this blog post are part of the latest node2nix release (version 1.1.0) that can be obtained from the NPM registry and the Nixpkgs collection.

Besides a new release, node2nix is now also used to generate the expressions for the set of NPM packages included with the development and upcoming 16.09 versions of Nixpkgs.

In earlier blog posts, I have covered various general concepts of the Nix package manager. For example, I have written three blog posts explaining Nix from a system administrator, programming language, and a sales perspective.

Furthermore, I have written a number of blog posts demonstrating how Nix can be used, such as managing a set of private packages, packaging binary-only software, and Nix's declarative nature -- the Nix package manager (as well as other tools in the Nix project), are driven by declarative specifications describing the structure of the system to deploy (e.g. the packages and its dependencies). From such a specification, Nix carries out all required deployment activities, including building the packages from its sources, distributing packages from the producer to consumer site, installation and uninstallation.

In addition to the execution of these activities, Nix provides a number of powerful non-functional properties, such as strong guarantees that dependency specifications are complete, that package builds can be reproduced, and that upgrades are non-destructive, atomic, and can always be rolled back.

Although many of these blog posts cover the activities that the Nix package manager typically carries out (e.g. the three explanation recipes cover building and the blog post explaining the user environment concepts discusses installing, uninstalling and garbage collection), I have not elaborated much about the distribution mechanisms.

A while ago, I noticed that people were using one of my earlier blog post as a reference. Despite being able to set up a collection of private Nix expressions, there were still some open questions left, such as fully setting up a private package repository, including the distribution of package builds.

In this blog post, I will explain Nix's distribution concepts and show how they can be applied to private package builds. As with my earlier blog post on managing private packages, these steps should be relatively easy to repeat.

Source and transparent binary deployments

As explained in my earlier blog post on packaging private software, Nix is in principle a source-based package manager. Nix packages are driven by Nix expressions describing how to build packages from source code and all its build-time dependencies, for example:

{ stdenv, fetchurl
, pkgconfig, perl, glib, gpm, slang, zip, unzip, file, gettext
, libX11, libICE, e2fsprogs
}:

stdenv.mkDerivation {
  name = "mc-4.8.12";

  src = fetchurl {
    url = http://www.midnight-commander.org/downloads/mc-4.8.12.tar.bz2;
    sha256 = "15lkwcis0labshq9k8c2fqdwv8az2c87qpdqwp5p31s8gb1gqm0h";
  };

  buildInputs = [ pkgconfig perl glib gpm slang zip unzip file gettext
      libX11 libICE e2fsprogs ];

  meta = {
    description = "File Manager and User Shell for the GNU Project";
    homepage = http://www.midnight-commander.org;
    license = "GPLv2+";
    maintainers = [ stdenv.lib.maintainers.sander ];
  };
}

The above expression (mc.nix) describes how to build Midnight Commander from source code and its dependencies, such as pkgconfig, perl and glib. Because the above expression does not specify any build procedure, the Nix builder environment reverts to the standard GNU Autotools build procedure, that typically consists of the following build steps: ./configure; make; make install.

Besides describing how to build a package, we must also compose a package by providing it the right versions or variants of the dependencies that it requires. Composition is typically done in a second expression referring to the former:

{ nixpkgs ? <nixpkgs>
, system ? builtins.currentSystem
}:

let
  pkgs = import nixpkgs { inherit system; };

  callPackage = pkgs.lib.callPackageWith (pkgs // pkgs.xlibs // self);

  self = rec {
    mc = callPackage ./mc.nix { };

    # Other package imports
  };
in
self

In the above expression (default.nix), the mc attribute imports our former expression and provides the build-time dependencies as function parameters. The dependencies originate from the Nixpkgs collection.

To build the Nix expression shown above, it typically suffices to run:

$ nix-build -A mc
/nix/store/jal99995sk6rixym4gfwcagmdiqrwv9a-mc-4.8.12

The result of a Nix build is a Nix store path in which the build result is stored.

As may be noticed, the prefix of the package name (jal99995sk6rixym4gfwcagmdiqrwv9a) is a SHA256 hash code that has been derived from all involved build dependencies, such as the source tarball, build-time dependencies and build scripts. Changing any of these dependencies (such as the version of the Midnight Commander) triggers a rebuild and yields a different hash code. Because hash codes ensure that the Nix store paths to packages will be unique, we can safely store multiple versions and variants of the same packages next to each other.

In addition to executing builds, Nix also takes as many precautions to ensure purity. For example, package builds are carried out in isolated environments in which only the specified dependencies can be found. Moreover, Nix uses all kinds of techniques to make builds more deterministic, such as resetting the timestamps of all files to 1 UNIX time, making build outputs read-only etc.

The combination of unique hash codes and pure builds results in a property called transparent binary deployments -- a package with an identical hash prefix results in a (nearly) bit-identical build regardless on what machine the build has been performed. If we want to deploy a package with a certain hash prefix that already exists on a trustable remote machine, then we can also transfer the package instead of building it again.

Distribution models

The Nix package manager (as well as its sub projects) support two kinds of distribution models -- push and pull deployment.

Push deployment

Push deployment is IMO conceptually the simplest, but at the same time, infrequently used on package management level and not very well-known. The idea of push deployment is that you take an existing package build on your machine (the producer) and transfer it elsewhere, including all its required dependencies.

With Nix this can be easily accomplished with the nix-copy-closure command, for example:

$ nix-copy-closure --to sander@consumermachine.org \
  /nix/store/jal99995sk6rixym4gfwcagmdiqrwv9a-mc-4.8.12

The above command serializes the Midnight Commander store path including all its dependencies, transfers them to the provided target machine through SSH, and then de-serializes and imports the store paths into the remote Nix store.

An implication of push deployment is that the producer requires authority over the consumer machine. Moreover, nix-copy-closure can transfer store paths from one machine to another, but does not execute any additional deployment steps, such as the "installation" of packages (in Nix packages become available to end-users by composing a Nix user environment that is in the user's PATH).

Pull deployment

With pull deployment the consumer machine is control instead of the producer machine. As a result, the producer does not require any authority over another machine.

As with push deployment, we can also use the nix-copy-closure command for pull deployment:

$ nix-copy-closure --from sander@producermachine.org \
  /nix/store/jal99995sk6rixym4gfwcagmdiqrwv9a-mc-4.8.12

The above command invocation is similar to the previous example, but instead copies a closure of the Midnight Commander from the producer machine.

Aside from nix-copy-closure, Nix offers another pull deployment mechanism that is more powerful and more commonly used, namely: channels. This is what people typically use when installing "end-user" packages with Nix.

The idea of channels is that they are remote HTTP/HTTPS URLs where you can subscribe yourself to. They provide a set of Nix expressions and a binary cache of substitutes:

$ nix-channel --add https://nixos.org/channels/nixos-unstable

The above command adds the NixOS unstable channel to the list of channels. Running the following command:

$ nix-channel --update

Fetches or updates the collection of Nix expressions from the channels allowing us to install any package that it provides. By default, the NIX_PATH environment variable is configured in such a way that they refer to the expressions obtained from channels:

$ echo $NIX_PATH
/nix/var/nix/profiles/per-user/root/channels/nixos:...

With a preconfigured channel, we can install any package we want including prebuilt substitutes, by running:

$ nix-env -i mc

The above command installs the Midnight Commander from the set of Nix expressions from the channel and automatically fetches the substitutes of all involved dependencies. After the installation succeeds, we can start it by running:

$ mc

The push deployment mechanisms of nix-copy-closure and pull deployment mechanisms of the channels can also be combined. When running the following command:

$ nix-copy-closure --to sander@consumermachine.org \
  /nix/store/jal99995sk6rixym4gfwcagmdiqrwv9a-mc-4.8.12 \
  --use-substitutes

The consumer machine first attempts to pull the substitutes of the dependency closure from the binary caches first, and finally the producer pushes the missing packages to the consumer machine. This approach is particularly useful if the connection between the producer and the consumer is slow, but the connection between the consumer and the binary cache is fast.

Setting up a private binary cache

With the concepts of push and pull deployment explained, you may probably wonder how these mechanisms can be used to your own private set of Nix packages?

Fortunately, with nix-copy-closure no additional work is required as it works on any store path, regardless of how it is produced. However, when it is desired to set up your own private binary cache, some additional work is required.

As channels/binary caches use HTTP as a transport protocol, we need to set up a web server with a document root folder serving static files. In NixOS, we can easily configure an Apache HTTP server instance by adding the following lines to a NixOS configuration: /etc/nixos/configuration.nix:

services.httpd = {
  enable = true;
  adminAddr = "admin@producer.org";
  hostName = "producer";
  documentRoot = "/var/www";
};

With the nix-push command we can generate a binary cache for our private package set (that includes the Midnight Commander and all its dependencies):

$ nix-push --dest /var/www $(nix-build default.nix)
/nix/store/jal99995sk6rixym4gfwcagmdiqrwv9a-mc-4.8.12

As may be observed by inspecting the document root folder, we now have a set of compressed NAR files representing the serialized forms of all packages involved and narinfo files capturing a package's metadata:

$ ls /var/www
007mwvk5kskrrgr0xgfnwxmwprd1jdvfxi079maq36mhg9dgwqlv.nar.xz
1c5z374p3z8065hfgyy82k2gi82wq147mrapkpivygcd6yjahcs6.nar.xz
a6gkm5vdylpjnsp9wzcb0c7kz11a3pak.narinfo
02jcvjlg2xha2bibl2ivb3nd0c3gh3nq.narinfo
1cgxv7679hqjvcs7jjjyjrk92n58ldj01yx1bsrvv6q2msh8r2m1.nar.xz
cfx2dvzj062qyj7qya9kcpwi1pk1bgin.narinfo
...

In addition to the binary cache, we also need to make the corresponding Nix expressions available. This can be done by simply creating a tarball out of the private Nix expressions and publishing the corresponding file through the web server:

tar cfvz /var/www/custompkgs.tar.gz custompkgs
custompkgs/
custompkgs/mc.nix
custompkgs/default.nix

On the customer machine, we need to configure the binary cache, by adding the following property to /etc/nix.conf:

binary-caches = http://producermachine.org/

In NixOS, this property can be set by adding the following property to /etc/nixos/configuration.nix:

nix.binaryCaches = [ "http://producermachine.org/" ];
nix.requireSignedBinaryCaches = false;

Additionally, we have to configure the NIX_PATH environment variable to refer to our tarball of Nix expressions:

$ export NIX_PATH=custompkgs=http://producermachine.org/custompkgs.tar.gz:$NIX_PATH

Now, when we run the following command-line instruction:

$ nix-env -f '<custompkgs>' -iA mc
downloading ‘http://producermachine.org/custompkgs.tar.gz’... [0/0 KiB, 0.0 KiB/s]
unpacking ‘http://producermachine.org/custompkgs.tar.gz’...
installing ‘mc-4.8.12’

We can install our custom Midnight Commander package by pulling the package from our own custom HTTP server without explicitly obtaining the set of Nix expressions or building it from source code.

Discussion

In this blog post, I have explained Nix's push and pull deployment concepts and shown how we can use them for a set of private packages, including setting up a private binary cache. The basic idea of binary cache distribution is quite simple: create a tarball of your private Nix expressions, construct a binary cache with nix-push and publish the files with an HTTP server.

In real-life production scenarios, there are typically more aspects you need to take into consideration beyond the details mentioned in this blog post.

For example, to make binary caches safe and trustable, it is also recommended to use HTTPS instead of plain HTTP connections. Moreover, you may want to sign the substitutes with a cryptographic key. The manual page of nix-push provides more details on how to set this up.

Some inconvenient aspects of the binary cache generation approach shown in this blog post (in addition to the fact that we need to set up an HTTP server), is that the approach is static -- whenever, we have a new version of a package built, we need to regenerate the binary cache and the package set.

To alleviate these inconveniences, there is also a utility called nix-serve that spawns a standalone web server generating substitutes on the fly.

Moreover, the newest version of Nix also provides a so-called binary cache Nix store. When Nix performs operations on the Nix store, it basically talks to a module with a standardized interface. When using the binary cache store module (instead of the standalone or remote Nix store plugin), Nix automatically generates NAR files for any package that gets imported into the Nix store, for example after the successful completion a build. Besides an ordinary binary cache store plugin, there is also plugin capable of uploading substitutes directly to an Amazon AWS S3 bucket.

Apart from the Nix package manager, also Nix-related projects use Nix's distribution facilities in some extent. Hydra, the Nix-based continuous integration server, also supports pull deployment as it can dynamically generate channels from jobsets. Users can subscribe to these channels to install the bleeding-edge builds of a project.

NixOps, a tool that deploys networks of NixOS configurations and automatically instantiates VMs in the cloud, as well as Disnix, a tool that deploys service-oriented systems (distributed systems that can be decomposed in a "distributable units", a.k.a. services) both use push deployment -- from a coordinator machine (that has authority over a collection of target machines) packages are distributed.

Concluding remarks

After writing this blog post and some thinking, I have become quite curious to see what a pull deployment variant of Disnix (and maybe NixOps) would look like.

Although Disnix and NixOps suit all my needs at the moment, I can imagine that when we apply the same concepts in large organizations with multiple distributed teams, it can no longer considered to be practical to work with a centralized deployment approach that requires authority over all build artefacts and the entire production environment.

It has been a while since I wrote my last blog post. Currently, I am enjoying a small Christmas break and I have spent a bit of my time playing around with a few of my old (pre-blog historical) projects.


Since the year 2016 is almost over, and because it has been roughly 20 years ago since the video game Duke Nukem 3D was released (1996), there was an abandoned project that particularly caught my interest.

Although the game was a very famous (or perhaps notorious) first-person shooter game known for having highly interactive environments, bad humour and non-linear levels, what I found particularly interesting is that it was also highly customizable -- it came with a level-editor (BUILD) allowing anyone to create their own custom maps, an art editor (EDITART) allowing people to create their own textures and sprites, and a scripting language (*.CON files) making it possible to modify or extend many aspects of the game, such as the behaviour of enemies, weapons, objects and the levels per episode.

Because of its hackability and the relative simplicity of the game engine and mechanics, these customization features were quite frequently used. Aside from many user created maps circling around the internet, there were also many total conversions available completely altering the game's behaviour. Some total conversions I found really interesting were Ages in Time (turning the game into a medieval setting), Platoon (a Vietnam war setting) and Wolf2Duke (a cross-over between Wolfenstein 3D and Duke Nukem 3D).

In 2003, the source code of Duke Nukem 3D was released under the GPLv2 (earlier in 2000, the source code of the BUILD engine that empowers Duke3D as well as several other games had already been released under Ken Silverman's BUILD license).

With the availability of the source code of both the game and the underlying engine, people have been making adjustments to make the game work on other operating systems than DOS (such as Windows and Linux) and enhancing its functionality to make it work better on modern systems.

For example, the most advanced source port of Duke Nukem 3D: EDuke32 complements the game engine with an OpenGL renderer with dynamic shading support. As a result, one of EDuke32's distinguished features is that it supports the fan-made High Resolution Pack enhancing the graphics quality of the textures and monsters.

While still being a teenager at high school, I have also spent quite a considerable amount of my spare time playing with these customization features. First, I started creating a couple of custom maps. Later, I started making other kinds of customizations, such as adjusting the monsters' behaviour to make them more dangerous.

I have combined all these modifications into my own total conversion consisting of two modified episodes containing 22 user maps in total. In this blog post, I will describe some of my development experiences.

Creating custom maps

Most of my work on the total conversion was spent on creating my own custom maps. Maps could be constructed with the BUILD editor included with the game. For example, the following command-line instruction allows us to open the first map of the single player episode of my total conversion:

> BUILD DDSL1.MAP

Starting the editor brings you a 2D view of the map shown from above:


Essentially, BUILD engine maps consist of 2D areas called sectors, denoted by areas surrounded by a white border, as shown in the picture above. It is also possible to create sub sectors (denoted by areas surrounded by a red border), to give sub areas in an existing sector different properties. I, for example, used them a lot for creating shadows.

Besides sectors, objects (such as monsters, items or weapons) are represented as sprites denoted by purple (or cyan) colored ovals with "sticks" attached to them.

By pressing the enter key on the numeric keypad, you could switch to 3D mode to see what the map would actually look like:


In 3D mode, there were many additional properties you could configure, such as the heights of the floor and ceilings, and the appearance of objects. For example, by pointing the crosshair to a wall, ceiling, floor, or sprite and pressing the 'V' key you could alter a texture or sprite's appearance:


Something that particularly puzzled me in the early beginning was constructing doors, such as garage doors that move up or down. I still remember very well that I had been studying the documentation for days, without any luck.

Eventually, by inspecting/disassembling other user maps I learned how the mechanics really worked -- the trick was to create a dedicated sector for the door and annotating the sector with a lotag and hitag value (that could be configured by pressing the 'T' key):


Lotag and hitag values (as denoted by the grey label in the picture above) were just metadata properties to the editor, but the game engine interpreted lotag values as functionality and hitag values as a means to group objects. The sector lotag value of: 20 corresponds to a function that moves the ceiling up to match the height of neighbouring ceiling once the player touches it. Any other object with the same hitag value would be triggered as well (that is why many doors in a map typically have unique hitag values, unless they were controlled by a key).

Besides configuring a sector to be a door, we also typically want to hear a sound effect when a player opens or closes it. Sound effects could be configured by placing a special-purpose sprite (MUSICANDSFX) (visible in the editor but invisible in the game), and annotating the sprite with a lotag (through Alt + 'T') corresponding to the number of the audio sample that should be played.

Additionally, we had to lower the door sector's ceiling in 3D mode in such a way that the it touches the ground, as a door is typically supposed to be closed by default. For example, if we "disassemble" a door by moving the ceiling up a bit in 3D mode, this is what we will see:


(As may be observed in the image above, the 'M'-sprite is the special-purpose sprite that configures the sound effect).

In addition to garage doors, there were many other kinds of interactivity patterns thay you could implement, such as rotating doors, elevators, moving vehicles (through moving sectors), shading effects, and objects that spawn if some action is triggered, all by annotating sectors and special-purpose sprites with lotags and hitags.

Despite the richness of features, the BUILD engine version used in Duke Nukem 3D also had a number of big limitations. For example, while 3D mode gave the player the illusion that he was observing a 3D world, most of its aspects were not truly 3D. For example, it was impossible to look up or down, because of the limitations of the renderer. Nonetheless, the game still gave the player the illusion that this was possible by simply stretching textures and adjusting the positions of the sprites a bit. (As a sidenote: after the source code of the BUILD engine was released, this issue was solved in the Polymost OpenGL-renderer).

Moreover, in 2D mode it was also possible to stack sectors on top of each other. However, in 3D mode you could only observe one of them at the time, preventing players to have a true room over room experience.

In some maps, a room over room experience was faked by "teleporting" a player from one sector to another. For example, when moving from an under water area to the surface, the player was technically "teleported" from one sector to another.


The above picture shows an area in my first map, in which I created a swimming pool. The swimming pool's surface is the rectangled sub sector on the right surrounded by a red border, while the under-water area is the rectangled sector on the left. By placing a sector effector sprite (the S-sprite shown below) in both sectors with a lotag value of: 7 and equal high tags, the teleportation could be controlled:


A similar fake experience was used for lifts -- when moving a lift up or down, the player was also technically teleported from one sector to another.

Another interesting feature of the game is that it was also possible to study and modify the maps that were included with the game. For example, the following command-line instruction opens the 9th level of the 3th episode:

> BUILD E3L9.MAP

(As a sidenote: if you may wonder why this works, the corresponding file: E3L9.MAP does not exists as a file, but is extracted from the game data's group file: DUKE3D.GRP).

Although I created nearly all my maps from scratch, the E3L9.MAP map, a football stadium in which you had to defeat the cycloid emperor boss, was a bit of a disappointment to me, because it was IMO too small and felt quite incomplete. As result, I created an extended version of the map adding many additional features, such as a stand and locker rooms:

Hacking the CON scripts

After creating a couple of maps and playing some total conversions, I also became interested in studying the CON scripts to see what kinds of modifications I could make. By skimming over them and playing around a bit, I made some interesting discoveries.

One of the things I learned in the BUILD editor is that you can create palette swapped sprites and textures. In the game, palette swaps were typically used to create color illuminated rooms or to change a monster's behaviour. A prominent example of such a monster is a blue colored lizard trooper that could be changed into a red colored lizard trooper. The latter was slightly stronger and had the ability to teleport.

Besides lizard troopers, it was also possible to create palette swaps of other enemies, such as pig cops, but I noticed that their behaviour did not change at all:


I wanted red colored pig cops to be more menacing than the blue ones. With a few small modifications to the GAME.CON file, I changed the behaviour of the red colored pig cops to shoot rockets (while keeping the blue colored pig cop's original behaviour intact), by changing the lines:

ifcanshoottarget
{
    sound PIG_ATTACK
    shoot SHOTGUN
    shoot SHOTGUN
    shoot SHOTGUN
    shoot SHOTGUN
    shoot SHOTGUN
}

into:

ifcanshoottarget
{
    ifspritepal 21
    {
        sound RPG_SHOOT
        shoot RPG
        shoot RPG
        shoot RPG
        shoot RPG
        shoot RPG
    }
    else
    {
        sound PIG_ATTACK
        shoot SHOTGUN
        shoot SHOTGUN
        shoot SHOTGUN
        shoot SHOTGUN
        shoot SHOTGUN
    }
}

To make a red colored pig cop drop an RPG instead of a shotgun when it gets killed, I changed:

ifrnd 16
    spawn SHIELD
else
    state drop_shotgun

into:

ifrnd 16
    spawn SHIELD
else
    ifspritepal 21
        state drop_rpg
    else
        state drop_shotgun

Similarly, I modified the code of the lizard mans in such a way that they would shoot with an expander if they were made red, or with a freezer when they are made blue.

In the game, there was also a miniature/weaker version of the first episode's end boss that behaved in a similar way. Also, when a miniature boss gets defeated, the game did not end.

While studying the CON files, I discovered that miniature versions of the second and third bosses also seemed to exist, but they were not used anywhere in the game. The only limitation that prevented them from being used is that their strength levels were set to 1 so that they would die almost instantly.

I modified their energy levels to make them much stronger, by taking the following lines:

ifspritepal 0
    ai AIBOSS2RUNENEMY
else
{
    strength 1
    sound BOS2_ATTACK ai AIBOSS2SHOOTENEMY
}

and changing the strength value into a something much higher than 1, e.g. 2000. One of the places where I used a miniature boss 2 is in my space station map:


Although these hacked mini bosses seemed to work, I observed that while fighting them, they would quite often hit themselves with their own rockets making it incredibly easy to beat them. :-)

Hacking on Wolf2Duke

Apart from my own, one of the total conversions I found quite interesting was Wolf2Duke creating a cross over between Duke Nukem 3D and Wolfenstein 3D.

Although I liked the idea, I was a bit disappointed about its game experience, because it did not really feel like being in a Wolfenstein 3D game at all. For example, most levels did not use any Wolfenstein 3D textures. Moreover, the enemies also behaved in odd ways and their attacks were extremely powerful.

After studying the total conversion's artifacts, such as the EDITART files, I noticed that there were many textures and sprites included from the original Wolfenstein 3D, but not used anywhere in the conversion. It even included images for items such as med kits, dogfood, and ammo clips, but no functionality was implemented to make them work.

I modified Wolf2Duke's CON scripts to add some of this missing functionality. I added the following code snippet to make the dog food item grant extra health (20 HP) to the player:

useractor notenemy DOGFOOD
  fall
  ifmove RESPAWN_ACTOR_FLAG
    state respawnit
  else
    ifp pshrunk nullop
    else
      ifp palive
        ifpdistl RETRIEVEDISTANCE
          ifcount 6
            ifphealthl MAXPLAYERHEALTH
              ifcanseetarget
      {
        addphealth 20
        quote 125
        ifspawnedby DOGFOOD
          state wolfgetcode
        else
          state wolfquikget
      }
enda

Moreover, I added some code to make the ammo clip work. When collecting a Wolfenstein 3D ammo clip, it would randomly grant the player pistol or chaingun ammo. How nice is that?

Besides hacking on the CON scripts, I also created two levels using my modified Wolf2Duke features in which I have been trying to create an experience that felt like the original game. Both levels were included as secret levels in my total conversion.

Reflection

I started the work on my own total conversion somewhere in 1997. It took me almost four years to complete it -- I finished my last map somewhere at the end of 2000.

Around 2005 (a few years after the open source release of Duke Nukem 3D and the release of EDuke32 with the High Resolution Pack (HRP)), I made some adjustments to the maps to make them look better when the engine's newer features were used.

Overall, I consider it to be a very interesting learning experience -- I learned a lot about the game engine, the game mechanics, and about game development in general. As a matter of fact, I liked tweaking and constructing my own maps much more than actually playing the game. :-)

With only limited resources, such as a slow dial-up internet connection and limited proficiency in English, I managed to figure out everything on my own. In total, I have created two episodes with 11 levels each. The maps were quite diverse -- for example, I have created city areas, space stations, landscapes, theme parks, factory buildings, and offices.


In my most ambitious map (as shown in the screenshot above), I was attempting to model the entire neighbourhood where I used to grow up. It took me nearly 7 months to complete the map and it became so big that it was pushing the engine to its limits. At some point, I had to make tradeoffs, such as sacrificing shadows, to prevent the game engine from hitting the maximum amount of sectors limit of 8192.

What I also learned is that game development is quite challenging and time consuming -- even without doing any real programming work and having a game engine at my disposal, it still takes a lot of time to construct something coherent, such as the levels, to test them and to tweak/optimize them to make them "feel right". To me, it is no surprise that many modern block buster games take many years to construct.

In addition to Duke Nukem 3D, I also created one user map for Shadow Warrior. Although it was using an improved version of the same game engine (BUILD), its game mechanics were completely different.

Furthermore, I made some attempts to construct maps for Unreal and Half-Life 1, but nothing successful came out of it. Due to lack of time and the huge amount of complexity I had to bridge, I lost my interest and changed my software development interests.

Today, it is my blog's sixth anniversary. As usual, this a good opportunity to reflect over last year's writings.

Disnix

Similar to last year, many of this year's blog posts have been covering Disnix-related aspects. In January, I released version 0.5 of Disnix, wrapping up most of the new functionality I have developed in the last months of 2015.

After releasing Disnix 0.5, I have been working on revising its architecture to provide the concept of containers, so that it became possible to deploy databases to multiple DBMS instances hosted on the same machine. Also, the notational conventions used in the Disnix infrastructure models have become much more structured.

In addition to the container concept, I modified Disnix in such a way that it can treat deployed services as containers, so that they become deployment targets for newly deployed services. Previously, it was required to deploy containers by other means first. With these new changes, complex Disnix deployments can now fully manage themselves.

Furthermore, I made some small adjustments so that Disnix can be used as a remote package deployer and I have extended the prototype Dynamic Disnix framework with new features to support state deployment and automatic port assignments to services. Most of these new features have become part of Disnix 0.6, released in June 2016.

node2nix

Another major accomplishment is my Nix/NPM integration work. I have (again!) reengineered my npm2nix fork to support NPM flat module installations.

Moreover, I rebranded my npm2nix fork into node2nix, integrated it into Nixpkgs (replacing the old npm2nix generated set of packages), and extended it with support for simulating NPM global package installations so that Node.js-related build tools, such as Grunt, can be conveniently used within the Nix ecosystem.

Nix/NixOS

Similar to previous years, I also did some general Nix/NixOS work -- I have modified Dysnomia (that used to be companion tool for Disnix) to integrate it with NixOS, so that it can also do state deployment on system level.

I also did some Nix promotion work. In March, I was invited to give a talk about Nix/NixOS at a Ruby-related conference. In this talk, I elaborated about Nix's declarative deployment properties.

Furthermore, I wrote a blog post explaining Nix's push and pull distribution mechanisms.

Miscellaneous stuff

Besides Nix/deployment-related work I have also written a blog post about integrating Node.js-style callback and Promise/A-based invocation patterns in JavaScript and a porting strategy providing XMPP messaging functionality to mobile apps using the Titanium framework.

Finally, in my Christmas break, I have been playing around with an old gaming project and wrote about my experiences.

Blog posts

Like all my previous reflections, I will publish the top 10 of my most frequently read blog posts. Surprisingly enough, not much has changed compared to last year:

1. On Nix and GNU Guix. Still remains my most popular blog post since 2012 and it appears that it is still attracting quite a few visitors.
2. An evaluation and comparison of Snappy Ubuntu. Still remains my second most popular blog post attracting almost as many visitors as the previous blog post.
3. Managing private Nix packages outside the Nixpkgs tree. This blog post is a practical hands on tutorial targeting Nix beginners. It has now moved to the third place and it seems to be quite frequently consulted.
4. Setting up a multi-user Nix installation on non-NixOS systems. This blog post also remains quite popular since 2014 and demonstrates that this area in the Nix user manual is still open for improvement.
5. An alternative explanation of the Nix package manager. Has slightly dropped in popularity, but is still frequent read.
6. Yet another blog post about Object Oriented Programming and JavaScript. This JavaScript-related blog post still remains incredibly popular. It now seems that I have become (sort of) an authority in explaining the prototype inheritance concept.
7. Asynchronous programming with JavaScript. Another JavaScript-related blog post that remains popular, although I have no idea why.
8. Composing FHS-compatible chroot environments with Nix (or deploying Steam in NixOS). Popular, but gradually dropping in popularity.
9. On NixOps, Disnix, service deployment and infrastructure deployment. This is the only blog post that was not in last year's top 10. I am actually quite happy to find out that there is an increasing amount of people taking interest in both NixOps and Disnix.
10. Using Nix while doing development. An explanation blog post that is still popular but gradually dropping in popularity.

Conclusion

I am still not out of ideas yet, so please stay tuned, because there will be more next year! The only thing I would like to say is:

HAPPY NEW YEAR!!!!!!!

It has been a while since I wrote my last programming-related blog post. In this blog post, I am going to elaborate about some of my experiences developing multi-process POSIX applications.

From my perspective, processes are an interesting operating systems concept, in particular in UNIX/POSIX-like operating systems, such as Linux.

The IEEE Std 1003.1 POSIX standard defines a "live process" as:

An address space with one or more threads executing within that address space, and the required system resources for those threads.

Within the boundaries of many (relatively simple) applications, process creation and management is typically not required. Nonetheless, decomposing a system into sub processes can be quite useful for a variety of reasons, such as:

• Improving a system's responsiveness by running slow/long-running tasks in the background, concurrently with other tasks. This is particularly useful to retain the ability to respond to user events, such as mouse clicks or keyboard input while data is being processed or to handle multiple connecting clients at the same time to a server application.
• Increased protection. If an incorrectly implemented or malicious task crashes during execution, it neither tears down the entire system nor affects the state of any other processes.
• More security. Child processes can be run under more restrictive user privileges, making it more difficult for the system to do any harm, such accessing privacy-sensitive filesystem areas.
• Portability. In a child process, we can invoke an external executable implemented in a different programming language.
• Scalability and performance. The execution of a collection of tasks can be parallelized by means of processes and their executions can be divided over multiple CPU cores by the operating system, potentially increasing the execution speed of a program.

Although multi-process applications may provide a number compelling benefits, programming such applications using the C programming language and the POSIX API is IMO not always straight forward -- I have found myself frequently repeating numerous patterns over and over again.

To alleviate the burden of repetition, I have identified a number of patterns, derived abstractions from them and constructed a library package, named libprocreact, providing these abstractions. The APIs that libprocreact provides are loosely inspired by reactive programming.

Programming patterns

When programming multi-process applications, there are many housekeeping tasks that need to be performed in order to properly organize them. Below, I have described a number of recurring patterns I typically implement:

Forking

The first and most prominent house keeping task is process creation. The key ingredient in creating processes is the fork() system call, as shown in the code example below:

#include <stdio.h>
#include <unistd.h>

pid_t pid = fork();

if(pid == -1)
    printf("The child process cannot be forked!\n");
else if(pid == 0)
{
    printf("Code executed by the child process\n");
    printf("It runs in the background!\n");
    _exit(0);
}
else
{
    printf("Code executed by the parent process!\n");
    printf("The pid of the child process is: %d\n", pid);
}

Forking is a relatively simple concept -- after successfully invoking the fork() function call (i.e. the return value is not -1), a child process gets created that appears as a nearly identical clone of the parent process. For example, their memory contents and file descriptors are identical.

Furthermore, a forked child process will be executed immediately in parallel to the parent process. Since a parent and child process are almost identical, we can use the return value of fork() to make a distinction between them -- in the parent, the fork() function call returns the PID of the child process so that it can monitor its status and interact with it. In the child process, fork() returns 0.

Although creating a clone of a parent process may sound very expensive, many POSIX-compliant operating systems have optimized this process by using a Copy-On-Write (COW) memory model -- instead of copying a parent process' memory, the memory between the parent and child processes is shared. The operating system maintains a table of shared and private memory pages for each process. When a process attempts to write to a shared memory page, then the corresponding memory page is copied and marked as private to the process.

Executing tasks

After forking a child process, we typically want it to execute a task so that we can use some of the process' positive traits in our advantage. In the if(pid == 0) { ... } block (shown in the previous example), we can put the code that the child process must execute.

For example, we can execute a long running task without blocking the parent process, such as doing an expensive linear search over an array of strings:

#include <string.h>

char *strings[] = { "foo", "bar", "baz", ..., NULL };
char *search = "baz";

...
else if(pid == 0)
{
    unsigned int i;

    for(i = 0; i < strlen(strings); i++)
    {
        if(strcmp(strings[i], search) == 0)
        {
            printf("%s has been found!\n", search);
            _exit(0);
        }
    }

    printf("%s cannot be found!\n", search);
    _exit(1);
}

(As may be observed in the example above, the string array is allocated by the parent process, but since the child process manifests itself as a duplicate on spawning, it has a reference to a "logical copy" as well).

We can also change the user permissions of the child process (for example, to restrict a task from having super-user permissions that might do potential harm):

#include <sys/types.h>
#include <unistd.h>

...
else if(pid == 0)
{
    if(setgid(100) == 0 && setuid(1000) == 0)
    {
        /* Execute some code with restrictive user permissions */
        ...
    }
    else
    {
        printf("Cannot change user permissions!\n");
        _exit(1);
    }
}

or invoking external (pre-built) executables, such as the cat command to stream the contents of a file to the standard output:

...
else if(pid == 0)
{
    char *const args[] = { "cat", "bigtextfile.txt", NULL };
    execvp(args[0], args);
    _exit(1);
}

Waiting

In addition to forking and carrying out tasks by child processes, the parent process must also take notice of a child process' status at some point. This is actually an obligation for certain events, for example, when a child process terminates -- a terminated child process remains a zombie until the parent process takes notice of it.

Taking notice of a process' status can be done by invoking a wait function, such as:

#include <sys/types.h>
#include <wait.h>

pid_t pid;
int wstatus;

/* Fork and execute */

pid_t ret_pid = waitpid(pid, &status, 0);

The above function call specifically waits for a process with a given PID to change state, and captures its wait status in the wstatus variable.

As a sidenote: besides waiting for a specific child process' state to change, it also possible to wait for any process in a process group to terminate (e.g. by invoking wait()). Furthermore, the wait function invocation blocks the parent process' execution by default, until a child process' state changes. We can also pass the WNOHANG flag to waitpid() to prevent it from blocking.

After a wait function invocation completes, we must interpret the return value and wait status:

if(ret_pid == -1)
    printf("Cannot obtain wait status of PID: %d\n", pid);
else if(!WIFEXITED(wstatus))
    printf("The process terminated abnormally!\n");
else if(WEXITSTATUS(wstatus) != 0)
    printf("The process execution failed!\n");
else
    printf("The process has completed its tasks successfully!\n");

In the above code fragment, we check for the following properties:

• Whether the wait status could be obtained. Sometimes this may not be possible, for example, if a process with a given PID does not exists or when it is beyond the parent process' control.
• Whether a process has been terminated abnormally or not. For example, abnormal termination happens when a process runs into a segmentation fault. In such cases, it may happen that a process still returns a zero exit status, incorrectly indicating that everything has succeeded.
• Whether a process has succeeded its tasks or not. By convention, a zero exit status indicates success, while any non-zero exit status indicates failure.

Output buffering

Sometimes, it may also be desired to propagate data back to the parent, for example, after the completion of a data transformation task. Since processes operate in their own private address space, we can no longer rely on shared memory, but we must use some means to transfer the data.

One of the possible means is using a pipe, as shown in the following code fragment:

int pipefd[2];

if(pipe(pipefd) == 0)
{
    pid_t pid = fork();

    if(pid == -1)
        fprintf(stderr, "Cannot fork process!\n");
    else if(pid == 0)
    {
        char *const args[] = { "sort", "words.txt", NULL };

        close(pipefd[0]); /* Close read-end of pipe */
        dup2(pipefd[1], 1); /* Attach write-end to the stdout */
        execvp(args[0], args);
        _exit(0);
    }
    else
    {
        close(pipefd[1]); /* Close write-end of pipe */
        /* Read from pipefd[0] */
        close(pipefd[0]);
    }
}
else
    fprintf(stderr, "Cannot construct a pipe!\n");

Before forking a child process, we construct a pipe consisting of two file descriptors -- a read and write end. In the child process, we close the read-end of the pipe (since it is not needed), and we write data to the write-end. In the parent, we read from the read-end and we discard the unneeded write-end.

When retrieving data from a pipe, we may want to capture its output in a data structure (such as a string, string array or struct). Capturing and transforming data to a specific structure is often not very straight forward, for example:

#define BUFFER_SIZE 1024

ssize_t bytes_read;
char *captured_string = NULL;
unsigned int captured_string_size = 0;

while((bytes_read = read(pipefd[0], buffer, BUFFER_SIZE)) > 0)
{
    char buffer[BUFFER_SIZE];

    captured_string = (char*)realloc(captured_string, captured_string_size + bytes_read);
    memcpy(captured_string + captured_string_size, buffer, bytes_read);
    captured_string_size += bytes_read;
}

/* Add NUL-termination */
captured_string = (char*)realloc(captured_string, captured_string_size + 1));
captured_string[captured_string_size] = '\0';

The purpose of above code fragment is to read data from a pipe and constructing a NUL-terminated string. It repeatedly reads chunks of data (of one kilobyte each) from the read-end of the pipe, dynamically extends the size of the buffer collecting the output, appends each chunk to the buffer, and finally appends a NUL-termination to the result.

As may be noticed, there are many concerns that we have to take care of and the resulting code is not trivial at all.

Orchestrating collections of processes

In addition to implementing the above patterns for a single child process running in the background, we may also want to apply them to collections of processes running concurrently.

Orchestrating collections of processes introduce many additional challenges beyond those described in the previous sections. For example, in order to read from the processes' pipes without blocking their executions (which could happen if any of their buffers gets full), we have to multiplex the house keeping operations in such a way that they read from each pipe breadth-first.

Multiplexing any of the previously shown patterns makes developing multi-process applications even more difficult.

A functional programming discipline

Because housekeeping tasks require us to supply many lines of boilerplate code, multi-process applications tend to become quite messy without any proper organization. For example, if I would take the sorting example (shown earlier) and extend it to capture the output of the process invocation into a string, I may end up writing:

#define BUFFER_SIZE 1024

int pipefd[2];

if(pipe(pipefd) == 0)
{
    pid_t pid = fork();

    if(pid == -1)
        fprintf(stderr, "Cannot fork process!\n");
    else if(pid == 0)
    {
        char *const args[] = { "sort", "words.txt", NULL };

        close(pipefd[0]); /* Close read-end of pipe */
        dup2(pipefd[1], 1); /* Attach write-end to the stdout */
        execvp(args[0], args);
        _exit(0);
    }
    else
    {
        ssize_t bytes_read;
        char *captured_string = NULL;
        unsigned int captured_string_size = 0;

        close(pipefd[1]); /* Close write-end of pipe */

        while((bytes_read = read(pipefd[0], buffer, BUFFER_SIZE)) > 0)
        {
            char buffer[BUFFER_SIZE];

            captured_string = (char*)realloc(captured_string, captured_string_size + bytes_read);
            memcpy(captured_string + captured_string_size, buffer, bytes_read);
            captured_string_size += bytes_read;
        }

        /* Add NUL-termination */
        captured_string = (char*)realloc(captured_string, captured_string_size + 1));
        captured_string[captured_string_size] = '\0';

        close(pipefd[0]); /* Close read-end of pipe */
    }
}
else
    fprintf(stderr, "Cannot construct a pipe!\n");

I do not expect anyone the study the above code fragment in detail, but just by looking at its structure and length, you could clearly see that this is not very appealing way to construct applications.

In contrast, the same task can be implemented in only one line of bash shell code:

captured_string=$(sort words.txt)

When I look at the previous code fragment from an abstract point of view, then it encapsulates an implementation of a specific concern (i.e. its primary task, such as sorting an array), and a number of general house-keeping concerns, such as constructing a pipe, waiting, output buffering, and closing file descriptors.

One possible way to structure the code in a better way is by function decomposition -- we can separate the specific and general concerns into functions and we can put the general house keeping aspects into a (reusable) library.

Consider the following synchronous function definition and invocation example that checks whether a given string exists in array of strings by doing a linear search:

#include <stdio.h>
#include <string.h>

int array_contains_string(const char **strings, const char *search)
{
    unsigned int i;

    for(i = 0; i < strlen(strings); i++)
    {
        if(strcmp(strings[i], search) == 0)
            return TRUE;
    }

    return FALSE;
}

int main(int argc, char *argv[])
{
    char *strings[] = { "foo", "bar", "baz", NULL };
    char *search  = "baz";
    int result = array_contains_string(strings, search);

    if(result)
        printf("The array does contain the string: %s\n", search);
    else
        printf("The array does not contain the string: %s\n", search);

    return 0;
}

Since searching (in particular linear searching) may take some time, we may want to change the function it into an asynchronous function running its primary task (the searching) in a child process. I can concisely express this primary concern in one single function:

#include <stdio.h>
#include <unistd.h>

pid_t array_contains_string_async(const char **strings, const char *search)
{
    pid_t pid = fork();

    if(pid == 0)
    {
        unsigned int i;

        for(i = 0; i < strlen(strings); i++)
        {
            if(strcmp(strings[i], search) == 0)
                _exit(0);
        }

        _exit(1);
    }

    return pid;
}

As may be noticed, the above function executes the same linear search procedure shown in the previous code fragment, with the following differences:

• The function forks a child process and carries out the search operation in the child process.
• Instead of returning a boolean value, it exits the child process with an exit status. By convention, a zero exit status indicates success while a non-zero exit status indicates failure.

We can capture the wait and exit status checking in a general utility function (procreact_wait_for_boolean()) that interprets the exit status of a child process as a boolean value (meaning that when a process exits with exit status 0 it returns TRUE and for any non-zero exit status, it returns FALSE):

#include <procreact_pid.h>

int main(int argc, char *argv[])
{
    char *strings[] = { "foo", "bar", "baz", NULL };
    char *search  = "baz";

    ProcReact_Status status;
    int result = procreact_wait_for_boolean(array_contains_string_async(strings, search), &status);

    if(status == PROCREACT_STATUS_OK)
    {
        if(result)
            printf("The array does contain the string: %s\n", search);
        else
            printf("The array does not contain the string: %s\n", search);
    }
    else
        fprintf(stderr, "The process terminated abnormally!\n");

    return 0;
}

As may be observed, by separating concerns and putting common operations into a library, we can accomplish the same result as the synchronous code fragment example, with relatively little overhead of boilerplate code.

Managing arbitrary output

The previously shown abstractions work well for functions returning a byte, boolean, or void-functions. However, it may also be desirable to implement asynchronous functions returning more complex data, such as strings or arrays of strings, for example:

#include <stdlib.h>
#include <string.h>

char *say_hello_to(const char *name)
{
    char *result = (char*)malloc(strlen(name) + 7 + 1);
    sprintf(result, "Hello %s!", name);
    return result;
}

The above function composes a string that greets a person with a given name. As explained earlier, implementing an asynchronous variant of the above function requires extra facilities to propagate the result back to the parent process, such as constructing a pipe, forcing us to do more housekeeping work.

In the previous examples, we were able to separate a task's primary concern into a function returning a PID and a function waiting and interpreting the exit status by using the PID reference. To manage complex data, we need to memorize more than just the PID -- we also need the pipe's file descriptors, store the buffered data, and the end result.

In some ways, a PID reference resembles another software abstraction -- a future in reactive programming or a promise in JavaScript. A future/promise is an object encapsulating a return value that will be provided at some point in the future.

We can encapsulate the entire housekeeping procedure for transferring and returning complex data in a ProcReact_Future struct:

#include <procreact_future.h>

ProcReact_Future say_hello_to_async(const char *name)
{
    ProcReact_Future future = procreact_initialize_future(procreact_create_string_type());

    if(future.pid == 0)
    {
        dprintf(future.fd, "Hello %s!", name);
        _exit(0);
    }

    return future;
}

The above code fragment looks similar to the synchronous function definition shown in the previous example, with the following differences:

• By constructing a ProcReact_Future struct, we no longer have to fork a child process and construct a pipe ourselves.
• The string composition step is carried out by the forked child process.
• Instead of returning a heap-allocated string, we write the resulting string to the write-end of the pipe provided by the future struct and we terminate the process by invoking the exit function call.

The procreact_initialize_future() function takes a parameter: a type, that is responsible for reading the output from the pipe and converting it into a representation of choice -- in this case a NUL-terminated string.

We can collect the return value of the function in the parent process by invoking the procreact_future_get() function:

ProcReact_Status status;
ProcReact_Future future = say_hello_to_async(name);
char *result = procreact_future_get(&future, &status);

if(status == PROCREACT_STATUS_OK && result != NULL)
    printf("%s\n", result);
else
    fprintf(stderr, "Some error occured!\n");

The procreact_future_get() function (that looks similar to a Future's .get() method or Promise's .then() method) takes care of reading from the pipe, buffering the output, converting the output to a string, waiting for the child process to terminate and closing the obsolete file descriptors.

Furthermore, analogous to a Future or Promise, when the retrieval function gets invoked for a second time, it will return its cached value instead of reading from the pipe again.

Orchestrating collections of processes

With concerns well separated, orchestration of collections of process also becomes easier. For example, we may want to execute multiple invocations to the following function in parallel:

ProcReact_Future return_count_async(unsigned int count)
{
    ProcReact_Future future = procreact_initialize_future(procreact_create_string_type());

    if(future.pid == 0)
    {
        dprintf(future.fd, "%u", count);
        _exit(0);
    }

    return future;
}

The purpose of the function shown above is to simply return a string presentation of a given numeric counter value.

As with the abstraction facilities shown previously (such as ProcReact_Future), we can also create similar abstractions for orchestrating collections of processes including processes whose output need to be captured:

ProcReact_FutureIterator iterator = procreact_initialize_future_iterator(has_next_count,
  next_count_process,
  complete_count_process,
&data);

The above function invocation: procreact_initialize_future_iterator() configures an ProcReact_FutureIterator struct. It takes the following parameters:

• A pointer to a function that indicates whether there is a next element in the collection.
• A pointer to a function that invokes the next process in the collection.
• A pointer to a function that gets invoked when a process completes.
• A void-pointer referring to an arbitrary data structure that gets passed to all functions above.

If I would like to invoke this function 5 times to say, count from 1 to 5, I can encapsulate the properties of this iteration process in the following data structure:

typedef struct
{
    unsigned int index;
    unsigned int amount;
    int success;
    char **results;
    unsigned int results_length;
}
IteratorData;

and compose the following instance from it:

IteratorData data = { 0, 5, TRUE, NULL, 0 };

The above struct maintains an index value indicating which element it currently processing, the amount value holds the amount of iterations that need to be executed, success is a boolean status flag indicating whether the iterations have all succeeded or not, and the results variable corresponds to an array of strings capturing the output of each function invocation.

The following function can be used to check whether we have completed the iteration or not:

static int has_next_count_process(void *data)
{
    IteratorData *iterator_data = (IteratorData*)data;
    return iterator_data->index < iterator_data->amount;
}

The following function executes each successive iteration step:

static ProcReact_Future next_count_process(void *data)
{
    IteratorData *iterator_data = (IteratorData*)data;
    ProcReact_Future future = return_count_async(iterator_data->index + 1);
    iterator_data->index++;
    return future;
}

The above function increases the index, invokes the function with the index as a parameter, and increases the index value.

The following function gets invoked when a process' execution finishes:

static void complete_count_process(void *data, ProcReact_Future *future, ProcReact_Status status)
{
    IteratorData *iterator_data = (IteratorData*)data;

    if(status == PROCREACT_STATUS_OK && future->result != NULL)
    {
        iterator_data->results = (char**)realloc(iterator_data->results, (iterator_data->results_length + 1) * sizeof(char*));
        iterator_data->results[iterator_data->results_length] = future->result;
        iterator_data->results_length++;
    }the 
    else
        iterator_data->success = FALSE;
}

The above function checks the status of the function invocation and captures the results that each function returns. When a process completes successfully (i.e. it does not terminate abnormally and provides a non-NULL result), it appends the result to the results array. In case of a failure, it sets the overall status flag of the iterator to FALSE.

With all iteration aspects abstracted away into a ProcReact_Future struct, we can execute the following function to do all iteration steps in parallel:

procreact_fork_in_parallel_buffer_and_wait(&iterator);

We can also limit the amount of processes that are allowed to run concurrently to a specific value, e.g. 2:

procreact_fork_buffer_and_wait_in_parallel_limit(&iterator, 2);

After executing all iterations, we can consult the data struct to figure out whether the iterations have succeeded and what their results are.

Asynchronously orchestrating collections of processes

The previous two collection examples are executed synchronously. This means that while the execution of each function that retrieves an element is done asynchronously, the overall iteration task blocks the parent process until it completes, which is not always desirable.

A possible solution to make iterations asynchronous is to fork another process and iterate over the collection in the child process, but this introduces another challenge when the collected data needs to be returned to the parent.

Instead of performing all iteration steps as a whole we can also control each iteration step ourselves. For example, the following command-line invocation executes a single iteration step that composes a ProcReact_Future struct:

if(procreact_spawn_next_future(&iterator))
    printf("Spawned a process and we have more of them!\n");
else
    printf("All processes have been spawned\n");

We can also run a each buffer iteration step ourselves and integrate that function call into a program's main loop:

while(TRUE)
{
    unsigned int running_processes = procreact_buffer(&iterator);

    if(running_processes == 0)
    {
        /* This indicates that there are no running processes anymore */
        /* You could do something with the end result here */
    }

    /* Do other stuff in the main loop */
}

The above code fragment allows us to evaluate processes' statuses and buffer their outputs without blocking the main loop.

Summary

In this blog post, I have described a number of common housekeeping tasks I typically need to implement when developing multi-process applications and I have described an API that abstracts over these common house keeping tasks.

To summarize, when we intend to execute task or a collection of tasks, we can use one of the following data structures, depending whether we need to evaluate a single value or a collection of values, and whether the type of the value is simple (e.g. a boolean, byte, or void) or complex (e.g. strings, arrays of strings, etc.):

	One	Many
Simple	pid_t	ProcReact_PidIterator
Complex	ProcReact_Future&ltProcReact_Type>	ProcReact_FutureIterator

Processing collections of tasks for each type can be done either synchronously or asynchronously, by picking the appropriate utility functions:

	Synchronous	Asynchronous
Simple	procreact_fork_in_parallel_and_wait() procreact_fork_and_wait_in_parallel_limit()	procreact_register_signal_handler() procreact_spawn_next_pid() procreact_complete_all_finished_processes()
Complex	procreact_fork_in_parallel_buffer_and_wait() procreact_fork_buffer_and_wait_in_parallel_limit()	procreact_spawn_next_future() procreact_buffer()

Motivation: Disnix

After reading this blog post, you may probably wonder why I have developed these abstractions? My main motivation is to organize Disnix's codebase in a better way.

Besides being a distributed deployment tool, all deployment activities that Disnix carries out (package management operations, state management operations, and communications) are carried out by external processes, for exactly the same reasons mentioned in the introduction.

Before deriving the abstractions described in this blog post, all process coordination in Disnix was hand coded -- Disnix's code was cluttered by boilerplate code doing output buffering, concurrency limiting, waiting, and resource allocation. As a result, some areas of the code were quite hard to read and difficult to extend. Moreover, it was also quite hard to ensure that the code is free from some serious issues, such as potential buffer overflows.

After restructuring the code to use these abstractions (between Git revisions 595c1ec and 4ba5e3d), I have managed to significantly reduce the amount of code. For example, the methods.c module (an RPC interface for all deployment operations that Disnix can execute remotely) previously consisted of 1976 LOC. After refactoring, it consists of only 488 LOC.

Moreover, the disnix-build utility's main module (build.c) size has been reduced from 250 LOC to 153 LOC. Each command-line utility that executes a deployment task (13 in total) each have at least 100 fewer lines lines of code.

In addition to reducing the size of many modules, I have also accomplished the following:

• Better separation of concerns. I have managed to clearly separate all asynchronous functions spawning processes into three separate libraries: (libpkgmgmt for package manager, libstatemgmt for state management and libinterface for communication) considerably simplifying the toolset's architecture.
• Performance improvements. Some tools (disnix-capture-infra and disnix-query) were still executing their tasks sequentially and were difficult to optimize. With these new abstractions, parallelizing their tasks became quite simple.
• Better error reporting. In older versions of Disnix, it was difficult to relate error messages to target machines. With the abstractions described in this blog post, implementing a translation process became much easier. As a result, all tools can now clearly report the origins of an error.

Discussion

Although the abstractions described in this blog post have allowed me to structure Disnix's code in a much better way, they are not a solution for everything.

For example, the ProcReact_Future abstraction buffers the process' output, which is quite inadequate for processes transferring large quantities of data. Moreover, each function invocation implies a fork. On a very large scale, this could become quite expensive, as it takes time to set up a child process and memory for allocating the Copy-on-Write (COW) table. In theory, we could also make it possible to reuse spawned processes among function invocations to optimize this, but currently no such optimizations have been implemented yet.

Furthermore, processes are not the only solution for implementing concurrent applications. When frequent interaction is required between parent and child, it may be better to use threads as their memory is shared (at the same time, this also has disadvantages). A major disadvantage of processes (compared to threads) is that all communication data needs to be serialized, transferred and deserialized, introducing quite a bit of communication overhead.

Apart from processes and threads, it is also possible to use a single threaded event loop, in which the programmer has the obligation to divide bigger tasks into small tasks. For example, this model is prominently used in Node.js. The advantage of this approach is that each additional concurrent task does not require the allocation of additional resources. This works, for example, well for applications that are mostly I/O-bound (I/O is typically thousands of times slower than a CPU).

The disadvantages of such an application organization is that it becomes a programmer's obligation to ensure that the main thread never blocks and that the code remains structured properly. Moreover, "context-switching" between single threaded tasks is more expensive than a context switch on CPU-level, making it quite inadequate for computationally intensive applications. Finally, the operating system cannot automatically divide tasks over multiple CPU cores.

Availability

libprocreact is part of the latest development version of Disnix. Additionally, I have isolated the library code and created a separate GitHub repository for it.

Although libprocreact can be used independently from Disnix, I currently do not have any plans to make it a fully fledged general purpose library. At the moment, it is not supposed to be used for any other use cases than Disnix's.

A while ago, I have ported the simple-xmpp library from the Node.js ecosystem to Appcelerator Titanium to enrich our company's product line with chat functionality. In addition, I have created a bare bones example app that exposes most of the library's features.



Although I am not doing that much front-end development these days, nor consider myself to be a Titanium-guru, I have observed that it is quite challenging to keep your app's code and organization clean.

In this blog post, I will report on my development experiences and describe the architecture that I have derived for the example chat application.

The Model-View-Controller (MVC) architectural pattern

Keeping the code of an end-user application sane is not unique to mobile applications or a specific framework, such as Titanium -- it basically applies to any system with a graphical user interface including desktop applications and web applications.

When diving into the literature or just by searching on the Internet, then you will most likely stumble upon a very common "solution" -- there is the Model-View-Controller (MVC) architectural pattern that can be used as a means to keep your system structured. It is a generically applicable pattern implemented by many kinds of libraries and frameworks for all kinds of domains, including the mobile application space.

The idea behind this pattern is that a system will be separated in three distinct concerns: the model, the view and the controller. The meaning of these concerns are somewhat ambiguously defined. For example, the design patterns book written by the gang of four (Erich Gamma, Richard Helm, Ralph Johnson and John Vlissides) says:

MVC consists of three kinds of objects. The Model is the application object, the View is its screen presentation, and the Controller defines the way the user interface reacts to user input.

The "problem" I have with the above explanation is that it is a bit difficult to grasp the meaning of an "application object". Moreover, the definition of the controller object used in the explanation above, states that it only has a relation with a user interface (a.k.a. the view) while I could also think of many scenarios in which external events are involved without invoking the user interface. I have no idea how to categorize these kinds of interactions by looking at the above description.

The paper that the book cites: "A Cookbook for Using the Model-View-Controller User Interface Paradigm in Smalltalk-80" (written by: Glenn E. Krasner and Stephen T. Pope) provides more detailed definitions. For example, it defines the model as:

The model of an application is the domain-specific software simulation or implementation of the application's central structure.

I particularly find the term "domain-specific" important -- it suggests that a model should encapsulate what matters to the problem domain, without any obfuscations of things not related to it, for example, user interface components.

The paper defines the view as follows:

In this metaphor, views deal with everything graphical: they request data from their model, and display the data

The above definition suggests that views are everything about presentation of objects belonging to the model.

Finally, the paper defines controllers as follows:

Controllers contain the interface between their associated models and views and the input devices (e.g., keyboard, pointing device, time)

In contrast to the design patterns book's definition of a controller, this definition also suggests that a controller has a relationship with the model. Moreover, it does not say anything about interactions with a physical user. Instead, it refers to input devices.

Although the paper provides more detailed definitions, it still remains difficult to draw a hard line from my perspective. For example, what is the scope of MVC? Should it apply to an entire system, or can it also be applied to components of which a system consists?

For example, in an earlier blog post, I wrote a blog post about some of my experiences with web development in which I have developed a simple library MVC-based library managing the layouts of web applications. The model basically encapsulates the structure of a web applications from an abstract point of view, but it only applies to a specific sub concern, not the system as a whole.

Despite its unclarities and ambiguities, I still think MVC makes sense, for the following reasons:

• View and controller code clutters the model with obfuscations making it much harder to read and maintain.
• There are multiple ways to present an object visually. With a clear separation between a model and view this becomes much more flexible.
• In general, more compact modules (in terms of lines of code) is many ways always better than having many lines of code in one module (for example for readability and maintainability). Separation of concerns stimulates reduction of the size of modules.

The Titanium and Alloy frameworks

As explained earlier, I have implemented the chat example app using the Titanium and Alloy frameworks.

Titanium is a framework targeting multiple mobile app platforms (e.g. Android, iOS, Windows and mobile web applications) using JavaScript as an implementation language providing a unified API with minor platform differences. In contrast to platforms such as Java, Titanium is not a write once, run anywhere approach, but a code reuse approach -- according to their information between 60 and 90% of the code can be reused among target platforms.

Moreover, the organization of Titanium's API makes a clear difference between UI and non-UI components, but does not impose anyone to strictly follow an MVC-like organization while implementing an application.

Alloy is a declarative MVC-framework that wraps around Titanium. To cite the Alloy documentation:

Alloy utilizes the model-view-controller (MVC) pattern, which separates the application into three different components:

• Models provide the business logic, containing the rules, data and state of the application.
• Views provide the GUI components to the user, either presenting data or allowing the user to interact with the model data.
• Controllers provide the glue between the model and view components in the form of application logic.

(As may be noticed, the above description introduces yet another slightly different interpretation of the MVC architectural pattern.)

The Alloy framework uses a number of very specific technologies to realize a MVC organization:

• For the models, it uses the backbone.js framework's model instances to organize the application's data. The framework supports automatic data binding to view components.
• Views use an XML data encoding capturing the static structure of the view. Moreover, the style of each view is captured in TSS stylesheet (having many similarities with CSS).
• The controllers are CommonJS modules using JavaScript as an implementation language.

Furthermore, the directory structure of an Alloy application also reflects separation of concerns. For example, each unit of an application stores each concern in a separate directory and file. For example, in the chat app, we can implement each concern of the contacts screen by providing the following files:

./app/views/contacts.xml
./app/controllers/contacts.js
./app/styles/contacts.tss

The above files reflect each concern of the contacts screen, such as the view, the controller and the style.

In addition to defining models, views, styles and controllers on unit-level, the app unit captures general properties applying of the app.

Organizing the example chat app

Despite the fact that the Alloy framework facilitates separation of concerns in some degree, I still observed that keeping the app's code structure sane remains difficult.

Constructing views

An immediate improvement of Alloy over plain Titanium is that the view code in XML is much better to read than constructing UI components in JavaScript -- the nesting of XML elements reflects the structure of the UI. Furthermore, the style of the UI elements can be separated from the layout improving the readability even further.

For example, the following snippet shows the structure of the login screen:

<Alloy>
<Window class="container">
<ScrollView>
<View>
<Label>Web socket URL</Label>
<TextField id="url" hintText="ws://localhost:5280/websocket/" />
</View>
<View>
<Label>Username</Label>
<TextField id="username" hintText="sander" />
</View>
<View>
<Label>Domain name</Label>
<TextField id="domain" hintText="localhost" />
</View>
<View>
<Label>Resource</Label>
<TextField id="resource" hintText="" />
</View>
<View>
<Label>Password</Label>
<TextField id="password" passwordMask="true" hintText="" />
</View>
<Button onClick="doConnect">Connect</Button>
</ScrollView>
</Window>
</Alloy>

As may be observed, by reading the above code fragment, it becomes quite obvious that we have a window with a scroll view inside. Inside the scroll view, we have multiple views containing a label and text field pair, allowing users to provide their login credentials.

Although implementing most screens in XML is quite straight forward as their structures are quite static, I have noticed that Alloy's technologies are not particularly useful to dynamically compose screen structures, such as the contacts overview that displaying a row for each contact -- the structure of this table changes whenever a new contact gets added or an existing contact removed.

To dynamically compose a screen, I still need to write JavaScript code in the screen's controller. Furthermore, UI elements composed in JavaScript do not take the style settings of the corresponding TSS file into account. As a result, we need to manually provide styling properties while composing the dynamic screen elements.

To keep the controller's code structured and avoiding code repetition, I have encapsulated the construction of table rows into functions.

Notifying views for changes

Another practical issue I ran into is updating the UI components when something changes, such as a receiving a text messaging or an updated status of a contact. An update to a backbone model automatically updates the attached view components, but for anything that is not backbone-based (such as XMPP's internal roster object) this will not work.

I ended up implementing my own custom non-backbone based data model, with my own implementation of the Observer design pattern -- each object in the data model inherits from the Observable prototype providing an infrastructure for observers to register and unregister themselves for notifications. Each view registers itself as an observer to the corresponding model object to update themselves.

The app's architecture

In the end, this is the architecture of the example chat app that I came up with:


The UML diagram shows the following aspects:

• All classes can be divided into four concerns: controllers, views, models, and utility classes. The observer infrastructure, for example, does in my opinion not belong to any of the MVC-categories, because they are cross cutting.
• The XMPPEventHandler is considered to be a controller. Despite not triggered by human actions, I still classify it as such. The event handler's only responsibility is to update the corresponding model objects once an event has been received from the XMPP server, such as a chat message.
• All model objects inherit from a custom-made Observable prototype so that views can register and unregister themselves for update notifications.
• Views extract information from the model objects to display. Furthermore, each view has its own controller responding to user input, such as button clicks.

Lessons learned

In addition to porting an XMPP library from the Node.js ecosystem to Titanium, I have also observed some recurring challenges when implementing the test application and keeping it structured. Despite the fact that the Alloy framework is MVC-based, it does not guarantee that your application's organization remains structured.

From my experiences, I have learned the following lessons:

• The roles of each concern in MVC are not well defined, so you need to give your own interpretation to it. For example, I would consider any controller to be an object responding to external events, regardless whether they have been triggered by humans or external systems. By following this interpretation, I ended up implementing the XMPP event handler as a controller.
• Similarly for the models -- the purpose of backbone.js models is mostly to organize data, but a model is more than just data -- from my perspective, the model encapsulates domain knowledge. This also means that non-backbone objects belong to this domain. The same thing applies to non-data objects, such as functions doing computations.
• You always have to look at your structure from an aesthetic point of view. Does it makes sense? Is it readable? Can it be simplified?
• Finally, do not rely on a framework or API to solve all your problems -- study the underlying concepts and remain critical, as a framework does not always guarantee that your organization will be right.
  
  Within the scope of Titanium/Alloy the problem is that models only make sense if you use backbone models. Using XML markup+TSS for views only make sense if your screen structure is static. The most logical outcome is to put all missing pieces that do not classify themselves into these categories into a controller, but that is probably the most likely reason why your code becomes a mess.

As a final note, the lessons learned do not apply to mobile applications or Titanium/Alloy only -- you will find similar challenges in other domains such as web applications and desktop applications.

In two earlier blog posts, I have described Dynamic Disnix, an experimental framework enabling self-adaptive redeployment on top of Disnix. The purpose of this framework is to redeploy a service-oriented system whenever the conditions of the environment change, so that the system can still meet its functional and non-functional requirements.

An important category of events that change the environment are machines that crash and disappear from the network -- when a disappearing machine used to host a crucial service, a system can no longer meet its functional requirements. Fortunately, Dynamic Disnix is capable of automatically responding to such events by deploying the missing components elsewhere.

Although Dynamic Disnix supports the recovery of missing services, there is one particular kind of failure I did not take into account. In addition to potentially crashing target machines that host the services of which a service-oriented systems consist, the coordinator machine that initiates the deployment process and stores the deployment state could also disappear. When the deployment state gets lost, it is no longer possible to reliably update the system.

In this blog post, I will describe a new addition to the Disnix toolset that can be used to cope with these kinds of failures by reconstructing a coordinator machine's deployment configuration from the meta data stored on the target machines.

The Disnix upgrade workflow

As explained in earlier blog posts, Disnix requires three kinds of deployment models to carry out a deployment process: a services model capturing the components of which a system consists, an infrastructure model describing the available target machines and their properties, and a distribution model mapping services in the services model to target machines in the infrastructure model. By writing instances of these three models and running the following command-line instruction:

$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix

Disnix will carry out all activities necessary to deploy the system: building the services and its intra-dependencies from source code, distributing the services and its intra-dependencies, and activating all services in the right order.

When changing any of the models and running the same command-line instruction again, Disnix attempts to upgrade the system by only rebuilding the aspects that have changed, and only deactivating the obsolete services and activating new services.

Disnix (as well as other Nix-related tools) attempt to optimize a redeployment process by only executing the steps that are required to reach a new deployment state. In Disnix, the building and distribution steps are optimized due to the fact that every package is stored in isolation the Nix store in which each package has a unique filename with a hash prefix, such as:

/nix/store/acv1y1zf7w0i6jx02kfa6gxyn2kfwj3l-firefox-48.0.2

As explained in a number of earlier blog posts, the hash prefix (acv1y1zf7w0i6jx02kfa6gxyn2kfwj3l...) is derived from all inputs used to build the package including its source code, build script, and libraries that it links to. That, for example, means that if we upgrade a system and nothing to the any of inputs of Firefox changes, we get an identical hash and if such a package build already exists, we do not have to build or transfer the package from an external site again.

The building step in Disnix produces a so-called low-level manifest file that is used by tools executing the remaining deployment activities:

<?xml version="1.0"?>
<manifest version="1">
<distribution>
<mapping>
<profile>/nix/store/aiawhpk5irpjqj25kh6ah6pqfvaifm53-test1</profile>
<target>test1</target>
</mapping>
</distribution>
<activation>
<mapping>
<dependsOn>
<dependency>
<target>test1</target>
<container>process</container>
<key>d500194f55ce2096487c6d2cf69fd94a0d9b1340361ea76fb8b289c39cdc202d</key>
</dependency>
</dependsOn>
<name>nginx</name>
<service>/nix/store/aa5hn5n1pg2qbb7i8skr6vkgpnsjhlns-nginx-wrapper</service>
<target>test1</target>
<container>wrapper</container>
<type>wrapper</type>
<key>da8c3879ccf1b0ae34a952f36b0630d47211d7f9d185a8f2362fa001652a9753</key>
</mapping>
</activation>
<targets>
<target>
<properties>
<hostname>test1</hostname>
</properties>
<containers>
<mongo-database/>
<process/>
<wrapper/>
</containers>
<system>x86_64-linux</system>
<numOfCores>1</numOfCores>
<clientInterface>disnix-ssh-client</clientInterface>
<targetProperty>hostname</targetProperty>
</target>
</targets>
</manifest>

The above manifest file contains the following kinds of information:

• The distribution element section maps Nix profiles (containing references to all packages implementing the services deployed to the machine) to target machines in the network. This information is used by the distribution step to transfer packages from the coordinator machine to a target machine.
• The activation element section contains elements specifying which service to activate on which machine in the network including other properties relevant to the activation, such as the type plugin that needs to be invoked that takes care of the activation process. This information is used by the activation step.
• The targets section contains properties of the machines in the network and is used by all tools that carry out remote deployment steps.
• There is also an optional snapshots section (not shown in the code fragment above) that contains the properties of services whose state need to be snapshotted, transferred and restored in case their location changes.

When a Disnix (re)deployment process successfully completes, Disnix stores the above manifest as a Disnix coordinator Nix profile on the coorindator machine for future reference with the purpose to optimize the successive upgrade step -- when redeploying a system Disnix will compare the generated manifest with the previously deployed generated instance and only deactivate services that have become obsolete and activating services that are new, making upgrades more efficient than fresh installations.

Unfortunately, when the coordinator machine storing the manifests gets lost, then also the deployment manifest gets lost. As a result, a system can no longer be reliably upgraded -- without deactivating obsolete services, newly deployed services may conflict with services that are already running on the target machines preventing the system from working properly.

Reconstructible manifests

Recently, I have modified Disnix in such a way that the deployment manifests on the coordinator machine can be reconstructed. Each Nix profile that Disnix distributes to a target machine includes a so-called profile manifest file, e.g. /nix/store/aiawhpk5irpjqj25kh6ah6pqfvaifm53-test1/manifest. Previously, this file only contained the Nix store paths to the deployed services and was primarily used by the disnix-query tool to display the installed set of services per machines.

In the latest Disnix, I have changed the format of the profile manifest file to contain all required meta data so that the the activation mappings can be reconstructed on the coordinator machine:

stafftracker
/nix/store/mi7dn2wvwvpgdj7h8xpvyb04d1nycriy-stafftracker-wrapper
process
process
d500194f55ce2096487c6d2cf69fd94a0d9b1340361ea76fb8b289c39cdc202d
false
[{ target = "test2"; container = "process"; _key = "4827dfcde5497466b5d218edcd3326327a4174f2b23fd3c9956e664e2386a080"; } { target = "test2"; container = "process"; _key = "b629e50900fe8637c4d3ddf8e37fc5420f2f08a9ecd476648274da63f9e1ebcc"; } { target = "test1"; container = "process"; _key = "d85ba27c57ba626fa63be2520fee356570626674c5635435d9768cf7da943aa3"; }]

The above code fragment shows a portion of the profile manifest. It has a line-oriented structure in which every 7 lines represent the properties of a deployed service. The first line denotes the name of the service, second line the Nix store path, third line the Dysnomia container, fourth line the Dysnomia type, fifth line the hash code derived of all properties, sixth line whether the attached state must be managed by Disnix and the seventh line an encoding of the inter-dependencies.

The other portions of the deployment manifest can be reconstructed as follows: the distribution section can be derived by querying the Nix store paths of the installed profiles on the target machines, the snapshots section by checking which services have been marked as stateful and the targets section can be directly derived from a provided infrastructure model.

With the augmented data in the profile manifests on the target machines, I have developed a tool named disnix-reconstruct that can reconstruct a deployment manifest from all the meta data the manifests on the target machines provide.

I can now, for example, delete all the deployment manifest generations on the coordinator machine:

$ rm /nix/var/nix/profiles/per-user/sander/disnix-coordinator/*

and reconstruct the latest deployment manifest, by running:

$ disnix-reconstruct infrastructure.nix

The above command resolves the full paths to the Nix profiles on the target machines, then downloads their intra-dependency closures to the coordinator machine, reconstructs the deployment manifest from the profile manifests and finally installs the generated deployment manifest.

If the above command succeeds, then we can reliably upgrade a system again with the usual command-line instruction:

$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix

Extending the self-adaptive deployment framework

In addition to reconstructing deployment manifests that have gone missing, disnix-reconstruct offers another benefit -- the self-adaptive redeployment framework described in the two earlier blog posts is capable of responding to various kinds of events, including redeploying services to other machines when a machine crashes and disappears from the network.

However, when a machine disappears from the network and reappears at a later point in time, Disnix no longer knows about its configuration. When such a machine reappears in the network, this could have disastrous results.

Fortunately, by adding disnix-reconstruct to the framework we can solve this issue:


As shown in the above diagram, whenever a change in the infrastructure is detected, we reconstruct the deployment manifest so that Disnix knows which services are deployed to it. Then when the system is being redeployed, the services on the reappearing machines can also be upgraded or undeployed completely, if needed.

The automatic reconstruction feature can be used by providing the --reconstruct parameter to the self adapt tool:

$ dydisnix-self-adapt -s services.nix -i infrastructure.nix -q qos.nix \
  --reconstruct

Conclusion

In this blog post, I have described the latest addition to Disnix: disnix-reconstruct that can be used to reconstruct the deployment manifest on the coordinator machine from meta data stored on the target machines. With this addition, we can still update systems if the coordinator machine gets lost.

Furthermore, we can use this addition in the self-adaptive deployment framework to deal with reappearing machines that already have services deployed to them.

Finally, besides developing disnix-reconstruct, I have reached another stable point. As a result, I have decided to release Disnix 0.7. Consult the Disnix homepage for more information.

In a number of previous blog posts, I have described node2nix, a tool that can be used to automatically integrate NPM packages into the Nix packages ecosystem. The biggest challenge in making this integration possible is the fact that NPM does dependency management in addition to build management -- NPM's dependency management properties conflict with Nix's purity principles.

Dealing with a conflicting dependency manager is quite simple from a conceptual perspective -- you must substitute it by a custom implementation that uses Nix to obtain all required dependencies. The remaining responsibilities (such as build management) are left untouched and still have to be carried out by the guest package manager.

Although conceptually simple, implementing such a substitution approach is much more difficult than expected. For example, in my previous blog posts I have described the following techniques:

• Extracting dependencies. In addition to the package we intend to deploy with Nix, we must also include all its dependencies and transitive dependencies in the generation process.
• Computing output hashes. In order to make package deployments deterministic, Nix requires that the output hashes of downloads are known in advance. As a result, we must examine all dependencies and compute their corresponding SHA256 output hashes. Some NPM projects have thousands of transitive dependencies that need to be analyzed.
• Snapshotting versions. Nix uses SHA256 hash codes (derived from all inputs to build a package) to address specific variants or versions of packages whereas version specifiers in NPM package.json configurations are nominal -- they permit version ranges and references to external artifacts (such as Git repositories and external URLs).
  
  For example, a version range of >= 1.0.3 might resolve to version 1.0.3 today and to version 1.0.4 tomorrow. Translating a version range to a Nix package with a hash code identifier breaks the ability for Nix to guarantee that a package with a specific hash code yields a (nearly) bit identical build.
  
  To ensure reproducibility, we must snapshot the resolved version of these nominal dependency version specifiers (such as a version range) at generation time and generate the corresponding Nix expression for the resulting snapshot.
• Simulating shared and private dependencies. In NPM projects, dependencies of a package are stored in the node_modules/ sub folder of the package. Each dependency can have private dependencies by putting them in their corresponding node_modules/ sub folder. Sharing dependencies is also possible by placing the corresponding dependency in any of the parent node_modules/ sub folders.
  
  Moreover, although this is not explicitly advertised as such, NPM implicitly supports cyclic dependencies and is able cope with them because it will refuse to install a dependency in a node_modules/ sub folder if any parent folder already provides it.
  
  When generating Nix expressions, we must replicate the exact same behaviour when it comes to private and shared dependencies. This is particularly important to cope with cyclic dependencies -- the Nix package manager does not allow them and we have to break any potential cycles at generation time.
• Simulating "flat module" installations. In NPM versions older than 3.0, every dependency was installed privately by default unless a shared dependency exists that fits within the required version range.
  
  In newer NPM versions, this strategy has been reversed -- every dependency will be shared as much as possible until a conflict has been encountered. This means that we have to move dependencies as high up in the node_modules/ folder hierarchy as possible which is an imperative operation -- in Nix this is a problem, because packages are cannot be changed after they have been built.
  
  To cope with flattening, we must compute the implications of flattening the dependency structure in advance at generation time.

With the above techniques it is possible construct a node_modules/ directory structure having a nearly identical structure that NPM would normally compose with a high degree of accuracy.

Impure version specifiers

Even if it would be possible to reproduce the node_modules/ directory hierarchy with 100% accuracy, there is another problem that remains -- some version specifiers always trigger network communication regardless whether the dependencies have been provided or not, such as:

[
  { "node2nix": "latest" }
, { "nijs": "git+https://github.com/svanderburg/nijs.git#master" }
, { "prom2cb": "github:svanderburg/prom2cb" }
]

When referring to tags or Git branches, NPM is unable to determine to which version a package resolves. As a consequence, it attempts to retrieve the corresponding packages to investigate even when a compatible version in the node_modules/ directory hierarchy already exists.

While performing package builds, Nix takes various precautions to prevent side effects from influencing builds including network connections. As a result, an NPM package deployment will still fail despite the fact that a compatible dependency has already been provided.

In the package builder Nix expression provided by node2nix, I used to substitute these version specifiers in the package.json configuration files by a wildcard: '*'. Wildcards used to work fine for old Node.js 4.x/NPM 2.x installations, but with NPM 3.x flat module installations they became another big source of problems -- in order to make flat module installations work, NPM needs to know to which version a package resolves to determine whether it can be shared on a higher level in the node_modules/ folder hierarchy or not. Wildcards prevent NPM from making these comparisons, and as a result, some package deployments fail that did not use to fail with older versions of NPM.

Pinpointing version specifiers

In the latest node2nix I have solved these issues by implementing a different substitution strategy -- instead of substituting impure version specifiers by wildcards, I pinpoint all the dependencies to the exact version numbers to which these dependencies resolve. Internally, NPM addresses all dependencies by their names and version numbers only (this also has a number of weird implications, because it disregards the origins of these dependencies, but I will not go into detail on that).

I got the inspiration for this pinpointing strategy from the yarn package manager (an alternative to NPM developed by Facebook) -- when deploying a project with yarn, yarn pinpoints the installed dependencies in a so-called yarn.lock file so that package deployments become reproducible when a system is deployed for a second time.

The pinpointing strategy will always prevent NPM from consulting external resources (under the condition that we have provided the package by our substitute dependency manager first) and always provide version numbers for any dependency so that NPM can perform flat module installations. As a result, the accuracy of node2nix with newer versions of NPM has improved quite a bit.

Availability

The pinpointing strategy is part of the latest node2nix that can be obtained from the NPM registry or the Nixpkgs repository.

One month ago, I have given a talk about node2nix at FOSDEM 2017 summarizing the techniques discussed in my blog posts written so far. For convenience, I have embedded the slides into this web page:

It has been a while since I wrote my last blog post. In the last couple of months, I have been working on many kinds of things, such as resurrecting the most important components of my personal web framework (that I have developed many years ago) and making them publicly available on GitHub.

There are a variety of reasons for me to temporarily switch back to this technology area for a brief period of time -- foremost, there are a couple of web sites still using pieces of my custom framework, such as those related to my voluntary work. I recently had to make changes, mostly maintenance-related, to these systems.

The funny thing is that most people do not consider me a web development (or front-end) person and this has an interesting history -- many years ago (before I started doing research) I always used to refer to "web technology" as one of my main technical interests. Gradually, my interest started to fade, up until the point that I stopped mentioning it.

I started this blog somewhere in the middle of my research, mainly to provide additional practical information. Furthermore, I have been using my blog to report on everything I do open-source related.

If I would have started this blog several years earlier, then many articles would have been related to web technology. Back then, I have spent considerable amounts of time investigating techniques, problems and solutions. Retrospectively, I regret that it took me so long to make writing a recurring habit as part of my work -- many interesting discoveries were never documented and have become forgotten knowledge.

In this blog post, I will reflect over my web programming experiences, describe some of the challenges I used to face and what solutions I implemented.

In the beginning: everything looked great

I vividly remember the early days in which I was just introduced to the internet, somewhere in the mid 90s. Around that time Netscape Navigator 2.0 was still the dominant and most advanced web browser.

Furthermore, the things I could do on the internet were very limited -- today we are connected to the internet almost 24 hours a day (mainly because many of us have smart phones allowing us to do so), but back then I only had a very slow 33K6 dial-up modem and internet access for only one hour a week.

Aside from the fact that it was quite an amazing experience to be connected to the world despite these limitations, I was also impressed by the underlying technology to construct web sites. It did not take long for me to experiment with these technologies myself, in particular HTML.

Quite quickly I was able to construct a personal web page whose purpose was simply to display some basic information about myself. Roughly, what I did was something like this:

<html>
<head>
<title>My homepage</title>
</head>

<body bgcolor="#ff0000" text="#000000">
<h1>Hello world!</h1>

<p>
      Hello, this is my homepage.
</p>
<p>
<img src="image.jpg" alt="Image">
</p>
</body>
</html>

It was simply a web page with a red colored background displaying some text, hyperlinks and images:


You may probably wonder what is so special about building a web page with a dreadful background color, but before I was introduced to web technology, my programming experience was limited to various flavours of BASIC (such as Commodore 64, AMOS, GW and Quick BASIC), Visual Basic, 6502 assembly and Turbo Pascal.

Building user interfaces with these kind of technologies was quite tedious and somewhat impractical compared to using web technology -- for example, you had to programmatically define your user interface elements, size them, position them, define style properties for each individual element and programming event handlers to respond to user events, such as mouse clicks.

With web technology this suddenly became quite easy and convenient -- I could now concisely express what I wanted and the browser took care of the rendering parts.

Unknowingly, I was introduced to a discipline called declarative programming -- I could describe what I wanted as opposed to specifying how to do something. Writing applications declaratively had all kinds of advantages beyond the ability to express things concisely.

For example, because HTML code is high-level (not entirely, but is supposed to be), it does not really matter much what browser application you use (or underlying platform, such as the operating system) making your application quite portable. Rendering a paragraph, a link, image or button can be done on many kinds of different platforms from the same specification.

Another powerful property is that your application can degrade gracefully. For example, when using a text-oriented browser, your web application should still be usable without the ability to display graphics. The alternate text (alt) attribute of the image element should ensure that the image description is still visible. Even when no visualization is possible (e.g. for visually impaired people), you could, for example, use a Text to Speech system to interpret your pages' content.

Moreover, the introduction of Cascading Style Sheets (CSS) made it possible to separate the style concern from the page structure and contents making the code of your web page much more concise. Before CSS, extensive use of presentational tags could still make your code quite messy. Separation of the style concern also made it possible to replace the stylesheet without modifying the HTML code to easily give your page a different appearance.

To deal with visual discrepancies between browser implementations, HTML was standardized and standards-mode rendering was introduced when an HTML doctype was added to a HTML file.

What went wrong?

I have described a number of appealing traits of web technology in the previous section -- programming applications declaratively from a high level perspective, separation of concerns, portability because of high-level abstractions and standardization, the ability to degrade gracefully and conveniently making your application available to the world.

What could possibly be the reason to lose passion while this technology has so many compelling properties?

I have a long list of anecdotes, but most of my reasons can be categorized as follows:

There are many complex additional concerns

Most web technologies (e.g. HTML, CSS, JavaScript) provide solutions for the front-end, mainly to serve and render pages, but many web applications are much more than simply a collection of pages -- they are in fact complex information systems.

To build information systems, we have to deal with many additional concerns, such as:

• Data management. User provided data must be validated, stored, transformed in something that can be visually represented, properly escaped so that they can be inserted into a database (preventing SQL injections).
• Security. User permissions must be validated on all kinds of levels, such as page level, or section level. User roles must be defined. Secure connections must be established by using the SSL protocol.
• Scalability. When your system has many users, it will no longer be possible to serve your web application from a single web server because it lacks sufficient system resources. Your system must be decomposed and optimized (e.g. by using caching).

In my experience, the tech companies I used to work for understood these issues, but I also ran into many kinds of situations with non-technical people not understanding that all these things were complicated and necessary.

One time, I even ran into somebody saying: "Well, you can export Microsoft Word documents to HTML pages, right? Why should things be so difficult?".

There is a lack of abstraction facilities in HTML

As explained earlier, programming with HTML and CSS could be considered declarative programming. At the same time, declarative programming is a spectrum -- it is difficult to draw a hard line between what and how -- it all depends on the context.

The same thing applies to HTML -- from one perspective, HTML code can be considered a "what specification" since you do not have specify how to render a paragraph, image or a button.

In other cases, you may want to do things that cannot be directly expressed in HTML, such as embedding a photo gallery on your web page -- there is no HTML facility allowing you to concisely express that. Instead, you must provide the corresponding HTML elements that implement the gallery, such as the divisions, paragraphs, forms and images. Furthermore, HTML does not provide you any facilities to define such abstractions yourself.

If there are many recurring high level concepts to implement, you may end up copying and pasting large portions of HTML code between pages making it much more difficult to modify and maintain the application.

A consequence of not being able to define custom abstractions in HTML is that it has become very common to generate pages server side. Although this suffices to get most jobs done, generating dynamic content is many times more expensive than serving static pages, which is quite silly if you think too much about it.

A very common sever side abstraction I used to implement (in addition to an embedded gallery) is a layout manager allowing you to manage static common sections, a menu structure and dynamic content sections. I ended up inventing such a component because I was used to frames, that became deprecated. Moving away from them required me to reimplement common sections of a page over and over again.

In addition to generated code, using JavaScript also has become quite common by dynamically injecting code into the DOM or transforming elements. As a result, quite a few pages will not function properly when JavaScript has been disabled or when JavaScript in unsupported.

Moreover, many pages embed a substantial amount of JavaScript significantly increasing their sizes. A study reveals that the total size of a quite a few modern web pages are equal to the Doom video game.

There is a conceptual mismatch between 'pages' and 'screens'

HTML is a language designed for constructing pages, not screens. However, information systems typically require a screen-based workflow -- users need to modify data, send their change requests to the server and update their views so that their modifications become visible.

In HTML, there are only two ways to propagate parameters to the server and getting feedback -- with hyperlinks (containing GET parameters) or forms. In both cases, a user gets redirected to another page that should display the result of the action.

For pages displaying tabular data or other complicated data structures, this is quite inconvenient -- we have to rerender the entire page each time we change something and scroll the user back to the location where the change was made (e.g. by defining anchors).

Again, with JavaScript this problem can be solved in a more proper and efficient way -- by programming an event handler (such as a handler for the click event), using the XMLHttpRequest object to send a change message to the server and updating the appropriate DOM elements, we can rerender only the affected parts.

Unfortunately, this again breaks the declarative nature of web technologies and the ability to degrade gracefully -- in a browser that lacks JavaScript support (e.g. text-oriented browsers) this solution will not work.

Also, efficient state management is a complicated problem, that you may want to solve by integrating third party JavaScript libraries, such as MobX or Redux.

Layouts are extremely difficult

In addition to the absence of abstractions in HTML (motivating me to develop a layout manager), implementing layouts in general is also something I consider to be notoriously difficult. Moreover, the layout concern is not well separated -- some aspects need to be done in your page structure (HTML code) and other aspects need to be done in stylesheets.

Although changing most visual properties of page elements in CSS is straight forward (e.g. adjusting the color of the background, text or borders), dealing with layout related aspects (e.g. sizing and positioning page elements) is not. In many cases I had to rely on clever tricks and hacks.

One of the weirdest recurring layout-tricks I used to implement is a hack to make the height of two adjacent floating divs equal. This is something I commonly used to put a menu panel next to a content panel displaying text and images. You do not know in advance the height of both panels.

I ended up solving this problems as follows. I wrapped the divs in a container div:

<div id="container">
<div id="left-column">
</div>
<div id="right-column">
</div>
</div>

and I provided the following CSS stylesheet:

#container
{
    overflow: hidden;
}

#left-column
{
    padding-bottom: 3000px;
    margin-bottom: -3000px;
}

#right-column
{
    padding-bottom: 3000px;
    margin-bottom: -3000px;
}

In the container div, I abuse the overflow property (disabling scroll bars if the height exceeds the screen size). For the panels themselves, I use a large padding value value and a equivalent negative margin. The latter hack causes the panels to stretch in such a way that their heights become equal:


(As a sidenote: the above problem can now be solved in a better way using a flexbox layout, but a couple of years ago you could not use this newer CSS feature).

The example shown above is not an exception. Another notable trick is the clear hack (e.g. <div style="clear: both;"></div>) to ensure that the height of the surrounding div grows automatically with the height of your inner divs.

As usual, JavaScript can be used to solved to abstract these oddities away, but breaks declarativity. Furthermore, when JavaScript is used for an essential part of your layout, your page will look weird if JavaScript has been disabled.

Interoperability problems

Many web technologies have been standardized by the World Wide Web Consortium (W3C) with the purpose to ensure interoperability among browsers. The W3C also provides online validator services (e.g. for HTML and CSS) that you can use to upload your code and check for its validity.

As an outsider, you may probably expect that if your uploaded code passes validation that your web application front-end is interoperable and will work properly in all browsers... wrong!

For quite some time, Internet Explorer 6 was the most dominant web browser. Around the time that it was released (2001) it completely crushed its biggest competitor (Netscape) and gained 95% market share. Unfortunately, it did not support modern web standards well (e.g. CSS 2 and newer). After winning the browser wars, Microsoft pretty much stopped its development.

Other browsers kept progressing and started to become much more advanced (most notably Mozilla Firefox). They also followed the web standards more faithfully. Although this was a good development, the sad thing was that in 2007 (6 years later) Internet Explorer 6 was still the most dominant browser with its lacking support of standards and many conformance bugs -- as a result, you were forced to implement painful IE-specific workarounds to make your web page work properly.

What I typically used to do is that I implemented a web page for "decent browsers" first (e.g. Firefox, Chrome, Safari), and then added Internet Explorer-specific workarounds (such as additional stylesheets) on top. By using conditional comments, an Internet Explorer-specific feature that treated certain comments as code, I could ensure that the hacks were not used by any non-IE browser. An example usage case is:

<!--[if lt IE 7]><link rel="stylesheet" type="text/css" href="ie-hacks.css"><![endif]-->

The above conditional comment states that if an Internet Explorer version lower than 7 is used, then the provided ie-hacks.css stylesheet should be used. Otherwise, it is treated as a comment and will be ignored.

Fortunately, Google Chrome overtook the role as the most dominant web browser and is developed more progressively eliminating most standardization problems. Interoperability today is still not a strong guarantee, in particular for new technologies, but considerably better around the time that Internet Explorer still dominated the browser market.

Stakeholder difficulties

Another major challenge are the stakeholders with their different and somewhat conflicting interests.

The most important thing that matters to the end-user is that a web application provides the information they need, that it can be conveniently found (e.g. through a search engine), and that they are not distracted too much. Visual appearance of your web site also matters in some extent (e.g. a dreadful appearance your web site will affect an end user's ability to find what they need and your credibility), but is typically not as important as most people think.

Most clients (the group of people requesting your services) are mostly concerned with visual effects and features, and not so much with the information they need to provide to their audience.

I still vividly remember a web application that I developed whose contents could be fully managed by the end user with a simple HTML editor. I deliberately kept the editor's functionality simple -- for example, it was not possible in the editor to adjust the style (e.g. the color of the text or background), because I believed that users should simply follow the stylesheet.

Moreover, I have spent substantial amounts of time explaining clients how to write for the web -- they need to structure/organize their information properly write concisely and structure/format their text.

Despite all my efforts in bridging the gap between end-users and clients, I still remember that one particular client ran away dissatisfied because of the lack of customization. He moved to a more powerful/flexible CMS, and his new homepage looked quite horrible -- ugly background images, dreadful text colors, sentences capitalized, e.g.: "WELCOME TO MY HOMEPAGE!!!!!!".

I also had been in a situation once in which I had to deal with two rivaling factions in a organization -- one being very supportive with my ideas and another being completely against them. They did not communicate with each other much and I basically had to serve as a proxy between them.

Also, I have worked with designers giving me mixed experiences. A substantial group of designers basically assumed that a web page design is the same thing as a paper design, e.g. a booklet.

With a small number of them I had quite a few difficulties explaining that web page designs need to be flexible and working towards a solution meeting these criteria -- people use different kinds of screen sizes, resolutions. People tend to resize their windows, adjust their font sizes, and so on. Making a design that is too static will affect how many users you will attract.

Technology fashions

Web technology is quite sensitive to technological fashions -- every day new frameworks, libraries and tools appear, sometimes for relatively new and uncommon programming languages.

While new technology typically provides added value, you can also quite easily shoot yourself in the foot. Being forced to work around a system's broken foundation is not particularly a fun job.

Two of my favorite examples of questionable technology adoptions are:

• No SQL databases. At some point, probably because of success stories from Google, a lot of people consider traditional relational databases (using SQL) not to be "web scale" and massively shifted to so-called "No SQL databases". Some well known NoSQL databases (e.g. MongoDB) sacrifice properties in service of speed -- such as consistency guarantees.
  
  In my own experience, many applications that I developed, the relational model made perfect sense. Also, consistency guarantees were way more important than speed benefits. As a matter of fact, most of my applications were fast enough. (As a sidenote: This does not disqualify NoSQL databases -- they have legitimate use cases, but in many cases they are simply not needed).
• Single threaded event loop server applications (e.g. applications built on Node.js).
  
  The single thread event loop model has certain kinds of benefits over the traditional thread/process per connection approach -- little overhead in memory and multitasking making it a much better fit to handle large numbers of connections.
  
  Unfortunately, most people do not realize that there are two sides of the coin -- in a single threaded event loop, the programmer has the responsibility to make sure that it never blocks so that that your application remains responsive (I have seen quite a few programmers who simply lack the understanding and discipline to do that).
  
  Furthermore, whenever something unexpected goes wrong you end up with an application that crashes completely making it much more sensitive to disruptions. Also, this model is not a very good fit for computationally intensive applications.
  
  (Again: I am not trying to disqualify Node.js or the single threaded event loop concept -- they have legitimate use cases and benefits, but it is not always a good fit for all kinds of applications).

My own web framework

I have been extensively developing a custom PHP-based web framework between 2002 and 2009. Most of its ideas were born while I was developing a web-based information system to manage a documentation library for a side job. I noticed that there were quite a few complexities that I had to overcome to make it work properly, such as database integration, user authentication, and data validation.

After completing the documentation library, I had been developing a number of similarly looking information systems with similar functionality. Over the years, I learned more techniques, recognized common patterns, captured abstractions, and kept improving my personal framework. Moreover I had been using my framework for a variety of additional use cases including web sites for small companies.

In the end, it evolved into a framework providing the following high level components (the boxes in the diagram denote packages, while the arrows denote dependency relationships):

• php-sbdata. This package can be used to validate data fields and present data fields. It can also manage collections of data fields as forms and tables. Originally this package was only used for presentation, but based on my experience with WebDSL I have also integrated validation.
• php-sbeditor. This package provides an HTML editor implementation that can be embedded into a web page. It can also optionally integrate with the data framework to expose a field as an HTML editor. When JavaScript is unsupported or disabled, it will fall back to a text area in which the user can directly edit HTML.
• php-sblayout. This package provides a layout manager that can be used to manage common sections, the menu structure and dynamic sections of a page. A couple of years ago, I wrote a blog post explaining how it came about. In addition to a PHP package, I also created a Java Servlet/JSP implementation of the same concepts.
• php-sbcrud is my partial solution to the page-screen mismatch problem and combines the concepts of the data management and layout management packages.
  
  Using the CRUD manager, every data element and data collection has its own URL, such as http://localhost/index.php/books to display a collection of books and http://localhost/index.php/books/1 to display an individual book. By default, data is displayed in view mode. Modifications can be made by appending GET parameters to the URL, such as: http://localhost/index.php/books/1?__operation=remove_book.
• php-sbgallery provides an embeddable gallery sub application that can be embedded in various ways -- directly in a page, as a collection of sub pages via the layout manager, in an HTML editor, and as a page manager allowing you to expose albums of the gallery as sub pages in a web application.
• php-sbpagemanager extends the layout manager with the ability to dynamically manage pages. The page manager can be used to allow end-users to manage the page structure and contents of a web application.
• php-sbbiblio is a library I created to display my bibliography on my personal homepage while I was doing my PhD research.

For nowadays' standards the features provided by the above package are considered to be old fashioned. Still, I am particularly proud of the following quality properties (some people may consider them anti-features these days):

• Being as declarative as possible. This means that the usage of JavaScript is minimized and non essential. Although it may not efficiently deal with the page-screen mismatch because of this deliberate choice, it does provide other benefits. For example, the system is still usable when JavaScript has been disabled and even works in text-oriented browsers.
• Small page sizes. The layout manager allows you to conveniently separate common aspects from page-specific aspects, including external stylesheets and scripts. As a result, the size of the rendered pages are relatively small (in particular compared to many modern web sites), making page loading times fast.
• Very thin data layer. The data manager basically works with primitive values, associative arrays and PDO. It has no strong ties to any database model or an Object-Relational-Mapper (ORM). Although this may be inconvenient from a productivity point of view, the little overhead ensures that your application is fast.

Conclusion

In this blog post, I have explained where my initial enthusiasm for the web came from, my experiences (including the negative ones), and my own framework.

The fact that I am not as passionate about the web anymore did not make me leave that domain -- web-based systems these days are ubiquitous. Today, much of the work I do is system configuration, back-end and architecture related. I am not so active on the front-end side anymore, but I still look at front-end related issues from time to time.

Moreover, I have used PHP for a very long time (in 2002 there were basically not that many appealing alternatives), but I have also used many other technologies such a Java Servlets/JSP, Node.js, and Django. Moreover, I have also used many client-side frameworks, such as Angular and React.

I was also briefly involved with the development of WebDSL, an ongoing project in my former research group, but my contributions were mostly system configuration management related.

Although these technologies all offer nice features and in some way impress me, it has been a very long time that I really felt enthusiastic about anything web related.

Availability

Three years ago, I have already published the data manager, layout manager and bibliography packages on my GitHub page. I have now also published the remaining components. They can be used under the terms and conditions of the Apache Software License version 2.0.

In addition to the framework components, I published a repository with a number of example applications that have comparable features to the information systems I used to implement. The example applications share the same authentication system and can be combined together through the portal application. The example applications are GPLv3 licensed.

You may probably wonder why I published these packages after such a long time? The are variety of reasons -- I always had the intention to make it open, but when I was younger I focused myself mostly on code, not additional concerns such as documenting how the API should be used or providing example cases.

Moreover, in 2002 platforms such as GitHub did not exist yet (there was Sourceforge, but it worked on a project-level and was not as convenient) so it was very hard to publish something properly.

Finally, there are always things to improve and I always run various kinds of experiments. Typically, I use my own projects as test subjects for other projects. I also have a couple of open ideas where I can use pieces of my web framework for. More about this later.

As explained in my previous blog post, I used to be quite interested in web technology and spend considerable amounts of time developing my own framework providing solutions for common problems that I used to face, such as layout management and data management.

Another challenge that you cannot avoid is the visual appearance of your web application. Today, there are many frameworks and libraries available allowing you to do impressive things, such as animated transitions, fade in/fade out effects and so on.

Unfortunately, many of these "modern" solutions also have number of big drawbacks -- typically, they are big and complex JavaScript-based frameworks significantly increasing the download size of pages and the amount of required system resources (e.g. CPU, GPU, battery power) to render a page. As a result, it is not uncommon that the download size of many web sites equal the Doom video game and may feel slow and sluggish.

Some people (such as the author of this satirical website) suggest that most (all?) visual aspects are unnecessary and that simply a "vanilla" page displaying information suffices to provide a user what he needs. I do not entirely agree with this viewpoint as many web sites are not just collections of pages but complex information systems. Complex information systems require some means of organizing data including a layout that reflects this, such as menu panels that guide a user through the desired sets of information.

I know many kinds of tricks (and hacks) to implement layouts and visual aspects, but one thing that I am not particularly good at is designing visuals myself. There is a big pool of layout considerations to choose from and many combinations that can be made. Some combinations of visual aspects are good, others are bad -- I simply lack the intuition to make the right choices. This does not apply to web design only -- when I had to choose furniture for my house I basically suffered from the same problem. As a result, I have created (by accident) a couple of things that look nice and other things that look dreadful :-)

Although I have worked with designers, it is not always possible to consult one, in particular for non-commercial/more technical projects.

In this blog post, I have gathered a number of minimalistic layout considerations that, regardless of the objective of the system and the absence of design intuition, I find worth to consider, with some supporting information so that rational decisions can be made. Furthermore, they are relatively simple to apply and do not require any framework.

Text size

A web application's primary purpose is providing information. As a consequence, how you present text is very important.

A number of studies (such as this one by Jakob Nielsen in 1997) show that visitors of web pages do not really read, but scan for information. Although this study was done many years ago it still applies to today's screens, such as tablets and devices that were designed for reading books, such as the Kindle.

Because users typically read much slower from a screen than from paper and hardly read sections entirely, it is a very good idea to pick a font size that is large enough.

In most browsers, the default font size is set to 16 pixels. Studies suggest that this size is all but too small, in particular for high resolution screens that we use nowadays. Moreover, a font size of 16px on modern screens, is somewhat equal to the font size of a (physical) book.

Moreover, CSS allows you to define the font sizes statically or relatively. In my opinion (supported by this article), it is a good practice to use relative font sizes, because it is a good habit to allow users to control the size of the fonts.

For me, typically the following CSS setting suffices:

body
{
    font-size: 100%;
}

Spacing

When I was younger, I had the tendency to put as much information on a page as possible, such as densily written text, images and tables. At some point, I worked with a designer who constantly reminded me that I should keep enough space between page elements.

(As a sidenote: space should not necessarily be white space, but could also be the background color or background gradient. Some designers call this kind of space "negative spacing").

Why is sufficient negative spacing a good thing? According to some sources, filling a page with too many details, such as images, makes it difficult to maintain a user's attention. For example, if a page contains too much graphics or have colors that appear unrelated to the rest of the page, a user will quickly skip details.

Furthermore, some studies suggest that negative spacing is an effective way to emphasize important elements of a page and a very effective way to direct the flow of a page.

From an implementation perspective, there are many kinds of HTML elements that could be adjusted (from the browser's default settings) to use a bit of extra spacing, such as:

• Text: I typically adjust the line-height to 1.2em increasing the space between each sentence of a paragraph.
• For divs and other elements that define sections, I increase their margins and paddings. Typically I would set them to at least: 1em.
• For table and table cells I increase their paddings to 0.5em.
• For preformatted text: pre I use a padding value of at least 1em.
• For list items (entries of an unordered or ordered list) I add a bit of extra spacing on top, e.g. li { margin: 0.2em 0; }.

Some people would argue that the above list of adjustments are still quite conservative and even more spacing should be considered.

Font color/foreground contrast

Another important thing is to pick the right foreground colors (such as the text color) and ensure that they have sufficient contrast -- for example, light grey colored text on a white background is very difficult for users to read.

In most browsers, the default setting is to have a white background and black colored text. Although this color scheme maximizes contrast, too much contrast also has a disadvantage -- it maximizes a user's attention span for a while, but a user cannot maintain such a level of attention indefinitely.

When displaying longer portions of text, it is typically better to lower the contrast a bit, but not too much. For example, when I want to display black colored text on a white background, I tune down the contrast a bit by setting the text color to dark grey: #444; as opposed to black: #000; and the background color to very light grey: #ddd; as opposed to white: #fff;.

Defining panels and sections

A web application is typically much more than just a single page of content. Typically, they are complex information systems displaying many kinds of data. This data has to be to be divided, categorized and structured. As a result, we also need facilities that guide the users through these sets of information, such as menu and content panels.

Creating a layout with "panels" turns out to be quite a complex problem, for which various kinds of strategies exist each having their pros and cons. For example, I have used the following techniques:

• Absolute positioning. We add the property: position: absolute; to a div and we use the left, right, top and bottom properties to specify the coordinates of the top left and bottom right position of the panel. As a result, the div automatically gets positioned (relative to the top left position of the screen) and automatically gets a width and height. For the sections that need to expand, e.g. the contents panel displaying the text, we use the overflow: auto; property to enable vertical scroll bars if needed.
  
  Although this strategy seems to do mostly what I want, it also has a number of drawbacks -- the position of the panels is fixed. For desktops this is usually fine, but for screens with a limited height (such as mobile devices and tablets) this is quite impractical.
  
  Moreover, it used to be a big problem when Internet Explorer 6 was still the dominant browser -- IE6 did not implement the property that automatically derives the width and height, requiring me to implement a workaround stylesheet using JavaScript to compute the width and heights.
• Floating divs is a strategy that is more friendly to displays with a limited height but has different kinds of challenges. Basically, by adding a float property to each panel, e.g. float: left; and specifying a width we can position columns next to each other. By using a clear hack, such as: <div style="clear: both;"></div> we can position a panel right beneath another panel.
  
  This strategy mostly makes sense despite the fact that the behaviour of floating elements in somewhat strange. One of the things that is very hard to solve is to create a layout in which columns have an equal height when their heights are not known in advance -- someone has written a huge blog post with all kinds of strategies. I, for example, implemented the "One True Layout Method" (a margin-padding-overflow hack) quite often.
• Flexbox is yet another (more modern) alternative and the most powerful solution IMO so far. It allows you to consicely specify the how divs should be positioned, wrapped and sized. The only downside I see so far, is that these properties are relatively new and require very new implementations of browser layout engines. Many users typically do not bother that much to upgrade their browsers, unless they are forced.

Resolution flexibility (a.k.a. responsive web design)

When I just gained access to the Internet, nearly all web page visitors used to be desktop users. Today, however, a substantial number of visitors use different kinds of devices, having small screens (such as phones and tablets) and very big screens (such as TVs).

To give all these visitors a relatively good user experience, it is important to make the layout flexible enough to support many kinds of resolutions. Some people call this kind of flexibility responsive web design, but I find that term somewhat misleading.

Besides the considerations shown earlier, I also typically implement the following aspects in order to become even more flexible:

• Eliminating vertical menu panels. When we have two levels of menu items, I typically display the secondary menu panel on the left on the screen. For desktops (and bigger screens) this is typically OK, but for smaller displays it eats too much space from the section that displays the contents. I typically use media queries to reconfigure these panels in such a way that the items on these menu panels are aligned horizontally by default and, when the screen width is big enough, they will be aligned vertically.
• Making images dynamically resizable. Images, such as photos in a gallery, may be too big to properly display on a smaller screen, such as a phone. Fortunately, by providing the max-width setting we can adjust the style of images in such a way that their maximum width never exceeds the screen size and their dimensions get scaled accordingly:
  
  

  img
  {
      max-width: 100%;
      width: auto;
      height: auto;
  }
  

• Adding horizontal scroll bars, when needed. For some elements, it is difficult to resize them in such a way that they never exceed the screen width, such as sections of preformatted text which I typically use to display code fragments. For these kinds of elements I typically configure the overflow-x: auto; property so that horizontal scroll bars appear when needed.

Picking colors

In addition to the text and the background colors, we may also want to pick a couple of additional colors. For example, we need different colors for hyperlinks so that it becomes obvious to users what is clickable and what not, and whether a link has already been visited or not. Furthermore, providing distinct colors for the menu panels, headers, and buttons would be nice as well.

Unfortunately, when it comes to picking colors, my intuition lets me down completely -- I do not know what "nice colors" are or which colors fit best together. As a result, I always have a hard time making the right choices.

Recently, I discovered a concept called "color theory" that lets you decide, somewhat rationally, what colors to pick. The basic idea behind color theory is that you pick a base color and then apply a color scheme to get a number of complementary colors, such as:

• Triadic. Composed of 3 colors on separate ends of the color spectrum.
• Compound. One color is selected in the same area of the color spectrum and two colors are chosen from opposite ends of the color spectrum.
• Analogous. Careful selection of colors in the same area of the color spectrum.

A particular handy online tool (provided by the article shown above) is paletton that seems to provide me good results -- it supports various color schemes, has a number of export functions (including CSS) and a nice preview function that shows you what a web page would look like if you apply the generated color scheme.

Unfortunately, I have not found any free/open-source solutions or a GIMP plugin allowing me to do the same.

Printing

Something that is typically overlooked by many developers is printing. For most interactive web sites this not too important, but for information systems including reservation systems, it is also a good practice to implement proper printing support.

I consider it to be a good practice to hide non-relevant panels, such as the menu panels:

@media only print
{
    #header, #menu
    {
        display: none;
    }
}

Furthermore, it is also a good practice to tune down the amount of colors a bit.

A simple example scenario

As an experiment to test the above listed concerns (e.g. text-size, foreground contrast, using paletton to implement color theory and NOT trusting my intuition), I ended up implementing the following "design":

This is what it looks on a desktop browser:


For printing, the panels and colors are removed:


The page is even functional in a text oriented browser (w3m):


By using media queries, we can adjust the positioning of the sub menu to make more space available for the contents panel on a mobile display (my Android phone):


The layout does not look shiny or fancy, but it appears functional to me, but hey, don't ask me too much since I simply lack the feeling for design. :-)

In my previous blog post, I have described my own custom web framework for which I released a number additional components. I have also implemented an example repository with simple demonstration applications. I have decided to use my "rationally crafted layout" to make them look a bit prettier.

Conclusion

In this blog post, I have gathered a collection of minimalistic layout considerations for myself that, regardless of the objective of a web application, I find worth to consider. Moreover, these concerns can be applied mostly in a rational way, which is good for people like me who lack design intuition.

Although I have described many aspects in this blog post, applying the above considerations will not produce any fancy/shiny web pages. For example, you can implement many additional visual aspects, such as rounded corners, shadows, gradients, animated transitions, fade-ins and fade-outs, and so on.

Moreover, from writing this blog post I learned a thing or two about usability and HTML/CSS tricks. However, I again observed that the more you know, the more you realize that you do not know. For example, there are even more specialized studies available, such as one about the psychology of colors that, for example, shows that women prefer blue, purple, and green, and men prefer blue, green, and black.

This, however, is where I draw the line when it comes to learning design skills and made me realize why I do not prefer to become a front-end/design specialist. Knowing some things about design is always useful, but the domain is much deeper and complex than I initially thought.

The only thing that I still like to do is finding additional, simple, rationally applicable design considerations. Does anyone have some additional suggestions for me?

It has been a while since I wrote a Nix-related blog post. In many of my earlier Nix blog posts, I have elaborated about various Nix applications and their benefits.

However, when you are developing a product or service, you typically do not only want to use configuration management tools, such as Nix -- you may also want to build a platform that is tailored towards your needs, so that common operations can be executed structurally and conveniently.

When it is desired to integrate custom solutions with Nix-related tools, you basically have one recurring challenge -- you must generate deployment specifications in the Nix expression language.

The most obvious solution is to use string manipulation to generate the expressions we want, but this has a number of disadvantages. Foremost, composing strings is not a very intuitive activity -- it is not always obvious to see what the end result would be by looking at the code.

Furthermore, it is difficult to ensure that a generated expression is correct and safe. For example, if a string value is not properly escaped, it may be possible to inject arbitrary deployment code putting the security of the deployed system at risk.

For these reasons, I have developed NiJS: an internal DSL for JavaScript, a couple of years ago to make integration with JavaScript-based applications more convenient. Most notably, NiJS is used by node2nix to generate Nix expressions from NPM package deployment specifications.

I have been doing PHP development in the last couple of weeks and realized that I needed a similar solution for this language. In this blog post, I will describe PNDP, an internal DSL for Nix in PHP, and show how it can be used.

Composing Nix packages in PHP

The Nix packages repository follows a specific convention for organizing packages -- every package is a Nix expression file containing a function definition describing how to build a package from source code and its build-time dependencies.

A top-level composition expression file provides all the function invocations that build variants of packages (typically only one per package) by providing the desired versions of the build-time dependencies as function parameters.

Every package definition typically invokes stdenv.mkDerivation {} (or abstractions built around it) that composes a dedicated build environment in which only the specified dependencies can be found and other kinds of precautions are taken to improve build reproducibility. In this builder environment, we can execute many kinds of build steps, such as running GNU Make, CMake, or Apache Ant.

In our internal DSL in PHP we can replicate these conventions using PHP language constructs. We can compose a proxy to the stdenv.mkDerivation {} invocation in PHP by writing the following class:

namespace Pkgs;
use PNDP\AST\NixFunInvocation;
use PNDP\AST\NixExpression;

class Stdenv
{
    public function mkDerivation($args)
    {
        return new NixFunInvocation(new NixExpression("pkgs.stdenv.mkDerivation"), $args);
    }
}

In the above code fragment, we define a class named: Stdenv exposing a method named mkDerivation. The method composes an abstract syntax tree for a function invocation to stdenv.mkDerivation {} using an arbitrary PHP object of any type as a parameter.

With the proxy shown above, we can create our own in packages in PHP by providing a function definition that specifies how a package can be built from source code and its build-time dependencies:

namespace Pkgs;
use PNDP\AST\NixURL;

class Hello
{
    public static function composePackage($args)
    {
        return $args->stdenv->mkDerivation(array(
"name" =>"hello-2.10",

"src" => $args->fetchurl(array(
"url" => new NixURL("mirror://gnu/hello/hello-2.10.tar.gz"),
"sha256" =>"0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"
            )),

"doCheck" => true,

"meta" => array(
"description" =>"A program that produces a familiar, friendly greeting",
"homepage" => new NixURL("http://www.gnu.org/software/hello/manual"),
"license" =>"GPLv3+"
            )
        ));
    }
}

The above code fragment defines a class named 'Hello' exposing one static method named: composePackage(). The composePackage method invokes the stdenv.mkDerivation {} proxy (shown earlier) to build GNU Hello from source code.

In addition to constructing a package, the above code fragment also follows the PHP conventions for modularization -- in PHP it is a common practice to modularize code chunks into classes that reside in their own namespace. For example, by following these conventions, we can also automatically load our package classes by using an autoloading implementation that follows the PSR-4 recommendation.

We can create compositions of packages as follows:

class Pkgs
{
    public $stdenv;

    public function __construct()
    {
        $this->stdenv = new Pkgs\Stdenv();
    }

    public function fetchurl($args)
    {
        return Pkgs\Fetchurl::composePackage($this, $args);
    }

    public function hello()
    {
        return Pkgs\Hello::composePackage($this);
    }
}

As with the previous example, the composition example is a class. In this case, it exposes variants of packages by calling the functions with their required function arguments. In the above example, there is only one variant of the GNU Hello package. As a result, it suffices to just propagate the object itself as build parameters.

Contrary to the Nix expression language, we must expose each package composition as a method -- the Nix expression language is a lazy language that only invokes functions when their results are needed, PHP is an eager language that will evaluate them at construction time.

An implication of using eager evaluation is that opening the composition module, triggers all packages to be built. By wrapping the compositions into methods, we can make sure that they only the requested packages are evaluated when needed.

Another practical implication of creating methods for each package composition is that it can become quite tedious if we have many of them. PHP offers a magic method named: __call() that gets invoked when we invoke a method that does not exists. We can use this magic method to automatically compose a package based on the method name:

public function __call($name, $arguments)
{
    // Compose the classname from the function name
    $className = ucfirst($name);
    // Compose the name of the method to compose the package
    $methodName = 'Pkgs\\'.$className.'::composePackage';
    // Prepend $this so that it becomes the first function parameter
    array_unshift($arguments, $this);
    // Dynamically the invoke the class' composition method with $this as first parameter and the remaining parameters
    return call_user_func_array($methodName, $arguments);
}

The above method takes the (non-existent) method name, converts it into the corresponding class name (by using the camel case naming convention), invokes the package's composition method using the composition object itself as a first parameter, and any other method parameters as successive parameters.

Converting PHP language constructs into Nix language constructs

Everything that PNDP does boils down to the phpToNix() function that automatically converts most PHP language constructs into semantically equivalent or similar Nix language constructs. For example, the following PHP language constructs are converted to Nix as follows:

• A variable of type boolean, integer or double are converted verbatim.
• A string will be converted into a string in the Nix expression language, and conflicting characters, such as the backslash and double quote, will be escaped.
• In PHP, arrays can be sequential (when all elements have numeric keys that appear in numeric order) or associative in the remainder of the cases. The generator tries to detect what kind of array we have. It recursively converts sequential arrays into Nix lists of Nix language elements, and associative arrays into Nix attribute sets.
• An object that is an instance of a class, will be converted into a Nix attribute set exposing its public properties.
• A NULL reference gets converted into a Nix null value.
• Variables that have an unknown type or are a resource will throw an exception.

As with NiJS (and JavaScript), the PHP host language does not provide equivalents for all Nix language constructs, such as values of the URL type, or encoding Nix function definitions.

You can still generate these objects by composing an abstract syntax from objects that are instances of the NixObject class. For example, when composing a NixURL object, we can generate a value of the URL type in the Nix expression language.

Arrays are a bit confusing in PHP, because you do not always know in advance whether it would yield a list or attribute set. To make these conversions explicit and prevent generation errors, they can be wrapped inside a NixList or NixAttrSet object.

Building packages programmatically

The PNDPBuild::callNixBuild() function can be used to build a generated Nix expression, such as the GNU Hello example shown earlier:

/* Evaluate the package */
$expr = PNDPBuild::evaluatePackage("Pkgs.php", "hello", false);

/* Call nix-build */
PNDPBuild::callNixBuild($expr, array());

In the code fragment above, we open the composition class file, named: Pkgs.php and we evaluate the hello() method to generate the Nix expression. Finally, we call the callNixBuild() function, in which we evaluate the generated expression by the Nix package manager. When the build succeeds, the resulting Nix store path is printed on the standard output.

Building packages from the command-line

As the previous code example is so common, there is also a command-line utility that can execute the same task. The following instruction builds the GNU Hello package from the composition class (Pkgs.php):

$ pndp-build -f Pkgs.php -A hello

It may also be useful to see what kind of Nix expression is generated for debugging or testing purposes. The --eval-only option prints the generated Nix expression on the standard output:

$ pndp-build -f Pkgs.js -A hello --eval-only

We can also nicely format the generated expression to improve readability:

$ pndp-build -f Pkgs.js -A hello --eval-only --format

Discussion

In this blog post, I have described PNDP: an internal DSL for Nix in PHP.

PNDP is not the first internal DSL I have developed for Nix. A couple of years ago, I also wrote NiJS: an internal DSL in JavaScript. PNDP shares a lot of concepts and implementation details with NiJS.

Contrary to NiJS, the functionality of PNDP is much more limited -- I have developed PNDP mainly for code generation purposes. In NiJS, I have also been exploring the abilities of the JavaScript language, such as exposing JavaScript functions in the Nix expression language, and the possibilities of an internal DSL, such as creating an interpreter that makes it a primitive standalone package manager. In PNDP, all this extra functionality is missing, since I have no practical need for them.

In a future blog post, I will describe an application that uses PNDP as a generator.

Availability

PNDP can obtained from Packagist as well as my GitHub page. It can be used under the terms and conditions of the MIT license.

In two earlier blog posts, I have described various pieces of my custom web framework that I used to actively develop many years ago. The framework is quite modular -- every concern, such as layout management, data management, the editor, and the gallery, are separated into packages that can be deployed independently, so that web applications only have to include what they actually need.

Although modularity is quite useful for a variety of reasons, the framework did not start out as being modular in the beginning -- when I just started developing web applications in PHP, I did not reuse anything at all. Slowly, I discovered similarities between my projects and started sharing snippets of common functionality between them. Gradually, I learned that keeping these common aspects up to date became a burden. As a result, I developed a "common framework" that I reused among all my PHP projects.

Having a common framework for my web application projects reduced the amount of required maintenance, but introduced a new drawback -- its size kept growing and growing. As a result, many simple web applications that only required a small subset of the framework's functionality still had to embed the entire framework, making them unnecessarily big.

Today, a bit of extra PHP code is not so much of a problem, but around the time I was still actively developing web applications, many shared web hosting providers only offered a small amount of storage capacity, typically just a few megabytes.

To cope with the growing size of the framework, I decided to modularize the code by separating the framework's concerns into packages that can be deployed independently. I "invented" my own conventions to integrate the framework packages into web applications:

• In the base directory of the web application project, I create a lib/ directory that contains symlinks to the framework packages.
• In every PHP script that displays a page (typically only index.php), I configure the include path to refer to the packages' content in the lib/ folder, such as:
  
  

  
  set_include_path("./lib/sblayout:./lib/sbdata:./lib/sbcrud");
  

  
  
• Each PHP module is responsible for loading the desired classes or utility functions from the framework packages. As a result, I ended up writing a substantial amount of require() statements, such as:
  
  

  
  require_once("data/model/Form.class.php");
  require_once("data/model/field/HiddenField.class.php");
  require_once("data/model/field/TextField.class.php");
  require_once("data/model/field/DateField.class.php");
  require_once("data/model/field/TextAreaField.class.php");
  require_once("data/model/field/URLField.class.php");
  require_once("data/model/field/FileField.class.php");
  

After my (approximately) 8 years of absence from the PHP domain, I discovered that a tool has been developed to support convenient construction of modular PHP applications: composer. Composer is heavily inspired by the NPM package manager, that is the defacto package delivery mechanism for Node.js applications.

In the last couple of months (it progresses quite slowly as it is a non-urgent side project), I have decided to get rid of my custom modularity conventions in my framework packages, and to adopt composer instead.

Furthermore, composer is a useful deployment tool, but its scope is limited to PHP applications only. As frequent readers may probably already know, I use Nix-based solutions to deploy entire software systems (that are also composed of non-PHP packages) from a single declarative specification.

To be able to include PHP composer packages in a Nix deployment process, I have developed a generator named: composer2nix that can be used to generate Nix deployment expressions from composer configuration files.

In this blog post, I will explain the concepts of composer2nix and show how it can be used.

Using composer

Using composer is generally quite straight forward. In the most common usage scenario, there is typically a PHP project (often a web application) that requires a number of dependencies. By changing the current working folder to the project directory, and running:

$ composer install

Composer will obtain all required dependencies and stores them in the vendor/ sub directory.

The vendor/ folder follows a very specific organisation:

$ find vendor/ -maxdepth 2 -type d
vendor/bin
vendor/composer
vendor/phpdocumentor
vendor/phpdocumentor/fileset
vendor/phpdocumentor/graphviz
vendor/phpdocumentor/reflection-docblock
vendor/phpdocumentor/reflection
vendor/phpdocumentor/phpdocumentor
vendor/svanderburg
vendor/svanderburg/pndp
...

The vendor/ folder structure (mostly) consists two levels: the outer directory defines the namespace of the packages and the inner directory the package names.

There are a couple of folders deviating from this convention -- most notably, the vendor/composer directory, that is used by composer to track package installations:

$ ls vendor/composer
autoload_classmap.php
autoload_files.php
autoload_namespaces.php
autoload_psr4.php
autoload_real.php
autoload_static.php
ClassLoader.php
installed.json
LICENSE

In addition to obtaining packages and storing them in the vendor/ folder, composer also generates autoload scripts (as shown above) that can be used to automatically make code units (typically classes) provided by the packages available for use in the project. Adding the following statement to one of your project's PHP scripts:

require_once("vendor/autoload.php");

suffices to load the functionality exposed by the packages that composer installs.

Composer can be used to install both runtime and development dependencies. Many development dependencies (such as phpunit or phpdocumentor) provide command-line utilities to carry out tasks. Composer packages can also declare which executables they provide. Composer automatically generates symlinks for all provided executables in the: vendor/bin folder:

$ ls -l vendor/bin/
lrwxrwxrwx 1 sander users 29 Sep 26 11:49 jsonlint -> ../seld/jsonlint/bin/jsonlint
lrwxrwxrwx 1 sander users 41 Sep 26 11:49 phpdoc -> ../phpdocumentor/phpdocumentor/bin/phpdoc
lrwxrwxrwx 1 sander users 45 Sep 26 11:49 phpdoc.php -> ../phpdocumentor/phpdocumentor/bin/phpdoc.php
lrwxrwxrwx 1 sander users 34 Sep 26 11:49 pndp-build -> ../svanderburg/pndp/bin/pndp-build
lrwxrwxrwx 1 sander users 46 Sep 26 11:49 validate-json -> ../justinrainbow/json-schema/bin/validate-json

For example, you can run the following command-line instruction from the base directory of a project to generate API documentation:

$ vendor/bin/phpdocumentor -d src -t out

In some cases (the composer documentation often discourages this) you may want to install end-user packages globally. They can be installed into the global composer configuration directory by running:

$ composer global require phpunit/phpunit

After installing a package globally (and adding: $HOME/.config/composer/vendor/bin directory to the PATH environment variable), we should be able to run:

$ phpunit --help

The composer configuration

The deployment operations that composer carries out are driven by a configuration file named: composer.json. An example of such a configuration file could be:

{
"name": "svanderburg/composer2nix",
"description": "Generate Nix expressions to build PHP composer packages",
"type": "library",
"license": "MIT",
"authors": [
      {
"name": "Sander van der Burg",
"email": "svanderburg@gmail.com",
"homepage": "http://sandervanderburg.nl"
      }
  ],

"require": {
"svanderburg/pndp": "0.0.1"
  },
"require-dev": {
"phpdocumentor/phpdocumentor": "2.9.x"
  },

"autoload": {
"psr-4": { "Composer2Nix\\": "src/Composer2Nix" }
  },

"bin": [ "bin/composer2nix" ]
}

The above configuration file declares the following configuration properties:

• A number of meta attributes, such as the package name, description, license and authors.
• The package type. The type: library indicates that this project is a library that can be used in another project.
• The project's runtime (require) and development (require-dev) dependencies. In a dependency object, the keys refer to the package names and the values to version specifications that can be either:
  
  • A semver compatible version specifier that can be an exact version (e.g. 0.0.1), wildcard (e.g. 1.0.x), or version range (e.g. >= 1.0.0).
  • A version alias that directly (or indirectly) resolves to a branch in the VCS repository of the dependency. For example, the dev-master version specifier refers to the current master branch of the Git repository of the package.
• The autoloader configuration. In the above example, we configure the autoloader to load all classes belonging to the Composer2Nix namespace, from the src/Composer2Nix sub directory.

By default, composer obtains all packages from the Packagist repository. However, it is also possible to consult other kinds of repositories, such as external HTTP sites or VCS repositories of various kinds (including Git, Mercurial and Subversion).

External repositories can be specified by adding a 'repositories' object to the composer configuration:

{
"name": "svanderburg/composer2nix",
"description": "Generate Nix expressions to build PHP composer packages",
"type": "library",
"license": "MIT",
"authors": [
      {
"name": "Sander van der Burg",
"email": "svanderburg@gmail.com",
"homepage": "http://sandervanderburg.nl"
      }
  ],
"repositories": [
      {
"type": "vcs",
"url": "https://github.com/svanderburg/pndp"
      }
  ],

"require": {
"svanderburg/pndp": "dev-master"
  },
"require-dev": {
"phpdocumentor/phpdocumentor": "2.9.x"
  },

"autoload": {
"psr-4": { "Composer2Nix\\": "src/Composer2Nix" }
  },

"bin": [ "bin/composer2nix" ]
}

In the above example, we have defined PNDP's GitHub repository as an external repository and changed the version specifier of svanderburg/pndp to use the latest Git master branch.

Composer uses a version resolution strategy that will parse composer configuration files and branch names in all repositories to figure out where a version can be obtained from and takes the first option that matches the dependency specification. Packagist is consulted last, making it possible for the user to override dependencies.

Pinpointing dependency versions

The version specifiers of dependencies in a composer.json configuration file are nominal and have some drawbacks when it comes to reproducibility -- for example, the version specifier: >= 1.0.1 may resolve to version 1.0.2 today and to 1.0.3 tomorrow, making it very difficult to exactly reproduce a deployment elsewhere at a later point in time.

Although direct dependencies can be easily controlled by the user, it is quite difficult to control the version resolutions of the transitive dependencies. To cope with this problem, composer will always generate lock files (composer.lock) that pinpoint the exact dependency versions (including all transitive dependencies) the first time when it gets invoked (or when composer update is called):

{
"_readme": [
"This file locks the dependencies of your project to a known state",
"Read more about it at https://getcomposer.org/doc/01-basic-usage.md#composer-lock-the-lock-file",
"This file is @generated automatically"
    ],
"content-hash": "ca5ed9191c272685068c66b76ed1bae8",
"packages": [
        {
"name": "svanderburg/pndp",
"version": "v0.0.1",
"source": {
"type": "git",
"url": "https://github.com/svanderburg/pndp.git",
"reference": "99b0904e0f2efb35b8f012892912e0d171e9c2da"
            },
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/svanderburg/pndp/zipball/99b0904e0f2efb35b8f012892912e0d171e9c2da",
"reference": "99b0904e0f2efb35b8f012892912e0d171e9c2da",
"shasum": ""
            },
"bin": [
"bin/pndp-build"
            ],
            ...
        }
        ...
    ]
}

By bundling the composer.lock file with the package, it becomes possible to reproduce a deployment elsewhere with the exact same package versions.

The Nix package manager

Nix is a package manager whose main purpose is to build all kinds of software packages from source code, such as GNU Autotools, CMake, Perl's MakeMaker, Apache Ant, and Python projects.

Nix's main purpose is not be a build tool (it can actually also be used for building projects, but this application area is still highly experimental). Instead, Nix manages dependencies and complements existing build tools by providing dedicated build environments to make deployments reliable and reproducible, such as clearing all environment variables, making files read-only after the package has been built, restricting network access and resetting the files' timestamps to 1.

Most importantly, in these dedicated environments Nix ensures that only specified dependencies can be found. This may probably sound inconvenient at first, but this property exists for a good reason: if a package unknowingly depends on another package then it may work on the machine where it has been built, but may fail on another machine because this unknown dependency is missing. By building a package in a pure environment in which all dependencies are known, we eliminate this problem.

To provide stricter purity guarantees, Nix isolates packages by storing them in a so-called "Nix store" (that typically resides in: /nix/store) in which every directory entry corresponds to a package. Every path in the Nix store is prefixed by hash code, such as:

/nix/store/2gi1ghzlmb1fjpqqfb4hyh543kzhhgpi-firefox-52.0.1

The hash is derived from all build-time dependencies to build the package.

Because every package is stored in its own path and variants of packages never share the same name because of the hash prefix, it becomes harder for builds to accidentally succeed because of undeclared dependencies. Dependencies can only be found if the environment has been configured in such a way that the Nix store paths to the packages are known, for example, by configuring environment variables, such as: export PATH=/nix/store/5vyssyqvbirdihqrpqhbkq138ax64bjy-gnumake-4.2.1/bin.

The Nix expression language and build environment abstractions have all kinds of facilities to make the configuration of dependencies convenient.

Integrating composer deployments into Nix builder environments

Invoking composer in a Nix builder environment introduces an additional challenge -- composer is not only a tool that does build management (e.g. it can execute script directives that can carry out arbitrary build steps), but also dependency management. The latter property conflicts with the Nix package manager.

In a Nix builder environment, network access is typically restricted, because it affects reproducibility (although it still possible to hack around this restriction) -- when downloading a file from an external site it is not known in advance what you will get. An unknown artifact influences the outcome of a package build in unpredictable ways.

Network access in Nix build environments is only permitted in so-called fixed output derivations. For a fixed output derivation, the output hash must be known in advance so that Nix can verify whether we have obtained the artifact we want.

The solution to cope with a conflicting dependency manager is by substituting it -- we must let Nix obtain the dependencies and force the tool to only execute its build management tasks.

We can populate the vendor/ folder ourselves. As explained earlier, the composer.lock file stores the exact versions of pinpointed dependencies including all transitive dependencies. For example, when a project declares svanderburg/pndp version 0.0.1 as a dependency, it may translate to the following entry in the composer.lock file:

"packages": [
    {
"name": "svanderburg/pndp",
"version": "v0.0.1",
"source": {
"type": "git",
"url": "https://github.com/svanderburg/pndp.git",
"reference": "99b0904e0f2efb35b8f012892912e0d171e9c2da"
        },
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/svanderburg/pndp/zipball/99b0904e0f2efb35b8f012892912e0d171e9c2da",
"reference": "99b0904e0f2efb35b8f012892912e0d171e9c2da",
"shasum": ""
        },
        ...
    }
    ...
]

As can be seen in the code fragment above, the dependency translates to two kinds of pinpointed source objects -- a source reference to a specific revision in a Git repository and a dist reference to a zipball containing a snapshot of the given Git revision.

The reason why every dependency translates to two kinds of objects is that composer supports two kinds of installation modes: source (to obtain a dependency directly from a VCS) and dist (to obtain a dependency from a zipball).

We can translate the 'dist' reference into the following Nix function invocation:

"svanderburg/pndp" = {
  targetDir = "";
  src = composerEnv.buildZipPackage {
    name = "svanderburg-pndp-99b0904e0f2efb35b8f012892912e0d171e9c2da";
    src = fetchurl {
      url = https://api.github.com/repos/svanderburg/pndp/zipball/99b0904e0f2efb35b8f012892912e0d171e9c2da;
      sha256 = "19l7i7adp76bjf32x9a2ykm0r5cgcmi4wf4cm4127miy3yhs0n4y";
    };
  };
};

and the 'source' reference to the following Nix function invocation:

"svanderburg/pndp" = {
  targetDir = "";
  src = fetchgit {
    name = "svanderburg-pndp-99b0904e0f2efb35b8f012892912e0d171e9c2da";
    url = "https://github.com/svanderburg/pndp.git";
    rev = "99b0904e0f2efb35b8f012892912e0d171e9c2da";
    sha256 = "15i311dc0123v3ppa69f49ssnlyzizaafzxxr50crdfrm8g6i4kh";
  };
};

(As a sidenote: we need the targetDir property to provide compatibility with the deprecated PSR-0 autoloading standard. Old autoload packages can be stored in a sub folder of a package residing in the vendor/ structure.)

To generate the above function invocations, we need more than just the properties provided by the composer.lock file. Since download functions in Nix are fixed output derivations, we must compute the output hashes of the downloads by invoking a Nix prefetch script, such as nix-prefetch-url or nix-prefetch-git. The composer2nix generator will automatically invoke the appropriate prefetch script to augment the generated expressions with output hashes.

To ensure maximum compatibility with composer's behaviour, the dependencies obtained by Nix must be copied into to the vendor/ folder. In theory, symlinking would be more space efficient, but experiments have shown that some packages (such as phpunit) may attempt to load the project's autoload script, e.g. by invoking:

require_once(realpath("../../autoload.php"));

The above require invocation does not work if the dependency is a symlink -- the require path resolves to a path in the Nix store (e.g. /nix/store/...). The parent's parent path corresponds to /nix where no autoload script is stored. (As a sidenote: I have decided to still provide symlinking as an option for deployment scenarios where this is not an issue).

After some experimentation, I discovered that composer uses the following file to track which packages have been installed: vendor/composer/installed.json. The contents appears to be quite similar to the composer.lock file:

[
    {
"name": "svanderburg/pndp",
"version": "v0.0.1",
"version_normalized": "0.0.1.0",
"source": {
"type": "git",
"url": "https://github.com/svanderburg/pndp.git",
"reference": "99b0904e0f2efb35b8f012892912e0d171e9c2da"
        },
"dist": {
"type": "zip",
"url": "https://api.github.com/repos/svanderburg/pndp/zipball/99b0904e0f2efb35b8f012892912e0d171e9c2da",
"reference": "99b0904e0f2efb35b8f012892912e0d171e9c2da",
"shasum": ""
        },
        ...
    },
    ...
]

Reconstructing the above file can be done by merging the contents of the packages and packages-dev objects in the composer.lock file.

Another missing piece in the puzzle is the autoload scripts. We can force composer to dump the autoload script, by running:

$ composer dump-autoload --optimize

The above command generates an optimized autoloader script. A non-optimized autoload script dynamically inspects the contents of the package folders to load modules. This is convenient in the development stage of a project, in which the files continuously change, but in production environments this introduces quite a bit of load time overhead.

Since packages in the Nix store can never change after they have been built, it makes no sense to generate a non-optimized autoloader script.

Finally, the last remaining practical issue, is PHP packages providing command-line utilities. Most executables have the following shebang line:

#!/usr/bin/env php

To ensure that these CLI tools work in Nix builder environments, the above shebang must be subsituted by the PHP executable that resides in the Nix store.

After carrying out the above described steps, running the following command:

$ composer install --optimize-autoloader

is simply just a formality -- it will not download or change anything.

Use cases

composer2nix has a variety of use cases. The most obvious one is to use it to package a web application project with Nix instead of composer. Running the following command generates Nix expressions from the composer configuration files:

$ composer2nix

By running the following command, we can use Nix to obtain the dependencies and generate a package with a vendor/ folder:

$ nix-build
$ ls result/
index.php  vendor/

In addition to web applications, we can also deploy command-line utility projects implemented in PHP. For these kinds of projects it make more sense generate a bin/ sub folder in which the executables can be found.

For example, for the composer2nix project, we can generate a CLI-specific expression by adding the --executable parameter:

$ composer2nix --executable

We can install the composer2nix executable in our Nix profile by running:

$ nix-env -f default.nix -i

and then invoke composer2nix as follows:

$ composer2nix --help

We can also deploy third party command-line utilities directly from the Packagist repository:

$ composer2nix -p phpunit/phpunit
$ nix-env -f default.nix -iA phpunit-phpunit
$ phpunit --version

The most powerful application is not the integration with Nix itself, but the integration with other Nix projects. For example, we can define a NixOS configuration running an Apache HTTP server instance with PHP and our example web application:

{pkgs, config, ...}:

let
  myexampleapp = import /home/sander/myexampleapp {
    inherit pkgs;
  };
in
{
  services.httpd = {
    enable = true;
    adminAddr = "admin@localhost";
    extraModules = [
      { name = "php7"; path = "${pkgs.php}/modules/libphp7.so"; }
    ];
    documentRoot = myexampleapp;
  };

  ...
}

We can deploy the above NixOS configuration as follows:

$ nixos-rebuild switch

By running only one simple command-line instruction, we have a running system with the Apache webserver serving our web application.

Discussion

In addition to composer2nix, I have also been responsible for developing node2nix, a tool that generates Nix expressions from NPM package configurations. Because composer is heavily inspired by NPM, we see many similarities in the architecture of both generators. For example, both generate the same kinds of expressions (a builder environment, a packages expression and a composition expression), have a similar separation of concerns, and both use an internal DSL for generating Nix expressions (NiJS and PNDP).

There are also a number of conceptual differences -- dependencies in NPM can be private to a package or shared among multiple packages. In composer, all dependencies in a project are shared.

The reason why NPM's dependency management is more powerful is because Node.js uses the CommonJS module system. CommonJS considers each file to be a unique module. This, for example, makes it possible for one module to load a version of a package from a certain filesystem location and another version of the same package from another filesystem location within the same project.

By contrast, in PHP, isolation is accomplished by the namespace declarations in each file. Namespaces can not be dynamically altered so that multiple versions can safely coexist in one project. Furthermore, the vendor/ directory structure makes it possible to store only one variant of a package.

Despite the fact that composer's dependency management is less powerful makes constructing a generator much more straightforward compared to NPM.

Another feature that composer supports for quite some time, and NPM until very recently is pinpointing/locking dependencies. When generating Nix expressions from NPM package configurations, we must replicate NPM's dependency resolving algorithm. In composer, we can simply take whatever the composer.lock file provides. The lock file saves us from replicating the dependency lookup process making the generation process considerably easier.

Acknowledgments

My implementation is not the first attempt that tries to integrate composer with Nix. After a few days of developing, I discovered another attempt that seems to be in a very early development stage. I did not try or use this version.

Availability

composer2nix can be obtained from Packagist and my GitHub page.

Despite the fact that I did quite a bit of research, composer2nix should still be considered a prototype. One of its known limitations is that it does not support fossil repositories yet.

In a number earlier blog posts, I have described two kinds of internal DSLs for Nix -- NiJS is a JavaScript-based internal DSL and PNDP is a PHP-based internal DSL.

These internal DSLs have a variety of application areas. Most of them are simply just experiments, but the most serious application area is code generation.

Using an internal DSL for generation has a number of advantages over string generation that is more commonly used. For example, when composing strings containing Nix expressions, we must make sure that any variable in the host language that we append to a generated expression is properly escaped to prevent code injection attacks.

Furthermore, we also have to take care of the indentation if we want to output Nix expression code that should be readable. Finally, string manipulation itself is not a very intuitive activity as it makes it very hard to read what the generated code would look like.

Translating host language objects to the Nix expression language

A very important feature of both internal DSLs is that they can literally translate some language constructs from the host language (JavaScript or PHP) to the Nix expression because they have (nearly) an identical meaning. For example, the following JavaScript code fragment:

var nijs = require('nijs');

var expr = {
  hello: "Hello",
  name: {
    firstName: "Sander",
    lastName: "van der Burg"
  },
  numbers: [ 1, 2, 3, 4, 5 ]
};

var output = nijs.jsToNix(expr, true);
console.log(output);

will output the following Nix expression:

{
  hello = "Hello",
  name = {
    firstName = "Sander";
    lastName = "van der Burg";
  };
  numbers = [
    1
    2
    3
    4
    5
  ];
}

In the above example, strings will be translated to strings (and quotes will be escaped if necessary), objects to attribute sets, and the array of numbers to a list of numbers. Furthermore, the generated code is also pretty printed so that attribute set and list members have 2 spaces of indentation.

Similarly, in PHP we can compose the following code fragment to get an identical Nix output:

use PNDP\NixGenerator;

$expr = array(
"hello" => "Hello",
"name" => array(
"firstName" => "Sander",
"lastName => "van der Burg"
  ),
"numbers" => array(1, 2, 3, 4, 5)
);

$output = NixGenerator::phpToNix($expr, true);
echo($output);

The PHP generator uses a number of clever tricks to determine whether an array is associative or sequential -- the former gets translated into a Nix attribute set while the latter gets translated into a list.

There are objects in the Nix expression language for which no equivalent exists in the host language. For example, Nix also allows you to define objects of a 'URL' and 'file' type. Neither JavaScript nor PHP have a direct equivalent. Moreover, it may be desired to generate other kinds of language constructs, such as function declarations and function invocations.

To still generate these kinds of objects, you must compose an abstract syntax tree from objects that inherit from the NixObjectprototype or class. For example, we can define a function invocation to fetchurl {} in Nixpkgs as follows in JavaScript:

var expr = new nijs.NixFunInvocation({
    funExpr: new nijs.NixExpression("fetchurl"),
    paramExpr: {
        url: new nijs.NixURL("mirror://gnu/hello/hello-2.10.tar.gz"),
        sha256: "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"
    }
});

and in PHP as follows:

use PNDP\AST\NixExpression;
use PNDP\AST\NixFunInvocation;
use PNDP\AST\NixURL;

$expr = new NixFunInvocation(new NixExpression("fetchurl"), array(
"url" => new NixURL("mirror://gnu/hello/hello-2.10.tar.gz"),
"sha256" => "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"
));

Both of the objects in the above code fragments translate to the following Nix expression:

fetchurl {
  url = mirror://gnu/hello/hello-2.10.tar.gz;
  sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
}

Transforming custom object structures into Nix expressions

The earlier described use cases are basically one-on-one translations from the host language (JavaScript or PHP) to the guest language (Nix). In some cases, literal translations do not make sense -- for example, it may be possible that we already have an application with an existing data model from which we want to derive deployments that should be carried out with Nix.

In the latest versions of NiJS and PNDP, it is also possible to specify how to transform custom object structures into a Nix expression. This can be done by inheriting from the NixASTNode class or prototype and overriding the toNixAST() method.

For example, we may have a system already providing a representation of a file that should be downloaded from an external source:

function HelloSourceModel() {
    this.src = "mirror://gnu/hello/hello-2.10.tar.gz";
    this.sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
}

The above module defines a constructor function composing an object that refers to the GNU Hello package provided by a GNU mirror site.

A direct translation of an object constructed by the above function to the Nix expression language does not provide anything meaningful -- it can, for example, not be used to let Nix fetch the package from the mirror site.

We can inherit from NixASTNode and implement our own custom toNixAST() function to provide a more meaningful Nix translation:

var nijs = require('nijs');
var inherit = require('nijs/lib/ast/util/inherit.js').inherit;

/* HelloSourceModel inherits from NixASTNode */
inherit(nijs.NixASTNode, HelloSourceModel);

/**
 * @see NixASTNode#toNixAST
 */
HelloSourceModel.prototype.toNixAST = function() {
    return this.args.fetchurl()({
        url: new nijs.NixURL(this.src),
        sha256: this.sha256
    });
};

The toNixAST() function shown above composes an abstract syntax tree (AST) for a function invocation to fetchurl {} in the Nix expression language with the url and sha256 properties a parameters.

An object that inherits from the NixASTNode prototype also indirectly inherits from NixObject. This means that we can directly attach such an object to any other AST object. The generator uses the underlying toNixAST() function to automatically convert it to its AST representation:

var helloSource = new HelloSourceModel();
var output = nijs.jsToNix(helloSource, true);
console.log(output);

In the above code fragment, we directly pass the construct HelloSourceModel object instance to the generator. The output will be the following Nix expression:

fetchurl {
  url = mirror://gnu/hello/hello-2.10.tar.gz;
  sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
}

In some cases, it may not be possible to inherit from NixASTNode, for example, when the object already inherits from another prototype or class that is beyond the user's control.

It is also possible to use the NixASTNode constructor function as an adapter. For example, we can take any object with a toNixAST() function:

var helloSourceWrapper = {
    toNixAST: function() {
        return new nijs.NixFunInvocation({
            funExpr: new nijs.NixExpression("fetchurl"),
            paramExpr: {
                url: new nijs.NixURL(this.src),
                sha256: this.sha256
            }
        });
    }
};

By wrapping the helloSourceWrapper object in the NixASTNode constructor, we can convert it to an object that is an instance of NixASTNode:

new nijs.NixASTNode(helloSourceWrapper)

In PHP, we can change any class into a NixASTNode by implementing the NixASTConvertable interface:

use PNDP\AST\NixASTConvertable;
use PNDP\AST\NixURL;

class HelloSourceModel implements NixASTConvertable
{
    /**
     * @see NixASTConvertable::toNixAST()
     */
    public function toNixAST()
    {
        return $this->args->fetchurl(array(
"url" => new NixURL($this->src),
"sha256" => $this->sha256
        ));
    }
}

By passing an object that implements the NixASTConvertable interface to the NixASTNode constructor, it can be converted:

new NixASTNode(new HelloSourceModel())

Motivating use case: the node2nix and composer2nix generators

My main motivation to use custom transformations is to improve the quality of the node2nix and composer2nix generators -- the former converts NPM package configurations to Nix expressions and the latter converts PHP composer package configurations to Nix expressions.

Although NiJS and PNDP provide a number of powerful properties to improve the code generation steps of these tools, e.g. I no longer have to think much about escaping strings or pretty printing, there are still many organizational coding issues left. For example, the code that parses the configurations, fetches the external sources, and generates the code are mixed. As a consequence, the code is very hard to read, update, maintain and to ensure its correctness.

The new transformation facilities allow me to separate concerns much better. For example, both generators now have a data model that reflects the NPM and composer problem domain. For example, I could compose the following (simplified) class diagram for node2nix's problem domain:


A crucial part of node2nix's generator is the package class shown on the top left on the diagram. A package requires zero or more packages as dependencies and may provide zero or more packages in the node_modules/ folder residing in the package's base directory.

For readers not familiar with NPM's dependency management: every package can install its dependencies privately in a node_modules/ folder residing in the same base directory. The CommonJS module ensures that every file is considered to be a unique module that should not interfere with other modules. Sharing is accomplished by putting a dependency in a node_modules/ folder of an enclosing parent package.

NPM 2.x always installs a package dependency privately unless a parent package exists that can provide a conforming version. NPM 3.x (and later) will also move a package into the node_modules/ folder hierarchy as high as possible to prevent too many layers of nested node_modules/ folders (this is particularly a problem on Windows). The class structure in the above diagram reflects this kind of dependency organisation.

In addition to a package dependency graph, we also need to obtain package metadata and compute their output hashes. NPM packages originate from various kinds of sources, such as the NPM registry, Git repositories, HTTP sites and local directories on the filesystem.

To optimize the process and support sharing of common sources among packages, we can use a source cache that memorizes all unique source referencess.

The Package::resolveDependencies() method sets the generation process in motion -- it will construct the dependency graph replicating NPM's dependency resolution algorithm as faithfully as possible, and resolves all the dependencies' (and transitive dependencies) metadata.

After resolving all dependencies and their metadata, we must generate the output Nix expressions. One Nix expression is copied (the build infrastructure) and two are generated -- a composition expression and a package or collection expression.

We can also compose a class diagram for the generation infrastructure:


In the above class diagram, every generated expression is represented a class inheriting from NixASTNode. We can also reuse some classes from the domain model as constituents for the generated expressions, by also inheriting from NixASTNode and overriding the toNixAST() method:

• The source objects can be translated into sub expressions that invoke fetchurl {} and fetchgit {}.
• The sources cache can be translated into an attribute set exposing all sources that are used as dependencies for packages.
• A package instance can be converted into a function invocation to nodeenv.buildNodePackage {} that, in addition to configuring build properties, binds the required dependencies to the sources in the sources cache attribute set.

By decomposing the expression into objects and combining the objects' AST representations, we can nicely modularize the generation process.

For composer2nix, we can also compose a class diagram for its domain -- the generation process:


The above class diagram has many similarities, but also some major differences compared to node2nix. composer provides so-called lock files that pinpoint the exact versions of all dependencies and transitive dependencies. As a result, we do not need to replicate composer's dependency resolution algorithm.

Instead, the generation process is driven by the ComposerConfig class that encapsulates the properties of the composer.json and composer.lock files of a package. From a composer configuration, the generator constructs a package object that refers to the package we intend to deploy and populates a source cache with source objects that come from various sources, such as Git, Mercurial and Subversion repositories, Zip files, and directories residing on the local filesystem.

For the generation process, we can adopt a similar strategy that exposes the generated Nix expressions as classes and uses some classes of the domain model as constituents for the generation process:

Discussion

In this blog post, I have described a new feature for the NiJS and PNDP frameworks, making it possible to implement custom transformations. Some of its benefits are that it allows an existing object model to be reused and concerns in an application can be separated much more conveniently.

These facilities are not only useful for the improvement of the architecture of the node2nix and composer2nix generators -- at the company I work for (Conference Compass), we developed our own domain-specific configuration management tool.

Despite the fact that it uses several tools from the Nix project to carry out deployments, it uses a domain model that is not Nix-specific at all. Instead, it uses terminology and an organization that reflects company processes and systems.

For example, we use a backend-for-frontend organization that provides a backend for each mobile application that we ship. We call these backends configurators. Optionally, every configurator can import data from various external sources that we call channels. The tool's data model reflects this kind of organization, and generates Nix expressions that contain all relevant implementation details, if necessary.

Finally, the fact that I modified node2nix to have a much cleaner architecture has another reason beyond quality improvement. Currently, NPM version 5.x (that comes with Node.js 8.x) is still unsupported. To make it work with Nix, we require a slightly different generation process and a completely different builder environment. The new architecture allows me to reuse the common parts much more conveniently. More details about the new NPM support will follow (hopefully) soon.

Availability

I have released new versions of NiJS and PNDP that have the custom transformation facilities included.

Furthermore, I have decided to release new versions for node2nix and composer2nix that use the new generation facilities in their architecture. The improved architecture revealed a very uncommon but nasty bug with bundled dependencies in node2nix, that is now solved.

In a number of earlier blog posts, I have elaborated about the development of custom configuration management solutions that use Nix to carry out a variety (or all) of its deployment tasks.

For example, an interesting consideration is that when a different programming language than Nix's expression language is used, an internal DSL can be used to make integration with the Nix expression language more convenient and safe.

Another problem that might surface when developing a custom solution is the scale at which builds are carried out. When it is desired to carry out builds on a large scale, additional concerns must be addressed beyond the solutions that the Nix package manager provides, such as managing a build queue, timing out builds that appear to be stuck, and sending notifications in case of a success or failure.

In such cases, it may be very tempting to address these concerns in your own custom build solution. However, some of these concerns are quite difficult to implement, such as build queue management.

There is already a Nix project that solves many of these problems, namely: Hydra, the Nix-based continuous integration service. One of Hydra's lesser known features is that it also exposes many of its operations through a REST API. Apart from a very small section in the Hydra manual, the API is not very well documented and -- as a result -- a bit cumbersome to use.

In this blog post, I will describe a recently developed Node.js package that exposes most of Hydra's functionality with an API that is documented and convenient to use. It can be used to conveniently integrate a Node.js application with Hydra's build management services. To demonstrate its usefulness, I have developed a simple command-line utility that can be used to remotely control a Hydra instance.

Finding relevant API calls

The Hydra API is not extensively documented. The manual has a small section titled: "Using the external API" that demonstrates some basic usage scenarios, such as querying data in JSON format.

Querying data is basically a nearly identical operation to opening the Hydra web front-end in a web browser. The only difference is that the GET request should include a header field stating that the output format should be displayed in JSON format.

For example, the following request fetches an overview of projects in JSON format (as opposed to HTML which is the default):

$ curl -H "Accept: application/json" https://hydra.nixos.org

In addition to querying data in JSON format, Hydra supports many additional REST operations, such as creating, updating or deleting projects and jobsets. Unfortunately, these operations are not documented anywhere in the manual.

By analyzing the Hydra source code and running the following script I could (sort of) semi-automatically derive the REST operations that are interesting to invoke:

$ cd hydra/src/lib/Hydra/Controller
$ find . -type f | while read i
do
    echo "# $i"
    grep -E '^sub [a-zA-Z0-9]+[ ]?\:' $i
    echo
done

Running the script shows me the following (partial) output:

# ./Project.pm
sub projectChain :Chained('/') :PathPart('project') :CaptureArgs(1) {
sub project :Chained('projectChain') :PathPart('') :Args(0) :ActionClass('REST::ForBrowsers') { }
sub edit : Chained('projectChain') PathPart Args(0) {
sub create : Path('/create-project') {
...

The web front-end component of Hydra uses Catalyst, a Perl-based MVC framework. One of the conventions that Catalyst uses is that every request invokes an annotated Perl subroutine.

The above script scans the controller modules, and shows for each module annotated subroutines that may be of interest. In particular, the sub routines with a :ActionClass('REST::ForBrowsers') annotation are relevant -- they encapsulate create, read, update and delete operations for various kinds of data.

For example, the project subroutine shown above supports the following REST operations:

sub project_GET {
    ...
}

sub project_PUT {
    ...
}

sub project_DELETE {
    ...
}

The above operations will query the properties a project (GET), create a new or update an existing project (PUT) or delete a project (DELETE).

With the manual, the extracted information and reading the source code a bit (which is unavoidable since many details are missing such as formal function parameters), I was able to develop a client API supporting a substantial amount of Hydra features.

API usage

Using client the API is relatively straight forward. The general idea is to instantiate the HydraConnector prototype to connect to a Hydra server, such as a local test instance:

var HydraConnector = require('hydra-connector').HydraConnector;

var hydraConnector = new HydraConnector("http://localhost");

and then invoke any of its supported operations:

hydraConnector.queryProjects(function(err, projects) {
    if(err) {
        console.log("Some error occurred: "+err);
    } else {
        for(var i = 0; i < projects.length; i++) {
            var project = projects[i];
            console.log("Project: "+project.name);
        }
    }
});

The above code fragment fetches all projects and displays their names.

Write operations require a user to be logged in with the appropriate permissions. By invoking the login() method, we can authenticate ourselves:

hydraConnector.login("admin", "myverysecretpassword", function(err) {
    if(err) {
        console.log("Login succeeded!");
    } else {
        console.log("Some error occurred: "+err);
    }
});

Besides logging in, the client API also implements a logout() operation to relinquish write operation rights.

As explained in an earlier blog post, write operations require user authentication but all data is publicly readable. When it is desired to restrict read access, Hydra can be placed behind a reverse proxy that requires HTTP basic authentication.

The client API also supports HTTP basic authentication to support this usage scenario:

var hydraConnector = new HydraConnector("http://localhost",
"sander",
"12345"); // HTTP basic credentials

Using the command-line client

To demonstrate the usefulness of the API, I have created a utility that serves as a command-line equivalent of Hydra's web front-end. The following command shows an overview of projects:

$ hydra-connect --url https://hydra.nixos.org --projects

As may be observed in the screenshot above, the command-line utility provides suggestions for additional command-line instructions that could be relevant, such as querying more detailed information or modifying data.

The following command shows the properties of an individual project:

$ hydra-connect --url https://hydra.nixos.org --project disnix

By default, the command-line utility will show a somewhat readable textual representation of the data. It is also possible to display the "raw" JSON data of any request for debugging purposes:

$ hydra-connect --url https://hydra.nixos.org --project disnix --json

We can also use the command-line utility to create or edit data, such as a project:

$ hydra-connect --url https://hydra.nixos.org --login
$ export HYDRA_SESSION=...
$ hydra-connect --url https://hydra.nixos.org --project disnix --modify

The application will show command prompts asking the user to provide all the relevant project properties.

When changes have been triggered, the active builds in the queue can be inspected by running:

$ hydra-connect --url https://hydra.nixos.org --status

Many other operations are supported. For example, you can inspect the properties of an individual build:

$ hydra-connect --url https://hydra.nixos.org --build 65100054

And request various kinds of build related artifacts, such as the raw build log of a build:

$ hydra-connect --url https://hydra.nixos.org --build 65100054 --raw-log

or download one of its build products:

$ hydra-connect --url https://hydra.nixos.org --build 65100054 \
  --build-product 4 > /home/sander/Download/disnix-0.7.2.tar.gz

Conclusion

In this blog post, I have shown the node-hydra-connector package that can be used to remotely control a Hydra server from a Node.js application. The package can be obtained from the NPM registry and the corresponding GitHub repository.

Roughly half a year ago, Node.js version 8 was released that also includes a major NPM package manager update (version 5). NPM version 5 has a number of substantial changes over the previous version, such as:

• It uses package lock files that pinpoint the resolved versions of all dependencies and transitive dependencies. When a project with a bundled package-lock.json file is deployed, NPM will use the pinpointed versions of the packages that are in the lock file making it possible to exactly reproduce a deployment elsewhere. When a project without a lock file is deployed for the first time, NPM will generate a lock file.
• It has a content-addressable cache that optimizes package retrieval processes and allows fully offline package installations.
• It uses SHA-512 hashing (as opposed to the significantly weakened SHA-1), for packages published in the NPM registry.

Although these features offer significant benefits over previous versions, e.g. NPM deployments are now much faster, more secure and more reliable, it also comes with a big drawback -- it breaks the integration with the Nix package manager in node2nix. Solving these problems were much harder than I initially anticipated.

In this blog post, I will explain how I have adjusted the generation procedure to cope with NPM's new conflicting features. Moreover, I have extended node2nix with the ability to generate Nix expressions from package-lock.json files.

Lock files

One of the major new features in NPM 5.0 is the lock file (the idea itself is not so new since NPM-inspired solutions such as yarn and the PHP-based composer already support them for quite some time).

A major drawback of NPM's dependency management is that version specifiers are nominal. They can refer to specific versions of packages in the NPM registry, but also to version ranges, or external artifacts such as Git repositories. The latter category of version specifiers affect reproducibility -- for example, the version range specifier >= 1.0.0 may refer to version 1.0.0 today and to version 1.0.1 tomorrow making it extremely hard to reproduce a deployment elsewhere.

In a development project, it is still possible to control the versions of dependencies by using a package.json configuration that only refers to exact versions. However, for transitive dependencies that may still have loose version specifiers there is only very little control.

To solve this reproducibility problem, a package-lock.json file can be used -- a package lock file pinpoints the resolved versions of all dependencies and transitive dependencies making it possible to reproduce the exact same deployment elsewhere.

For example, for the NiJS package with the following package.json configuration:

{
"name": "nijs",
"version": "0.0.25",
"description": "An internal DSL for the Nix package manager in JavaScript",
"bin": {
"nijs-build": "./bin/nijs-build.js",
"nijs-execute": "./bin/nijs-execute.js"
  },
"main": "./lib/nijs",
"dependencies": {
"optparse": ">= 1.0.3",
"slasp": "0.0.4"
  },
"devDependencies": {
"jsdoc": "*"
  }
}

NPM may produce the following partial package-lock.json file:

{
"name": "nijs",
"version": "0.0.25",
"lockfileVersion": 1,
"requires": true,
"dependencies": {
"optparse": {
"version": "1.0.5",
"resolved": "https://registry.npmjs.org/optparse/-/optparse-1.0.5.tgz",
"integrity": "sha1-dedallBmEescZbqJAY/wipgeLBY="
    },
"requizzle": {
"version": "0.2.1",
"resolved": "https://registry.npmjs.org/requizzle/-/requizzle-0.2.1.tgz",
"integrity": "sha1-aUPDUwxNmn5G8c3dUcFY/GcM294=",
"dev": true,
"requires": {
"underscore": "1.6.0"
      },
"dependencies": {
"underscore": {
"version": "1.6.0",
"resolved": "https://registry.npmjs.org/underscore/-/underscore-1.6.0.tgz",
"integrity": "sha1-izixDKze9jM3uLJOT/htRa6lKag=",
"dev": true
        }
      }
    },
    ...
}

The above lock file pinpoints all dependencies and development dependencies including transitive dependencies to exact versions, including the locations where they can be obtained from and integrity hash codes that can be used to validate them.

The lock file can also be used to derive the entire structure of the node_modules/ folder in which all dependencies are stored. The top level dependencies property captures all packages that reside in the project's node_modules/ folder. The dependencies property of each dependency captures all packages that reside in a dependency's node_modules/ folder.
If NPM 5.0 is used and no package-lock.json is present in a project, it will automatically generate one.

Substituting dependencies

As mentioned in an earlier blog post, the most important technique to make Nix-NPM integration work is by substituting NPM's dependency management activities that conflict with Nix's dependency management -- Nix is much more strict with handling dependencies (e.g. it uses hash codes derived from the build inputs to identify a package as opposed to a name and version number).

Furthermore, in Nix build environments network access is restricted to prevent unknown artifacts to influence the outcome of a build. Only so-called fixed output derivations, whose output hashes should be known in advance (so that Nix can verify its integrity), are allowed to obtain artifacts from external sources.

To substitute NPM's dependency management, populating the node_modules/ folder ourselves with all required dependencies and substituting certain version specifiers, such as Git URLs, used to suffice. Unfortunately, with the newest NPM this substitution process no longer works. When running the following command in a Nix builder environment:

$ npm --offline install ...

The NPM package manager is forced to work in offline mode consulting its content-addressable cache for the retrieval of external artifacts. If NPM needs to consult an external resource, it throws an error.

Despite the fact that all dependencies are present in the node_modules/ folder, deployment fails with the following error message:

npm ERR! code ENOTCACHED
npm ERR! request to https://registry.npmjs.org/optparse failed: cache mode is 'only-if-cached' but no cached response available.

At first sight, the error message suggests that NPM always requires the dependencies to reside in the content-addressable cache to prevent it from downloading it from external sites. However, when we use NPM outside a Nix builder environment, wipe the cache, and perform an offline installation, it does seem to work properly:

$ npm install
$ rm -rf ~/.npm/_cacache
$ npm --offline install

Further experimentation reveals that NPM augments the package.json configuration files of all dependencies with additional metadata that are prefixed by an underscore (_):

{
"_from": "optparse@>= 1.0.3",
"_id": "optparse@1.0.5",
"_inBundle": false,
"_integrity": "sha1-dedallBmEescZbqJAY/wipgeLBY=",
"_location": "/optparse",
"_phantomChildren": {},
"_requested": {
"type": "range",
"registry": true,
"raw": "optparse@>= 1.0.3",
"name": "optparse",
"escapedName": "optparse",
"rawSpec": ">= 1.0.3",
"saveSpec": null,
"fetchSpec": ">= 1.0.3"
  },
"_requiredBy": [
"/"
  ],
"_resolved": "https://registry.npmjs.org/optparse/-/optparse-1.0.5.tgz",
"_shasum": "75e75a96506611eb1c65ba89018ff08a981e2c16",
"_spec": "optparse@>= 1.0.3",
"_where": "/home/sander/teststuff/nijs",
"name": "optparse",
"version": "1.0.5",
  ...
}

It turns out that when the _integrity property in a package.json configuration matches the integrity field of the dependency in the lock file, NPM will not attempt to reinstall it.

To summarize, the problem can be solved in Nix builder environments by running a script that augments the package.json configuration files with _integrity fields with the values from the package-lock.json file.

For Git repository dependency specifiers, there seems to be an additional requirement -- it also seems to require the _resolved field to be set to the URL of the repository.

Reconstructing package lock files

The fact that we have discovered how to bypass the cache in a Nix builder environment makes it possible to fix the integration with the latest NPM. However, one of the limitations of this approach is that it only works for projects that have a package-lock.json file included.

Since lock files are still a relatively new concept, many NPM projects (in particular older projects that are not frequently updated) may not have a lock file included. As a result, their deployments will still fail.

Fortunately, we can reconstruct a minimal lock file from the project's package.json configuration and compose dependencies objects by traversing the package.json configurations inside the node_modules/ directory hierarchy.

The only attribute that cannot be immediately derived are the integrity fields containing hashes that are used for validation. It seems that we can bypass the integrity check by providing a dummy hash, such as:

integrity: "sha1-000000000000000000000000000=",

NPM does not seem to object when it encounters these dummy hashes allowing us to deploy projects with a reconstructed package-lock.json file. The solution is a very ugly hack, but it seems to work.

Generating Nix expressions from lock files

As explained earlier, lock files pinpoint the exact versions of all dependencies and transitive dependencies and describe the structure of the entire dependency graph.

Instead of simulating NPM's dependency resolution algorithm, we can also use the data provided by the lock files to generate Nix expressions. Lock files appear to contain most of the data we need -- the URLs/locations of the external artifacts and integrity hashes that we can use for validation.

Using lock files for generation offer the following advantages:

• We no longer need to simulate NPM's dependency resolution algorithm. Despite my best efforts and fairly good results, it is hard to truly make it 100% identical to NPM's. When using a lock file, the dependency graph is already given, making deployment results much more accurate.
• We no longer need to consult external resources to resolve versions and compute hashes making the generation process much faster. The only exception seems to be Git repositories -- Nix needs to know the output hash of the clone whereas for NPM the revision hash suffices. When we encounter a Git dependency, we still need to download it and compute the output hash.

Another minor technical challenge are the integrity hashes -- in NPM lock files integrity hashes are in base-64 notation, whereas Nix uses heximal notation or its own custom base-32 notation. We need to convert the NPM integrity hashes to a notation that Nix understands.

Unfortunately, lock files can only be used in development projects. It appears that packages that are installed directly from the NPM registry, e.g. end-user packages that are installed globally through npm install -g, never include a package lock file. (It even seems that the NPM registry blacklist the lock files when publishing a package in the registry).

For this reason, we still need to keep our own implementation of the dependency resolution algorithm.

Usage

By adding a script that augments the dependencies'package.json configuration files with _integrity fields and by optionally reconstructing a package-lock.json file, NPM integration with Nix has been restored.

Using the new NPM 5.x features is straight forward. The following command can be used to generate Nix expressions for a development project with a lock file:

$ node2nix -8 -l package-lock.json

The above command will directly generate Nix expressions from the package lock file, resulting in a much faster generation process.

When a development project does not ship with a lock file, you can use the following command-line instruction:

$ node2nix -8

The generator will use its own implementation of NPM's dependency resolution algorithm. When deploying the package, the builder will reconstruct a dummy lock file to allow the deployment to succeed.

In addition to development projects, it is also possible to install end-user software, by providing a JSON file (e.g. pkgs.json) that defines an array of dependency specifiers:

[
"nijs"
, { "node2nix": "1.5.0" }
]

A Node.js 8 compatible expression can be generated as follows:

$ node2nix -8 -i pkgs.json

Discussion

The approach described in this blog post is not the first attempt to fix NPM 5.x integration. In my first attempt, I tried populating NPM's content-addressable cache in the Nix builder environment with artifacts that were obtained by the Nix package manager and forcing NPM to work in offline mode.

NPM exposes its download and cache-related functionality as a set of reusable APIs. For downloading packages from the NPM registry, pacote can be used. For downloading external artifacts through the HTTP protocol make-fetch-happen can be used. Both APIs are built on top of the content-addressable cache that can be controlled through the lower-level cacache API.

The real difficulty is that neither the high-level NPM APIs nor the npm cache command-line instruction work with local directories or local files -- they will only add artifacts to the cache if they come from a remote location. I have partially built my own API on top of cacache to populate the NPM cache with locally stored artifacts pretending that they were fetched from a remote location.

Although I had some basic functionality supported, it turned out to be much more complicated and time consuming to get all functionality implemented.

Furthermore, the NPM authors never promised that these APIs are stable, so the implementation may break at some point in time. As a result, I have decided to look for another approach.

Availability

I just released node2nix version 1.5.0 with NPM 5.x support. It can be obtained from the NPM registry, Github, or directly from the Nixpkgs repository.