Implementing a wrapper for .NET applications

using System;
using System.Reflection;
using System.IO;

namespace HelloWorldWrapper
{
    class HelloWorldWrapper
    {
        private String[] AssemblySearchPaths = {
          @"C:\cygwin\nix\store\23ga...-ALibrary",
          @"C:\cygwin\nix\store\833p...-BLibrary"
        };

        private String ExePath =
          @"C:\cygwin\nix\store\27f2...-Executable\Executable.exe";

        private String MainClassName =
          "SomeExecutable.Executable";

        public HelloWorldWrapper(string[] args)
        {
            // Attach the resolve event handler to the AppDomain
            // so that missing library assemblies will be searched
            AppDomain currentDomain = AppDomain.CurrentDomain;
            currentDomain.AssemblyResolve +=
              new ResolveEventHandler(MyResolveEventHandler);

            // Dynamically load the executable assembly
            Assembly exeAssembly = Assembly.LoadFrom(ExePath);

            // Lookup the main class
            Type mainClass = exeAssembly.GetType(MainClassName);

            // Lookup the main method
            MethodInfo mainMethod = mainClass.GetMethod("Main");

            // Invoke the main method
            mainMethod.Invoke(this, new Object[] {args});
        }

        static void Main(string[] args)
        {
            new HelloWorldWrapper(args);
        }

        private Assembly MyResolveEventHandler(object sender,
          ResolveEventArgs args)
        {
            // This handler is called only when the common language
            // runtime tries to bind to the assembly and fails.

            Assembly MyAssembly;
            String assemblyPath = "";
            String requestedAssemblyName =
              args.Name.Substring(0, args.Name.IndexOf(","));

            // Search for the right path of the library assembly
            foreach (String curAssemblyPath in AssemblySearchPaths)
            {
                assemblyPath = curAssemblyPath + "/" +
                  requestedAssemblyName + ".dll";

                if (File.Exists(assemblyPath))
                    break;
            }

            // Load the assembly from the specified path. 
            MyAssembly = Assembly.LoadFrom(assemblyPath);

            // Return the loaded assembly.
            return MyAssembly;
        }

    }
}

namespace SomeNamespace
{
    class SomeClass
    {
       static void Main(string[] args)
       {
       }
    }
}

Usage

{dotnetenv, MyAssembly1, MyAssembly2}:

dotnetenv.buildWrapper {
  name = "My.Test.Assembly";
  src = /path/to/source/code;
  slnFile = "Assembly.sln";
  assemblyInputs = [
    dotnetenv.assembly20Path
    MyAssembly1
    MyAssembly2
  ];
  namespace = "TestNamespace";
  mainClassName = "MyTestApplication";
  mainClassFile = "MyTestApplication.cs";
}

Conclusion

References

• This page explains how the reflection API can be used to dynamically load assemblies: http://www.codeproject.com/KB/cs/DynLoadClassInvokeMethod.aspx
• The following page explains various ways to load library assemblies beyond the base directory of the executable. It also includes an explanation of the AssemblyResolve event: http://support.microsoft.com/kb/837908

What is software deployment?

Software deployment refers to all the activities that make a software system
available for use

• Building software components from source code
• Packaging software
• Transferring the software from the producer site to consumer site
• Installation of the software system
• Activation of the software system
• Software upgrades

Relationship to software engineering

Software Engineering (SE) is the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software, and the study of these approaches; that is, the application of engineering to software.

Why is software deployment complicated?

Importance of software deployment

• First of all, (not surprisingly) software systems become bigger and increasingly more complex. Nowadays, some software systems are not only composed of many components, but these components are also distributed and deployed on various machines in a network working together to achieve a common goal. For example, service-oriented systems are composed this way. Deploying such systems manually is a very time consuming, complex, error prone and tedious process. The bigger the system gets, the more likely it becomes that an error occurs.
• We have to be more flexible in reacting to events. For example, in a cloud infrastructure, if a machine breaks, we must be able to redeploy the system in such a way that services are still available to end-users, limiting the impact as much as possible.
  
• We want to push changes to a system in production environment faster. Because systems become increasingly more complex, an automated deployment solution is essential. In Agile software development projects, a team wants to generate value as quickly as possible, for which it is essential to have a working system in a production environment as soon as possible. To achieve this goal, it is crucial that the deployment process can be performed without much trouble. A colleague of mine (Rini van Solingen), who is also a Scrum consultant, has covered this importance in a video blog interview.
  

Research

• Mechanics. This field concerns the execution of the deployment activities. How can we make these steps reproducible, reliable, efficient? Most of the research that I do covers deployment mechanics.
• Deployment planning. Where to place a component in a network of machines? How to compose components together?
• Empirical research covering various aspects of deployment activities, such as: How to quantify build maintenance effort? How much maintenance is needed to keep deployment specifications (such as build specifications) up to date?

Conclusion

References

• [1] Carzaniga et al., A Characterization Framework for Software Deployment Technologies, 1998
• [2] The Joint Task Force on Computing Curricula, Software Engineering 2004

Modifications to Disnix

• mssql-database. This activation script loads a schema on initial startup if the database does not exists. It uses the OSQL.EXE tool included with Microsoft SQL server, to automatically execute the SQL instructions from shell scripts to check whether the database exists and to create the tables if needed.
• iis-webapplication. This activation script activates or deactivates a web application on Microsoft Internet Information Services. It uses the MSDeploy.exe tool to automatically activate a web application or deactivate a web application.

Installing Disnix on Windows

• I have used the following instructions to install Microsoft SQL server: http://visualcsharptutorials.com/2011/05/installing-sql-server-2008-express. Important is that you need to configure a SQL server user account with administration rights, a.k.a. the 'sa' user, which must use SQL server authentication. More information about this can be found here: http://www.eukhost.com/forums/f15/login-failed-user-sa-microsoft-sql-server-error-18456-a-12544/
• Microsoft IIS was already installed on my machine, but perhaps this page can act as a reference: http://learn.iis.net/page.aspx/28/installing-iis-on-windows-vista-and-windows-7. Important is that you include the Web Deployment tool, which is not enabled by default.

Running the Disnix daemon

$ cygrunsrv -I dbus -p /usr/sbin/dbus-daemon.exe -a \
    '--system --nofork'

$ cygrunsrv -I disnix -p /usr/local/bin/disnix-service.exe -a \
  '--activation-modules-dir /usr/local/libexec/disnix/activation-scripts' \
  -e 'PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin' \
  -y dbus

$ ssh-host-config

Example case

Distributed deployment

{dotnetenv}:
{zipcodes}:

dotnetenv.buildSolution {
  name = "ZipcodeService";
  src = ../../../../services/webservices/ZipcodeService;
  baseDir = "ZipcodeService";
  slnFile = "ZipcodeService.csproj";
  targets = "Package";
  preBuild = ''
    sed -e 's|.\SQLEXPRESS|${zipcodes.target.hostname}\SQLEXPRESS|' \
        -e 's|Initial Catalog=zipcodes|Initial catalog=${zipcodes.name}|' \
        -e 's|User ID=sa|User ID=${zipcodes.target.msSqlUsername}|' \
        -e 's|Password=admin123$|Password=${zipcodes.target.msSqlPassword}|' \
        Web.config
  '';
}

$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix

Limitations

• Disnix is used to manage the service components of a system. Disnix does not deploy infrastructure components, such as web server or database server. DisnixOS is a NixOS based extension that takes care of this. However, DisnixOS cannot be used on Windows, because SQL server and IIS are tightly integrated into the Windows operating system and registry. We cannot use the Nix store to safely isolate them. You need to either install these infrastructure components manually or use other deployment solutions.
• As mentioned in our previous blog post about .NET deployment, the .NET framework needs to be installed manually in order to be able to build Visual Studio projects. On most Windows installations, however, the .NET framework is already included.
• The .NET build functions and activation scripts are quite new and not very well tested. Moreover, they could also break, because we currently have no way to automatically test them like we do with Linux software.
• Also not all desired deployment features may be supported. I'd like to have feedback on this :-)
  

References

• To use the .NET deployment features described in this blog post, you need to obtain the latest pre-release of Disnix, which can be downloaded from the Disnix Hydra project page
• The StaffTracker .NET example case, can also be downloaded from the Disnix Hydra project page and can be freely used under the MIT license.
• A description of the StaffTracker example case can be found in our WASDeTT paper, titled: 'Disnix: A toolset for distributed deployment' which can be obtained from my publications page.
• I have used the following web page as a reference for writing WCF web services: http://www.codeproject.com/KB/WCF/first_WCF_Service.aspx
• The following page is used as a reference for ADO.NET: http://www.csharp-station.com/Tutorials/AdoDotNet/Lesson02.aspx
• The W3Schools tutorial about ASP.NET was sufficient for me, to understand it.
• I have recently presented this subject at Philips. The slides can be found on my talks page.

What is the Filesystem Hierarchy Standard?

• It defines directory names and their purposes. For example, the bin directory is for storing executable binaries, sbin is for storing executable binaries only accessible by the super-user, lib is for storing libraries.
• It defines several hierarchies within the filesystem, which have separate purposes:
  
  The primary / hierarchy contains essential components for the boot process and system recovery. The secondary hierarchy: /usr contains components not relevant for booting and recovery. Moreover, files in this directory should be shareable across multiple machines, e.g. through a network filesystem. The tertiary hierarchy: /usr/local is for the system administrator to install software locally.
  
  Hierarchies and special purpose directories are usually combined. For example, the /bin directory contains executable binaries relevant for booting and recovery which can be used by anyone. The /usr/bin directory contains executable binaries not relevant for booting or recovery, such as a web browser, which can be shared across multiple machines. The /usr/local/bin directory contains local executables installed by the system administrator.
  
  The FHS also defines the /opt directory for installing add-on application software packages. This directory can also be considered a separate hierarchy, although it is not defined as such in the standard. This directory also contains the same special purpose directories, such as bin and lib like the primary, secondary and tertiary hierarchies. Furthermore, also the /opt/<package name> convention may be used, which stores files specific to a particular application in a single folder.
  
• It also makes a distinction between static and variable parts of a system. For example, the contents of the secondary hierarchy /usr are static and could be stored on a read-only mounted partition. However, many programs need to modify their state at runtime and store files on disks, such as cache and log files. These files are stored in variable storage directories, such as /var and /tmp.
  
• It defines what the contents of some folders should look like. For example, a list of binaries which should reside in /bin, such as a Bourne compatible shell.
  
• The standard has some practical problems. Some aspects of the filesystem are undefined and need clarification, such as a convention to store cross compiling libraries. Furthermore, the standard is quite old and newer Linux features such as the /sys directory are not defined in the standard. To cope with these issues, many distributions have additional clarifications and policies, such as the Debian FHS policy.

Linux Standard Base

NixOS and the FHS

/nix/store/r8vvq9kq18pz08v249h8my6r9vs7s0n3-firefox-8.0.1

Discussion

• How to determine files for the boot process and recovery? Because we don't use hierarchies, files relevant for the boot process can't be found in /bin and /lib. In NixOS, we basically generate a small base system as a Nix component serving this purpose, which is stored as a single component in the Nix store.
  
• How to share components? Because we have no secondary hierarchy, you can't share components by storing the /usr directory on a network file system. In Nix, however, the entire Nix store is static and shareable. Furthermore, it's even more powerful, because the hash codes inside the component names prevent different variants of NixOS to use the incorrect versions of a component. For example, you can safely share the same Nix store across 32-bit and 64-bit machines, because this parameter is reflected in the hash.
  
  In fact, we already use sharing extensively in our build farm, to generate virtual machines to perform testcases. Apart from a Linux kernel image and an initial RAM disk, the components of the entire virtual network are shared through the host system's Nix store, which is mounted as a network file system.
• How to find dependencies? Because we store all software packages in the Nix store, it is harder to address components in e.g. scripts and configuration files, because all these components are stored in separate directories, and aren't necessarily in the user's PATH.
  
  I have to admit that this is inconvenient in some cases, however in NixOS we usually don't manually edit configuration files and write scripts.
  NixOS is a distribution that is unconventional in this sense, because its goal is to make the entire deployment process model-driven. Instead of manually installing packages and adapting configuration files in e.g. /etc. In NixOS, we generate all the static parts of a system from Nix expressions, including all configuration files and scripts. The Nix expression language provides the paths to the components.

Improvements

• The directory names and purposes within the hierarchies can be more consistently used within Nix packages. Essentially, you could say that every component in the Nix store is a separate hierarchy.
  
  For example, I think it's a good thing to use /nix/store/<package>/bin for storing binaries, and /nix/store/<package>/lib for libraries. Sometimes installation scripts of packages do not completely adhere to the FHS and need some fixing. For example, some packages install libraries in the libexec directory, which is not defined by the FHS.
  
  We could make an additional check in the generic builder of Nixpkgs, checking the structure of a package and to report inconsistencies.
  
• For the variable parts of the system, we can adhere better to the FHS. For example, the current version of the FHS defines the /srv directory used for serving files to end-users, through a service (e.g. a web server). In NixOS, this directory is not used.
• Because the FHS standard has some gaps, we could also define some additional clarification, like other Linux distributions do.

Conclusion

Related work

• GoboLinux is also a Linux distribution not obeying the FHS, with a custom filesystem organization. They also have the vision that the filesystem can act as a database for organizing packages.
• The Mancoosi project is also a research project investigating package management related problems, like we do. Their research differs from us, because they don't break away from the traditional filesystem organization and management of Linux systems. For example, a lot of their research deals with maintainer scripts of packages, which goal is to "glue" files from a package to files on the system, e.g. by imperatively modifying configuration files.
  
  By keeping the traditional management intact, this introduces a whole bunch of challenges to make deployment efficient, reliable and reproducible. For example, they have investigated modeling techniques for maintainer scripts and system configurations, to simulate whether an upgrade succeed and to determine inverse operations to perform rollbacks.
  
  In our research, we achieve more reliable deployment by breaking from the traditional model.
  

Filesystem organization

Package organization

• Because every version is stored in its own directory, we can safely store multiple versions next to each other, without having to worry that a file of another version gets overwritten.
• By using a symlink, pointing to the current version, we can atomically switch between versions by flipping the symlink (also used by Nix to switch Nix profiles), which ensures that there is no situation in which a package contains both files of the old version and new version.

• It does not reflect which optional features are enabled in a package. For example, many libraries and programs have optional dependencies on other libraries. Some programs may require a particular variant of a library with a specific option enabled.
• It does not take the build-time dependencies, such as build tools or library versions into account, such as the version of GCC or glibc. Older versions may have ABI incompatibilities with newer versions.
  
• It does not take the architecture of the binaries into account. Using this naming scheme, it is harder to store 32-bit and 64-bit binaries safely next to each other.
  

System composition

• Kernel/, contains everything related to the Linux kernel, such as Boot/ storing kernel images, Modules/ storing kernel modules and Objects/ storing device files.
  
• Links/, is a big symlink tree composing the contents of all packages that are currently in use. This symlink tree makes it possible to refer to executables in Executables/, libraries in Libraries/ and other files from a single location. The Environment/ directory is used to set essential environment variables for the installed programs.
• Settings/ contains configuration files, which in other distributions are commonly found in /etc. Almost all files are symlinks to the Settings/ folder included in each program directory.
• Variable/ contains all non-static (variable) data, such as cache and log files. In conventional distributions these files are stored in /var.

Recipes

compile_version=1.8.2
url="$httpSourceforge/lesstif/lesstif-0.95.2.tar.bz2"
file_size=2481073
file_md5=754187dbac09fcf5d18296437e72a32f
recipe_type=configure
make_variables=(
   "ACLOCALDIR=$target/Shared/aclocal"
   "aclocaldir=$target/Shared/aclocal"
)

Building packages

Conclusion

• There are many deployment related aspects (and perhaps other aspects as well) that can be solved and improved by just using the filesystem. I often see that people write additional utilities, abstraction layers and databases to perform similar things, while you can also use symlink trees and bind mounts to provide abstractions and compositions. Additionally, this also shows that the filesystem, which is an essential key component for UNIX-like systems, is still important and very powerful.
• Storing packages in separate directories is a simple way to manage their contents, store multiple versions safely next to each other and to make upgrades more reliable.

1. Deal with the deployment complexity from the beginning in your development process. Although this lesson isn't technical and may sound very obvious to you, I can ensure you that if you have a codebase which grows rapidly, without properly thinking about deployment, it becomes a big burden to implement the techniques described in this blog post later in the development process. In some cases, you may even have to re-engineer specific parts of your system, which can be very expensive to do, especially if your codebase consists of millions of lines of code.
   
   
2. Make your deployment process scriptable. This lesson is also an obvious one, especially if you have adopted an agile software development process, in which you want to create value as soon as possible (without a deployed system there is no value).
   
   Although it's obvious, I have seen many cases where developers still build and package components of a software system by manually performing build and package tasks in the IDE and by doing all the installation steps by hand (with some deployment documentation). Such deployment processes typically take a lot of time and are subject to errors (because they are performed manually and humans make mistakes).
   
   Quite often these developers, who perform these manual/ad-hoc deployment processes, tell me that it's much too complicated to learn how a build system works (such as GNU Make or Apache Ant) and to maintain these build specifications.
   
   The only thing I can recommend to them is that it really is worth the effort, because it significantly reduces the time to make a new release of the product and also allows you test your software sooner and more frequently.
   
   
3. Decompose your system and their build processes. Although I have seen many software projects using some kind of automation, I have also encountered a lot of build tools which treat the the complete system as a single monolithic blob, which must be built and deployed as a whole.
   
   In my opinion it's better to decompose your system in parts which can be built separately, for the following reasons:
   
   
   • It may significantly reduce build times, because only components that have been changed have to be rebuilt. There is no need to reexamine the complete codebase for each change.
   • It increases flexibility, because it becomes easier to replace specific parts of a system with other variants.
   • It allows a better means of sharing components across systems and better integration in different environments. As I have stressed out in an earlier blog post, software nowadays is rarely self-contained and run in many types of environments.
   • It allows you to perform builds faster, because they can be performed in parallel.
     
   
   In order to decompose a system properly, you have to think early about a good architecture of your system and keep reflecting about it. As a general rule, the build process of a sub component should only depend on the source code, build script and the build results of the dependencies. It should not be necessary to have cross references between source files among components.
   
   Typically, a bad architecture for a system also implies a very complex and inefficient build process.
   
   I have encountered some extreme cases, in which the source code system is one big directory of files, containing hundreds of dependencies including third-party libraries in binary form, infrastructure components (such as the database and web servers) containing many tweaks and custom configuration files.
   
   Although such code bases can be automatically deployed, it offers almost no flexibility, because it is a big burden to replace a specific part (such as a third party library) and still ensure that the system works as expected. Also it requires the system to be deployed in a clean environment because it offers no integration. Typically these kind of systems aren't updated that frequently either.
   
   
4. Make build-time and run-time properties of components configurable. Another obvious lession, but I quite frequently encounter build processes which make implicit assumptions about the locations where specific dependencies can be found.
   
   For example, in some Linux packages, I have seen a number of Makefiles which have hardcoded references to the /usr/bin directory to find specific build tools, which I have to replace myself. Although most Linux systems have these build-time dependencies stored in this folder, there are some exceptions such as NixOS and GoboLinux.
   
   In my opinion it's better to make all locations and settings configurable, either by allowing it to be specified as a build parameter, environment variable (e.g. PATH) or stored in a configuration file, which can be easily edited.
   
   
5. Isolate your build artifacts. I have noticed that some development environments store the output of a build in separate folders (for example Eclipse Java projects) whereas others store all the build artifacts of all projects in a single folder (such as Visual Studio C# projects). Similarly, with end user installations in production environments, libraries are also stored in global directories, such as /usr/lib on Linux environments, or C:\Windows\System32 on Windows.
   
   Based on my experience with Nix, I think it's better to store the build output of every component in isolation, by storing them in separate folders. Although this makes it harder to find components, it also offers some huge benefits, such as the ability to store multiple versions next to each other. Also upgrades can be made more reliable, because you can install another version next to an existing one. Furthermore, builds can also be made more pure and reproducible, because it's required to provide the locations of these components explicitly. In global folders such as /usr/lib a dependency may still be implicitly found even if it's not specified, which cannot happen if all the dependencies are stored in separate locations. More information about isolation can be found in my blog post about NixOS and the FHS.
   
   
6. Think about a component naming convention. Apart from storing components in separate directories which offers some benefits, it's also important to think about how to name them. Typically, the most common naming scheme that's being used consists of a name and version number. This naming scheme can be insufficient in some cases, for example:
   
   
   • It does not take the architecture of the component into account. In a name-version naming scheme a 32-bit variant and a 64-bit variant are considered the same, while they may not be compatible with another component.
   • It does not take the optional features into account. For example, another component may require certain optional features enabled in order to work. It may be possible that a program incorrectly uses a component which does not support a required optional feature.
   
   I recommend you to also take relevant build-time properties into account when naming a component. In Nix you get such a naming convention for free, although the naming convention is also very strict. Nix hashes all the build-time dependencies, such as the source code, libraries, compilers and build scripts and makes this hash code part of the component name, for example: /nix/store/r8vvq9kq18pz08v24918my6r9vs7s0n3-firefox-8.0.1. Because of this naming scheme, different variants of a component, which are (for example) compiled with another version of a compiler of for a different system architecture do not share the same name.
   
   In some cases, it may not be necessary to adopt such a strict naming scheme, because the underlying component technology used is robust enough to offer enough flexibility. In one of my case studies covering Visual Studio projects, we only distinguish between variants of component by taking the name, version, interface and architectures into account. because they have no optional features and we assume the .NET framework is always compatible. We organize the filesystem in such a way that 32-bit and 64-bit variants are stored in separate directories. Although a naming scheme like this is less powerful than Nix's, it already offers huge benefits compared to traditional deployment approaches.
   
   Another example of a more strictly named component store is the .NET Global Assembly Cache (GAC), which makes a distinction between components based on the name, version, culture and a cryptographic key. However, the GAC is only suitable for .NET library assemblies and not for other types of components and cannot take other build properties into account.
   
   
7. Think about component composition. A system must be able to find its dependencies at build-time and run-time. If the components are stored in a separate folders, you need to take extra effort in order to compose them. Some methods of addressing dependencies are:
   
   
   • Modify the environment. Most platforms use environment variables to address components, such as PATH for looking up executables, CLASSPATH for looking up Java class files, PERL5LIB for addressing Perl libraries. In order to perform a build these environment variables must be adapted to contain all the dependencies. In order to run the executable, the executable needs to be wrapped in a process, which sets these environment variables. Essentially, this method binds dependencies statically to another component.
   • Composing symlink trees. You can also provide a means of looking up components from a single location by composing the contents of the required dependencies in a symlink tree and by referring to its contents. In conjunction with a chroot environment, and a bind mount of the component directory, you can make builds pure. Because of this dynamic binding approach, you can replace dependencies, without a rebuild/reconfiguration, but you cannot easily run one executable that uses a particular variant of a library, while another runs another variant. This method of addressing components is used by GoboLinux.
   • Copy everything into a single folder. This is a solution which always work, however it is also a very inefficient as you cannot share components.
   
   Each composition approach has its own pros and cons. The Nix package manager supports all three approaches, however the first approach is mostly used for builds as its the most reliable one, because static binding always ensures that the dependencies are present and correct and that you can safely run multiple variants of compositions simultaneously. The second approach is used by Nix for creating Nix profiles, which end users can use to conveniently access their installed programs.
   
   
8. Granularity of dependencies. This lesson is very important while deploying service-oriented systems in a distributed environment. If service components have dependencies on components with a large level of granularity, upgrades may become very expensive because some components are unnecessarily reconfigured/rebuilt.
   
   I have encountered a case in which a single configuration file was used to specify the locations of all service components of a system. When the location of a particular service component changes, every service component had to be reconfigured (even services that did not need access to that particular service component) which is very expensive.
   
   It's better to design these configuration files in such a way that they only contain properties that a service components need to know.
   
   

Conclusion

Motivation

• Sometimes I have to solve engineering problems, instead of doing science (whatever that means). In my opinion, this is also interesting and useful to report about. For example, while most of my research is about Nix, NixOS, Disnix, sometimes I have to apply my work in specific scenarios/environments which is challenging enough to get it to work. A notable example is the deployment .NET software. Without proper application of the stuff I'm investigating, the usefulness of my research isn't so great either.
• Sometimes I have to express ideas and opinions. For example, what others say about the research we're doing or about general software engineering issues.
• Sometimes I need to do knowledge transfer about issues I have to keep explaining over and over again, like the blog post about software deployment in general.
• Sometimes I have some fun projects also well.

Writing for a blog

• Because I'm doing research in software deployment and because this is a neglected research topic, I have to adapt my paper to fit in the scope of a conference I have to submit to, because they are typically about something else. This is not always easy.
• I have to identify the audience of the particular conference and learn about the subjects and concepts they talk about.
• I have to read a lot of related work papers to determine the exact scientific contribution and to describe what the differences are.
• I have to integrate my idea into the concepts of the given conference.
• I have to explain the same concepts over and over again, because they are not generally understood. What is software deployment? Why is software deployment difficult and important? What is Nix? Why is Nix conceptually different compared to conventional package managers? etc. etc.
• In each paragraph, I have to convince the reader over and over again.
• It should fit within the page limit
• I have to keep myself aware of the fact that my paper must be scientific paper and not an engineering paper.

Blog posts

1. Software deployment complexity. I'm happy that my general blog post about software deployment complexity has raised so much attention, because within the software engineering research community it's typically an ignored subject. In just one week, it surpassed the number of hits of all my other blog posts.
2. Second computer. For some reason my blog post about my good ol' Commodore Amiga is very popular. For a long time, this was actually my most popular blog post and it's not even research related! It seems that 19 years after Commodore's demise, the Amiga is far from dead.
3. First blog post. This is the first blog post in which I briefly introduce myself. Apparently, people want to know who I am :-)
4. NixOS: A purely functional Linux distribution. This is a general blog post explaining the idea and features of NixOS, the Linux distribution built on top of the Nix package manager, which we use as a foundation for our current research project.
5. First computer. Apart from my good ol' Commodore Amiga, my first computer the: Commodore 128 also still lives on!
6. The Nix package manager. This blog post covers the Nix package manager, on which NixOS is built. Apparently people are interested in the main Nix concepts as well.
7. Concepts of programming languages. This blog post is about the 'Concepts of programming languages' course taught at the TU Delft, for which I was a teaching assistent. This course covers various programming languages, which are conceptually different from each other. I actually have no idea why this blog post is so popular.
8. Using NixOS for declarative deployment and testing. This blog post covers two very nice applications of NixOS in which you can automatically deploy a distributed network of NixOS machines and use the same specification to generate a network of virtual machines for testing. I have presented this topic at FOSDEM, Europe's biggest free and open source software event, and my presentation was well received there :-).
9. On Nix, NixOS and the Filesystem Hierarchy Standard (FHS). This blog post responds to one of the common criticisms I have received about Nix and NixOS from other distributions. In this blog post I explain why we are different and that it's for a very good reason.
10. Self-adaptive deployment with Disnix. This blog post is about one of my biggest research contributions, which I have presented at the SEAMS symposium (co-located with ICSE) in Hawaii. In this blog post I have built a framework on top of Disnix and the underlying purely functional deployment model of Nix to make systems self-adaptable by redeployment in case of an event.

Conclusion

Geek Gadgets

• ixemul.library. A BSD 4.3 kernel running under AmigaOS, which internally translates a number system API calls to AmigaOS kernel calls. The API is specifically modelled after NetBSD.
• libnix. An ANSI C API library directly using the AmigaOS libraries, instead of the Unix compatibility layer. This library is better suited for developing native AmigaOS applications not requiring any UNIX functionality.
• GNU Compiler Collection. The free/open-source/ubiquitous compiler suite used on many platforms, such as Linux, Cygwin, FreeBSD, OpenBSD, NetBSD and Mac OS X.
• Korn Shell. Geek Gadgets provides the pdksh shell as a Bourne-compatible shell instead of bash, which is commonly used on other GNU systems. Probably because it's smaller and faster.

Using Geek Gadgets

Deploying Geek Gadgets

• The latest stable version is much too old, dating from 1998. The Geek Gadgets documentation is quite outdated and does not accurately describe what packages you need from the latest snapshot.
• In order to use any Geek Gadgets tools, I need to manually set up the Geek Gadgets environment each time, by executing the GG:Sys/S/GG-Startup script and by raising the stack size, because otherwise I can get mysterious lock-ups/crashes. Of course, I could also add these instructions to S:User-Startup, but I don't want the Geek Gadgets to spoil up my regular system when I use native AmigaOS applications.
• Geek Gadgets uses a traditional UNIX filesystem organisation, with all its disadvantages. For example, packages are deployed in an impure manner because their contents is scattered across the filesystem. I don't like it very much to let development packages pollute my filesystem.

Building AmigaOS applications with Nix

Creating a disk image

• UAE must be configured to use a host filesystem directory as a hard drive partition. In the UAE GUI this can be enabled by clicking on 'New filesystem...' button in the 'Harddisks' tab.
• The Amiga Workbench 3.1 diskettes are required to perform a hard drive installation of the Amiga Workbench. The install disk contains a wizard which takes care of these steps. I performed an 'Immediate user installation' in which I only installed the English language locale, no printers and an American keymap.
• I have also installed the LhA archiver, because this is the most widely used archiver on the Amiga. I have obtained this version from Aminet: http://aminet.net/package/util/arc/lha. This file is a self extracting archive, which can installed by running the following instructions from the command-line interface:
  

  T:
  Protect lha.run +e
  lha.run
  Copy lha_68k C:lha
  

• I used this simple README file from Aminet as a reference describing the base packages I need: http://de4.aminet.net/dev/gg/0README-GG.txt
• I did not download the Geek Gadgets packages from Aminet, but instead I downloaded the latest snapshot, which I found here: ftp://ftp.back2roots.org/pub/geekgadgets/amiga/m68k/snapshots/990529/bin.
• Instead of gcc, I have downloaded egcs. Additionally, I have downloaded a2ixlibrary and gawk, which are frequently required to build packages. libg++ is not included in the latest snapshot and apparently also not required.
• I've extracted all the tarballs on the host system into the GG subfolder of the Amiga drive, because this is much faster. I had to fix to bin/sh symlink to the ksh manually, because the symlink in the tarball is incorrect.
• A GG: assignment is required so that Geek Gadgets knows where to find its files. This assignment can be made by adding the following line to the S:User-Startup file:
  

  Assign >NIL: GG: DH0:GG
  

• If you want to create native AmigaOS applications, you need to copy the Amiga system headers into GG:os-include. I have found a collection of Geek Gadgets compatible os-include headers here: http://ftp.back2roots.org/back2roots/cds/fred_fish/geekgadgets_vol1_9610/ade-bin
• In order to automatically perform builds from Nix expressions, we need to start the build process right after the system is booted. To make this possible I commented out the last two lines from the S:Startup-Sequence, to disable the Workbench and to boot into command-line mode:
  

  ; C:LoadWB
  ; EndCLI >NIL:
  

  
  I have added the following lines to the S:User-Startup script to initialize the Geek Gadgets environment and to automatically start the build process:
  

  Execute GG:Sys/S/GG-Startup
  Stack 200000
  
  DH0:T
  sh build.sh
  

Writing a Nix function

{name, src, buildCommand, buildInputs ? []}:

with import <nixpkgs> {};

let
  amigaBase = ./amigabase;
  kickRom = ./kickrom/kick.rom;
in

stdenv.mkDerivation {
  inherit name;

  buildInputs = [ uae procps ];

  buildCommand = ''

    mkdir hd
    cd hd

    for i in ${amigaBase}/{C,Classes,Expansion,Fonts,L,Libs,Locale} \
      ${amigaBase}/{Prefs,Rexxc,S,Storage,System,Tools,Utilities}
    do
        ln -s "$i"

        # Symlink icons
        if [ -f "$i.info" ]
        then
            ln -sf "$i.info"
        fi
    done

    cp -av ${amigaBase}/GG .
    chmod 755 GG

    cd GG

    for i in `find . -type d` `find . -type f`
    do
        chmod 755 $i
    done

    for i in ${toString buildInputs}
    do
        cp -av $i/* .
    done

    cd ..

    mkdir T
    cd T

    stripHash ${src}
    cp -av ${src} $strippedName

    if [ -d "$strippedName" ]
    then
        chmod -R 755 "$strippedName"
    fi

    echo "src=$strippedName" > buildinstructions.sh

    cat >> buildinstructions.sh << "EOF"
    ${buildCommand}
    EOF

    cat > build.sh << "EOF"
    ( sh -e buildinstructions.sh

      if [ $? = 0 ]
      then
          echo "success" > /OUT/done
      else
          echo "failure" > /OUT/done
      fi
    ) 2>&1 | tee /OUT/log.txt
    EOF

    cd ../..

    # Create UAE config file
    export HOME=$(pwd)

    cat > .uaerc <<EOF
    config_description=UAE default configuration
    use_gui=no
    kickstart_rom_file=${kickRom}
    sound_output=none
    fastmem_size=8
    chipmem_size=4
    cpu_speed=max
    cpu_type=68ec020
    gfx_linemode_windowed=double
    filesystem2=rw,:HD0:$(pwd)/hd,1
    filesystem=rw,HD0:$(pwd)/hd
    filesystem2=rw,:OUT:$out,0
    filesystem=rw,OUT:$out
    EOF

• We disable the GUI, because this prevents us running UAE non-interactively.
• We need to specify where the Kickstart ROM image can be found. This property is defined earlier in the kickRom variable.
• We disable sound support, because we don't need it and it may cause lock ups in some cases.
• The Geek Gadgets tools require quite an amount of RAM to run properly, so we've configured the UAE instance to use 2 MiB of Chip RAM and 8 MiB of Fast RAM.
• Normally the UAE emulator tries to accurately match the speed of a 7 MHz Motorla 68000 chip, which makes building software very slow compared to nowadays' standards. Fortunately, UAE can also use the host CPU as efficiently as possible, which we do in our build function to make builds significantly faster.
• We also configured two hard drive instances. The HD0: drive corresponds to the disk image we have created earlier containing the Workbench and Geek Gadgets. The OUT: drive corresponds to the Nix store output path, in which we have to store our build result.

    mkdir -p $out

    export DISPLAY=:0

    # Start UAE
    uae &

    while [ ! -f $out/done ]
    do
        sleep 1
    done

    # Kill the emulator
    pkill uae
    sleep 1

    # Check the build status
    [ "$(cat $out/done)" = "success" ]

    cd $out
    rm done

    rm -f `find . -name _UAEFSDB.___`
  '';
}

Using the Nix function

import ./amigaosbuild.nix {
  name = "hello";
  src = fetchurl {
    url = mirror://gnu/hello/hello-2.1.1.tar.gz;
    sha256 = "1md7jsfd8pa45z73bz1kszpp01yw6x5ljkjk2hx7wl800any6465";
  };
  buildCommand = ''
    tar xfvz $src
    cd hello-2.1.1
    ./configure --prefix=/OUT
    make || true
    make install
  '';
}

$ nix-build hello.nix

#include <exec/types.h>
#include <intuition/intuition.h>
#include <intuition/intuitionbase.h>
#include <intuition/screens.h>

#include <clib/exec_protos.h>
#include <clib/dos_protos.h>
#include <clib/intuition_protos.h>

#include <stdio.h>

#define INTUITION_VERSION 37

struct Library *IntuitionBase = NULL;

struct TagItem win_tags[] = {
    {WA_Left, 20},
    {WA_Top, 20},
    {WA_Width, 300},
    {WA_Height, 120},
    {WA_CloseGadget, TRUE},
    {WA_Title, "Hello world!"},
    {WA_IDCMP, IDCMP_CLOSEWINDOW},
    {TAG_DONE, NULL}
};

void handle_window_events(struct Window *window)
{
    WaitPort(window->UserPort);
}

int main(int argc, char *argv[])
{
    IntuitionBase = OpenLibrary("intuition.library",
      INTUITION_VERSION);

    if(!IntuitionBase)
    {
        fprintf(stderr, "Error opening intuition library!\n");
        return 1;
    }
    else
    {
        struct Window *window = OpenWindowTagList(NULL, win_tags);

        if(window == NULL)
            fprintf(stderr, "Window failed to open!\n");
        else
        {
            handle_window_events(window);
            CloseWindow(window);
        }

        CloseLibrary(IntuitionBase);
        return 0;
    }
}

Conclusion

References

• In order to use this build function you need an Amiga Kickstart ROM image and the Amiga Workbench diskettes. If you don't have these, the legal way to obtain these is by ordering one of the Cloanto's Amiga Forever discs. Alternatively, you could try AROS, a free/open-source operating system reimplementing the AmigaOS APIs, although I have no experience with it so far.
• Performing builds in virtual machines inside a Nix expression is nothing new. We have similar approaches in our Hydra build farm, in which we generate OpenSUSE, Ubuntu and Fedora machines to build RPM and Debian packages from Nix expressions. NixOS virtualization provides additional advantages, which I have described here.

Availability

This is a remarkable piece of engineering work.

Software engineering conferences

Difference between engineering and science

Research in engineering is directed towards the efficient accomplishment of specific tasks and towards the development of tools that will enable classes of tasks to be accomplished more efficiently.

The problem-solving paradigm of the practicing engineer generally involves a sequence of systematic selection or design decisions which progressively narrow down alternative options for accomplishing the task until a unique realization of the task is determined.

The research engineer may use the paradigms of mathematics and physics in the development of tools for the practicing engineer, but is much more concerned with the practical implications of his research than the empirical scientist or mathematician.

Academic contributions

• You shouldn't merely solve a problem for which you have to engineer something. It should be related to software engineering processes.
• Your contribution should fill a gap within in a software engineering process. For example, it should offer an improvement or a new insight everybody should know about. Furthermore, this contribution must be novel.
• The contribution should be significant enough.
• You need strong motivation why this contribution is needed, which is not always obvious. You also need to look at what has been done before.
• You need to prove what you claim is valid (or at least strong enough) and you need to describe how it compares to (possibly) related work.
• And of course, you need to explain what you did

Reflection

• The motivation is unclear or misunderstood.
• Perhaps the paper may contain too many implementation details.
• There is something wrong with the evaluation method or the evaluation is not strong enough.
• Reviewers have different expectations about submissions. Maybe this is also due to the fact that software deployment is a very uncommon research subject.
• A solution is/looks too simple

Engineering

• Typically, many software engineering problems arise in practice. Being in the field gives you a lot of experience and a better understanding of particular software engineering problems.
• Many software engineering contributions are implemented as tools. Tools need to be constructed from various components by using various means. So in order to fill a gap in software engineering processes, you also have to do some software engineering.
• Tools need to be applied in real world scenarios. In such cases, you may have to study existing systems, extend/integrate your tooling, or even implement work-arounds because you're tooling assumes ideal behaviour.
• There may be lessons that you have learned in your research that every software engineer should know about.

So what's the big deal anyway?

Conclusion

Purely functional deployment

• The result of a derivation exclusively depends on the specified inputs of a derivation. If a dependency is not specified, it cannot be found and thus a build fails.
• In a Nix build environment, many side effects are removed. For example, most environment variables are cleared or set to non-existing values, such as HOME=/homeless-shelter. Also certain commands cannot be used because they have side effects, such as date or hostname.
• Various common build tools (e.g. GCC and binutils) have been patched to ignore impure directories, such as /usr/lib and /usr/include which may accidentally allow a build to succeed, with improper dependency specifications.
• Nix stores every component in a so-called Nix store using a special filename, such as: /nix/store/a92pq...-firefox-9.0.1 in which the former part: a92pq... is a hash code derived from all build-time dependencies. This filesystem organisation prevents that dependencies can be implicitly found unless they are explicitly specified in a build process, e.g. through configuration flags or environment variables.
• To improve purity, builds can be optionally performed in a chroot() environment.
• Nix components which have been successfully built, are made immutable by removing the write permission bits, analogous to purely functional language objects which are immutable.

Mutable components

• Mutable components are typically hosted inside a container, such as a DBMS or application server. In order deploy a mutable component filesystem operations are not always enough; also the container must be notified.
• The state of mutable components is very often stored in an unisolated directory and in a format close to the underlying implementation of the container. In order to transfer a mutable component from one machine to another, copying state files may (in many cases) not work, because they are incompatible with the target architecture or temporarily inconsistent, because of locks and/or unfinished write operations.

Dysnomia

Extending Disnix

• The simple version. In the transition phase, we capture all state of mutable components, transfer the state and restore the state. During this phase access may be blocked and can be very expensive for large datasets.
• The optimised version. In this version, we transfer the state outside the transition phase so that the system is still accessible. After transferring the state, we enter the transition phase and we transfer an incremental dump of the mutable state components to take changes into account which have been made meanwhile. In this version, the blocking time window is reduced to a minimum reducing the costs of blocking as much as possible.

Discussion

• The filesystem is used as an intermediate layer, which may be very expensive for very large databases. Database replication tools can solve these problems much more efficiently.
• It is not part of Disnix yet. I'm still thinking of integrating it, but it radically changes the behaviour of Disnix. Perhaps it may become an optional feature in the future.
• Also, the costs of managing state may be higher than the benefits that it gives.
• The number of supported mutable components is still limited, and not all types of mutable components can be efficiently snapshotted.

Where comes the name from?

• Nix is derived from the Dutch word for nothing: 'Niks' (because if you don't specify dependencies, Nix cannot find anything) combined with UNIX, hence: Nix. Coincidentally, Nix is also a mythological figure as well as a name for a moon of the dwarf planet Pluto which is part of the Kuiper belt.
• Disnix and NixOS also use the original naming convention. For example, Disnix is DIStributed NIX.
• Hydra, our continuous integration server and Charon, our cloud deployment tool are names based on the other moons of Pluto.
• Apparently, Pluto has a fourth moon, which is recently discovered, but it does not have a name yet. Furthermore, I don't think this hacky tool is also worthy to bear that new name.
• I took another dwarf planet from the Kuiper belt: Eris, which happens to be the largest dwarf planet discovered so far. Apparently it has a moon named: Dysnomia. So that explains where the name comes from. Furthermore, I also think it's better to take a moon of a different dwarf planet, because Dysnomia could also be used without Nix (and related applications).

Conclusion

• Academic research publication libraries: ACM Digital Library, IEEE eXplore, SpringerLink
• Conferences:
  • Academic: ICSE, ASE, ISSRE, SPLASH
  • Industrial:
  • Technology-related: Oracle JavaOne, Microsoft TechEd
  • "Trade shows": CeBIT
  • Free and Open-Source software related: FOSDEM, EclipseCon, LinuxCon
• Internet services:
  • Question and answers websites: Stack overflow
  • Technology related websites: Slashdot, Phoronix, Ars Technica
  • Social news websites: Reddit
  • Source code sharing and collaboration: Sourceforge, Github
  • (Micro)blogs: Twitter, Blogger
  • Messaging: IRC, Mailing Lists

Academic fraction

Research in engineering is directed towards the efficient accomplishment of specific tasks and towards the development of tools that will enable classes of tasks to be accomplished more efficiently.

Industry fraction

Community project fraction

"The System"

Implications of "The System" on the academic fraction

Implications of "The System" on the industry fraction

• Developing software is not necessarily a process that merely costs some amount of money. Software development is an investment and also gives benefits. The benefits are often overlooked by a lot of managers. Reducing costs, e.g. by hiring cheaper employees with less knowledge, may typically result in fewer benefits.
• In order to reach your primary goals, you also have to reach secondary goals. For example, banks are not really software companies, but they have to become software companies because their organization depend completely on software.
  
  I think the same is true for software companies doing software engineering. People developing financial software, may not want to think about build management complexity issues, but they have to, because otherwise it is impossible to properly engineer systems for end-users.

Implications of "The System" on community projects

Discussion

• The Dutch government as well as the industry in the Netherlands, spent a relatively small amount of money in research, while they want our country to be a 'knowledge economy'. It's actually the least of all countries in the European Union. The government is planning to reduce these investments even more. They expect that companies invest more in research, but in my experience, apart from a few notable exceptions, most of companies are reluctant or have no clue what is going on in the research world. We have very good researchers in the Netherlands, but their work isn't applied that well in industry. In my opinion, that's a shame and a waste. (I have used this blog post as a reference (In Dutch) )
• As mentioned in an earlier blog post, industry participation at academic conferences is low. Industry people often have different interests than academic people. Nowadays, they rather attend technology related conferences. For industry people, application and benefits of tools and technology is important. More important than a mathematical proof or evaluation showing numbers, which they don't understand.
  
  I have encountered several boring presentations at academic conferences, showing lots of "greek symbols" and all kinds of complicated things I didn't understand. I'm pretty sure that if the presenter would have attendees from industry, they have no clue what they are talking about and they quickly lose interest.
• Sometimes people reinvent the wheel, but in a crappier way, which they have to maintain themselves. In my research, for example, I have seen many custom build systems that have a significant maintenance burden. People usually stick to these suboptimal solutions for a long time, while there are many solutions available that are more convenient, more powerful and easier to maintain.

Recommendations

• I think "The System" of academic researchers shouldn't be merely about publications. Furthermore, they shouldn't be merely about these fixed collection of conferences/journals each having their own "borders". Publishing stuff is not a bad thing in my opinion, as concepts must be properly explained, evaluated and peer-reviewed. But concepts are useless without any deliverables that can be applied.
  
  "Playing darts" by only aiming at "sections on a dart board" is bad for research. For example, Edsger Dijkstra, one of the most famous computer scientists published a lot, but most of his publications were his "EWDs"; manuscripts, which he wrote about any subject he want, whenever he wanted, without aiming at anything or keeping some kind of "System" satisfied.
• As an academic software engineering researcher, you are typically investigating stuff for some kind of "audience" (e.g. developers or testers) and it may possibly be related to certain kind of technology (e.g. Java, Eclipse, Linux etc.). Therefore, I think it's also important to directly work with people from these fractions, address them regularly (e.g. visit them and participate in their technology-related conferences) and see whether you can make your work interesting for them.
• It's also a good thing to make your tools available by some means. Perhaps joining a community project or start your own community project can be good thing.
• Companies must know that developing software is an investment and that secondary goals have to be reached in order to achieve primary goals. Some secondary goals cannot be solved efficiently by third parties, as it's too specific for the domain of the company.
• Companies should not be afraid in participating in other fractions' community means. In fact, they should be more eager in finding out what other fractions have to offer them.

Why am I writing this?

• Techniques and lessons for improvement of deployment processes
• An evaluation and comparison of GoboLinux
• On Nix, NixOS and the Filesystem Hierarchy Standard (FHS)
• Deploying .NET services with Disnix
• Deploying .NET applications with the Nix package manager

Conclusion

Some clarification

Background

Motivation

• How are source files combined into a final executable (e.g. static linking, dynamic linking)?
• What licenses govern the (re)use of source files that were used to build the binary?
• How can we derive the license of the resulting binary from the build graph containing the source files?

Approach

Tracing system calls

• File related system calls, e.g. open(), execve()
• Process related system calls, e.g. fork(), clone(), vfork()

• We have to translate all relative paths to absolute paths
• In Linux, pids wrap around if they exceed 32767, so we have to use a different attribute to distinguish between processes
• Cycles appear in the graph, if files are read/written multiple times, which we have to remove
• The are coarse grained processes, such as the install process, which install multiple files in one run. In the resulting graph it looks like the resulting artifact is dependent on all other artifiacts installed by the same process, which is not true. We have to identify these processes and rewrite them.
• We don't want to know anything about the dependencies of the build tools themselves, because these are not considered derived work.

Build trace graphs

Discussion

References

What is the IFF file format?

IFF concepts

• The FORM chunk, contains an arbitrary collection of data chunks or other group chunks, as shown in the picture above. Our example, defines a FORM which has the type ILBM. In principle, every application file format is essentially a form in which the form type refers to the application file format identifier. In the body of the FORM several data chunks can be found:
  • The BMHD defines the bitmap header containing various general settings, such as the width, height and the amount of colors used.
  • The CMAP defines the red, green and blue color channel values of each color in the palette.
  • The BODY chunk contains pixel data.
• The CAT chunk may only contain a collection of group chunks, that is only FORM, CAT or LIST chunks.
• The LIST chunk is an extended CAT chunk that also contains a number of PROP chunks. PROP chunks are group chunks which may only reside in a list and contain a collection of data chunks. These data chunks are shared properties of all group chunks inside the LIST. For example, a LIST containing ILBM FORM chunks, may use a PROP chunk containing a CMAP chunk, which purpose is to share the same palette over a number of bitmap images.

• If a chunk size is odd, then the chunk data must be padded with an extra 0 byte, so that the next chunk is always stored on an even address in memory (as shown in the example form). This requirement was introduced, because the 68000 processor (which the Amiga uses) processes integers much faster on even addresses in memory. In our example form shown earlier, the CMAP chunk is padded.
• Also application file format attributes of word and long word sizes, must be word aligned (stored on even addresses in memory).
• All integers must be big-endian, because the Amiga was a big-endian system. This means that on little-endian systems, such as PCs, the byte order of integers has to be reversed.

IFF file format support

• Support for most IFF application formats is not present in many common viewers and players. However, some more advanced programs support it. For example, Paint Shop Pro and the SDL_image library have support for viewing ILBM images.
• These applications all have their own implementation of a specific IFF application format. Some implementations are good, others lack certain features. For example, I have seen several viewers not supporting the special Amiga screen modes, such as Extra HalfBrite (EHB) and Hold-and-Modify (HAM) or the color range cycle chunks.
• Applications can open simple IFF files that consist of a single FORM, but do not know how to deal with IFF scrap files, i.e. CATs/LISTs containing multiple FORMs of various types, possibly with shared options.
• Most applications can view IFF application formats, but cannot write them or check for their validity, which may result in crashes if invalid IFF files are opened.
• A number of open file formats have generic parser libraries, e.g. PNG (libpng), JPEG (libjpeg), GIF (giflib), Ogg (libogg), Vorbis (libvorbis) etc. that applications use to open, parse and save files. There is no equivalent for ILBM and other IFF application formats.

IFF libraries experiment

Implementing a SDL ILBM viewer

• libiff. This library implements the properties defined in the IFF specification, such as parsing data chunks and groups chunks. Furthermore, it also supports writing IFF files as well as conformance checking.
• libilbm. This library implements the application chunks as well as the byte run compression algorithm defined in the ILBM specification. Furthermore, it supports several extension chunks and the file format used by the PC version of Deluxe Paint (which has several minor differences compared to Amiga version). Application chunks can be parsed, by defining a table with function pointers to the ILBM functions that handle these and to pass the table to the IFF library functions.
• libamivideo. This library acts as a conversion library for Amiga graphics data. As explained earlier, the Amiga uses bitplanes to organise graphics and has several special screen modes (Extra-Halfbrite (EHB) and Hold-and-Modify (HAM)) to display more colors out of the predefined color registers. In the SDL viewer we use the libamivideo library to convert Amiga graphics data to chunky or RGB graphics and to emulate the special screen modes.
  
  Images saved by the PC version of Deluxe Paint however (which have the PBM form type instead of ILBM), do not use bitplanes but chunky graphics, and thus conversion is not necessary.
  
• SDL_ILBM. This package contains a high level SDL library as well as the ilbmviewer command-line tool, directly generating SDL surfaces from IFF files containing ILBM images as well as performing the required conversions automatically.

$ ilbmviewer picture.IFF

$ iffjoin Champagne Venus.lores Sample.8SVX > join.IFF 
$ ilbmviewer join.IFF

Implementing a SDL 8SVX player

• lib8svx. This library implements the application chunks as well as the fibonacci-delta compression method defined in the 8SVX specification. As with libilbm, it also defines a table with function pointers handling application specific chunks to the IFF parser.
• libresample. This library is used to convert sample rates. 8SVX samples have variable sample rates, while on the PC hardware samples are typically passed to audio buffers with a fixed sample rate. Therefore, we have to convert them.
• SDL_8SVX. This package contains a library as well as the 8svxplayer command-line tool. Sample rate conversion is automatically done by the SDL library.

$ iffjoin Picture.ILBM Sample.8SVX > join.IFF
$ 8svxplayer join.IFF 

Backporting the ILBM viewer to AmigaOS

Conclusion

• The IFF project on Sourceforge, is a C++ library using uSTL, which deviates on some aspects. For example, it stores integers in little-endian format. Furthermore, I haven't seen any application file formats using this library.
• I also found a project named libiff on Google Code. It seems to have no releases and very little documentation. I have no clue about its capabilities and features.

DSM: Workshop on Domain-Specific Modeling (2008)

HotSWUp: Workshop on Hot Topics in Software Upgrades (2008)

OOPSLA: Object-Oriented Programming, Systems, Languages & Applications (2008)

FOSDEM: Free and Open Source Software Developers' European Meeting (2009)

ICSE: International Conference on Software Engineering (2009)

ICSE-Cloud: ICSE Workshop on Software Engineering Challenges in Cloud Computing (2009)

Conclusion

FOSDEM: Free and Open Source Software Developers' European Meeting (2010)

SEAA: EUROMICRO Conference on Software Engineering and Advanced Applications (2010)

WASDeTT: Workshop on Academic Software Development Tools and Techniques (2010)

ASE: International Conference on Automated Software Engineering (2010)

ISSRE: International Symposium on Software Reliability Engineering (2010)

LAC: Landelijk Architectuur Congres (translation: 'National Architecture Congress') (2010)

Conclusion

FOSDEM: Free and Open Source Software Developers' European Meeting (2011)

SEAMS: Software Engineering for Adaptive and Self-Managing Systems (2011)

ICSE: International Conference on Software Engineering (2011)

FOSDEM: Free and Open Source Software Developers' European Meeting (2012)

HotSWUp: Workshop on Hot Topics in Software Upgrades (2012)

ICSE: International Conference on Software Engineering (2012)

Conclusions

On leaving academia

1. My working style does not completely map to the job description of a researcher. Apart from doing the work to get things published, I have also invested a significant amount of time and effort in the tools and their practical usage.
   
   As I have also presented frequently to people outside the academic community, I typically use a lot of practical examples and I often have a number of demos showing good practical usage. These aspects are nice, but are (in principle) not part of my job description. Quite often, outsiders become happy after seeing these aspects and think that my research is about doing that kind of stuff, which is not true.
   
   
2. It's about publishing. As I have explained earlier in a number of blog posts, researchers have to publish and often it's used as the primary means to measure their productivity.
   
   I know some people that have left academia because they hate writing papers and the pressure that it sometimes gives. I don't really hate writing papers, although I've had my frustrating moments sometimes -- actually, if I have a good idea I'm very eager to write them down.
   
   What bothers me is that my research subject -- software deployment -- is not a very popular topic in the software engineering research world. If I would continue my career in research, I have to produce more publications, preferably at highly ranked conferences and journals.
   
   It's difficult to achieve this goal with my current research topic and working style. As a consequence, I'm more or less forced to pick subjects that are better publishable, which basically forces me to check the interests of the program committee members and do stuff that I'm not always interested in. I'm not somebody who likes to work that way.
   
   Some research topics are hyper-specialized, which is not always a good thing. I have noticed that even for my research (that already looks quite practical to most people in the research community) is difficult to sell conceptually to non-academic people, unless I do it the right way. For a number of presentations that I have seen at conferences, I'm pretty sure that people for industry would lose their interest in a matter of minutes and have no clue what the speaker is talking about.
   
   For many of these papers, the "real world" does not care about and as a consequence they tend to become forgotten knowledge. I'm not somebody who likes to do stuff that nobody cares about and do not bring any value to the software engineering community as a whole.
   
   Furthermore, if I would spend my time to publish more, there is less time to do engineering work and to produce/improve tools that can actually do something.
   
   I've picked my current research project, because I like solving problems that I really believe that they are relevant and I want to add value. However, the "deliverables" of my style unfortunately don't always map to publishable units at academic events.
   
   
3. I want to add more practical value. I have always been a person that likes to build stuff. If I have a nice idea in mind, I'm always very eager to try it out and to make piece of software that does something, although it is not always very relevant. For the last 6 months, I had no time for doing stuff like this. Nearly all my time was consumed by finishing publications, writing my thesis and preparing/visiting a conference.
   
   

Looking for a new job

My new job: Software Architect @ Conference Compass

The future of Disnix and other components of the Nix project

The future of research in software deployment

Concluding remarks

The Nix package manager

Purely functional programming languages

.486
.MODEL FLAT
.CODE
PUBLIC _myFunc
_myFunc PROC
  ; Subroutine Prologue
  push ebp     ; Save the old base pointer value.
  mov ebp, esp ; Set the new base pointer value.
  sub esp, 4   ; Make room for one 4-byte local variable.
  push edi     ; Save the values of registers that the function
  push esi     ; will modify. This function uses EDI and ESI.
  ; (no need to save EBX, EBP, or ESP)

  ; Subroutine Body
  mov eax, [ebp+8]   ; Move value of parameter 1 into EAX
  mov esi, [ebp+12]  ; Move value of parameter 2 into ESI
  mov edi, [ebp+16]  ; Move value of parameter 3 into EDI

  mov [ebp-4], edi   ; Move EDI into the local variable
  add [ebp-4], esi   ; Add ESI into the local variable
  add eax, [ebp-4]   ; Add the contents of the local variable
                     ; into EAX (final result)

  ; Subroutine Epilogue 
  pop esi      ; Recover register values
  pop  edi
  mov esp, ebp ; Deallocate local variables
  pop ebp ; Restore the caller's base pointer value
  ret
_myFunc ENDP
END

int sum(int a, int b)
{
    return a + b;
}

int main()
{
    return sum(1, 2);
}

x = y => f(x) = f(y)

int v = 0;

int add(int a)
{
    v = v + a;
    return v;
}

int main()
{
    int p = add(1); /* 1 */
    int q = add(1); /* 2 */
}

Purely functional package management

{stdenv, fetchurl}:

stdenv.mkDerivation {
  name = "hello-2.6";

  src = fetchurl {
    url = ftp://ftp.gnu.org/gnu/hello/hello-2.6.tar.gz;
    sha256 = "1h6fjkkwr7kxv0rl5l61ya0b49imzfaspy7jk9jas1fil31sjykl";
  };

  meta = {
    homepage = http://www.gnu.org/software/hello/manual/;
    license = "GPLv3+";
  };
}

rec {
  stdenv = ...;

  fetchurl = import ../build-support/fetchurl {
    inherit stdenv curl;
  };

  hello = import ../applications/misc/hello {
    inherit stdenv fetchurl;
  }

  ...
}

• To ensure that packages cannot be overridden or removed by build functions of other packages, we store the result of each build in a separate location on the filesystem, in directories that are made immutable (by removing write permission bits), so that they cannot be changed. To create, such a unique location, we store component in a store called the Nix store and we use hashing (derived from all inputs to the function, such as their dependencies, sources and build scripts) to generate a unique directory name, e.g. /nix/store/xq2bfcqdsbrmfr8h5ibv7n1qb8xs5s79-openssl-1.0.0c. If we, for example, change any of its build parameters, such as the used compiler, the hash will differ and thus its safe to allow multiple variants to coexist as they never share the same name.
• By using the Nix store concept, we get another important property "for free" -- because every package variant is stored in a unique location, as opposed to global locations, such as /usr/lib, we have stricter guarantees that dependency specifications are complete, as they cannot be implicitly found. In order to allow a dependency to be found we have to explicitly specify it, for example, by adding it to the PATH environment variable.
• To reduce the chances on side effects even more, we run build scripts in isolated environments with unprivileged user rights and we can optionally use a chroot() environment to limit access to the host filesystem even more.
• We have also patched several common UNIX utilities, such as gcc and ld to ignore global locations, such as /usr/lib.

Nix applications

• NixOS is a GNU/Linux distribution completely built around the Nix package manager. Apart from the fact that every component (including the Linux kernel and configuration files) are managed by Nix, it is also used to deploy entire system configurations from declarative specifications.
• Disnix is an extension to Nix developed by me to automatically deploy service-oriented systems composed of distributable components, in networks of machines.
• Hydra is a continuous build and integration server, built around Nix and using the Nix expression language to define build and test jobs.

Why am I writing this blog post?

Downsides of this alternative explanation

Conclusion

References

• I have taken the example of Leibniz' Principe from Edsger Dijkstra's nice hand written manuscript (EWD 1073) titled: 'How computing science created a new mathematical style', in which he explains why computing science deviates from ordinary mathematics. A presentation from Erik Meijer that I have attended earlier, gave me the inspiration to check this manuscript.
• The paper titled: 'Conception, evolution, and application of functional programming languages' written by Paul Hudak gives a good overview on how functional programming languages arised and evolved, including Haskell -- a well-known purely functional programming language.
• I took the code fragment showing an Intel assembly calling convention, from the following web page: http://www.cs.virginia.edu/~evans/cs216/guides/x86.html.

Packaging the Android SDK

• The Android SDK from the website is not a source package. Google does not seem to provide any proper source releases, except for obtaining the sources from Git yourself. The downloadable distribution is a zip archive with Java JAR files and a hybrid of native i686 and x86_64 executables. Native executables do not work with Nix out of the box, as they try to lookup their run-time dependencies from global locations, which are not present on NixOS. Therefore, they must be patched using PatchELF.
• The Android SDK is not self-contained. It requires developers to install a number of add-ons, such as platform tools, platform SDKs, system images, and support libraries, by running:
  
  $ android update
  
  
• In the normal workflow, these additions are downloaded by the android utility and stored in the same base directory as the SDK, which is an imperative action. This conflicts with the Nix deployment model, as components are made immutable after they have been built. Moreover, these additions must be installed non-interactively.

Android SDK base package

$ ldd ./emulator-x86 
linux-gate.so.1 =>  (0xf76e4000)
libdl.so.2 => /nix/store/7dvylm5crlc0sfafcc0n46mb5ch67q0j-glibc-2.13/lib/libdl.so.2 (0xf76de000)
libpthread.so.0 => /nix/store/7dvylm5crlc0sfafcc0n46mb5ch67q0j-glibc-2.13/lib/libpthread.so.0 (0xf76c4000)
librt.so.1 => /nix/store/7dvylm5crlc0sfafcc0n46mb5ch67q0j-glibc-2.13/lib/librt.so.1 (0xf76af000)
libstdc++.so.6 => not found
libm.so.6 => /nix/store/7dvylm5crlc0sfafcc0n46mb5ch67q0j-glibc-2.13/lib/libm.so.6 (0xf7689000)
libutil.so.1 => /nix/store/7dvylm5crlc0sfafcc0n46mb5ch67q0j-glibc-2.13/lib/libutil.so.1 (0xf7684000)
libgcc_s.so.1 => not found
libc.so.6 => /nix/store/7dvylm5crlc0sfafcc0n46mb5ch67q0j-glibc-2.13/lib/libc.so.6 (0xf7521000)
/nix/store/7dvylm5crlc0sfafcc0n46mb5ch67q0j-glibc-2.13/lib/ld-linux.so.2 (0xf76e5000)

cd tools

for i in dmtracedump emulator emulator-arm emulator-x86 hprof-conv \
  mksdcard sqlite3
do
  patchelf --set-interpreter ${stdenv.gcc.libc}/lib/ld-linux.so.2 $i
  patchelf --set-rpath ${stdenv.gcc.gcc}/lib $i
done

for i in etc1tool zipalign
do
  patchelf --set-interpreter ${stdenv.gcc.libc}/lib/ld-linux.so.2 $i
  patchelf --set-rpath ${stdenv.gcc.gcc}/lib:${zlib}/lib $i
done

for i in ddms draw9patch monkeyrunner monitor lint traceview
do
    shebangfix $i
done

SDL init failure, reason is: No available video device

$ strace -f ./emulator
...
rt_sigaction(SIGINT, {SIG_DFL, [INT], SA_RESTART}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_DFL, [QUIT], SA_RESTART}, {SIG_DFL, [], 0}, 8) = 0
futex(0xfffffffff7705064, FUTEX_WAKE_PRIVATE, 2147483647) = 0
open("./lib/tls/i686/sse2/libX11.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("./lib/tls/i686/libX11.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("./lib/tls/sse2/libX11.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("./lib/tls/libX11.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("./lib/i686/sse2/libX11.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("./lib/i686/libX11.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("./lib/sse2/libX11.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("./lib/libX11.so.6", O_RDONLY)     = -1 ENOENT (No such file or directory)
...
open("/nix/store/cy8rl8h4yp2j3h8987vkklg328q3wmjz-gcc-4.6.3/lib/libXext.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/nix/store/7dvylm5crlc0sfafcc0n46mb5ch67q0j-glibc-2.13/lib/libXext.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
...
open("/nix/store/cy8rl8h4yp2j3h8987vkklg328q3wmjz-gcc-4.6.3/lib/libXrandr.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/nix/store/7dvylm5crlc0sfafcc0n46mb5ch67q0j-glibc-2.13/lib/libXrandr.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, "SDL init failure, reason is: No "..., 55SDL init failure, reason is: No available video device
) = 55
unlink("/home/sander/.android/avd/foo.avd/hardware-qemu.ini.lock") = 0
exit_group(1) 

for i in emulator emulator-arm emulator-x86
do
    wrapProgram `pwd`/$i \
      --prefix LD_LIBRARY_PATH : `pwd`/lib:${libX11}/lib:\
${libxcb}/lib:${libXau}/lib:${libXdmcp}/lib:\
${libXext}/lib
done

Supporting plugins and optional packages

$ android list sdk

Fetching https://dl-ssl.google.com/android/repository/addons_list-2.xml
  Validate XML
  Parse XML
  Fetched Add-ons List successfully
  Refresh Sources
  Fetching URL: https://dl-ssl.google.com/android/repository/repository-7.xml
  Validate XML: https://dl-ssl.google.com/android/repository/repository-7.xml
  Parse XML:    https://dl-ssl.google.com/android/repository/repository-7.xml
  Fetching URL: https://dl-ssl.google.com/android/repository/addon.xml
  Validate XML: https://dl-ssl.google.com/android/repository/addon.xml
  Parse XML:    https://dl-ssl.google.com/android/repository/addon.xml

Platform tools

Platform SDKs and system images

<sdk:sdk-repository ...>
<sdk:platform>
<sdk:version>2.2</sdk:version>
<sdk:api-level>8</sdk:api-level>
<sdk:codename/>
<sdk:revision>03</sdk:revision>
<sdk:min-tools-rev>
<sdk:major>8</sdk:major>
</sdk:min-tools-rev>
<sdk:description>Android SDK Platform 2.2_r3</sdk:description>
<sdk:desc-url>http://developer.android.com/sdk/</sdk:desc-url>
<sdk:archives>
<sdk:archive arch="any" os="any">
<sdk:size>74652366</sdk:size>
<sdk:checksum type="sha1">231262c63eefdff8...</sdk:checksum>
<sdk:url>android-2.2_r03-linux.zip</sdk:url>
</sdk:archive>
</sdk:archives>
<sdk:layoutlib>
<sdk:api>4</sdk:api>
</sdk:layoutlib>
</sdk:platform>

  ...
</sdk:sdk-repository>

let buildPlatform = ...
in
{
  ...  

  platform_8 = buildPlatform {
    name = "android-platform-2.2";
    src = fetchurl {
      url =
        https://dl-ssl.google.com/android/repository/android-2.2_r03-linux.zip;
      sha1 = "231262c63eefdff8fd0386e9ccfefeb27a8f9202";
    };
    meta = {
      description = "Android SDK Platform 2.2_r3";
      url = http://developer.android.com/sdk/;
    };
  };

  ...
}

Other additions

Building Android applications

android create project --target android-8 --name MyFirstApp \
--path /home/sander/MyFirstApp --activity MainActivity \
--package com.example.myfirstapp

$ ant debug

$ keytool --genkeypair --alias sander

key.store=/home/sander/.keystore
key.alias=sander
key.store.password=foobar
key.alias.password=foobar

$ ant release

{androidenv}:

androidenv.buildApp {
  name = "MyFirstApp";
  src = ../../src/myfirstapp;
  platformVersions = [ "8" ];
  useGoogleAPIs = true;

  release = true;
  keyStore = /home/sander/keystore;
  keyAlias = "sander";
  keyStorePassword = "foobar";
  keyAliasPassword = "foobar";
}

rec {
  androidenv = import ./androidenv { ...  };

  myfirstapp = import ./myfirstapp {
    inherit androidenv;
  };
  ...
}

$ nix-build pkgs.nix -A myfirstapp
/nix/store/11fz1yxx33k9f9ail53cc1n65r1hhzlg-MyFirstApp
$ ls result/
MyFirstApp-release.apk

Emulating Android apps

$ android create avd -n device -t android-8

$ emulator -avd device -no-boot-anim -port 5554

$ adb -s emulator-5554 wait-for-device

$ adb -s emulator-5554 shell getprop dev.bootcomplete
1

$ adb -s emulator-5554 install result/MyFirstApp-release.apk

$ adb -s emulator-5554 shell am start -a android.intent.action.MAIN \
  -n com.example.my.first.app/.MainActivity

{androidenv, myfirstapp}:

androidenv.emulateApp {
  name = "MyFirstApp";
  app = myfirstapp;
  platformVersion = "16";
  useGoogleAPIs = false;
  package = "com.example.my.first.app";
  activity = "MainActivity";
}

$ nix-build pkgs.nix -A emulate_myfirstapp
./result/bin/run-test-emulator

Deploying Android apps on real devices

$ adb -d install result/MyFirstApp-release.apk

Conclusion

• androidenv.buildApp builds an Android App for a particular class of Android devices.
• androidenv.emulateApp generates a script that launches a particular emulator instance and automatically starts an App in it.

Availability

GNU Guix is a purely functional package manager, and associated free software distribution, for the GNU system. In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection.

GNU Guix

A comparison of package specifications

{stdenv, fetchurl}:

stdenv.mkDerivation {
  name = "cpio-2.11";

  src = fetchurl {
    url = mirror://gnu/cpio/cpio-2.11.tar.bz2;
    sha256 = "1gavgpzqwgkpagjxw72xgxz52y1ifgz0ckqh8g7cckz7jvyhp0mv";
  };

  patches = [ ./cpio-gets-undeclared.patch ];

  meta = {
    homepage = http://www.gnu.org/software/cpio/;
    longDescription = ''
      GNU cpio copies ...
    '';
    license = "GPLv3+";
  };
}

rec {
  stdenv = ...

  fetchurl = import ../pkgs/build-support/fetchurl {
    ...
  };

  cpio = import ../pkgs/tools/archivers/cpio {
    inherit stdenv fetchurl;
  };

  ...
}

$ nix-build all-packages.nix -A cpio
/nix/store/pl12qa4q1z...-cpio-2.11

(define-module (distro packages cpio)
  #:use-module (distro)
  #:use-module (guix packages)
  #:use-module (guix download)
  #:use-module (guix build-system gnu))

(define-public cpio
  (package
    (name "cpio")
    (version "2.11")
    (source
     (origin
      (method url-fetch)
      (uri (string-append "mirror://gnu/cpio/cpio-"
                          version ".tar.bz2"))
      (sha256
       (base32
        "1gavgpzqwgkpagjxw72xgxz52y1ifgz0ckqh8g7cckz7jvyhp0mv"))))
    (build-system gnu-build-system)
    (arguments
     `(#:patches (list (assoc-ref %build-inputs
                                  "patch/gets"))))
    (inputs
     `(("patch/gets" ,(search-patch "cpio-gets-undeclared.patch"))))
    (home-page "https://www.gnu.org/software/cpio/")
    (synopsis
     "A program to create or extract from cpio archives")
    (description
     "GNU Cpio copies ...")
    (license "GPLv3+")))

$ guix-build hello
/nix/store/pl12qa4q1z...-cpio-2.11

Possible advantages of Guix over Nix

• because it rocks!
• because it's GNU!
• it has a compiler, Unicode, gettext, libraries, etc.
• it supports embedded DSLs via macros
• can be used both for composition and build scripts

My thoughts

The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this.

• It's interesting to compare an internal vs external DSL approach in deployment. Although Ludovic has listed several potential benefits, I still don't see that these are proven yet by Guix. Some questions that came into my mind are:
  • Does compiling Guix specifications give any benefits, let's say in performance or in a different aspect?
  • Why do we need modules, such as gettext, for package specifications?
  • Can Guix expressions be better debugged than Nix expressions? (I know that the Nix expression language is a lazy purely functional language and that errors are very hard to debug).
  • I know that by using an internal DSL the language is extensible, but for what purposes is this useful and what can you do it what Nix currently cannot?
    
    On the other hand, I think the Nix expression language is also extensible in the sense that you can call any process from a derivation function (implementing an operation) and encapsulate derivation into a function with a nice interface. For example, I have used this for Disnix to integrate deployment planning algorithms. Maybe there are different kind of extensions or more efficient integration patterns possible with Guix? So far, I haven't seen anything concrete yet.
• I also find the integration aspect with Nix interesting. I have seen many language/environment specific package managers, for e.g. Perl, Python, Eclipse and they all solve deployment issues in their own way. Furthermore, they do not offer all the features we care about, such as transactional upgrades and reproducible builds. By making it possible to integrate language specific package managers with Nix, we can remove this burden/annoyance.
• If GNU packages can be easily packaged in a functional way, it will also make the lives of Nix packagers easier, as we don't have to implement any hacks/workarounds anymore and we can (semi-)automatically convert Nix and Guix expressions.
• It's also good to have a critical look at the foundations Nix/Nixpkgs, such as the bootstrap. In Guix, this entire process is reimplemented that may yield useful techniques/lessons that we can apply in Nix as well.
• To have Nix and the purely functional deployment model is the media always good, as we want to break conventional thoughts/ideas.

• A more concise syntax, which is shorter and better comprehensible. However, I have to admit that I'm a bit biased on this, because I only have little hands-on experience with the Scheme programming language and quite some experience with the Nix expression language.
• Better static consistency checking. External DSLs can produce more understandable error messages, whereas host language "abusement" may create all kinds of complex structures significantly making error reports much more complicated and difficult to grasp.
  
  Furthermore in an embedded DSL, you can also use the host language to do some unintended operations. For example, we could use Scheme to imperatively modify a variable, that is used by another package, affecting reproducibility (although the build of a derivation itself is pure, thanks to Nix). A packager has to manually take care that no side-effects are specified, while the Nix expression language prevents these side-effects to be programmed.
  
  Again, I have not done any comparisons with Nix and Guix, but I have worked several years in a research group (which besides me, includes Eelco Dolstra, the author of Nix) in which external DSLs are investigated, such as WebDSL, a domain-specific language for web applications.
  
  My former colleague Zef Hemel (along with a few others) wrote a series of blog posts covering issues with internal DSLs (one of them: 'When rails fails', covering some problems with internal DSLs in the Ruby programming language, has raised quite some (controversial) attention) and a paper titled: 'Static consistency checking of web applications with WebDSL' reporting on this matter.
• External DSLs have often smaller dependencies. If an internal DSL (embedded in a host language) is used in a tool, it often needs the entire host language runtime, which for some programming languages is quite big, while we only need a small subset of it. One of the questions we have encountered frequently in the Nix community, is why we didn't use Haskell (a lazy purely functional language) to specify package configurations in Nix. One of the reasons is that the Haskell runtime is quite big and has many (optional) dependencies.

• The Nix expression language is a lazy purely functional language, whereas Guix uses eager evaluation, although the derivation files that are produced and built are still processed lazily by the Nix worker. In Nix, laziness offers various benefits, such as that only the desired packages and its required dependencies are built, while everything we don't need is not, improving efficiency. In Guix, other means have to be used to achieve this goal and I'm not sure if Guix has a better approach.
• Guix's license is GPLv3 including the package descriptions, whereas Nix's license is LGPLv2.1 and the Nixpkgs is MIT licensed. Guix has a stronger copyleft than Nix. I don't want to have a debate about these licenses and the copyleft here, but for more information I'd like to redirect readers to an earlier blog post about free and open-source software and to form an opinion on this.

Concluding remarks