Tobias Watzek

Weeknumberology in .NET

2024-12-15T00:00:00Z

I’m currently writing a little program in C# (.NET 9) that needs to do some calculations with weeks. Specifically ISO weeks which start on Monday and the first week of a year is defined as the first week that contains the first Thursday of a year. I started to write some code that gets me the first day of the first week of the year when I stumbled over the System.Globalization.ISOWeek class. This humble static class contains six methods that cover the common things one may want to do with week numbers.

GetWeekOfYear(DateTime) returns the week number of a date. For example, GetWeekOfYear(new DateTime(2024, 12, 15)) will return 50.

The method GetWeeksInYear(Int32) returns the number of weeks in a given year. For 2024 this method returns 52.

GetYear(DateTime) may surprise you but according to the ISO week numbering system a day in a year may not be in that year if its week isn’t part of the year. Namely, the 1st January 2023 is actually part of the 52 week of 2022. When you pass it to GetYear(DateTime) you get 2022.

In order to get the start and end date of a year when working with week the methods GetYearStart(Int32) and GetYearEnd(Int32) can be used. For instance, the start of the year 2023 is the 2nd January 2023 and the end of 2022 is the 1st January 2023.

Last but not least there is ToDateTime(Int32, Int32, DayOfWeek) which will calculate the date of a weekday of a week in a year. When called like this ToDateTime(2024, 50, DayOfWeek.Sunday) the method will return the 15th December 2024.

I was relieved that I found this small but very helpful class. This once again demonstrates the vast collection of things that come with the .NET standard library that make our lives as developers easier.

Time Tracking Like a Nerd

2024-03-15T00:00:00Z

Tracking the time I spend on projects is often necessary but always annoying. In my professional life I have worked with various tools to accomplish this daunting task. They ranged from shiny web apps to desktop apps that were so complicated to use that it was easier to keep track of time in a separate Excel file and copy data over to the app every couple of days. If you look online for applications to track time you will find a myriad of results and each one has a ton of features that will probably never be used by the likes of me. I was looking for a solution that allows me to quickly and easily enter data while keeping everything stored locally. Luckily I did not have to look far. I’m already using hledger to keep track of my finances in plain text and recently learned that it has support to read and report on two different plain text time logging formats: timeclock and timedot.

Timeclock originates from the Emacs timeclock.el. It’s a bit harder for humans to read and write but allows logging the start and end of a task with second precision. Each line that starts with an i marks the start of a task. The next line that isn’t a comment (lines starting with #, ; or *) or blank must start with o which marks the end of the task. After an i or an o comes a date and time. The date and time in starting lines has to be followed by an account or project name. Separated by two spaces from the account optional comments and tags can be added.

i 2024-03-15 07:00:00 blogging  writing about time tracking after 2 spaces ; optional comment, tags:
o 2024-03-15 07:30:00
i 2024-03-15 07:30:00 eating  breakfast with coffee
o 2024-03-15 08:00:00

By running hledger commands with the above sample a number of reports can be generated. For example to get an overview over the hours tracked to each account the balance command can be used.

$ hledger -f sample.timeclock balance
               0.50h  blogging
               0.50h  eating
--------------------
               1.00h

To get a detailed list of all the logged hours in a specific month the register command with the --period option comes in handy.

$ hledger -f sample.timeclock register --period 2024-03
2024-03-15 writing about time tracking after 2 spaces  (blogging)  0.50h   0.50h
2024-03-15 breakfast with coffee                       (eating)    0.50h   1.00h

In order to get a report of all the hours that were logged to each account per month the register command in combination with the -M option can be used.

$ hledger -f sample.timeclock register -M
2024-03  blogging    1.00h    1.00h
         eating      0.50h    1.50h

In contrast to the timeclock format the timedot format is a bit easier to read for humans. It achieves this by removing the exact timestamps that mark the start and end of a task. Instead, it simply adds blocks of time to accounts on a day. There are multiple types of blocks but the one that gives the timedot format its name is the period . which is interpreted as 15 minutes. To illustrate this look at the following timedot example that tracks the same amount of time as the timeclock example above.

2024-03-15
blogging  ....
eating    ..

All the same commands that can be used with timeclock files can also be used with timedot files.

I’m still undecided which format I like more. At the moment I lean more towards the timedot format because I seldomly need to know the start and end times of the work I keep track of. In my experience time tracking is a very emotional topic and I don’t know many people that like it. I think that by using a plain text file to keep track of the time I spend on projects I am less distracted from the many features most time tracking applications offer and spend more time on the actual problems I try to solve.

How I Satisfied My Bookmarking Needs

2023-09-17T00:00:00Z

Some months ago, I was looking for the link to an article I read a while ago. I remembered some keywords and painstakingly sifted through search results of search engines, my browser history, and chats where I might have posted the article in question. All my efforts were in vain. I was furious that I did not save the link when I read the article. At that time, the only bookmarking functionality I used was the one offered by my browser. However, I only use my browser bookmarks to store links to sites I visit regularly and do not use them as a well-organized knowledge base. I decided it was high time that I found a solution for my bookmarking needs.

At first, I looked into the native capabilities of my browser. I use Mozilla Firefox on all of my devices. Bookmarks can easily be stored, tagged with keywords, organized in folders, and synchronized between devices. Although this would probably be enough for my needs, I want a separation between bookmarks of sites I visit regularly, which are more like shortcuts, and my archive of links, a collection of tools, articles, and other intriguing stuff. Because I want to manage these things separately, I decided against using my browsers bookmarks to build my archive.

Whenever I’m looking for a tool, one of the first places I visit is the Awesome Selfhosted list. It’s a long list of self-hosted tools. I found what I looked for in the “Bookmarks and Link Sharing” section. The one entry that stood out was linkding. It is described as a “Self-hosted bookmark manager that is designed to be minimal, fast, and easy to set up using Docker.”. This description is on point. I won’t get into how I host it because I already wrote about it in How I Host My Stuff, it actually is surprisingly easy to host.

An essential aspect of managing my bookmarks is how they are added. It would be a big turnoff when I would have to open my bookmarking application each time I wanted to add a new bookmark and manually add it. Fortunately, linkding offers browser extensions for Firefox and Chrome and provides instructions on adding bookmarks on mobile devices via bookmarklets or shortcuts.

Links added to linkding can be tagged with keywords so they can be filtered easily. This makes possible to search for links concerning a specific topic. Additionally, it is possible to add notes to each link, which comes in handy when I want to add some additional information to a link. Another cool feature is the possibility to mark links as unread. I often stumble across interesting tools or articles but might not have the time to read them. By adding them to linkding and marking them as unread, I can quickly find and read them later. Finally, I can close tabs in my browser with things that might be interesting rather than keeping them open for weeks.

After hosting linkding for some time, I must say that it is a solid piece of software. It runs without issues, and all updates have been applied effortlessly. The browser add-on and shortcuts enable me to quickly add new bookmarks to my collection.

In summary, linkding has become a significant part of my digital life. It has many features without feeling bloated and integrates seamlessly into browsers and mobile devices. I would suggest it to anyone looking for a minimal self-hosted bookmarking solution. Admittedly, more recently, I’ve been looking into more sophisticated applications because I would like to effectively manage and connect the entirety of my notes, links, and other information. Stay tuned for more information about this topic in the future.

Solving the Dictionary JSON Serialization Puzzle in .NET

2023-09-16T00:00:00Z

My last post Fine-Tuning JSON Serialization in .NET introduced JSON converters that can be used to serialize and deserialize types to and from JSON in .NET. The methods I described in that post work for most use cases. Unfortunately, I ran into some issues when working with dictionaries. In this post, I want to explain these problems and show how they can be solved.

To demonstrate how to serialize dictionaries to JSON, look at the following example, which only uses primitive types as keys and values that can be serialized and deserialized out of the box by the JsonSerializer provided by System.Text.Json.

using System.Collections.Generic;
using System.Text.Json;

var d = new Dictionary<string, int>()
{
    { "foo", 42 }
};

var options = new JsonSerializerOptions { WriteIndented = true };
var json = JsonSerializer.Serialize(d, options);

As expected, this code works, and the resulting JSON looks like this:

{
  "foo": 42
}

However, using a custom type as the key of a dictionary that should be serialized to JSON will result in a runtime exception. It will even fail when a JsonConverter<T>, implemented as described in my previous post, is used. Let’s try to serialize a dictionary that uses a custom Currency type as its key with a custom JSON converter.

using System;
using System.Collections.Generic;
using System.Text.Json;
using System.Text.Json.Serialization;

var eur = new Currency("EUR");

var d = new Dictionary<Currency, int>(){
    { eur, 42 }
};

var options = new JsonSerializerOptions
{
    WriteIndented = true,
    Converters = { new CurrencyJsonConverter() }
};
var json = JsonSerializer.Serialize(d, options);

public sealed record Currency(string Value);

public sealed class CurrencyJsonConverter : JsonConverter<Currency>
{
    public override Currency Read(
        ref Utf8JsonReader reader,
        Type typeToConvert,
        JsonSerializerOptions options) =>
            new Currency(reader.GetString());

    public override void Write(
        Utf8JsonWriter writer,
        Currency currency,
        JsonSerializerOptions options) =>
            writer.WriteStringValue(currency.Value);
}

Running this example will result in a System.NotSupportedException with the message “The type ‘Currency’ is not a supported dictionary key using converter of type ‘CurrencyJsonConverter’.”. So it seems that the CurrencyJsonConverter is used, but the implementation of the Write method is not used to serialize the dictionary key. A look at the stack trace of the exception, provides insight into where the exception occurred and how the problem might be resolved.

at System.Text.Json.ThrowHelper.ThrowNotSupportedException_DictionaryKeyTypeNotSupported(
  Type keyType,
  JsonConverter converter)
at System.Text.Json.Serialization.JsonConverter`1.WriteAsPropertyName(
  Utf8JsonWriter writer,
  T value,
  JsonSerializerOptions options)
...

The last call of the stack trace is just a helper method that throws the exception, but the call before to the JsonConverter.WriteAsPropertyName method looks interesting. A peak at the documentation reveals that this method “Writes a dictionary key as a JSON property name.”. Furthermore, the method is marked virtual so it can be overridden in the CurrencyJsonConverter.

 public sealed class CurrencyJsonConverter : JsonConverter<Currency>
 {
     public override Currency Read(
         ref Utf8JsonReader reader,
         Type typeToConvert,
         JsonSerializerOptions options) =>
             new Currency(reader.GetString());

     public override void Write(
         Utf8JsonWriter writer,
         Currency currency,
         JsonSerializerOptions options) =>
             writer.WriteStringValue(currency.Value);

+    public override void WriteAsPropertyName(
+        Utf8JsonWriter writer,
+        Currency currency,
+        JsonSerializerOptions options) =>
+            writer.WritePropertyName(currency.Value);
 }

It’s crucial to notice that, other than the Write method, the new WriteAsPropertyName method uses the WritePropertyName method of the Utf8JsonWriter instead of WriteStringValue. Running the example from above with the updated JSON converter works and produces the following JSON:

{
  "EUR": 42
}

This is great, but what if some JSON should be deserialized to a dictionary as demonstrated below?

using System;
using System.Collections.Generic;
using System.Text.Json;
using System.Text.Json.Serialization;

var json = """
{
  "EUR": 42
}
""";

var options = new JsonSerializerOptions
{
    Converters = { new CurrencyJsonConverter() }
};
var d = JsonSerializer.Deserialize<Dictionary<Currency, int>>(json, options);

Similar to the serialization example, the deserialization also throws a System.NotSupportedException with the message “The type ‘Currency’ is not a supported dictionary key using converter of type ‘CurrencyJsonConverter’.”. Something is still missing from the CurrencyJsonConverter. A look at the stack trace reveals a call to a JsonConverter.ReadAsPropertyName method. This method deserializes a JSON property name to a dictionary key and can be overridden, just like the JsonConverter.WriteAsPropertyName method.

 public sealed class CurrencyJsonConverter : JsonConverter<Currency>
 {
     public override Currency Read(
         ref Utf8JsonReader reader,
         Type typeToConvert,
         JsonSerializerOptions options) =>
             new Currency(reader.GetString());

     public override void Write(
         Utf8JsonWriter writer,
         Currency currency,
         JsonSerializerOptions options) =>
             writer.WriteStringValue(currency.Value);

     public override void WriteAsPropertyName(
         Utf8JsonWriter writer,
         Currency currency,
         JsonSerializerOptions options) =>
             writer.WritePropertyName(currency.Value);

+    public override Currency ReadAsPropertyName(
+        ref Utf8JsonReader reader,
+        Type typeToConvert,
+        JsonSerializerOptions options) =>
+            Read(ref reader, typeToConvert, options);
 }

In the override of the ReadAsPropertyName method, the already implemented Read method of the CurrencyJsonConverter can be used. With the updated CurrencyJsonConverter, the previously introduced deserialization example runs without issues.

I didn’t expect any problems when serializing and deserializing dictionaries to JSON, and was surprised to encounter them, although I was using JSON converters. After some debugging and digging into the documentation, finding the solution wasn’t as cumbersome as I initially anticipated. Simply implementing the methods WriteAsPropertyNameand ReadAsPropertyName on a custom JSON converter makes it possible to use custom types as keys for dictionaries that are serialized to or deserialized from JSON. At last, I hope my findings are valuable for people struggling with the same issues.

Fine-Tuning JSON Serialization in .NET

2023-08-23T00:00:00Z

Working with JSON in .NET is pretty easy with the types provided by System.Text.Json. Nevertheless, it is sometimes necessary to implement custom converters to serialize or deserialize some structures. To illustrate this, take the following example where I declare some simple types that are used to represent money.

public sealed record Money(Amount Amount, Currency Currency);
public sealed record Amount(decimal Value);
public sealed record Currency(string Value);

In order to serialize an instance of Money to JSON, the JsonSerializer.Serialize method from the System.Text.Json namespace can be used.

using System.Text.Json;

var oneEur = new Money(new Amount(1), new Currency("EUR"));
var options = new JsonSerializerOptions { WriteIndented = true };
var json = JsonSerializer.Serialize(oneEur, options);

The JSON looks like this:

{
  "Amount": {
    "Value": 1
  },
  "Currency": {
    "Value": "EUR"
  }
}

The produced JSON is nested two levels deep because of the Amount and Currency records. Neither is this efficient nor is it especially pretty. I would like the JSON to look like this:

{
  "Amount": 1,
  "Currency": "EUR"
}

To accomplish this, I need to implement one JSON converter for Amount and another one for Currency. A JSON converter can be created by inheriting from the JsonConverter<T> class and implementing the Write and Read method. The converters for Amount and Currency can be seen in the next example.

using System.Text.Json;
using System.Text.Json.Serialization;

public sealed class AmountJsonConverter : JsonConverter<Amount>
{
    public override Amount Read(
        ref Utf8JsonReader reader,
        Type typeToConvert,
        JsonSerializerOptions options) =>
            new Amount(reader.GetDecimal());

    public override void Write(
        Utf8JsonWriter writer,
        Amount amount,
        JsonSerializerOptions options) =>
            writer.WriteNumberValue(amount.Value);
}

public sealed class CurrencyJsonConverter : JsonConverter<Currency>
{
    public override Currency Read(
        ref Utf8JsonReader reader,
        Type typeToConvert,
        JsonSerializerOptions options) =>
            new Currency(reader.GetString());

    public override void Write(
        Utf8JsonWriter writer,
        Currency currency,
        JsonSerializerOptions options) =>
            writer.WriteStringValue(currency.Value);
}

The converters can be used by adding them to the JsonSerializerOptions that are passed to the JsonSerializer.Serialize method.

using System.Text.Json;

var oneEur = new Money(new Amount(1), new Currency("EUR"));
var options = new JsonSerializerOptions
{
    WriteIndented = true,
    Converters = {
        new AmountJsonConverter(),
        new CurrencyJsonConverter()
    }
};
var json = JsonSerializer.Serialize(oneEur, options);

Now the produced JSON is no longer unnecessary nested.

{
  "Amount": 1,
  "Currency": "EUR"
}

Another way to reach the same goal would be to implement a JSON converter for Money instead of implementing converters for Amount and Currency. In this converter, an intermediate type can be used for serialization and deserialization. This is especially useful when extensive customizations are necessary. A converter for Money could look like this:

public sealed class MoneyJsonConverter : JsonConverter<Money>
{
    private sealed record JsonMoney(decimal Amount, string Currency);

    public override Money Read(
        ref Utf8JsonReader reader,
        Type typeToConvert,
        JsonSerializerOptions options)
    {
        var jm = JsonSerializer.Deserialize<JsonMoney>(ref reader, options);
        return new Money(new Amount(jm.Amount), new Currency(jm.Currency));
    }

    public override void Write(
        Utf8JsonWriter writer,
        Money money,
        JsonSerializerOptions options)
    {
        var jm = new JsonMoney(money.Amount.Value, money.Currency.Value);
        JsonSerializer.Serialize(writer, jm, options);
    }
}

Custom JSON converters are a powerful tool to finely control the serialization and deserialization of complex data structures. I often use them when I have to conform to a specific JSON format while still being able to use expressive data structures in the code.

How I Back Up My Stuff

2023-04-23T00:00:00Z

In my last article, I wrote about how I host my stuff. A topic that goes hand in hand with hosting is backing up data. There are many strategies when it comes to backups. Depending on the type of data that should be backed up, they range from naive to doomsday prepper. For my simple applications, I apply a strategy that is more on the naive side because it will not hurt immensely if I should lose the data.

I started by choosing a place where I wanted to store my backups^[1]. My applications run on virtual private servers that I rent. Those servers run in a data center in Germany. Each of those servers has snapshot backup enabled, which makes it possible to reset the server to a previous snapshot for some days. This is the only backup I have enabled for servers that just serve statically generated sites. If something were to happen to those servers, I can just deploy the site to another server and nothing will be lost. For servers that host more dynamic sites that have a database or store uploaded files, I decided to use a tool called restic and back up my data via SFTP to rented storage hosted in Finland. This makes it less likely that my data will be lost if something happens to the data center in Germany.

Restic is a tool written in Go that can securely back up data to different storage types. It is pretty simple to set up and even supports cleaning up old backups via policies. I have it scheduled on my servers via Cron and configured to keep seven daily, four weekly, and three monthly backups. This tool makes it easy to back up all user-generated content of my applications.

Backups always seemed like a very complicated and exhausting topic to me, but after I researched a bit, it wasn’t as tricky as I made it out to be. My final setup utilizing server snapshots, restic, and hosted storage is simple but effective. Of course, if you have to securely back up critical data, it may be best to give this topic a tad more thought.

Something that comes to mind here is the 3-2-1 rule for backups. Three copies of data, two on different media, one is kept off-site. ↩︎

How I Host My Stuff

2023-04-22T00:00:00Z

This blog isn’t the only website or webapp that I operate from my living room. I operate several small sites and tools that run on rented virtual private servers. Until recently, I ran those applications directly on the machine and configured everything manually. On the one hand, this can be a lot of fun but on the other hand, sometimes it also causes great pain. Triggered by the need to host new applications, which were a bit more complex, I started to tinker with my setup. First, I gave Coolify a shot which is a cool project, but it did too much magic for my taste. Then I went down another rabbit hole and looked into simple Kubernetes variants like k3s, fortunately, this adventure didn’t last long. My goal was to find a way to run containerized versions of applications while still keeping everything as simple as possible. I found the solution in Docker Compose in combination with Traefik Proxy. Docker Compose is a neat way of defining and running applications that can consist of one or more containers, volumes, and networks. Traefik Proxy is a reverse proxy that can be hosted via Docker, dynamically load configuration from Docker containers, and automatically manage Let’s Encrypt certificates for applications. In the following, I want to show you how I use this combination to host my stuff.

To start everything off, I created a new Git repository that holds all my infrastructure configuration files. Never again would I edit a file somewhere on a remote machine and then lose all changes because I didn’t open it as root. Having all configurations in Git makes it easy to roll back changes and investigate what I did to bring down my server.

Now that I had a repository, I added some files, namely compose files. Each application has a docker-compose.yaml file that instructs Docker Compose on how to run the application. My initial compose file for linkding, a self hosted bookmark service can be seen in the following example.

version: "3"

services:
  linkding:
    image: "sissbruecker/linkding:1.17.2"
    container_name: "links"
    ports:
      - "9090:9090"
    volumes:
      - "links-data:/etc/linkding/data"
    restart: unless-stopped
volumes:
  links-data:

Here you can see that the file starts with information about the Docker Compose version, then comes a section that defines all services that should be run, and at the end, I define a single volume that is used by the service to persist data. This compose file can be started with the Docker Compose CLI by running docker-compose up. After doing this, the service can be accessed on port 9090. That’s a great start, but I needed the services to be accessible from the internet, and of course, they have to be secured with a TLS certificate.

That sounds like a job for a reverse proxy, and my reverse proxy of choice happens to be Traefik Proxy. It can be hosted via Docker, and dynamically loads its configuration from Docker. This means that I can spin up containers, and when they have some labels the proxy will automatically route traffic to them and even manage certificates. The next example shows the docker-compose.yaml for Traefik Proxy.

version: "3"

services:
  traefik:
    image: "traefik:v2.9.10"
    container_name: "traefik"
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.network=traefik_traefik"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.web.http.redirections.entrypoint.permanent=true"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.letsencrypt.acme.tlschallenge=true"
      - "--certificatesresolvers.letsencrypt.acme.email=REDACTED"
      - "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
    ports:
      - "443:443"
      - "80:80"
    volumes:
      - "traefik-letsencrypt:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    networks:
      - "traefik"
    restart: unless-stopped
volumes:
  traefik-letsencrypt:
networks:
  traefik:

The basic structure of this compose file is similar to the one I use for linkding. As you can see, I expose the default ports for HTTP 80 and HTTPS 443 traffic. To automatically detect configuration from other containers Traefik needs access to the /var/run/docker.sock socket which is solved by mounting it as a volume. Another thing I did was to put all applications that should be routed by Traefik into the same “traefik” network (called “traefik_traefik” when used by other applications). In the command section of the traefik service configuration, you can see the various command line arguments I pass to Traefik’s static configuration. These arguments instruct Traefik to load dynamic configuration from all Docker containers in the “traefik_traefik” network, redirect all traffic from the “web” endpoint on port 80 to the “websecure” endpoint on port 443, and lastly, I define a certificate resolver “letsencrypt” that can be used to fetch Let’s Encrypt certificates for applications.

Next, I wanted to route traffic to my linkding container. This can be achieved by simply placing some labels on the container. These labels are used to enable Traefik, define rules used to route traffic, set the entry points used by the container, and finally, the last label sets the certificate resolver that should be used. Additionally, the container has to be in the “traefik_traefik” network. The ports section is no longer needed. The changes I made to the compose file are shown in the following example.

  version: "3"

  services:
    linkding:
      image: "sissbruecker/linkding:1.17.2"
      container_name: "links"
-     ports:
-       - "9090:9090"
+     labels:
+       - "traefik.enable=true"
+       - "traefik.http.routers.links.rule=Host(`example.org`)"
+       - "traefik.http.routers.links.entrypoints=websecure"
+       - "traefik.http.routers.links.tls.certresolver=letsencrypt"
      volumes:
        - "links-data:/etc/linkding/data"
+     networks:
+       - "traefik_traefik"
      restart: unless-stopped
  volumes:
    links-data:
+ networks:
+   traefik_traefik:
+     external: true

After configuring some applications in the same way and running them on my small server, I ran into a problem with my server’s resources. Sometimes some of the applications took up all of the available memory and the host system had to take them down. Fortunately, there is a simple way to avoid this by setting some limits for the services defined in the compose files. These limit the memory and CPU power a service is able to use, and I highly recommend setting them. In this last example, you can see how I configured the limits for the linkding service.

  version: "3"

  services:
    linkding:
      image: "sissbruecker/linkding:1.17.2"
      container_name: "links"
      # ...
      restart: unless-stopped
+     deploy:
+       resources:
+         limits:
+           cpus: "0.25"
+           memory: "200M"
  # ...

Hosting my applications as containers works really well, and with all their configuration in a Git repository, setting everything up is fast and easy. As with all servers, it’s important to consider security every step of the way so please research best practices and stay safe while self-hosting.

Beautiful Architecture Diagrams

2023-03-30T00:00:00Z

Anybody who works with me knows that I’m a big fan of visualizing software with diagrams. In my experience, having a diagram of an application’s structure or processes performed by the application makes it a lot easier to talk about. High-level diagrams can be especially helpful when communicating with non-technical people. A model I really like to use for architecture diagrams is the C4 model by Simon Brown. This model takes a hierarchical approach and introduces four core diagram types: system context, container, component, and code. These diagrams visualize a software system from the highest to the lowest level.

Until recently I always used the well-known diagrams.net (formerly draw.io) with this C4 plugin to draw such diagrams. As I already wrote in this article about Mermaid I prefer to create diagrams using plain text tools. Mermaid provides an experimental C4 diagram type which looks promising, but is not ready for professional use. While exploring the C4 model website for the hundredth time, something caught my eye. There on the bottom was a logo and the text “The example diagrams were created with Structurizr”. Looking at the Structurizr site, I got excited because this was the tool I was looking for.

Structurizr introduces a sane syntax for C4 diagrams that feels right for me. There are customization options, but the generated diagrams look good enough out of the box.

To demonstrate the syntax, I created a small set of diagrams that describe this blog. Because of this blog’s simplicity, I only added a system context and a container diagram. In the following code snippet, the source code for the diagrams is shown. You can actually copy the code and paste it to the Structurizr DSL page to play around with it.

workspace {

    model {
        reader = person "Reader" "Surves the web in the search for interesting content."
        author = person "Author" "Writes articles."
        blog = softwareSystem "Blog" "A website that serves articles and pages." {
            webserver = container "Web Server" "Serves websites over the internet." "Caddy" {
                reader -> this "visits"
            }
            folder = container "Folder" "Contains HTML files and other assets." {
                webserver -> this "serves"
            }
            ssg = container "Static Site Generator" "Takes input files and generates a static website." "Eleventy" {
                this -> folder "generates website"
                author -> this "writes source files"
            }
        }
    }

    views {
        systemContext blog {
            include *
            autolayout lr
        }

        container blog {
            include *
            autolayout lr
        }

        theme default
    }
}

Structurizr diagrams are composed of a model and a views section that are contained in a workspace. In the first section all systems, persons, containers, and components of the diagrams and their relationships, are defined. The second section defines which views should be generated from the model and how they should look. The above code generates the following two diagrams.

In the above diagram, you can see the blog displayed as a software system and two persons that interact with the blog. The persons represent the readers and the author of the blog.

Going down one layer in the diagram, the containers that make up the blog are described. It is composed of a web server, a folder in the file system, and a static site generator that generates HTML files from source files.

If you have to draw some architecture diagrams, I can only recommend that you give Structurizr a try. I’m really impressed by the clear and expressive language, the tooling, and the generated diagrams.

Sounds like Vue

2023-02-26T00:00:00Z

Recently I built a custom audio player for one of my websites. I quickly created the first version using Alpine.js. But after looking at the kilobytes shipped to the user, I decided to try to build a more lightweight version with Vue.js.

Last, I used Vue in a customer project three or four years ago. Since then, Vue version 3 was released which introduced a new API called the composition API. This was the first time I was using the new API. There were some initial learnings, but I quickly got used to the new way of building Vue components. Honestly, Vue is a lot of fun to work with.

Although Vue can be used without any build process, the real fun starts with single-file components. A single-file component is a file with the .vue extension. Such a file contains the logic, template, and styles of a single component. No special syntax has to be learned as it is mostly just JavaScript in a script tag, HTML in a template tag, and CSS in a style tag. The following figure shows what a simple “Hello, World!” component looks like.

<script setup>
  const name = "Tobias";
</script>

<template>
  <div>Hello, {{ name }}!</div>
</template>

<style scoped>
  div {
    font-size: 2rem;
    font-weight: bold;
  }
</style>

Source and rendered output of the “Hello, World!” Vue component.

Although this component is pretty simple many things are going on. First, you might spot the setup attribute on the script tag. This tells Vue that this component uses the composition API and that some boilerplate for this API can be prepared by Vue. Inside the template tag, the variables and functions defined in the script can be used. In this component, simple text interpolation using the double curly braces is used to print the value of the name variable. At the end of the single-file component comes the style tag which has a special scoped attribute. This attribute instructs Vue to only apply the given styles to the current component. Therefore, I can just write a CSS rule targeting div without having to worry about breaking the styles of everything else on the page.

This is already pretty nice but to implement an audio player some interactivity is necessary. So let’s take a look at how Vue deals with that.

<script setup>
  import { reactive } from "vue";

  const state = reactive({
    stepSize: 1,
    gauge: 0,
    countDecrease: 0,
    countIncrease: 0,
  });

  function onDecrease() {
    state.gauge -= state.stepSize;
    state.countDecrease += 1;
  }

  function onIncrease() {
    state.gauge += state.stepSize;
    state.countIncrease += 1;
  }
</script>

<template>
  <div>
    <div>Gauge: {{ state.gauge }}</div>
    <div>Increase Count: {{ state.countIncrease }}</div>
    <div>Decrease Count: {{ state.countDecrease }}</div>
    <div>
      <label for="interactive-step-size">Step size</label>
      <input
        id="interactive-step-size"
        type="number"
        step="1"
        min="1"
        max="99"
        :value="state.stepSize"
        @input="state.stepSize = parseInt($event.target.value, 10)"
      />
    </div>
    <button @click="onDecrease">Decrease</button>
    <button @click="onIncrease">Increase</button>
    <button
      @click="
        state.stepSize = 1;
        state.gauge = 0;
        state.countDecrease = 0;
        state.countIncrease = 0;
      "
    >
      Reset
    </button>
  </div>
</template>

Source and rendered output of an interactive Vue component.

The component above consists of an input, three buttons, and three values. The decrease and the increase button decrease or increase the gauge by the number of steps configured via the step-size input. The third button can be used to reset the state of the component. The values that are displayed by the component are the current value of the gauge, the number of times the increase button was clicked, and the number of times the decrease button was clicked.

First, let’s take a look at the template tag. There you can see that text interpolation is used to display the three values. More interesting are the input and the three buttons. Some attributes start with an @ sign. These are event handlers. If you compare them you can see that it is possible to either pass a function that should be called each time the event occurs, for example onDecrease, or to handle the event inline and use the $event variable to access the event. Double curly braces cannot be used to dynamically bind the values of attributes. Instead, to dynamically bind a value to an attribute the attribute has to be prefixed with :. This is what happens to the value attribute of the input. It is bound to the state.stepSize variable.

If you take a look at the script tag you can see that the reactive function is imported from Vue. This function can be used to create reactive objects in Vue. By using this function Vue can track what happens to the object and update the UI if it needs to. The rest of the script only contains the two event handlers to increase or decrease the gauge. With this example, all basics of Vue should be clear and we can get into how I built the audio player, if you want to know more about Vue just head over to its great documentation.

At the core of my custom audio player lies an HTML audio element. Without any customization, this element looks different in each browser. Just like you can see and try below.

HTML audio element playing:
“Heroic Age” Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 4.0 License

I wanted my audio player to have the following features:

play/pause
go back 10 seconds
go forward 30 seconds
show progress
allow position seeking
display the current position
display the total duration
provide accessible labels

To implement the first three features only the basics of Vue and some knowledge of the HTML audio element are required. There needs to be some kind of state that holds information about the current playing state, position, and total duration. Then some buttons with event handlers are needed to toggle play and pause, go back 10 seconds, and go forward 30 seconds. These can be implemented via the reactive function and some @ event handlers as shown earlier. But somehow the audio element needs to be controlled and the total duration of the audio has to be known to correctly calculate the position when going forward. To do this two Vue functions are used.

First, we need the ref function which can either be used similarly to the reactive function or to hold a reference to a real DOM element, which has to be marked with the ref attribute, in a variable. With a ref variable that holds an audio element its JavaScript functions and properties can be accessed, for example, play(), pause(), duration, or currentTime. So now the element can be controlled by accessing the value of the ref variable that contains it. Additionally, I added event handlers to the play and pause events of the audio element to update the state of the component in case an external input starts or stops audio, for example media controls of the operating system.

Second, to set the total duration of the audio an event listener that listens to the loadedmetadata event can be registered. This event listener can store the duration in the reactive state. This will work sometimes but more often this event is not dispatched because the metadata of the audio element was already fetched before Vue registers its handlers. To still get the duration from the element a Vue lifecycle hook called onMounted can be used which is called after Vue created and inserted the components’ DOM tree. In this hook, the audio element can be accessed and its duration can be stored.

Without further ado here is the very simple audio player in all its glory.

<script setup>
  import { ref, reactive, onMounted } from "vue";

  const state = reactive({
    playing: false,
    position: 0,
    duration: 0,
  });

  const audioElement = ref(null);

  function togglePlayPause() {
    if (state.playing) {
      audioElement.value.pause();
    } else {
      audioElement.value.play();
    }
  }

  function goBack() {
    setPosition(Math.max(0, state.position - 10));
  }

  function goForward() {
    setPosition(Math.min(state.duration, state.position + 30));
  }

  function setPosition(value) {
    state.position = value;
    audioElement.value.currentTime = state.position;
  }

  onMounted(() => {
    audioElement.value.readyState > 0
      ? (state.duration = audioElement.value.duration)
      : undefined;
  });
</script>

<template>
  <div>
    <audio
      ref="audioElement"
      src="/path/to/audio.mp3"
      preload="metadata"
      @loadedmetadata="state.duration = $event.target.duration"
      @pause="state.playing = false"
      @play="state.playing = true"
      @timeupdate="state.position = $event.target.currentTime"
    ></audio>
    <div>
      <button @click="goBack()">-10 Seconds</button>
      <button @click="togglePlayPause()">
        {{ state.playing ? "Pause" : "Play" }}
      </button>
      <button @click="goForward()">+30 Seconds</button>
    </div>
  </div>
</template>

Simple audio player playing:
“Heroic Age” Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 4.0 License

After getting the basic functionality working it was time to look into how I could implement the progress and position-seeking. I used an HTML input of type range as the base for this functionality. The biggest challenge was styling this 🤬 control, which I will not discuss here. Just know that if you want to style such an input you are going to write many different styles for different browsers, which, as everybody knows, is always fun.

Back to the matter at hand. To implement seeking and keeping track of the progress I introduced a new property called seekingPosition to the reactive state of the component. This property is undefined until the range input is touched. While the range input is in use the seekingPosition holds the value of the input. When the range input is no longer used the seekingPosition is set back to undefined and the audio plays from the last value that was assigned to the seekingPosition. This means that the component can be in two different modes: playing mode or seeking mode. In playing mode the range input shows the current position. In seeking mode the range input shows the position the user is currently at with the handle. When switching to seeking mode the audio must continue to play until the handle is let go and a change event is dispatched by the input.

Most of the functionality from the simple player can be used as is. There are just some changes required to incorporate the seekingPosition. The setPosition function now resets the seekingPosition to undefined in case the position was set via the change event of the range input. Instead of an inline listener to the timeupdate event of the audio element that sets the position a function is introduced which does not update the position if the seekingPosition has a value. The seekingPosition is set via an inline event handler which reacts to the input event of the range input. Another Vue feature called a computed property is used for this part of the player. A computed property gets calculated based on other reactive variables and can be created with the computed function. In this case, the currentPosition is a computed property that is either set to the position or to the seekingPosition depending on the current mode of the component. This computed property is used to bind the value of the range input and to display the value beside the input formatted as hours, minutes, and seconds.

The following code snippet shows how the seeking and playing mode work. Some irrelevant parts have been removed from the snippet so please do not try to execute this.

<script setup>
  import { ref, reactive, computed } from "vue";

  const state = reactive({
    playing: false,
    seekingPosition: undefined,
    position: 0,
    duration: 0,
  });

  const audioElement = ref(null);

  const currentPosition = computed(() =>
    Math.floor(
      state.seekingPosition == undefined
        ? state.position
        : state.seekingPosition
    )
  );

  function setPosition(value) {
    state.position = value;
    audioElement.value.currentTime = state.position;
    state.seekingPosition = undefined;
  }

  function onTimeupdate(e) {
    if (state.seekingPosition != undefined) {
      return;
    }

    state.position = e.target.currentTime;
  }
</script>

<template>
  <audio
    ref="audioElement"
    src="/path/to/audio.mp3"
    preload="metadata"
    @loadedmetadata="state.duration = $event.target.duration"
    @pause="state.playing = false"
    @play="state.playing = true"
    @timeupdate="onTimeupdate"
  ></audio>
  <input
    type="range"
    aria-label="Position"
    :max="Math.floor(state.duration)"
    :value="currentPosition"
    :aria-valuetext="formatSecondToValuetext(currentPosition)"
    @input="state.seekingPosition = $event.target.value"
    @change="setPosition($event.target.value)"
  />
  <time :datetime="formatSecondsToDuration(currentPosition)"
    >{{ formatSecondsToDisplay(currentPosition) }}</time
  >
</template>

Last but not least I want to mention some techniques I used to try to make the audio player as accessible as possible. To make the player accessible for people that use assistive technologies like screen readers I: set labels for all controls, used the aria-valuetext attribute on the range input to provide a more meaningful textual representation of its value, declared a region around the player with the label “Audio Player”, and used HTML time elements to represent the duration and position with valid duration strings as datetime attribute values. Just remember that audio is inherently inaccessible for certain groups ant it should be accompanied by a textual representation whenever possible similar to images alt text.

Finally, the following audio player is the result of this endeavor. You can find its source in this GitHub Gist which you can use however you want. I hope you learned something from this article and will try out Vue if you haven’t already.

Audio player playing:
“Heroic Age” Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 4.0 License

Fun with YAML

2023-01-31T00:00:00Z

Today the release pipeline at work broke. No harm was done, but finding the cause of the problem was difficult. While I analyzed the issue, once again, I learned something surprising, nay horrifying, about YAML.

Our application is deployed to a Kubernetes cluster via Helm. Helm uses so-called Helm charts to package applications and their Kubernetes configuration together. These packages can then be used to install or upgrade applications on Kubernetes clusters. Charts are composed of YAML files that will be processed by a template engine during the deployment. One of our use cases for this feature is the replacement of placeholder tokens with environment-specific values.

Now to the helpful error message that was logged when the error occurred during the deployment. I formatted it for better readability and removed some irrelevant details.

Error: UPGRADE FAILED: YAML parse error on chart/templates/deployment.yaml:
error converting YAML to JSON:
yaml: invalid map key:
map[interface {}]interface {}{
  "required \"A .Values.someValue value is required.\" .Values.appSettings.someValue":
  interface {}(nil)
}

Only two lines were added to the deployment.yaml file mentioned in the error message. These two lines contained a placeholder that should be replaced by .Values.someValue that is passed to the Helm upgrade command. After I completely ignored the error message, I checked the usual suspects. First, I inspected if .Values.someValue contained a value. Second, I ensured that the deployment.yaml file is correctly indented. Neither of these theories checked out, so I finally read the error message and realized that the message was new to me. Unfortunately, I only understood that something went wrong during the parsing of the deployment.yaml file and that Helm is written in Go (interface {} ❤️).

So let’s take a look at the two lines that were added.

- name: "SomeName"
  value: { { required "A .Values.someValue value is required." .Values.someValue } }

It’s actually pretty easy to spot the issue in these two lines. Especially if I tell you that placeholders in Helm chart template files are supposed to start with two opening curly braces {{ and end with two closing curly braces }}. So the issue here is the space between the curly braces { { and } }. After the space was removed, the release worked again.

I could have stopped there, but one thing that left me puzzled was the error message. It isn’t simply saying something like: “Hey, you added some weird braces in a YAML file which isn’t valid YAML, duh!” but it actually tells me something like: “Hey, I tried to put your stuff into a map type but failed, and by the way, I use Go!”. So I assumed the YAML must be at least partially valid if the parser does not stop at the first curly brace it encounters.

It’s YAML specification reading time, yay. Specifically, I looked at the current YAML specification, which at the time of writing is version 1.2.2 and can be found at yaml.org. This is a pretty long document spanning ten chapters. So I did the first thing that came to my mind — activate the browsers search and type in “curly” — Bingo! I found exactly one match for the word, which led me to Chapter 7. Flow Style Production in the specification. If you want to know in detail what YAML’s flow styles are, just read the specification. I will only tell you that the following sample is 100% valid YAML and 97.8% valid JSON. Only the last value of the structure is not valid JSON.

{
  "is": "this",
  "really": "valid",
  "YAML?": true,
  "btw": [
    "this",
    "is",
    "also",
    "YAML"
  ],
  "oh": no
}

This explains why the parser didn’t fail when it encountered curly braces in the YAML but instead tried to do its best with what it got. After looking into this issue in detail, the error message also seems less cryptic. At its core, the error message says “invalid map key”. This means that a map, sometimes called a dictionary or object, is created, and the given key is invalid. The key is the construct inside the outer curly braces, which once again is a map. But a map cannot be used as the key of a map. Therefore, the error is raised.

It’s not over yet. The snippet above can be translated differently depending on the YAML specification that is implemented by the parser. Specifically, the last value, no, is problematic. In YAML version 1.2.2, this would be interpreted as the string "no" because you do not always need quotes around strings in YAML. But in earlier versions, no will be translated to the boolean false (see YAML v1.1 bool draft). Just search for the Norway problem online, in case you want to know more about this hilarious topic.

Once again, I know more about YAML than I did before, which will probably make it easier to avoid and detect bugs in the future. Generally, I think YAML has its place, but it is way more complex than JSON, making mistakes more common. The German phrase “So ein Jammer!” can roughly be translated to “What a pity!” and I often have to say “So ein YAML!” whenever I have to touch YAML.

DIY Visual Studio Code Autocomplete

2023-01-04T00:00:00Z

Several months ago, I received a little consulting job that required me to create a structured report.Naturally, I wrote a custom application called dokumentatorin to convert YAML files to other formats. The transformed files can be processed further by tools like Pandoc to generate nice-looking documents. Although the generated reports were guaranteed to look beautiful, I still had to create dozens of YAML files and fill them with data. My editor of choice for such a task is Visual Studio Code.

So there I was, creating the first YAML file and adding about ten fields to it. Some of the fields contained simple strings, while others expected a certain format. After finishing the third file, I decided that I could not possibly continue like this. I am used to all of my code editors supporting me whenever I edit structured data or code. I wanted to have the same experience for these YAML files. Otherwise, I could have just used some office text editor like a sane person.

My tool that processes the YAML files could already validate the files using a JSON Schema. So why not reuse the schema that I already built? In general, a JSON Schema is used to describe and validate the structure of a JSON document while also using the JSON data format itself. The following is an example JSON Schema for a person with a name, an age, and a favorite ice cream flavor. Each field has a type and some annotations used to validate and describe the data and mark which fields are required.

{
  "type": "object",
  "description": "A person.",
  "additionalProperties": false,
  "properties": {
    "name": {
      "type": "string",
      "description": "First and last name of the person."
    },
    "age": {
      "type": "integer",
      "description": "Age of the person.",
      "minimum": 0
    },
    "iceCreamFlavor": {
      "type": "string",
      "description": "Favourite ice cream flavor of the person.",
      "enum": ["vanilla", "chocolate", "strawberry", "blue"]
    }
  },
  "required": ["name", "age", "iceCreamFlavor"]
}

I spent some time searching around and found out that with the help of the YAML extension for Visual Studio Code and some settings, I could build my own autocomplete and validation using my JSON Schema. To demonstrate this imagine a directory structured like this:

.vscode/
- settings.json
data/
- john.yaml
schema.json

I want the JSON Schema defined above and stored in the schema.json file to provide autocompletion and validation for all YAML files in the data directory. To accomplish this, I have to add the following setting to the .vscode/settings.json file.

{
  "yaml.schemas": {
    "schema.json": "data/*.yaml"
  }
}

Now I can press CTRL + Space in the john.yaml file and get suggestions for the complete structure defined in the JSON Schema.

In my opinion, this feature is powerful and can be used in many situations. It certainly made my job a lot easier and prevented some mistakes.

Drawing Diagrams with Mermaids

2022-07-31T00:00:00Z

An image says more than a thousand words. This old saying is very true when it comes to software development. It is way easier to discuss a topic with multiple people when you can draw an image of the thing you are talking about. Otherwise, everybody will probably have a slightly different understanding of the topic. There are many tools out there that make it easy to quickly draw some boxes and arrows like draw.io, Excalidraw, or Miro. I have used all of those tools and I can recommend them but one thing all of them have in common is that they are graphical tools that require the user to think about distracting stuff like layout or colors when thinking about a problem. As a programmer, my brain is most comfortable with thinking in plain text and several tools out there make it possible to write text that gets turned into beautiful diagrams. In this post, I want to tell you about Mermaid which works well for most of my use-cases but there exist quite a few similar tools like Graphviz or PlantUML which are also quite interesting.

Mermaid describes itself as a “JavaScript based diagramming and charting tool that renders Markdown-inspired text definitions to create and modify diagrams dynamically”. It does an extraordinary job. The syntax of the different diagram types is intuitive and easy to remember. There are many different diagram types like flowcharts, sequence diagrams, class diagrams, state diagrams, and many more. For quick designs, there exists a live editor with share and export functionality. Even platforms like GitHub or Azure DevOps offer support for embedded Mermaid diagrams in Markdown. This makes Mermaid a great choice to document software. I use the Mermaid live editor to quickly sketch out parts of a system and then discuss them with colleagues. These diagrams can then be used to implement the design and also end up in the documentation. Due to the plain text source of those diagrams, it is also possible to quickly change the documentation without having to redraw everything.

As I mentioned at the beginning an image says more than a thousand words so let me show you some diagrams and their source code so you get a better feeling for Mermaid.

The following is a diagram that creates a flowchart for “make it work, make it right, make it fast”. In my opinion, the source code can be understood without looking into the documentation of Mermaid.

graph TD
    A(Develop feature)
    work{Does it work?}
    makeWork[make it work]

    right{Is it right?}
    makeRight[make it right]

    fast{Is it fast enough?}
    makeFast[make it fast]

    done(feature done)

    A --> work
    work -- no --> makeWork
    work -- yes --> right
    makeWork --> work
    right -- no --> makeRight
    makeRight --> right
    right -- yes --> fast
    fast -- no --> makeFast
    makeFast --> fast
    fast -- yes --> done

The next diagram is a sequence diagram of a pizza order. This type of diagram demonstrates that not a lot of code is needed to draw a quite complex diagram that would be hard to draw in a graphical tool.

sequenceDiagram
    Me->>Pizza-Place: I'd like to order a pizza
    Pizza-Place->>+Me: Order Confirmation
    Pizza-Place-)+Cook: Make a new pizza
    Cook--)-Pizza-Place: Pizza
    Pizza-Place-)+Delivery Person: Deliver this pizza
    Delivery Person->>Me: Pizza
    Delivery Person--)-Pizza-Place: Pizza delivered

As you can see in the examples Mermaid is a very powerful tool. It allows everyone to quickly draw good-looking diagrams that are easy to maintain and understand. I suggest that you consider it the next time you need a diagram. Here is a final example of a pie chart.

pie title Which tools were used to draw diagrams in this post?
    "Mermaid" : 100

Faster Frontend Builds

2022-07-21T00:00:00Z

At work, we are developing a single-page application with React. It used the Create React App build tools that, albeit they were not the fastest, did their job. I was bothered by this as it often slowed me down when developing locally but I also had some issues with our build pipeline. Just imagine that you write a fix for a problem and you have to wait about ten minutes for your pull request build and after that, you have to wait the same amount of time for the release build. For an impatient millennial this is unacceptable. I had to change something.

It is common knowledge that in the JavaScript world every problem can be solved by the inclusion of yet another package. So I did exactly that. Instead of using Create React App I updated the project to use …drumroll please… Vite. Locally this made things fast, very fast. Starting the local development server now takes about three seconds instead of thirty. This was already a huge win for us. On the build server, this change did not provide any noticeable speed improvements. That is because, at the moment of writing, Vite uses different mechanisms for development and optimized production builds.

After I was satisfied with the improvements to development builds I started to look at how the Azure DevOps build pipeline that I wrote half a year ago could be improved. This pipeline runs for each pull request and the release build. Its output is a Docker image that includes a web server that serves the built single-page application. Inside the pipeline, a multi-stage Docker build was run. This build used a Dockerfile which first built the application and ran tests inside of a Node.js base image and then copied the built application from the first stage to the final stage that was based on the image of a web server. The major reason for building and testing the application during the Docker build was to ensure that the build is independent of dependencies installed on the build server. This can be a valid reason but in our case, all steps of the build run on a hosted agent which is a container that gets created for the build job and is destroyed after the job finishes. So we do not have to worry about someone installing updates on the shared build server and with that breaking all the builds that relied on a previous version of some dependency. For the frontend build, I decided it was perfectly safe to drop the build inside of Docker, build everything on the agent itself, and as a final step create a Docker image with the finished application. This change alone chopped off one and a half minutes of build time.

Finally, I could rest and enjoy the fruits of my labor, or could I? There was still one thing that bothered me. At the beginning of each automated build, about thirty seconds were spent downloading NPM packages. These packages seldom change and would therefore be a perfect candidate for some sort of caching. This is where the Cache task comes in. This simple task can be used to cache dependencies for a given cache key which can also be the contents of a file. By using the package.lock.json file as a cache key all NPM packages are cached until the file changes. By applying this final improvement most of the builds only spend a tiny amount of time restoring the package cache instead of downloading the whole internet over and over again.

It’s over it’s done. I could now finally move my attention to the features I should have been building the whole time, but now I can develop them without having to wait for eons (aka seconds) until my development environment spins up and my code is released to production. Unless by coincidence some new JavaScript engine named like a baked good comes along that makes everything run faster I am finished with performance improvements of my build.

Favicon Quantization

2022-04-28T00:00:00Z

Recently I stumbled upon the tool Schrödinger Hat which can be used to transform images to only use the colors of specific color palettes. What does that mean? Look at the following image that I took during my trip to Bruges.

After I processed it with the tool using the Tokyo Night palette the old houses of Bruges look more like a place that can be found in the Feywild than a place in Belgium.

Seeing those images I got an idea for a new favicon. I wanted to retire my previous favicon for quite some time now so I started experimenting. After looking at the website of the tool I found out that it is based on the Python library ImageGoNord. It was time to warm up my rusty Python skills and fire up a Jupyter notebook.

Looking at the library and experimenting with some images I found out that one of the methods that the library uses is color quantization. This image processing method can be used to reduce the number of colors in an image. Usually, the goal is to use fewer colors than the original image uses while still maintaining the original look of the image. One of the reasons to do this is to reduce the size of an image. If you want to know more about this process I suggest you take a look at the color quantization article on Wikipedia.

Unfortunately, I was not happy with the results I got from the Python library so I looked into some alternatives in the .NET/C# space with which I am more familiar. Luckily the great ImageSharp library which I already had the pleasure of working with in the past has quite an impressive repertoire of image processing options. One of them being color quantization. I played around for some time and finally reached a result with which I was happy.

In the following code block you can see the whole program I used to process the image. First, the image called me.jpg gets loaded. Then a palette using the primary colors of the website is created and used to initialize a PaletteQuantizer. Using the Mutate function a chain of modifications is applied to the image. In this chain the image is converted to black and white, then the contrast is increased and finally, the quantization is applied. The only step left is to save the created image. As a side note, I want to mention that I did all this using a .NET Interactive Notebook which was a pleasant experience.

using SixLabors.ImageSharp;
using SixLabors.ImageSharp.Processing;

using var image = Image.Load("me.jpg");

var palette = new []{
    // Grey
    new Color(new Rgb24(0x2E, 0x34, 0x40)),
    // Green
    new Color(new Rgb24(0x48, 0xbb, 0x78)),
    // Pink
    new Color(new Rgb24(0xf6, 0x87, 0xb3)),
};

var quantizer = new PaletteQuantizer(palette);

image.Mutate(x => x.BlackWhite()
    .Contrast(1.5f)
    .Quantize(quantizer));

image.Save("favicon.png");

Now for the big reveal here are the source photo I used for my favicon and the processed version of the image using the code above.

As you can see, all it takes is one library and a handful of lines of C# code to apply some major modifications to an image. The next time I have to resize some pictures I will think twice if I open GIMP or instead write a quick C# program to do the job.

Rider Plugin Recommendations

2022-04-26T00:00:00Z

At work, I mostly write C# code using the great JetBrains Rider IDE. To add some features to the already great feature set I use some plugins that I will share in this article.

Cognitive Complexity

The Cognitive Complexity plugin tells you when your methods start to get out of hand. It calculates a score for each method that is based on the paper G. Ann Campbell in Cognitive Complexity - A new way of measuring understandability. This score is composed primarily of the nesting of statements and the control flow structures used in the method. The plugin gives you a nice badge above each method that indicates how complicated it seems according to its score. The score is updated while the code is written so you get instant feedback.

I like this plugin because it offers an objective rating of the complicatedness of a method which can be discussed with other developers. Like with every metric this one should not be used without thinking. Just because a method is rated “simple enough” by the plugin it does not mean that is not actually “very complex” and vice versa.

String Manipulation

All your String Manipulation needs will be satisfied by this one. It can do so many things: converting casing like PascalCase to snake_case, escaping text, encoding text, sorting lines, trimming, aligning multiple carets, converting dates, or even inserting a sequence of numbers. I do not need this often but when I do this plugin never disappoints me.

Nyan Progress Bar

Progress bars are boring and I have to admit that Rider does show them occasionally. To make the experience more entertaining I highly recommend installing the Nyan Progress Bar 🐈 and if you can’t get enough just visit nyan.cat.

Type Alias

2021-09-24T00:00:00Z

While working on projects sometimes the need to add a thing that behaves like a primitive type but only allows a defined set of values. Many people would just use an enum and get on with their lives. I’m not one of those people therefore I built myself something I call a TypeAlias which I use in many projects.

Imagine you have the assignment to store identifiers for different types of animals (Dolphin, Hippopotamus, and Tiger) and also have some code that behaves differently depending on the animal type. Of course, you do not simply want to use strings or magic numbers to do this.

First, let me demonstrate how this could be solved with an enum and tell you why this might not be the best solution.

The following code is the definition of the AnimalTypeEnum enum.

public enum AnimalTypeEnum
{
    Dolphin = 1,
    Hippopotamus = 2,
    Tiger = 3
}

So far so good. One requirement may be to store or transmit a textual representation of the AnimalTypeEnum instead of the numeric value of the enum member. This could be done by a piece of code similar to the following.

var tigerString = AnimalTypeEnum.Tiger.ToString();

Now for the inverse, the following can be used.

if (Enum.TryParse("Dolphin", out AnimalTypeEnum a))
{
    // it worked
}

Now for some fun stuff. What happens if you work with the numerical values of the enum and you want to convert this number back to the enum?

var animalType = (AnimalTypeEnum) 1;
Console.WriteLine(animalType.ToString());
// prints "Dolphin"

The example above would work but what would happen if there was once the enum member AnimalTypeEnum.Dodo with the value 4. Because of its extinction, it was removed from the code but some entries that are using the value still exist in the database.

var animalType = (AnimalTypeEnum) 4;
Console.WriteLine(animalType.ToString());
// prints "4"

This behavior certainly is not pretty. There is no warning of any kind. The correct way is to always check if the provided value is even defined before casting it. This is demonstrated in the next snippet.

var v = 4;
if (Enum.IsDefined(typeof(AnimalTypeEnum), v))
{
    var animalType = (AnimalTypeEnum) v;
    Console.WriteLine(animalType.ToString());
}
// no output

When using enums this validation should be made in every function that accepts enum values to protect the program from invalid enum values. A similar case of this happens when an uninitialized variable of the enum type is used. Per default, such a variable has the value 0. Therefore it is recommended to define an enum member with the value 0 that is defined as nothing or unknown. Otherwise, the following case is possible.

public class Thing
{
    public AnimalTypeEnum AnimalType { get; }
}

var thing = new Thing();
Console.WriteLine(thing.AnimalType.ToString());
// prints "0"

It may seem petty to not use enums because of the issues mentioned above but when I write software I like to make impossible states impossible and enums provide way too many ways in which they could be misused.

This is where the following construct comes in to save the day. The following snipped contains an abstract class that can be inherited from to build strongly typed aliases for types like strings or integers.

/// <summary>
/// A generic class that can be used
/// to create a strongly typed alias of a type.
/// </summary>
/// <typeparam name="T">
/// Type of the value.
/// </typeparam>
public abstract class TypeAlias<T>
    : IEquatable<TypeAlias<T>>,
      IEquatable<T> where T : notnull
{
    /// <summary>
    /// Value of the instance.
    /// </summary>
    public T Value { get; }

    protected TypeAlias(T value)
    {
        Value = value;
    }

    public static implicit operator T?(TypeAlias<T>? typeAlias)
    {
        return typeAlias is null ? default : typeAlias.Value;
    }

    /// <inheritdoc />
    public override string ToString()
    {
        return Value.ToString() ?? string.Empty;
    }


    /// <inheritdoc />
    public bool Equals(TypeAlias<T>? other)
    {
        if (ReferenceEquals(null, other))
        {
            return false;
        }

        if (ReferenceEquals(this, other))
        {
            return true;
        }

        return EqualityComparer<T>.Default.Equals(Value, other.Value);
    }

    /// <inheritdoc />
    public override bool Equals(object? obj)
    {
        if (ReferenceEquals(null, obj))
        {
            return false;
        }

        if (ReferenceEquals(this, obj))
        {
            return true;
        }

        if (obj.GetType() != this.GetType())
        {
            return false;
        }

        return Equals((TypeAlias<T>)obj);
    }

    /// <inheritdoc />
    public bool Equals(T? other)
    {
        return Value.Equals(other);
    }

    /// <inheritdoc />
    public override int GetHashCode() =>
     EqualityComparer<T>.Default.GetHashCode(Value);

    public static bool operator ==(TypeAlias<T>? objA, TypeAlias<T>? objB)
    {
        return Equals(objA, objB);
    }

    public static bool operator !=(TypeAlias<T>? objA, TypeAlias<T>? objB)
    {
        return !(objA == objB);
    }

    public static bool operator ==(TypeAlias<T>? objA, T? objB)
    {
        if (objA is null)
        {
            return objB == null;
        }

        return objA.Value.Equals(objB);
    }

    public static bool operator !=(TypeAlias<T>? objA, T? objB)
    {
        return !(objA == objB);
    }

    public static bool operator ==(T? objA, TypeAlias<T>? objB)
    {
        return objB == objA;
    }

    public static bool operator !=(T? objA, TypeAlias<T>? objB)
    {
        return !(objB == objA);
    }
}

The TypeAlias<T> has a Value property which stores the value of the instance. It implements IEquatable<TypeAlias<T>> in a way that uses its value to determine equality with other instances. Additionally IEquatable<T> is implemented so it can be compared with values of type T. This means that for example the equality between a TypeAlias<string> and a string can be determined. The corresponding operators == and != are also overloaded. Last but not least the implicit cast operator is implemented for T? which makes it possible to for example cast a TypeAlias<int> to an int.

So how would I implement the AnimalType enum from above as a TypeAlias and perform similar operations as above? The implementation of the TypeAlias is provided in the next snippet.

public class AnimalType : TypeAlias<string>
{
    private AnimalType(string value)
        : base(value) { }

    public static AnimalType Dolphin { get; } =
        new("Dolphin");
    public static AnimalType Hippopotamus { get; } =
        new("Hippopotamus");
    public static AnimalType Tiger { get; } =
        new("Tiger");

    public static IImmutableSet<AnimalType> PossibleValues { get; } =
        new HashSet<AnimalType>() { Dolphin, Hippopotamus, Tiger }
            .ToImmutableHashSet();

    public static AnimalType? FromString(string value)
    {
        StringToAnimalType.TryGetValue(value, out var animalType);
        return animalType;
    }

    private static readonly IImmutableDictionary<string, AnimalType> StringToAnimalType =
        PossibleValues.ToImmutableDictionary(e => e.Value, e => e);
}

Using the new AnimalType as string can be achieved by calling the ToString() method similar to the enum example above or by using the implicit conversion as demonstrated below.

string dolphin = AnimalType.Dolphin;

There is no need to validate if a value is defined or not when using a variable of type AnimalType because it is not possible to produce an instance of the AnimalType that contains an invalid value. To create an instance of AnimalType from a string the FromString method can be used that returns null when the value is invalid.

Not only does using the TypeAlias class enable the creation of safer code it also works nicely together with EntityFramework Core value conversions to convert between raw values and TypeAlias implementations. This means that all application code can use the safe TypeAlias and values of primitive types and validations of those values only have to be present at the edges of the application.

Handcrafting Enums Sucks

2020-10-22T00:00:00Z

Go is one of my go-to languages (pun intended) when I want to quickly build something or try something out. Recently I built a small HTTP API that returned an entity as JSON. This entity had a member that was an enum type. Go doesn’t have enums like some other languages but it allows the imitation of enums by using a type alias, constant groups, and the keyword iota.

type Pet int

const (
    cat Pet = iota
    dog
    pig
)

The values of the Pet type above will start at 0 for the cat and end with 2 for the pig. That’s the way how the iota keyword works. Now I defined the enum but when I serialize one of its values to JSON it is transformed into a number. An alternative would be to define the enum by using a string alias and assigning string values.

type PetString string

const (
    cat PetString = "cat"
    dog = "dog"
    pig = "pig"
)

By using the strings to define the values of the enum a conversion to JSON would return string values. But now I’ve lost the cool numbering features of iota and everything feels a bit repetitive.

Being the responsible software engineer I am I went on to find an alternative. Naturally the great go generate command came to my rescue. This command is bundled with the Go tools and can be used to execute commands that are specified inside of Go source files in special comments. Such a comment can look like this //go:generate touch foo. This would execute the touch command on a file called foo and therefore update the files timestamp or create the file if it didn’t exist. The funny thing is that you can actually run any Go source file by calling go run mycode.go in a Go generate comment. That means that you can write source code that generates other source code.

Now I knew what I wanted to do. I wanted to write a code generator that generates my enums and all the custom JSON conversion logic from and to strings. To generate my code I wrote a Go file that uses the jennifer package to generate Go code in a fluent fashion. After passing the values of my Pet enum to my generator the following code is generated. I removed the package, import, and JSON serialization code from the code for brevity.

type Pet int

const (
    cat Pet = iota
    dog
    pig
)

func (s Pet) String() string {
    return toStringPet[s]
}

var toStringPet = map[Pet]string{
    cat: "cat",
    dog: "dog",
    pig: "pig",
}
var toIdPet = map[string]Pet{
    "cat": cat,
    "dog": dog,
    "pig": pig,
}

func (s Pet) MarshalJSON() ([]byte, error) {
    // ... marshaling logic
}
func (s *Pet) UnmarshalJSON(b []byte) error {
    // ... unmarshaling logic
}

The complete code for my generator can be found on GitHub. After the enums are generated I can reference the enums in my code and serialize them to JSON as strings. Another benefit of using a code generator is that I could actually generate the enums by calling another API or reading in a file that provides the values. All in all, I am very happy with my solution and think that it may come in quite handy in some situations.

Privacy for My Azure SQL Database

2020-10-11T00:00:00Z

I’m a bit of an Microsoft Azure novice. Recent tasks lead to me learning some of the possibilities offered by Microsofts cloud computing service. Diving into this world feels like exploring a deep dungeon filled with great treasures but also with great dangers. One who ventures into this world has to learn new terminologies, access long-forgotten networking knowledge, and have a keen mind to avoid paying for things that aren’t required.

First, let me tell you what I want to run in the cloud. I have an ASP.NET web application that uses a Microsoft SQL database as data storage. The classic approach to this setup would be to either run the application and the database server on one machine or to separate the application and the database by using two machines. It doesn’t matter if the machines are physical or virtual as long as they are running Microsoft Windows. As intriguing as it seems to be running and administrating two virtual machines in Azure it didn’t seem like this is the cloud way to go.

After some research, I found the two services I wanted to use. My application should run inside of an Azure App Service which is more or less a container that only runs my application and makes it accessible to the web. To avoid having to manage a Microsoft SQL Server I opted in to use an Azure SQL Database. This database is mostly compatible with Microsoft SQL Server and completely managed.

The initial setup in which I tried to just make it work was fairly simple. I created an app service and a SQL database via the Azure web interface. After that, I published my application to the app service with the help of Visual Studio. Configuring the app service to use the Azure SQL database worked by configuring a connection string in the settings of the service. I opened the web app in the browser and everything worked fine.

Now that everything worked I wanted to secure the setup. In the current state, the database could be accessed from everywhere by everyone who has the credentials. But I only want my application to be able to access the database. In a classical scenario, I would put the database and the application together in a private network so they could communicate without using the public internet. As expected Azure provides a service called Virtual Network which provides exactly what I need. It allows the creation of private networks that can be used with other Azure services.

So I created a Virtual Network in the web interface. Navigated to the database server firewall settings disabled all public access and added the newly created network. Then I navigated to the App Service and added the virtual network in the networking tab. I opened the application in the browser and was greeted with an error. The application could not reach the database. After some cursing and applying the old “Have you tried turning it off and on again?” fix nothing changed.

Beaten down I did the one thing only a desperate person does: I read the documentation. There is a page about integrating app services with virtual networks which mentions a configuration key called WEBSITE_VNET_ROUTE_ALL. I skimped over it a couple of times but then I finally understood my problem. Azure SQL Databases get a hostname that is resolved to a public IP. An app service with a virtual network by default only sends RFC1918 traffic through the virtual network. Meaning that it only sends traffic that is intended for an IP in one of the private IP ranges. All other traffic is routed over the public internet. This means that requests to the database came from a public IP and not from within the virtual network. The firewall didn’t allow those requests and the access failed. After I enabled the WEBSITE_VNET_ROUTE_ALL configuration key for the app service by setting it to 1 I could access the database again. I also tried to access the database from my local machine which didn’t work. I finally reached my goal.

Cloud services like Microsoft Azure make it very easy to get an application up and running fast. But to keep your programs and data safe it is important to read the documentation carefully and to make yourself comfortable with the platform by trying things out.

Light It Up

2020-07-31T00:00:00Z

My team and I at work are currently working on a website that is built in a very classic way with a content management system in the backend that renders pages directly on the server. There is not a lot of action going on in the user interface as it mostly displays text and image content. But one or two of our tasks required us to implement some dynamic elements for the site. Dynamic elements in this context mean elements on the page that instantly react to user input. For example, a select box which filters a list of items when a user selects a value. After some consideration, we decided to build custom Web Components for those elements. Web Components describe a collection of browser technologies that make it possible to define so-called custom elements. A custom element can be used like any other HTML element (e.g. p, a, input,…).

Although it is possible to build Web Components only with the APIs provided by browsers our choice was it to use the LitElement library as it provides a declarative way to define Web Components paired with excellent TypeScript support. To make it easier to build Web Components the library provides some decorators, lifecycle events, templating functions, and styling techniques.

To demonstrate some functionalities of the library in this article I decided to write a small component which is called light-it-up. It shows a pulsating round light in a box. It has the properties boxColor and lightColor which control which colors are used. Below is a code block that shows how the component is used in this article and the rendered component itself.

<light-it-up boxColor="#03071e" lightColor="#dc2f02"> </light-it-up>

Now to the fun part. The code of the whole component:

@customElement("light-it-up")
export class LightItUp extends LitElement {
  @property({ type: String }) public boxColor = "#000";
  @property({ type: String }) public lightColor = "#3F88C5";

  public static get styles(): CSSResult {
    return css`
      .box {
        // ...
      }
      .light {
        // ...
      }
    `;
  }

  // Implement `render` to define a template for your element.
  protected render(): TemplateResult | "" {
    return html`<div
      class="box"
      style="${styleMap({ backgroundColor: this.boxColor })}"
    >
      <div
        class="light"
        style="${styleMap({ backgroundColor: this.lightColor })}"
      ></div>
    </div>`;
  }
}

By adding @customElement("light-it-up") to the class it is registered as a custom element with the name light-it-up. The @property decorators make it possible to bind values from attributes to the members boxColor and lightColor. Another benefit that @property provides is the re-rendering of the component when their values change. The static get styles() getter returns a tagged template string that is used as a stylesheet for the component. And finally, there is the render() method that gets called each time the component is rendered to the browser. It uses a tagged template string html`...` which gets processed by the great lit-html templating library.

This is just a simple example as to how Web Components and libraries like LitElement can be used. Of course, there are pretty complex components that can be built without the need of including a big JavaScript UI framework into your project. The complete code for the light-it-up component can be found in the following repository.

All Shapes Are Beautiful

2020-06-07T00:00:00Z

This post contains some CSS style demonstrations which may only work in recent versions of modern browsers. A feed reader, Lynx, or Internet Explorer will probably not display this article in all its beauty.

What if I told you, that the web doesn’t have to consist only of boxes. This is something I just recently discovered after I had to build the header of a website in the shape of a trapezoid. At first, I wanted to use images to accomplish this but the site should be able to scale for multiple resolutions which made this approach quite difficult. After some research, I found out about the CSS clip-path property. This property makes it possible to define basic shapes like polygons, ellipses, or circles. When a clip-path is applied to an element only the defined region of the element is visible. Most modern browsers support clip-path. To see which browsers should work you can take a look at the following “Can I use…” table. Below you should see three elements that are shaped like a circle, a diamond, and an ellipse.

These are not images but actual div elements. I apply one of the following classes to each of the elements to clip them into the right shape.

.circle {
  clip-path: circle(50% at 50% 50%);
}
.diamond {
  clip-path: polygon(50% 0, 100% 50%, 50% 100%, 0 50%);
}
.ellipse {
  clip-path: ellipse(25% 40% at 50% 50%);
}

It is quite nice to be able to transform elements into simple shapes but the real power of clip-path is that it can use SVG to define what should be clipped away. In the image below you can see my cat. An SVG clip-path of a brushstroke is applied to it which gives the image a creative border.

The code to use an SVG as a clip-path is very similar to the simple shapes above. A SVG clipPath element is referenced by id via the url('#cat-clip-path') function. The referenced SVG is included in the source code of this post.

.cat {
  clip-path: url("#cat-clip-path");
}

To find out more about CSS clip-path I recommend that you look at the guides provided by MDN. A nice tool that can help to build clip paths is Clippy. I hope this article offered an interesting introduction to the CSS clip-path property and why it is pretty awesome.

Counting Goats

2020-06-02T00:00:00Z

I would describe myself as a privacy-conscious person. Therefore it wasn’t an easy choice for me if I should implement some kind of analytics on my website. But in the end, my curiosity won and I decided to add some kind of tracking/analytics to my site. The solution had to be as privacy-aware as possible while still providing useful data. So I didn’t even consider tools like Google Analytics which have a very bad reputation regarding privacy.

After some research I found three possible candidates Fathom Analytics, GoAccess, and GoatCounter.

I rejected Fathom Analytics early on as it is only available via a paid subscription which would not be a viable option for my homepage. There is also a lite version available that can be hosted but it requires cookies to work which also was a no-go for me.

GoAccess takes a different approach to aggregate visitor information than most solutions. Instead of a client-side script, it analyzes the server access logs for the page in real-time. The results can either be viewed in the terminal or the browser. And I have to say that the web view is pretty sexy but see for yourself in this live demo or the screenshot below. GoAccess provides many functionalities and presents the data very nicely. But server access logs have to be enabled and GoAccess has to have access to them. Because of its complexity and because it didn’t quite feel like the right tool for the job I rejected GoAccess after playing around with it for some time.

This left only GoatCounter as a possible analytics tool. It is built and operated by Martin Tournoij aka arp242 and the source code is available on GitHub. On the landing page, it advertises features like privacy-awareness, small size, accessibility, and data ownership. All of those features lead me to chose this solution. Besides the usage is free for non-commercial projects which also made it attractive for me. The user interface is pretty simple but it provides clear information about all the data that I care about. A screenshot can be seen below. Getting the whole thing to work took surprisingly little time. I only had to create an account and insert a small script tag at the bottom of my page and suddenly the page visits were visible in the backend.

As you can see there are plenty of privacy-focused web analytics tools out there. It is just pretty difficult to choose one of them. Time will show if I made the right decision by choosing GoatCounter but I have a very good feeling that it was the right choice.

MIDI for the Home Office

2020-03-22T00:00:00Z

I’m one of the many employees who work from home because of the currently ongoing COVID-19 pandemic. This is not my normal work environment and I had to make some adjustments to my workspace at home to make it work. One of the most fun and probably one of the most unnecessary ones was it to make a USB MIDI controller act as an input device for frequently used functions or hotkeys on my Windows work computer.

Fortunately, I had an old USB MIDI controller lying around. It is an AKAI LPD8 but any MIDI controller should work. This controller has eight pads and eight control dials. Additionally, the controller can switch between four programs which can change the signals of each input. Therefore it would be possible to configure 64 different functions. I currently use four pads and one dial.

I had the idea to use this controller after my first day working from home. Three things annoyed me: controlling my music, activating push to talk in the work voice chat, and controlling the volume of my music player. At first, I just installed the software for the controller and tried to find out if this would be sufficient. But the software can only be used to change the MIDI signals (e.g. which pad is mapped to which note) that the controller emits. This didn’t stop me and I started to look into alternatives. Then I found AutoHotKey which is best described by this sentence on their website:

The ultimate automation scripting language for Windows.

It makes it possible to react to keypresses and execute complex actions. For example, it is possible to send key combinations, build simple GUIs, or execute programs. But there seemed to be one drawback. AutoHotKey normally does not understand MIDI signals. A bit more digging and I stumbled upon midi4ahk. This project provides a DLL and AutoHotKey scripts that can be linked in custom scripts to react to MIDI events.

Let’s take a quick look at how a MIDI message is structured. It consists of a status byte that describes the type of the message and sometimes the channel on which the message was sent. After the status byte are two additional bytes which are the parameters of the type. The three messages which were most important to me were “Note On”, “Note Off”, and “Control Change”. A “Note On” message has the status byte 1001 0000₂ or a bit shorter as hexadecimal number 90₁₆. The first half of the status byte indicates the “Note On” type and the second half is the channel number. The parameters are the musical note (0-127) and the velocity (0-127) or how hard the button was pressed. The first bit of both bytes is always 0 therefore only 128 values are possible. The “Note Off” message is the same except for the status byte which is 80₁₆. “Control Change” messages have the status byte B0₁₆. Similar to the note messages the first half of the byte indicates that the message is of the type “Control Change” and the second half is the channel number. This message type has as parameters the controller number (0-127) and the controller value (0-127).

Now I could finally start with the implementation. The library midi4ahk provides a function called listenNote which can be used to execute another function when a specific note is played. I wanted to use pad 8 of my controller to play or pause my music. The software of the MIDI controller showed me that this pad produces the note with the number 41 (F₂). So I wrote the following script:

; 1. param note number
; 2. param function name
; 3. param MIDI channel (0 == all)
listenNote(41, "playPauseMusic", 0)
return

; 1. param note number
; 2. param velocity (0-127)
playPauseMusic(note, vel)
{
    if (vel)
    {
        Send, {Media_Play_Pause}
    }
    return
}

As expected it didn’t work when I tried it out. After some MIDI debugging with the excellent Pocket MIDI tool I found out the following. My MIDI controller first sends a “Note On” message with the velocity set according to how strong I press the pad. When I release the pad it sends a “Note Off” message with the velocity set to 127. My script does not differentiate between “Note On” and “Note Off” events. Hence it pauses the music for a short moment when receiving the “Note On” message and shortly after it starts the music again when the “Note Off” message is received. To overcome this and only react to a “Note On” event I used a small hack. I only send the play or pause signal when the velocity is above zero but below 127. Because “Note Off” events always have a velocity of 127 this works as long as I don’t smash the pad with full force. The following code snippet shows the final implementation of the playPauseMusic function.

playPauseMusic(note, vel)
{
    if (vel > 0 and vel < 127)
    {
        Send, {Media_Play_Pause}
    }
    return
}

And to demonstrate that it works here is a short video of me pressing the pad on the MIDI controller and my music player reacting to it by pausing the playback.

Sorry, the video is not supported by your browser.

After implementing the pads to act as media control keys and to send a push to talk key combination I moved on to controlling the volume of my music player. When I’m working I often listen to music in the background but I also want to hear my colleagues talking in the voice chat. This makes me change the volume of my music player often and it takes a fair number of clicks to do it. The control dials on my MIDI controller seemed to be a perfect fit for the task at hand. When they are turned they emit a “Control Change” message with their position mapped from 0 to 127. All I thought I had to do was listen to “Control Change” messages with the listenCC function and then AutoHotKey will surely offer a function to change the volume of a program. The first part of it was true but AutoHotKey does not offer such functionality. I had to wander again into the depths of the internet to find an answer. This time I came back with the tool NirCmd. It is a command-line program that can be used to execute a set of tasks without any GUIs. One of those tasks is called setappvolume and it can be used to set the volume of a specific application to a value between 0 and 1. So I wrote the following script.

; 1. param control number
; 2. param function name
; 3. param MIDI channel (0 == all)
listenCC(4, "musicVolume", 0)
return

; 1. param control number
; 2. param control value
musicVolume(num, val)
{
    volume := ((100/127)*val)/100
    Run, nircmd.exe setappvolume MyMusicPlayer.exe %volume%
    return
}

I register the musicVolume function to be executed every time the control with the number 4 is activated. The musicVolume function receives the control number and the control value which is between 0 and 127 as inputs. In the function, I map the value to a number between 0 and 1 and execute the NirCmd setappvolume command with the name of my music player and the calculated volume. This did work but I saw some problems with the implementation. For each value sent by the control dial, this function is executed and the volume updated. The control dial sends a value for each position that is passed while turning it. This could lead to a lot of calls and wasted CPU usage. I monitored my CPU usage when I turned the control dial and each turn let the CPU usage climb 40% higher. Unfortunately, AutoHotKey has no automatic debounce utilities when functions are used so I had to implement a simple version myself as shown in the following snippet.

; 1. param control number
; 2. param control value
musicVolume(num, val)
{
    static lastCall := -1
    thisCall := A_TickCount
    if (lastCall = -1)
    {
        lastCall := thisCall
    }
    if ((thisCall - lastCall) < 16)
    {
        lastCall := thisCall
        return
    }
    lastCall := thisCall
    volume := ((100/127)*val)/100
    Run, nircmd.exe setappvolume MyMusicPlayer.exe %volume%
    return
}

To implement a simple debounce I used a static function variable which keeps its value across executions of a function and the built-in variable A_TickCount which contains the number of milliseconds since the system was started. By remembering when the function was called the last time I was able to only execute the NirCmd setappvolume command when the last function execution was longer than 16 milliseconds ago.

The following video shows me turning the control dial and the volume bar of my music player moving up and down in sync with it.

Sorry, the video is not supported by your browser.

I highly recommend that you get your hands on a MIDI controller and use it to control your computer. It makes working from home feel like controlling a space ship 🚀.

I Ate Some Delicious HTTPie

2020-03-15T00:00:00Z

Everyone who works with websites and web APIs at one point will have to test or debug some HTTP calls. A vanilla HTTP GET request can be done with your browser of choice by entering the URL into the address bar. But what do you do if you want to make a call with another HTTP verb or set some headers? There are many graphical applications like Insomnia or Postman and the almighty cURL. I have used all three of them to work with HTTP endpoints and they do a great job but there is a new kid in town.

HTTPie is a command-line tool that can be used to interact with HTTP endpoints. It provides features like colorized output, JSON support, and intuitive usability.

> http httpbin.org/status/418
HTTP/1.1 418 I'M A TEAPOT
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 135
Date: Sun, 15 Mar 2020 17:36:47 GMT
Server: gunicorn/19.9.0
x-more-info: http://tools.ietf.org/html/rfc2324


    -=[ teapot ]=-

       _...._
     .'  _ _ `.
    | ."` ^ `". _,
    \_;`"---"`|//
      |       ;/
      \_     _/
        `"""`

The example above makes an HTTP GET request to http://httpbin.org/status/418 and prints the headers and body of the response. In the terminal, the headers are syntax highlighted. It looks similar to the following example.

HTTP/1.1 418 I'M A TEAPOT
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 135
Date: Sun, 15 Mar 2020 17:36:47 GMT
Server: gunicorn/19.9.0
x-more-info: http://tools.ietf.org/html/rfc2324

It is also possible to set headers for your request by writing them in the format Header:Value after the URL. In the following example, I set the User-Agent header for the request to Ichiban. The --body flag instructs HTTPie to only print the response body.

> http --body httpbin.org/headers User-Agent:Ichiban
{
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "Host": "httpbin.org",
        "User-Agent": "Ichiban",
        "X-Amzn-Trace-Id": "Root=1-5e6e6be0-f16095947012ee8d94f75fe9"
    }
}

If another HTTP method then GET should be used the name of the method has to be written before the URL. Like in the following example where a DELETE call is made to http://httpbin.org/anything/42.

> http --body DELETE httpbin.org/anything/42
{
    "args": {},
    "data": "",
    "files": {},
    "form": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "Content-Length": "0",
        "Host": "httpbin.org",
        "User-Agent": "HTTPie/0.9.8",
        "X-Amzn-Trace-Id": "Root=1-5e6e6e0d-fb2d984b569fadc4f6fddc92"
    },
    "json": null,
    "method": "DELETE",
    "origin": "91.115.48.164",
    "url": "http://httpbin.org/anything/42"
}

In my opinion, this tool doesn’t replace the other applications that I mentioned above. But when you quickly want to test if a call is working this is the right tool for the job. It is cross-platform, quick, and easy to use.

All the Headers

2020-03-08T00:00:00Z

I recently found out about the Mozilla Observatory. It is a website scanner that rates a website based on the implemented security features. Among others, it checks for the encryption standards used for HTTPS and the headers sent by the server. My page initially scored a C+. Thanks to Let’s Encrypt which provides my TLS certificate and the great Caddy web server which serves my website and automatically provides HTTPS there were no improvements to be made in this area. But some important HTTP headers were missing. I actually didn’t know a lot about those headers and I want to document what I learned.

At the time of writing a call to curl -I https://watzek.dev, which can be used to make an HTTP HEAD call that only fetches the headers of a document, returns the following.

  HTTP/2 200
  accept-ranges: bytes
  cache-control: no-cache
+ content-security-policy: default-src 'none'; frame-ancestors 'none'; form-action 'self'; base-uri 'none'; img-src 'self'; script-src 'self'; style-src 'self'; manifest-src 'self'; block-all-mixed-content; upgrade-insecure-requests;
  content-type: text/html; charset=utf-8
  etag: "q67wby54s"
+ feature-policy: microphone 'none'; geolocation 'none'; autoplay 'none'; accelerometer 'none'; ambient-light-sensor 'none'; battery 'none'; camera 'none'; display-capture 'none'; document-domain 'none'; encrypted-media 'none'; fullscreen 'none'; gyroscope 'none'; magnetometer 'none'; midi 'none'; payment 'none'; publickey-credentials 'none'; sync-xhr 'none'; usb 'none'; wake-lock 'none'; xr-spatial-tracking 'none'
  last-modified: Mon, 24 Feb 2020 17:52:46 GMT
+ referrer-policy: same-origin
  server: Caddy
+ strict-transport-security: max-age=63072000; includeSubDomains; preload
  x-clacks-overhead: GNU Terry Pratchett
+ x-content-type-options: nosniff
+ x-frame-options: DENY
+ x-xss-protection: 1; mode=block
  content-length: 6652
  date: Sat, 07 Mar 2020 17:41:33 GMT

The important headers are highlighted in the code block above. They are content-security-policy, feature-policy, referrer-policy, strict-transport-security, x-content-type-options, x-frame-options, and x-xss-protection.

Content-Security-Policy

A visit to a website leads to the download of multiple resources. When the homepage of my site is loaded an HTML file, a CSS stylesheet, a JavaScript script, an image, and favicon images are downloaded. By default, the browser would just load the files without looking if I request them from the same origin or from any other URL.

This is where the Content-Security-Policy (CSP) header comes into play. Among other directives, it makes it possible to explicitly define which origins are allowed for resources. In my case I set the allowed origins for resources I use to 'self' which means that the browser should only load them if they are on the same origin. This also disallows inline scripts or stylesheets.

In addition to controlling the allowed origins of resources this header also controls to which origins forms can send data with the form-action policy.

Setting the frame-ancestors policy is similar to setting the X-Frame-Options header. It controls where a website may be embedded.

My site also sets the block-all-mixed-content policy and the upgrade-insecure-requests policy. The first policy disallows the loading of resources that are loaded with HTTP when the site is loaded via HTTPS. The second policy instructs the browser to also use HTTPS when a resource is actually linked via HTTP.

Feature-Policy

Modern browsers provide a variety of features that can be used by a website. Examples for such features are access to microphones or cameras of the user. Normally all those features are available and could be used by any code that is running on a site. The Feature-Policy header can be used to explicitly set the features that are used by a page. This prevents the site to use features that are not explicitly enabled. For my site, I disabled nearly every feature because it doesn’t use any of them.

Referrer-Policy

This header can be used to prevent the leak of sensitive information and provide visitors of the site with more privacy. Browsers usually send a Referer [sic] header which contains the address of the previously visited page. This information can be quite useful for some features but unfortunately, it can also be used for tracking or other malicious activities. For my site, I set the header to same-origin. This means that for requests on the same origin the whole address is set as Referer header value but for requests to other sites, so-called cross-origin requests, no value will be set. This prevents other sites from tracking that visitors are coming from my site.

Strict-Transport-Security

My website should only be reached via TLS encryption. By setting the Strict-Transport-Security header browsers know that the site should only be accessed with HTTPS. This is also known as HTTP Strict Transport Security (HSTS).

The header has the value max-age=63072000; includeSubDomains; preload. This means that browsers should remember to only access the site via HTTPS for 63072000 seconds (2 years). All subdomains should also be loaded via HTTPS. The preload attribute is special as it is actually not part of the specification. It allows the site to be included in the HSTS list of browsers. By being included in the list browsers know that the site should be loaded via HTTPS only and they won’t even try to use HTTP. Actually all .dev domains are automatically included in the list.

X-Content-Type-Options

When the stylesheet /styles/styles.css of this site is requested a Content-Type header is sent along with the response. This header has the value text/css; charset=utf-8. By looking at the value of the header the browser knows how the response of the server should be interpreted. In addition to looking at the value of the header browsers sometimes perform operations to determine the data type of the response. If this is the case the browser probably won’t use the content type from the header. This could potentially lead to the transformation of non-executable types to executable ones. By setting the X-Content-Type-Options header the browser adheres to the value of the Content-Type header and won’t perform any of those operations.

X-Frame-Options

Some HTML elements like <iframe> or <object> allow a website to embed another site. This can be used to make visitors think that they are on one site when the site is actually only embedded into another website. Consider for example an online banking site that is embedded into another malicious site. To prevent browsers from embedding my site I set the X-Frame-Options header to DENY.

X-XSS-Protection

This header is not implemented in all browsers and support for it will probably be removed in the future. It instructs the browser what to do if it detects a cross-site scripting (XSS) attack. The value 1; mode=block instructs the browser to not render the page when an attack is detected.

Conclusion

It was amazing for me to see how many headers are available to secure websites. They provide fine-grained control over the security of websites. By understanding and using them properly the web can get saver for users and also the privacy of users can be protected.

Static Is Fantastic

2020-02-23T00:00:00Z

Static site generators are fantastic. They make it easy to build websites without the need for server-side code or databases. Most static site generators work similarly. They take several templates and some content as input and output a folder with a generated website. Some generators are very opinionated about how the source of the website is structured, which template types can be used, or what can be used as content. Other static site generators take a different approach and are more open about how the website is built. For the redesign of my page, I chose Eleventy as static site generator.

Eleventy is written in JavaScript and uses the Node.js runtime. This makes it easy to customize the build by using packages from the vast selection offered by the npm software registry. But not only is Eleventy easily extensible it also provides a lot of functionality out of the box.

For my website, I decided to write my templates in the Nunjucks template language. It is a very powerful templating language and similar to other templating languages I used before. The template for the header of a blog post looks similar to the following example.

<header>
    <h1 class="mb-0 font-bold">
        {{ title }}
    </h1>
    {% if description %}
    <p class="text-sm italic leading-tight">
        {{ description }}
    </p>
    {% endif %}
    <p class="text-sm">
        <time datetime="{{ date | htmlDateString }}">
            {{ date | dateDisplay }}
        </time>
    </p>
</header>

It is possible to print out data from the processed content like the title or the description. Additionally, Eleventy provides functionalities to create filters or functions that can be used by multiple template engines. In the example above the htmlDateString or the dateDisplay filter are such functions.

Eleventy does not only support a variety of template languages it also makes it possible to use different kinds of data. For example, JSON files can be used as a data source or even JavaScript scripts that return data from an external source like a web API.

In contrast to other static site generators, Eleventy does not provide tools to process assets like stylesheets, scripts, and images by itself. In my opinion, this is a feature. It allows Eleventy to concentrate on the job of generating a folder structure and HTML files from the given data sources. Processing of assets can be added by choosing tools from the npm software repository. I integrated tools that process my stylesheets and images into the build. For example, the great postcss package enables me to use tools like cssnano which minifies the size of my CSS. And for the optimization of images, I use imagemin.

With the templates and build process in place, it is now easy for me to write blog posts without interruptions. I have to create a new Markdown file in the posts folder add a title, a date, and optionally a description as front matter and can start writing. The front matter of this article looks similar to the following snippet.

---
title: "Static is fantastic"
date: 2020-02-23
description: "Generating my website statically."
---

Deployments of a new version of the site are very boring (in a positive way). I have to take the folder that is produced from the build process and transfer it onto my webserver. I don’t have to restart any services, run migrations, or do anything else. In the future, I want to look at projects other than blogs to see how static site generation can be a useful alternative to dynamic server-side rendered sites.

The Quest for the Website Accessibility Grail

2020-02-15T00:00:00Z

One of the key requirements that I set for the redesign of my website is accessibility. For me, accessibility means to enable as many people as possible to use my website without any additional effort for them. Information that I provide should be available to everyone. I don’t want to erect any barriers that could prevent people from accessing my content.

But how can a website be made accessible for people with and without disabilities, mobile users, or people with slow internet connections? In the following, I will explain how I tried to make my website accessible to everyone.

Without structure we are barbarians

Using the right HTML elements provides a good base for an accessible website. The markup language HTML offers a multitude of elements or tags that can be used to express the meaning of the content of a website, but the right elements must be used in the right context. Often HTML elements are misused. For example, a web application could use <a> elements to trigger JavaScript handlers. Which would be wrong because the meaning of an <a> element is it to introduce a hyperlink. The element that should be used instead is a <button> element.

Not only does HTML give meaning to the content it also structures the page. The following code block shows a simplified version of the markup of the <body> element of this page.

<body>
  <header></header>
  <main></main>
  <footer></footer>
</body>

Can you guess where the sections of this page are in the markup? The <header> element contains the main navigation of the page wrapped in <nav> tags. Inside of the <main> element is the article you are currently reading. This element contains the primary content of a page and an HTML document may only contain one <main> element that is not hidden. The <footer> contains some links to additional information about me and the website. If you look inside the <main> element you will find that it is structured similarly to the <body> element. In the following code block, you find a simplified structure of the <main> element.

<main>
  <article>
    <header></header>
    <div></div>
    <footer></footer>
  </article>
</main>

Here it is visible that the <main> content contains an <article> element. An <article> element is used for self-contained content that can be reused on its own. An <article> element can have a <header> and a <footer> just like the <body> element. The difference is that these elements have to be related to the enclosing <article> element. Assistive technologies like screen readers can make use of those elements to allow visitors easier navigation on the site.

Those points only scratch the surface of what has to be thought of when structuring the HTML of an accessible website. An excellent source of information is this MDN article about accessible HTML.

Paint me a picture with words

Another important practice is it to provide textual descriptions of images. This can be done by setting alt attributes on <img> elements.

For example, the image above is represented in the HTML with the following code.

<img src="./src/images/rooster.jpg" alt="Johann the rooster" />

There is an alt attribute that describes what can be seen in the image. This description can be used by assistive technology so that information provided by images is not lost. But alt attributes can also be useful for people with slow network connections that have disabled the loading of images. Useful tips about the usage of the alt attribute can be found in this article about the alternate text attribute from The A11Y Project.

Grow into your design

A practical approach to designing a website is to start from mobile screen sizes and to work your way up. This so-called mobile-first approach ensures that a website works on the device with the most limitations in terms of screen size and network bandwidth first. As the development continues and the screen size grows features can be added if necessary. This design process feels more natural than going form big screen sizes to small screens and often leads to better results than other approaches.

In terms of accessibility this means that the page uses an appropriate font size for smaller displays, links or buttons are big enough to be tapped on a touchscreen, and the site is also usable with a slow network connection.

Be fast

It is a privilege to have access to a fast network connection and a website should not be built in a way that renders it useless when accessed via a slower connection.

In my quest to make my website small I abstained from using big CSS frameworks or custom fonts. Additionally, I decided to generate this website statically which reduces the response time of the server.

Look at the colors

A website without colors can look pretty dull. But colors should not be chosen carelessly. There are many forms of color blindness and colors should be chosen to still be distinguishable from each other for people with such disabilities. Another requirement for colors is their contrast. The contrast between background and foreground must be high enough so the can be distinguished from each other.

Question everything

Because there are a lot of things to consider it is important to test if there are no errors that negatively influence the accessibility of the website. To do so there are automatic tools that can identify accessibility problems like WAVE, the Accessibility Inspector built into the Firefox developer tools, or Google Lighthouse. Unfortunately, not everything can be tested automatically and it is a good practice to do some additional manual testing. For example try to navigate the website with the keyboard only, use a screen reader to explore the website, surf around with a throttled network connection, and check if everything works on an older smartphone.

So in the spirit of questioning everything please let me know if you find any accessibility issues or other problems with the redesign of my website. Hopefully this article helped you to lear a bit more about web accessibility and don’t forget to apply the things you learned to websites you build yourself.

Image Processing Experiments (3) – F#

2019-10-17T00:00:00Z

TL;DR My F# version of the image processing experiments may not be as short as the bash version but it is by far the most beautiful to look at. Find the source code in this repository: tobiaswatzek/image-processing-experiments

The introduction to the image processing experiments can be found in my first article of the series.

Just to recall the program should do the following:

Read in an image as the first argument passed to it on the command line.
Print the file path, width and height of the image to the console.
Display the original image and a blurred version of the image.
Convert the image to grayscale and save it besides the original image as gray-IMAGENAME.

After implementing a solution in Rust and in Bash I decided to build the final implementation with a functional programming language. The first language I wanted to use was Haskell but I was not familiar with the development processes and tooling of the language and I had a deadline to hold which repelled me from using it. Just recently I had looked into F# and because my professional job has provided me with some expertise with the .NET ecosystem I chose F# as the language for the third version of the program.

Programming in F# or other functional languages requires a different thinking process opposed to non functional languages. Combined with the syntax of the language this leads to more concise and readable code. While the syntax of the language is very concise and abstains from the myriad of brackets that other languages require it initially is a bit tricky and the indent based delimitation of code blocks can lead to some funny frustrating errors. A great page to learn the basics of the F# syntax is F# syntax in 60 seconds from F# for fun and profit by Scott Wlaschin.

let showImage path =
        let imgProcessInfo = ProcessStartInfo()
        imgProcessInfo.UseShellExecute <- true
        imgProcessInfo.FileName <- path
        Diagnostics.Process.Start(imgProcessInfo) |> ignore

As an example for a function I wrote you can see the showImage function that accepts a path to an image as parameter. Some interesting operators are the assignment operator <- which can assign values to variables and the forward pipe operator |> which can be used to pass the result of a function on the left side to the function on the right side. The ignore function that gets the result of the Diagnostics.Process.Start(imgProcessInfo) call can be used to explicitly ignore the result of a function. If the ignore function would not be used in this case a compile time warning would occur that the result of the function is not used. This prevents implicit side effects and unhandled results of function calls.

Because F# is part of the .NET ecosystem all .NET libraries can be used by it. This made finding an image library fairly easy. I chose the great SixLabors.ImageSharp library which provided all the functionalities I needed and more. Using this .NET library which is written in C# is a bit different than using native F# libraries. Functions need brackets around them and parameters are savagely separated by commas instead of spaces only. Apart from those small annoyances using the library was really easy and enabled me to quickly implement a solution for the problem at hand.

In this article I wrote a little bit about F# and using it to implement a simple program. This was the last article of the image processing experiments series. The source code for all the implementations can be found in this repository: tobiaswatzek/image-processing-experiments.

Image Processing Experiments (2) - Bash/ImageMagick

2019-06-19T00:00:00Z

TL;DR My bash version of the image processing experiments that I did in three languages is by far the shortest. Find the source code in this repository: tobiaswatzek/image-processing-experiments

The introduction to the image processing experiments can be found in my first article of the series.

Just to recall the program should do the following:

Read in an image as the first argument passed to it on the command line.
Print the file path, width and height of the image to the console.
Display the original image and a blurred version of the image.
Convert the image to grayscale and save it besides the original image as gray-IMAGENAME.

After implementing a solution in Rust I decided to look into Bash and ImageMagick which led to the following script.

#!/bin/bash
magick identify -format "Using image file: %f, width %w, height %h\n" "${1}"
magick convert "${1}" -blur 0x5 - | \
     magick montage -geometry +1+1 "${1}" - - | \
     magick display&
magick convert "${1}" -colorspace gray \
     -set filename:f "%d/gray-%f" "%[filename:f]"

The first thing in this Bash script is the so called shebang #!/bin/bash. This combination of a number sign and an exclamation mark which is followed by a path and optionally arguments is used by the operating system to determine the interpreter that should execute the script. In my case I want to execute the script with Bash and my Bash executable can be called via /bin/bash.

In my script I am executing a series of ImageMagick magick commands that use the first parameter $1 passed to the script as image path. The zero parameter $0 would return the name of the script. To access the value of the $1 parameter I use the ${1} syntax instead of just accessing the variable via $1 because it prevents errors if variables are used inside of strings.

ImageMagick provides the identify command which can extract information from an image. The -format flag can be used to customize which values are returned and how they are printed to the console.

In the third line (continued via \ for readability) the input image gets blurred, combined with the original image, and then displayed. The convert and montage commands do not write the generated image to a file but instead output the image to stdout via the - parameter. By using pipes | the output of one command is passed to the input of the next command. Another operator that is used at the end of the line is the ampersand & operator. This operator instructs the shell to fork an asynchronous sub-shell which executes the command while the parent-shell continues to execute the rest of the script. By using this operator the script is not waiting on the display command but goes on to execute the last line of the script.

By using the mighty convert command the image can be easily converted to grayscale. The -set parameter allows the creation of a template that is used to determine the output file path and name by combining the original directory %d with the original file name %f. Therefore the grayscale image will be saved besides the original image but with the filename prefixed with gray-.

In this article I wrote about some features of Bash and ImageMagick. Hopefully you found it interesting. The next article of this series will discuss my solution in F#. The source code for all the implementations can be found in this repository: tobiaswatzek/image-processing-experiments.

Image Processing Experiments (1) - Rust

2019-05-18T00:00:00Z

TL;DR I implemented an image processing experiment in Rust and was positively surprised. Find the source code in this repository: tobiaswatzek/image-processing-experiments.

In one of my courses I recently had to implement a program that satisfies the following criteria in three programming languages.

The program should do the following:

Read in an image as the first argument passed to it on the command line.
Print the file path, width and height of the image to the console.
Display the original image and a blurred version of the image.
Convert the image to grayscale and save it besides the original image as gray-IMAGENAME.

There was no constraint on the programming language that we used so i decided to implement my solution in Rust, Bash (with ImageMagick), and F#. I dabbled in each of those languages a bit before but none of them can be considered a language in which I am fluent in.

In this article I would like to discuss my takeaways from implementing the assignment in Rust.

Rust is a systems programming language that can be compared with C or C++ in that it does not run your program inside of a runtime or use a garbage collector. It can be used when performance is one of the main concerns but in opposite to C or similar languages it provides a memory management model that leads to memory and thread safety.

The installation instructions that are provided on the Rust website worked without any problems on my Laptop that runs Arch Linux. I used VS Code with the Rust (rls) extension to implement the assignment.

Rust has a package manager named Cargo that can be used to manage dependencies (in Rust slang those are called crates) of the application - much like NPM for JavaScript or Nuget for .NET. In addition to package management Cargo can be used to run and build the project.

I did not want to write the image loading and transformation code from scratch so I looked for libraries that would do that task for me. I stumbled upon the image crate which can be used to work with images in rust. In addition to the image crate I found the imageproc crate which extends the image crate with image processing functionalities. To fulfil the 3rd point of the requirements “Display the original image and a blurred version of the image.”

I wanted to use a GUI library or bindings to existing GUI libraries. I tried to use the GTK bindings for Rust but because of my lack of experience with Rust and GTK things got complicated really fast and I thought that there must be an easier way to display two images. As it is often the case the simple path was right in front of me the whole time. The imageproc crate provides the method imageproc::window::display_multiple_images which was exactly what I needed. If you think that just adding the library to the dependencies file (Cargo.toml) and calling the method would be enough you are wrong. I had to learn the hard way that crates can have feature flags that enable certain additional features of a dependency. In this case the display-window flag has to be added which will internally add a dependency to sdl2 which provides rust bindings for Simple Directmedia Layer. Going down this rabbit hole has shown me once again how important it is to be able to read other people’s code or I would never have found the definition of the features inside of the source code of the imageproc crate.

After I had all the libraries and a working development environment it was surprisingly easy to implement the task. Two things that seem to be common in Rust were Option types returned by functions and macros like println!.

The Option type can be seen as an alternative to returning null. It can either have a value Some or it can have no value None. I recognized the construct from functional languages like F# or Haskell. The great thing about option types is that they make it explicit that a function could return nothing aka null and that those cases should be handled or else you end up in the WhoopsieThereIsNothingException club (aka NullPointerException club or NullReferenceException club).

Macros in Rust can be compared to function templates that will be replaced with a piece of code at compile time. Therefore the println! macro can actually parse its format string and error if it is not given enough parameters.

This is the first article I wrote and I hope it contained something useful for you. I will follow up with articles about the two other implementations in Bash and F#. The source code for all the implementations can be found in this repository: tobiaswatzek/image-processing-experiments.