Colin the Geek

Stylish Markdown

2018-08-07T00:00:00+00:00

I love Markdown… I mean, who doesn’t? I use it for everything: wiki pages, release notes, API documentation, deployment guides, to do lists for my wife. Ok maybe not that last one, I’m usually the recipient of the to do lists!

When distributing documentation as part of a release, we tend to convert the Markdown into HTML with hyperlinks, logos, fonts and colour schemes; then use wkhtmltopdf to convert it into a PDF.

Recently I’ve been trialing a different approach - wrapping the Markdown inside a thin layer of HTML, then converting it into HTML on the fly using JavaScript. The advantage is that the document remains diff-able, so the recipient can compare the current document with a previous version by using standard text-based differencing tools on the HTML source code (which is mainly Markdown).

This approach is really useful for things like API specifications, were every small change needs to be carefully scrutinised. With PDF documents you usually only have a few short entries in a version table (if you’re lucky).

See the Pen Stylish Markdown by Colin The Geek (@colinthegeek) on CodePen.

The Markdown is nested inside a script tag with a custom text/markdown type. Keeping it inside a script tag is important, it means the browser won’t try to HTML encode any special characters in the Markdown. The JavaScript is pretty simple, it finds all of the Markdown scripts, uses Showdown to convert them into HTML, then places them into the same location in the document. I’ve added an unindent function that trims out any leading white space so the Markdown can be indented properly within the HTML.

Then you can apply your own CSS to give your document a bit of style! In my example I’m using some GitHub CSS maintained by Sindre Sorhus, looks the part!

Time Travel DSL

2017-11-25T00:00:00+00:00

I’ve designed a little domain specific language in C# to manage changes to the system clock. The objective was to simulate a sequence of transactions running against my domain model with realistic time intervals between each transaction.

An example use case would be building an order history:

using (TimeTravel.To(2.MonthsAgo()))
{
    var order = Order.Create();

    using (TimeTravel.Forward(45.To(60).Minutes()))
    {
        order.Submit();

        using (TimeTravel.Forward(4.To(6).Days() + 2.To(8).Hours()))
        {
            order.Dispatch();
        }
    }

    Console.WriteLine(order.ReportHistory());
}

// Outputs:
//   Created - 25/09/2017 11:33
//   Submitted - 25/09/2017 12:25
//   Dispatched - 30/09/2017 17:25

A well designed DSL should make the code easier to understand and reason about. Although C# isn’t the most expressive language on the block, it still has some nice features that can be used to design an internal DSL.

Encapsulating DateTime.Now

The first step to achieve time travel (other than finding a flux capacitor) is to encapsulate the standard .NET system clock so you can offset the current time:

public static class SystemTime
{
    public static TimeSpan Offset;
    public static DateTime Now { get { return DateTime.Now + Offset; } }
    public static DateTime Today { get { return SystemTime.Now.Date; } }
}

TimeSpan is a standard .NET struct that represents a time interval in days, hours, minutes and seconds. They can be added or subtracted from a date and time, I’m using one above to offset DateTime.Now.

Now you can refactor all of your domain logic to use SystemTime.Now and SystemTime.Today instead of DateTime.Now and DateTime.Today. During normal operations the offset will be zero so these properties will return the current date and time.

TimeSpan extension methods

The next step is to use extension methods to build a succinct language for expressing time spans:

// for example: 45.Minutes()
public static TimeSpan Minutes(this int value)
{
    return TimeSpan.FromMinutes(value);
}

// for example: 8.Hours()
public static TimeSpan Hours(this int value)
{
    return TimeSpan.FromHours(value);
}

// for example: 4.Days()
public static TimeSpan Days(this int value)
{
    return TimeSpan.FromDays(value);
}

The nice thing about time spans is that they can be added together to allow us to compose more complex expressions using our DSL:

4.Days() + 6.Hours() + 45.Minutes()

In the order history example above, I use another extension to simulate random, but realistic, time intervals like 45.To(60).Minutes():

public static int To(this int start, int end)
{
    var range  = Enumerable.Range(start, end - start);
    var random = new Random();
    return range.ElementAt(random.Next(range.Count()));
}

Chaining this with the TimeSpan extension methods makes our time interval code read like a normal English sentence… if you ignore all the full stops and brackets.

My Generating Sample Data article has more tips for generating random data.

Scoping the time travel

We can make use of the using statement to provide a scope for our time travel, ensuring we don’t get stuck in the past. Nesting the using statements provides a convenient way do relative jumps through time:

// SystemTime.Now = 12:00

using (TimeTravel.Forward(5.Minutes))
{
    // SystemTime.Now = 12:05

    using (TimeTravel.Back(3.Minutes))
    {
        // SystemTime.Now = 12:02
    }

    // SystemTime.Now = 12:05
}

// SystemTime.Now = 12:00

The trick here is to write a class that implements IDisposable. The constructor captures the current SystemTime.Offset before any changes are made… then when the class is disposed (as it exits the using block) it restores the original offset.

public class TimeTravel : IDisposable
{
    private readonly TimeSpan previousOffset;

    public TimeTravel()
    {
        previousOffset = SystemTime.Offset;
    }

    public void Dispose()
    {
        SystemTime.Offset = previousOffset;
    }
}

The last piece of the puzzle is a few static methods on our TimeTravel class to adjust the offset:

// for example: TimeTravel.Forward(5.Minutes())
public static TimeTravel Forward(TimeSpan offset)
{
    var token = new TimeTravel();
    SystemTime.Offset += offset;
    return token;
}

// for example: TimeTravel.Back(5.Days())
public static TimeTravel Back(TimeSpan offset)
{
    var token = new TimeTravel();
    SystemTime.Offset -= offset;
    return token;
}

You can find a full copy of the code used in this article on this gist.

Using SQL LIKE with Entity Framework

2017-10-18T00:00:00+00:00

I found out this week that using the LIKE operator in an Entity Framework query is maddeningly difficult 😳. I reckon there are three options, each with different trade-offs and levels of difficulty.

Option 1: Entity SQL

According to Microsoft the Entity SQL Language is a storage-independent query language that is similar to SQL. It allows you to write queries like this:

var query = context.PersonQuery.Where(
    "it.FirstName LIKE @pattern",
    new ObjectParameter("pattern", "Col%")
);

where it represents the entity being queried.

Unfortunately this won’t work with your normal Entity Framework datasets, instead you have to expose an additional ObjectSet for each type of entity you wish to query:

public class DataContext : DbContext
{
    // Normal sets used in EF code-first
    public DbSet<Person> People { get; set; }

    // Special Entity SQL object sets
    public ObjectContext ObjectContext   => ((IObjectContextAdapter)this).ObjectContext;
    public ObjectSet<Person> PersonQuery => ObjectContext.CreateObjectSet<Person>();
}

Advantages:
✔ Really easy to setup
✔ Works with all databases supported by EF

Disadvantages:
✘ No in-memory version for unit testing
✘ Cannot be mixed with standard LINQ to make more complex queries
✘ Uses magic strings that won’t survive a refactor and aren’t type checked
✘ Has a rather verbose syntax for passing parameters

Option 2: SQL Functions

Entity Framework ships with a set of SQL Functions that can be used in LINQ queries. When Entity Framework executes the query, it will use the database function with the matching name.

As there’s no LIKE function available, we have to use PATINDEX instead which performs a pattern match and returns the position of the first occurrence:

var query =
    from person in context.People
    where SqlFunctions.PatIndex("C%", person.FirstName) > 0
    select person;

Advantages:
✔ No setup required
✔ Standard LINQ so it can be mixed into more complex queries
✔ Strongly typed and refactor safe

Disadvantages:
✘ No in-memory version for unit testing, it throws a not supported exception
✘ Only works with SQL Server

Option 3: Write a Custom SQL Function

Neither of the options above really float my boat. The lack of support for in-memory sequences makes it impossible to unit test complex queries without using the database… I’d like to be able to write a query like this:

var query =
    from person in context.People
    where person.FirstName.Like("Col%")
    select person;

First we have to implement an in-memory version of the Like function to support unit testing. It’s reasonably easy to do this using regular expressions:

public static class Functions
{
    [DbFunction("CodeFirstDatabaseSchema", "Like")]
    public static bool Like(this string target, string pattern)
    {
        // Escape all the special regex characters by default
        pattern = Regex.Escape(pattern);

        // Add regex equivalents for the various SQL LIKE characters
        pattern = pattern.Replace("%", ".+");
        pattern = pattern.Replace("_", ".");
        pattern = pattern.Replace(@"\[", "[");
        pattern = pattern.Replace(@"\]", "]");
        pattern = pattern.Replace(@"[\^", "[^");

        // Match against the entire string
        pattern = "^" + pattern + "$";

        return Regex.IsMatch(target, pattern, RegexOptions.IgnoreCase);
    }
}

The [DbFunction] attribute tells Entity Framework to map this method to a database function called “Like” when transposed into a SQL query… so lets add this function to the database:

CREATE FUNCTION [dbo].[Like]
(
    @target nvarchar(max),
    @pattern nvarchar(max)
)
RETURNS BIT
AS
BEGIN
    RETURN (SELECT CASE WHEN @target LIKE @pattern THEN 1 ELSE 0 END)
END

In an ideal world that would be it, however Entity Framework needs a bit more metadata to map this SQL function to the C# one. There are third party libraries that can add this metadata automatically, like this one by moozzyk, but they don’t support extension methods.

Here’s how to do it explicitly for our Like function (brace yourself):

public class FunctionsConvention : IStoreModelConvention<EdmModel>
{
    public void Apply(EdmModel item, DbModel model)
    {
        var payload = new EdmFunctionPayload
        {
            Schema = "dbo",
            StoreFunctionName = "Like",
            Parameters = new[]
            {
                FunctionParameter.Create("target" , getPrimitiveType(model, PrimitiveTypeKind.String), ParameterMode.In),
                FunctionParameter.Create("pattern", getPrimitiveType(model, PrimitiveTypeKind.String), ParameterMode.In),
            },
            ReturnParameters = new[]
            {
                FunctionParameter.Create("result", getPrimitiveType(model, PrimitiveTypeKind.Boolean), ParameterMode.ReturnValue)
            },
        };

        item.AddItem(EdmFunction.Create("Like", "CodeFirstDatabaseSchema", item.DataSpace, payload, null));
    }

    private static EdmType getPrimitiveType(DbModel model, PrimitiveTypeKind typeKind)
    {
        return model
            .ProviderManifest
            .GetStoreType(TypeUsage.CreateDefaultTypeUsage(PrimitiveType.GetEdmPrimitiveType(typeKind)))
            .EdmType;
    }
}

Finally this convention needs to be registered in your data context:

public class DataContext : DbContext
{
    public DbSet<Person> People { get; set; }

    protected override void OnModelCreating(DbModelBuilder modelBuilder)
    {
        modelBuilder.Conventions.Add(new FunctionsConvention());
    }
}

Advantages:
✔ In-memory version for unit testing or filtering results after the query executes
✔ Standard LINQ so it can be mixed into more complex queries
✔ Strongly typed and refactor safe
✔ Works with all databases supported by EF

Disadvantages:
✘ A large amount of up-front setup required

Conclusion

SQL Functions (option 2) is probably the quickest and easiest one to get up and running, but if the lack of in-memory support is a deal breaker then writing a custom SQL function may be worth the extra effort.

As a bonus, this approach could easily be adapted to unlock other SQL features like soundex or geospatial queries.

Generating Sample Data

2017-02-01T00:00:00+00:00

Recently I’ve been thinking a lot about manual exploratory testing and performance testing, both of which require large sets of realistic test data.

On most projects we can’t use real user data for testing (due to data security rules) so instead I’ve come up with a few simple techniques to generate large sets of random, but realistic, sample data.

Selecting a random item from a sequence

The following extension method returns a random item from a sequence:

public static T Random<T>(this IEnumerable<T> source, int? seed = null)
{
    var random = new Random();
    if (seed != null) random = new Random(seed.Value * source.Count());
    return source.ElementAt(random.Next(source.Count()));
}

…and here’s how it can be used in practice:

var names = new[] { "Homer", "Marge", "Bart", "Lisa", "Maggie" };

for (var i = 0; i < 5; i++)
{
    var name = names.Random(i);
}

// Generates: Lisa, Marge, Maggie, Bart, Homer

Notice that the method has an optional parameter for a seed - providing this parameter makes the selection pseudo-random, i.e. it always picks the same random item each time it’s called (provided the underlying sequence isn’t changed). This can be quite useful when generating test data as you’ll get the same sample data every time you tear down and rebuild the database.

Of course in the example above you would need a much larger pool of sample names than our favourite cartoon family, how about names from the 1990 US census?

A little multiline editing in your favourite editor and you end up with this:

public static class SampleData
{
    public static string[] FemaleNames = new[]
    {
        "Mary",
        "Patricia",
        "Linda",
        "Barbara",
        "Elizabeth",
        "Jennifer",
        "Maria",
        ...
    }

    public static string[] Surnames = new[]
    {
        "Smith",
        "Johnson",
        "Williams",
        "Jones",
        "Brown",
        "Davis",
        "Miller",
        ...
    }
}

…with the usage:

var person = new Person
{
    FirstName = SampleData.FemaleNames.Random(i),
    LastName  = SampleData.Surnames.Random(i),
};

Weighting values

Sometimes your random data can be a little too random and to make your data realistic you need to make some values occur more frequently, that’s where the Weight extension comes in:

public static IEnumerable<T> Weight<T>(this IEnumerable<T> source, int weight)
{
    IEnumerable<T> result = new T[0];

    for (int i = 0; i < weight; i++)
    {
        result = result.Concat(source);
    }

    return result;
}

var titles =
    new[] { "Mrs", "Miss", "Ms" }.Weight(5)
    .Concat(new[] { "Dr", "Prof", "Rev" });

for (var i = 0; i < 5; i++)
{
    var title = titles.Random(i);
}

// Generates: Miss, Ms, Mrs, Prof, Mrs

So “Mrs”, “Miss” and “Ms” are five times more likely to occur in the sample data than the other professional titles.

Ranges

Unlike Ruby, C# has no built in support for ranges of values. So here’s an extension to add this support:

public static IEnumerable<int> To(this int start, int end)
{
    return Enumerable.Range(start, end - start);
}

This can be used in conjunction with the random method to generate numeric data like phone numbers:

for (var i = 0; i < 5; i++)
{
    var phone = "028 9012 " + 1000.To(9999).Random(i);
}

// Generates:
//   028 9012 7535
//   028 9012 1284
//   028 9012 4033
//   028 9012 6781
//   028 9012 9529

Happy testing 👓

Dockerizing my Development Process

2016-06-07T00:00:00+00:00

I’m a big fan of using Vagrant for development. I really like the approach of installing the framework, database and development tools on a project-specific virtual machine that can be shared with the other folks on the team.

But Docker is the shiny new toy that promises development / production parity, allowing infrastructure to be scripted alongside the code and run in these light-weight container thingy-me-bobs.

Does Docker live up to its promises? Is it time to evict the Vagrant? I’ve developed a simple web app to find out…

The Basics - describes the installation process and explains the basics of running Docker commands.
Introducing Quotster - introduces the sample application.
Getting Started - running an interactive terminal in Docker to install npm packages and run a simple web server.
Prototyping - using Docker Compose to run the web server, and a custom Docker image that restarts it when the code changes.
Adding the Database - links in a MongoDB container to store the data.
Scaling Up - adds a load balancer and scales to multiple instances of the web container.
Releasing to Production - using Docker Machine to create a Digital Ocean droplet to run the application in the cloud.

The Basics

The first task is to install the Docker Toolbox. If you’re using OSX or Windows, this will create a Linux VM (using VirtualBox) that runs Docker. It also installs a command-line tool called Docker Machine which forwards docker commands from your terminal to the VM.

But wait… weren’t we supposed to be getting rid of VMs? Well at the moment Docker is only available on Linux, and even when it arrives for Windows / Mac you won’t be able to run a Windows container on a Linux base machine (or vice versa). Since I’m ultimately deploying my application to a Linux server, I can live with the cost of a single Docker VM on my laptop.

Once you have Docker installed you can open the “Docker Quickstart Terminal” and run the following command:

$ docker run ubuntu echo Hello World

Hello World

…where ubuntu is an image and echo Hello World is a command.

Images are virtual file systems that have software installed on them, in this case it’s Ubuntu. If Docker cannot find the image locally, it will download it from the Docker Hub where you can find thousands of pre-defined images.

Docker creates a container, i.e. an instance of the image, and runs the command inside it. Once the command completes the container is spent. Although you can restart a container, it’s better to think of them as disposable runtimes for your commands.

You can list the spent containers:

$ docker ps -a

CONTAINER ID        IMAGE               COMMAND              CREATED             STATUS                     PORTS               NAMES
49df8a0b5a76        ubuntu              "echo Hello World"   2 minutes ago       Exited (0) 2 seconds ago                       stoic_curie

…and clean them up by using the first few characters of their ID:

$ docker rm 49d

You can also view and manage your local images by using the docker images command.

Introducing Quotster

My sample app is a quote wall where you can record insightful, life-changing nuggets of information; or mildly amusing nonsense that you find on Twitter:

It’s a Node web server backed by a MongoDB database. The web application can be scaled out behind a round-robin load balancer called HAProxy.

Getting Started

The first task is to get a simple web server running. Normally I would create a new Vagrant machine, connect to it using SSH, and install the version of Node that I want for the project. Then I would use npm from the Vagrant machine to install a web server package into my project folder (which is shared automatically between the VM and host).

The good news is that you can do the same thing using Docker… with a few magic switches:

$ docker run -i -t -v $(PWD):/code -w /code -p 3000:3000 node bash

Let’s start on the right-hand side, I’m telling Docker that I want to run the command bash on a container. The image for the container is the official node image from Docker Hub.

The -i -t switches tell Docker to run an interactive shell, without these the container will run bash and exit immediately. -v $(PWD):/code will mount the current directory into the container and -w /code sets it as the working directory.

Finally -p 3000:3000 forwards port 3000 on the container to the host. We’ll use this later to test the web application from a browser.

Now we can use npm from inside the container to create a package.json and install the Koa web server package:

$ npm init

name: quotster
version: 1.0.0
description: A sample web application to try out Docker.
entry point: server.js
test command:
git repository:
keywords:
author: Colin Orr
license: MIT

$ npm install koa --save

Note: I’ve changed the entry point from the default index.js to server.js.

Here’s the code for a simple “Hello World” web app, use your favourite text editor to save it to server.js in the root of your project:

// server.js

const koa = require('koa');
const app = koa();

app.use(function *() {
  this.body = 'Hello World';
});

app.listen(3000, () => console.log('server listening on port 3000'));

Start your web server by running the following command from your Docker terminal:

$ npm start

npm info it worked if it ends with ok
npm info using npm@3.8.9
npm info using node@v6.2.0
npm info lifecycle quotster@1.0.0~prestart: quotster@1.0.0
npm info lifecycle quotster@1.0.0~start: quotster@1.0.0

> quotster@1.0.0 start /code
> node server.js

server listening on port 3000

Now you can visit the website on http://192.168.99.100:3000.

Prototyping

The next stage is to do some prototyping, installing more npm packages and scaffolding out the most of the application using static data. Repeatedly typing that docker command with all of those switches is going to get tiresome really quickly!

There’s another tool called docker-compose which allows you to move these switches into a configuration file:

# docker-compose.yml

version: '2'
services:
  web:
    image: node
    working_dir: /code
    command: npm start
    volumes:
      - .:/code
    ports:
      - 3000:3000

Save this file as docker-compose.yml in the root of the project folder and launch your container by running the following command:

$ docker-compose up

This will start the container and run the web server. You can connect to the container and run interactive commands by opening a second Docker Quickstart Terminal and running:

$ docker-compose exec web bash

…where web is the service name from the yaml file, and bash is the command to run.

From here we can install some more packages:

$ npm install koa-logger --save
$ npm install koa-route --save
$ npm install koa-body --save
$ npm install koa-pug --save

…and enhance our application to include logging, routing and rendering HTML using Pug (formally called Jade):

// server.js

const koa    = require('koa'),
      logger = require('koa-logger'),
      route  = require('koa-route'),
      body   = require('koa-body'),
      Pug    = require('koa-pug');

const app = koa();
const pug = new Pug();

app.use(logger());
app.use(body());
pug.use(app);

let quotes = [
  { id: 1, quote: 'You can do anything, but not everything.', author: 'David Allen' },
  { id: 2, quote: 'Always forgive your enemies; nothing annoys them so much.', author: 'Oscar Wilde' },
  { id: 3, quote: 'Choose a job you love, and you will never have to work a day in your life.', author: 'Confucius' }
];

app.use(route.get('/', function *() {
  this.render('client', quotes);
}));

app.listen(3000, () => console.log('server listening on port 3000'));

Note: if you’re following along at home, you’ll need to grab a copy of my HTML template and save it to client.pug.

One annoyance is having to stop and start the web server each time we make a change to the source code. To avoid this, we could use nodemon to watch for changes and restart the server.

Nodemon in an npm package that has to be installed globally. Although we could do this on the running container, it would be lost when the container is stopped and discarded. It’s better to add it to the image, and that means writing a custom Dockerfile:

# Dockerfile

FROM node
RUN npm install -g nodemon

Now we can change our docker-compose.yml to build this image and execute the server using nodemon:

version: '2'
services:
  web:
    build: .
    working_dir: /code
    command: nodemon -L
    volumes:
      - .:/code
    ports:
      - 3000:3000

Apply these changes by running the following commands on your host:

$ docker-compose down
$ docker-compose up

Docker Compose will build your custom image and use it instead of the official node image. Now you’ll find that the server restarts automatically as you save the source code.

Adding the Database

Docker Compose can manage multiple containers for a project, creating a private network between them so that they can communicate. Let’s add the official mongo image into the mix to save our data:

# docker-compose.yml

version: '2'
services:
  db:
    image: mongo
  web:
    build: .
    working_dir: /code
    command: nodemon -L
    volumes:
      - .:/code
    ports:
      - 3000:3000
    depends_on:
      - db

So we’ve configured MongoDB to run under a service called db. I’ve added a depends_on clause to our web application, this helps Docker Compose decide the order in which to load the containers.

Services can be referenced by other containers by using their name, db in this case. Take a look at the final version of the server and you’ll see the database URL at the bottom:

// server.js

...

mongoose
  .connect('mongodb://db/quotster')
  .then(() => console.log('database connection established'))
  .then(() => app.listen(3000, () => console.log('server listening on port 3000')));

Docker sets up a private network between containers so your database never needs to be exposed externally. Another advantage - no more configurable connection strings that change between deployments 😃.

You can still access the data directly by starting an interactive session on the database container, then query the database using the mongo command-line client:

$ docker-compose exec db bash
$ mongo

> use quotster
switched to db quotster
> db.quotes.find()
{ "_id" : ObjectId("57518fa80331e31400751728"), "quote" : "You can do anything, but not everything.", "author" : "David Allen", "date" : ISODate("2016-06-03T14:09:44.483Z"), "__v" : 0 }
{ "_id" : ObjectId("57518fc70331e31400751729"), "quote" : "Always forgive your enemies; nothing annoys them so much.", "author" : "Oscar Wilde", "date" : ISODate("2016-06-03T14:10:15.869Z"), "__v" : 0 }
{ "_id" : ObjectId("57518fdb0331e3140075172a"), "quote" : "Choose a job you love, and you will never have to work a day in your life.", "author" : "Confucius", "date" : ISODate("2016-06-03T14:10:35.696Z"), "__v" : 0 }

Note: the mongo image has different options to control where the container stores the physical data.

Scaling Up

Scaling and load balancing is something I’ve never attempted on my development rig, but Docker makes it easy!

Lets introduce the haproxy load balancer from Docker Hub:

# docker-compose.yml

version: '2'
services:
  db:
    image: mongo
  web:
    build: .
    working_dir: /code
    command: nodemon -L
    volumes:
      - .:/code
    depends_on:
      - db
    expose:
      - 3000
  lb:
    image: dockercloud/haproxy
    links:
      - web
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - 80:80

I’ve removed the port forwarding from the web application, instead I’m using expose to state the port that it’s using. The load balancer uses links to identify the services to balance, and port 80 is exposed to the outside world.

When you do a docker-compose up you can see the load balancer has been added to the mix from the server logs, and you can visit the site on http://192.168.99.100.

In a separate terminal, scale up the web instances with the following command:

$ docker-compose scale web=3

Refresh the homepage a few times and you’ll see the load balancer in action from the server logs:

web_1  |   <-- GET /
web_1  |   --> GET / 200 7ms 1.03kb
web_3  |   <-- GET /
web_3  |   --> GET / 200 213ms 1.03kb
web_2  |   <-- GET /
web_2  |   --> GET / 200 206ms 1.03kb
web_1  |   <-- GET /
web_1  |   --> GET / 200 5ms 1.03kb

Running docker-compose ps will show the containers and port mappings:

$ docker-compose ps

Name                  Command             State                   Ports                 
--------------------------------------------------------------------------------------------
quotster_db_1    /entrypoint.sh mongod         Up      27017/tcp                             
quotster_lb_1    tini -- dockercloud-haproxy   Up      1936/tcp, 443/tcp, 0.0.0.0:80->80/tcp
quotster_web_1   nodemon -L                    Up      3000/tcp                              
quotster_web_2   nodemon -L                    Up      3000/tcp                              
quotster_web_3   nodemon -L                    Up      3000/tcp                              

Running docker-compose stop will bring down all the containers in one go.

Releasing to Production

The final test for Docker is to take my micro server farm of containers and run it on a Digital Ocean droplet in the cloud.

My web application is currently running the code from a shared volume on my laptop, when I push to the cloud this won’t be available so I need to tweak the Dockerfile to build the code into the image:

# Dockerfile

FROM node
RUN npm install -g nodemon
COPY . /code

Now when I call docker-compose build the resulting image will include the Quotster source code, but docker-compose.yml still has the configuration which overrides this folder with the shared volume. That’s OK for development, but we don’t want this to happen for production.

To solve this problem, Docker Compose has a nice feature that allows you to split you configuration file into a base file with overrides for different environments:

# docker-compose.yml

version: '2'
services:
  db:
    image: mongo
  web:
    build: .
    working_dir: /code
    depends_on:
      - db
    expose:
      - 3000
  lb:
    image: dockercloud/haproxy
    links:
      - web
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - 80:80

# docker-compose.override.yml

version: '2'
services:
  web:
    command: nodemon -L
    volumes:
      - .:/code

# docker-compose.prod.yml

version: '2'
services:
  db:
    restart: always
  web:
    command: npm start
    restart: always
  lb:
    restart: always

So the common configuration remains in docker-compose.yml, while docker-compose.override.yml contains the development specific tweaks and docker-compose.prod.yml contains the production tweaks.

I’ve added a restart: always setting to the production configuration, this instructs Docker to restart the container automatically if it crashes, or if the server running the containers is restarted.

You can continue to use docker-compose up for development, if you want to run with the production configuration you run the following command:

$ docker-compose -f docker-compose.yml -f docker-compose.prod.yml up

We’re ready to push to Digital Ocean. You’ll need to setup an account and generate a personal access token (instructions). Once you have your token, you can run the following command:

$ docker-machine create --driver digitalocean --digitalocean-access-token <YOUR TOKEN> --digitalocean-region lon1 digitalocean

This will create a new $5 a month droplet on Digital Ocean with Docker installed and ready to use.

You can connect to the droplet using Docker Machine:

$ eval $(docker-machine env digitalocean)
$ docker-machine active
digitalocean

This switches the current terminal session to forward all Docker commands to digitalocean rather than your local Docker VM. So interacting with Docker in the cloud is exactly the same experience as working with it locally.

Now we can spin up our application:

$ docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d

Note: the -d switch runs docker in ‘detached’ mode - it will run in the background and you can close the terminal session without stopping the containers.

Finally, scale up to 3 instances of the web containers:

 $ docker-compose -f docker-compose.yml -f docker-compose.prod.yml scale web=3

You should be able to hit the site from the browser. You can find the public IP address from the Digital Ocean dashboard.

Conclusion

Docker containers do feel a lot faster and more light-weight than Vagrant VMs, but they are rather arduous to manage from the command line.

Docker Compose solves this problem by allowing configuration to be stored in a separate file. It does an excellent job of abstracting the more complicated aspects of Docker like creating private networks and coordinating multiple containers.

Docker Machine is the icing on the cake, allowing you to switch your terminal between development / staging / production servers and run the same commands across the board.

This trio of tools give Docker a compelling story for development, it’s definitely worth trialling on your next project!

Regex Find & Replace

2016-04-03T00:00:00+00:00

I repeatedly forget and rediscover this powerful technique for finding and replacing text.

I wanted to change the name of a CSS class across a large codebase. A simple approach would be to find and replace for all *.html files:

Find:    class="selected"
Replace: class="active"

But this doesn’t cope with elements with multiple CSS classes:

class="selected"
class="btn selected"
class="selected link"
class="selected-item"    « this one needs to be left alone
class="item-selected"    « as does this one

The solution was to use regular expression find and replace (this feature is available in most editors, usually enabled by toggling the .* button). Regex capture groups can be added to match the other CSS classes:

Find:    class="(.+\s)?selected(\s|")
Replace: class="$1active$2

Strings captured by the regular expressions be added back into the replacement text by using the dollar - $1 and $2 in the example above. If you want to brush up on your regular expressions, check out the Regex Crossword it’s like Sudoku for uber geeks!

Introducing Penfold

2016-03-22T00:00:00+00:00

I’ve convinced myself that .NET needs a new behaviour driven development library. Here are my simple demands:

Specification statements that can be written with spaces, punctuation and, most importantly, emojis 😜.
Nested contexts that can group assertions into logical collections and share variables, setup code and scope.
Support for different styles of specification, or even the ability to roll your own specification DSL.
The option to enhance the textual output of the specification with extra information and comments.

Now I know that RSpec, Jasmine and Mocha tick all of these boxes, but I haven’t found a satisfactory alternative for .NET. So I’ve had a crack at writing one - It’s called Penfold and is available now on Nuget.

[TestFixture]
public class CalculatorSpecification : Specification
{
    public CalculatorSpecification()
    {
        var calculator = new Calculator();

        before_each = () => calculator.Clear();

        describe["Addition"] = () =>
        {
            context["adding two and three"] = () =>
            {
                before = () => calculator.Key(2).Add(3);

                it["sets the total to five"] = () =>
                {
                    calculator.Total.ShouldEqual(5);
                };

                it["sets the equation history to:"] = () =>
                {
                    log(calculator.Equation.ShouldEqual("2 + 3"));
                };
            };

            context["adding two, three and four"] = () =>
            {
                before = () => calculator.Key(2).Add(3).Add(4);

                it["sets the total to nine"] = () =>
                {
                    calculator.Total.ShouldEqual(9);
                };

                it["sets the equation history to:"] = () =>
                {
                    log(calculator.Equation.ShouldEqual("2 + 3 + 4"));
                };
            };
        };
    }
}

which outputs the following to the console:

CalculatorSpecification
  Addition
    adding two and three
      sets the total to five
      sets the equation history to:
        2 + 3
    adding two, three and four
      sets the total to nine
      sets the equation history to:
        2 + 3 + 4

Note: if you check out my calculator implementation, you’ll see that I’m not expecting to be head-hunted by Casio any time soon!

The Basics

Penfold specifications inherit from a Specification base class and are written in the default constructor. Each specification must have at least one assertion to be executed, so the bare minimum would be:

[TestFixture]
public class CalculatorSpecification : Specification
{
    public CalculatorSpecification()
    {
        it["can add two numbers together"] = () =>
        {
            new Calculator().Key(1).Add(2).Total.ShouldEqual(3);
        };
    }
}

You’ll probably recognize the [TestFixture] annotation - that’s because Penfold is built on top of good old NUnit so in theory you can use any test runner that supports NUnit^[1].

Penfold doesn’t include an assertion library, all of the code in this article uses my favourite - Machine.Specifications.Should, but you can use any assertion framework you like.

Assertions can be grouped together inside a context where they share variables and setup code. Contexts are defined by using the describe and context keywords:

describe["Addition"] = () =>
{
    var calculator = new Calculator();

    context["adding two and three"] = () =>
    {
        before = () => calculator.Key(2).Add(3);

        it["sets the total to five"] = () =>
        {
            calculator.Total.ShouldEqual(5);
        };

        it["sets the equation history to '2 + 3'"] = () =>
        {
            calculator.Equation.ShouldEqual("2 + 3");
        };
    };

    context["adding two, three and four"] = () =>
    {
        ...

The calculator variable is scoped to the “Addition” context, it can be accessed from within the nested test steps but is not available outside of the describe block.

Gherkin Syntax

Penfold also supports Gherkin syntax:

[TestFixture]
public class CalculatorFeature : Specification
{
    public CalculatorFeature()
    {
        var calculator = new Calculator();

        comment = @"
            as a math idiot
            I want to use a calculator
            so I don't make mistakes with simple arithmetic
        ";

        Scenario["Addition"] = () =>
        {
            Given["I have pressed clear"]    = () => calculator.Clear();
            When["I key in two"]             = () => calculator.Key(2);
            When["I add three"]              = () => calculator.Add(3);
            Then["the total is five"]        = () => calculator.Total.ShouldEqual(5);
            Then["the equation history is:"] = () => log(calculator.Equation.ShouldEqual("2 + 3"));
        };

        Scenario["Division"] = () =>
        {
            Given["I have pressed clear"]    = () => calculator.Clear();
            When["I key in twelve"]          = () => calculator.Key(12);
            When["I divide by four"]         = () => calculator.Divide(4);
            Then["the total is three"]       = () => calculator.Total.ShouldEqual(3);
            Then["the equation history is:"] = () => log(calculator.Equation.ShouldEqual("12 / 4"));
        };
    }
}

which outputs the following to the console:

CalculatorFeature
  as a math idiot
  I want to use a calculator
  so I don't make mistakes with simple arithmetic
  Scenario: Addition
    Given I have pressed clear
    When I key in two
    When I add three
    Then the total is five
    Then the equation history is:
      2 + 3
  Scenario: Division
    Given I have pressed clear
    When I key in twelve
    When I divide by four
    Then the total is three
    Then the equation history is:
      12 / 4

In the example above I’ve used inline lambdas for the step definitions to aid readability, but you can still use block lambdas for multiple lines of code.

Penfold actually makes it really easy to specify your own keywords, but that’s a blog for a different day. If you’re interested, checkout the Specification.cs source code to see how the standard keywords are defined.

Logging

Sometimes you end up repeating yourself between the specification statement and the code. For example:

it["sets the equation history to '2 + 3'"] = () =>
{
    calculator.Equation.ShouldEqual("2 + 3");
};

The log method allows you to enrich the specification output from within the code, so the example above could be rewritten as:

it["sets the equation history to:"] = () =>
{
    var expected = "2 + 3";
    log(expected);
    calculator.Equation.ShouldEqual(expected);
};

which results in the following output:

sets the equation history to:
  2 + 3

One of the cool features of Machine.Specifications.Should is that the ShouldEqual extension returns the expected string, so we can rewrite this as a one-liner:

it["sets the equation history to:"] = () =>
{
    log(calculator.Equation.ShouldEqual("2 + 3"));
};

Ignored, Pending & Categorization

You can ignore a step by using the x or @ignore keywords. Individual steps can be ignored, or you can ignore an entire context.

x = it["I'm being ignored"] = () =>
{
    ...
};

@ignore = it["so am I!"] = () =>
{
    ...
};

Steps which haven’t been implemented can be marked as pending by setting their action to null:

it["I'm not implemented yet"] = null;

This flags the step as inconclusive to the test runner and outputs:

I'm not implemented yet [PENDING]

Finally, categories can be added by using the following syntax:

@_["Category A"] =
@_["Category B"] =
it["does something"] = () =>
{
    ...
};

Categories can be applied to individual assertions, or to an entire context.

Expecting Exceptions

You can use the Catch method as an easy way to catch an exception and return it:

context["dividing by zero"] = () =>
{
    it["explodes 💣💥☠️"] = () =>
    {
        Catch(() => calculator.Key(2).Divide(0))
            .ShouldBeOfExactType<DivideByZeroException>();
    };
};

I told you there’d be emojis!

Summary

This started out as a bit of an experiment, but I’m quite happy with the result. I plan to try it on a couple of side projects and see how it goes.

One minor niggle is that Visual Studio doesn’t support code folding on the lambda expressions. Once a specification gets long enough, it’s useful to be able to fold away certain contexts that you aren’t interested in. I’ve found the C# outline extension which sorts this out.

If you decide to use Penfold on your own project, please give the repo a star on GitHub so that I know it’s being used.

Oh and why call it Penfold? Because unlike Danger Mouse, he needs specs!

^[1] I’ve had some difficulty with TestDriven.net, it can’t identify the tests because they are specified inside the class constructor. I’ve developed an adapter that allows the specification to be executed, but it can only execute the whole specification and not individual assertions or contexts.

Visual Studio Code Snippets

2016-01-21T00:00:00+00:00

Over the years I’ve developed a number of snippets to help me write unit tests using the MSpec framework. After numerous requests from envious co-workers (actually one request from a mildly curious co-worker), I’ve finally open sourced them on Github:

https://github.com/ColinOrr/vstudio-snippets

Installation

Run the following commands to clone the snippets from my Github repository:

cd "Documents\Visual Studio 2013\Code Snippets\Visual C#"
git clone https://github.com/ColinOrr/vstudio-snippets "My Code Snippets"

Visual Studio watches this folder and automatically loads the snippets, so no need for a restart!

test

The built-in test snippet generates an MSTest method (yuck!). Since we use MSpec, I’ve replaced this snippet with an MSpec test container.

lam

MSpec uses lambdas to link specification statements with real code. I find them awkward to type, especially since Visual Studio formatting goes haywire and messes up the indentation. The lam snippet fixes the indentation and drops you straight into the lambda body, ready to start writing test code.

argnull

For production code, you can use the argnull snippet to speed up your argument guards. This snippet uses the same parameter for both the guard and the error message, so you only have to type the argument once. You get an extra boost since intellisense kicks in to auto-complete the argument name.

testnull

The ying to argnull’s yang, testnull generates a test to exercise your argument null check.

Note: this test inherits from a base class called Context which isn’t standard MSpec. I always create a context class for each test file to share context and setup between multiple tests.

guid

It can be handy to seed well known GUIDs in order to test retrieval by ID. But no one ever remembers how may digits are in the stupid things - guid snippet to the rescue!

Note: you can provide two sample digits in case you make it all the way up to 10.

any

More finger-gymnastics are required when using NSubstitute’s catch all argument matcher, the any snippet makes life easier.

Bonus Boxcar Script

MSpec expects your specification statements to_be_in_boxcar_case. This is tedious to type compared with using the space key. I’ve included a AutoHotKey script to convert anything surrounded in double quotes on the current line into the correct format.

Install AutoHotKey
Double-click on Boxcar.ahk in your snippets folder
Create a shortcut to this file into your startup folder

Once the script is running you can press Right-Alt + Return to format the current line.

JavaScript Gotcha: Loop Scope

2015-09-01T00:00:00+00:00

This JavaScript gotcha had me scratching my head for the best part of an hour. I was attempting to create a number of files on the file system using the following code:

var files = ['folder1/a.txt', 'folder1/b.txt', 'folder2/c.txt'];

for (var i = 0; i < files.length; i++) {
  var file = files[i];
  mkdirp(path.dirname(file), function () {       // Create the folder
    fs.writeFile(file, "Content", function () {  // Create the file
      console.log(file + " created");
    });
  });
}

Surprisingly this didn’t work, instead I got the following output:

folder2/c.txt created
folder2/c.txt created
folder2/c.txt created

Why did it try to create the last file three times? Because JavaScript loops don’t have their own scope like most other languages do… no seriously… I’m not joking, try this out:

for (var i = 0; i < 3; i++) {

}

console.log(i);  // outputs '3', OUTSIDE of the loop

So if the for loop contains asynchronous callbacks they will get the final value of any variable set in the loop, not their current value.

You can fix this by using the forEach method of the array instead:

files.forEach(function (file) {
  mkdirp(path.dirname(file), function () {       // Create the folder
    fs.writeFile(file, "Content", function () {  // Create the file
      console.log(file + " created");
    });
  });
});

Every day’s a school day.

Continuous Delivery Tips

2015-08-05T00:00:00+00:00

I recently gave a friend some advice on setting up his first Continuous Delivery Pipeline. We’ve been running pipelines on our .NET web projects for the last couple of years so I had a few tips that I’ll document here for posterity.

A continuous delivery pipeline is the practice of breaking your automated build and test process into a number of stages. Early stages build and unit test the software, providing fast feedback on the basic correctness of the code. Later stages perform lengthier tests like integration tests, performance tests and manual exploratory testing. Each stage provides increasing confidence, so when a release candidate makes it to the end of the pipeline it can be deployed into production. For further information, check out Martin Fowler’s article.

Tip #1: Use a decent scripting language

To set up a pipeline, you’ll need to automate lots of tasks… and that means writing a large number of scripts. Currently we have scripts for:

Building the code into binaries
Running automated tests
Generating documentation
Publishing documentation
Applying version numbers
Changing configuration files
Deploying web applications
and much more…

Get used to the idea that these scripts will be first class citizens in your codebase that need the same care and attention as your production code.

We use Ruby, more specifically Rake, for all of our scripting needs. Now choosing Rake for scripting .NET projects has raised a few eyebrows at the office, but it actually works really well. The Albacore project extends Rake to add tasks for building and testing .NET:

build :rebuild do |b|
  b.sln    = 'MyProject.sln'
  b.target = ['Clean', 'Rebuild']

  b.prop 'Configuration', 'Release'
end

One of the advantages of using Rake is that you’re actually writing Ruby code. That means you can write custom tasks directly in the Rakefile using the full power of Ruby with its clean and readable syntax. Adding a Gemfile to your project allows you to download third party libraries (Gems) for all kinds of tasks such as XML manipulation, sending emails, making HTTP calls etc.

Finally, avoid declarative XML based task systems (I’m looking at you MSBuild). Their syntax is verbose and horrible to read, especially if looping or branching is required in your scripts. Custom tasks are generally farmed out to another scripting language like PowerShell, or worse yet compiled into DLLs, so even the simplest scripting task becomes a real chore.

Tip #2: Don’t marry your CI platform

Continuous integration platforms like TeamCity provide all sorts of fancy features to entice you into a long term relationship. They can detect the type of project in your source code and automatically build it, provide runners for every testing framework known to mankind, even publish your libraries to package managers and your websites to the cloud!

Don’t be tempted! The problem here is that the logic needed to build and release your software gradually moves from your source controlled codebase into your CI platform’s configuration screens.

I worked on one project that totally relied on the CI server to build and package the code. No one knew how to do it manually if the build server was offline - you couldn’t even produce a custom build from your development environment to use in a sales demo.

Here’s the golden rule:

Every task performed on the build server can also be done by a developer, from the command line, on their local development environment

If you follow this rule then developers can write and test their scripts in isolation, without interrupting the delivery pipeline. It also makes it easier to reproduce and debug issues with your scripts.

For us, this means that our TeamCity server simply runs a series of Rake tasks at each stage of the delivery pipeline; and we use a clone of our development VM (built using Vagrant) as the TeamCity build agent.

Tip #3: Use traceable version numbers

All being well, your pipeline will be pumping out multiple release candidates every day… one for each push that a developer makes to the repository.

Different releases will be deployed to your various test, demo and production environments, so it’s useful to have a numeric version number that can easily be traced back to your source control. This helps with questions such as:

does this release include the fix for bug #99?

… and:

is our demo server on the same version as production?

We use a fairly standard numeric version number like v1.4.2.357 made up of major.minor.patch.build parts. The first three parts are set manually by adding an annotated tag to the commit in our Git repository.

$ git tag -a v1.4.2 -m 'Tagging v1.4.2'
$ git push --tags

The build part of the version number is automatically generated based on the number of commits since the last tag. Git will tell you this when you use the describe command:

$ git describe
v1.4.2-357-gec812be

This approach allows you to trace a version back to the specific commit in source control that it was built from. If you keep your code in GitHub this works with their Releases feature as it also uses annotated tags in the same manner.

Tip #4: Quarantine flakey tests

There’s nothing worse than a non-deterministic test. They never misbehave while you’re watching them, they can be very tricky to diagnose, and they always start failing while you’re trying to get a critical fix through the pipeline.

Setup a quarantine stage early in the project that runs after your main testing stages, but doesn’t block a release candidate from progressing through the rest of the pipeline. When a test starts failing intermittently, place it in the quarantine until you have a chance to properly investigate and fix it.

There shouldn’t be too many tests in the quarantine, you should be actively working on improving their reliability, so that means the quarantine stage can be re-run quickly, and repeatedly, until the tests go green. If the tests aren’t passing, even after a few re-runs, then there’s probably a real issue with the functionality.

We prefer to use a quarantine instead of simply ignoring or removing the test. There is still value in running your non-deterministic tests on a regular basis, and it’s much harder to forget about them when they are clearly visible as a stage in your pipeline.

For further reading, check out Eradicating Non-Determinism in Tests.

Tip #5: Read the book

My final tip is to read the book. It’s an excellent read that clearly explains the practice of continuous delivery, and provides practical advice on how to set it up.

You can also read the chapter on deployment pipelines as a free download - what wonderful chaps!

Simplifying Node with Koa

2015-07-07T00:00:00+00:00

I really enjoy writing web applications in Node JS, but sometimes the asynchronous programming style makes my head hurt! This code sample is from my current side project, commentator. It retrieves comments stored as Markdown files from the local file system and returns them as a JSON array:

var folder = path.join('data', req.url);
var files  = fs.readdirSync(folder);

var comments = [];
for (var i = 0; i < files.length; i++) {
  var file     = path.join(folder, files[i]);
  var markdown = fs.readFileSync(file, 'UTF-8');

  var comment = parse(file, markdown);
  comments.push(comment);
}

res.send(comments);

Dead simple, right? It looks for files in the requested folder, loops through each file and reads the contents, then parses the contents into a comment and adds it to an array.

But show this simple, readable code to an experienced Node programmer and they’ll laugh you out of the room. They’ll point out that using fs.readdirSync or fs.readFileSync in a web application will tie up your single JavaScript thread with lengthy file system operations. The application won’t be able to serve requests from multiple users while it’s reading from the file system leading to a very slow website when placed under load.

To allow this code to scale we need to use the asynchronous version of the file system methods… but what does that look like?

var folder = path.join('data', req.url);
fs.readdir(folder, function (err, files) {
  if (err) throw err;

  var comments = [];
  for (var i = 0; i < files.length; i++) {
    var file = path.join(folder, files[i]);
    fs.readFile(file, 'UTF-8', function (err, markdown) {
      if (err) throw err;

      var comment = parse(file, markdown);
      comments.push(comment);

      if (comments.length == files.length) {
        res.send(comments);
      }
    });
  }
});

Yuck! My original 10 lines of code have increased to 16; the maximum indentation has gone up to 4 levels; and I’ve had to sprinkle the code with if (err) throw err; statements to manually check for, and re-throw errors! 🙀

There’s also complexity around when the response is ready to be sent. The code loops over the files and kicks off asynchronous readFile operations in parallel. The callbacks may come out of order, so I have added an if statement into each one to check if all the comments have been added before sending the response.

Enter Koa. It’s a new web framework from the team behind Express, so it has some pedigree. Koa allows you to write the following:

app.use(function *() {

  var folder = path.join('data', this.request.url);
  var files  = yield fs.readdir(folder);

  var comments = [];
  for (var i = 0; i < files.length; i++) {
    var file     = path.join(folder, files[i]);
    var markdown = yield fs.readFile(file, 'UTF-8');

    var comment = parse(file, markdown);
    comments.push(comment);
  }

  this.body = comments;

});

This code is identical to the simple version except that it uses asynchronous file system operations preceded by the yield keyword. Koa cleverly ~~abuses~~ uses a new feature of JavaScript called generators to pause the code mid-flow, wait for the asynchronous operation to complete, then continue executing.

There is one catch - you can only yield asynchronous operations that return a promise. However, the file system operations in the code above use the standard Node callback pattern, so I’ve added the following wrapper to convert the callbacks to promises:

var fs = require('fs'),
    q  = require('q');

module.exports.readdir  = q.denodeify(fs.readdir);
module.exports.readFile = q.denodeify(fs.readFile);

Hopefully I’ve convinced some of you to give Koa a try. I’m excited about how the code is shaping up in commentator, and since Koa is developed by the team who wrote Express - I have a feeling that it may be here to stay.

Ruby on Rails, on Dokku!

2015-04-29T00:00:00+00:00

In my last post I showed how to setup Dokku and deploy a simple “Hello World” web application. Now let’s turn it up to 11 and build a fully-blown Ruby on Rails application with its own database, taking a look at some of Dokku’s management features along the way.

The app will be flicklog - a journal of movies that you’ve seen recently at the cinema with a short synopsis and a star rating.

Ready? Adequately caffeinated? Let’s go.

Create a Rails Application

To get started you’ll need a development environment with Ruby, Rails 4 and PostgreSQL. Setting this up can be a little tricky, I would recommend that you install Vagrant and use my lovingly crafted Vagrantfile to provision a development VM. That way we’ll all be on the same page.

On your development VM, run the following commands to create a database and a new Rails application, specifying that it should use PostgreSQL as the database:

createdb flicklog_development
rails new flicklog --database=postgresql
cd flicklog

Next, we have to add a couple of gems to the Gemfile:

# Gemfile
gem 'therubyracer', platforms: :ruby
gem 'rails_12factor', group: :production

therubyracer provides Rails with a JavaScript runtime and the rails_12factor gem is needed for the logging to work with Dokku.

We can now install the gems and run the web application:

bundle install
rails s -b 0.0.0.0

This starts up the web application on port 3000, you can visit the site on http://localhost:3000. It will show the default “Welcome aboard” page with getting started instructions.

Next, we want to create a movie entity that has a title, synopsis, release date and star rating. Rails provides a scaffold command that generates the code for this automatically:

rails generate scaffold movie title synopsis:text released:date rating:integer
rake db:migrate

The scaffold command creates a model, view, controller and database migration for our movie entity. The migrate command applies the database migration to our database, creating a new table to store our movies.

Finally, edit the default route to make the home page our list of movies:

# config/routes.rb
root 'movies#index'

If you visit your local site now, you will see an empty table of movies and you can start adding movies to your log.

Deploy to Dokku

Now that we’ve got our simple app working, it’s time to deploy it to Dokku. My previous post explains how to setup a Dokku server, I’m going to push my app to flicklog.colinthegeek.com.

Dokku uses Git to deploy your application, so your code has to be checked into a Git repository before it can be deployed. You may already use an online service like GitHub or BitBucket to store your code, but for the purposes of this article I’m going to add the code to a local repository on my development VM:

git init
git add .
git commit -m 'Initial Commit'

Now the code is in a repository, we add a remote that points to our Dokku server, don’t forget to substitute colinthegeek.com with your own server address:

git remote add geek dokku@colinthegeek.com:flicklog

…where “geek” is the name of the remote, “dokku” is the Git user name, “colinthegeek.com” is the server, and “flicklog” is the name of the app.

That’s the configuration complete, lets push the code to Dokku and see what happens:

git push geek master

The code should deploy successfully and return the URL at the bottom of the deployment log:

=====> Application deployed:
       http://flicklog.colinthegeek.com

Viewing the Logs

If you visit the site now, you’ll be disappointed to see the following error message:

Something has gone wrong, and Rails won’t tell us what because it’s running in production mode. It does suggest that we check the logs, so lets try that. SSH onto your server and execute dokku to see the commands that are available:

> ssh root@colinthegeek.com
> dokku

Usage: dokku COMMAND <app> [command-specific-options]

apps:create <app>                               Create a new app
apps:destroy <app>                              Permanently destroy an app
apps                                            List your apps
backup:export [file]                            Export dokku configuration files
backup:import [file]                            Import dokku configuration files
config <app>                                    Display the config vars for an app
config:get <app> KEY                            Display a config value for an app
config:set <app> KEY1=VALUE1 [KEY2=VALUE2 ...]  Set one or more config vars
config:unset <app> KEY1 [KEY2 ...]              Unset one or more config vars
domains:add <app> DOMAIN                        Add a custom domain to app
domains <app>                                   List custom domains for app
domains:clear <app>                             Clear all custom domains for app
domains:remove <app> DOMAIN                     Remove a custom domain from app
help                                            Print the list of commands
logs <app> [-t]                                 Show the last logs for an application (-t follows)
nginx:build-config <app>                        (Re)builds nginx config for given app
nginx:import-ssl <app>                          Imports a tarball from stdin; should contain server.crt and server.key
plugins-install                                 Install active plugins
plugins                                         Print active plugins
plugins-update                                  Update active plugins
run <app> <cmd>                                 Run a command in the environment of an application
url <app>                                       Show the first URL for an application (compatibility)
urls <app>                                      Show all URLs for an application
version                                         Print dokku's version

The command that we are looking for is dokku logs:

> dokku logs flicklog

Started GET "/" for 82.132.212.222 at 2015-04-27 18:15:15 +0000

PG::ConnectionBad (could not connect to server: No such file or directory
	Is the server running locally and accepting
	connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
):

The logs tell us that our application cannot find the PostgreSQL server, that’s OK, we haven’t created one yet. We’ll do that in the next section.

PostgreSQL Plugin

Dokku doesn’t have any support for databases, but it does have rich support for plugins. There are a number of community plugins for PostgreSQL, my favourite one is by kloadut.

Install the plugin by executing the following commands on your Dokku server:

cd /var/lib/dokku/plugins
git clone https://github.com/Kloadut/dokku-pg-plugin postgresql
dokku plugins-install

The plugin extends the available Dokku commands:

> dokku
...
postgresql:console <db>                        Open a PostgreSQL console
postgresql:create <db>                         Create a PostgreSQL container
postgresql:delete <db>                         Delete specified PostgreSQL container
postgresql:dump <db> > dump_file.sql           Dump database data
postgresql:info <db>                           Display database informations
postgresql:link <app> <db>                     Link an app to a PostgreSQL database
postgresql:list                                Display list of PostgreSQL containers
postgresql:logs <db>                           Display last logs from PostgreSQL container
postgresql:restore <db> < dump_file.sql        Restore database data from a previous dump

All we have to do is create a database and link it to our application:

dokku postgresql:create flicklog-db
dokku postgresql:link flicklog flicklog-db

Finally, lets run the migration to create our movies table:

dokku run flicklog rake db:migrate

Now you can visit the site and start logging flicks!

Managing Data

The PostgreSQL plugin includes some handy commands for exporting and importing data. These commands use the postgresql-client behind the scenes, so you must install it on your Dokku server before you use them:

sudo apt-get install postgresql-client

You can export the database with the dump command:

dokku postgresql:dump flicklog-db > ~/flicklog-db.sql

This creates a full backup of your database schema and data. You can restore a backup with the restore command:

dokku postgresql:restore flicklog-db < ~/flicklog-db.sql

Finally, the console command is useful if you want to run SQL queries against your database. The following commands open a console, selects all of the movies and quits:

dokku postgresql:console flicklog-db
select * from movies;
\q

Hopefully this was guide was helpful, I’ve been using Dokku to host side projects for a while and I find it very useful. You can find the source code from in this article on my GitHub: https://github.com/ColinOrr/dokku-rails

Setting up a Dokku Development PaaS

2015-02-04T00:00:00+00:00

I have a bad case of multiple technology disorder. At the last count I have six side projects on the go, and not one of them uses the same language or framework! Don’t worry - I’m sure the doctor will prescribe me some pills to sort this out, but until then I’d like somewhere to host these projects online to show them off to my friends and colleagues.

Heroku would be the ideal solution, it’s a popular Platform as a Service (PaaS) that allows you to deploy applications written Ruby, Node.js, Python and PHP by pushing your source code to their servers using Git. The issue with Heroku is that it gets rather expensive when you want to do anything serious. The web dyno that runs your site has a habit of going to sleep on the free plan, which is a bit embarrassing during a demo… also you can’t have SSL without moving to a paid plan.

Enter Dokku. It’s a mini version of Heroku that can run on your own Linux machine. It uses Heroku’s open source build packs to configure a Docker container to run your application. So if your app works on Heroku, it’ll work on Dokku.

I’m running Ruby on Rails, Express, Laravel and static HTML sites all on the same 512MB Ubuntu server for only $5 a month with DigitalOcean! The rest of this article explains how to set it up. Although I’m using DigitalOcean, the same instructions should apply to other cloud providers.

Step 1: Create an Ubuntu VM

At the time of writing, Dokku is tested on Ubuntu 14.04 x64. With DigitalOcean, you just create a new $5 per month droplet for this version of Ubuntu. Don’t forget to add your SSH key, although it’s optional you’ll need it later for working with Dokku. If you don’t have an SSH key, GitHub has a good article explaining how to create one.

DigitalOcean will create your droplet in about 60s, just enough time to go and register a new .sexy domain name… personally I’m going to stick with colinthegeek.com :-)

DigitalOcean will report your static IP address on the Droplets page, in my case it’s 178.62.99.39. Point your new domain to this IP address by adding two alias “A” records in the DNS settings of your domain provider:

Host Name	IP Address
@	178.62.99.39
*.colinthegeek.com	178.62.99.39

The first “@” entry points the top-level domain to your new server, the second entry is for subdomains. The subdomain configuration is important as this is how Dokku will make your application available e.g. chitter.colinthegeek.com.

Next, we have to connect onto the server and create some swap space. For some reason cloud providers don’t configure this by default, and if you don’t have any you’ll quickly run into memory issues. My server has 512 MB of RAM, so I’m going to give it an additional 512 MB of swap space.

Connect to the server using SSH:

ssh root@colinthegeek.com

Create a 512MB swap file and enable it:

sudo fallocate -l 512M /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Make the change permanent by modifying the fstab file:

sudo nano /etc/fstab

Finally, add the following line to the bottom of the file and save it:

/swapfile   none    swap    sw    0   0

This DigitalOcean article gives a detailed description of what just happened.

Step 2: Install Dokku

Dokku is installed using a single shell script. Head over to Dokku’s GitHub page to find the the latest stable version of the bootstrapper command. Currently it’s on v0.3.13.

Connect to your server and run the bootstrapper command:

ssh root@colinthegeek.com
wget -qO- https://raw.github.com/progrium/dokku/v0.3.13/bootstrap.sh | sudo DOKKU_TAG=v0.3.13 bash

This will install all Dokku dependencies and clone their repository to ~/dokku in your home directory. You can use this repo to upgrade Dokku in the future. It also adds a user called dokku, and their home directory /home/dokku is where your applications will be deployed.

The next step is to register your domain name with Dokku:

echo colinthegeek.com > /home/dokku/VHOST

When it comes to deploying your application, you’ll use a git push to upload it to the dokku user. This means you have to add your SSH public key as an authorised key for the dokku user account. Your public key will be on the local machine in a ~/.ssh/id_rsa.pub file by convention.

Exit the SSH session to get back to your local machine:

exit

Copy the contents of your SSH public key file to the server and add it to the dokku user’s authorised keys:

cat ~/.ssh/id_rsa.pub | ssh root@colinthegeek.com "sudo sshcommand acl-add dokku colinthegeek"

Note: don’t forget to change the server name (colinthegeek.com) to your own domain name. The last argument “colinthegeek” is just a label to help distinguish your key from others on the server, so it doesn’t really matter what you name it.

You can give your friends access to push to your server by running the command above with their public keys.

Step 3: Push a Sample Application

And now the moment you’ve all been waiting for… time to push an application and see what breaks!

Clone Heroku’s sample application for Node JS:

git clone https://github.com/heroku/node-js-sample.git
cd node-js-sample

Add a new remote repository that points to your Dokku server:

git remote add geek dokku@colinthegeek.com:hello-world

…where “geek” is the name of the remote, “dokku” is the Git user name, “colinthegeek.com” is the server, and “hello-world” is the name of the app.

Push the code to Dokku:

git push geek

You’ll see some log messages as Dokku deploys the application, all being well it should end with:

=====> Application deployed:
       http://hello-world.colinthegeek.com

Go ahead and visit your new “Hello World” app, show it off to your significant other, receive a blank look from them as you marvel at your own feat of engineering, then go and have well deserved cup of tea!

In my next article I’ll show you how to manage your Dokku applications, setup a Postgres database and get a simple Rails website up and running.

Hello World

2015-01-31T00:00:00+00:00

Well it’s January 2015 and the snow falling outside has given me a good excuse to avoid my primary New Year’s resolution of a bi-weekly run! So I’ve decided to start working on my fallback resolution and setup a blog. Self-confessed geek that I am, I quite enjoyed evaluating the different blogging platforms out there. I looked at WordPress, Ghost and Jekyll.

I discounted WordPress fairly quickly. It’s a comprehensive one-stop-shop for blogging and content management; but it was too heavy weight for my simple blog and the user interface felt a bit bloated and clunky.

Ghost was a real contender. It was easy to setup by simply downloading the code and running npm install. The user interface is super slick with a simple split-screen Markdown editor that shows the Markdown on the left and a preview on the right. The way Ghost handles images is particularly clever, you add a placeholder for the image in your Markdown, and an upload panel appears in situ on the preview pane where you can drag-and-drop your image.

I had a couple of niggles with Ghost. I couldn’t get spell check to work, even though my browser has a built in spell checker. Although the preview is excellent, it doesn’t show your post styled in your chosen theme… in order to see the finished article you have to publish it, which seemed a little odd.

Jekyll also uses Markdown syntax, but unlike Ghost it generates static HTML for your site upfront so you can host it pretty much anywhere. Jekyll is a Ruby Gem, you install it and scaffold a template site from the command line:

gem install jekyll
jekyll new colinthegeek
cd colinthegeek
jekyll serve

This will start your new site on port 4000. As you edit your blog article or theme, Jekyll will monitor your files and regenerate the site each time you save. If you’re a Ruby on Rails developer Jekyll will feel very familiar.

Jekyll requires that you to add metadata to your Markdown to help the theme generate your site. They call this Front Matter and takes the format of YAML at the top of you Markdown file:

---
layout: post
title: Hello World
---
Well it's January 2015...

This turns out to be a really flexible approach. I added a new summary property to my posts then edited the home page template to use the summary rather than display the entire post. I did this without needing to extend the framework or register the new attribute, it just worked!

When it comes time to publish your blog, you run a jekyll build command to compile your Markdown posts into a static HTML site that can be hosted on a basic HTTP server.

But here’s where it gets really cool, GitHub Pages will host your Jekyll site, for free! You simply push your Jekyll site to a GitHub repository called <your_username>.github.io, and GitHub will compile the site and make it available on http://<your_username>.github.io. They even allow you to point a custom domain name to your site. Thanks GitHub!

If you’re interested, you can view the source code of this blog on GitHub:

https://github.com/ColinOrr/colinorr.github.io.

Jekyll won’t be for everyone, it’s very much a developer platform for writing a blog and the blogger must be familiar with Markdown, GitHub and Ruby to get started. Ghost is also a great choice with a lower barrier to entry, however, you’d need to find somewhere to host it and think about your backup strategy which isn’t an issue when using GitHub Pages.

Colin the Geek

Stylish Markdown

Time Travel DSL

Encapsulating DateTime.Now

TimeSpan extension methods

Scoping the time travel

Using SQL LIKE with Entity Framework

Option 1: Entity SQL

Option 2: SQL Functions

Option 3: Write a Custom SQL Function

Conclusion

Generating Sample Data

Selecting a random item from a sequence

Weighting values

Ranges

Dockerizing my Development Process

Table of Contents

The Basics

Introducing Quotster

Getting Started

Prototyping

Adding the Database

Scaling Up

Releasing to Production

Conclusion

Regex Find & Replace

Introducing Penfold

The Basics

Gherkin Syntax

Logging

Ignored, Pending & Categorization

Expecting Exceptions

Summary

Visual Studio Code Snippets

Installation

test

lam

argnull

testnull

guid

any

Bonus Boxcar Script

JavaScript Gotcha: Loop Scope

Continuous Delivery Tips

Tip #1: Use a decent scripting language

Tip #2: Don’t marry your CI platform

Tip #3: Use traceable version numbers

Tip #4: Quarantine flakey tests

Tip #5: Read the book

Simplifying Node with Koa

Ruby on Rails, on Dokku!

Create a Rails Application

Deploy to Dokku

Viewing the Logs

PostgreSQL Plugin

Managing Data

Setting up a Dokku Development PaaS

Step 1: Create an Ubuntu VM

Step 2: Install Dokku

Step 3: Push a Sample Application

Hello World