Summer vacation times are over. Well, for some of us at least, clearly some are still lucky enough to be off, which is showing itself a bit (see below). But as both conference organisation and participation throughout the rest of the year is starting to be clear, I figure it's time to share some updates around different ones.

Postgres Open SV

First of all - if you haven't already, don't forget to register for Postgres Open SV in San Francisco in two weeks time! Registration for the main US West Coast/California PostgreSQL community conference will close soon, so don't miss your chance. I'm looking forward to meeting many old and new community members there.

PostgreSQL Conference Europe

Next up after Postgres Open will be pgconf.eu, the main European PostgreSQL community conference of 2018. The planning for this years conference is at full speed, but unfortunately we are slightly behind. In particular, we were supposed to be notifying all speakers today if they were accepted or not, and unfortunately our program committee are a bit behind schedule on this one. We had over 200 submissions this year which makes their work even bigger than usual. But speakers will be receiving their notification over the upcoming couple of days.

Hopefully once all speakers have been able to confirm their attendance, we will also have a schedule out soon. Until then, you can always look at the list of accepted talks so far. This list is dynamically updated as speakers get approved and confirm their talks.

We have already sold approximately half of the tickets that we have available this year, so if you want to be sure to get your spot, we strongly recommend that you register as soon as you can! And if you want to attend the training sessions, you should hurry even more as some are almost sold out!

PGConf.ASIA

Work on the program committee of PGConf.ASIA has also been going on over the late summer, and is mostly done! The schedule is not quite ready yet, but expected out shortly. You can look forward to a very interesting lineup of speakers, so if you are in the Asian region, I strongly recommend keeping an eye out for when the registration opens, and join us in Tokyo!

FOSDEM PGDay

As has been announced, PostgreSQL Europe will once again run a FOSDEM PGDay next to the big FOSDEM conference in Brussels in February next year. We hope to also run our regular booth and developer room during FOSDEM, but those are not confirmed yet (more info to come). The Friday event, however, is fully confirmed. Of course not open for registration yet, but we'll get there.

Nordic PGDay

Nordic PGDay has been confirmed for March 19th next year. The format will be similar to previous years, and we will soon announce the location. For now, mark your calendars to make sure you don't double book! And rest assured, the conference will take place somewhere in the Nordics!

Usergroups and PGDays

Then there are a number of smaller events of course. Next week, I will speak at the Prague PostgreSQL Meetup. We should be kicking off the Stockholm usergroup. PDXPUG runs a PGDay in Portland in September (which I unfortunately won't be able to attend). In general, it seems like usergroups are starting to get going again after the summer break, so check with your local group(s) what's happening!

This post will show you how to get started with using Terraform, an open-source tool that lets you build your infrastructure using code.

Some time ago, the Varnish Cache project released a new upstream version 6.0 of Varnish Cache. I updated the fedora rawhide package a few weeks ago. I have also built a copr repo with varnish packages for el6 and el7 based on the fedora package. A selection of matching vmods is also included.

Packages are available at https://copr.fedorainfracloud.org/coprs/ingvar/varnish60/

The following vmods are available:

Included in varnish-modules:
vmod-bodyaccess
vmod-cookie
vmod-header
vmod-saintmode
vmod-tcp
vmod-var
vmod-vsthrottle
vmod-xkey

Packaged separately:
vmod-curl
vmod-digest
vmod-geoip
vmod-memcached
vmod-querystring
vmod-uuid

Please test and report bugs. If there is enough interest, I may consider pushing these to fedora as well.

We build our network in order to simultaneously achieve high availability and maximum utilisation of available bandwidth. To that end, we are using Multi-Chassis Link Aggregation (MLAG) between our data centre switches running Cumulus Linux and our firewall cluster

I spent some time in the United States last month. Equipped with SIM cards from both Tele2 Sweden (MCCMNC 24007) and Telenor Norway (MCCMNC 24201), I set out to test IPv6 while roaming, as I usually do while abroad.

The previous posts in this series are:

• IPv6 roaming in Belgium and Romania
• IPv6 roaming in the United Kingdom
• IPv6 roaming in Czechia
• IPv6 roaming in Sweden

There were two mobile networks that I was able to register in and test: AT&T (3G/4G, no 2G) and T-Mobile (2G/3G/4G). The test results are found below.

I could also see three other 4G networks show up in a network scan (Caprock Cellular, Sprint and Verizon), but none of those were available to me. Presumably neither Tele2 nor Telenor have roaming arrangements with any of those operators.

AT&T - MCCMNC 310410

T-Mobile - MCCMNC 310260

3GPP cause code #33 means «requested service option not subscribed», while cause code #50 means «PDP type IPv4 only allowed». The latter doesn’t indicate a fatal error, as I do automatically get a working IPv4-only connection to the Internet.

The fact that IPv6 consistently fails on 2G/3G is in all likelihood due to Tele2/Telenor’s HLR removing the IPv6 capabilities from my subscriber profile before transmitting it to AT&T/T-Mobile’s vSGSN.

Tele2 and Telenor do this to forestall the possible IPv6-related failures described in RFC 7445 sections 3 and 6. In this case, it is in all likelihood an unnecessary precaution, considering that both AT&T and T-Mobile are known to have deployed IPv6 to their own customers.

With the upcoming elections in PostgreSQL Europe, I'm excited to see that we have more candidates than ever before. But during the FOSDEM conference we just finished in Brussels, that also lead to a fairly large number of people who asked me the simple question "what does it actually mean to be on the board of PostgreSQL Europe". So I think it's time to summarize that for both those standing for election, and for the members of the organisation in general.

For a TL; DR; version, being on the board basically means a lot of administrative work :) But read on for some details.

Mule 4 will be released soon. Along with Mule 4 a new Mule SDK is relased which can be used to extend the functionality of Mule with custom modules. The Mule SDK replaces Devkit for developing connectors.

Documentation on the Mule SDK can so far ...

If OpenStack is something you have heard of, but aren’t too familiar with, you probably think “virtual servers” - or maybe “automatic server provisioning”. That is in my opinion not what OpenStack is about. It is about being able to integrate with “everything infrastructure-related”, and enforcing access in a secure manner. OpenStack is a very good starting point for achieving that, giving you virtual servers, networking, storage management, and at it’s core: identity management. Installing virtual servers and provisioning resources for them are well and good, but that’s just an appliance. The fun part comes when we can Build Things.

One of the OpenStack-projects named Keystone, provide identity management. This component is responsible for dealing with the service catalogue and API endpoints in your OpenStack cloud. User authentication happen through this, and users usually have roles assigned on one or more projects. When an end user want to create a virtual server, the user passes an authentication token to the compute service API (nova), which it verifies with keystone before performing any actions. Through this mechanism, user authentication and authorization is nicely decoupled from the services that end up delivering infrastructure. It is meant to be used as a mean to extend OpenStack, and it’s pretty much straight forward.

Prerequisites

• An OpenStack-solution
• Ability to write a small web service application, for example using something like Flask. Can be done in any programming language, but python give you access to a bunch of SDKs for speaking with the OpenStack APIs which can come in handy.
• Something you want to implement

So if you have those three things, you’re good to go. If you don’t have those - but wanna play around and see what you can do, I can push you in the right direction to get started.

• You don’t have an OpenStack-solution set up? DevStack can be set up to give you a functioning minimal OpenStack-rig - you shouldn’t use it for anything important - but it’s an excellent development tool.
• You don’t know how to write a small web service application? Learn Flask!

Preparing OpenStack

The two first steps of adding the API to the service and endpoint catalogue aren’t mandatory - but it’s a polite thing to do. By doing so, users of the cloud can see the URLs where your API is available.

## As an admin user, perform the following steps:

# openstack service create myservice --name myservice
+---------+----------------------------------+
| Field   | Value                            |
+---------+----------------------------------+
| enabled | True                             |
| id      | b81a620dfdb54592b97c67816d00fc43 |
| name    | myservice                        |
| type    | myservice                        |
+---------+----------------------------------+

# openstack endpoint create myservice admin http://ctrl-node.mycloud.example.org:1337
# openstack endpoint create myservice public http://ctrl-node.mycloud.example.org:1337
# openstack endpoint create myservice internal http://ctrl-node.mycloud.example.org:1337
[...]

# openstack endpoint list --service myservice
+----------------------------------+--------+--------------+--------------+---------+-----------+-----------------------------------------------+
| ID                               | Region | Service Name | Service Type | Enabled | Interface | URL                                           |
+----------------------------------+--------+--------------+--------------+---------+-----------+-----------------------------------------------+
| 1fbbbcc57108422596e07ab4bd713631 | None   | myservice    | myservice    | True    | internal  | http://ctrl-node.mycloud.example.org:1337     |
| 25edd1c28178462980e90bce2b8140f2 | None   | myservice    | myservice    | True    | public    | http://ctrl-node.mycloud.example.org:1337     |
| 78b2090f1df4450fb4544abf1eee842a | None   | myservice    | myservice    | True    | admin     | http://ctrl-node.mycloud.example.org:1337     |
+----------------------------------+--------+--------------+--------------+---------+-----------+-----------------------------------------------+

The more important step is to create a service user, which will be used to perform administrative tasks - such as validating authentication tokens.

# openstack user create --email myservice@localhost --password myservicesecretpassword myservice
+---------------------+----------------------------------+
| Field               | Value                            |
+---------------------+----------------------------------+
| domain_id           | default                          |
| email               | myservice@localhost              |
| enabled             | True                             |
| id                  | ab3e5a51a87a44898c8543e0a254bd3b |
| name                | myservice                        |
| options             | {}                               |
| password_expires_at | None                             |
+---------------------+----------------------------------+

# openstack role add --project services --user myservice admin

At this point, we have our service user.

The web service

You need to handle the API calls, and you need to verify authenticated requests.

Validating a token can be done with the following code:

from keystoneauth1.identity import v3
from keystoneauth1 import session
from keystoneclient.v3 import client

# Authenticate with the service user, and create a keystone client
auth = v3.Password(auth_url='https://ctrl-node.mycloud.example.org:35357/v3',
                   username='myservice',
                   password='myservicesecretpassword',
                   project_name='services',
                   user_domain_id='default',
                   project_domain_id='default')
sess = session.Session(auth=auth)
keystone = client.Client(session=sess)

# Validator-class that can easily be extended with functions
# that test role/project-access. self.validate contains a hash
# with information about this.
class Validator:
    def __init__(self, token):
        self.validate = keystone.tokens.validate(token)

Tokens are passed between OpenStack-services using the x-auth-token http-header. In Flask, picking this out in your service routes can be as simple as request.headers.get('x-auth-token').

Using the validator-class above in a route can be

from flask import Flask
from flask import request
from flask import abort
from flask import jsonify

from validator import Validator
from thing import Thing

# A helper function to wrap the raised exception in the case 
# of an invalid token.
def create_validator():
    try:
        v = Validator(request.headers.get('x-auth-token'))
    except:
        abort(403)
    return v

# Routes below.
# The /-route can be used as a liveliness and ready probe.
@app.route("/")
def check():
    return jsonify({"myservice": 0.1})

# Authenticated route.  Passing the validator object to the
# constructor of Thing(), and calling a method - returning its
# output as json.
@app.route("/v1/thing/list")
def thing_list():
    return jsonify(Thing(create_validator()).list_things())

What is Thing()? It’s a class we’ve made to split application logic from routes. At the route level, we have ensured that the request is authenticated now we want to do something.

class Thing:
    def __init__(self, validator):
        self.v = validator

    def list_things(self):
        return { 'things': ['apple', 'chair', 'banana', 'television'] }

The Thing class takes a validator in its constructor for two reasons.

1. It’s nice to authenticate the request early on.
2. You get access to role and project-information from within the class, and can make decisions based on that.

A slightly more complete version of this skeleton is available on GitHub.

Testing it out

Loading our liveliness route works fine without any authentication.

$ curl http://ctrl-node.mycloud.example.org:1337/
{
  "myservice": 0.1
}

Our web service for listing things requires authentication:

$ curl http://ctrl-node.mycloud.example.org:1337/v1/thing/list
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>403 Forbidden</title>
<h1>Forbidden</h1>
<p>You don't have the permission to access the requested resource. It is either read-protected or not readable by the server.</p>

So we need to issue a token. We can do that by calling the identity API manually, or if one have set up a CLI to our cloud - one can use that to issue a token.

$ openstack token issue
+------------+----------------------------------+
| Field      | Value                            |
+------------+----------------------------------+
| expires    | 2018-01-27T23:23:16+0000         |
| id         | 39465e6698b7409cb926a870cef1e5bb |
| project_id | dafe2c2a6d6e4bfc9146ea975b2897e0 |
| user_id    | 830ec37bc8774738904731a7bd55cb58 |
+------------+----------------------------------+

Inserting the token id as the http header ‘x-auth-token’ will authenticate requests against our OpenStack cloud, including our custom built web service:

$ curl -H 'x-auth-token: 39465e6698b7409cb926a870cef1e5bb' http://ctrl-node.mycloud.example.org:1337/v1/thing/list
{
  "things": [
    "apple",
    "chair",
    "banana",
    "television"
  ]
}

The web service has the information it needs about the calling user by validating the token. Which user, domain, project, and what roles the user have.

At this point one are free to implement whatever one pleases. One can integrate with some in-house system that shows invoice information for a customer - or may just be to create some helper APIs that simply re-uses the provided token to chain-run a series of other OpenStack APIs.

If you want to provide a user friendly web-interface for your APIs, stay tuned for Part II.

Links

• Example API skeleton
• Extending the OpenStack - Part II: The web interface (Upcoming)

Jenkins Pipeline is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins. I will be using this to deploy a Mule application to Anypoint Platform Runtime Manager and store the delivery in ...

Sometimes you disagree

A sometime

# Lets extract the scripts (and then some) from the mysql-server-5.5 package:
root@dbserver:/tmp/mysql-server-5.5_control# dpkg --control /var/cache/apt/archives/mysql-server-5.5_5.5.58-0ubuntu0.14.04.1_amd64.deb
# And pick put the parts relevant to the import/export directory from the post-install script
root@dbserver:/tmp/mysql-server-5.5_control# grep mysql_filesdir DEBIAN/postinst
    mysql_filesdir=/var/lib/mysql-files
    if [ ! -d "$mysql_filesdir"      -a ! -L "$mysql_filesdir"      ]; then mkdir "$mysql_filesdir"; fi
    chown -R mysql:mysql $mysql_filesdir
    chmod 700 $mysql_filesdir
root@dbserver:/tmp/mysql-server-5.5_control# 

drwx------ 2 mysql mysql 4096 nov.  23 10:09 /var/lib/mysql-files/

Chmod as a solution

Chmod reverted

Sudo as a non-solution

Different ways to get our will

An ensured failure

Be a puppet master

Call on a daemon related to a Greek primordial deity

File: /etc/cron.d/fix_permissions_mysql-files

# Ensure correct permissions for /var/lib/mysql-files directory
* * * * * root /bin/chmod 0770 /var/lib/mysql-files

rm  /etc/cron.d/fix_permissions_mysql-files

Immediate (i)notification and reaction

[..] the “inotify cron” system. It consist of a daemon and a table manipulator. You can use it a similar way as the regular cron. The difference is that the inotify cron handles filesystem events rather than time periods

File: /usr/local/bin/RL_fix_permissions_mysql-files.sh

#!/bin/bash

logger -t incron -p daemon.notice "Chmodding /var/lib/mysql-files back to 0770"
/bin/chmod 0770 /var/lib/mysql-files

File: /etc/incron.d/RL_fix_permissions_mysql-files

# Ensure correct permissions for /var/lib/mysql-files directory
# chmod if metadata changes, but disable while doing this, so we don't loop.

/var/lib/mysql-files IN_ATTRIB,IN_NO_LOOP /usr/local/bin/RL_fix_permissions_mysql-files.sh

rm /usr/local/bin/RL_fix_permissions_mysql-files.sh
rm /etc/incron.d/RL_fix_permissions_mysql-files

React instead of random act

File: /etc/apt/apt.conf.d/98RL_fix_permissions_mysql-files

# Ensure correct permissions after dpkg invocations
DPkg::Post-Invoke { "/bin/chmod -v 0770 /var/lib/mysql-files"; };

root@host:~# aptitude install cowsay
The following NEW packages will be installed:
  cowsay
0 packages upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
[..]
Setting up cowsay (3.03+dfsg1-6) ...
mode of ‘/var/lib/mysql-files’ changed from 0700 (rwx------) to 0770 (rwxrwx---)

rm /etc/apt/apt.conf.d/98RL_fix_permissions_mysql-files

Isn’t this really a bug?

if [ ! -d "$mysql_filesdir" -a ! -L "$mysql_filesdir" ]; then mkdir "$mysql_filesdir"; fi
[..]
chown -R mysql:mysql $mysql_filesdir
chmod 700 $mysql_filesdir

if [ ! -d "$mysql_filesdir" -a ! -L "$mysql_filesdir" ]; then
   mkdir "$mysql_filesdir"
   chown -R mysql:mysql $mysql_filesdir
   chmod 700 $mysql_filesdir
fi

rm /etc/apt/apt.conf.d/98RL_fix_permissions_mysql-files

Closing remarks

 _________________________
< A very merry Christmas! >
 -------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

The Raspberry Pi 3 is the third generation Raspberry Pi, on this I will be installing Mulesoft enterprise runtime with latest Java 8 running inside Kubernetes. The pods will register themselves with Anypoint platform runtime manager. This ...

There’s evil in the air and there’s thunder in sky – Meatloaf “Bat out of hell”

yum install foo [..] Error: foo conflicts with bar 

Again I have had the dubious pleasure of having dependencies between RPM-packages ending my attempt to install a single package because of a deep-rooted fear of removing core packages from production ...

OpenStack Neutron has since Mitaka had extensions available for availability zone based scheduling of routers and DHCP-agents. This is really useful if you have a cluster that is otherwise partitioned into availability zones (on the Cinder and Nova-levels.).

Setting up the server-side end

First of all, you will need:

• An OpenStack cluster, using Mitaka or newer
• Neutron-networking handling routers and/or dhcp-agents

Then you need to:

• Set availability zone in agent configuration by setting the availability_zone parameter in the [AGENT]-block of l3_agent.ini and dhcp_agent.ini
• Ensure that the availability_zone-related extensions are active within Neutron. openstack extension list, you should see availability_zone, network_availability_zone and router_availability_zone in there.

Using it

As Mirantis have explained in a pretty good blogpost, the Availability Zone-concept behaves somewhat different in nova vs. cinder vs. neutron. What it boils down to, the Neutron implementation adds some scheduling hints - helping Neutron eliminating candidates for the resources you want to set up. If a L3 agent (router) were eligible to be scheduled on any network node in the cluster, setting an availability zone hint will limit the candidates to those network nodes which remain in the availability zones listed. No availability zone hint, means that anything goes.

Using the feature is as simple as adding --availability-zone-hint AZ to any CLI calls, or set the ”availability_zone_hints”: ['AZ'] attribute when calling it through the API. As you may have noticed, the API takes an array, while the command line does not - you can however specify the parameter multiple times, with multiple values if you would like to specify multiple AZ hints over the CLI.

Does it work?

At my day job, we have been running this feature since we established our cluster right after Mitaka were released - and even though it has worked pretty well for us, we have noticed that basically no tools have caught up and adopted this feature. The only two places we can see the support for this is in the networking API (duh) and the official CLIs. This mean that it hasn’t been available in the OpenStack-dashboard, and anyone trying to use common cloud orchestration libraries against it haven’t been able to set these values.

So I’ve added support for it in Horizon 1, 2 and Shade. A coworker has done some work and fixed support for it in Gophercloud 1 , 2 and the OpenStack provider for Terraform. Another coworker has spent some time getting Heat up to speed with the functionality, but this has not yet been merged.

End users should have easier access to this feature as of the upcoming Queens-release - which will arrive two years after the initial release of the feature.

I’ve been playing with AWS for a few years, on and off. I’ve used many individual pieces of what’s available - but I haven’t yet tried to combine a larger set of feature into a thing. Until now. This blog.

This blog is powered by jekyll. Jekyll can build your blog from markdown files into a static tree that you can serve from any old webserver. With no server-side code, and a pretty small file size, it would have been a pretty good fit for the small web hotel that was included in my Internet-subscription back in 1998 (10MB for free!). Jekyll is used in Github pages, my daytime job blogs (1, 2), among other things.

After much trying and failing, I ended up with a Cloud Formation-template which assemble together a stack of resources that are used to deliver this site. The template can be invoked again to create another stack. I have kept two resources outside the template: The SSL wildcard certificate for my domain, and the Route 53 hosted DNS zone.

The stack

• S3 website-bucket
• Code Commit-repository
• Lambda function
• Cloudfront-Distribution
• Route 53 recordsets (IPv4 and IPv6)
• IAM Group
• Two IAM Roles
• Instance profile
• Extra set of Cloudfront-distribution and Route 53 recordsets for handling redirect from the www.-prefix to the proper URL.

So a decent shopping list.

The bucket

A simple S3 bucket, configured as a website bucket, using index.html as an index page, and 404.html as an error page. This bucket can be addressed directly by HTTP, and one can go by with only a bucket and DNS if HTTP is enough. However, adding CloudFront on top of the stack does provide the usual benefits of a CDN, and SSL/HTTP2-support - last but not least: lower bandwidth costs than transporting data from an AWS region for edge traffic in Europe/America.

The git-repository

CodeCommit lets you create git-repositories. You can push code to them, branch, create merge requests - the usual git-stuff. These can have triggers, which can be a Lambda-function, or a SNS-topic. I want the site to be built when I push to the repository, so I make the trigger call a Lambda-function.

The pushTrigger-function

Lambda can be used to run some piece of code without deploying it to a instance. In this case we want the function to spawn an instance which will set up an environment which can be used to build the site - clone the repository, run a script, then shut itself down (and terminate itself). The following python code does the trick.

import os
import boto3

def pushHandler(event, context):
    client = boto3.client('ec2', region_name=os.environ.get('REGION'))
    res = client.run_instances(ImageId='ami-5e29aa31',
                               InstanceType='m3.medium',
                               MinCount=1, MaxCount=1,
                               InstanceInitiatedShutdownBehavior='terminate',
                               IamInstanceProfile= { "Arn": os.environ.get('INSTANCEPROFILE') },
                               UserData="""#!/bin/bash
    yum install aws-cli git -y
    yum install -y curl gpg gcc gcc-c++ make
    gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
    curl -sSL https://get.rvm.io | bash -s stable
    usermod -a -G rvm ec2-user
    cd ~ec2-user
    sudo -u ec2-user git config --global credential.helper '!aws codecommit credential-helper $@'
    sudo -u ec2-user git config --global credential.UseHttpPath true
    sudo -u ec2-user git clone https://git-codecommit."""+os.environ.get('REGION')+""".amazonaws.com/v1/repos/"""+os.environ.get('REPONAME')+"""
    cd ~ec2-user/"""+os.environ.get('REPONAME')+"""
    ls -la
    sudo -u ec2-user bash sync.bash
    aws s3 cp /var/log/cloud-init-output.log s3://"""+os.environ.get('TARGETBUCKET')+"""/output-last.txt
    poweroff
    """)
    if len(res['Instances']) == 1:
        return {"message": "success"}
    else:
        raise Exception('I failed!')

It starts an instance, passes a bash script which will be run during first boot of the instance. “But where are the credentials?” you might say. Let’s talk about roles.

The pushTrigger-role

IAM-policies and roles may be my favourite feature of AWS. Not because dealing with permissions is in any way fun. It’s not. The AWS SDK will look for its access keys in multiple locations; It will use the access keys if provided to the function, it will use environment variables if set, and it will query the metadata-API and use those - if provided. When a resource has a role applied on itself, the AWS SDK will find the access keys on its own. If these keys are compromised, the damage is limited - the keys are only valid for a limited time - and only when used from a certain location.

  PushTriggerRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - lambda.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: "/"
      Policies:
        - PolicyName:
            Fn::Join:
            - "-"
            - - Ref: AWS::StackName
              - "PushTriggerPolicy"
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              Effect: Allow
              Action:
                - iam:PassRole
                - ec2:RunInstances
              Resource:
                - !GetAtt UpdateBucketRole.Arn
                - "arn:aws:ec2:*:*:subnet/*"
                - "arn:aws:ec2:*::image/ami-5e29aa31"
                - "arn:aws:ec2:*:*:instance/*"
                - "arn:aws:ec2:*:*:volume/*"
                - "arn:aws:ec2:*:*:security-group/*"
                - "arn:aws:ec2:*:*:network-interface/*"

The Lambda-function has access to create an ec2-instance with one specific image, and to pass one specific AWS role along to the instance. The resource section can be further stripped down to only allow the instance to live in a certain network, using a specific security group.

The UpdateBucket-role

In the pushTrigger-role above, the instance has a role passed on to it. This is for the access the ec2 instance need to deploy the site. It needs to be able to clone the CodeCommit-repository, and it needs to be able to write data to the S3 bucket. The gist of how the policy document is set up is pretty similar; A list of actions and resources. But a difference between a role for a lambda function, and a role for an ec2-instance is that the instance also need an instance-profile resource.

  UpdateBucketProfile:
    Type: "AWS::IAM::InstanceProfile"
    Properties:
      Roles:
      - Ref: UpdateBucketRole
  UpdateBucketRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
              - ec2.amazonaws.com
          Action:
            - sts:AssumeRole
      Path: "/"
      Policies:
        - PolicyName:
            Fn::Join:
            - "-"
            - - Ref: AWS::StackName
              - "UpdateBucketPolicy"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              Effect: Allow
              Action:
                - codecommit:GetRepositoryTriggers
                - s3:*
                - codecommit:GetTree
                - codecommit:GitPull
                - codecommit:BatchGetRepositories
                - codecommit:GetObjectIdentifier
                - codecommit:GetBlob
                - codecommit:GetReferences
                - codecommit:CancelUploadArchive
                - codecommit:GetCommit
                - codecommit:GetUploadArchiveStatus
                - codecommit:GetCommitHistory
                - codecommit:GetRepository
                - codecommit:GetBranch
              Resource:
                - Fn::Join:
                  - ""
                  - - "arn:aws:s3:::"
                    - Ref: Hostname
                - Fn::Join:
                  - ""
                  - - "arn:aws:s3:::"
                    - Ref: Hostname
                    - "/*"
                - Fn::Join:
                  - ":"
                  - - "arn:aws:codecommit"
                    - Ref: AWS::Region
                    - Ref: AWS::AccountId
                    - Ref: RepoName

The policy gives full access to the S3 bucket, and complete read access to the code commit repository. One can probably cut down access to some CodeCommit functions, and for example disable the ability to delete the S3 bucket to further restrict to need-to-access.

The IAM Group

When creating all these resources as part of a stack, it’s nice to think of “what kind of access is needed to work on this solution?” - and create a set of groups that provide that access. That way, you can create separate users in your AWS account that have access to only what they need. In this case, the stack consists of a group with access to the CodeCommit-repository, and to the S3 bucket. The latter is mostly for debugging and convenience.

We’ve covered the bucket that contains the site, we’ve covered the repository, the push-trigger, and the role based accesses that connect these things. But it is still not available to access. Let’s talk about CloudFront (CDN) and Route 53 (DNS).

The CloudFront Distribution

We set up a distribution that use the SSL certificate that I already have created in Amazon Certificate Manager, limit this to SNI, turn on HTTP2 and IPv6 (I can’t really see any reason not to). Since the origin is a S3 bucket, we strip query strings and cookies from any requests - allowing the cache to be as efficient as can be. All requests will be redirected to https.

A quirk here is that the origin is the website endpoint of the bucket. You can use a bucket as the backend directly - but in this configuration, the index-documents will only work on the top level. As Jekyll generates pages in subdirectories, and by default will require the index-document to be looked for - we need to use the website endpoint.

CloudFront supports invalidation requests for the cache, and if it weren’t for the fact that you currently can not grant access to creating invalidation requests on a single distribution - it’s either all distributions or none - I would allow the ec2 instance to invalidate the entire cache on deployment as a convenience.

The Route53 Recordsets

We create two recordsets in our hosted zone; one A-record, and one AAAA-record. These are both AliasTargets against the DNS-name of the cloud front distribution. AliasTargets works around the limitation that you can’t use a CNAME to alias a different domain on top level. This means that trygvevea.com can be an alias to a cloudfront-distribution - which will move around, depending on what Amazon wants to do.

The redirect-resources

We have another S3-bucket, CloudFront-Distribution and two Route53 recordsets. These are created to handle a redirect from www.trygvevea.com to trygvevea.com. The redirect itself is performed by the S3 bucket. The Cloudfront-distribution will terminate SSL, and just relay the redirect configured in the S3 bucket. The recordsets alias cloud front. This isn’t necessary if you accept that the www.-prefix would be a dead page.

The end result

Is this over-engineered?

Yes. No. Maybe. Depends.

For a simple blog? Yes. As an exercise to get hands-on experience using the AWS-toolset to create a world class scalable stack to deliver an application? Not at all.

Back in 2012, I created a small web based game called Grabitty LITE (I am still doing work on Grabitty - which I hope to release someday) - this was served from a small web server. The game was covered by a live streamer with about 3000 viewers, who wanted to play the game all at once. This managed to saturate the bandwidth of my webserver. After this happened, I moved the game over to a S3 website bucket - and never had the problem again.

This skill set, and this way of thinking / solving problems, does, and will in the future, serve me well.

Architecture Security

The exposed attack-surface is very limited. There’s basically no software that can become compromised - and if I were to forget to touch this blog for a few years, it will still be there just as I left it. The only real attack vector is the credentials of any user that has access to update the code. Again, that can become a significantly more difficult attack to execute with multi factor authentication for the user.

There are some weaknesses with the instance - it sets up everything on each run, and pipe some URL to bash as root. That is probably the worst part of this design. This can be remedied by preparing an AMI in advance that is set up with the tools you need. In that case, you don’t have to worry about attacks coming from upstream code updates.

Costs?

It’s hard to tell what this costs. My experience with AWS is that “the first shot is free” is a quite accurate description. However, the design I’ve described above limits the potential cost pretty much to bandwidth - most other things are either really cheap, or within the free tier.

• CodeCommit: Very unlikely that I will exceed the never-expiring free tier. (5 users, 50GB storage, ~10000 “git requests” per month)
• S3 storage: The blog is only a few MB large, and it’s unlikely that it will exceed 1GB. That’s $0.0245 today. Most transfer costs will be limited by caching in the CDN.
• EC2: Since the instance sets up everything every time, it takes ~15-20 minutes before it has performed a deploy. That amounts to a couple of cents for every push. This can be reduced by preparing an AMI in advanced. At one weekly blogpost, this isn’t something I’d be worried about.
• CloudFront: A page view of this blog is ~250KB or so, which would mean that 4000 page views is around 1GB - at $0.085.

So I would expect the total costs to be less than a dollar a month. The wildcard is bandwidth.

Links

• The template I’ve used is available on Github

A feature of Neutron that is probably less known, is that most network resources have a description field available in their APIs. This is pretty useful to pass some readable information about what a resource is for. This exists for everything from routers and networks to security group rules.

I noticed that for Floating IP addresses, this field was not exposed in the OpenStack dashboard - so I wrote a small patch. It is probably the OpenStack-patch that have gone through with the least effort.

• Commit: Floating IP: Expose description field in form and tables

Description field for Floating IP addresses. Coming to Horizon in Queens.

I read Tolkien’s “canon”, that is, The Hobbit, The Lord of the Rings, and The Silmarillion, around Christmas every year. So also this year.

One of the most fascinating stories in The Silmarillion is of course the story of Túrin Turambar. He is regarded as one of the major heroes of his age. At the Council of Elrond, Elrond himself lists the great men and elf-friends of old, Hador and Húrin and Túrin and Beren. But while reading through the Silmarillion, there are few among mortal men that have also added so much pain and disaster to the elves. While a great war hero, Húrin was also responsible for the slaying of the greatest hunter of the elves, Beleg Cúthalion, the strong bow. Being the war hero, he turned the people of Nargothrond away from the wisdom of their history, and even their king, and made the hidden kingdom available for the enemy. How many elves were cruelly slain or taken to captivity in Angband because of Turin’s pride? Thousands! Perhaps even tens of thousands? So how come the elves, ages later, still reckoned Túrin son of Húrin as one of the great elf-friends?

In a Nordic saga style stunt, Túrin finally slew his greatest enemy, Glaurung the great fire-breathing dragon. Glaurung had been a continous danger to all peoples of Middle-Earth, and the end of that worm was of course a great relief to all the elves, even Elrond’s ancestors, the kings of Doriath and Gondolin. Also, we must remember that the lives of the elves are different from that of men. When the elves’ bodies die, their spirits go to Mandos, where they sit in the shadow of their thought, and from where they may even return, like Glorfindel of both Gondolin and Rivendell. But when men die, they go to somewhere else, and are not bound to the world. It seems that elves are more willing to forgive and let grief rest for wisdom over time, than are men’s wont. Even the Noldor who survived the passing of the Helcaraxë forgave and united the Noldor of Fëanor’s people that left them at the burning of the ships at Losgar.

Perhaps that is one of the lessons learned from the tragic story of Túrin. From all his unhappy life, good things happened, and afterwards, the elves forgave and even mourned him and his family.

I read Tolkien’s “canon”, that is, The Hobbit, The Lord of the Rings, and The Silmarillion, around Christmas every year. So also this year.

2017 was a great year for Tolkien fans. It was the 125th anniversary of the Professor’s birth, and the 80th anniversary for the Hobbit. We also got the magnificent news that Amazon will produce a TV series based on “previously unexplored stories based on J.R.R. Tolkien’s original writings“. So what storylines would that be? A reboot of the 2001-03 trilogy is out of the question, as Peter Jackson explored and extended more than enough already. So, what do we have left? A lot! Let’s have a look.

The Lord of the Rings and its appendices tells stories in several different timelines. Long before (as in hundreds, and even thousands of years) before the main story, just before the main story (like a few decennials), parallel to the main story, and after.

One storyline could follow the ancient history of Gondor and Arnor. There are lots and lots of substories there. If I should pick one I would like to see, it would be the stories of the kings Arvedui of Arnor and  Eärnil II of Gondor, perhaps started with the Firiel incident. There are lots of exciting points to pick up there. Gondor throne heiritage politics, the war against, and the prediction of the downfall of the Witch King, the flight to Forochel, with the disastrous ship’s wreck in the ice, and the loss of the palantiri.

For the “near history” before The War of the Ring, the obvious choice would be a “The young Aragorn” series, where we could follow Aragorn in his many guises, riding with the Rohirrim, going on raids with Gondor against Harad, in and in constant conflict with Denethor. And his love life, of course, with his meeting and very long-term relationship with Arwen. And speaking of Arwen, her family story is a good storyline, with the love of Celebrían and Elrond, travelling from Lorien to Rivendell, and her abduction, and Elladan and Elrohir’s rescue of her from the orcs. Parallel to that, the story I would most love to see, would be, the story of Denethor. His tragic life is worth a season alone. Another storyline from the years just before The War of the Ring, could be Balin’s attempt to retake Moria, and build  a colony of dwarves. Lots of gore and killing of goblins to depict!

Parallel to the War of the Ring, there are a lot of things going on, that are merely mentioned in the book, and completely forgotten in the movies. The fight in Dale. The Ents’ war against the orcs after the capture of Isengard, the loss of Osgiliath and Cair Andros, to name just a few.

And of course, even after the the War of the Ring, and the Return of the King, there are stories to follow up. Aragorn’s “negotiations” for peace with his neighbouring peoples, with armed battle as alternative, supported by Eomer of Rohan. The sweet but bitter death of Aragorn and Arwen. The reign of King Eldarion.

I’m optimistic! This is going to be great!

We hope you have enjoyed the articles in our third SysAdvent season!

This is the last post in this years sysadvent. If you want to read more, we have other blog entries at our main site, our techblog, our employees have personal blogs that are aggregated at

I read Tolkien’s “Canon”, that is, The Hobbit, The Lord of the Rings, and The Silmarillion, every year about Christmas. These year, it’s even The Hobbit’s 80th Anniversary, and to celebrate, I have of course read through The Hobbit again.

So many have said so much about this book, so I’d rather show off my newest addition to my Tolkien bookshelf. This is the Swedish 1962 edition of The Hobbit, Bilbo, En Hobbits Äventyr (Bilbo, A Hobbit’s Adventure), and it has quite an interesting history.

In the 50s and 60s, Astrid Lindgren, maybe most famous for her children’s books about Pippi Longstocking, worked as an editor at the department for Children’s literature at Rabén & Sjögren, who published Tolkien’s works in Sweden. Lindgren was very interested in Tolkien’s work, and while she later denied Tolkien as an inspiration for it, she published the quite Lord of the Rings reminiscing Mio my Son in 1954, and later the world beloved classic children’s fantasy novels The Brothers Lionheart and Ronia, the Robber’s daughter.

In the early 60s Lindgren was not content* with the current Swedish translation of The Hobbit, Hompen (translation by Tore Zetterholm, 1947), and wanted to better it. So she opted for a new translation and got hold of Britt G. Hallqvist for the job. For illustrations, she contacted her friend Tove Jansson, now World famous for her Moomin Valley universe. Jansson had already had success with her Moomintrolls, and had previously made illustrations for a Swedish edition of Lewis Carrol’s classic poem Snarkjakten (The Hunting of the Snark, 1959), so a successful publication seemed likely.

Hallqvist translated, Jansson drew, Lindgren published it, and it flopped! Tolkien fans didn’t enjoy Jansson’s drawings much, and the illustrations were not used** again before 1994. By then, the 1962 version was cherished by Tove Jansson fans and Tolkien collectors over the World, and it had become quite hard to find. The 1994 edition was sold out in a jiffy. The illustrations were finally “blessed” by the Tolkien Estate, when they were used for the 2016 Tolkien Calendar.

Jansson’s illustrations were also used in the 2016 Tolkien calendar, which I’m, afraid to say, have not acquired (yet).

I was lucky and found a decent copy of the 1962 edition in a Japanese(!) bookstore on the Net. Now I LOVE this book. Its illustrations are absolutely gorgeous.

The destruction of Lake Town and the death of Smaug is my personal favourite

This book makes a great additon to my ever growing list of Hobbits.

It would be a pity to let this book stay alone without decent Janssonic company, so I searched a few weeks, was lucky again and found a nice copy of the mentioned Snarkjakten by Lewis Carrol, and an almost mint copy of the absolutely fantastic (in all meanings of that word) Swedish 1966 edition of Alice i underlandet (Alice in Wonderland). If you enjoy Alice, you will love Janssons’ illustrations, even outshining her work on The Hobbit.

Alice were later used in a lot of versions, among them, Finnish, American, British, and Norwegian editions." />Alice were later used in a lot of versions, among them, Finnish, American, British, and Norwegian editions." src="http://ingvar.blog.redpill-linpro.com/files/20171219_003943.jpg.small_-300x225.jpg" width="300" height="225" />

Janssons illustrations of Alice were later used in a lot of versions, among them, Finnish, American, British, and Norwegian editions.

For an intensely interesting read about Jansson’s artistic work on these classics: Read Olga Holownia’s essay at barnboken.net.

That’s it. Merry Christmas and happy Youletide everybody!

*) Neither was Tolkien himself. He specially disliked the translation of Elvish names into Swedish, like Esgaroth -> Snigelby (ie. Snail Town!!!). Also interesting: Svensson, Louise, Lost in Translation? – A Comparative Study of Three Swedish Translations of J.R.R. Tolkien’s ‘The Hobbit’, Lund University 2016

**) Actually, there were other versions with Jansson’s illustrations; the Finnish Hobbit Lohikäärme-vouri (The Dragon mountain) from 1973, and the updated Finnish translation in 2003. The illustrations were also used in this year’s Finnish 80th Anniversary edition of The Hobbit.

As mentioned in the previous ansible post, we use ansible quite a lot for day to day operations. While we prefer Puppet for configuration management, ansible is excellent for automation of maintenance procedures.

One such procedure is gracefully applying package upgrades, including any required reboot, of application servers. In ...

Personally, I consider the script parameter in a Vagrantfile to be a feature that is not abused enough. It’s got a lot of potential - every script can have a parameter (or several). Modifying your Vagrant use to include this gives you a more flexible and reliable ...

Sometimes you need just a little something on your network to do a simple task. Perhaps you need a small router, a firewall or load balancer. Currently, the most popular option is to deploy a little Linux server. There is a downside though, a Linux is rather heavy. It requires ...

Distro packages are a blessing that most of us take for granted (thank you and sorry package maintainers everywhere!). They make installation, maintenance and even removal of both simple and complex software a breeze.

Sometimes you disagree

But sometimes you disagree with a decision made in the distro package. ...

S2I, Source-To-Image, is a toolkit for building Docker images with minimum effort. The S2I project description describes itself like this:

Source-to-Image (S2I) is a toolkit and workflow for building reproducible Docker images from source code. S2I produces ready-to-run images by injecting source code into a Docker container and letting ...

While Varnish is most famous for its speedy caching capabilities, it is also a general swiss army knife of web serving. In the spirit of Christmas, here’s Twelve Days of Varnish Cache, or at least, twelve Varnish use cases. Read the rest of this post on Redpill Linpro’s Sysadvent calendar.

While Varnish is most famous for its speedy caching capabilities, it is also a general swiss army knife of web serving. In the spirit of Christmas, here’s Twelve Days of Varnish Cache, or at least, twelve use cases.

So, the database is slow - why?

There can be several reasons for this. Perhaps a few very heavy queries are bogging down the database. In this case, you’d typically set up slow query logging and find them in the slow.log. However, sometimes the reason is simply lots of ...