Planet Redpill Linpro

01 December 2018

Tore Anderson

Enabling IPv6 on the Huawei ME906s-158 / HP lt4132

Last year I wrote a post about my difficulties getting IPv6 to work with the Huawei ME906s-158 WWAN module. I eventually gave up and had it replaced with a module from another manufacturer.

Not long ago, though, I received an e-mail from another ME906s-158 owner who told me that for him, IPv6 worked just fine. That motivated me to brush the dust off my module and try again. This time, I figured it out! Read on for the details.

The Carrier PLMN List

The ME906s-158 comes with a built-in list of nine different PLMN profiles. This list can be managed with proprietary AT command AT^PLMNLIST, which is documented on page 209 of the module’s AT Command Interface Specification.

To interact with the AT command interface, use the option driver. More details on that here.

This is the complete factory default list:


^PLMNLIST: "00000",00000,23106,26207,23802,23806
^PLMNLIST: "20205",26801,20205,26202,26209,27201,27402,50503,54201,53001,40401,40405,40411,40413,40415,40420,40427,40430,40443,40446,40460,40484,40486,40488,40566,40567,405750,405751,405752,405753,405754,405755,405756,20404,20601,20810,21401,21670,22210,22601,23003,23415,24405,24802,27602,27801,28001,28602,28802,29340,42702,60202,62002,63001,63902,64004,64304,65101,65501,90128,23201,28401,64710,46601,42602,22005,41302,29403,50213,50219,21910,25001,27077,52505,23801,40004,42403,46692,52503,73001,24602,24705
^PLMNLIST: "26201",26201,23001,20416,23203,23207,21901,21630,23102,29702,29401,26002,20201,23431,23432
^PLMNLIST: "21403",20610,20801,20802,21403,22610,23101,23430,23433,26803,26003
^PLMNLIST: "50501",50501,50571,50572
^PLMNLIST: "22801",22801,29501
^PLMNLIST: "21407",21405,21407,23402
^PLMNLIST: "99999",24491,24001,23820
^PLMNLIST: "50502",50502 


Each ^PLMNLIST: line represents a single pre-defined PLMN profile, identified by the first MCCMNC number (in double quotes). The "26201" profile is for Deutsche Telekom, the "50501" profile is for Telstra Mobile, and so on.

The rest of the numbers on each line is a list of MCCMNCs that will use that particular profile. For example, if you have a SIM card issued by T-Mobile Netherlands (MCCMNC 20416), then the ME906s-158 will apply the Deutsche Telekom profile ("26201").

Unfortunately, the documentation offers no information on how the PLMN profiles differ and how they change the way the module work.

My provider (Telenor Norway, MCCMNC 24201) is not present in the factory default list. In that case, the module appears to use the "00000" PLMN profile («Generic») as the default, and that one disables IPv6! Clearly, Huawei haven’t read RFC 6540

In any case, this explains why I failed to make IPv6 work last year, and why it worked fine for the guy who mailed me - his provider was among those that used the Deutsche Telekom PLMN profile by default.

Modifying the Carrier PLMN List

The solution seems clear: I need to add my provider’s MCCMNC to an IPv6-capable PLMN profile. The Deutsche Telekom one would probably work, but "99999" («Generic(IPV4V6)») seems like an even more appropriate choice.

Adding MCCMNCs is done with AT^PLMNLIST=1,"$PLMNProfile","$MCCMNC", like so:



(To remove an MCCMNC, use AT^PLMNLIST=0,"$PLMNProfile","$MCCMNC" instead.)

I can now double check that the "99999" PLMN profile includes 24201 for Telenor Norway:


^PLMNLIST: "99999",24491,24001,23820,24201


To make the new configuration take effect, it appears to be necessary to reset the module. This can be done with the AT^RESET command.

Confirming that IPv6 works

It is possible to query the module directly about IPv6 support in at least two different ways:





+CGDCONT: (0-11),"IP",,,(0-2),(0-3),(0,1),(0,1),(0-2),(0,1)
+CGDCONT: (0-11),"IPV6",,,(0-2),(0-3),(0,1),(0,1),(0-2),(0,1)
+CGDCONT: (0-11),"IPV4V6",,,(0-2),(0-3),(0,1),(0,1),(0-2),(0,1)


The ^IPV6CAP: 7 response means «IPv4 only, IPv6 only and IPv4v6» (cf. page 336 of the AT Command Interface Specification), and the +CDGCONT: responses reveal that the modem is ready to configure PDP contexts using the IPv6-capable IP types. Looking good!

Of course, the only test that really matters is to connect it:

$ mmcli --modem 0,ip-type=ipv4v6
successfully connected the modem
$ mmcli --bearer 0                                                                                            
Bearer '/org/freedesktop/ModemManager1/Bearer/0'
  Status             |   connected: 'yes'
                     |   suspended: 'no'
                     |   interface: 'wwp0s20f0u3c3'
                     |  IP timeout: '20'
  Properties         |         apn: ''
                     |     roaming: 'allowed'
                     |     IP type: 'ipv4v6'
                     |        user: 'none'
                     |    password: 'none'
                     |      number: 'none'
                     | Rm protocol: 'unknown'
  IPv4 configuration |   method: 'static'
                     |  address: ''
                     |   prefix: '30'
                     |  gateway: ''
                     |      DNS: '', ''
                     |      MTU: '1500'
  IPv6 configuration |   method: 'static'
                     |  address: '2a02:2121:2c4:7105:5a2c:80ff:fe13:9208'
                     |   prefix: '64'
                     |  gateway: '::'
                     |      DNS: '2001:4600:4:fff::52', '2001:4600:4:1fff::52'
                     |      MTU: '1500'
  Stats              |          Duration: '59'
                     |    Bytes received: 'N/A'
                     | Bytes transmitted: 'N/A'


Sat 01 Dec 2018, 00:00

28 November 2018

Ingvar Hagelund

Updated packages of varnish-6.0.2 matching vmods, for el6 and el7

Recently, the Varnish Cache project released an updated upstream version 6.0.2 of Varnish Cache. This is a maintenance and stability release of varnish 6.0, which you may consider as the current “LTS” branch of varnish. I have updated the fedora rawhide package, and also updated the varnish 6.0 copr repo with packages for el6 and el7 based on the fedora package. A selection of matching vmods is also included in the copr repo.

Packages are available at

The following vmods are available:

Included in varnish-modules:

Packaged separately:

Please test and report bugs. If there is enough interest, I may consider pushing these to fedora as well.

Varnish Cache is a powerful and feature rich front side web cache. It is also very fast, and that is, fast as in powered by The Dark Side of the Force. On steroids. And it is Free Software.

Redpill Linpro is the market leader for professional Open Source and Free Software solutions in the Nordics, though we have customers from all over. For professional managed services, all the way from small web apps, to massive IPv4/IPv6 multi data center media hosting, and everything through container solutions, in-house, data center, and cloud, contact us at

by ingvar at Wed 28 Nov 2018, 10:02

19 November 2018

Magnus Hagander

PGConf.EU 2018 - the biggest one yet!

It's now almost a month since PGConf.EU 2018 in Lisbon. PGConf.EU 2018 was the biggest PGConf.EU ever, and as far as I know the biggest PostgreSQL community conference in the world! So it's time to share some of the statistics and feedback.

I'll start with some attendee statistics:

451 registered attendees 2 no-shows 449 actual present attendees

Of these 451 registrations, 47 were sponsor tickets, some of who were used by sponsors, and some were given away to their customers and partners. Another 4 sponsor tickets went unused.

Another 52 were speakers.

This year we had more cancellations than we've usually had, but thanks to having a waitlist on the conference we managed to re-fill all those spaces before the event started.

by (Magnus Hagander) at Mon 19 Nov 2018, 20:01

15 November 2018

Redpill Linpro Techblog

Local development environment for OpenShift

When developing software it makes sense to be able to work on local files, while the source code should be served from a controlled environment (a container) to prevent pollution of the developer workstation.

In this article I will describe the evolution of a development workflow for deploying applications on OpenShift. The ultimate goal is to make it possible to maximize dev/prod parity, while minimizing the idle time in the change/test cycle.


Thu 15 Nov 2018, 23:00

13 November 2018

Bjørn Ruberg

Updating wordlists from Elasticsearch

Among the many benefits of running a honeypot is gathering the credentials intruders try in order to log in. As explained in some earlier blog posts, my Cowrie honeypots are redirecting secondary connections to another honeypot running INetSim. For example, an intruder logged in to a Cowrie honeypot may use the established foothold to make […]

by bjorn at Tue 13 Nov 2018, 07:34

08 November 2018

Redpill Linpro Techblog

OpenShift with Jenkins for dev/prod parity

The 12-factor app presents a number of guidelines to achieve DevOps compliancy. One of the guidelines specifies dev/prod parity, which in OpenShift can be implemented by re-using a single container image for all steps within an applications lifecycle. Here we will describe how dev/prod parity can be achieved within OpenShift by using the pipeline support of the OpenShift BuildConfig object type.


Thu 08 Nov 2018, 23:00

05 November 2018

Magnus Hagander

Tracking foreign keys throughout a schema

I recently ran into the need with a customer to track the usage of a specific key throughout the schema. Basically, "what are all the tables and columns referencing this key, directly or indirectly". Luckily, with a little bit of catalog query, that's not hard:

WITH RECURSIVE what (tbl) AS (
   VALUES ('')
t (oid, key, constrid) AS (
 SELECT tbl::regclass::oid, conkey, NULL::oid
  FROM what INNER JOIN pg_constraint ON (contype='p' AND conrelid=tbl::regclass)
 SELECT conrelid, conkey, c.oid
 FROM pg_constraint c
 INNER JOIN t ON (c.confrelid=t.oid AND c.confkey=t.key)
 WHERE contype='f'
SELECT nspname, relname, key, ARRAY(
    SELECT attname FROM pg_attribute a WHERE a.attrelid=t.oid AND attnum=ANY(key)
INNER JOIN pg_class cl ON cl.oid=t.oid
INNER JOIN pg_namespace n ON n.oid=cl.relnamespace

The output can be similar to:

 nspname | relname | key | array 
 public  | tt      | {1} | {ttt}
 public  | foo1    | {1} | {a}
 public  | foo2    | {3} | {z}

for a single column key (tt being the table with the primary key in, and the foo1 and foo2 tables referencing it directly or through the other one), or:

 nspname | relname |  key  | array 
 public  | m1      | {1,2} | {a,b}
 public  | m2      | {1,2} | {a,b}

for a multi-column foreign key.

In this particular use-case, it was an efficient way to track down key usage where naming standards for using the key had not always been followed. And of course, we also found a couple of cases where the column had the correct name but lacked the actual FOREIGN KEY definition, but that was done by just looking at the column names.

by (Magnus Hagander) at Mon 05 Nov 2018, 13:44

11 October 2018

Ingvar Hagelund

Updated packages of varnish-6.0.1 with matching vmods, for el6 and el7

Recently, the Varnish Cache project released an updated upstream version 6.0.1 of Varnish Cache. This is a maintenance and stability release of varnish 6.0. I have updated the fedora rawhide package, and also updated the varnish 6.0 copr repo with packages for el6 and el7 based on the fedora package. A selection of matching vmods is also included in the copr repo.

Packages are available at

The following vmods are available:

Included in varnish-modules:

Packaged separately:

Please test and report bugs. If there is enough interest, I may consider pushing these to fedora as well.

Varnish Cache is a powerful and feature rich front side web cache. It is also very fast, and that is, fast as in powered by The Dark Side of the Force. On steroids. And it is Free Software.

Redpill Linpro is the market leader for professional Open Source and Free Software solutions in the Nordics, though we have customers from all over. For professional managed services, all the way from small web apps, to massive IPv4/IPv6 multi data center media hosting, and everything through container solutions, in-house, data center, and cloud, contact us at

by ingvar at Thu 11 Oct 2018, 09:24

22 August 2018

Magnus Hagander

Updates about upcoming conferences

Summer vacation times are over. Well, for some of us at least, clearly some are still lucky enough to be off, which is showing itself a bit (see below). But as both conference organisation and participation throughout the rest of the year is starting to be clear, I figure it's time to share some updates around different ones.

Postgres Open SV

First of all - if you haven't already, don't forget to register for Postgres Open SV in San Francisco in two weeks time! Registration for the main US West Coast/California PostgreSQL community conference will close soon, so don't miss your chance. I'm looking forward to meeting many old and new community members there.

PostgreSQL Conference Europe

Next up after Postgres Open will be, the main European PostgreSQL community conference of 2018. The planning for this years conference is at full speed, but unfortunately we are slightly behind. In particular, we were supposed to be notifying all speakers today if they were accepted or not, and unfortunately our program committee are a bit behind schedule on this one. We had over 200 submissions this year which makes their work even bigger than usual. But speakers will be receiving their notification over the upcoming couple of days.

Hopefully once all speakers have been able to confirm their attendance, we will also have a schedule out soon. Until then, you can always look at the list of accepted talks so far. This list is dynamically updated as speakers get approved and confirm their talks.

We have already sold approximately half of the tickets that we have available this year, so if you want to be sure to get your spot, we strongly recommend that you register as soon as you can! And if you want to attend the training sessions, you should hurry even more as some are almost sold out!


Work on the program committee of PGConf.ASIA has also been going on over the late summer, and is mostly done! The schedule is not quite ready yet, but expected out shortly. You can look forward to a very interesting lineup of speakers, so if you are in the Asian region, I strongly recommend keeping an eye out for when the registration opens, and join us in Tokyo!


As has been announced, PostgreSQL Europe will once again run a FOSDEM PGDay next to the big FOSDEM conference in Brussels in February next year. We hope to also run our regular booth and developer room during FOSDEM, but those are not confirmed yet (more info to come). The Friday event, however, is fully confirmed. Of course not open for registration yet, but we'll get there.

Nordic PGDay

Nordic PGDay has been confirmed for March 19th next year. The format will be similar to previous years, and we will soon announce the location. For now, mark your calendars to make sure you don't double book! And rest assured, the conference will take place somewhere in the Nordics!

Usergroups and PGDays

Then there are a number of smaller events of course. Next week, I will speak at the Prague PostgreSQL Meetup. We should be kicking off the Stockholm usergroup. PDXPUG runs a PGDay in Portland in September (which I unfortunately won't be able to attend). In general, it seems like usergroups are starting to get going again after the summer break, so check with your local group(s) what's happening!

by (Magnus Hagander) at Wed 22 Aug 2018, 12:39

13 August 2018

Redpill Linpro Techblog

Getting started with Terraform

This post will show you how to get started with using Terraform, an open-source tool that lets you build your infrastructure using code.


Mon 13 Aug 2018, 22:00

09 August 2018

Ingvar Hagelund

Packages of varnish-6.0 with matching vmods, for el6 and el7

Some time ago, the Varnish Cache project released a new upstream version 6.0 of Varnish Cache. I updated the fedora rawhide package a few weeks ago. I have also built a copr repo with varnish packages for el6 and el7 based on the fedora package. A selection of matching vmods is also included.

Packages are available at

The following vmods are available:

Included in varnish-modules:

Packaged separately:

Please test and report bugs. If there is enough interest, I may consider pushing these to fedora as well.

by ingvar at Thu 09 Aug 2018, 13:47

19 March 2018

Bjørn Ruberg

X-Forwarded-For DDoS

A discussion forum of one of Redpill Linpro‘s customers has been under attack lately, through a number of DoS and DDoS variants. Today’s attack strain was of the rather interesting kind, as one of its very distinctive identifiers was a suspicious, not to say ridiculous, amount of IP addresses in the incoming X-Forwarded-For HTTP header. […]

by bjorn at Mon 19 Mar 2018, 21:55

25 February 2018

Redpill Linpro Techblog

Layer 3 routing on Cumulus Linux MLAG

We build our network in order to simultaneously achieve high availability and maximum utilisation of available bandwidth. To that end, we are using Multi-Chassis Link Aggregation (MLAG) between our data centre switches running Cumulus Linux and our firewall cluster

Sun 25 Feb 2018, 23:00

16 February 2018

Tore Anderson

IPv6 roaming in the United States

I spent some time in the United States last month. Equipped with SIM cards from both Tele2 Sweden (MCCMNC 24007) and Telenor Norway (MCCMNC 24201), I set out to test IPv6 while roaming, as I usually do while abroad.

The previous posts in this series are:

There were two mobile networks that I was able to register in and test: AT&T (3G/4G, no 2G) and T-Mobile (2G/3G/4G). The test results are found below.

I could also see three other 4G networks show up in a network scan (Caprock Cellular, Sprint and Verizon), but none of those were available to me. Presumably neither Tele2 nor Telenor have roaming arrangements with any of those operators.

AT&T - MCCMNC 310410

Home PLMN Tech IPV6 PDP context IPV4V6 PDP context
Tele2 SE 3G Fails (cause code 33) IPv4-only (cause code 50)
Tele2 SE 4G Works perfectly Works perfectly
Telenor NO 3G Fails (cause code 33) IPv4-only (cause code 50)
Telenor NO 4G Works perfectly Works perfectly

T-Mobile - MCCMNC 310260

Home PLMN Tech IPV6 PDP context IPV4V6 PDP context
Tele2 SE 2G Fails (cause code 33) IPv4-only (cause code 50)
Tele2 SE 3G Fails (cause code 33) IPv4-only (cause code 50)
Tele2 SE 4G Works perfectly Works perfectly
Telenor NO 2G Fails (cause code 33) IPv4-only (cause code 50)
Telenor NO 3G Fails (cause code 33) IPv4-only (cause code 50)
Telenor NO 4G Works perfectly Works perfectly

3GPP cause code #33 means «requested service option not subscribed», while cause code #50 means «PDP type IPv4 only allowed». The latter doesn’t indicate a fatal error, as I do automatically get a working IPv4-only connection to the Internet.

The fact that IPv6 consistently fails on 2G/3G is in all likelihood due to Tele2/Telenor’s HLR removing the IPv6 capabilities from my subscriber profile before transmitting it to AT&T/T-Mobile’s vSGSN.

Tele2 and Telenor do this to forestall the possible IPv6-related failures described in RFC 7445 sections 3 and 6. In this case, it is in all likelihood an unnecessary precaution, considering that both AT&T and T-Mobile are known to have deployed IPv6 to their own customers.

Fri 16 Feb 2018, 00:00

07 February 2018

Magnus Hagander

What does it mean to be on the board of PostgreSQL Europe

With the upcoming elections in PostgreSQL Europe, I'm excited to see that we have more candidates than ever before. But during the FOSDEM conference we just finished in Brussels, that also lead to a fairly large number of people who asked me the simple question "what does it actually mean to be on the board of PostgreSQL Europe". So I think it's time to summarize that for both those standing for election, and for the members of the organisation in general.

For a TL; DR; version, being on the board basically means a lot of administrative work :) But read on for some details.

by (Magnus Hagander) at Wed 07 Feb 2018, 16:49

01 February 2018

Redpill Linpro Techblog

Mule 4 SDK custom connector example

Mule 4 will be released soon. Along with Mule 4 a new Mule SDK is relased which can be used to extend the functionality of Mule with custom modules. The Mule SDK replaces Devkit for developing connectors.

Documentation on the Mule SDK can so far ...

Thu 01 Feb 2018, 23:00

28 January 2018

Trygve Vea

Extending the OpenStack - Part I: APIs

If OpenStack is something you have heard of, but aren’t too familiar with, you probably think “virtual servers” - or maybe “automatic server provisioning”. That is in my opinion not what OpenStack is about. It is about being able to integrate with “everything infrastructure-related”, and enforcing access in a secure manner. OpenStack is a very good starting point for achieving that, giving you virtual servers, networking, storage management, and at it’s core: identity management. Installing virtual servers and provisioning resources for them are well and good, but that’s just an appliance. The fun part comes when we can Build Things.

One of the OpenStack-projects named Keystone, provide identity management. This component is responsible for dealing with the service catalogue and API endpoints in your OpenStack cloud. User authentication happen through this, and users usually have roles assigned on one or more projects. When an end user want to create a virtual server, the user passes an authentication token to the compute service API (nova), which it verifies with keystone before performing any actions. Through this mechanism, user authentication and authorization is nicely decoupled from the services that end up delivering infrastructure. It is meant to be used as a mean to extend OpenStack, and it’s pretty much straight forward.


  • An OpenStack-solution
  • Ability to write a small web service application, for example using something like Flask. Can be done in any programming language, but python give you access to a bunch of SDKs for speaking with the OpenStack APIs which can come in handy.
  • Something you want to implement

So if you have those three things, you’re good to go. If you don’t have those - but wanna play around and see what you can do, I can push you in the right direction to get started.

Preparing OpenStack

The two first steps of adding the API to the service and endpoint catalogue aren’t mandatory - but it’s a polite thing to do. By doing so, users of the cloud can see the URLs where your API is available.

## As an admin user, perform the following steps:

# openstack service create myservice --name myservice
| Field   | Value                            |
| enabled | True                             |
| id      | b81a620dfdb54592b97c67816d00fc43 |
| name    | myservice                        |
| type    | myservice                        |

# openstack endpoint create myservice admin
# openstack endpoint create myservice public
# openstack endpoint create myservice internal

# openstack endpoint list --service myservice
| ID                               | Region | Service Name | Service Type | Enabled | Interface | URL                                           |
| 1fbbbcc57108422596e07ab4bd713631 | None   | myservice    | myservice    | True    | internal  |     |
| 25edd1c28178462980e90bce2b8140f2 | None   | myservice    | myservice    | True    | public    |     |
| 78b2090f1df4450fb4544abf1eee842a | None   | myservice    | myservice    | True    | admin     |     |

The more important step is to create a service user, which will be used to perform administrative tasks - such as validating authentication tokens.

# openstack user create --email myservice@localhost --password myservicesecretpassword myservice
| Field               | Value                            |
| domain_id           | default                          |
| email               | myservice@localhost              |
| enabled             | True                             |
| id                  | ab3e5a51a87a44898c8543e0a254bd3b |
| name                | myservice                        |
| options             | {}                               |
| password_expires_at | None                             |

# openstack role add --project services --user myservice admin

At this point, we have our service user.

The web service

You need to handle the API calls, and you need to verify authenticated requests.

Validating a token can be done with the following code:

from keystoneauth1.identity import v3
from keystoneauth1 import session
from keystoneclient.v3 import client

# Authenticate with the service user, and create a keystone client
auth = v3.Password(auth_url='',
sess = session.Session(auth=auth)
keystone = client.Client(session=sess)

# Validator-class that can easily be extended with functions
# that test role/project-access. self.validate contains a hash
# with information about this.
class Validator:
    def __init__(self, token):
        self.validate = keystone.tokens.validate(token)

Tokens are passed between OpenStack-services using the x-auth-token http-header. In Flask, picking this out in your service routes can be as simple as request.headers.get('x-auth-token').

Using the validator-class above in a route can be

from flask import Flask
from flask import request
from flask import abort
from flask import jsonify

from validator import Validator
from thing import Thing

# A helper function to wrap the raised exception in the case 
# of an invalid token.
def create_validator():
        v = Validator(request.headers.get('x-auth-token'))
    return v

# Routes below.
# The /-route can be used as a liveliness and ready probe.
def check():
    return jsonify({"myservice": 0.1})

# Authenticated route.  Passing the validator object to the
# constructor of Thing(), and calling a method - returning its
# output as json.
def thing_list():
    return jsonify(Thing(create_validator()).list_things())

What is Thing()? It’s a class we’ve made to split application logic from routes. At the route level, we have ensured that the request is authenticated now we want to do something.

class Thing:
    def __init__(self, validator):
        self.v = validator

    def list_things(self):
        return { 'things': ['apple', 'chair', 'banana', 'television'] }

The Thing class takes a validator in its constructor for two reasons.

  1. It’s nice to authenticate the request early on.
  2. You get access to role and project-information from within the class, and can make decisions based on that.

A slightly more complete version of this skeleton is available on GitHub.

Testing it out

Loading our liveliness route works fine without any authentication.

$ curl
  "myservice": 0.1

Our web service for listing things requires authentication:

$ curl
<title>403 Forbidden</title>
<p>You don't have the permission to access the requested resource. It is either read-protected or not readable by the server.</p>

So we need to issue a token. We can do that by calling the identity API manually, or if one have set up a CLI to our cloud - one can use that to issue a token.

$ openstack token issue
| Field      | Value                            |
| expires    | 2018-01-27T23:23:16+0000         |
| id         | 39465e6698b7409cb926a870cef1e5bb |
| project_id | dafe2c2a6d6e4bfc9146ea975b2897e0 |
| user_id    | 830ec37bc8774738904731a7bd55cb58 |

Inserting the token id as the http header ‘x-auth-token’ will authenticate requests against our OpenStack cloud, including our custom built web service:

$ curl -H 'x-auth-token: 39465e6698b7409cb926a870cef1e5bb'
  "things": [

The web service has the information it needs about the calling user by validating the token. Which user, domain, project, and what roles the user have.

At this point one are free to implement whatever one pleases. One can integrate with some in-house system that shows invoice information for a customer - or may just be to create some helper APIs that simply re-uses the provided token to chain-run a series of other OpenStack APIs.

If you want to provide a user friendly web-interface for your APIs, stay tuned for Part II.


by Trygve Vea at Sun 28 Jan 2018, 00:00

25 January 2018

Redpill Linpro Techblog

Jenkinsfile to Anypoint platform and Artifactory

Jenkins Pipeline is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins. I will be using this to deploy a Mule application to Anypoint Platform Runtime Manager and store the delivery in ...

Thu 25 Jan 2018, 23:00

23 January 2018

Sigurd Urdahl

Making it right all the time - or that time I disagreed with a distro package about file permissions

This was originally posted to my employer Redpill Linpro's annual pre-Christmas Sysadvent calendar of tech blogging

Distro packages are a blessing that most of us take for granted (thank you and sorry package maintainers everywhere!). They make installation, maintenance and even removal of both simple and complex software a breeze.

Sometimes you disagree

But sometimes you disagree with a decision made in the distro package. Or maybe just have a use-case that is specific enough that the generalized setup provided by the distro package collides with your needs.

A sometime

A while ago I had one such “sometime”. Let me explain: a lot of our customers here at Redpill Linpro Managed Services use Mysql. Among them is one of my customers, - the official Norwegian national education and career portal. From time to time their developers need to export data from Mysql to CSV file (that is comma seperated values). The best way to do get CSV from Mysql is to use SELECT .. INTO OUTFILE. The resultset from such an SELECT is dropped as a CSV file in the import/export directory configured with the variable secure_file_priv. runs on Ubuntu, and in Ubuntu the mysql-server-5.5 package creates a directory intended for this usage - /var/lib/mysql-files - in it’s ‘‘postinst’’ script:
# Lets extract the scripts (and then some) from the mysql-server-5.5 package:
root@dbserver:/tmp/mysql-server-5.5_control# dpkg --control /var/cache/apt/archives/mysql-server-5.5_5.5.58-0ubuntu0.14.04.1_amd64.deb
# And pick put the parts relevant to the import/export directory from the post-install script
root@dbserver:/tmp/mysql-server-5.5_control# grep mysql_filesdir DEBIAN/postinst
if [ ! -d "$mysql_filesdir" -a ! -L "$mysql_filesdir" ]; then mkdir "$mysql_filesdir"; fi
chown -R mysql:mysql $mysql_filesdir
chmod 700 $mysql_filesdir
This code ensures a directory like this exist:
drwx------ 2 mysql mysql 4096 nov.  23 10:09 /var/lib/mysql-files/
Since my customer wants to use SELECT .. INTO OUTFILE to export data it is not surprising that they also want to access the produced files with another user than mysql. (And as a sysadmin I strongly prefer them to not use “my” system users, but rather “their” application users. And since they are smart they want me to prefer them to not use “my” system users.) Basically the customer wants their developers to ‘just be able to access the files’, and I want to facilitate that in a simple, effective and non-convoluted way.
Non-convoluted is important since we work in teams in Redpill Linpro Managed Services. The easier it is for my co-workers to see how something works on a server, the better services we can provide. So we prefer to avoid implementing things in complicated ways with possibly suprising behaviour.

Chmod as a solution

For our customer and us the preferred solution is to have the directory accesible by their application user. An easy way to do that is to add the application user to the mysql group and chmod it so that it is accessible for members of the mysql group.
chmod 0770 /var/lib/mysql-files

Chmod reverted

But as you probably noticed above - the postinst script will revert this. After every run of the postinst script - thus efter every time the mysql-server-5.5 package is upgraded - the directory will revert to being only accessible to the mysql user.

Sudo as a non-solution

Giving the customer sudo access to copy the files acting as the mysql user is easy, but it forces the developers to remember that they can’t just copy the files as they are used to, but have to use sudo. It also opens us up to security problems with globbing arguments in our sudo rules, unless we force them to decide on filenames in advance. A helper script to which we provide sudo access would remove the security concerns, but still leaves us with a cumbersome way of getting where we want. And for all we know maybe they don’t really want to copy the file, but to do something to it? The developers just want to access their exported data, and since the developers are our friends I want to help them do just what they want - without hassle.

Different ways to get our will

We just need to make sure that it is we, not the postinst script, that get our will when it comes to these permissions. Preferably in an elegant way.
Luckily there are lots of ways to solve this on a Unix-based system. Some ways not as good as others. Some not so elegant.

An ensured failure

The most straighforward solution is to remember to run the chmod command every time the package is upgraded. That is of course not going to happen, it will be forgotten. And with all these computers around there isn’t really any reason why we should handle this manually.

Be a puppet master

At Redpill Linpro Managed Services we use Puppet for configuration management, so we could let Puppet handle this. Most likely noone will try to “SELECT .. INTO OUTFILE” in the time between the upgrade and the next Puppet run, right? Well, probably not, but I bet that the one time when it is important to get access those data really quickly will be between a package upgrade and a Puppet run.
Cron is is the daemon who execute scheduled commands on your Linux system.
So we can just make it right all the time (well, once a minute that is) with the help of cron
File: /etc/cron.d/fix_permissions_mysql-files

# Ensure correct permissions for /var/lib/mysql-files directory
* * * * * root /bin/chmod 0770 /var/lib/mysql-files
This of course is overkill, in the sense that a package upgrade will set the wrong permissions a handful times a year and this cron job will try to fix it a little over half a million times a year. The strain this puts on the server should be negliable, but the strain such a solution puts on the mental health and well being of many sysadmins probably isn’t. And that is not really elegant, is it?
rm  /etc/cron.d/fix_permissions_mysql-files

Immediate (i)notification and reaction

A more elegant, and thus attractive, solution would be if we could trigger a chmod when there is a change to the directory, not when there is a change to what time it is. And the Linux kernel provides us with a way to do just that through the inotify subsystem. Inotify is a facility that makes it possible to do things when something changes in a filesystem. Luckily, since throwing together small C programs using kernel system calls isn’t modus operandi for most of us, there are user space tools that makes it possible to utilise the inotify functionality quite easily in shell scripts (and if your weapon of choice is C, Haskell, ocaml, python, ruby or basically anything else, most likely there is a library to help you getting inotified).
One of the tools available is incron:
[..] the “inotify cron” system. It consist of a daemon and a table manipulator. You can use it a similar way as the regular cron. The difference is that the inotify cron handles filesystem events rather than time periods
So this gives ut a method to revert the postinst script’s revert of our chmod immediately. This script will do the actual chmod (and leave a trace of this in the syslog)
File: /usr/local/bin/


logger -t incron -p daemon.notice "Chmodding /var/lib/mysql-files back to 0770"
/bin/chmod 0770 /var/lib/mysql-files
And this is the configuration for incron
File: /etc/incron.d/RL_fix_permissions_mysql-files

# Ensure correct permissions for /var/lib/mysql-files directory
# chmod if metadata changes, but disable while doing this, so we don't loop.

/var/lib/mysql-files IN_ATTRIB,IN_NO_LOOP /usr/local/bin/
The IN_NO_LOOP ensures that our chmodding of the file doesn’t trigger a new inotification and thus a new chmod and a new inotification and a new chmod and ….
But every time a file inside the directory is changed we also change the metadata for the directory, and then we trigger a chmod. A completely unecessary chmod. And that is not really elegant, is it?
rm /usr/local/bin/
rm /etc/incron.d/RL_fix_permissions_mysql-files

React instead of random act

It would be better if we could react only when our chmod is reverted by the postinst script. It would be neat if we could hook into the package handling. And we can!
We can configure apt to trigger a script after running dpkg:
File: /etc/apt/apt.conf.d/98RL_fix_permissions_mysql-files

# Ensure correct permissions after dpkg invocations
DPkg::Post-Invoke { "/bin/chmod -v 0770 /var/lib/mysql-files"; };
Now every time dpkg runs we will make sure the permissions are as we want them. Let me install an old favourite to demonstrate
root@host:~# aptitude install cowsay
The following NEW packages will be installed:
0 packages upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Setting up cowsay (3.03+dfsg1-6) ...
mode of ‘/var/lib/mysql-files’ changed from 0700 (rwx------) to 0770 (rwxrwx---)
You noticed that last line, right?
“But this too is a sub-optimal solution, isn’t it?”, you’re probably thinking, and you are absolutely right. It will run chmod every time dpkg has been invoked from apt, which is quite often. And if we for some reason or not download the package and install it directly with dpkg our chmod will not be ran. The latter could be handled if we configured this with a ‘‘post-invoke’’ hook in dpkg.conf instead of treating it as apt configuration, but we would still only solve this for ourselves, and only on the systems where we add this configuration. And that is not really elegant, is it?
It is tempting to just
rm /etc/apt/apt.conf.d/98RL_fix_permissions_mysql-files
but I’m not doing that until we have a better solution in place.

Isn’t this really a bug?

The elegant and community oriented solution is to remove the cause for this behaviour, and make sure we don’t need to go in and chmod the directory ever again.
Rembemer, the problem comes from this code in the postinst script:
if [ ! -d "$mysql_filesdir" -a ! -L "$mysql_filesdir" ]; then mkdir "$mysql_filesdir"; fi
chown -R mysql:mysql $mysql_filesdir
chmod 700 $mysql_filesdir
and could be resolved if we move the chmod (and that chown that could lead to the same kind of problems) inside the if statement:
if [ ! -d "$mysql_filesdir" -a ! -L "$mysql_filesdir" ]; then
mkdir "$mysql_filesdir"
chown -R mysql:mysql $mysql_filesdir
chmod 700 $mysql_filesdir
To that end I have proposed this as a bug to the Ubuntu package maintainers, and hopefully I can soon
rm /etc/apt/apt.conf.d/98RL_fix_permissions_mysql-files
You can track the progress of the bug and my patch at Ubuntu Launchpad

Closing remarks

I hope you have enjoyed this little walk through my sometime episode. If you learned something new that’s great. All of the approaches above can prove useful for solving problems completely different from this one. You mighe have suggestions to other ways of solving this, and that’s great - one of my favourite things with Unix-based systems is that there are so many ways you can solve a problem, and thus so many new things you still can learn even after having been in the game for a couple of decades.
From all of us (including my old friend) to all of you:
< A very merry Christmas! >
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||

by Sigurd Urdahl ( at Tue 23 Jan 2018, 09:55

22 January 2018

Redpill Linpro Techblog

Mulesoft Runtime on Raspberry with Kubernetes

The Raspberry Pi 3 is the third generation Raspberry Pi, on this I will be installing Mulesoft enterprise runtime with latest Java 8 running inside Kubernetes. The pods will register themselves with Anypoint platform runtime manager. This ...

Mon 22 Jan 2018, 23:00

21 January 2018

Redpill Linpro Techblog

yum shell - bat out of dependency hell

There’s evil in the air and there’s thunder in sky – Meatloaf “Bat out of hell”

yum install foo [..] Error: foo conflicts with bar 

Again I have had the dubious pleasure of having dependencies between RPM-packages ending my attempt to install a single package because of a deep-rooted fear of removing core packages from production ...

Sun 21 Jan 2018, 23:00

Trygve Vea

Availability Zones in Neutron

OpenStack Neutron has since Mitaka had extensions available for availability zone based scheduling of routers and DHCP-agents. This is really useful if you have a cluster that is otherwise partitioned into availability zones (on the Cinder and Nova-levels.).

Setting up the server-side end

First of all, you will need:

  • An OpenStack cluster, using Mitaka or newer
  • Neutron-networking handling routers and/or dhcp-agents

Then you need to:

  • Set availability zone in agent configuration by setting the availability_zone parameter in the [AGENT]-block of l3_agent.ini and dhcp_agent.ini
  • Ensure that the availability_zone-related extensions are active within Neutron. openstack extension list, you should see availability_zone, network_availability_zone and router_availability_zone in there.

Using it

As Mirantis have explained in a pretty good blogpost, the Availability Zone-concept behaves somewhat different in nova vs. cinder vs. neutron. What it boils down to, the Neutron implementation adds some scheduling hints - helping Neutron eliminating candidates for the resources you want to set up. If a L3 agent (router) were eligible to be scheduled on any network node in the cluster, setting an availability zone hint will limit the candidates to those network nodes which remain in the availability zones listed. No availability zone hint, means that anything goes.

Using the feature is as simple as adding --availability-zone-hint AZ to any CLI calls, or set the ”availability_zone_hints”: ['AZ'] attribute when calling it through the API. As you may have noticed, the API takes an array, while the command line does not - you can however specify the parameter multiple times, with multiple values if you would like to specify multiple AZ hints over the CLI.

Does it work?

At my day job, we have been running this feature since we established our cluster right after Mitaka were released - and even though it has worked pretty well for us, we have noticed that basically no tools have caught up and adopted this feature. The only two places we can see the support for this is in the networking API (duh) and the official CLIs. This mean that it hasn’t been available in the OpenStack-dashboard, and anyone trying to use common cloud orchestration libraries against it haven’t been able to set these values.

So I’ve added support for it in Horizon 1, 2 and Shade. A coworker has done some work and fixed support for it in Gophercloud 1 , 2 and the OpenStack provider for Terraform. Another coworker has spent some time getting Heat up to speed with the functionality, but this has not yet been merged.

End users should have easier access to this feature as of the upcoming Queens-release - which will arrive two years after the initial release of the feature.

by Trygve Vea at Sun 21 Jan 2018, 00:00

17 January 2018

Sigurd Urdahl

yum shell - bat out of dependency hell

There's evil in the air and there's thunder in sky
(Meatloaf "Bat out of hell")

# yum install foo
Error: foo conflicts with bar

Again I have had the pleasure of having dependencies between RPM-packages ending my attempt to install a single package with a suggestion of removing core packages. I think this most often happen with Mysql or Percona packages, but I am sure MariaDB will be able to give you the same situation too. It's not the first time I have been here..

[root@ftp01-prod ~]# yum install Percona-Server-client-57
Loaded plugins: fastestmirror, priorities
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package Percona-Server-client-57.x86_64 0:5.7.20-19.1.el7 will be installed
--> Processing Dependency: Percona-Server-shared-57 for package: Percona-Server-client-57-5.7.20-19.1.el7.x86_64
--> Running transaction check
---> Package Percona-Server-shared-57.x86_64 0:5.7.20-19.1.el7 will be installed
--> Processing Dependency: Percona-Server-shared-compat-57 for package: Percona-Server-shared-57-5.7.20-19.1.el7.x86_64
--> Running transaction check
---> Package Percona-Server-shared-compat-57.x86_64 0:5.7.20-19.1.el7 will be installed
--> Processing Conflict: Percona-Server-shared-compat-57-5.7.20-19.1.el7.x86_64 conflicts Percona-Server-shared-56
--> Finished Dependency Resolution
Error: Percona-Server-shared-compat-57 conflicts with Percona-Server-shared-56-5.6.38-rel83.0.el7.x86_64
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest
[root@ftp01-prod ~]#

So if I want to install Percona-Server-client-57 I have to install Percona-Server-shared-compat-57 too, and that I can't because of the already installed Percona-Server-shared-56. OK, so I will just remove Percona-Server-shared-56 and then install Percona-Server-shared-compat-57 before doing the install I first tried to do:

[root@ftp01-prod ~]# yum remove Percona-Server-shared-56
Dependencies Resolved

 Package                           Arch            Version                      Repository                 Size
 Percona-Server-shared-56          x86_64          5.6.38-rel83.0.el7           @percona-release          3.4 M
Removing for dependencies:
 MySQL-python                      x86_64          1.2.5-1.el7                  @centos_os                284 k
 fail2ban                          noarch          0.9.7-1.el7                  @epel                     0.0
 fail2ban-sendmail                 noarch          0.9.7-1.el7                  @epel                      11 k
 perl-DBD-MySQL                    x86_64          4.023-5.el7                  @centos_os                323 k
 postfix                           x86_64          2:2.10.1-6.el7               @centos_os                 12 M
 redhat-lsb-core                   x86_64          4.1-27.el7.centos.1          @anaconda                  45 k

Transaction Summary
Remove  1 Package (+6 Dependent packages)

Installed size: 16 M
Is this ok [y/N]:

Very much not OK. I'd like to at least keep things with descriptions like this

Description : The Linux Standard Base (LSB) Core module support
: provides the fundamental system interfaces, libraries,
: and runtime environment upon which all conforming
: applications and libraries depend.

The problem seems to be that Postfix needs which is provided by both Percona-Server-shared-56 and Percona-Server-shared-compat-57. So I just need to swap the former for the latter, and then I can run my original install.

OK, so I both want to remove a package and install a package. And I want to do it at the same time, so that I don't have to remove things like redhat-lsb-core. Did you notice the use of the word transaction in "Transaction Summary" from yum? A transaction is actually what I want. Luckily yum provides a way of doing this, and have probably done since forever, but I didn't learn about it till today. And as so many times before, it is a shell that solves our problems:

[root@ftp01-prod ~]# yum shell
Loaded plugins: fastestmirror, priorities
> remove Percona-Server-shared-56
> install Percona-Server-shared-compat-57
Loading mirror speeds from cached hostfile
> run
--> Running transaction check
---> Package Percona-Server-shared-56.x86_64 0:5.6.38-rel83.0.el7 will be erased
---> Package Percona-Server-shared-compat-57.x86_64 0:5.7.20-19.1.el7 will be installed
--> Finished Dependency Resolution

 Package                             Arch       Version                Repository            Size
 Percona-Server-shared-compat-57     x86_64     5.7.20-19.1.el7        percona-release      1.2 M
 Percona-Server-shared-56            x86_64     5.6.38-rel83.0.el7     @percona-release     3.4 M

Transaction Summary
Install  1 Package
Remove   1 Package

Total download size: 1.2 M
Is this ok [y/d/N]:

Yes, very much thank you! And then finally:

[root@ftp01-prod ~]# yum install Percona-Server-client-57
Dependencies Resolved

 Package                        Arch         Version                  Repository             Size
 Percona-Server-client-57       x86_64       5.7.20-19.1.el7          percona-release       7.2 M
Installing for dependencies:
 Percona-Server-shared-57       x86_64       5.7.20-19.1.el7          percona-release       747 k

Transaction Summary
Install  1 Package (+1 Dependent package)

Total download size: 7.9 M
Installed size: 41 M
Is this ok [y/d/N]:y

Done and done :-)

by Sigurd Urdahl ( at Wed 17 Jan 2018, 23:00

14 January 2018

Trygve Vea

This blog: A serverless experiment

I’ve been playing with AWS for a few years, on and off. I’ve used many individual pieces of what’s available - but I haven’t yet tried to combine a larger set of feature into a thing. Until now. This blog.

This blog is powered by jekyll. Jekyll can build your blog from markdown files into a static tree that you can serve from any old webserver. With no server-side code, and a pretty small file size, it would have been a pretty good fit for the small web hotel that was included in my Internet-subscription back in 1998 (10MB for free!). Jekyll is used in Github pages, my daytime job blogs (1, 2), among other things.

After much trying and failing, I ended up with a Cloud Formation-template which assemble together a stack of resources that are used to deliver this site. The template can be invoked again to create another stack. I have kept two resources outside the template: The SSL wildcard certificate for my domain, and the Route 53 hosted DNS zone.

The stack

  • S3 website-bucket
  • Code Commit-repository
  • Lambda function
  • Cloudfront-Distribution
  • Route 53 recordsets (IPv4 and IPv6)
  • IAM Group
  • Two IAM Roles
  • Instance profile
  • Extra set of Cloudfront-distribution and Route 53 recordsets for handling redirect from the www.-prefix to the proper URL.

So a decent shopping list.

The bucket

A simple S3 bucket, configured as a website bucket, using index.html as an index page, and 404.html as an error page. This bucket can be addressed directly by HTTP, and one can go by with only a bucket and DNS if HTTP is enough. However, adding CloudFront on top of the stack does provide the usual benefits of a CDN, and SSL/HTTP2-support - last but not least: lower bandwidth costs than transporting data from an AWS region for edge traffic in Europe/America.

The git-repository

CodeCommit lets you create git-repositories. You can push code to them, branch, create merge requests - the usual git-stuff. These can have triggers, which can be a Lambda-function, or a SNS-topic. I want the site to be built when I push to the repository, so I make the trigger call a Lambda-function.

The pushTrigger-function

Lambda can be used to run some piece of code without deploying it to a instance. In this case we want the function to spawn an instance which will set up an environment which can be used to build the site - clone the repository, run a script, then shut itself down (and terminate itself). The following python code does the trick.

import os
import boto3

def pushHandler(event, context):
    client = boto3.client('ec2', region_name=os.environ.get('REGION'))
    res = client.run_instances(ImageId='ami-5e29aa31',
                               MinCount=1, MaxCount=1,
                               IamInstanceProfile= { "Arn": os.environ.get('INSTANCEPROFILE') },
    yum install aws-cli git -y
    yum install -y curl gpg gcc gcc-c++ make
    gpg --keyserver hkp:// --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
    curl -sSL | bash -s stable
    usermod -a -G rvm ec2-user
    cd ~ec2-user
    sudo -u ec2-user git config --global credential.helper '!aws codecommit credential-helper $@'
    sudo -u ec2-user git config --global credential.UseHttpPath true
    sudo -u ec2-user git clone https://git-codecommit."""+os.environ.get('REGION')+""""""+os.environ.get('REPONAME')+"""
    cd ~ec2-user/"""+os.environ.get('REPONAME')+"""
    ls -la
    sudo -u ec2-user bash sync.bash
    aws s3 cp /var/log/cloud-init-output.log s3://"""+os.environ.get('TARGETBUCKET')+"""/output-last.txt
    if len(res['Instances']) == 1:
        return {"message": "success"}
        raise Exception('I failed!')

It starts an instance, passes a bash script which will be run during first boot of the instance. “But where are the credentials?” you might say. Let’s talk about roles.

The pushTrigger-role

IAM-policies and roles may be my favourite feature of AWS. Not because dealing with permissions is in any way fun. It’s not. The AWS SDK will look for its access keys in multiple locations; It will use the access keys if provided to the function, it will use environment variables if set, and it will query the metadata-API and use those - if provided. When a resource has a role applied on itself, the AWS SDK will find the access keys on its own. If these keys are compromised, the damage is limited - the keys are only valid for a limited time - and only when used from a certain location.

    Type: AWS::IAM::Role
        Version: '2012-10-17'
        - Effect: Allow
          - sts:AssumeRole
      Path: "/"
        - PolicyName:
            - "-"
            - - Ref: AWS::StackName
              - "PushTriggerPolicy"
            Version: '2012-10-17'
              Effect: Allow
                - iam:PassRole
                - ec2:RunInstances
                - !GetAtt UpdateBucketRole.Arn
                - "arn:aws:ec2:*:*:subnet/*"
                - "arn:aws:ec2:*::image/ami-5e29aa31"
                - "arn:aws:ec2:*:*:instance/*"
                - "arn:aws:ec2:*:*:volume/*"
                - "arn:aws:ec2:*:*:security-group/*"
                - "arn:aws:ec2:*:*:network-interface/*"

The Lambda-function has access to create an ec2-instance with one specific image, and to pass one specific AWS role along to the instance. The resource section can be further stripped down to only allow the instance to live in a certain network, using a specific security group.

The UpdateBucket-role

In the pushTrigger-role above, the instance has a role passed on to it. This is for the access the ec2 instance need to deploy the site. It needs to be able to clone the CodeCommit-repository, and it needs to be able to write data to the S3 bucket. The gist of how the policy document is set up is pretty similar; A list of actions and resources. But a difference between a role for a lambda function, and a role for an ec2-instance is that the instance also need an instance-profile resource.

    Type: "AWS::IAM::InstanceProfile"
      - Ref: UpdateBucketRole
    Type: AWS::IAM::Role
        Version: '2012-10-17'
        - Effect: Allow
            - sts:AssumeRole
      Path: "/"
        - PolicyName:
            - "-"
            - - Ref: AWS::StackName
              - "UpdateBucketPolicy"
            Version: "2012-10-17"
              Effect: Allow
                - codecommit:GetRepositoryTriggers
                - s3:*
                - codecommit:GetTree
                - codecommit:GitPull
                - codecommit:BatchGetRepositories
                - codecommit:GetObjectIdentifier
                - codecommit:GetBlob
                - codecommit:GetReferences
                - codecommit:CancelUploadArchive
                - codecommit:GetCommit
                - codecommit:GetUploadArchiveStatus
                - codecommit:GetCommitHistory
                - codecommit:GetRepository
                - codecommit:GetBranch
                - Fn::Join:
                  - ""
                  - - "arn:aws:s3:::"
                    - Ref: Hostname
                - Fn::Join:
                  - ""
                  - - "arn:aws:s3:::"
                    - Ref: Hostname
                    - "/*"
                - Fn::Join:
                  - ":"
                  - - "arn:aws:codecommit"
                    - Ref: AWS::Region
                    - Ref: AWS::AccountId
                    - Ref: RepoName

The policy gives full access to the S3 bucket, and complete read access to the code commit repository. One can probably cut down access to some CodeCommit functions, and for example disable the ability to delete the S3 bucket to further restrict to need-to-access.

The IAM Group

When creating all these resources as part of a stack, it’s nice to think of “what kind of access is needed to work on this solution?” - and create a set of groups that provide that access. That way, you can create separate users in your AWS account that have access to only what they need. In this case, the stack consists of a group with access to the CodeCommit-repository, and to the S3 bucket. The latter is mostly for debugging and convenience.

We’ve covered the bucket that contains the site, we’ve covered the repository, the push-trigger, and the role based accesses that connect these things. But it is still not available to access. Let’s talk about CloudFront (CDN) and Route 53 (DNS).

The CloudFront Distribution

We set up a distribution that use the SSL certificate that I already have created in Amazon Certificate Manager, limit this to SNI, turn on HTTP2 and IPv6 (I can’t really see any reason not to). Since the origin is a S3 bucket, we strip query strings and cookies from any requests - allowing the cache to be as efficient as can be. All requests will be redirected to https.

A quirk here is that the origin is the website endpoint of the bucket. You can use a bucket as the backend directly - but in this configuration, the index-documents will only work on the top level. As Jekyll generates pages in subdirectories, and by default will require the index-document to be looked for - we need to use the website endpoint.

CloudFront supports invalidation requests for the cache, and if it weren’t for the fact that you currently can not grant access to creating invalidation requests on a single distribution - it’s either all distributions or none - I would allow the ec2 instance to invalidate the entire cache on deployment as a convenience.

The Route53 Recordsets

We create two recordsets in our hosted zone; one A-record, and one AAAA-record. These are both AliasTargets against the DNS-name of the cloud front distribution. AliasTargets works around the limitation that you can’t use a CNAME to alias a different domain on top level. This means that can be an alias to a cloudfront-distribution - which will move around, depending on what Amazon wants to do.

The redirect-resources

We have another S3-bucket, CloudFront-Distribution and two Route53 recordsets. These are created to handle a redirect from to The redirect itself is performed by the S3 bucket. The Cloudfront-distribution will terminate SSL, and just relay the redirect configured in the S3 bucket. The recordsets alias cloud front. This isn’t necessary if you accept that the www.-prefix would be a dead page.

The end result

Drawing of the flow when updating or visiting the site

Is this over-engineered?

Yes. No. Maybe. Depends.

For a simple blog? Yes. As an exercise to get hands-on experience using the AWS-toolset to create a world class scalable stack to deliver an application? Not at all.

Back in 2012, I created a small web based game called Grabitty LITE (I am still doing work on Grabitty - which I hope to release someday) - this was served from a small web server. The game was covered by a live streamer with about 3000 viewers, who wanted to play the game all at once. This managed to saturate the bandwidth of my webserver. After this happened, I moved the game over to a S3 website bucket - and never had the problem again.

This skill set, and this way of thinking / solving problems, does, and will in the future, serve me well.

Architecture Security

The exposed attack-surface is very limited. There’s basically no software that can become compromised - and if I were to forget to touch this blog for a few years, it will still be there just as I left it. The only real attack vector is the credentials of any user that has access to update the code. Again, that can become a significantly more difficult attack to execute with multi factor authentication for the user.

There are some weaknesses with the instance - it sets up everything on each run, and pipe some URL to bash as root. That is probably the worst part of this design. This can be remedied by preparing an AMI in advance that is set up with the tools you need. In that case, you don’t have to worry about attacks coming from upstream code updates.


It’s hard to tell what this costs. My experience with AWS is that “the first shot is free” is a quite accurate description. However, the design I’ve described above limits the potential cost pretty much to bandwidth - most other things are either really cheap, or within the free tier.

  • CodeCommit: Very unlikely that I will exceed the never-expiring free tier. (5 users, 50GB storage, ~10000 “git requests” per month)
  • S3 storage: The blog is only a few MB large, and it’s unlikely that it will exceed 1GB. That’s $0.0245 today. Most transfer costs will be limited by caching in the CDN.
  • EC2: Since the instance sets up everything every time, it takes ~15-20 minutes before it has performed a deploy. That amounts to a couple of cents for every push. This can be reduced by preparing an AMI in advanced. At one weekly blogpost, this isn’t something I’d be worried about.
  • CloudFront: A page view of this blog is ~250KB or so, which would mean that 4000 page views is around 1GB - at $0.085.

So I would expect the total costs to be less than a dollar a month. The wildcard is bandwidth.

by Trygve Vea at Sun 14 Jan 2018, 00:00

07 January 2018

Trygve Vea

Floating IP Description in Horizon

A feature of Neutron that is probably less known, is that most network resources have a description field available in their APIs. This is pretty useful to pass some readable information about what a resource is for. This exists for everything from routers and networks to security group rules.

I noticed that for Floating IP addresses, this field was not exposed in the OpenStack dashboard - so I wrote a small patch. It is probably the OpenStack-patch that have gone through with the least effort.

Description field for Floating IP addresses. Coming to Horizon in Queens.

by Trygve Vea at Sun 07 Jan 2018, 00:00

29 December 2017

Ingvar Hagelund

J.R.R. Tolkien: The Silmarillion

I read Tolkien’s “canon”, that is, The Hobbit, The Lord of the Rings, and The Silmarillion, around Christmas every year. So also this year.

One of the most fascinating stories in The Silmarillion is of course the story of Túrin Turambar. He is regarded as one of the major heroes of his age. At the Council of Elrond, Elrond himself lists the great men and elf-friends of old, Hador and Húrin and Túrin and Beren. But while reading through the Silmarillion, there are few among mortal men that have also added so much pain and disaster to the elves. While a great war hero, Húrin was also responsible for the slaying of the greatest hunter of the elves, Beleg Cúthalion, the strong bow. Being the war hero, he turned the people of Nargothrond away from the wisdom of their history, and even their king, and made the hidden kingdom available for the enemy. How many elves were cruelly slain or taken to captivity in Angband because of Turin’s pride? Thousands! Perhaps even tens of thousands? So how come the elves, ages later, still reckoned Túrin son of Húrin as one of the great elf-friends?

In a Nordic saga style stunt, Túrin finally slew his greatest enemy, Glaurung the great fire-breathing dragon. Glaurung had been a continous danger to all peoples of Middle-Earth, and the end of that worm was of course a great relief to all the elves, even Elrond’s ancestors, the kings of Doriath and Gondolin. Also, we must remember that the lives of the elves are different from that of men. When the elves’ bodies die, their spirits go to Mandos, where they sit in the shadow of their thought, and from where they may even return, like Glorfindel of both Gondolin and Rivendell. But when men die, they go to somewhere else, and are not bound to the world. It seems that elves are more willing to forgive and let grief rest for wisdom over time, than are men’s wont. Even the Noldor who survived the passing of the Helcaraxë forgave and united the Noldor of Fëanor’s people that left them at the burning of the ships at Losgar.

Perhaps that is one of the lessons learned from the tragic story of Túrin. From all his unhappy life, good things happened, and afterwards, the elves forgave and even mourned him and his family.

by ingvar at Fri 29 Dec 2017, 16:27

25 December 2017

Ingvar Hagelund

J.R.R. Tolkien: The Lord of the Rings

I read Tolkien’s “canon”, that is, The Hobbit, The Lord of the Rings, and The Silmarillion, around Christmas every year. So also this year.

2017 was a great year for Tolkien fans. It was the 125th anniversary of the Professor’s birth, and the 80th anniversary for the Hobbit. We also got the magnificent news that Amazon will produce a TV series based on “previously unexplored stories based on J.R.R. Tolkien’s original writings“. So what storylines would that be? A reboot of the 2001-03 trilogy is out of the question, as Peter Jackson explored and extended more than enough already. So, what do we have left? A lot! Let’s have a look.

The Lord of the Rings and its appendices tells stories in several different timelines. Long before (as in hundreds, and even thousands of years) before the main story, just before the main story (like a few decennials), parallel to the main story, and after.

One storyline could follow the ancient history of Gondor and Arnor. There are lots and lots of substories there. If I should pick one I would like to see, it would be the stories of the kings Arvedui of Arnor and  Eärnil II of Gondor, perhaps started with the Firiel incident. There are lots of exciting points to pick up there. Gondor throne heiritage politics, the war against, and the prediction of the downfall of the Witch King, the flight to Forochel, with the disastrous ship’s wreck in the ice, and the loss of the palantiri.

For the “near history” before The War of the Ring, the obvious choice would be a “The young Aragorn” series, where we could follow Aragorn in his many guises, riding with the Rohirrim, going on raids with Gondor against Harad, in and in constant conflict with Denethor. And his love life, of course, with his meeting and very long-term relationship with Arwen. And speaking of Arwen, her family story is a good storyline, with the love of Celebrían and Elrond, travelling from Lorien to Rivendell, and her abduction, and Elladan and Elrohir’s rescue of her from the orcs. Parallel to that, the story I would most love to see, would be, the story of Denethor. His tragic life is worth a season alone. Another storyline from the years just before The War of the Ring, could be Balin’s attempt to retake Moria, and build  a colony of dwarves. Lots of gore and killing of goblins to depict!

Parallel to the War of the Ring, there are a lot of things going on, that are merely mentioned in the book, and completely forgotten in the movies. The fight in Dale. The Ents’ war against the orcs after the capture of Isengard, the loss of Osgiliath and Cair Andros, to name just a few.

And of course, even after the the War of the Ring, and the Return of the King, there are stories to follow up. Aragorn’s “negotiations” for peace with his neighbouring peoples, with armed battle as alternative, supported by Eomer of Rohan. The sweet but bitter death of Aragorn and Arwen. The reign of King Eldarion.

I’m optimistic! This is going to be great!

by ingvar at Mon 25 Dec 2017, 07:00

24 December 2017

Redpill Linpro Sysadvent

Thank you for visiting our SysAdvent Blog!

We hope you have enjoyed the articles in our third SysAdvent season!

This is the last post in this years sysadvent. If you want to read more, we have other blog entries at our main site, our techblog, our employees have personal blogs that are aggregated at

Sun 24 Dec 2017, 23:00

Ingvar Hagelund

J.R.R. Tolkien: The Hobbit

I read Tolkien’s “Canon”, that is, The Hobbit, The Lord of the Rings, and The Silmarillion, every year about Christmas. These year, it’s even The Hobbit’s 80th Anniversary, and to celebrate, I have of course read through The Hobbit again.

So many have said so much about this book, so I’d rather show off my newest addition to my Tolkien bookshelf. This is the Swedish 1962 edition of The Hobbit, Bilbo, En Hobbits Äventyr (Bilbo, A Hobbit’s Adventure), and it has quite an interesting history.

In the 50s and 60s, Astrid Lindgren, maybe most famous for her children’s books about Pippi Longstocking, worked as an editor at the department for Children’s literature at Rabén & Sjögren, who published Tolkien’s works in Sweden. Lindgren was very interested in Tolkien’s work, and while she later denied Tolkien as an inspiration for it, she published the quite Lord of the Rings reminiscing Mio my Son in 1954, and later the world beloved classic children’s fantasy novels The Brothers Lionheart and Ronia, the Robber’s daughter.

In the early 60s Lindgren was not content* with the current Swedish translation of The Hobbit, Hompen (translation by Tore Zetterholm, 1947), and wanted to better it. So she opted for a new translation and got hold of Britt G. Hallqvist for the job. For illustrations, she contacted her friend Tove Jansson, now World famous for her Moomin Valley universe. Jansson had already had success with her Moomintrolls, and had previously made illustrations for a Swedish edition of Lewis Carrol’s classic poem Snarkjakten (The Hunting of the Snark, 1959), so a successful publication seemed likely.

Hallqvist translated, Jansson drew, Lindgren published it, and it flopped! Tolkien fans didn’t enjoy Jansson’s drawings much, and the illustrations were not used** again before 1994. By then, the 1962 version was cherished by Tove Jansson fans and Tolkien collectors over the World, and it had become quite hard to find. The 1994 edition was sold out in a jiffy. The illustrations were finally “blessed” by the Tolkien Estate, when they were used for the 2016 Tolkien Calendar.

Jansson’s illustrations were also used in the 2016 Tolkien calendar, which I’m, afraid to say, have not acquired (yet).

I was lucky and found a decent copy of the 1962 edition in a Japanese(!) bookstore on the Net. Now I LOVE this book. Its illustrations are absolutely gorgeous.




The destruction of Lake Town and the death of Smaug are my personal favourites

The destruction of Lake Town and the death of Smaug is my personal favourite

It makes a great additon to my ever growing list of Hobbits.

This book makes a great additon to my ever growing list of Hobbits.

It would be a pity to let this book stay alone without decent Janssonic company, so I searched a few weeks, was lucky again and found a nice copy of the mentioned Snarkjakten by Lewis Carrol, and an almost mint copy of the absolutely fantastic (in all meanings of that word) Swedish 1966 edition of Alice i underlandet (Alice in Wonderland). If you enjoy Alice, you will love Janssons’ illustrations, even outshining her work on The Hobbit.

For an intensely interesting read about Jansson’s artistic work on these classics: Read Olga Holownia’s essay at

That’s it. Merry Christmas and happy Youletide everybody!

*) Neither was Tolkien himself. He specially disliked the translation of Elvish names into Swedish, like Esgaroth -> Snigelby (ie. Snail Town!!!). Also interesting: Svensson, Louise, Lost in Translation? – A Comparative Study of Three Swedish Translations of J.R.R. Tolkien’s ‘The Hobbit’, Lund University 2016

**) Actually, there were other versions with Jansson’s illustrations; the Finnish Hobbit Lohikäärme-vouri (The Dragon mountain) from 1973, and the updated Finnish translation in 2003. The illustrations were also used in this year’s Finnish 80th Anniversary edition of The Hobbit.

by ingvar at Sun 24 Dec 2017, 07:00

23 December 2017

Redpill Linpro Sysadvent

Using Ansible for system updates

As mentioned in the previous ansible post, we use ansible quite a lot for day to day operations. While we prefer Puppet for configuration management, ansible is excellent for automation of maintenance procedures.

One such procedure is gracefully applying package upgrades, including any required reboot, of application servers. In ...

Sat 23 Dec 2017, 23:00