When we introduced the network configuration using Ansible and AWX at a customer, we gradually extended the configuration scope. Over time, more and more configuration got added into the configuration pool and this lead to longer and longer run-times for the playbooks.

While the job-execution got really simple by using AWX instead of the plain CLI method for Ansible, the time to finish drew heavily on that benefit.

A complete job-run over the network infrastructure took at least ...

We’ve been to FOSDEM in Belgium this year. A couple of Many of Redpill’s Agents (so called: Consultants) have made the trip to Belgium to join the annual conference taking place at Université libre de Bruxelles (ULB).

This year the conference was held during the first weekend in February, at the 1st and 2nd.

For those who don’t know: FOSDEM is a free software developer conference where you can attend talks about various different tools, processes and ideas and ...

If you are using Azure PostgreSQL and have upgraded your client side libpq to version 12 (which can happen automatically for example if you use the PostgreSQL apt repositories), you may see connection attempts fail with symptoms like:

$ psql -hZZZZZZ.postgres.database.azure.com -dpostgres -UXXXXX_dba@ZZZ-db01
psql: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

With no log information whatsoever available. This can happen if your client is in a Kerberos environment and has valid Kerberos credentials (which can be verified with the klist command). In this case, PostgreSQL 12 will attempt to negotiate GSSAPI encryption with the server, and it appears the connection handler in Azure PostgreSQL is unable to handle this and just kills the connection.

When running the same thing against a local PostgreSQL server prior to version 12, a message like the following will show up in the log:

2020-02-20 10:48:08 CET [35666]: [2-1] client=1.2.3.4 FATAL:  unsupported frontend protocol 1234.5680: server supports 2.0 to 3.0

This is a clear indicator of what's going on, but unfortunately the information isn't always available when connecting to a managed cloud service, such as Azure PostgreSQL. The hard error from Azure also prevents libpq from retrying without GSSAPI encryption, which is what would happen when connecting to a regular PostgreSQL backend or for example through pgbouncer.

The fix/workaround? Disable GSSAPI encryption in the client:

$ export PGGSSENCMODE=disable
$ psql -hZZZZZZ.postgres.database.azure.com -dpostgres -UXXXXX_dba@ZZZ-db01
Password for user XXXXX_dba@ZZZ-db01:
psql (11.6 (Ubuntu 11.6-1.pgdg16.04+1), server 9.5.20)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

postgres=>

If you have this type of issue, it's probably worth putting this environment variable in your startup scripts. It can also be set using the gssencmode parameter as part of the connection string, in environments where this is more convenient.

While the AWS console gives you a nice point and click interface, and really helps you explore the vast service catalog of AWS, the use of the CLI should not be neglected.

Some of the advantages of the CLI:

• Reusable, can the same command multiple times, perhaps with slight modification for quickly creating multiple instances of similar resources.
• Reproducible, can run the same command, to reproduce exactly the same kind of resource as has been created before.
...

This year we intend to upgrade all the routers in our network backbone to a brand new platform based on open networking devices from Edge-Core running Cumulus Linux. In this post - replete with pictures - we will take a close look at the new routers and the topology of our new network backbone.

Why upgrade?

Our network backbone is today based on the Juniper MX 240 routing platform. Each of them occupy 5

Sometimes I need to quickly remove one of our data centre switches from production. Typically this is done in preparation of scheduled maintenance, but it could also be necessary if I suspect that it is misbehaving in some way. Recently I stumbled across an undocumented feature in Cumulus Linux that significantly simplified this procedure.

The key is the file /cumulus/switchd/ctrl/shutdown_linkdown. This file does normally not exist, but if it is created with the contents 1, it changes ...

Some time back in 2019, Varnish Software and the Varnish Cache project released a new LTS upstream version 6.0.5 of Varnish Cache. I updated the fedora 29 package, and added a modularity stream varnish:6.0 for fedora 31. I have also built el6 and el7 packages for the varnish60 copr repo, based on the fedora package. A snapshot of matching varnish-modules, and a selection of other misc vmods are also available.

Packages may be fetched from https://copr.fedorainfracloud.org/coprs/ingvar/varnish60/.

vmods included in varnish-modules:
vmod-bodyaccess
vmod-cookie
vmod-header
vmod-saintmode
vmod-tcp
vmod-var
vmod-vsthrottle
vmod-xkey

vmods packaged separately:
vmod-blobsynth
vmod-rfc6052
vmod-querystring
vmod-blobdigest
vmod-memcached
vmod-digest
vmod-geoip
vmod-basicauth
vmod-curl
vmod-uuid

A version of this text was presented as the lecture for Creation Day, Holmlia Church, 2019-06-19.

[Introduction: Excerpts from The Ainulindalë accompagnied by folk music improvisation on organ and violin]

Some of you may know that I’m a Tolkien enthusiast. I give away Tolkien books on my own birthday. Sometimes I feel like going door-to-door with The Lord of the Rings and its gospel; *Ding* *dong* Goood Morning! Did you know that Tolkien’s books may change your life? (What is that? Yes, Good Morning in all meanings of that expression, thank you). Now, as I can present this before you here in church, I probably won’t have to.

For many, the language professor John Ronald Reuel Tolkien only means his books The Hobbit and The Lord of the Rings. Some have even not read any of his books, but may have seen films with strange wizards, orcs, elves, and a good deal of fighting. But this is Creation Day, so in this small lecture there will be less orcs, Gandalf, Bilbo, Frodo, and the Ring. Instead I will talk a bit about Tolkien’s thoughts on God as the Creator, his Creation, and Men, as God’s sub-creators.

In the introduction, we heard lines from Tolkien’s creation myth, the Ainulindalë, that is, The Music of the Ainur: God gives the Ainur, that is, his angels, a theme to improvise over. Then he lets the song unfold, and when the song is finished, he shows them what they have sung. He says: Ëa! Let this world be! And the song is the World. When the song is sung, its life is the history of the World unfolding. Isn’t that just incredibly beautiful?

The Ainur enjoys the high mountains and the deep valleys, and the sea, and the elves, and the trees, and the flowers, and the animals they have sung about. But in the middle of the harmonies, Melkor’s dissonance is heard. The mightiest of the angels sets his own thoughs above God’s thoughs, and wants to rule, and in pride, fill the void with subjects under his dominion. But what first sounds like destroying God’s theme, is itself taken up in the song, and makes it even more fulfilled.

In the motion of the sea, the song is most clearly heard. Now further in the Ainulindalë, we hear how Melkor in his rebellion makes extreme cold, freezing the water, and uncontrolled heat, boiling it to steam. But in the midst of the freezing cold, we get beautiful snowflakes, and from the heat and steam, there are clouds and life-giving rain. Tolkien shows us that even when the Creation is challenged by evil, God can always turn the evil to something good in the end. God doesn’t want evil to happen, but when it happens, hope is always there. And when Time comes to its end, and the final chord is sung, we may see that hope and faith in the middle of evil, gave the most beautiful music played in God’s honor.

Those reading Tolkien’s books will soon observe his joy of nature. The books are swarming of life. There are bushes and flowers and trees of all kinds, and everything has value; from pipe weed to oak trees. There are insects and foxes, eagles and ravens, bears and elephants, and even the simplest flower may be important and save lives. Tolkien loved the landscape were he grew up, with meadows, woods, small rivers, hills, and the other crossroads with an inn with good beer. But he also loved the snow in the high mountains, the mighty large rivers, the deep cloven valleys, the sun in the sky, the stars of Elbereth, thunder claps and storm over mountains, and the wind of the sea. There is a lot of God’s creation wihin Tolkien’s Middle Earth.

Tolkien criticize those who says that fairy-tales and fantastic stories are just escapism, and have nothing to do with reality. In one of his most known lectures, he turns this upside-down: In a World of evil, somebody wants to tell that there is Light in the darkness and make stories of Hope. What is wrong with that? And Escaping means getting from prison to freedom. That is a Good Thing!

Tolkien says that one of the most important features of a fairy-tale, is to experience anew the small and large wonders of the World. When in The Lord of the Rings we read about Frodo coming to the elven wood Lothlórien; For the first time in his life, he realizes what a Tree really is. He feels the bark, the trunk, the branches, and the leaves. They are full of color and smell and sound and Life. The Ents, the sheperds of the Trees, that watches over the woods of Fangorn Forest, sing and talk to their trees, and mourns them when they die. Trees are so much more than something that’s just there. Go and watch and smell and enjoy the life of the trees in the grove you pass on the way to work every day.

Aragorn and his rangers have watched over Hobbiton and Bree, and held evil forces away, without the people living there knowing about this. When you get to live in freedom and peace, remember in thankfulness who built the peace, and who is watching over it. After reading about the faithful friendship between Sam and Frodo, find again the joy in the relations to your friends. When the story about Aragorn and Arwen’s long awaited marriage is told, or Faramir’s spontanous proposal to Eowyn, or Rose and Sam’s happy wedding, renew the joy of your partner, and delight in your choice. Fantasy and fairy-stories gives us the opportunity to recovery, to find again the fantastic from the domestic.

Man is special in God’s creation. Tolkien meant that God has put a spark of his creating power within us, making us more than animals. In telling myth and stories, we make new things that weren’t there before. We are sub-creators.

When we make new stories, or tell or retell myths, they are of course not the Truth. But as the light is spread through a prism making a spectrum of colors, our stories are created from the True Light. Thus, Myth and stories may show us a glimpse of the Truth. This is good, and not only because they come of God’s true Light. When light is broken into colors, they are no longer perfect white: Some becomes red, some blue, some yellow, some violet. But in this spectrum of colors, something new has been created, that earlier was not. And it has value in itself.

Unfortunately, we can not all write like Tolkien. There are those that try, and you get … things … like Game of Thrones and other garbage. But when we use our talents, we are sub-creators too. If that is being a priest, or taking pictures, or making music, or doing accounting, or sports, or teaching, or baking, or programming, or carpentry; That is fullfilment of the potential of God’s light through us. With all our strange shapes and colors, we bring fourth a richness that would not exist without us. And though our sub-creation is not perfect, it still has its source in God’s unbroken bright light.

I read Tolkien’s “Canon”, that is, The Hobbit, The Lord of the Rings, and The Silmarillion, every year about Christmas. So also this year.

What was Bilbo up to after he left Hobbiton, and until Frodo met him again in Rivendell. While there are few explicit mentions, there are some cues that we may explore.

First, when Bilbo was packing and leaving Bag End after his long expected party, he was again going with dwarves. They are not named, but it seems likely that they are the same who delivered goods from Dale to the party, and have probably stayed in the guest rooms of Bag End since. No dwarves were mentioned at the party, and I guess they would have, had they been present. So Bilbo goes with the dwarves, and as he tells to Frodo later, he goes on his last journey all the way to The Mountain, that is, Erebor, and to Dale. He comes too late to visit his old friend Balin – he had left for Moria. Then Bilbo returned to Rivendell. No more is told about his travels back, though it is easy to speculate. When he left the Mountain, returning homewards the previous time, he was invited to the halls of his friend the Elven King, that is Thranduil of Mirkwood/Greenwood the Great, but gently rejected the offer. It would be natural to pay him a visit on his second return westwards. The elves would give him safe journey through the forest. By legend, he was probably well known to the Beornings too, and I would guess he got a safe and well escorted journey back over the Misty Mountains.

Back in Rivendell, Frodo got acquainted to Aragorn the Ranger. If Bilbo uses one year on his journey to Erebor and back to Rivendell, he is 112, and Aragorn would be at the frisky age of 71. While Aragorn is often away, helping in the watch of the Shire, or on errantry for Gandalf, like going hunting for Gollum, he is probably often back in Rivendell. Bilbo speaks of him as his good friend, the Dùnadan, and when they sneak away in the Hall of Fire, it sounds like it is not the first time they redraw to look over his verses.

So what has Bilbo done over the next 16 years? Like the Asbjørnsen and Moe, or the Grimm brothers, he has literally collected fairy tales. The Red Book of Westmarch that goes from Bilbo and Frodo to Sam at the end of the story, contains several long stories and verse translated from Elvish by Bilbo. Within this frame, this is what we may call the Silmarillion Traditions. And based on this, he may have written quite a few verses of his own. When he recites for Erestor and other elves in the Hall of Fire, it is clear that this is not the first time he does this, though he does not often get asked for a second hearing.

Finally in Rivendell, Bilbo got his own parlor. After Frodo’s reception dinner, and all the singing and reciting of verse in the Hall of Fire, we are told that Frodo and Bilbo retreats to Bilbo’s room, where they can exit to a veranda that looks out over a garden and the river. We know Bilbo was always fond of his garden, and it is nice to know that the elves of Rivendell provided him with one just outside his room.

If I had to grow old in solitude, I’d like a room at the Rivendell Resort for the Resting, please.

I read Tolkien’s “Canon”, that is, The Hobbit, The Lord of the Rings, and The Silmarillion, every year about Christmas. So also this year.

There is said so much about this book already, so instead of adding more non-interesting chatter to the World, I’d rather again this year show off my latest acquisition to my Hobbit collection: The annotated Hobbit:

This is a true treasure for Hobbit fans. In addition to the actual text, it contains tons of information, like the contemporary context for the book, different versions and updates among the many editions, possible inspirations and related texts, fun facts, illustrations from Hobbit variants of the World,

notes on the meaning of names and places, and so much more.

It even contains the full text of The Quest of Erebor, that was meant as an appendix for The Lord of the Rings, but was cut before its release.

This is the revised and expanded version of The Annotated Hobbit. We owe great thanks to Douglas A. Anderson who must have gone to extremes while researching for this edition.

This book is greatly recommended for those who enjoy being immersed in footnotes, distractions, and fun facts while reading. Ah, that would be the typical Tolkien fan, I guess.

It is another great addition to my ever growing list of Hobbits.

(Update: This post has been updated to reflect changing backup tool from WAL-E to WAL-G. WAL-G is a more modern and faster implementation of cloud backups for postgreSQL)

Several Redpill Linpro customers are now in the kubernetes way of delivery. Kubernetes has changed the way they work, and is acting as an effective catalyst empowering their developers. For these customers, the old-school way of running PostgreSQL is becoming a bit cumbersome:

The typical PostgreSQL installation has been based on bare ...

Domain Name System Security Extensions (DNSSEC) is a technology that uses cryptographic signatures to make the Domain Name System (DNS) tamper-proof, safeguarding against DNS hijacking. If your ISP or network operator cares about your online security, their DNS servers will validate DNSSEC signatures for you. DNSSEC is widely deployed: here in Scandinavia, about 80% of all DNS lookups are subject to DNSSEC validation (source). Wondering whether or not your DNS server validates DNSSEC signatures? www.dnssec-or-not.com ...

I recently needed to remove a couple of decommissioned switches from one of our data centres. This turned out to be quite an ordeal. The reason? The ill-conceived way the rack mount brackets used by most data centre switches are designed. In this post, I will use plenty of pictures to explain why that is, and propose a simple solution on how the switch manufacturers can improve this in future.

Rack switch mounting 101

(Note: this is a repost of an article from the Redpill Linpro techblog.)

We have all done it. When SSH asks us this familiar question:

$ ssh redpilllinpro01.ring.nlnog.net
The authenticity of host 'redpilllinpro01.ring.nlnog.net (2a02:c0:200:104::1)' can't be established.
ECDSA key fingerprint is SHA256:IM/o2Qakw4q7vo9dBMLKuKAMioA7UeJSoVhfc5CYsCs.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

…we just answer yes - without bothering to verify the fingerprint shown.

Many of us will even automate answering yes to this question by adding StrictHostKeyChecking accept-new to our ~/.ssh/config file.

Sometimes, SSH will be more ominous:

$ ssh redpilllinpro01.ring.nlnog.net
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:IM/o2Qakw4q7vo9dBMLKuKAMioA7UeJSoVhfc5CYsCs.
Please contact your system administrator.
Add correct host key in /home/tore/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/tore/.ssh/known_hosts:448
ECDSA host key for redpilllinpro01.ring.nlnog.net has changed and you have requested strict checking.
Host key verification failed.

This might make us stop a bit and ask ourselves: «Has a colleague re-provisioned this node since the last time I logged in to it?»

Most of the time, the answer will be: «Yeah, probably», followed by something like sed -i 448d ~/.ssh/known_hosts to get rid of the old offending key. Problem solved!

These are all very understandable and human ways of dealing with these kinds of repeated questions and warnings. SSH certainly does «cry wolf» a lot! Let us not think too much about what happens that one time someone actually is «DOING SOMETHING NASTY», though…

Another challenge occurs when maintaining a large number of servers using automation software like Ansible. Manually answering questions about host keys might be impossible, as the automation software likely needs to run entirely without human interaction. The cop out way of ensuring it can do so is to disable host key checking altogether, e.g., by adding StrictHostKeyChecking no to the ~/.ssh/config file.

DNSSEC-validated SSH host key fingerprints in DNS

Fortunately a better way of securely verifying SSH host keys exists - one which does not require lazy and error-prone humans to do all the work.

This is accomplished by combining DNS Security Extensions (DNSSEC) with SSHFP resource records.

To make use of this approach, you will need the following:

1. The SSH host keys published in DNS using SSHFP resource records
2. Valid DNSSEC signatures on the SSHFP resource records
3. A DNS recursive resolver which supports DNSSEC
4. A stub resolver that is configured to request DNSSEC validation
5. A SSH client that is configured to look for SSH host keys in DNS

I will elaborate on how to implement each of these requirements in the sections below.

1. Publishing SSHFP host keys in DNS

The ssh-keygen utility provides an easy way to generate the correct SSHFP resource records based on contents of the /etc/ssh/ssh_host_*_key.pub files. Run it on the server like so:

$ ssh-keygen -r $(hostname --fqdn).
redpilllinpro01.ring.nlnog.net. IN SSHFP 1 1 5fca087a7c3ebebbc89b229a05afd450d08cf9b3
redpilllinpro01.ring.nlnog.net. IN SSHFP 1 2 cdb4cdaf7734df343fd567e0cab92fd6ac5f2754bfef797826dfd4bcf90f0baf
redpilllinpro01.ring.nlnog.net. IN SSHFP 2 1 613f389a36cf33b67d9bd69e381785b275e101cd
redpilllinpro01.ring.nlnog.net. IN SSHFP 2 2 8a07b97b96d826a7d4d403424b97a8ccdb77105b527be7d7be835d02fdb9cd58
redpilllinpro01.ring.nlnog.net. IN SSHFP 3 1 3e46cecd986042e50626575231a4a155cb0ee5ca
redpilllinpro01.ring.nlnog.net. IN SSHFP 3 2 20cfe8d906a4c38abbbe8f5d04c2cab8a00c8a803b51e252a1585f739098b02b

These entries can be copied and pasted directly into the zone file in question so that they are visible in DNS:

$ dig +short redpilllinpro01.ring.nlnog.net. IN SSHFP | sort
1 1 5FCA087A7C3EBEBBC89B229A05AFD450D08CF9B3
1 2 CDB4CDAF7734DF343FD567E0CAB92FD6AC5F2754BFEF797826DFD4BC F90F0BAF
2 1 613F389A36CF33B67D9BD69E381785B275E101CD
2 2 8A07B97B96D826A7D4D403424B97A8CCDB77105B527BE7D7BE835D02 FDB9CD58
3 1 3E46CECD986042E50626575231A4A155CB0EE5CA
3 2 20CFE8D906A4C38ABBBE8F5D04C2CAB8A00C8A803B51E252A1585F73 9098B02B

How to automatically update the SSHFP records in DNS when a node is being provisioned is left as an exercise for the reader, but one nifty little trick is to run something like ssh-keygen -r "update add $(hostname --fqdn). 3600". This produces output that can be piped directly into nsupdate(1).

If you for some reason can not run ssh-keygen on the server, you can also use a tool called sshfp. This tool will take the entries from ~/.ssh/known_hosts (i.e., those you have manually accepted earlier) and convert them to SSHFP syntax.

2. Ensuring the DNS records are signed with DNSSEC

DNSSEC signing of the data in a DNS zone is a task that is usually performed by the DNS hosting provider, so normally you would not need to do this yourself.

There are several web sites that will verify that DNSSEC signatures exist and validate for any given host name. The two best known are:

• DNSViz
• Verisign Labs’s DNSSEC Debugger

If DNSViz shows that everything is «secure» in the left column (example) and the DNSSEC Debugger only shows green ticks (example), your DNS records are correctly signed and the SSH client should consider them secure for the purposes of SSHFP validation.

If DNSViz and the DNSSEC Debugger give you a different result, you will most likely have to contact your DNS hosting provider and ask them to sign your zones with DNSSEC.

3. A recursive resolver that supports DNSSEC

The recursive resolver used by your system must be capable of validating DNSSEC signatures. This can be verified like so:

$ dig redpilllinpro01.ring.nlnog.net. IN SSHFP +dnssec
[...]
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 7, AUTHORITY: 0, ADDITIONAL: 1
[...]

Look for the ad flag («Authenticated Data») in the answer, If present, it means that the DNS server confirms that the supplied answer has a valid DNSSEC signature and is secure.

If the ad flag is missing when querying a hostname known to have valid DNSSEC signatures (e.g., redpilllinpro01.ring.nlnog.net), your DNS server is probably not DNSSEC capable. You can either ask your ISP or IT department to fix that, or change your system use a public DNS server known to be DNSSEC capable.

Cloudflare’s 1.1.1.1 is one well-known example of a public recursive resolver that supports DNSSEC. To change to it, replace any pre-existing nameserver lines in /etc/resolv.conf with the following:

nameserver 1.1.1.1
nameserver 2606:4700:4700::1111
nameserver 1.0.0.1
nameserver 2606:4700:4700::1001

4. Configuring the system stub resolver to request DNSSEC validation

By default, the system stub resolver (part of the C library) does not set the DO («DNSSEC OK») bit in outgoing queries. This prevents DNSSEC validation.

DNSSEC is enabled in the stub resolver by enabling EDNS0. This is done by adding the following line to /etc/resolv.conf:

options edns0

5. Configuring the SSH client to look for host keys in DNS

Easy peasy: either you can add the line VerifyHostKeyDNS yes to your ~/.ssh/config file, or you can supply it on the command line using ssh -o VerifyHostKeyDNS=yes.

Verifying that it works

If you have successfully implemented steps 1-5 above, we are ready for a test!

If you have only done step 3-5, you can still test using redpilllinpro01.ring.nlnog.net (or any other node in the NLNOG RING for that matter). The NLNOG RING nodes will respond to SSH connection attempts from everywhere, and they have all DNSSEC-signed SSHFP records registered.

$ ssh -o UserKnownHostsFile=/dev/null -o VerifyHostKeyDNS=yes no-such-user@redpilllinpro01.ring.nlnog.net
no-such-user@redpilllinpro01.ring.nlnog.net: Permission denied (publickey).

Ignore the fact that the login attempt failed with «permission denied» - this test was a complete success, as the SSH client did not ask to manually verify the SSH host key.

UserKnownHostsFile=/dev/null was used to ensure that any host keys manually added to ~/.ssh/known_hosts at an earlier point in time would be ignored and not skew the test.

It is worth noting that SSH does not add host keys verified using SSHFP records to the ~/.ssh/known_hosts file - it will validate the SSHFP records every time you connect. This ensures that even if the host keys change, e.g., due to the server being re-provisioned, the ominous «IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY» warning will not appear - provided the SSHFP records in DNS have been updated, of course.

Trusting the recursive resolver

The setup discussed in this post places implicit trust in the recursive resolver used by the system. That is, you will be trusting it to diligently validate any DNSSEC signatures on the responses it gives you, and to only set the «Authenticated Data» flag if those signatures are truly valid.

You are also placing trust in the network path between the host and the recursive resolver. If the network is under control by a malicious party, the DNS queries sent from your host to the recursive resolver could potentially be hijacked and redirected to a rogue recursive resolver.

This means that an attacker with the capability to hijack or otherwise interfere with both your SSH and DNS traffic could potentially set up a fraudulent SSH server for you to connect to, and make your recursive resolver lie about the SSH host keys being correct and valid according to DNSSEC. The SSH client will not be able to detect this situation on its own.

In order to detect such attacks, it is necessary for your host to double-check the validity of answers received from the recursive resolver by performing local DNSSEC validation. How to set up this will be the subject of a future post here on the Redpill Lipro techblog. Stay tuned!

We have all done it. When SSH asks us this familiar question:

$ ssh redpilllinpro01.ring.nlnog.net The authenticity of host 'redpilllinpro01.ring.nlnog.net (2a02:c0:200:104::1)' can't be established. ECDSA key fingerprint is SHA256:IM/o2Qakw4q7vo9dBMLKuKAMioA7UeJSoVhfc5CYsCs. Are you sure you want to continue connecting (yes/no/[fingerprint])? 

…we just answer yes - without bothering to verify the fingerprint shown.

Many of us will even automate answering yes to this question by adding StrictHostKeyChecking accept-new to our ~/.ssh/config ...