Scenario: I had the need for a small tool that would parse logfiles of approximately 6 million lines. To each line apply two regexps to extract a few values, and using two separate dictionaries/hashmaps (choose your poison wrt terminology) calculate how many times each capture group int he regexp shows up.

All in all, this is not going to be a very performance critical run, as it will only parse about 4 times these 6 million lines per day, so even at worst case we're talking less than 100M regexp matches per day. Piece of cake for any language. And since it's a weekend and I don't have any time when I have to deliver this finished script, I set out to prove to myself that python is going to be fast enough for this.

In the beginning of 2019 Oracle stopped releasing free-of-charge updates to their JDK, except for personal use. At the same time Oracle started requiring a subscription for Oracle JDK use in production environments. In this blog post we will look into OpenJDK as an alternative to the Oracle provided JDK and how one would migrate to it.

...

The Suricata IDS is an extremely nice piece of software with multiple deployment scenarios, including inline and with mirroring or taps. The process itself usually runs without any problems, but as with any setup with multiple moving parts there are things that can go wrong. Monitoring and making sure the Suricata processes itself is easy […]

I’ve recently revamped my home network security monitoring. Currently I’m capturing and streaming all network traffic on my MikroTik router’s outside interface to a remote sensor, namely a Raspberry Pi 4 with 4 GB RAM running Suricata IDS. Suricata’s log is read by Elastic’s Filebeat and shipped to an Elasticsearch instance, making the data available […]

Now we continue improving the VPC template from my previous blog entry “Starting with Cloudformation templates”

What we ended up with there was a VPC with one subnet connected to the Internet. Or what is know in AWS lingo as a “Public Subnet”.

The goal now is a VPC with presence in tree Availability Zones with a “Public Subnet” in each, and a “Private Subnet” in each as well.

Humble beginnings

Before we go all out on tree ...

I’ve recently revamped my home network security monitoring. Currently I’m capturing and streaming all network traffic on my MikroTik router’s outside interface to a remote sensor, namely a Raspberry Pi 4 with 4 GB RAM running Suricata IDS. Suricata’s log is read by Elastic’s Filebeat and shipped to an Elasticsearch instance, making the data available […]

I’ve recently revamped my home network security monitoring. Currently I’m capturing and streaming all network traffic on my MikroTik router’s outside interface to a remote sensor, namely a Raspberry Pi 4 with 4 GB RAM running Suricata IDS. Suricata’s log is read by Elastic’s Filebeat and shipped to an Elasticsearch instance, making the data available […]

The number of great webservers on our toolbelt is constantly growing. From the venerable Apache httpd over lighttpd to nginx - and for the reverse proxy space pound, varnish and also nginx - the number just keeps growing. Caddy is a newcomer in this field, yet its features are already impressive. Lets take a brief look.

...

In a samba setup where users and groups are fetched from Active Directory to be used in a unix/linux environment, AD may prohibit the samba winbind tools like wbinfo to recurse into its group structure. You may get groups and users and their corresponding gids and uids, but you may not get the members of a group.

It is usually possible to do the opposite, that is, probing a user object and get the groups that user is member of. Here is a little script that collects all users, probing AD for the groups of each and every user, and sorting and putting it together. In perl of course.

https://github.com/ingvarha/groupmembers

The plan

One of Redpill Linpro’s customers - had parts of their web presence managed by another provider - “P”. The customer wanted to migrate the server operations to Redpill Linpro (“RL”), including moving their Elasticsearch cluster to Redpill Linpro’s Elasticsearch-as-a-Service environment, preferrably without downtime. While exporting and importing by using snapshots was an option, the better option would be a live migration.

A really useful Elasticsearch feature is the ...

Working from home encourages home office optimization. During the COVID-19 period, with way more video conferences than usual, certain improvements were found necessary. I guess everyone that’s been in a video meeting where more than one participant used a regular mic and regular speakers has experienced the wonders of audio feedback. That prompted me to […]

As we saw in the introduction to ActiveMQ Artemis post, in ActiveMQ Artemis the implementation is separated from the configuration and data, requiring one to create a broker instance after installation of the implementation. One of the advantages of doing this, is that it makes upgrades much easier. Lets take a detailed look at that now.

...

Apache ActiveMQ is a modern, open source messaging platform. For years now, the community has been working on the successor to the venerable ActiveMQ - now sometimes referred to as ActiveMQ classic.

The new project is named ActiveMQ Artemis, at least until the time that it will be re-branded as ActiveMQ 6.0. Apparently that will happen once ActiveMQ Artemis has all features of ActiveMQ, but since no one knows if and when this feature parity is achieved lets take a look at ActiveMQ Artemis - the future of the ActiveMQ project.

...

This is not the place to tell anyone why Infrastructure as Code is a good idea. For that I can point the potential readers to a blog by my colleague Yngve about that: Why code your infrastructure?

I a short series of blogs, I intend to demonstrate building infrastructure in AWS in steps, where I will be building upon previous entries. Basic knowledge of network and VPC is assumed.

Note that following these instructions can and will incur costs ...

I'm sure everybody is aware you can have PostgreSQL fetch data live across the internet (or locally of course) as part of your queries and use the data directly. In fact there are a large number of drivers available to access different kinds of data for download. But in the simplest case, we can also just use the file_fdw wrapper that's included in the standard PostgreSQL packages, together with everybody's http-swiss-army-knife, curl.

In attempting to adapt this post for the public, what more time-relevant dataset to work off in these pandemic-affected times than the open data provided by the ECDC, being both open data and very current. In particular for this example, they provide public datasets with COVID numbers from across the world (actual public data, requiring no registration to read, and actual data, not just a limited API).

So, let's see how we can access this data from PostgreSQL:

CREATE EXTENSION file_fdw;

CREATE SERVER curly FOREIGN DATA WRAPPER file_fdw;

CREATE FOREIGN TABLE _rawdata (
 daterep text not null,
 day int not null,
 month int not null,
 year int not null,
 cases int not null,
 deaths int not null,
 countries text not null,
 geoid text not null,
 countrycode text null,
 popdata int null,
 continent text not null,
 cumulative14days float null
)
SERVER curly
OPTIONS (
 PROGRAM 'curl -s https://opendata.ecdc.europa.eu/covid19/casedistribution/csv/',
 FORMAT 'csv',
 HEADER 'on'
);

And to use this we can simply query the foreign table, which will then trigger a http GET of the data:

covid=# SELECT count(*) FROM _rawdata;
 count 
-------
 41837
(1 row)

However, there is an unfortunate interaction with LIMIT. So if we for example try to get just the first 10 rows:

covid=# SELECT * FROM _rawdata LIMIT 10;
ERROR:  program "curl -s https://opendata.ecdc.europa.eu/covid19/casedistribution/csv/" failed
DETAIL:  child process exited with exit code 23

I’m sure everybody is aware you can have PostgreSQL fetch data live across the internet (or locally of course) as part of your queries and use the data directly. In fact there are a large number of drivers available to access different kinds of data for download. But in the simplest case, we can also just use the file_fdw wrapper that’s included in the standard PostgreSQL packages, together with everybody’s http-swiss-army-knife, curl.

In attempting to adapt this ...

So, you have a partitioned table. And you want to change your mind. Re-partitioning is "easy" if you can take downtime -- just create a new table with a new name and copy all the data over. But what if we want to try to do it without downtime? Logical replication enhancements in PostgreSQL 13 brings us some new options for this!

But first a disclaimer -- this is definitely not pretty! And does not take into consideration things like foreign keys and similar. But sometimes a quick hack can be the best hack.

So let's go!

As always, scammers and phishers use newsworthy events to their advantage. The coronavirus pandemic is no exception. All over the worlds, security researchers observe phishing and scam attempts. Samples for studying and for awareness training are collected at various sites, including https://coronavirusphishing.com/. A large number of security researchers have joined forces to establish a cyber […]

When we introduced the network configuration using Ansible and AWX at a customer, we gradually extended the configuration scope. Over time, more and more configuration got added into the configuration pool and this lead to longer and longer run-times for the playbooks.

While the job-execution got really simple by using AWX instead of the plain CLI method for Ansible, the time to finish drew heavily on that benefit.

A complete job-run over the network infrastructure took at least ...

We’ve been to FOSDEM in Belgium this year. A couple of Many of Redpill’s Agents (so called: Consultants) have made the trip to Belgium to join the annual conference taking place at Université libre de Bruxelles (ULB).

This year the conference was held during the first weekend in February, at the 1st and 2nd.

For those who don’t know: FOSDEM is a free software developer conference where you can attend talks about various different tools, processes and ideas and ...

One of my honeypots runs INetSim which, among many other services, emulates an SMTP server. The honeypot is frequently used by spammers who think they’ve found a mail server with easily guessed usernames and passwords. Obviously I’m logging the intruders’ activities, so I’m shipping the logs to Elasticsearch using Filebeat. Shipping the regular INetSim activity […]

If you are using Azure PostgreSQL and have upgraded your client side libpq to version 12 (which can happen automatically for example if you use the PostgreSQL apt repositories), you may see connection attempts fail with symptoms like:

$ psql -hZZZZZZ.postgres.database.azure.com -dpostgres -UXXXXX_dba@ZZZ-db01
psql: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

With no log information whatsoever available. This can happen if your client is in a Kerberos environment and has valid Kerberos credentials (which can be verified with the klist command). In this case, PostgreSQL 12 will attempt to negotiate GSSAPI encryption with the server, and it appears the connection handler in Azure PostgreSQL is unable to handle this and just kills the connection.

When running the same thing against a local PostgreSQL server prior to version 12, a message like the following will show up in the log:

2020-02-20 10:48:08 CET [35666]: [2-1] client=1.2.3.4 FATAL:  unsupported frontend protocol 1234.5680: server supports 2.0 to 3.0

This is a clear indicator of what's going on, but unfortunately the information isn't always available when connecting to a managed cloud service, such as Azure PostgreSQL. The hard error from Azure also prevents libpq from retrying without GSSAPI encryption, which is what would happen when connecting to a regular PostgreSQL backend or for example through pgbouncer.

The fix/workaround? Disable GSSAPI encryption in the client:

$ export PGGSSENCMODE=disable
$ psql -hZZZZZZ.postgres.database.azure.com -dpostgres -UXXXXX_dba@ZZZ-db01
Password for user XXXXX_dba@ZZZ-db01:
psql (11.6 (Ubuntu 11.6-1.pgdg16.04+1), server 9.5.20)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

postgres=>

If you have this type of issue, it's probably worth putting this environment variable in your startup scripts. It can also be set using the gssencmode parameter as part of the connection string, in environments where this is more convenient.

While the AWS console gives you a nice point and click interface, and really helps you explore the vast service catalog of AWS, the use of the CLI should not be neglected.

Some of the advantages of the CLI:

• Reusable, can the same command multiple times, perhaps with slight modification for quickly creating multiple instances of similar resources.
• Reproducible, can run the same command, to reproduce exactly the same kind of resource as has been created before.
...

A customer of my employer Redpill Linpro was recently the target of a DDoS attack. While investigating the attack, we found a large number of HTTP requests with the User-Agent named CITRIXRECEIVER. The clients performed GET requests to multiple URLs on the customer’s web site at the rate of several thousand packets per second. The […]

This year we intend to upgrade all the routers in our network backbone to a brand new platform based on open networking devices from Edge-Core running Cumulus Linux. In this post - replete with pictures - we will take a close look at the new routers and the topology of our new network backbone.

Why upgrade?

Our network backbone is today based on the Juniper MX 240 routing platform. Each of them occupy 5