Planet Redpill Linpro

Call for Papers - PostgreSQL Conference Europe 2012

2012-05-15T19:49:00+00:00

The call for papers for PostgreSQL Conference Europe 2012 in Prague, the Czech Republic has now been posted. As usual, we are looking for talks on all topics related to PostgreSQL. At this point, we are looking for submissions for regular conference sessions - we will post a separate call for papers for lightning talks at a later time.

We are also still looking for sponsors - please see our website for details about the sponsor benefits and the costs.

Follow the news feed on our site, or our Twitter feed, for news updates!

CSS crazy – or why we haven’t learnt anything from IE6 (part 1)

2012-05-07T08:12:10+00:00

Disclaimer: This isn’t going to be about php, so if that doesn’t tickly your fancy feel free to skip this entry. It’s also a bit of a rant.

Every now and then you figure out you want to do some cool css trick or update some css or markup that’s been lagging behind. Last friday I was in total weekend mode so I figured I’d update some of the old markup and css in the project I’m currently working on. The header area in that project was one of the first pieces of html and css created when we started developing it, and it has been left largely untouched since those early days in 2009. It was in desperate need of some tender CSS love.

The first thing I wanted to fix was the header background. It’s got a gradient background image, with a logo that has the gradient as background color. Depending on which browser you’re using, the background image and the logo is always one or two pixels off, because that’s just the way the markup is. We still have to support relatively old and obscure browsers like IE7, so keeping backwards compatibility is important to not unnecessarily annoy our customer(s). So, since we’re living in 2012 nowadays it should be as easy as adding a css background gradient with a fallback to the background image:

background: url("/images/navbar_top.png") repeat-x; /* legacy background image */
background: linear-gradient(to bottom, #434A50 0%,#2B3036 80%); /* W3C */

Yeah, not so much. The CSS level 3 specification for background gradients is still not finished and all browsers that has implemented it so far has done so based on early drafts, their own interpretation of the specification as well as their own desires about how the specification should work. According to the MDN page about CSS linear gradients, achieving this effect across all browsers is not at all trivial.

First of all: the background gradient CSS3 specification is of course not yet final and it is still subject to changes. Also, every browser on the planet has its own way of specifying background gradients, and they’re all non-compatible as well as wildly different. If that wasn’t enough to completely turn you off CSS gradients, bear in mind that each browser also has different browser-/vendor-specific implementations depending on the browser version, as the recommended implementation has been changed quite a lot since Apple introduced this feature back in 2008. To end it all, the currently final version of the CSS3 recommended standard has switched around one of the keywords – instead of specifying where you want the gradient to start (“top”), you specify the direction (“to bottom”) – making the final recommendation inversely compatible with the currently available browser-specific implementations. So, to implement something as trivial as a background gradient you need to take into consideration several different versions of a multitude of browsers, as well as a totally moving specification. With that in mind, here is the bit of CSS I ended up using for that header background gradient:

background: url("/images/navbar_top.png") repeat-x; /* legacy background image */
background: -moz-linear-gradient(top, #434A50 0%, #2B3036 80%); /* FF3.6+ */
background: -khtml-gradient(linear, left top, left bottom, color-stop(0%,#434A50), color-stop(80%,#2B3036)); /* KHTML */
background: -webkit-gradient(linear, left top, left bottom, color-stop(0%,#434A50), color-stop(80%,#2B3036)); /* Chrome, Safari4+ */
background: -webkit-linear-gradient(top, #434A50 0%,#2B3036 80%); /* Chrome10+, Safari5.1+ */
background: -o-linear-gradient(top, #434A50 0%,#2B3036 80%); /* Opera11.10+ */
background: -ms-linear-gradient(top, #434A50 0%,#2B3036 80%); /* IE10+ */
filter: progid:DXImageTransform.Microsoft.gradient( startColorstr='#434A50', endColorstr='#2B3036',GradientType=0 ); /* IE6-9 */
background: linear-gradient(to bottom, #434A50 0%,#2B3036 80%); /* Current W3C recommendation */

I kid you not.
Luckily, with this blob of CSS we’re able to support pretty much all browsers in use today. Yay for us! Only marginally smaller than the byte size of the image we’re currently using!

I’ll follow up with a new post soon, with more juicy CSS fun. Comments are much appreciated!

Using a mysql replicate for backup: Remember to set log_slave_updates

2012-05-07T07:51:05+00:00

The other day I was setting up a mysql slave in a an application where we needed to daisy-chain one slave from another. With just adding the slave the standard way, the first slave kept happily replicating, but we got no updates on the second slave. After a bit of debugging, we found log_slave_updates in the documentation. This makes the slave actually binlog changes from the master. Without this, only changes done one the slave is added to the binlog. Adding log_slave_updates to my.cnf on the first slave, the second in the chain started replicating as well.

So far, so good. But considering using replication for backup purposes, which is something I guess is quite common, the binlog of changes to the database is necessary to be able to do a point in time recovery. To get that binlog, you have to add log_slave_updates. Unless you have backup of the binlog files from the master, of course, but the point was using the replicate for backup, wasn’t it?

If you are unsure, compare the size of the binlog files on the master and the backup replicate slave.

How to Google more efficiently

2012-05-01T14:37:00+00:00

Most people these days use google.com as their preferred search engine, and most times a regular search will bring you the results you are looking for. But what if you want a more complex search? Or want to filter out some results? Well, here's a few pointers on how to Google a bit more efficiently, and some other neat tricks.

A regular search

Say that I want to search for a review for the cellphone HTC Desire Z. I could always use the string "HTC Desire Z review". Or, we could do a more efficient search.

Take the following search string as an example:

site:engadget.com ~review "htc desire z" -amazon 2010..2012

Let's break it down:

site:engadget:com - Only searches the pages of the site engadget.com
~review - Also searches related words, such as "check, examine, opinion".
htc desire z - Searches for the exact phrase, and not each word separately.
-amazon - Excludes this term from the search.
2010..2012 - Shows all results from within the specified timerange.

The last element above not only works with a time-range, but also with any other type of range. For instance:

$100..$200

..gives you results based on a range from $100 to $200.

Searching for filetypes

In certain cases you want to search for specific types of file. For instance,

filetype:pdf linux intitle:best *distribution

Again, let's break this down:

filetype:pdf - Searches only results of the file type you specify. .doc, .jpg, .torrent, the options are endless. Close to, anyway.
linux - Just a regular search string.
intitle:best - Only shows results with that word in the title (in this case, 'best')
*distribution - Replaces itself with common terms in your search (in this case 'Arch Linux distribution', 'Red Hat distribution', will both be searched, among many others)

Definitions

To find the definition of a word, use:

define: exquisite

Unit converter

Converting from one unit to another made easy:

Distance

54 km to miles

Weight

2 kg to lbs

Currency

150 NOK to USD

Calculator

Also works great as a calculator:

20*(2+4)/5-1

Time

Check out what the time is in another part of the world:

time Hong Kong

Weather

Also handy for checking the weather:

weather Oslo

Just a few of the things one can do with the Google search engine!

FreeRADIUS and CRLs

2012-04-26T17:44:00+00:00

Deploying FreeRADIUS, you may want to use a Certificate Revocation List (CRL) to revoke access to users that has been issued a valid certificate at some point.

How to actually do this isn't completely self-evident. Less so if you use multiple sub-CAs.

FreeRADIUS basically uses OpenSSL's routines and options for this. And OpenSSL doesn't currently refresh CRLs in a running process, so after updating a CRL you need to restart the freeradius daemon (easily scripted).

The important point in freeradius is in /etc/freeradius/eap.conf, specifically check_crl and CA_path in the tls section. Also, comment out CA_file, since CA_path and CA_file are intended for the same purposes, but CRLs only work with the CA_path method.

   #  Check the Certificate Revocation List
   #
   #  1) Copy CA certificates and CRLs to same directory.
   #  2) Execute 'c_rehash '.
   #    'c_rehash' is OpenSSL's command.
   #  3) uncomment the line below.
   #  5) Restart radiusd
   check_crl = yes
   CA_path = /etc/freeradius/certs/CA/

Be sure not to concatenate certificates in your CA_path. One CA, one file. Then, after the c_rehash, your directory should look something like this:

root@radius:~# ls -l /etc/freeradius/certs/CA/
lrwxrwxrwx 1 root freerad    9 2012-04-26 03:15 12345678.0 -> local-sub-ca-1.pem
lrwxrwxrwx 1 root freerad   13 2012-04-26 03:15 12345678.r0 -> sub-ca-1-crl.pem
lrwxrwxrwx 1 root freerad   11 2012-04-26 03:15 23456789.0 -> local-root-ca.pem
lrwxrwxrwx 1 root freerad    8 2012-04-26 03:15 3456789a.0 -> local-sub-ca-2.pem
lrwxrwxrwx 1 root freerad   12 2012-04-26 03:15 3456789a.r0 -> sub-ca-2-crl.pem
-rw-r--r-- 1 root freerad 1360 2012-04-24 15:04 local-root-ca.pem
-rw-r--r-- 1 root freerad 1360 2012-04-24 15:06 local-sub-ca-1.pem
-rw-r--r-- 1 root freerad 1384 2012-04-24 14:58 local-sub-ca-2.pem
-rw-r--r-- 1 root freerad 1251 2012-04-26 03:15 sub-ca-1-crl.pem
-rw-r--r-- 1 root freerad  739 2012-04-26 03:15 sub-ca-2-crl.pem

Thanks to Yasuhiro ABE for pointing me in the right direction. ありがとうございました、あべさん！それはよかったよ！

BASH autocomplete for SSH

2012-04-18T08:48:00+00:00

If you, like me, find yourself SSH'ing all day long, this little piece of code will do wonders. Well, it'll save you some keystrokes at least. This line parses your .bash_history-file and tries to autocomplete the host you are trying to SSH to, given that you've already used SSH to access the host in question at least once before.

So, to make this work, place the following in your .bashrc or .bash_profile file:

complete -W "$(echo $(grep '<sup>ssh</sup> ' .bash_history | sort -u | sed 's/<sup>ssh</sup> //'))" ssh

Once that's done, either log out, and in again, or type:

source .bashrc

..to apply the change made to your .bashrc, if that's the file you've made the change to. You should now be able to autocomplete hosts when SSH'ing with TAB.

Update: I am also (now) aware of, thanks to my colleagues, that you have to tweak and sometimes install additional packages to have a similar functionality provided by your system's package manager. I find the solution above the easiest one, as it's only a single line that needs to be included in a file, without any additional tweaks or packages necessary. But in the end, it's your call!

CentOS/RHEL/SL 6: root filesystem on tmpfs

2012-04-15T19:44:29+00:00

There are several scenarios where conventional hard drives are not really needed. Examples are HPC cluster nodes, virtualization nodes, home theater streaming PCs, silent desktops, internet cafés and embedded systems. Hard drives tend to fail, they are slow, they consume power, they generate heat and noise, and they are quite expensive if you need/want something faster and more reliable than SATA.

This post will show how to run CentOS 6 directly from tmpfs backed by memory, without using the (standard) 512 MB writable overlay. The procedure should be similar for RHEL and Scientific Linux 6.

The resulting boot process will be:

Boot a node off a PXE enabled DHCP server.
Chainload into iPXE.
Download vmlinuz and a rather large initrd containing the entire filesystem over ftp/http(s). Try to avoid tftp when downloading the initrd because of its file size limitation and slow transfer speeds.
Once downloaded, the kernel will start and the initrd will be mounted.
The modified dracut scripts in the initrd will create a tmpfs partition in memory with the same size as your filesystem image included in the initrd.
Your entire filesystem image will be copied to the tmpfs partition and attached to a loop device.
This loop device will be used as the new root device, and the boot process continues as usual.

This is a screenshot from an ongoing boot process:

Now to the procedure:

First, create a custom kickstart file. I’ve included the specialties below:

bootloader --location=mbr --append="toram"
clearpart --all
firstboot --disabled
install
lang en_US.UTF-8
network --bootproto dhcp --device eth0 --onboot yes
part / --fstype=ext4 --size=2048
reboot
zerombr

%packages
patch

%post
cat > /etc/fstab << END
tmpfs      /         tmpfs   defaults         0 0
devpts     /dev/pts  devpts  gid=5,mode=620   0 0
tmpfs      /dev/shm  tmpfs   defaults         0 0
proc       /proc     proc    defaults         0 0
sysfs      /sys      sysfs   defaults         0 0
END

# The patch is base64 encoded to avoid having to escape it manually.
cat > /root/dmsquash-live-root.base64 << EOF_patch
MjFhMjIKPiBnZXRhcmcgdG9yYW0gJiYgdG9yYW09InllcyIKMTM0YzEzNSwxMzgKPCAgICAgZG9f
bGl2ZV9mcm9tX2Jhc2VfbG9vcAotLS0KPiAgICAgIyBDcmVhdGUgb3ZlcmxheSBvbmx5IGlmIHRv
cmFtIGlzIG5vdCBzZXQKPiAgICAgaWYgWyAteiAiJHRvcmFtIiBdIDsgdGhlbgo+ICAgICAgICAg
ZG9fbGl2ZV9mcm9tX2Jhc2VfbG9vcAo+ICAgICBmaQoxNjNjMTY3LDIxMwo8ICAgICBkb19saXZl
X2Zyb21fYmFzZV9sb29wCi0tLQo+ICAgICAjIENyZWF0ZSBvdmVybGF5IG9ubHkgaWYgdG9yYW0g
aXMgbm90IHNldAo+ICAgICBpZiBbIC16ICIkdG9yYW0iIF0gOyB0aGVuCj4gICAgICAgICBkb19s
aXZlX2Zyb21fYmFzZV9sb29wCj4gICAgIGZpCj4gZmkKPiAKPiAjIEkgdGhlIGtlcm5lbCBwYXJh
bWV0ZXIgdG9yYW0gaXMgc2V0LCBjcmVhdGUgYSB0bXBmcyBkZXZpY2UgYW5kIGNvcHkgdGhlIAo+
ICMgZmlsZXN5c3RlbSB0byBpdC4gQ29udGludWUgdGhlIGJvb3QgcHJvY2VzcyB3aXRoIHRoaXMg
dG1wZnMgZGV2aWNlIGFzCj4gIyBhIHdyaXRhYmxlIHJvb3QgZGV2aWNlLgo+IGlmIFsgLW4gIiR0
b3JhbSIgXSA7IHRoZW4KPiAgICAgYmxvY2tzPSQoIGJsb2NrZGV2IC0tZ2V0c3ogJEJBU0VfTE9P
UERFViApCj4gCj4gICAgIGVjaG8gIkNyZWF0ZSB0bXBmcyAoJGJsb2NrcyBibG9ja3MpIGZvciB0
aGUgcm9vdCBmaWxlc3lzdGVtLi4uIgo+ICAgICBta2RpciAtcCAvaW1hZ2UKPiAgICAgbW91bnQg
LW4gLXQgdG1wZnMgLW8gbnJfYmxvY2tzPSRibG9ja3MgdG1wZnMgL2ltYWdlCj4gCj4gICAgIGVj
aG8gIkNvcHkgZmlsZXN5c3RlbSBpbWFnZSB0byB0bXBmcy4uLiAodGhpcyBtYXkgdGFrZSBhIGZl
dyBtaW51dGVzKSIKPiAgICAgZGQgaWY9JEJBU0VfTE9PUERFViBvZj0vaW1hZ2Uvcm9vdGZzLmlt
Zwo+IAo+ICAgICBST09URlNfTE9PUERFVj0kKCBsb3NldHVwIC1mICkKPiAgICAgZWNobyAiQ3Jl
YXRlIGxvb3AgZGV2aWNlIGZvciB0aGUgcm9vdCBmaWxlc3lzdGVtOiAkUk9PVEZTX0xPT1BERVYi
Cj4gICAgIGxvc2V0dXAgJFJPT1RGU19MT09QREVWIC9pbWFnZS9yb290ZnMuaW1nCj4gCj4gICAg
IGVjaG8gIkl0J3MgdGltZSB0byBjbGVhbiB1cC4uICIKPiAKPiAgICAgZWNobyAiID4gVW1vdW50
aW5nIGltYWdlcyIKPiAgICAgdW1vdW50IC1sIC9pbWFnZQo+ICAgICB1bW91bnQgLWwgL2Rldi8u
aW5pdHJhbWZzL2xpdmUKPiAKPiAgICAgZWNobyAiID4gRGV0YWNoICRPU01JTl9MT09QREVWIgo+
ICAgICBsb3NldHVwIC1kICRPU01JTl9MT09QREVWCj4gCj4gICAgIGVjaG8gIiA+IERldGFjaCAk
T1NNSU5fU1FVQVNIRURfTE9PUERFViIKPiAgICAgbG9zZXR1cCAtZCAkT1NNSU5fU1FVQVNIRURf
TE9PUERFVgo+ICAgICAKPiAgICAgZWNobyAiID4gRGV0YWNoICRCQVNFX0xPT1BERVYiCj4gICAg
IGxvc2V0dXAgLWQgJEJBU0VfTE9PUERFVgo+ICAgICAKPiAgICAgZWNobyAiID4gRGV0YWNoICRT
UVVBU0hFRF9MT09QREVWIgo+ICAgICBsb3NldHVwIC1kICRTUVVBU0hFRF9MT09QREVWCj4gCj4g
ICAgIGVjaG8gIlJvb3QgZmlsZXN5c3RlbSBpcyBub3cgb24gJFJPT1RGU19MT09QREVWLiIKPiAg
ICAgZWNobwo+IAo+ICAgICBsbiAtcyAkUk9PVEZTX0xPT1BERVYgL2Rldi9yb290Cj4gICAgIHBy
aW50ZiAnL2Jpbi9tb3VudCAtbyBydyAlcyAlc1xuJyAiJFJPT1RGU19MT09QREVWIiAiJE5FV1JP
T1QiID4gL21vdW50LzAxLSQkLWxpdmUuc2gKPiAgICAgZXhpdCAwCjE2OWMyMTksMjIxCjwgICAg
IGVjaG8gIjAgJCggYmxvY2tkZXYgLS1nZXRzeiAkQkFTRV9MT09QREVWICkgc25hcHNob3QgJEJB
U0VfTE9PUERFViAkT1NNSU5fTE9PUERFViBwIDgiIHwgZG1zZXR1cCBjcmVhdGUgLS1yZWFkb25s
eSBsaXZlLW9zaW1nLW1pbgotLS0KPiAgICAgaWYgWyAteiAiJHRvcmFtIiBdIDsgdGhlbgo+ICAg
ICAgICAgZWNobyAiMCAkKCBibG9ja2RldiAtLWdldHN6ICRCQVNFX0xPT1BERVYgKSBzbmFwc2hv
dCAkQkFTRV9MT09QREVWICRPU01JTl9MT09QREVWIHAgOCIgfCBkbXNldHVwIGNyZWF0ZSAtLXJl
YWRvbmx5IGxpdmUtb3NpbWctbWluCj4gICAgIGZpCg==
EOF_patch

cat /root/dmsquash-live-root.base64 | base64 -d > /root/dmsquash-live-root.patch

patch /usr/share/dracut/modules.d/90dmsquash-live/dmsquash-live-root /root/dmsquash-live-root.patch

ls /lib/modules | while read kernel; do
  echo " > Update initramfs for kernel ${kernel}"
  initrdfile="/boot/initramfs-${kernel}.img"

  /sbin/dracut -f $initrdfile $kernel
done
%end

%post --nochroot

echo "Copy initramfs outside the chroot:"
ls $INSTALL_ROOT/lib/modules | while read kernel; do
  src="$INSTALL_ROOT/boot/initramfs-${kernel}.img"
  dst="$LIVE_ROOT/isolinux/initrd0.img"
  echo " > $src -> $dst"
  cp -f $src $dst
done
%end

Explaination: The post script will apply a patch to /usr/share/dracut/modules.d/90dmsquash-live/dmsquash-live-root before regenerating the initramfs. This patch will add support for the ‘toram’ boot parameter. Then, the initramfs is being copied to the isolinux directory outside the filesystem image.

Second, use livecd-creator and livecd-iso-to-pxeboot from the livecd-tools package to convert the kickstart file into a bootable vmlinuz and initrd:

$ sudo livecd-creator --config=centos6.ks fslabel=centos6
$ sudo livecd-iso-to-pxeboot centos6.iso

The commands above will create tftpboot/vmlinuz0 and tftpboot/initrd0.img. Put these files on your boot server and create a suitable PXE configuration. livecd-iso-to-pxeboot will create tftpboot/pxelinux.cfg/default which can be used as a template.

Now you are ready to boot one or multiple CentOS 6 in-memory instances over the network!

Another screenshot:

Feature request upstream.

P4 nettradio på Linux

2012-04-13T07:19:00+00:00

Skulle du ønske å høre på P4 sin nettradio uten å måtte ha et nettleservindu åpen, kan du bare peke din mediaspiller (som f.eks. vlc eller mplayer) direkte til strømmen. Hvis vi bruker mplayer:

For P4:

mplayer mms://mms-cluster2.nextra.no/P4_Norge

For P4 Bandit

mplayer mms://mms-cluster2.nextra.no/P4_bandit

Bare erstatt mplayer med vlc, eller noe annet, skulle du ønske å endre mediaspiller som benyttes.

Ubuntu, Citrix, and SSL-error 61

2012-04-03T08:01:00+00:00

I just attempted to install the newest Linux Citrix ICA client for a colleague, on Ubuntu, and upon a successful install, an error similar to this one popped up when trying to run the client from Firefox:

SSL error
Contact your help desk with the following information:
You have not chosen to trust"/C=US/ST=/L=/O=Equifax/OU=Equifax Secure Certificate Authority/CN=",
the issuer of the server's security certificate ((SSL error 61).

The solution to this? Make Firefox's certificates accessible to Citrix, like so:

sudo ln -s /usr/share/ca-certificates/mozilla/* /opt/Citrix/ICAClient/keystore/cacerts

That should do the trick!

Simple video streaming with apache to Samsung Galaxy Tab

2012-03-30T21:15:18+00:00

Ok, so I’d like to watch videos on my Samsung Galaxy Tab without the hassle of copying them from my computer. I thought streaming with VLC would be a nice option. It probably is, but I didn’t get it to work. The tab has great support for media formats which means I only need streaming - no converting, so I thought why not use plain old apache? And it worked pretty well, playback is smooth and I can jump around nearly instantaneously. This is my setup:

First if you don’t have it install apache2 and symlink your video folder into /var/www. Then check your servers ip adress with ifconfig

$ sudo apt-get install apache2
$ sudo ln -s /home/bb/Video /var/www/video
$ ifconfig

After that just open up the browser in the tab (with your servers ip adress of course)

http://192.168.1.100/video

Depending on file type I get a choice of programs to handle the file, for some reason .mp4 files triggers a dialog with just Android-system and Browser, just choose Android-system and you’ll get a proper dialog of choices. One problem I got was that the standard video player can’t handle all the formats, I use Dice Player which works very well for me.

PassiveDNS 0.5.0

2012-03-29T15:18:45+00:00

I have pushed PassiveDNS version 0.5.0.

According to the roadmap, I have been at 0.5.0 for a while, and even started to implement stuff for the 1.5.0 version. But my real aim is the 1.0.0 release, and I have started all the activities for the 1.0.0 release, but I lack the statistics that I set in the roadmap when PassiveDNS ends. I have played it against pcaps with DNS attacks, Im fuzzing pcaps being read by PassiveDNS etc. so a 1.0.0 is hopefully not that far away

Some of the changes since my last blog post (v0.2.9):

* Logging of NXDOMAINs (-Xx -L nxdomain.log)
* DNS over UDP/TCP on IPv4 and IPv6 (Used to be just IPv4+UDP)
* Logging to stdout (-L – / -l -), both for NXDOMAINS and other DNS records.
* Implemented some hardening, including checking that client TID match server TID etc.
* Other small optimization and fixing a small memleak etc.

The way I implemented NXDOMAINS in PassiveDNS for now, makes it compete with the memory pool from “normal” domains/records. So if you have a fastflux or someone just querying for generated b0gus domains on your network, you might push out valid domains from the cache in favor for a NXDOMAIN. The reason I did this, is that it was faster than implementing an own memory pool for the NXDOMAINS and it give the possibility to log NXDOMAINS in current version with out to much hassle. If this way of implementing NXDOMAINS turns out to fight for memory more aggressively than one would like, one can always start two instances of PassiveDNS, one just looking for NXDOMAINS, and the other one looking for the regular domains. As I gain more experience with NXDOMAINS in PassiveDNS and get more feedback, Ill reconsider the implementation if needed

One note, the current logfile format will be stable until the 1.5.0 release (that is my intention at least), After that, my plan is to implement a customizable log format, and also more fields of interest will be available. If you have any additional data that you want to output and thoughts about how the output for those data should be, don’t hesitate to let me know

I ran into a security related bug on my Ubuntu 10.04 which might be triggered running PassiveDNS. I have emailed the Debian package maintainer and reported the bug to security@ubuntu.com and also filed a bug report. The bug is fixed upstream in ldns long time ago, so hopefully it will be fixed soon in Ubuntu 10.04 too

For reporting issues or making feature request, please do so here.

Happy DNS Archiving

PGConf.EU 2012 - announcement and call for sponsors

2012-03-27T14:59:00+00:00

It's time to mark your calendars - PostgreSQL Conference Europe 2012 will be held at the Corinthia Hotel in Prague, the Czech Republic, on October 23-26 2012. As previous years there will be one day of professional training (Tuesday 23rd) and then three days of regular talks.

At this point, we are also opening our sponsorship program. We are looking for sponsors at all levels, from Bronze to Platinum. Please see our website for details about the sponsor benefits and the costs.

Follow the news feed on our site, or our Twitter feed, for further information as we finalize details.

Cucumber testing with Kyuri in a browser

2012-03-19T14:01:56+00:00

Cucumber is a really nice testing framework, but I dislike having to implement the step definitions in ruby when the project itself is not in ruby, but what (web) project use javascript so why not do cucumber testing with javascript?

There are a couple of solutions for this, Cucumber JVM (former cuke4duke) is an official java based version of cucumber that let’s you use any jvm language. That’s cool, and probably a good way to go, but not fun enough for me. It messes with my mojo to start the jvm for something like this. And besides, there is no documentation on defining steps in javascript. Cucumer-js is a pure javascript alternative that looks good, uses node and seems very cool. But it lacks i18n support, it seems to be actively developed on so this is certainly something to keep an eye on. Cucumber-js also has support for running it in the browser via browserify. A third cool alternative is Kyuri, another pure javascript implementation, and it has support for i18n! Kyuri is also node based, can it be used in a browser? This is the steps that I took to get it running.

Prerequisites Install browserify and kyuri with npm.

$ sudo npm --global browserify
$ npm install kyuri

Browserify Kyuri actually has two ways to define steps, as “cucumber” and vows, vows seems to be the main target, but it doesn’t translate well to the browser. It has dependencies that browserify can’t handle. So we cut it out, which means a little bit of tweaking:

$ browserify -r kyuri  -i /node_modules/kyuri/node_modules/vows/lib/vows.js -i ./kyuri/runners/vows -o kyuri-bundle.js

This will give you a kyuri-bundle.js, it lacks vows.js and its runner but it still tries to require it so we need to edit it a bit. Remove this line (it’s line 375)

kyuri.runners.vows = require('./kyuri/runners/vows');

And these just below:

// Set default runner to vows
kyuri.runner = kyuri.runners.vows;

Browser This is a sample html page to test it:

<!DOCTYPE html>
<html>
<head>
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <title>Kyuri</title>
    <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js" type="text/javascript" language="javascript" charset="utf-8" ></script>
    <script src="kyuri-bundle.js" type="text/javascript" language="javascript" charset="utf-8" ></script>
    <script type="text/javascript" language="javascript" charset="utf-8">
        var kyuri = require('kyuri');
        
        var Runner = kyuri.runners.cucumber;

        Runner.on('beforeTest', function (done) {
          console.log('beforeTest event');
          done();
        });

        Runner.on('beforeBackground', function (done) {
          console.log('beforeBackground event');
          done();
        });

        Runner.on('afterTest', function (done) {
          console.log('afterTest event');
          done();
        });
        
        $(document).ready(function(){
            
            //features
            var features = [];
            $('.feature').each(function(){
                console.log(kyuri.parse($(this).text()));
                features.push(kyuri.parse($(this).text()));
            });
            
            //step defitions
            var Steps = require('kyuri').Steps;

            Steps.Given(/^I have entered (\d+) into the calculator$/, function (step, num) {
              console.log('Calculator: ' + num);
              step.done();
            });
            
            var module = {};
            Steps.export(module);
            
            var steps  = module.exports;
            
            //run it
            kyuri.runners.cucumber.run(features, steps, function () {
                console.log("Done!");
            });
            
        });
    </script>
</head>
<body>
    
<pre class="feature">
Feature: Addition
    In order to avoid silly mistakes
    As a math idiot
    I want to be told the sum of two numbers

    Scenario: Add two numbers
        Given I have entered 50 into the calculator
        And I have entered 70 into the calculator
        When I press add
        Then the result should be 120 on the screen
</pre>
</body>
</html>

Caveats

This only works in Firefox for me, Chrome has a minor error (probably easy to fix).
It logs console color codes… looks weird

Private repositories in Sonatype Nexus

2012-03-14T16:36:00+00:00

Sonatype Nexus is an artifact repository manager very much used when maven is your build tool for a java project. You can use it as a proxy for other repositories but also to host your own artifacts.

By default in Nexus, all repositories are reachable via the anonymous user.

Here is what to do to configure your Nexus server so that (at least) some of your repositories will require credentials to access them.

First you have to prevent the anonymous user to have access to all the repositories:

Create a new Privilege that gives access to you public group (or individual repositories) NOTE: Assigning access to a group is equivalent to assigning these privileges to all of the repositories in the group.
1. Login to nexus as an administrator.
2. Click on Privileges in the left menu.
3. Click Add.
4. Use the following values:
  1. Name: M2 Public Repositories (Group)
  2. Description: Access to Public Repositories (Group)
  3. Repository: Public Repositories (Group)
  4. Repository Target: All (Maven2)
5. Save.
Repeat the previous step for all you public groups and/or respositories.
Create a new Role and assign this new privilege to it.
1. Click on Roles in the left menu.
2. Click Add.
3. User the following values:
  1. Role Id: repo-public-read
  2. Name: Repo: All Public Repositories (read)
  3. Description: Read only access to all public repositories.
  4. Session Timeout: 60
  5. Selected Roles / Privileges: M2 Public Repositories (Group) - (read)
  6. NOTE: Include all of the roles you created in the first step.
4. Save.
Remove the Grant Read all role from the Anonymous user and add the new role.
1. Click on Users in the left menu.
2. Click on the anonymous user.
3. Remove the role 'Repo: All Repositories (read)'
4. Add the role 'All Public Repositories (read)'
5. Save.

Once you have done this, you need to create users to access all the non public repositories.

You first create a privilege for your given repo (Security->Privileges then Add)
Then a role (add the view and read part of the privilege created above)
And finally a user and assign him the role just create
Repeat this for as many times as you have repos. If you want to have several users for one repo, then just repeat 3.

The final step is to configure the client side with the proper settings in .m2/settings.xml and in the project pom.xml. like this:
settings.xml

<servers>
   <server>
       <id>myserver</id>
       <username>mrx</username>
       <password>pwd</password>
   </server>
<servers>

pom.xml

<repositories>
   <repository>
       <id>myserver</id>
       <name>myserver</name>
       <url>http://server:8081/nexus/content/repositories/your_repo</url>
   </repository>
</repositories>

Now your repository should be accessible with user and password only.

Some links:
Managing Security with Nexus
Depend on a password protected nexus repository
Can I make a repository private without disabling anonymous access?

varnish-3.0.2 for fedora

2012-03-08T08:22:08+00:00

I finally got around to wrap up varnish-3.0.2 for fedora 17 and rawhide. Please test and report karma.

In this release, I have merged changes from the upstream rpm, and added native systemd support for f17 and rawhide. It also builds nicely for epel5 and epel6, providing packages quite similar to those available from the varnish project repo.

As epel does not allow changes in a package API after release, varnish-3.0.2 won’t be available through epel5 or epel6, so use the varnish project repo, or my precompiled packages for epel 4, 5 and 6 available here.

As always, feedback is very welcome.

Bridging between JMS and RabbitMQ (AMQP) using Spring Integration

2012-03-08T05:48:00+00:00

An old customer recently asked me if I had a solution for how to integrate between their existing JMS infrastructure on Websphere MQ with RabbitMQ.

Although I know that RabbitMQ has the shovel plugin which can bridge between Rabbit instances I've yet not found a good plugin for JMS -> AMQP forwarding.
The first thing that came to my mind was to utilize a Spring Integration mediation as SI has excellent support for both JMS and Rabbit.

Curious as I am I started a PoC and this is the result. It takes messages of a JMS queue and forwards to an AMQP exchange that is bound to a queue the consumer application is supposed to listen to. I used an external HornetQ instance in JBoss 6.1 as the JMS Provider, but I am 100% secure that the same setup would work for Websphere MQ as they both implement JMS.

Be aware that I've done no performance tweaking or QoS setup yet as this is just a proof-of-concept. For a real setup you'd probably have to think about delivery guarantees versus performance and etc...

The code will be available at a GitHub repository near you soon..

SpringContext in XML:

<?xml version="1.0" encoding="UTF-8"?>
<beans:beans xmlns="http://www.springframework.org/schema/integration"
    xmlns:beans="http://www.springframework.org/schema/beans"
    xmlns:aop="http://www.springframework.org/schema/aop"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:int-amqp="http://www.springframework.org/schema/integration/amqp"
    xmlns:rabbit="http://www.springframework.org/schema/rabbit"
    xmlns:int-jms="http://www.springframework.org/schema/integration/jms"
    xmlns:int="http://www.springframework.org/schema/integration"
    xsi:schemaLocation="http://www.springframework.org/schema/integration/amqp http://www.springframework.org/schema/integration/amqp/spring-integration-amqp-2.1.xsd
        http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-3.0.xsd
        http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration-2.0.xsd
        http://www.springframework.org/schema/integration/jms http://www.springframework.org/schema/integration/jms/spring-integration-jms-2.1.xsd
        http://www.springframework.org/schema/rabbit http://www.springframework.org/schema/rabbit/spring-rabbit-1.0.xsd
        http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd">
        
    <beans:bean id="jndiTemplate" class="org.springframework.jndi.JndiTemplate">
        <beans:property name="environment">
            <beans:props>
                <beans:prop key="java.naming.factory.initial">org.jnp.interfaces.NamingContextFactory</beans:prop>
                <beans:prop key="java.naming.provider.url">jnp://localhost:1099</beans:prop>
                <beans:prop key="java.naming.factory.url.pkgs">org.jnp.interfaces:org.jboss.naming</beans:prop>
            </beans:props>
        </beans:property>
    </beans:bean>

    <beans:bean id="jmsQueueConnectionFactory"
      class="org.springframework.jndi.JndiObjectFactoryBean">
        <beans:property name="jndiTemplate">
            <beans:ref bean="jndiTemplate"/>
        </beans:property>
        <beans:property name="jndiName">
            <beans:value>ConnectionFactory</beans:value>
        </beans:property>
    </beans:bean>

    <!-- Channels and adapters for SI -->
    <int-jms:message-driven-channel-adapter connection-factory="jmsQueueConnectionFactory" destination-name="myJmsQueue" channel="rabbitChannel"/>
    <channel id="rabbitChannel"/>
    <int-amqp:outbound-channel-adapter channel="rabbitChannel" exchange-name="fromJmsExchange" amqp-template="rabbitTemplate"/>
    
    <!-- Connectivity to Rabbit -->
    <rabbit:template id="rabbitTemplate" connection-factory="cf"/>
    <rabbit:connection-factory id="cf" host="localhost"/>
    
    <!-- Rabbit entities, to be created at context startup -->
    <rabbit:admin connection-factory="cf"/>
    <rabbit:queue name="fromJMS"/>
    <rabbit:direct-exchange name="fromJmsExchange">
        <rabbit:bindings>
            <rabbit:binding queue="fromJMS"/>
        </rabbit:bindings>
    </rabbit:direct-exchange>
</beans:beans>

Maven POM:


<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.rl</groupId>
    <artifactId>si.jmstorabbit</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>si.jmstorabbit</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <hornet.version>2.2.5.Final</hornet.version>
        <spring.integration.version>2.1.0.RELEASE</spring.integration.version>
    </properties>

    <repositories>
        <repository>
            <id>springsource-release</id>
            <url>http://repository.springsource.com/maven/bundles/release</url>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>springsource-external</id>
            <url>http://repository.springsource.com/maven/bundles/external</url>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>
    </repositories>

    <dependencies>
    <dependency>
       <groupId>org.springframework.integration</groupId>
       <artifactId>spring-integration-core</artifactId>
       <version>${spring.integration.version}</version>
    </dependency>
    <dependency>
       <groupId>org.springframework.integration</groupId>
       <artifactId>spring-integration-file</artifactId>
       <version>${spring.integration.version}</version>
    </dependency>
    <dependency>
       <groupId>org.springframework.integration</groupId>
       <artifactId>spring-integration-amqp</artifactId>
       <version>${spring.integration.version}</version>
    </dependency>
    <dependency>
       <groupId>org.springframework.integration</groupId>
       <artifactId>spring-integration-jms</artifactId>
       <version>${spring.integration.version}</version>
    </dependency>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>3.8.1</version>
      <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework</groupId>
        <artifactId>spring-context</artifactId>
        <version>3.0.7.RELEASE</version>
    </dependency>
    
    <dependency>
        <groupId>jboss</groupId>
        <artifactId>jnp-client</artifactId>
        <version>4.2.2.GA</version>
    </dependency>
            
    
    <dependency>
            <groupId>org.hornetq</groupId>
            <artifactId>hornetq-core-client</artifactId>
            <version>${hornet.version}</version>
        </dependency>
        <dependency>
            <groupId>org.hornetq</groupId>
            <artifactId>hornetq-jms-client</artifactId>
            <version>${hornet.version}</version>
        </dependency>
        <dependency>
            <groupId>org.hornetq</groupId>
            <artifactId>hornetq-jms</artifactId>
            <version>${hornet.version}</version>
        </dependency>
           <dependency>
            <groupId>jboss</groupId>
            <artifactId>jboss-common-client</artifactId>
            <version>3.2.3</version>
        </dependency>
        <dependency>
            <groupId>org.jboss.netty</groupId>
            <artifactId>netty</artifactId>
            <version>3.2.7.Final</version>
        </dependency>
        <dependency>
            <groupId>javax.jms</groupId>
            <artifactId>jms</artifactId>
            <version>1.1</version>
        </dependency>
  </dependencies>
  
</project>

Practicing the fundamentals: Timezone handling

2012-02-28T15:00:49+00:00

Timezone handling can sometimes be a bitch. Turns out, using a couple of PHPs classes and functions, it’s quite easy.

Let’s say you have a web application with the following scenario:
Users from around the planet, and your server is located in the UK.

You would initialize your DateTime object like this:

$timezoneUTC = new DateTimeZone('UTC');
$dateTime = new DateTime('2012-02-23 10:22', $timezoneUTC);

echo $dateTime->format('Y-m-d H:i e').'<br/>';

This should output:
2012-02-23 10:22 UTC

For an international user, seeing his or her local time would be nice.
To do this, simply change the timezone of the existing DateTime object, like this:

$timeZoneSweden = new DateTimeZone('Europe/Stockholm');
$dateTime->setTimeZone($timeZoneSweden);
echo $dateTime->format('Y-m-d H:i e').'<br/>';

$timeZoneNY = new DateTimeZone('America/New_York');
$dateTime->setTimeZone($timeZoneNY);
echo $dateTime->format('Y-m-d H:i e').'<br/>';

$timeZoneBKK = new DateTimeZone('Asia/Bangkok');
$dateTime->setTimeZone($timeZoneBKK);
echo $dateTime->format('Y-m-d H:i e').'<br/>';

This should output:
2012-02-23 11:22 Europe/Stockholm
2012-02-23 05:22 America/New_York
2012-02-23 17:22 Asia/Bangkok

The final code should look something like this:

$timezoneUTC = new DateTimeZone('UTC');
$timeZoneSweden = new DateTimeZone('Europe/Stockholm');
$dateTime = new DateTime('2012-02-23 10:22', $timezoneUTC);
$dateTime->setTimeZone($timeZoneSweden);

This will leave a DateTime object set with the users timezone.

And that’s it.

New Munin-plugin for HP P2000 StorageWorks

2012-02-20T13:09:23+00:00

I committed a new Munin-plugin for HP P2000 StorageWorks. It monitors through the HTTP XML-RPC interface. It detects all disks, vdisks and controllers automatically – all you need to provide is the username/password, IP-address, and set up some symlinks.

It produces 10 graphs, + subgraphs.

Graph-gallery:

VDISK:

Disk bandwidth and iops (rootgraphs provides enclosures, subgraphs provides disks):

Controller bandwidth and iops:

Controller Cache:

Controller CPU-load:

A sawtooth-graph to indicate uptime statistics for the controllers:

Use Munin? Got a HP P2000 StorageWorks SAN? Why don’t you give it a try?

hackeriet festival

2012-02-10T11:02:44+00:00

If you haven’t heard already, Hackeriet – the Oslo Hackerspace – is hosting a full day of talks and workshops and party tomorrow Saturday. Come on by from 11:00am!
Check out the program at
http://events.hackeriet.no,
keywords to look out for are: arduinos, mesh networks, crypto, cyberwar, datalove and chipmusic :-))

Adventures in bash - catching several exit values in a piped set of commands

2012-02-08T15:53:00+00:00

"All in all, very odd, bash continues to be the most bizarre of languages, convoluted, twisted, but with strange solutions thrown in just when you are about to give up hope entirely." (forum post at Techpatterns)

Yesterday I was re-working a database backup script at one of my customers and stumbled onto a problem when I wanted to have both proper error handling and at the same time avoid filling the disk.

The code providing the challenge was this

$MYSQLDUMP $MYSQLDUMP_OPTS $DB | gzip -9 > $BACKUP_FILE

I need to pipe the output of mysqldump to gzip, because otherwise I run into problems with the disk filling up. And yes, having to it like this also means that doing restores are quite a pain, but that is another problem.

Normally I do error handling in scripts by evaluating $?, but to have proper error handling in here I need to capture the exit value of both mysqldump and gzip. And $? only gives med the exit value of gzip - the least important of the two.

Luckily, and as expected, I'm not the first person to run into this problem, ad by way of googling I found that Bash actually have a built-in way of giving me both exit values - the array $PIPESTATUS. $PIPESTATUS is an array with all the exit values from you last command line. $PIPESTATUS[0] contains the first exit value, $PIPESTATUS[1] the second and so on

sigurdur@ifconfig:~$ true | false
sigurdur@ifconfig:~$ echo ${PIPESTATUS[0]}
0
sigurdur@ifconfig:~$ true | false
sigurdur@ifconfig:~$ echo ${PIPESTATUS[1]}
1

You can also get the entire array

sigurdur@ifconfig:~$ true | false |false |true

sigurdur@ifconfig:~$ echo ${PIPESTATUS[@]}

0 1 1 0

A single, non-piped command is considered to be a "pipe of one", thus leaving you with a $PIPESTATUS array with one value. Since $PIPESTATUS is updated after every command line I had to copy the array before extracting the exit values.

So my code ended up like this:

$MYSQLDUMP $MYSQLDUMP_OPTS $DB | gzip -9
# We want the exit values of both mysqldump and gzip
exitarray=("${PIPESTATUS[@]}")
mydumpexitcode=${exitarray[0]}
gzipexitcode=${exitarray[1]}

PIPESTATUS have probably been part of Bash since forever, but to me it was new - and it solved my problem. Fun stuff:-)

dnstool - curses-based DNS zone administration

2012-02-06T15:27:00+00:00

Out now! Get it while the bits are still hot!

https://github.com/Redpill-Linpro/dnstool

The Fine Manual is plain text and even has screenshots, perfect for the old-school lynx enthusiast ;)

If your dns server supports RFC2136 dynamic updates, you might want to look at this. BIND9 does.

Patches always welcome!

pixie dust

2012-02-02T00:20:13+00:00

we’ve booted backtrack off usb before, now that’s kinda
boring and installing backtrack onto the usb with unetbootin
is painfully slow and not the same as bootin strait off the
usb which is what we want in this case; not an install
but a fresh copy every boot

there is someone disagreeing in the back of the room, now
wouldn’t this be a lot more complicated? No sir. on the contrary
booting fresh every time makes work a lot simpler; you gain a
direct relationship to what you store where, and where you
access your data from

but there is another one in the front;you sir, you feel that
one would have to sacrifice many of the comforts such as all
any tools of the trade at hand and permanent local storage -
but at best this is a lazy roadblock to salvation; by booting
off of local storage we have local storage at hand in a more
practical format, be that even a microscopic carrier can be
removed and replaced with sufficient storage for everything
and then some

the medium can be embedded, destroyed or ingested, so
the impermiableness of accidentally recorded data and the
robustness, accessability and portability of removable storage
comes very much in hand upon situations that either require
inconspiciousness, anonymity, covertness, plausible deniability
or a high degree of reliability in day-to-day computing

the totalality of the system given to remaining only in memory
causes it to be independent of other storage for operations, and when
operations cease from loss of any exterior preconditions, the
system simply ceases. when preconditions reoccur – by powering on
and executing the first block – the system can be relied upon to
simply starts afresh, completely unperturbed by any previous history

should the need arise to patch the system; say some new app or
capability is called for where there is no time to rebuild,
a patch should be scripted always when there is certanity that
the capability will require a repeat performance. It is advised
to devise a patch which includes all dependencies.

thus the fresh system becomes more capable and more accessible
over time, just like an install. patches can then easily be
rolled into the system should they proove useful to others.

But how does one do it? Well, it’s easy but unfortunately
not as easy as overwriting the boot device; it’s just not
practical because partitioning is always an individual consideration

. there are often other files on the block device

. choice of filesystem and memory technology has much bearing

. the block device is larger or smaller than expected

instead, we allow any bootable partition scheme and any
filesystem and memory technology, as long as the storage
requirements of the system are met;

here’s to clone how:

cp -a boot/ apt/ casper/ gone/ preseed/ syslinux/
syslinux /dev/partition
mbr /dev/device

but that’s fine, it’s been done and all, but even the ability to
boot the system with precisely zilch local storage comes in
handy, and for that we have pixie dust.

pixie daemon and tiny ftp should be pointing a path
exactly matching the dhcp-provided patch.. otherwise
you will have worries!

/etc/pxe.conf:

interface=eth1
service=X86PC,0,0,local,Local boot
service=X86PC,0,0,pxelinux,PXELinux
tftpdbase=/var/lib/tftpboot
domain=truly.yours

/etc/default/tftpd-hpa:TFTP_DIRECTORY=”/var/lib/tftpboot/”

/etc/dnsmasq.conf:

dhcp-boot=/var/lib/tftpboot/pxelinux,vulcano,10.10.10.86

“high speed” tftp daemons and multicast can be found but it is
advised to stick to tftpd-hpa and dnsmasq with no esoterics due
to the sheer amount of variables introduced.

/var/lib/tftpboot/pxelinux.cfg/default:

# not strictly necessary but makes the menu pretty
menu hshift 13
menu width 49
menu margin 8

menu title BackTrackBoot
default vesamenu.c32
display f.txt
timeout 600

label local
menu label Local Harddisk
localboot 0

menu begin bt
menu title BackTrack 5
# ok here comes the real shit
label backtrack5
menu label BackTrack R1
kernel bt5/vmlinuz
append boot=casper netboot=nfs nfsroot=vulcano:/mnt/bt5 initrd=bt5/initrd.gz text splash vga=791 file=/cdrom/preseed/custom.seed --
menu end

you’ll need to copy to tftpboot/bt5 the initrd.gz and vmlinuz from the backtrack ISO /casper folder (which you can mount -o loop -t iso9660 bt5.iso /mnt/bt5

the rest of the files you provide to the bootee over NFS

/etc/exports:

/mnt/bt5 10.10.3.0/24(rw,sync,no_subtree_check) 10.10.10.0/24(rw,sync,no_subtree_check)
mount -t iso9660 -o loop BT5R1-GNOME-32.iso /mnt/bt5

add a http server with kickstart / preseed files for an ever more powerful setup,
in which case you replace the file= stanza in the append line with
url=http://host/path/to/preseed
more on preseeds… maybe later.

Now restart all dependent services:

/etc/init.d/nfs-kernel-server restart
/etc/init.d/tftpd-hpa restart
/etc/init.d/apache2 restart
/etc/init.d/pxe restart

debugging this setup usually requires tracing the process that is failing, so:
- dhcp options tracing (dnsmasq verbose and tcpdump / wireshark)
- verbose pxe
- verbose foreground tftpd-hpa : in.tftpd -v -v -L /var/lib/tftpboot

Presentation: Writing modules for nginx

2012-02-01T08:55:09+00:00

I held a presentation at Redpill Linpros “C / C++ / Embedded” Competence Group-meeting yesterday.

It was about writing modules for nginx. It contains a lot of source code. I believe it’s an OK primer.

Extending functionality in nginx, with modules!

View more presentations from Trygve Vea

The example code can be found on github.

https://github.com/kvisle/nginx_module_samples

Finding gaps in partitioned sequences

2012-01-27T16:53:52+00:00

There are an almost unlimited number of articles on the web about how to find gaps in sequences in SQL. And it doesn't have to be very hard. Doing it in a "partitioned sequence" makes it a bit harder, but still not very hard. But when I turned to a window aggregate to do that, I was immediately told "hey, that's a good example of a window aggregate to solve your daily chores, you should blog about that". So here we go - yet another example of finding a gap in a sequence using SQL.

I have a database that is very simply structured - it's got a primary key made out of (groupid, year, month, seq), all integers. On top of that it has a couple of largish text fields and an fti field for full text search. (Initiated people will know right away which database this is). The sequence in the seq column resets to zero for each combination of (groupid, year, month). And I wanted to find out where there were gaps in it, and how big they were, to debug the tool that wrote the data into the database. This is really easy with a window aggregate:


SELECT * FROM (
   SELECT
      groupid,
      year,
      month,
      seq, 
      seq-lag(seq,1) OVER (PARTITION BY groupid, year, month ORDER BY seq) AS gap FROM mytable
) AS t
WHERE NOT (t.gap=1)
ORDER BY groupid, year, month, seq

One advantage to using a window aggregate for this is that we actually get the whole row back, and not just the primary key - so it's easy enough to include all the data you need to figure something out.

What about performance? I don't really have a big database to test this on, so I can't say for sure. It's going to be a sequential scan, since I look at the whole table,and not just parts of it. It takes about 4 seconds to run over a table of about a million rows, 2.7Gb, on a modest VM with no actual I/O capacity to speak of and a very limited amount of memory, returning about 100 rows. It's certainly by far fast enough for me in this case.

And as a bonus, it found me two bugs in the loading script and at least one bug in somebody elses code that I'm now waiting on to get fixed...

New Munin-plugin for HAProxy

2012-01-23T06:00:44+00:00

I committed a new Munin-plugin for HAProxy. It’s a multigraph plugin, it discovers all the configured frontends and backends automatically – all you need to provide is the username/password for the haproxy status page.

It produces 8 graphs, + subgraphs for some of the backends, where it presents the same graphs, but with server-specific metrics.

Some of the root-graphs:

Do you use HAProxy and Munin? Check it out!

Finding what binaries to restart

2012-01-19T21:39:06+00:00

When I started working with Linux system administration a few years ago, restarting services after a package upgrade was fairly easy. If the package didn’t restart itself, one could always ask lsof for help:

lsof +L1 | egrep 'bin/|lib/'

Now, on later Linux distributions, the usage of prelink has changed this, so one usually gets a lot of false positives, and could never trust the result of that good old lsof output.

Finding running executables is possible using some perl magic (Yes, I’m pretty sure you perl guys can write this more compressed) along the lines of this, at least on RHEL5:

perl -e ' for $i (glob "/proc/[1-9]*/exe") { $f=readlink $i; if ( $f=~ /([^\0]+)\0.*deleted/ ) { print "$1\n" }} ' | sort | uniq

But this won’t help us finding what service to restart after a dependency library was updated. So I finally wrote this script to help me. My boxes are mostly Fedora and RHEL, so it uses the fact that installed binaries on Red Hat based systems have installation time stored in the rpm database (rpm tag %{INSTALLTIME} ). This script abuses rpm heavily, and may use some time to finish on a busy system.

http://users.linpro.no/ingvar/check_newlibs

Test run:

[root]# /home/ingvar/check_newlibs
Warning: Needs restart: /sbin/agetty, pids 6067
Warning: Needs restart: /usr/bin/tail, pids 7315
Warning: Needs restart: /usr/bin/vim, pids 19759
Warning: Needs restart: /usr/sbin/sendmail.sendmail, pids 10645 10637
Warning: Needs restart: /usr/sbin/acpid, pids 5259
Warning: Needs restart: /usr/sbin/crond, pids 5567
Warning: Needs restart: /bin/bash, pids 26074 17731 16848 15718 30753 6120 32704
Warning: Needs restart: /sbin/mingetty, pids 6071 6069 6076 6068 6072 6070
Warning: Needs restart: /sbin/portmap, pids 5082
Warning: Needs restart: /usr/sbin/smartd, pids 20948
Warning: Needs restart: /sbin/multipathd, pids 20170
Warning: Needs restart: /usr/sbin/atd, pids 5969
Warning: Needs restart: /usr/sbin/sshd, pids 19863
Warning: Needs restart: /usr/libexec/mysqld, pids 17775

PassiveDNS 0.2.9

2012-01-17T15:17:07+00:00

I added some features and changes to PassiveDNS. The most important change is that the output now contains the TTL value, so you need to use the current tools/* (if you use them) as they are also changed to work with this new output format (or update your own tools).

I also added the ability to specify the DNS record types that you want to log from the command line and I added support for more record types. PassiveDNS now should be able to track: A, AAAA, CNAME, DNAME, NAPTR, SOA, PTR, RP, SRV, TXT, MX and NS.

Support for chroot and dropping privileges are also added.

I also added some features to tools/pdns2db.pl while I was at it:
1) You can now process a passivedns.log file in “batch” mode, exiting when finished.
2) You can now specify a file with a list of domains or IPs to skip insertion to the DB.
3) You can now specify a file with a list of PCRE (Perl Compatible Regular Expressions) of “domains/IPs” to skip insertion to the DB.
4) You can now specify a file with a list of domains or IPs to alert on!
5) You can now specify a file with a list of PCRE of “domains/IPs” to alert on!
6) You can now specify a file with a list of domains to whitelist and not alert on.
7) You can now specify a file with a list of PCRE of “domains/IPs” to whitelist and not alert on.

The skiplists will be checked first, and if the domain/IP is found/matched there, whitelist and blacklist will be ignored and insertion to DB will be ignored.

Next the whitelists will be checked, and if a domain/IP is found there or match a PCRE that you have defined it will not be checked by the blacklist.

Last the blacklists is checked, and if a domain/IP is found there or match a PCRE that you have defined, it will write the PassiveDNS record to the alert file that you specify (Default: /var/log/passivedns-alert.log).

There are different sources for getting lists of known bad domains. Here is one if you want to test the blacklist functionality: http://isc.sans.edu/feeds/suspiciousdomains_High.txt

Im pretty far as what it comes to planed features at this stage. Please try out PassiveDNS and beat the crap out of it I will probably “up” the version to 0.5.0 soon and from there on, it is just testing and testing and more testing before it will be a “one dot O” release.

If you have any issues with PassiveDNS, please submit them here.

PRADS, and how it compares to pads and p0fv2 and p0fv3

2012-01-12T10:49:01+00:00

The question was brought up to me late last night on IRC, as p0fv3 RC was recently announced. This is a short answer to that question:

“People that find the PRADS page and already know p0f or pads may be interested in a comparison or essentially arguments why you would use one over the other.”

First off, its exiting to see Michal Zalewski back with p0fv3 I quickly read through his code yesterday and tested it out, and its rather interesting how he solves things. The fingerprint database at the moment is limited, but expect that to grow in the near future. I also love his non formal output in his applications

[PRADS vs PADS]
So, back to the questions. First off, pads “Passive Asset Detection System” uses regexp syntax to look for common bytes in payload to identify server application. So if the server says ” Server: Apache/2.2.3 (Linux/SUSE)” that is collected as what service is running on the server port where this was detected. The “rules” can be written more specifically for each server software, but are rather general and small today. Some pads “rules” look for ASCII strings, and some for different bytes in hex etc. to identify stuff like SSL/TLS. Pads is no longer actively developed by the original author, but I do maintain a fork of the last version with enhancements added.

PRADS extended the way pads does asset detection. We have build in IPv6 support in PRADS, so it also detects asset listening on IPv6 addresses. We also have build in connection tracking, so that we can cut off detection in a stream after an amount of packets or bytes seen from the client or server. This to drop trying to look for server/client assets in connections that transfers big files or are encrypted etc. Most “banners/identifiers” are in the first packet etc. so limiting for how many packets in a stream to do detection on helps on performance etc.

To extend pads a bit, we also added detection for client applications using the same method as for detecting server applications.

My future thoughts on enhancing the pads/PRADS asset rules are to make them more like the Snort/Suricata rule language and use fast pattern matching before invoking the pcre method etc. Pads does no OS fingerprinting per say btw.

[PRADS vs p0f]
PRADS tcp fingerprinting was based on the p0fv2 way as p0f had the fingerprint DB and we thought that reusing the fingerprints would make it easier for people to migrate if they wanted, instead of recollecting and adding fingerprints. PRADS also added some touches of its own (for IPv6 etc) and the way we match the fingerprints (and fuzzing). We have thought about extending the fingerprints and re-write them, but thats in the future. Right now they are doing a good job. We also added all the p0fv2 ways of fingerprinting to the whole tcp session, from the syn to the rst/fin. p0fv2 could just use one method at a time, depending on how you started p0fv2. PRADS outputs all the info it gathers, and leaves the final correlation to the end user/program etc. A good example on that is prads-asset-report and prads2snort, which ads wight to each type of fingerprints, ranging the syn and syn+ack higher than stray-ack, rst and fin etc. You can also base the final guess on client or server applications to, say if the User-Agent contains: “Linux” or “Windows NT 6.1″ or “Macintosh; Intel Mac OS X 10.7″ etc.
or if the Server string of the web server is: “Microsoft-IIS 6.0″ or “Apache 2.2.15 (FreeBSD)” or “Apache 2.2.3 (Red Hat)” etc.

The p0fv3 tcp fingerprints are new in the way they are written. A new fingerprint file format, that makes it easy to add different types of fingerprints into one and same file (TCP/HTTP/SMTP etc). The most significant enhancement in the TCP fingerprints that I see is the MSS and MTU multiplier field. p0fv3 also detects new quirks not measured in p0fv2. The rules are now also more human readable, Example:

# RULE
label = s:unix:Linux:2.6.x
sig = *:64:0:*:mss*4,6:mss,sok,ts,nop,ws:df,id+:0

# Will match:
.-[ X.X.X.X/58435 -> Y.Y.Y.Y/22 (syn) ]-
|
| client = X.X.X.X/58435
| os = Linux 2.6.x
| dist = 9
| params = none
| raw_sig = 4:55+9:0:1460:mss*4,6:mss,sok,ts,nop,ws:df,id+:0
|
`—-

The way the tcp fingerprints are matched are also changed a bit, and I believe Michal Zalewski has done this for good reasons and that it will enhance the detection.

Beside the new tcp fingerprint changes, p0fv3 also has application layer detection added. I looked at the HTTP stuff, and p0fv3 matches also on the HTTP header order and dont blindly trust the User-Agent, as we do in PRADS. We have thought about extending the “rule/signature” in PRADS to be more Snort/Suricata like, so you can have more content matches etc, but more accuracy can be achieved today using the pcre language, to verify header order etc, before blindly trusting the UA, but pcre is way too expensive used alone I think, so organizing the signatures/rules better internally and having something like a fast_pattern matcher would help alot. Quick pcre example for a User-Agent with specific HTTP header order:

# Detects Firefox/3.6.X with HTTP header order to add confidence in the match.
# PRADS rule:
http,v/MFF 3.6.X/$1//,\r\nHost: .*\r\nUser-Agent: Mozilla\/5\.0 (.*Firefox\/3\.6\..*)\r\nAccept: .*\r\nAccept-Language: .*\r\nAccept-Encoding: .*\r\nAccept-Charset:

Running it in PRADS on an old pcap gives me:

# Client IPs deducted just to be kind
[client:MFF 3.6.X (X11; U; Linux x86_64; en-US; rv:1.9.2.12) Gecko/20101027 Ubuntu/10.04 (lucid) Firefox/3.:80:6],[distance:8]
[client:MFF 3.6.X (X11; U; Linux x86_64; en-GB; rv:1.9.2.12) Gecko/20101027 Ubuntu/10.10 (maverick) Firefox:80:6],[distance:11]
[client:MFF 3.6.X (Windows; U; Windows NT 5.1; de; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 ( .NET CLR 3.:80:6],[distance:10]
[client:MFF 3.6.X (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 (.NET CLR :80:6],[distance:14]
[client:MFF 3.6.X (X11; U; Linux x86_64; en-US; rv:1.9.2.12) Gecko/20101027 Linux Mint/10 (Julia) Firefox/3:80:6],[distance:15]
[client:MFF 3.6.X (X11; U; Linux x86_64; en-US; rv:1.9.2.12) Gecko/20101027 Ubuntu/10.10 (maverick) Firefox:80:6],[distance:9]
[client:MFF 3.6.X (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12:80:6],[distance:6]
[client:MFF 3.6.X (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6:80:6],[distance:12]
[client:MFF 3.6.X (Windows; U; Windows NT 6.1; es-ES; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12:80:6],[distance:14]
[client:MFF 3.6.X (X11; U; Linux x86_64; en-US; rv:1.9.2.10) Gecko/20101005 Fedora/3.6.10-1.fc14 Firefox/3.:80:6],[distance:8]
[client:MFF 3.6.X (X11; U; Linux x86_64; en-US; rv:1.9.2.12) Gecko/20101027 Ubuntu/10.04 (lucid) Firefox/3.:80:6],[distance:12]

Not the whole User-Agent is grabbed, and we need to extend that in the future. But the pcre language makes it possible to match on as much content as you want, to have the confidence you need in your signatures/rules for detecting assets. PRADS looks for client and server applications on all ports and both UDP and TCP and for IPv4 and IPv6.

[PRADS vs The World]
Right now we are working on adding the DHCP OS fingerprinting and ICMP OS fingerprinting. DHCP is pushed to the git master on github but is not fully integrated into the PRADS core yet, but printing and matching is working, so you can help add fingerprints if you want . The ICMP part is tricky as I want to fingerprint on the protocol layer, and also the payload, so I kind of have to combine the p0f way with the pads way of detecting and matching.

PRADS has also lots of other stuff, like connection tracking/Flow gathering with output compatible with cxtracker and sancp. I have also been working on my passivedns project, and I tend to port the relevant function over to PRADS, so we can have domain names mapped with assets to.

p0fv3 has an API so you can talk to it, to fetch relevant info about the IPs it knows about. I see p0fv3 with this functionality aimed at mail and web servers etc, to determine if this is spam or ham stuff coming its way, but you can use it in lots of cool ways.
I know PRADS is used in much the same way from people I have talked too. An example that Kacper put up can be found on http://prads.delta9.pl/. On the road map for upcoming PRADS releases, we have access to assets via shared memory. That will make it easier for extracting info from the running PRADS process that is current. PRADS also ships with prads2db.pl which parses a prads asset log-file and inserts the info to a DB so you can query it for info.

PRADS philosophy is something like: “If it can be detect passively, PRADS should probably do it.”

So if you are comparing for deciding which application to go for, I would say use them all, and correlate the the knowledge that each tool gives you. You can even add the output from the active fingerprinting tool nmap into the mix.

That said, much of my view on PRADS comes from that I use it in my Network Security Monitoring setup and from my wish to “know as much as possible about my assets”. If you have any wishes or suggestions, god or bad etc, feel free to contact us.

Suricata and some phun with flowints

2012-01-09T07:09:06+00:00

I have been looking into malware traffic that is hard to make signatures for in a “regular” way. I’m not a malware reverser, so I don’t dig into a malware to determine byte-testes and jumps etc. in binary protocols. This lead me to use a lot of flowbits at first, for making my sigs, but the performance in Snort and Suricata was “crap” to say it nice. So I talked to Victor Julien, lead programmer of Suricata, discussing implementing packet and byte counting in Suricata. I want to count each packet sent by a client and server and the total amount of bytes sent by client and server. Talking back and forth, Victor convinced me that I might be best to go for byte count for reassembled streams. So I added a feature request to Suricata. I since then updated the feature request to add the packet and byte counters, as I think they will do great use.

Talking to Matt Jonkman (Emerging Threats Pro), he pointed me to flowint in Suricata to try to solve my packet counting. So in Suricata 1.1.1, you can do something like this to initialize the packet counters:

# Initialize the packet counter (Suricata 1.1.1 and some older versions)
#alert ip $HOME_NET any -> $EXTERNAL_NET any (msg:”Generic Client Established Flow IP Packet Counter set”; flow:established,from_client; flowint:client_packet,notset; flowint:client_packet,=,0; flowbits:noalert; classtype:not-suspicious; sid:1; rev:1;)

#alert ip $EXTERNAL_NET any -> $HOME_NET any (msg:”Generic Server Established Flow IP Packet Counter set”; flow:established,from_server; flowint:server_packet,notset; flowint:server_packet,=,0; flowbits:noalert; classtype:not-suspicious; sid:2; rev:1;)

In Suricata 1.2dev (rev 4c1e417) (I did my test for the blog on this version) and newer, you dont need to initialize the counter, as it will automagical be initialized to zero, so you don’t need sid:1 and sid:2:

## Generic packet counter: (This could be better done internally in Suricata/Snort? and not with rules?)
alert ip $HOME_NET any -> $EXTERNAL_NET any (msg:”Generic Client Established Flow IP Packet Counter”; flow:established,from_client; flowint:client_packet,+,1; flowbits:noalert; classtype:not-suspicious; sid:3; rev:1;)

alert ip $EXTERNAL_NET any -> $HOME_NET any (msg:”Generic Server Established Flow IP Packet Counter”; flow:established,from_server; flowint:server_packet,+,1; flowbits:noalert; classtype:not-suspicious; sid:4; rev:1;)

So, what can you do with packet counters?

First off, lets look at some generic rules I made up to test with, which basically should limit the detections in streams to the first 29 packets from the client:

# GENERiC GET
alert http $HOME_NET any -> $EXTERNAL_NET any (msg:”GENERIC GET (classic)”; flow:from_client,established; content:”GET “; depth:4; content:!”connection: keep-alive”; nocase; http_header; classtype:not-suspicious; sid:5; rev:1;)

alert http $HOME_NET any -> $EXTERNAL_NET any (msg:”GENERIC GET (flowint)”; flow:from_client,established; flowint:client_packet,,30; content:”GET “; depth:4; content:!”connection: keep-alive”; nocase; http_header; classtype:not-suspicious; sid:6; rev:1;)

# GENERiC UA
alert http $HOME_NET any -> $EXTERNAL_NET any (msg:”GENERIC User-Agent (classic)”; flow:from_client,established; content:”User-Agent: “; http_header; content:!”connection: keep-alive”; nocase; http_header; classtype:not-suspicious; sid:7; rev:1;)

alert http $HOME_NET any -> $EXTERNAL_NET any (msg:”GENERIC User-Agent (flowint)”; flow:from_client,established; flowint:client_packet,,30; content:”User-Agent: “; http_header; content:!”connection: keep-alive”; nocase; http_header; classtype:not-suspicious; sid:8; rev:1;)

Sid 5 and 6 looks for a HTTP GET request that is not a HTTP keep-alive. Sid 7 and 8 is looking for User-Agent in non HTTP keep-alive request. Common for the flowint versions of the rules, are that they are just limited to the first 29 packets in an established flow. So running Suricata against 2009-04-20-09-05-46.dmp etc. shows some interesting results:

Num Rule Gid Rev Ticks % Checks Matches Max Ticks Avg Ticks Avg Match Avg No Match
——– ———— ——– ——– ———— —— ——– ——– ———– ———– ———– ————–
1 4 1 1 1695335708 67.74 510720 510720 6412616 3319.50 3319.50 0.00
2 3 1 1 581354624 23.23 508970 82175 3602972 1142.22 3061.99 772.59
3 7 1 1 135943292 5.43 7900 2352 499972 17208.01 16156.62 17653.74
4 5 1 1 43040648 1.72 3313 2517 199052 12991.44 16247.74 2694.82
5 8 1 1 29172972 1.17 7900 2352 434592 3692.78 6588.51 2465.18
6 6 1 1 17917112 0.72 3313 2517 353684 5408.12 6528.93 1864.06

Sorry for the formating
First, if we look at sid 5 and 6, we see that they both where checked 3313 times, and matched 2517 times. If we look at total ticks, sid 5 uses 43040648 ticks and sid 6 (flowint) uses 17917112 ticks. Average ticks for sid 5 is 12991.44 ticks and 5408.12 ticks for sid 6 (flowint).

Looking at sid 7 and 8, we see that they both where checked 7900 times, and matched 2352 times. If we look at total ticks, sid 7 uses 135943292 ticks and sid 8 (flowint) uses 29172972 ticks. Average ticks for sid 7 is 17208.01 ticks and 3692.78 ticks for sid 8 (flowint).

A basic conclusion for this test, is that the rules with the flowint check are faster and will give you the same alerts.
But if we look at the ticks sid 3 and 4 uses to count the all the packets, they are high in total, but low on average ticks. So they are not expensive for each check, but since they are checked (and possibly incremented) for each packet, the total ticks are relative high. Having this in the core of Suricata and Snort, would probably make them less expensive (hint hint).

So what more c00l stuff can we do with packet counters?

Some malware I stumbled upon will give you an example (Mostly used in the Gheg Spam bot, aka Tofsee/Mondera)
b31e4624cdc45655b468921823e1b72b
3c453e40ff63da3c2a914c29b6c62ee0
e8034335afb724d8fe043166ba57cd23

It seems to communicate in a binary way (encrypted), but looking at 5 different pcaps I got, I saw a pattern and my flowint counters came to good use. It seems like the client and server sends packets with a specific payload size in different parts of the communication. I did not see any obvious content to match on, so content matches didn’t seem trivial, and this is a great way to demonstrate my point: Flowint+packet counters to the rescue! Here is a tcpdump output of traffic on port 443 (not including the port 22050 traffic, which is much longer, but the start is the same), so you can see the packets sizes and in which order they do come in this short sessions:

reading from file b31e4624cdc45655b468921823e1b72b.pcap, link-type EN10MB (Ethernet)
03:47:02.571111 IP 192.168.1.10.1031 > 216.246.8.230.443: Flags [S], seq 910650996, win 65535, options [mss 1460,nop,nop,sackOK], length 0
03:47:02.608784 IP 216.246.8.230.443 > 192.168.1.10.1031: Flags [S.], seq 442582883, ack 910650997, win 5840, options [mss 1380,nop,nop,sackOK], length 0
03:47:02.608977 IP 192.168.1.10.1031 > 216.246.8.230.443: Flags [.], ack 1, win 65535, length 0
03:47:02.646959 IP 216.246.8.230.443 > 192.168.1.10.1031: Flags [P.], seq 1:201, ack 1, win 5840, length 200
03:47:02.647342 IP 192.168.1.10.1031 > 216.246.8.230.443: Flags [P.], seq 1:142, ack 201, win 65335, length 141
03:47:02.685098 IP 216.246.8.230.443 > 192.168.1.10.1031: Flags [.], ack 142, win 6432, length 0
03:47:02.718986 IP 216.246.8.230.443 > 192.168.1.10.1031: Flags [P.], seq 201:676, ack 142, win 6432, length 475
03:47:02.718999 IP 216.246.8.230.443 > 192.168.1.10.1031: Flags [F.], seq 676, ack 142, win 6432, length 0
03:47:02.719268 IP 192.168.1.10.1031 > 216.246.8.230.443: Flags [.], ack 677, win 64860, length 0
03:47:02.719584 IP 192.168.1.10.1031 > 216.246.8.230.443: Flags [F.], seq 142, ack 677, win 64860, length 0
03:47:02.757350 IP 216.246.8.230.443 > 192.168.1.10.1031: Flags [.], ack 143, win 6432, length 0

And here is how I sigged it:

# Backdoor:Win32/Tofsee (aka: Gheg / Mondera)
alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:”Possible Tofsee server Packet 2 (200 Bytes)”; flow:established,from_server; flowint:server_packet,=,2; dsize:200; flowbits:set,Tofsee_SERVER_200; flowbits:noalert; classtype:trojan-activity; sid:9; rev:1;)

alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:”Possible Tofsee client Packet 3 (141 Bytes)”; flow:established,from_client; flowint:client_packet,=,3; dsize:141; flowbits:isset,Tofsee_SERVER_200; flowbits:set,Tofsee_CLIENT_141; flowbits:noalert; classtype:trojan-activity; sid:10; rev:1;)

alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:”Possible Tofsee server Packet 4(475 Bytes)”; flow:established,from_server; flowint:server_packet,=,4; dsize:475; flowbits:isset,Tofsee_CLIENT_141; classtype:trojan-activity; sid:11; rev:1;)

Sid 9 looks only for the 2. packet in an established flow from the Server (C&C) and the packet has to have payload size/dsize 200. It then sets the flowbit Tofsee_SERVER_200 if this hits and the rule has noalert, because this could easily trigger a false positive just this check. So we got to do some more checks. Sid 10 checks only Client packet 3, it has to have a payload size/dsize of 141 and flowbit Tofsee_SERVER_200 has to be set for this too match. Sid 10 is also no alert, as we still can check some more, to not be spammed by falses. So sid 11 checks if server packet 4 has payload size/dsize 475, and that flowbit Tofsee_CLIENT_141 is set. No we can give an alert, as this would probably be an unique set of conditions. So testing again with out 2009-04-20-09-05-46.dmp test pcap, we get:

Num Rule Gid Rev Ticks % Checks Matches Max Ticks Avg Ticks Avg Match Avg No Match
——– ———— ——– ——– ———— —— ——– ——– ———– ———– ———– ————–
1 4 1 1 1727862376 63.39 510720 510720 14059784 3383.19 3383.19 0.00
2 3 1 1 508719672 18.66 508970 82176 3689732 999.51 2830.58 646.95
3 7 1 1 140271824 5.15 7900 2352 1013800 17755.93 18570.93 17410.42
4 9 1 1 101662288 3.73 28419 0 6625384 3577.26 0.00 3577.26
5 11 1 1 84264720 3.09 32938 0 612848 2558.28 0.00 2558.28
6 10 1 1 71553560 2.62 32938 0 576132 2172.37 0.00 2172.37
7 5 1 1 42053248 1.54 3313 2517 805736 12693.40 15831.10 2771.81
8 8 1 1 31547660 1.16 7900 2352 153972 3993.37 7039.04 2702.21
9 6 1 1 17944504 0.66 3313 2517 292508 5416.39 6476.95 2062.83

Overall, sid 9, 10 and 11 did not do that bad here. And the best thing is, they all have 0 matches. I ran this on many of my test pcaps, and I’ve not been close to false positives. Sid 10 seems to fire some times, but not the others, so rather unique combo of packets in a stream I guess and a way to sig malware like this. Also, we could add check for the TCP “PUSH” flag in sid 9, 10 and 11 etc to be more accurate if we need.

So the proof of the pudding, running it against a pcap of the malware:

Num Rule Gid Rev Ticks % Checks Matches Max Ticks Avg Ticks Avg Match Avg No Match
——– ———— ——– ——– ———— —— ——– ——– ———– ———– ———– ————–
1 3 1 1 443120 33.03 165 158 102108 2685.58 2731.72 1644.00
2 11 1 1 310420 23.14 259 2 2860 1198.53 2478.00 1188.58
3 4 1 1 302944 22.58 269 269 15376 1126.19 1126.19 0.00
4 10 1 1 257896 19.22 259 3 16484 995.74 7446.67 920.14
5 9 1 1 27088 2.02 10 3 7448 2708.80 5080.00 1692.57

Events:

[**] [1:11:1] Possible Tofsee server Packet 4(475 Bytes) [**] {TCP} 216.246.8.230:443 -> 192.168.1.10:1031
[**] [1:11:1] Possible Tofsee server Packet 4(475 Bytes) [**] {TCP} 84.16.252.136:22050 -> 192.168.1.10:1032

My Tofsee rules fire on all 5 pcaps I looked at initially (and lots more pcaps I tested after that), so hopefully it will fire on all current Tofsee traffic.

I also replied on an e-mail to the snort-user list 3. of November, making the same feature request as I did for Suricata. No one followed up :/ The email should probably be directed to the snort-devel list some time in the future…

I hope this post has been useful, and hopefully we can get some more flowint rules out there, and maybe even get native packet and byte counting in Snort and Suricata one day

Security thoughts for 2012+

2011-12-30T14:15:21+00:00

Quoting Richard Bejtlich: “Prevention will eventually fail!”

And I have always agreed. Accidents do happen, the world is not perfect. So when companies that really spend time and money on security get breached (RSA, Lockheed, Google, [place your company here?]) then you could work out from the theory that you eventually will get breached too.

When you realize and accept that, you may need to redefine the way you think of IT security. You should prepare for the worst, so identifying what would be “the worst” for you (your company) and then identifying you most critical assets should be on the top of your list, and you should start focusing your effort on securing them the most.

Limit the users that have access to the most critical assets (and work on sensitive projects etc). The users also need special attention when it comes to awareness training and follow up. They should also have a good communication with the security staff making it easy to report anything that seems suspicious and get positive feedback no matter what. They are an important part of picking up security issues where your technology fails! So you need them.

The most critical assets needs to be monitored as close to real-time as it gets. The time it takes for an incident detection and till your response should be a minimum, even outside working hours and weekends.

Then the users who has access to theses critical systems should also have special attention/hardening on their OS’s etc. Use a modern operating system and enabling the security functionality all ready there and making sure that executables cant be executed from temporary directory etc. When you got basic security features in place (Including Anti-Virus), you should start looking at centralized logging and alerting on suspicious activities from the logs.
You should also look into implementing different ways of monitoring anomalies for the users usage. When do they normally log on? From where do they normally log on? Are they fetching lots of documents from the file servers? etc. And did they access the fake “secret document” that is there just for catching any suspicious activity? (You need to define your own anomalies).

When the inner core (most valued assets + its users) are “secured”, you should strive to maintain an acceptable level of security on the rest of the corporate office network and also importantly the public facing part. Compromises here can be used to escalate into the “inner core” or to damage your reputation and business affairs, so keeping an acceptable level of security here “as always” is good.

As “Prevention will eventually fail!”, you need to have sufficient logging up and running. So when you do have an incident, the analyst has sufficient data to work with and this will also keep the cost down, as the time it takes to handle an incident will be lower. I’m mostly into Network Security Monitoring, so for me, NetFlow type data, IDS events, full packet capture, proxy logs, and DNS queries logs are some key logs from network that will help me. On the more host side of logging, the more logging, the better… web, email, proxy, spam, anti-virus, file-access, local client logs, syslogs/eventlogs, and so on…..

And remember – if you cant spot any badness, you are not looking hard enough
I always work on the theory that something in my networks are p0wned. That keeps me on my toes and keeps me actively finding new ways to spot badness.

With that – I wish you all a hacky new year!