Planet Redpill Linpro

Nicolai Langfeldt: It's just too easy

noreply@blogger.com (Nicolai) — Thu, 02 Sep 2010 03:07:00 +0000

I'm often among early adopters. Not so with flat screen TVs. Just got a one of the way flat Samsung "LED" ones. My wife wanted white frame, I wanted DLNA support. Digital Living Network Alliance has made a UPnP based standard for connecting multimedia appliances to network resources. Like my (*cough* our) new TV. All our network resources run Linux though, so windows software is no good for us. One of my colleagues figured this out for us, there are multiple DLNA servers for Linux. One of the most promising was TVMOBiLi. And they make Debian and Fedora packages. To get it working with Samsungs you, right now, need a release candidate (I installed 1.4.0.2196).

Start it. Point browser to http://localhost:30888/ where there is a nice web GUI. Click the "SETTINGS" link, and set up all the different directories we use for movies, mp3s and pictures. Fire up media player app on TV, and it already knows about the tvmobili server. And browsing the My Content folder on the TV showed all our files. And Samsung firmwares supports quite a lot of video and sound codecs.

Sometimes being slow to adopt new stuff saves a lot of work. And deprived me of a learning adventure. I still don't know much about DLNA.

Happy though.

Kristian Lyngstol: Wiki.sh mediawiki for geeks

Tue, 31 Aug 2010 15:26:17 +0000

We at Varnish Software use mediawiki internally for various documentation-like tasks, as does Redpill Linpro and a number of projects I’ve been involved with.

The problem is that Firefox and Chrome are web browsers, not editors. I prefer vim. A solution was needed. A geeky one.

I checked around for cli-based mediawiki-editing tools, and found a bunch of broken crappy-ass tools. There were at least two different approaches to fuse-based mediawiki-editing, none of them really worked or were possible to hack without investing far more effort into the matter than it was worth. (Somehow, when people measure the quality of FUSE filesystems and playing games in wine, the definition of “works well” seem to be “it’s theoretically possible with a lot of effort to get it to mostly do something somewhat similar to what you should want it to do.”). Then there were some other tools that were even written in perl(!). The horror.

So I wrote my own. Couldn’t be THAT hard, right? And it gave me the opportunity to hack around with Awk, which is always a bonus.

Presenting wiki.sh

Turns out, it wasn’t that hard.

wiki.sh is 219 line shell script, or 149 lines once comments and blank lines are removed. It uses basic auth to authenticate and uses the mediawiki api to fetch and edit content.

It can be reviewed at:

http://github.com/KristianLyng/wikish/raw/master/wiki.sh

The code repository living at http://github.com/KristianLyng/wikish

As it’s mainly tested by myself on a single wiki, it’s likely to break for other setups. But it’s a simple shell script, and should be fairly straight forward to work with.

It uses curl, “supports” XDG basedirs for configuration (aka: let’s test how many environmental variables are never ever use before we try the default location), allows multiple configs based on PWD, has explicit POST and GET commands, or the more convenient EDIT command, which will fetch a page, open it in your $EDITOR then post the new version if you changed it. It fetches edit-tokens in mediawiki-style, but should probably not be used in environments where you anticipate edit conflicts yet, since it fetches the edit token right before the post, not when it fetches the content.

All in all, it’s a cute little hack. So far, I haven’t been able to make it really break after the initial protocol-mick-mockery was resolved.

Demo

Step 1, pick a page to edit.

Step 2, it’s now fetched and open in vim.

Step 3, edit it however you want, save in the normal vim-way.

Step 4, wikish updates the wiki for you.

The gpg-output is because I keep the password in a gpg-encrypted file. The result and title is to verify that it posted it to the right title.

Step 5: Showing off the horribly complex configuration.

Cute tricks

If you use rewrite rules for mediawiki, and not the overly ugly wikipedia naming scheme of having ‘wiki’ at least three or four times in a url, you’ll want to set ‘$wgUsePathInfo = false’ in mediawiki. This isn’t specific to wiki.sh, but anything that wants to use the api. Otherwise, you’re likely to find yourself editing /Mediawiki/Api.php instead of whatever you were really intending to edit.

An other neat trick I use is to execute “cd /home/kristian/tmp/wikish/” at the top of the config. That way, I can keep all the trash in one place – with the added benefit of being able to do open old wiki pages directly in vim – even if I can’t edit them directly.

I also keep the passwords in a gpg file, as the example points out.

You will almost certainly want to symlink wiki.sh to /usr/local/bin/wikiedit for example. wiki.sh checks how it was invoked: “wikiedit PAGE” is the equivalent of “./wiki.sh EDIT PAGE”, “wikipost PAGE” == “./wiki.sh POST PAGE” etc. PAGE being the actual title you want to edit.

Feel free to post issues on the github issue tracker. I do review them, though if they are not relevant to my own setup, you may want to add a patch too

Jorge Enrique Barrera: Recommended Firefox addons

Fri, 27 Aug 2010 09:15:56 +0000

This post never gets old. I constantly keep finding new and useful addons for Firefox. Here’s the list of what I consider to be some of the best Firefox addons out there.

Adblock Plus

Ever been annoyed by all those ads and banners on the internet that often take longer to download than everything else on the page? Install Adblock Plus now and get rid of them.

MediaplayerConnectivity

Allow you to launch embed video of website in an external application with a simple click.

StartupMaster

Asks for the master password at startup (fixes multiple password prompt).

Tab Mix Plus

Tab Mix Plus enhances Firefox’s tab browsing capabilities. It includes such features as duplicating tabs, controlling tab focus, tab clicking options, undo closed tabs and windows, plus much more. It also includes a full-featured session manager.

Tiny Menu

Replace the standard menu bar with a tiny menu popup.

Tree Style Tab

Show tabs like a tree.

Xmarks

Xmarks is the #1 bookmarking add-on. Install it on all your computers to keep your bookmarks and (optionally) passwords backed up and synchronized. Xmarks also helps you uncover the best of the web based on what millions of people are bookmarking.

It’s all Text!

Right click on a text area, select “It’s All Text!” and edit the text in the editor of your choice.

Web Developer

The Web Developer extension adds a menu and a toolbar with various web developer tools.

BarTab

BarTab can intercept when tabs are loaded in the background or restored after a browser restart and will only load the content when the tab is actually visited. It also allows you to free memory by unloading already loaded tabs, either manually or automatically.

Lars Strand: Bubba Two - some performance numbers

Wed, 25 Aug 2010 18:14:22 +0000

Here are my experience with the Bubba Two NAS so far. Below are some performance numbers.

Write locally:

lks@bubba:~$ dd if=/dev/zero of=bigfile bs=1M count=1K

1024+0 records in

1024+0 records out

1073741824 bytes (1.1 GB) copied, 96.8614 seconds, 11.1 MB/s

Testing raw network traffic:

lks@bubba:~$ iperf -c 192.168.1.10

------------------------------------------------------------

Client connecting to 192.168.1.10, TCP port 5001

TCP window size: 16.0 KByte (default)

------------------------------------------------------------

[  3] local 192.168.1.3 port 54697 connected with 192.168.1.10 port 5001

[  3]  0.0-10.0 sec  76.4 MBytes  64.1 Mbits/sec

Write over NFS (exported with UDP):

root@titan:/mnt/lks# dd if=/dev/zero of=bigfile bs=1M count=1K

1024+0 records in

1024+0 records out

1073741824 bytes (1.1 GB) copied, 174.882 s, 6.1 MB/s

During the test, the CPU clocks in at about 70%. Since the disk can deliver almost the double of that, I guess the bottleneck is the bus.

Downloading over ftp give around 7-8MB/s (~55-65Mbits/s). Samba around 5-6MB/s (~40-48Mbit/s).

These numbers are more than adequate for most of my multimedia needs. A 720p HD film encoded in MPEG2 needs around 20Mbits/s (~2.4MB/s). But since most films are encoded using MPEG4 (or similar) - a proper encoded 1080p movie will only use around 2-3MB/s.

Lars Strand: The Bubba Two NAS

Mon, 23 Aug 2010 20:00:56 +0000

My balcony server finally died on my the other day. It has been running 24/7 for four years in all kinds of weather. I wasn't very surprised - in fact I've been waiting for it to happen. The motherboard had died. I've replaced the motherboard, and its back up. But for how long before a disk or something else fails?

I have backup of (mostly) everything here and there, but I would like to have everything on a separate NAS box. One of the most exciting NAS boxes on the market right now is something called Bubba|Two.

Bubba Two is produced by the Swedish company Excito. Its basically a small Linux server with a big disk. You can use the slick web-interface, or you can ssh into the NAS and treat it like an ordinary Linux-server. It is a LAMP server with SSH running Debian Etch. Samba, proftpd and Mediatomb (upnp) provide the box with file-server capabilities. It even have Squeezecenter installed if you have a Squeezebox (which I happen to have).

Its a ARM processor clocked at 333MHz with 256MB RAM and a 2TB disk. It uses ridiculous low amount of power (max 12W). There is no fan, so the only noise is from the HDD itself - which is barely audible.

Since the default apt-repositories are no longer working (Etch is too old), I change sources.list to:

  deb http://archive.debian.org/debian/ etch main

I can now proceed to install NFS-server, Munin-node and Bind. A couple of minutes later, and its all running smoothly. Too easy.

Jean-Marc Reymond: How to update Smooks in JBoss ESB

noreply@blogger.com (Bambitroll) — Mon, 23 Aug 2010 11:51:00 +0000

The latest JBoss SOA platform (5.0) uses JBoss ESB 4.7 (https://www.jboss.com/products/platforms/soa/components/) which itself uses Smooks 1.2.3 (http://docs.jboss.org/jbossesb/docs/4.7/manuals/html/ReleaseNotes.html)
The latest and greatest Smooks is 1.3.1 and contains certain new features.
So if you absolutely need them in your JBoss Soa Platform 5.0, you will have to update Smooks manually. To do so, download the zip file from http://downloads.sourceforge.net/smooks/smooks-1.3.1.zip, unpack it and replace all the milyn-* from $JBOSS_ESB_HOME/server/default/deployers/esb.deployer/lib/ with the ones from the lib folder in the zip file.

Warning: This configuration is most likely not supported in case you need support from Red Hat.

Ingvar Hagelund: RPMS of varnish-2.1.3

Tue, 17 Aug 2010 13:17:46 +0000

Varnish is a high performance, state of the art HTTP accelerator, used by sites like Facebook and Twitter.

I recently built RPMS of varnish-2.1.3 for RHEL and derivates. Source and binary packages for el4 and el5 on x86_64 are available from http://sourceforge.net/projects/varnish/files/.

Packages for other arches and test builds for el6 may be found here.

As before, packages in the 2.1 series will not reach EPEL4 and EPEL5, as they are stuck at the 2.0 version, and updating to 2.1 needs manual intervention, and possibly changing of the VCL configuration. I’ll try to push an update to 2.1 in EPEL6 soon.

Espen Braastad: munincollector-ng

Mon, 16 Aug 2010 20:15:53 +0000

Munincollector-ng is a perl script that collects graphs from multiple munin installations to display them in one page. A scenario where this is helpful is when you have (too) many munin clients on (too) many munin masters, and you want to look through some of the graphs – i.e. the Disk usage in percent (aka df) plugin – without spending/wasting too much time browsing through the less important graphs.

It consists of one perl script and one configuration file. It is being executed regularly by cron. At each run, it iterates through the configuration file; downloads the graphs to a local directory and generates an html file:

Below is some example configuration that will gather the week and month graphs from the df plugin from four separate munin masters (three without authentication and one with authentication). The graphs will be downloaded to /var/www/munincollector-ng/:

# General configuration
graph.plugin df
graph.type week month
graph.log /var/log/munincollector-ng.log
graph.dir /var/www/munincollector-ng

# Configuration per munin master you want to collect graphs from.
# The format is: <id>.<option> <value>

# Three munin installations with no authentication
uio.url http://munin.ping.uio.no
foo.url http://foo.com/munin/
bar.url http://bar.com/munin/

# One munin master that requires authentication
baz.url http://baz.com/munin/
baz.realm Munin
baz.username user1
baz.password pass1
baz.netloc baz.com:80

An example cron job that will execute the script once per day (make sure user have write permissions in /var/www/munincollector-ng/):

8 8 * * * user /usr/local/bin/munincollector-ng -c /etc/munincollector-ng/example.conf

The current (as of this writing) latest revision is available for download. I might put it in some publicly available revision system one day.

PS: Put the logo.png and style.css from your /etc/munin/templates/ directory into /var/www/munincollector-ng/ to make it look a bit nicer.

David Jensen: Chroot'd Ubuntu on Samsung Galaxy S (Android)

Fri, 13 Aug 2010 22:12:00 +0000

So I got my hands on a Samsung Galaxy S (or SGS) a while ago, and it’s a really nice phone using Android 2.1 as it OS. Before I settled on the SGS the Nokia N900 had caught my eye. The 900 has great hardware and its OS is more of a “standard” Linux. Its Maemo OS is really just a Linux distro with all that entails (X11,QT,GTK), this means you can easily run any Linux app you can compile, and since it’s easy to install Debian or Ubuntu chroot’d on it you can even run things like OpenOffice etc (But the speed and screen real estate won’t be good). The problem is that Maemo is being merged into to Meego and there is only one phone, the N900, which means no apps what so ever except open source ones.

Android is linux based, but it doesn’t use X11 and most applications are written in Java and executed on a custom JVM called ‘Dalvik’. Therefore I was a bit sceptical of buying a android device, it felt a bit stupid to not take advantage of the massive amount of open source programs, libraries and toolkits out there. I still do, but android struck me as a much more mature OS, an it’s very nice to have lot’s of apps :-)

Anyway, having linux underneath means that we can easily install a chroot’d Ubuntu in Android, giving us access to many nice applications… like openssh, python, webservers, transsmission (with web gui) etc. This is nothing new, it has been done before, both with Debian and Ubuntu, but here is how I did it on my SGS. If you don’t know what chroot is, read up before you do anything. For X11 application there is a trick with VNC to get them running, I haven’t tried it yet though.

A couple of prerequisites:

Root access (su)
busybox

I got both by rooting my phone after this guide .

Partition

First of we need somewhere to put the system, android uses FAT32 for the SD card and that’s no good for us. We need a real file system, like ext4. Unfortunately the kernel in my SGS didn’t support it. You can get a list of supported filesystems with ‘cat /proc/filesystems’. This is from my SGS:

$ cat /proc/filesystems
nodev   sysfs
nodev   rootfs
nodev   bdev
nodev   proc
nodev   cgroup
nodev   binfmt_misc
nodev   sockfs
nodev   pipefs
nodev   anon_inodefs
nodev   tmpfs
nodev   inotifyfs
nodev   devpts
    ext3
    ext2
    cramfs
nodev   ramfs
    vfat
    msdos
    romfs
    rfs
    j4fs

So we settle for ext3. The easiest way to partition the SD card is to connect it to you’re computer (not through the phone), and use gparted. I used the SD card reader on my laptop with the use of an adapter that came with the card when I bought it. It’s best to just shrink the vfat partition because if you remove it your phone is going to complain that the card is broken or needs formatting. If your starting from scratch just make sure the vfat partition is the first partition.

The size of the ext3 partition of course depends on the size of your SD card. But at least 1 gigabyte is probably a good idea. I used a 2 gigabyte card, 512 megabytes for the vfat, the rest for the ext3 partition. Don’t forget to tell gparted to format your new partition.

Create a rootfs with rootstock

Rootstock is a great tool for building a basic ubuntu image to use with your ARM based device. It uses good old debootstrap, running the second stage of the installation (the one you usually need to run on the device) under qemu. Pretty smart. This means that the tarball you get is a fully finished root base file system. You can even specify some extra packages that rootstock can install. A SSH server is a good idea, I like less, screen, ne (nice editor, like nano but with ordinary key-bindings, like ctrl-s etc), build-essential so that we can compile things.

Note that you specify a user and a password. You should choose a good password.

Create a rootfs tarball:

$ sudo rootstock --fqdn ubuntu --login sgs --password temporary --imagesize 1G --seed openssh-server,less,build-essential,screen,ne

Unpack the tarball and copy the content to your fine new partition on the SD card. Then set up googles DNS server (the ext3 partition is mounted on /mnt):

$ mv mv armel-rootfs-201008012234.tgz /mnt/
$ cd /mnt
$ tar xvzf armel-rootfs-201008012234.tgz
$ rm armel-rootfs-201008012234.tgz 
$ echo 'nameserver 8.8.8.8' > etc/resolv.conf

It’s time to put the card back into the phone…

Now on your phone

For the rest you need to drop into root:

$ su
#

First mount the ext3 partition, I use a dir on the SD card called “ubuntu”, you can put it anywhere you want though. Since the SGS has internal storage too the SD card has the device /dev/block/mmcblk1p2

# busybox mount /dev/block/mmcblk1p2 /sdcard/sd/ubuntu

chroot by itself is not enough, we need to mount /proc,/dev and /sysfs too. An the path on android is a bit strange so we need to set that too.

Tap this on your phone for a glorious chrooted ubuntu:

# busybox mount -t proc proc /sdcard/sd/ubuntu/proc
# busybox mount --bind /sys /sdcard/sd/ubuntu/sys
# busybox mount --bind  /dev /sdcard/sd/ubuntu/dev
# export PATH=/bin:/usr/bin:/sbin:/usr/sbin:$PATH
# busybox chroot /sdcard/sd/ubuntu

Log in via SSH

When your chroot’d you can start the SSH server and log into it that way. This will give you full terminal emulation (as opposed to “adb shell”), i.e.bash completion, emacs, vim etc.

Start server and check what ip nr your phone has

# /etc/init.d/ssh start
# ifconfig | grep "inet addr"

Note that since this is Ubuntu the root user doesn’t have any password so you have to ssh to it with the user name and password you specified to rootstock. In my example above that’s “sgs” and “temporary”. To get root privileges use sudo, remember this is Ubuntu.

A script to help

If you think tapping all that mounts and path etc on your phone is a bit tiresome to tap you can make a script of it. Shell scripting is not my strong point so it can probably be improved, but it seems to work for me. It mounts the ext3 partition and sets up everything needed. Note that it does not drop you into a chroot’d ubuntu, it just starts the ssh server.

#!/bin/sh
#we need to setup test [
busybox ln -s /system/xbin/busybox /system/xbin/[

if [ -d /sdcard/sd/ubuntu/etc ] ; then
    echo "ubuntu partition is already mounted"
else
    echo "Mounting ext3 partition to /sdcard/sd/ubuntu"
    busybox mount -o noatime /dev/block/mmcblk1p2 /sdcard/sd/ubuntu

    echo 'Mounting proc,sys and dev'
    busybox mount -t proc proc /sdcard/sd/ubuntu/proc
    busybox mount --bind /sys /sdcard/sd/ubuntu/sys
    busybox mount --bind  /dev /sdcard/sd/ubuntu/dev
fi

echo 'Exporting a proper path'
export PATH=/bin:/usr/bin:/sbin:/usr/sbin:$PATH

echo 'Chrooting and starting ssh server'
busybox chroot /sdcard/sd/ubuntu /etc/init.d/ssh start

echo "And your ip is:"
busybox ifconfig | busybox grep "inet addr"

Put it on the sdcard or the on the internal storage and execute it like this:

$ su
# sh /sdcard/sd/ubuntu.sh

Kristian Lyngstol: Stripping cookies with VCL

Fri, 13 Aug 2010 19:07:30 +0000

The easiest way to get Varnish to help you out of the box, is to strip cookies you don’t want. Other than that, the default settings are pretty good.

So to ease that whole mess, I wrote a bash script for the occasion, maintained as of now at http://varnish-cache.org/browser/trunk/varnish-tools/cookie-stripper. It can create everything you need for a VCL file to just remove cookies and define a web server. A copy of the current version is here:

#!/bin/sh
# Author: Kristian Lyngstol
# Date: August 2010
# More info: http://kristianlyng.wordpress.com
# License: Consider it public domain.
#

usage() {
	echo
	echo "Varnish Cookie-strip VCL generator"
	echo "Copyright (C) 2010 Kristian Lyngstol "
	echo
	echo "$0 [-b backend:port] [-i indentation] [-c cookie] [-c cookie] ..."
	echo "Generates VCL for Varnish to strip cookies."
	echo " -c \tStrip cookie named \`cookie'. Can be"
	echo "            \tspecified multiple times"
	echo " -b \tAlso generate the backend definition to use"
	echo "               \ta web server at host:port as the backend. Both"
	echo "               \thost AND port, separated by colon, must be specified"
	echo " -t \tUse \`indent' when indenting the code, instead of"
	echo "            \t4 spaces."
	echo " -i         \tCreate case-insensitive cookie matching"
	echo
	echo "Example: $0 -b localhost:8181 -c __utmz -c __utma > /etc/varnish/default.vcl"
	echo
	echo "The VCL generated is for Varnish 2.1 and beyond and with the"
	echo "-b option there is no need to add or modify the final VCL to"
	echo "make Varnish \"just work\""
}
ind() {
	echo -n "$indent"
}

TEMP=`getopt -o hb:c:it: -n $0 -- "$@"`

if [ $? != 0 ] ; then echo "$(usage)" >&2 ; exit 1 ; fi

eval set -- "$TEMP"

icase=""
indent="    "
backend=""
cookiestring=""

while true ; do
	case "$1" in
		-b) backend="$2"; shift 2;;
		-i) icase="(?i)"; shift 1;;
		-t) indent="$2"; shift 2;;
		-c) cookiestrip="$cookiestrip $2"; shift 2;;
		-h) usage; exit 0 ;;
		--) shift; break ;;
		*) echo "Internal error! See $0 -h" ; exit 1 ;;
	esac
done

host=`echo $backend | sed s/:.*//`
port=`echo $backend | sed s/.*://`

if [ ! -z "$backend" ]; then
	if [ -z "$host" ] || [ -z "$port" ]; then
		echo "Invalid backend \"$backend\"" 1>&2
		echo "$(usage)" 1>&2
		exit 1;
	fi

	echo "backend default {"
	ind
	echo ".host = \"$host\";"
	ind
	echo ".port = \"$port\";"
	echo "}"
	echo
fi

if [ -z "$cookiestrip" ] && [ -z "$backend" ]; then
	echo "No -c or -b option specified." 1>&2
	echo "$(usage)" 1>&2
	exit 2;
fi

echo "sub vcl_recv {"
for a in $cookiestrip; do
	ind
	echo "# Remove cookie $a"
	ind
	echo -n 'set req.http.cookie = regsub(req.http.cookie,';
	echo -n "\"${icase}$a=[^;]*;?( |$)\""
	echo ',"");'
done

echo
ind
echo "# Remove the cookie header if it's empty after cleanup"
ind
echo 'if (req.http.cookie ~ "^ *$") {'
ind
ind
echo 'remove req.http.cookie;'
ind
echo '}'
echo '}'

Since I don’t have any cookie-needing web sites or servers, I haven’t tested it properly. I figure someone else will test it if it’s a worthwhile script to have around.

Feedback welcome.

Jorge Enrique Barrera: Introduction to irssi

Tue, 10 Aug 2010 07:09:26 +0000

There are a load of IRC clients for UNIX-based systems out there, but I personally feel that irssi suits me best. Incredibly flexible, adjustable for every user, and doesn’t depend on X to be able to run it. Haven’t really found a guide out there that explains things in an easy and understandable manner, aimed particularly at new users. So here’s my attempt at it.

Table of contents

Getting started

What we need, of course, is irssi itself. You can get it at http://www.irssi.org/download, or use your system’s package manager to download it.

yum install irssi

..for RPM-based systems.

sudo aptitude install irssi

..for Debian/Ubuntu type of distributions.

Now, once installed, you can fire up irssi by typing

irssi

in the commandline. You should now be running irssi, whoo!

Customizing the settings

Now that irssi is up and going, you can change the settings, adjust it so that if suits you. By typing:

/SET <KEYWORD>

you will get a list of settings. For instance, let’s try out /SET nick. This is what pops up:

<09:07> [completion]
<09:07> completion_nicks_lowercase = OFF
<09:07> [lookandfeel]
<09:07> hilight_nick_matches = ON
<09:07> query_track_nick_changes = ON
<09:07> show_nickmode = ON
<09:07> show_nickmode_empty = ON
<09:07> show_own_nickchange_once = OFF
<09:07> [misc]
<09:07> netjoin_max_nicks = 10
<09:07> netsplit_max_nicks = 10
<09:07> netsplit_nicks_hide_threshold = 15
<09:07> [server]
<09:07> alternate_nick = joba7_
<09:07> nick = joba7

Here you see the various settings containing the word nick. irssi will automatically set your nick based on your account name. So if your username on your machine is foo, irssi will set your nick to foo. You can of course change this:

/SET nick joba7
/SET alternate_nick joba7_

..followed up by a /SAVE and /RELOAD to save and reload the settings.

Note: Every time you change a default setting, you will have to use /SAVE to save the configuration.

Connecting to an IRC-network

The usual way to connect to an IRC-network is to use /SERVER or /CONNECT. After having started up irssi, you can type:

/SERVER powertech.no.eu.dal.net

This will then connect me to the server powertech.no.eu.dal.net on the DALnet network. At the same time though, I’d like to be on the EFnet network. You can connect to multiple networks by typing:

/CONNECT irc.homelien.no

This connects you to the EFnet network as well. You are perhaps wondering how you can change from one network to the other? Look at your status-window. This is usually window number 1. Use ALT-1 or ESC-1 to change to that window. Right now my active network is EFnet, on the server irc.homelien.no. Which means that I’m seeing this:

[08:38] [joba7(+i)] [1:homelien (change with ^X)]

«Change with ^X» – ^X means Ctrl-X, so upon pressing it, you will change your active network. You will see something like this in your status window:

08:38 -!- Irssi: Changed to server powertech.no.eu.dal.net

To disconnect from one of the servers, or stop irssi’s attempt at reconnecting, use:

/DISCONNECT <network> – Disconnects server with tag network.
/DISCONNECT recon-1 – Cancels an attempt to connect to the server named recon-1.
/RMRECONNS – Cancels all attempts to reconnect.
/RECONNECT recon-1 – Will attempt to reconnect you to recon-1.
/RECONNECT ALL – Will attempt to reconnect to all the servers in the reconnection queue.

You can scroll upwards/downwards with the Page Up / Page Down keys. If they aren’t working for you, you can use Meta-p / Meta-n, where Meta can be ESC and/or ALT.

Default behavior is that irssi pretty much creates hidden windows for just about everything. Hidden windows are created every time you join a channel or start a conversation. There are different ways to switch between them:

Meta-1 … Meta-0- Switch between window 1-10.
Meta-q … Meta-o – Switch between window 11-19.
/WINDOW <number> – Switch to window number <number>.
Ctrl-p / Ctrl-n – Switch to the previous/next window.

Adding networks, servers and channels

On the network DALnet you have a function called Nickserv, which lets you register a nick, so that you can become the owner of it. To be able to use this nick you will have to identify to it when logging on to the IRC network. It’s a bit tiresome having to type

/nickserv identify <yournick> <yourpassword> every time you connect. I’m assuming you want this a bit more automated? I at least do. :) But before we do this we have to do a couple of things. Like adding a network:

/IRCNET ADD <YourNetwork>

For example:

/IRCNET ADD DALnet

If you’d like to execute a command when logging on, as in the example above with nick-identification, use -autosendcmd:

/IRCNET ADD -autosendcmd "/nickserv identify <YourNick> <ThePassword>;wait -DALnet 2000" DALnet

The command above automatically sends the identify-command to Nickserv when you log on. The -wait 2000 tells irssi to wait two seconds after having logged on to the server before sending the command.

Now that we’ve added a network, we need servers to go along with it. The command to do so is:

/SERVER ADD -ircnet DALnet powertech.no.eu.dal.net 6667

If you’d like to automatically connect to this server when you start irssi, simply add an -auto right after ADD. Specifying the port is optional.

So, first network, then server, now for a channel:

/CHANNEL ADD -auto #linuxhjelp DALnet [password]

[password] is optional, use of it depends on if the channel you’re trying to join needs a password to be accessed.

Logging

irssi has the ability to log both away-messages and regular conversations in channels. Logging for away messages is enabled by default, but not regular logging. First, it’s possible to define what the awaylog should log:

/SET awaylog_level MSGS HILIGHT – Defines what kind of messages to log.
/SET awaylog_file ~/.irssi/away.log – Defines the file the log should write to.

The easiest way of enabling logging, and not just away messages, is to execute:

/SET autolog ON

Per default irssi logs to ~/irclogs/<Servername or Networkname>/<Channel or nick>.log. This can be defined by:

/SET autolog_path ~/irclogs/$tag/$0.log

That is the default value. The path will automatically be created if it doesn’t exist. $0 points to the target (channel/nick). You can make irssi to automatically rotate the logs by adding a time/date stamp to the filename. You find more about the formatting used by typing man strftime in your commandline. But for example:

/SET autolog_path ~/irclogs/%Y/$tag/$0.%m-%d.log

%Y for year, %m for month and %d for day.

For more information about logging have a look at:

/HELP LOG

Scripts

If you’ve used other IRC clients, you’ve probably encountered the possibility to use scripts. Scripts can be used for various things, from showing a channel what song you’re currently playing to flaunting your machine’s hardware specs. A good site for scripts is http://scripts.irssi.org.

The scripts you download should be placed in ~/.irssi/scripts/. If the folder doesn’t exist, create it yourself. Once the script is placed there, you can load it in your irssi by doing a:

/SCRIPT LOAD ~/.irssi/scripts/TheScript.pl

This is a good approach if you don’t want certain scripts to load when irssi starts. If you do however want them to load when starting irssi, create a symlink in the folder ~/.irssi/scripts/autorun, just create the folder if it doesn’t exist. Symlink, what’s that? Well, it’ss a special type of file that contains a reference to another file or directory in the form of an absolute or relative path. So in other words:

cd ~/
wget http://scripts.irssi.org/scripts/splitlong.pl
mkdir -p ~/.irssi/scripts/autorun
mv ~/splitlong.pl ~/.irssi/scripts
cd ~/.irssi/scripts/autorun
ln -s ../splitlong.pl

So the file is really in ~/.irssi/scripts, but you’ve made a shortcut to it from ~/.irssi/scripts/autorun. Pretty neat if you ask me.

To unload a script simply use:

/SCRIPT UNLOAD %lt;Name of script&gt;

Minor adjustments

It’s not just what I’ve mentioned above that you can adjust and customize to your own liking. Another thing is the use of themes. You’re probably using the default blue theme that irssi comes with, which is just fine for my use, but perhaps not for you. I do however like the minor modification just for kicks.

Every time you use /SAVE, a message like this will appear in your status window:

08:38 -!- Irssi: Saved configuration to file /home/jorge/.irssi/config
08:38 -!- Irssi: Theme saved to /home/jorge/.irssi/default.theme

Settings are stored in the configuration file named config and the theme in default.theme. Both files are editable through a text editor. I’ve never really changed things manually in the config-file, but I’ve tampered with default.theme a bit, like what my own nick appears to me. Default appearance of your nick is like so:

< YourNick>

Where YourNick is in bold white, while the inequality signs are in gray.

Open the file ~/.irssi/default.theme and find the line that says something like:

ownmsgnick = "{msgnick $0 $1-}";

Change this to:

ownmsgnick = "%R<%n$0$1-%R>%n %|";

This will make the inequality signs turn red. Your nick will still be in white bold. But you can always change the inequality signs to something else. For instance:

ownmsgnick = "%Y(%n$0$1-%Y)%n %|";

Will make the parenthesis turn yellow. If you want to change the color of your nick, from default white to something else, find this line:

ownnick = "%W$*%n";

..and replace it with the color you want:

ownnick = "%G$*%n";

For a nice overview of colors, check out http://irssi.org/documentation/formats, and for more information about irssi, visit http://www.irssi.org/.

And here’s a screenshot of irssi:

Good luck!

Kacper Wysocki: tune2fs and green disks

Thu, 05 Aug 2010 10:05:22 +0000

Hey folks,
old news I’m sure, but if you get tempted into buying the new WD Caviar “Green Power” disks there is something you need to know about them: they fake 512-byte blocksizes while in reality having 4096-byte blocks! The move to 4K blocks is reasonable considering we just busted the 2 terabyte barrier, but the disk firmware is faking 512-byte blocks in the name of compatibility (read: so windows xp won’t shit itself).

Unfortunately, running in bs512 mode makes the disk exactly 3x slower than it should be!
The fix: line up your partitions at 4k boundries, so start partition one at block 64, 1024 or even 2048 (the win7 start block) not the default, 63, in most partitioning software. Start fdisk with the -u parameter and carefully specify the start block. In gparted you’ll have to unhook the “snap to cylinder boundries” checkbox, and then I suppose you could even move a partition to the right block, but expect this to take an inordinate amount of time!

On a related note, fsck’ing an ext filesystem on boot is a drag, and fsck’ing 2TB file systems is a huge drag. Sure you should be running the fsck but it has a nasty tendency to happen on your workstation precisely when you can’t afford the extra 5 minute delay!

I bump the default 10 mounts count to 0 (disabling mount count fscking) and auto-fsck my disks every 99 days, staggered so not all disks get checked on the same day. Do this with the tune2fs command:

wasp:~# tune2fs -c 0 -i 99d /dev/sda1
tune2fs 1.41.12 (17-May-2010)
Setting maximal mount count to -1
Setting interval between checks to 8553600 seconds

out.

PS I recently managed to achieve sustained throughputs of 110MB/s with these WD disks and properly aligned partitions:

7516192768 bytes (7.5 GB) copied, 68.4392 s, 110 MB/s
115+0 records in
114+0 records out

yes that’s disk-to-disk with ext4 and one large file, no fragmentation.

PPS the defaults have nowadays changed to 120 days and 39 mounts, to which I say -1 mounts is better anyway!

edit: Now that your files are aligned, you can specify a block size to mkfs as well, which might avoid unaligned fragments: mkfs.ext4 -b 4096 -L gigantor -O sparse_super /dev/sdb1

Edward Bjarte Fjellskål: snort-2.8.6.1 debian/ubuntu packages

Thu, 05 Aug 2010 09:58:31 +0000

Back from vacation

I did pack 2.8.6.0, but it never made it to the public before I went on vacations :/

You can find 2.8.6.1 here:
http://debs.gamelinux.org/snort/hardy/

-*> Snort! <*-
Version 2.8.6.1 IPv6 GRE (Build 39)

Sigvard Lyth: A vacation in Bornholm

noreply@blogger.com (Sigvard) — Thu, 05 Aug 2010 09:33:00 +0000

My family and I decided a year or so ago, that we wanted to take a trip together. We wanted something active, and maybe a little sportish, and also wanted it to be possible to change location from time to time. My wife quite quickly found Bornholm for a location, and bicycling as the activity. We invited all the familly to join us, but sadly only got my mom to join us, as the final decissions came quite late.

So, finally, we were ready to lieve. For easy departure we went to my grandma, who lives in Akershus, the day before our flight. I bother to mention this because an important part happend at this point... I lost my smartphones during this bit...That means that I didn't write this on my htc Magic. :) Rather I write it on my wife's eee 701, running ubutu 10.04.

Christopher Grenness: Solving Android Market Download Problems

Tue, 03 Aug 2010 18:53:51 +0000

I can’t promise that I have the solutions for all, but hopefully others can chip in with their tricks, workarounds and hopefully positive results.

So, for the past few months, I have struggled when trying to download apps from Android Market.
I had no problems downloading apps at work, and it also worked over 3G (but as I don’t have an unlimited data plan, it’s not preferred…).
At the same time, logging into Gtalk is impossible.

I found several threads online discussing the problem – even an “official” thread started by a Google employee, which unfortunately has been closed without offering a definite solution, at least not for me – but you can still read it here.

So that leads me to the two solution suggestions I have to offer, but if anyone have other suggestions please add them as comments.

If you are a Google user in UK, and have been using @googlemail.com as your email address and Google login name, try to replace @googlemail.com with @gmail.com, and if that doesn’t solve it, follow this how-to to clear caches etc (see the comments below the article)
If you’re not a UK Google user, but still face problems like I described, try what solved it for me – access your firewall (either a standalone box or typically integrated in your DSL modem, your wifi access point, or similar), and open up for traffic on TCP and UDP ports 5228. That’s all I had to do, and I hope this will solve it for others as well.

Please add comments if none of these suggestions are helping, or if you have other solutions to this problem.

Kristian Lyngstol: Varnish backend selection through DNS

Mon, 02 Aug 2010 20:19:25 +0000

A common challenge to using a cache is maintaining a mapping between public site names and actual web servers (backends). If you only have one type of web server (or maybe two?), and it’s fairly static, this isn’t a big deal. However, if your infrastructure spans tens of different types of web servers, then it starts getting iffy. Here’s an example of how this could look:

director sports round-robin {
    { .backend = { .host = "sports1.internal.example.net"; .port = "80"; } }
    { .backend = { .host = "sports2.internal.example.net"; .port = "80"; } }
    { .backend = { .host = "sports3.internal.example.net"; .port = "80"; } }
}
director shop round-robin {
    { .backend = { .host = "shop1.internal.example.net"; .port = "80"; } }
    { .backend = { .host = "shop2.internal.example.net"; .port = "80"; } }
    { .backend = { .host = "shop3.internal.example.net"; .port = "80"; } }
}
director economy round-robin {
    { .backend = { .host = "economy1.internal.example.net"; .port = "80"; } }
    { .backend = { .host = "economy2.internal.example.net"; .port = "80"; } }
    { .backend = { .host = "economy3.internal.example.net"; .port = "80"; } }
}
director main round-robin {
    { .backend = { .host = "main1.internal.example.net"; .port = "80"; } }
    { .backend = { .host = "main2.internal.example.net"; .port = "80"; } }
    { .backend = { .host = "main3.internal.example.net"; .port = "80"; } }
}
sub vcl_fetch {
    if (req.http.host ~ "sports.example.net$") {
        set req.backend = sports;
    } elsif (req.http.host ~ "shop.example.net$") {
        set req.backend = shop;
    } elsif (req.http.host ~ "economy.example.net$") {
        set req.backend = economy;
    } else {
        set req.backend = main;
    }
}

This is obviously a bit of a drag, and so far we only added four sites.

Enter the DNS director

The DNS director allows you to define a single director containing one or more backends, just like any other backend director, but uses DNS to decide which one to pick. Simply put, it does a DNS lookup on the Host header and sees if it has a backend that matches.

Notice that it does NOT automatically try whatever IP the Host header resolves to. It has to know about the backend in advance. This might sound like a bit of a major flaw, but I choose to look at it as a safety net.

The DNS director also allow you to add a postfix to the host-name before it is looked up, so www.example.com could become www.example.com.internal.example.net. It has rudimentary DNS round-robin support and caches the DNS lookups (both successful lookups and misses). Since there isn’t a practical way of obtaining the TTL of a DNS result except apparently hand-coding the resolver or possibly adding some obscure dependency, the life-time of the DNS cache is defined by a setting in the director, cleverly named .ttl.

As a last added bonus, I also added a really easy way to shoot yourself in the leg. With the DNS director, you can specify a range of backends using .list and a acl-like syntax. However, remember that adding 10.0.0.0/8 means Varnish will internally generate 16 million backends. That’s PROBABLY not a good idea. So do use some moderation. A /24 or two shouldn’t be a big deal, but I’d try to narrow it down as much as possible.

Here’s the above example, except using FOO.example.net.internal.example.net instead of FOO<N>.internal.example.net, and assuming that the web servers are all in the 192.168.0.0/24 range or 172.16.0.0/24

director mydir dns {
    .list = {
        .port = "80";
        .connection_timeout = 0.4;
        "192.168.0.0"/24;
        "172.16.0.0"/24;
    }
    .ttl = 5m;
    .suffix = "internal.example.net"
}
sub vcl_recv {
    set req.backend = mydir;
}

Specifying connection timeout and similar attributes is optional in .list, but has to be before the list of IPs. You do not have to use .list, you can also add backends the same way you would with the random or round-robin director.

The above examples caches the DNS results for 5 minutes. I’ve also added some counters (visible through varnishstat): Number of DNS lookups, DNS cache hits, failed DNS lookups and how often the DNS cache is full. You may still want to do some basic sanitizing of domain names so as to reduce DNS spam, but now you can probably just use one regsub to match a number of sites.

Availability

The DNS director was committed to Varnish development trunk yesterday (Sunday, August 1st 2010) and I expect it to be available in Varnish 2.1.4. It has already been used in production at a few customer sites, with good results. Like any non-trivial piece of code, there are certain aspects of it I want to improve, but I do not foresee that as a blocker for including it in a release. It does not affect the rest of Varnish at all if it not used (unless you count adding 4 counters to varnishstat).

If you want to test it, you’ll have to use Varnish trunk. Alternatively, you can check out my github repo which is currently sitting at Varnish 2.1.2 + DNS director + return(refresh). It does look like you’ll have to compile from source, though. Unless, of course, you’re a Varnish Software customer, then you just drop us a mail and you’ll get your rpms or .debs shortly. (My marketing hat is currently firmly planted on my head).

The development of this feature was sponsored by Globo and Mercado Libre and implemented by myself/Varnish Software.

Shervin Asgari: Migrating from JODConverter 2 to JODConverter 3

noreply@blogger.com (Shervin Asgari) — Mon, 02 Aug 2010 14:44:00 +0000

In the previous posting I showed you how you could automate conversions of documents to PDF & PDF/A using JODConverter 2.

JODConverter 3.0.beta has been out for some time, and even though it is still beta, it is very stable. Maybe even more stable than JODConverter 2.

In this blog posting I will highlight the benefits of JODConverter 3 compared to its predecessor and show you how you can modify your code to create PDF/A documents with JODConverter 3.

JODConverter 2 versus 3
JODConverter 3 still uses OpenOffice.org to perform its conversion. It is still a wrapper to the OOo API. It is only a complete rewrite of the JODConverter core library which is much cleaner and easier to use.

Whats new?

No more init script(!)

You don't have to manually start OpenOffice.org as a service anymore. This will be handled automatic.
You can even create multiple processes which is useful for multi-core CPU's. Best practise is one process for each CPU core.

Automatically restart an OOo instance if it crashes.

If for some reason your process crashes, JODConverter will detect this, and restart the process automatic. This was a hassle with JODConverter 2, as you needed to manually do this in Linux.

Abort conversions that take too long (according to a configurable timeout parameter)
Automatically restart an OOo instance after n conversions (workaround for OOo memory leaks)

Additionally the new architecture will make it easier to use the core JODConverter classes as a generic framework for working with OOo - not just limited to document conversions.
I am sure there will be more features when JODConverter 3 goes final.

Configuration

All you need to do do is point your OpenOffice.org installation to the OfficeManager, and you are good to go.

OfficeManager officeManager = new DefaultOfficeManagerConfiguration()
        .setOfficeHome("/usr/lib/openoffice")
        .buildOfficeManager().start();

This manager will use the default settings for Task Queue Timeout, Task Execution Timeout, Port Number etc but you can easily change them

OfficeManager officeManager = new DefaultOfficeManagerConfiguration()
        .setOfficeHome("/usr/lib/openoffice")
        .setTaskExecutionTimeout(240000L)
        .setTaskQueueTimeout(60000L)
        .buildOfficeManager().start();

If you want to utilize piping (Recommended is one process per CPU-core), you will need to set VM argument and point java.library.path to the location of $URE_LIB which on my Ubuntu machine is /usr/lib/ure/lib/
For instance:

-Djava.library.path="/usr/lib/ure/lib"

And then you can change your OfficeManager.

OfficeManager officeManager = new DefaultOfficeManagerConfiguration()
        .setOfficeHome("/usr/lib/openoffice")
        .setConnectionProtocol(OfficeConnectionProtocol.PIPE)
        .setPipeNames("office1","office2") //two pipes
        .setTaskExecutionTimeout(240000L) //4 minutes
        .setTaskQueueTimeout(60000L)  // 1 minute
        .buildOfficeManager().start();

ConverterService3Impl
The following codes performs all the converting. It supports a File or byte[] as input.

This is how you use it:
Lets say you have a PDF file as byte[], and you want to convert this byte to PDF/A as byte.
All you would have to do is call method:

byte[] pdfa = converterService.convertToPDFA(pdfFile);

Similarly, if you have a Document (say a OpenOffice.org writer document) and you want to convert this to PDF you would call the method:

File doc = new File("myDocument.odt");
File pdfDocument = converterService.convert(doc, ".pdf");

Note that you will always get a PDF/A compliant pdf. All you need to do is change the extension from ".pdf" to ".html" and the converter would do the magic.

Here is the source. Please read the comments in the source code if you want to understand it, or just ask in the comment section below.

import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.ConnectException;
import java.util.HashMap;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

import javax.ejb.Local;
import javax.ejb.Stateless;

import lombok.Cleanup;

import org.apache.commons.io.FilenameUtils;
import org.apache.commons.io.IOUtils;
import org.artofsolving.jodconverter.OfficeDocumentConverter;
import org.artofsolving.jodconverter.document.DefaultDocumentFormatRegistry;
import org.artofsolving.jodconverter.document.DocumentFamily;
import org.artofsolving.jodconverter.document.DocumentFormat;
import org.artofsolving.jodconverter.document.DocumentFormatRegistry;

/**
 * This service converts files from one thing to another ie ODT to PDF, DOC to ODT etc
 * @author Shervin Asgari
 *
 */
@Stateless
@Local(ConverterService.class)
public class ConverterService3Impl implements ConverterService {

  private static final String PDF_EXTENSION = ".pdf";
  private static final String PDF = "pdf";  

  // Uncomment these when we want to use them

  // private final int PDFXNONE = 0;
  private final int PDFX1A2001 = 1;
  // private final int PDFX32002 = 2;
  // private final int PDFA1A = 3;
  // private final int PDFA1B = 4; 


  @Logger //Your favourite logger (ie Log4J) could be injected here 
  private Log log;

  public File convert(File inputFile, String extension) throws IOException, ConnectException {
    if (inputFile == null) {
      throw new IOException("The document to be converted is null");
    }

    Pattern p = Pattern.compile("^.?pdf$", Pattern.CASE_INSENSITIVE);
    Matcher m = p.matcher(extension);
    OfficeDocumentConverter converter;
    
    //If inputfile is a PDF you will need to use another FormatRegistery, namely DRAWING
    if(FilenameUtils.isExtension(inputFile.getName(), PDF) && m.matches()) {
      DocumentFormatRegistry formatRegistry = new DefaultDocumentFormatRegistry();
      formatRegistry.getFormatByExtension(PDF).setInputFamily(DocumentFamily.DRAWING);
      converter = new OfficeDocumentConverter(officeManager, formatRegistry);
    } else {
      converter = new OfficeDocumentConverter(officeManager);
    }
    
    String inputExtension = FilenameUtils.getExtension(inputFile.getName());
    File outputFile = File.createTempFile(FilenameUtils.getBaseName(inputFile.getName()), extension);

    try {
      long startTime = System.currentTimeMillis();
      //If both input and output file is PDF
      if (FilenameUtils.isExtension(inputFile.getName(), PDF) && m.matches()) {
        //We need to add the DocumentFormat with DRAW
        converter.convert(inputFile, outputFile, toFormatPDFA_DRAW());
      } else if(FilenameUtils.isExtension(outputFile.getName(), PDF)) {
        converter.convert(inputFile, outputFile, toFormatPDFA());
      } else {
        converter.convert(inputFile, outputFile);
      }
      long conversionTime = System.currentTimeMillis() - startTime;
      log.info(String.format("successful conversion: %s [%db] to %s in %dms", inputExtension, inputFile.length(), extension, conversionTime));

      return outputFile;
    } catch (Exception exception) {
      log.error(String.format("failed conversion: %s [%db] to %s; %s; input file: %s", inputExtension, inputFile.length(), extension, exception, inputFile.getName()));
      exception.printStackTrace();
      throw new IOException("Converting failed");
    } finally {
      outputFile.deleteOnExit();
      inputFile.deleteOnExit();
    }
  }
  
  /**
   * Convert pdf file to pdf/a
   * You will need to install OpenOffice extension (pdf viewer) to get it working
   * @param pdf
   * @return Byte array
   * @throws IOException
   */
  public byte[] convertToPDFA(byte[] pdfByte) throws IOException, ConnectException {
    @Cleanup InputStream is = new ByteArrayInputStream(pdfByte);
    File pdf = createFile(is, PDF_EXTENSION);
    log.debug("PDF is: #0 #1", pdf.getName(), pdf.isFile());
    return convert(pdf);
  }

  /**
   * Returns the content of the file as byte[]
   */
  public byte[] obtainByteData(File file) throws IOException {
    @Cleanup ByteArrayOutputStream baStream = obtainStreamFromData(file);
    // Convert the contents of the output stream into a byte array
    byte[] byteData = baStream.toByteArray();
    return byteData;
  }

  /**
   * Copy bytes from an InputStream to an OutputStream. 
   * This method buffers the input internally, so there is no need to use a BufferedInputStream. Large streams (over 2GB) will
   * return a bytes copied value of -1 after the copy has completed since the correct number of bytes cannot be returned as an int. 
   * 
   * @param file
   * @return ByteArrayOutputStream
   * @throws IOException
   */
  private ByteArrayOutputStream obtainStreamFromData(java.io.File file) throws IOException {
    @Cleanup final InputStream inputStream = new FileInputStream(file);
    ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
    
    //For large streams use the copyLarge(InputStream, OutputStream) method. (Over 2GB)
    int byteCopied = IOUtils.copy(inputStream, outputStream);
    log.debug("Copied #0 bytes", byteCopied);
    if(byteCopied == -1) {
      throw new IOException("The file is too big (over 2GB). Does not support so large files");
    }
    return outputStream;
  }
  
  private byte[] convert(File pdf) throws IOException {
    if (pdf == null) {
      throw new IOException("The document to be converted is null");
    }

    File convertedPdfA = convert(pdf, PDF_EXTENSION);
    @Cleanup final InputStream inputStream = new BufferedInputStream(new FileInputStream(convertedPdfA));
    byte[] pdfa = IOUtils.toByteArray(inputStream);
    return pdfa;
  }

  /**
   * Creates a temp file and writes the content of InputStream to it. doesn't close input
   * 
   * @return File
   */
  private java.io.File createFile(InputStream in, String extension) throws IOException {
    java.io.File f = File.createTempFile("tmpFile", extension);
    @Cleanup BufferedOutputStream out = new BufferedOutputStream(new FileOutputStream(f));
    IOUtils.copy(in, out);
    return f;
  }
  
  /**
   * This DocumentFormat must be used when converting from document (not pdf) to pdf/a
   * For some reason "PDF/A-1" is called "SelectPdfVersion" internally; maybe they plan to add other PdfVersions later.
   */
  private DocumentFormat toFormatPDFA() {
    DocumentFormat format = new DocumentFormat("PDF/A", PDF, "application/pdf");
    Map<String, Object> properties = new HashMap<String, Object>();
    properties.put("FilterName", "writer_pdf_Export");

    Map<String, Object> filterData = new HashMap<String, Object>();
    filterData.put("SelectPdfVersion", this.PDFX1A2001);
    properties.put("FilterData", filterData);

    format.setStoreProperties(DocumentFamily.TEXT, properties);

    return format;
  }
  
  /**
   * This DocumentFormat must be used when converting from pdf to pdf/a
   * For some reason "PDF/A-1" is called "SelectPdfVersion" internally; maybe they plan to add other PdfVersions later.
   */
  private DocumentFormat toFormatPDFA_DRAW() {
    DocumentFormat format = new DocumentFormat("PDF/A", PDF, "application/pdf");
    Map<String, Object> properties = new HashMap<String, Object>();
    properties.put("FilterName", "draw_pdf_Export");

    Map<String, Object> filterData = new HashMap<String, Object>();
    filterData.put("SelectPdfVersion", this.PDFX1A2001);
    properties.put("FilterData", filterData);

    format.setStoreProperties(DocumentFamily.DRAWING, properties);

    return format;
  }

}

Remember to close the connection when your application is quit/shutdown

Kacper Wysocki: CPM: Reliable multiuser password management

Mon, 02 Aug 2010 14:29:05 +0000

Sup all,
summer is drawing to a close and vacation is definitely over, but I for one welcome the chance to think and act again. Some time ago our managed services department started complaining about various shoddy password management solutions. Truth be told we already had a good solution, CPM (”Console Password Management”) but the software had fallen into disrepair due to seldom and untidy updates from its author. A new maintainer was desired and a project to fix the software was decreed and the result fell into my lap so to speak.

What sets CPM apart from other password management solutions is that it supports multiple users and goes to great lengths to keep your passwords secure while at the same time being very simple in its design: CPM locks its XML-formatted hierarchical password database in non-swappable private memory (so your passwords don’t get written in cleartext to disk while swapping), and encrypts the database with an arbitrary amount of GnuPG public keys.

All this makes CPM quite nice for storing and sharing secrets in a nice curses-based searchable console interface.

For the longest time I’ve been keeping the hundred-odd passwords I can’t remember on notepads and in random text files, thinking that surely I should start employing some sort of password management before I go crazy or my passwords leak. The congruence of my wishes with the scope of this project, so I picked up CPM and gave it a little love, and the result can be found at

GitHub CPM with CPM packages for debian in the downloads section.

CPM crash course

Requirements: Gnu Privacy Guard, and a GPG keypair.

First, install CPM:

dpkg -i cpm_0.25~beta-2debian2_amd64.deb

Then, create a password database, adding your key to the recipient list when prompted.

create-cpmdb

Then, use CPM from the console:

cpm

CPM should now ask you for you GPG key password and display an empty database.

CPM is controlled with the arrow keys, Enter and some control keys.
Hitting Control-H will bring you to the Help screen which explains the control keys.

By default CPM organises your passwords in a structure of hosts that have several services which may have one or more users. Hosts, services, users and passwords are nodes in the tree and a node is added by hitting Control-A and given an appropriate name.

For instance, if I were to add a password ch1ckens0up to user lolarun on the wiki service of host fragglepop.info, I would create the following node structure:

  host:fragglepop.info
      \-->service:wiki
              \-->user:lolarun
                      \-->password:ch1ckens0up

Of course there is no need to follow this anal layout, and you may even change the node structure by editing the template names in CPM by hitting Control-N or modifying the /etc/cpmrc config file.

To have CPM generate a random password for you, hit Control-P.
Your changes are not saved unless you hit Control-W or quit the program by hitting ESC enough times. Quitting through Control-C will not save the database.

Future work includes pushing the package into Debian.

What you don’t get (yet) is a GTK-based GUI, or a wrapper to pull the password database out of GIT and commit it again after modification nor integration with gpg-agent, probably (?) due to a bug in gpgme.

Enjoy this lovely piece of software and leave a comment after testing it!

Kristian Lyngstol: Varnish 2.1.3 Use it!

Fri, 30 Jul 2010 12:55:15 +0000

We just released Varnish 2.1.3. And it’s good.

I am a person who value stability, predictability and stability over pretty much everything else. I will gladly use 3-year old software over the latest version if it’s working well. I only upgrade my desktop if I see a really good reason to do it. I rather wait 2 years for a new package to enter my favorite distribution than grab it from a source package to get a new feature.

Why do I tell you this? Because I am now ready to truly whole-heartedly recommend Varnish 2.1.3. Varnish 2.0.6 was a great release. It was stable, it worked well, we know how to get the most of it. There were no un-knowns. When we released Varnish 2.1.0, we knew that it was going to take a release or two to get the 2.1 releases equally good. I finally believe we are there, and that using Varnish 2.1.3 is (almost) as safe as 2.0.6. This is the version I will be recommending to our customers.

Varnish 2.0.6 compared to 2.1

Varnish 2.1 represents two years of development. Roughly. The performance of Varnish 2.1.3 is roughly the same as that of Varnish 2.0.6, with a few exceptions.

First, we now use the “critbit” hashing algorithm instead of “classic” as default. This switch revealed a few weaknesses in the implementation that we gradually resolved between Varnish 2.1.0 and 2.1.3. The benefit of critbit is that it requires far less locking to deliver content. It scales better with large data sets and is generally a nice thing to have.

We have also re-factored much of the code that relates to directors, which allowed us to add multiple new directors. Including directors to pick a backend based on source-ip, URL hash etc.

With Varnish 2.1.3 we also added a “log” command to VCL, which allows you to add generic log messages to the SHM-log through VCL, a much-requested feature.

We have also added basic support for Range-headers. This is not the smartest version of it around, but it fits into the KISS-approach of Varnish. When Range-support is enabled, Varnish will fetch the entire object upon a Range request, but deliver only the range that the client requested. This allows Varnish to cache the entire object, but deliver it in smaller segments.

An other important change between Varnish 2.0 and 2.1 is the removal of the object workspace. The most immediate effect of this is that you will have to write “beresp” instead of “obj” in the vcl_fetch part of your VCL. The bigger consequence is that you no longer have a obj_workspace parameter and all work previously done in obj_workspace is now done in sess_workspace, then Varnish allocates exactly as much space as it needs for the object once it’s finished. This should save you some memory on large data sets.

Now, there are several other changes, but most of them are internal. This is partly to make way for persistent storage, and also for general house-keeping. An other important reason why the perceived difference between Varnish 2.0.6 and 2.1 is not that big is that many of the features that were written for Varnish 2.1 were ported to Varnish 2.0. This includes new purging mechanisms, saint mode, resets in vcl_error and numerous bug fixes.

What’s next?

We are still working on persistent storage. It is available in Varnish 2.1 as an experimental feature, but it is missing certain key aspects – like LRU support. You can compare this to how critbit evolved: Critbit was available in Varnish 2.0, but not stable. We used the 2.0 release to fine tune critbit, and we will use 2.1 to improve persistent storage.

For Varnish 2.1.4, I will be merging the DNS director, which has been ready for some time now. I wanted to investigate some reports of memory leaks before I merged it, and those seem to be debunked now.

I will also be merging my return(refresh) code, which is fairly simple stuff. All it does is “guarantee” a cache miss, even if there is valid content in the cache. The use-case for this is when you update content and want to control who does the initial waiting. The typical example is when your front page updates, you send a script to it with a magic header (X-Refresh: Yes, for instance), then you look for that in VCL and make sure the client is coming from an allowed IP (if (client.ip ~ purgers), for example) and issue return(refresh), which will (oddly enough) refresh the content. Your clients wont have to wait and the front-page is updated immediately.

In the longer run, we are also looking at proper support for gzip. For the uninitiated, it should be emphasized that for normal operation, Varnish doesn’t need to support gzip. Normally, Varnish will simply forward the Accept-Encoding header to the web server, which will compress the content as it sees fit and return it with a Vary header. That way, Varnish can deliver compressed content without having to compress it itself. This works fine, until you introduce Edge Side Includes (ESI) into the mix. With ESI, Varnish has to parse the content returned to check for ESI commands, and it can’t do that if the content is compressed. So today, Varnish only supports uncompressed ESI. We wish to solve that. Properly.

I am sure I have forgotten some key elements, but this should hopefully be enough to make this a worth-while read.

Kristian Lyngstol: Smart bans with Varnish

Wed, 28 Jul 2010 10:47:51 +0000

Banning – or what is commonly referred to as purging – means adding an expression to the ban list. I wrote an introduction to purging a while back: http://kristianlyng.wordpress.com/2010/02/02/varnish-purges/, and today I’ll introduce you to the ban lurker.

First, let me apologize for the name confusion. We’ve realized that calling it “purging” seems to indicate that the objects are removed from the cache. As they are not, it’s more reasonable to call it banning – and that’s what Varnish calls it internally. As far as VCL and CLI goes, it’s still called purging.

A quick reminder on how Varnish bans objects

Varnish bans objects by adding an expression to a list, and then when an object is “hit”, it is tested against all expressions on the list since the last time it was hit. So only when an object is hit can it be evaluated.

You can ban on both req.* and obj.* items. When you ban in VCL, there are THREE contexts involved:

1. The VCL context that puts the ban on the ban-list.

2. The context of the next request that hits the object

3. The context of the object

This means that a purge for:

purge ("req.url == " req.url " && obj.http.magic == " req.http.purge-magic)

all three purges are used. The “req.url” outside the quotations marks (the second one) is in the context of the VCL that triggered this purge() function. The “req.url” inside the quotation marks is in the context of the request that will hit the object next – whatever that may be. The “obj.http.magic” variable will be in the context of the actual object on the cache, while the last “req.http.purge-magic” is in the current vcl-context.

If that was confusing, it’s ok. If it wasn’t confusing, I’m impressed.

Enter the ban-lurker

So the problem with Varnish bans is that not all objects are hit constantly. In fact, some objects are rarely ever hit. And they will linger even if they are banned. To solve this, the ban-lurker was created. Put simply, the ban-lurker will check older objects against the current ban-list to see if they are worth keeping around. This serves two functions: 1. Banned objects can be discarded. 2. The size ban-list can be reduced.

Now, the problem with the ban-lurker is that it doesn’t stem from a request. In other words, it doesn’t HAVE a “req.” structure. It doesn’t have a URL. So if your bans are for (“req.url == ” req.url), the ban lurker can’t help you, because it doesn’t have a req.url to test against. It can ONLY test against obj.

As most bans are for req.url, this means that the ban-lurker isn’t very useful out-of-the-box. Unless you adapt your approach to bans a bit.

Writing ban-lurker-friendly bans

To utilize the ban-lurker, there are two things you need to do:

1. Turn it on with “param.set ban_lurker_sleep 0.1″ (for example. Any non-zero value enables it)

2. Do not use req.* in your bans.

Now, how to avoid using req? Well, the simples method is probably to store req.url on the object, and ban on that. Example:

sub vcl_fetch {
    set obj.http.x-url = req.url;
}
sub vcl_deliver {
    unset resp.http.x-url; # Optional
}
sub vcl_recv {
    if (req.request == "PURGE") {
          if (client.ip !~ purgers) {
              error 401 "Not allowed";
           }
          purge("obj.http.x-url ~ " req.url); # Assumes req.url is a regex. This might be a bit too simple
    }
}

The above code-snippet simply stores the URL on the object as a HTTP header called x-url, then uses that instead of req.url to ban. It also removes the x-url header before it’s returned to the clients. It may not look as intuitive (… because “req.url == ” req.url is clearly intuitive), but it allows you to take advantage of the ban-lurker.

Conclusion

We (Varnish Software and the general Varnish community) don’t know how useful the ban-lurker will prove to be, nor do we know what the best grace period for the ban_lurker_sleep parameter is. This is all fairly new, and not used much. We’re confident in it’s stability, but not the benefit.

However, it’s clear that if you have a possibly large data-set and you do frequent purges, using the ban-lurker can help keep the size of the ban-list within manageable limits, as well as help you throw banned content out faster. We can’t really see the downside of it.

However, using the ban-lurker without adjusting your approach to actual bans is pointless. It wont be able to do anything except wake up every second and discover it can’t do anything.

I hope this helped. Happing banning!

Roger Kind Kristiansen: Android: Map tiles not loading – and possible fixes

Sun, 25 Jul 2010 10:12:25 +0000

I have encountered two reasons for the Google Maps tiles not loading. One is rather common, and the other one only seem to affect some phones:

Missing/wrong Google Maps API key
Permissions placed after application tag in the AndroidManifest.xml

Missing/wrong Google Maps API key

Usage of the Google Maps API key is properly documented many places, like in tutorials, the official Android developer site and the Google API project pages.

Wrong placement of permissions in the AndroidManifest.xml

The part about wrong placement of the permissions in the manifest file bit me when trying to get my map based app working on the Sony Ericsson Xperia X10 mini pro. The app was working like a charm in every configuration I tried, except on this little beauty. The configurations I tried included:

HTC Desire running 2.1
T-Mobile G1 running 1.6
Various emulators running 1.5, 1.6, 2.1 and 2.2 with various screen sizes. Also one that matched the x10 mini pro screen size.
Of course the X10 mini pro running 1.6

I had defined my permissions below the application definition, and after a while I moved the tags above the tag, and this fixed it.

So this is not OK on the X10 mini pro (AndroidManifest.xml):

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
      package="com.test.myapp" android:versionCode="1" android:versionName="1.0">
    <application android:label="@string/app_name" android:icon="@drawable/launcher_icon">
    	<uses-library android:name="com.google.android.maps" />
        <activity android:label="@string/app_name"
                  android:alwaysRetainTaskState="true" android:name=".views.Map">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />
                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
        </activity>
    </application>

   <uses-permission android:name="android.permission.INTERNET"></uses>
</manifest>

While this works:

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
      package="com.test.myapp" android:versionCode="1" android:versionName="1.0">

    <uses-permission android:name="android.permission.INTERNET"></uses>

    <application android:label="@string/app_name" android:icon="@drawable/launcher_icon">
    	<uses-library android:name="com.google.android.maps" />
        <activity android:label="@string/app_name"
                  android:alwaysRetainTaskState="true" android:name=".views.Map">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />
                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
        </activity>
    </application>
</manifest>

If someone can tell me why defining permissions below the application tag works on most set-ups, but not on the X10 mini, please enlighten me.

Magnus Hagander: PGDay.EU announced and call for papers

Thu, 22 Jul 2010 17:57:00 +0000

PGDay.EU 2010 has finally been announced. It will be in Stuttgart, Germany, on December 6th to 8th. More details available on the conference website.

We have also sent out our call for papers. If you have done something interesting with PostgreSQL, please go ahead and submit a talk! We are currently looking for talks in both English and German!

Kristian Lyngstol: A Varnish Crash Course for aspiring sysadmins

Thu, 15 Jul 2010 12:11:03 +0000

Ok, so you’ve just started working in a Big Sysadmin Department, and they use Varnish. You may have to deal with it, but are not going to be working on it on a daily basis. This is some of what you should know about varnish in case you have to extinguish some fires.

This post is heavily inspired by my visit to Paris last week, and as such, I’d like to say hello to Ludo and everyone else (this is mostly written on my plane ride back home).

1. What Varnish is, and how it fits into the web

Varnish is a reverse proxy. It operates on layer 7, unlike your typical load balancer (though some of them have layer 7 functionality, like basic health checks, they can hardly compare to Varnish). For all intents and purposes, Varnish is the first “web server” that a browser hits.

Varnish has a simple, yet powerful configuration language – the Varnish Configuration Language (VCL) – which allows it to make intelligent decisions on what to cache, and how. It can rewrite urls, it can perform redirects and it can direct traffic at different web servers based on both the specific request it is dealing with (for example, if the request is for “http://www.example.com/sports”, it might go to a dedicated sport-server) and on the state of the web server.

By default, Varnish should Just Work. It will only cache content it is certain can be safely cached. That means that it will only cache GET requests, and only content where no cookies are involved. This behavior can – and most likely will – be overridden by your VCL.

In regards to the RFC2616 (The HTTP RFC), Varnish is not really what they are talking about when they refer to proxies and caches. Varnish belongs on the origin-server side of the equation, as the same people who control the web server can also control the cache. But not entirely, so you we have done some approximations using common sense to make Varnish fit into the RFC.

2. Configuration, stopping and starting.

Varnish has two separate types of configuration types. VCL defines policy, and only policy. It can be used to implement features that are otherwise not present (for example the ability to purge content simply by adding “?purge=yes” to the url, or any other scheme like that).

The second type of configuration is program arguments – or parameters. Parameters relate to how Varnish behaves on the machine it is running it. You can define the size of the cache, the user and group to run Varnish under, what ports are used by Varnish, where varnish will find its VCL, and so forth.

On Debian-based systems (including Ubuntu), you will typically find the parameters defined in /etc/default/varnish. Similarly, you will find them in /etc/sysconfig/varnish on Red Hat.

The VCL is often stored in /etc/varnish/, and typically /etc/varnish/default.vcl – but that is defined by a startup argument.

Additionally, you can change MOST of the parameters while Varnish is running. There is only requirement for this: Varnish needs to be started with a -T argument, to enable the management interface.

The best way to use the management interface is by telnetting to it. This gives you a nice and simple interface. With a “help” command. You can run the same commands with the “varnishadm” tool, but that will not give you as extensive error messages. The rule of thumb that I use is “telnet for interactive commands, varnishadm for scripting purposes”.

Parameters are changed by using “param.set” after reviewing them with “param.show”. Changes are applied immediately, but it might take some time for them to be visible. A good example of this delay is if you change the default ttl, as that applies to new objects in the cache from that point on.

You can also load and use new VCL while Varnish is running. This does not introduce any (known) glitches or slowdowns or delays in Varnish, as it is a matter of switching 4 pointers around – and varnish will still keep the old configurations around while they are still used (for instance if a backend is still in use).

Loading VCL is done with the “vcl.load ” command. That will only _load_ the configuration, not actually use it. If your VCL has syntax errors, this is when they will show up. After it has loaded, you can switch to the new config with “vcl.use “. Alternatively use my varnish_reload script. Be very careful with “reload” arguments to init scripts, as they may be implemented using a restart.

Starting Varnish is left as an exercise to the reader.

3. Quickly getting an overview of Varnish

varnishstat.

Run it, look at it, and understand it. This previous blog post gives a quick introduction to varnishstat: http://kristianlyng.wordpress.com/2009/12/08/varnishstat-for-dummies/.

One thing you should always check is the uptime of varnish – in varnishstat. Because Varnish will “never” crash permanently. It can – however – crash repeatedly. This is because Varnish has two processes: A management process and a “everything else” process. If you have a bug, it is likely to take down the “everything else” process, and the management process will notice this and restart it immediately. This is logged to syslog in detail.

On Debian systems (including Ubuntu), /var/log/syslog is more extensive than plain /var/log/messages.

You also want to look out for disk activity. This is the number one killer of Varnish-performance. If you are seeing extensive disk activity, it might make sense to reduce the size of the cache so it fits in memory. And tune the caching scheme to make sure there are no duplicate objects (for instance a www.example.com/foo object and a separate example.com/foo object). Or buy more memory. Or make sure the disk is an SSD. (Hint: vmstat 5).

One trick that is commonly applied to avoid disk activity is to put the shmlog on a tmpfs. This is generally not required, but doesn’t hurt. The shmlog and the compiled VCL is typically stored at /usr/var/varnish/(hostname)/ or similar.

Do not let the virtual memory usage of Varnish scare you. But do be afraid of the resident memory usage. It is not uncommon to see Varnish use 80G of virtual memory but only 28G resident on a machine with 32G.

Lastly you have varnishlog. This will tell you exactly what varnish is doing with each request, but is extremely extensive. Keep in mind that varnishlog can filter. Here are some freebies, and I’ll leave it to the reader to expand upon them (or a future blog post):

varnishlog -o RxUrl /some/url
List requests with “/some/url” in the url. Including /some/url, /some/url/blah and /blah/some/url/bla.

varnishlog -b -o
Only list backend traffic.

varnishlog -b -o -i TxURL
List URLS going to a backend.

varnishlog -c -o RxHeader FireFox
List requests from clients with “FireFox” present in any of the HTTP headers supplied.

varnishlog -o TxStatus 500
List all requests sent back to a client with status code 500.

4. Clear part of the cache

Let’s say your web server misbehaved and your VCL wasn’t smart enough to spot it, and you’ve cached malformed data. The easiest way to clean that up is using the “purge” function through the management interface. It will not free up memory, but it will make sure the content is refreshed the next time someone asks for it.
This supports regular expressions similar to what you find in VCL. I wrote a detailed post about purging: http://kristianlyng.wordpress.com/2010/02/02/varnish-purges/

Some highlights:

purge req.url == /some/specific/url && req.http.host == “www.example.com”
purge req.url ~ “^/some/generic/prefix” && req.http.host ~ “^(www\.)?example\.(com|org|net)”
purge obj.http.cache-control ~ “max-age=2222222222222″

5. Final advice

Don’t use 32 bit.

Read my post on best practices: http://kristianlyng.wordpress.com/2010/01/26/varnish-best-practices/

Don’t over do it. KISS is king. Standard OS packages, as simple a VCL as you can, as little tuning of parameters as you can. Etc.

Remember: If Varnish is told not to cache based on response from a web server, it will cache the decision not to cache. This means that if your front page temporarily fails and varnish is told not to cache it, make sure the TTL is low, otherwise all traffic to the front page will go to a web server until the hitpass object has expired.

Ask for advice early. It’s much harder to fix a system that is deployed and in production than help you avoid problems. You can get commercial help from us at Varnish Software or the Varnish community.

Roger Kind Kristiansen: Flattr: Mikrobetaling (mikrodonasjon) for nettinnhold

Wed, 14 Jul 2010 23:32:45 +0000

Flattr er et ferskt system (i beta fra mars 2010) for mikrodonasjoner, som tar sikte på å gjøre det mye enklere å donere småpenger til innholdsprodusenter. Kort fortalt fungerer det slik:

Du setter inn en fast sum hver måned
I løpet av måneden klikker du på flattr-knappene til de tingene du mener har gjort seg fortjent til en økonomisk påskjønnelse
Ved månedens utløp blir din sum delt likt mellom alle tingene du har flattret i løpet av måneden

Foreløpig bærer Flattr preg av at det fortsatt befinner seg på betastadiet; brukermassen er ikke er all verden og utvalget av ting som kan flattres/blir flattret er deretter. Det bedrer seg dog fortløpende. Er dette måten å tjene penger på nettinhold i fremtiden?

Flattr er i åpen beta, men er du utålmodig (og tidlig ute med å lese denne posten), kan du forsøke en av disse invitasjonskodene:

288a1a706ac7a139b
0dd1624666273b5c3
66a27863d0a49c785

Jan Henning Thorsen: My x200 used to say "beep"

Tue, 13 Jul 2010 07:00:00 +0000

I’ve been using my Thinkpad x200 for about a year now, and I’m pretty sure it started beeping some time this winter: A short “beep” occurred each time it came back from suspend or the power cord was plugged in. It has been pretty annoying, but it has not been annoying enough for me to “fix it” until yesterday. So I started googling the issue, since it used to be the “pcspkr” module loaded by Ubuntu which resulted in the annoying beeping, but was not the case since it wasn’t even loaded in 10.4.

So I gave up on google after some time and decided to seek answer where no answer has been found before: I entered the BIOS, (I haven’t looked for this answer in the bios, it’s just that I consider the BIOS to be pretty useless) poked around, disabled some features and the beep went away.

Press Thinkvantage, and then F1 to enter the BIOS (right after the power is turned on)
Then go to “Config” -> “Beep and alarm”
Simply select DISABLE on each of the choices, if you dislike beeps as much as I do.
Press F10 to save and exit the BIOS, and you’re beep free!

I’m not going to investigate this any further, but I’m still curious about when the beeping started… Had it always been beeping or has there been some kernel updates, whick resulted in the beeping to start? I’m guessing the “kernel update” is the right answer, but I have no idea how to disable it from Linux. Is it somewhere in /sys or /proc, or…?

Roger Kind Kristiansen: HDD vs. SSD comparison

Mon, 12 Jul 2010 20:07:19 +0000

My employer was nice enough to give me an SSD for my laptop and wanted me to do a quick comparison between this and my previous drive, so here it is.

All the benchmarks have been performed on my Lenovo X61s running Ubuntu 10.4. The SSD is an Intel X25-M 80GB SSD while the HDD is an Hitatchi TravelStar 60GB 7200RPM.

The first impression after booting the laptop with a fresh install of Ubuntu 10.4 on the X25 was that everything seemed faster. Booting and loading apps was done in a snap and the entire user experience was more fluent.

Startup times

	HDD	SDD
System boot	1m 27s	20s
Gnome	44s	8.3s
Firefox		2.6s
Thunderbird 3	11.9	4.9
Eclipse	29.4	15.3

File operations

My current project at work has a rather sizable subversion repository (XXXX MB). I performed a ‘du -h’ on this repository to calculate its size, and also grepped for a word in the entire repository.

Lastly I run postmark with all settings set to defaults, except files and transactions set to 50000. Postmark tests read/writes of lots of small files.

	HDD	SSD
du -h	11m 34s	1m 56s
grep -R	13m 48s	3m 18s
postmark	Time: 729 seconds total 514 seconds of transactions (97 per second)

Files:
75138 created (103 per second)
Creation alone: 50000 files (285 per second)
Mixed with transactions: 25138 files (48 per second)
24905 read (48 per second)
25084 appended (48 per second)
75138 deleted (103 per second)
Deletion alone: 50276 files (1256 per second)
Mixed with transactions: 24862 files (48 per second)

Data:
136.54 megabytes read (191.79 kilobytes per second)
428.74 megabytes written (602.24 kilobytes per second)

“Time:
22 seconds total
16 seconds of transactions (3125 per second)

Files:
75138 created (3415 per second)
Creation alone: 50000 files (12500 per second)
Mixed with transactions: 25138 files (1571 per second)
24905 read (1556 per second)
25084 appended (1567 per second)
75138 deleted (3415 per second)
Deletion alone: 50276 files (25138 per second)
Mixed with transactions: 24862 files (1553 per second)

Data:
136.54 megabytes read (6.21 megabytes per second)
428.74 megabytes written (19.49 megabytes per second)”

Battery lifetime

Peformed by letting the laptop run unattended and not performing any particular task. Time time given is the time laptop ran before the battery ran out.

HDD	SSD
4h 47m	4h 53m

Conclusion

Even though there’s not much of an improvement to battery life, SSD rocks when it comes to overall system responsiveness and speed.

No big surprises here, move along.

Jan Henning Thorsen: Xperia X10 mini

Mon, 12 Jul 2010 07:00:00 +0000

So I’ve been using my X10 mini for ten days now, and it has been quite a bumpy ride:

Last weekend I had to charge it about every six/eight hour. I have to admit that I did use it a lot in the beginning, but i was still shocked to see how bad the battery was: I mean a mobile phone which requires a cord isn’t really mobile :S

After I got back from my weekend I decided to change back to my Nokia 5800, just to have a phone I could use over night without charging it. At the same time I also sent an email to CDON telling them that I wanted to return the phone. (yeah, I know it’s a bit strange to bue a phone from cdon…) CDON was really nice and explained to me (without question) how I could return the phone. Even though I had given up on the battery, I really thought the phone was excellent, so the next day I decided to start reading reviews and blogs to figure out if there where anything I could do to extend the battery life. I also asked a co-worker who gave me some information I found hard to believe….

Even so, after a day of reading and charging the phone, I figured out that it wasn’t all that crappy as expected and I’ve now managed to get the lifespan from eight hours to more than two days! So this is what I did:

Co-worker + blogs stated that the battery actually gets better over time! I have no idea what kind of crazy chemical reaction is involved, but it must be true: Swapping software just can’t make such an improvement by it self (?)
“Supercharged it”. Meaning I charged it to full and then simply kept on charging it. I pulled to cord out a couple of times, but plugged it in again when I saw that the battery indicator dropped instantly.
Replaced whatever SE got pre-installed with ”helix” launcher.
Installed “Advanced task killer” and killed a lot of SE background processes I didn’t need anyway. ATK also periodically kills background processes which is just idling.
Installed “BatteryTime”. Doesn’t do much for the battery time, but simply more detailed information about how much battery/time is left to use for various tasks.
I got bluetooth, GPS and wireless turned off most of the time. I know this kind of sucks, but since I hardly ever use bluetooth, nor wireless anyway it’s not so bad. I will try to use GPS more, just to see how it impacts the battery, but for now I got it turned off and use AGPS instead.

The crappy thing about the bullets above is that I did it all at once, so I cannot really tell which one worked, but at least doing them all works for me and I’m now quite happy with the phone! Of course having a smart phone, battery lasting for days, big screen, small form factor…. It’s just not possible these days for me to achieve the ultimate phone, so I need to settle for something.

Other useful/cool applications

Bump - Share data (pictures, apps, contacts++) by bumping two phones with “bump” installed
ConnectBot - SSH client. Pretty useless on x10, since of the screen size, but still a nice SSH client.
Goggles - “Use pictures to search the web”
Google Sky Map - Move your phone around to see how the universe looks from your perspective.
GPS status - Digital compass and speedometer.
Opera mini 5 beta - I would like to replace the “internal browser” with Opera, but I still don’t know how. (TODO)
Startup manager - Make sure not too many useless processes start when the phone boots.
SyncMyPix - Sync between facebook and address book.

Todo

Root the phone. Replace the rest of the crap with Android 2.x.

Sources

Ingvar Hagelund: Changing a process’ file descriptor on the fly

Sat, 10 Jul 2010 00:31:01 +0000

I come back to a specific problem every once in a while: Changing a program’s file descriptor while the same program is running.

From time to time, we do stupid things, like running a very important shell command, and adding debug output to see that it works, then realizing that it will take hours to finish, and spitting gigabytes of debug to an xterm through ssh does not help.

An other typical example could be a finding an ill managed system with some daemon without proper logfile handling. Restarting that process right now is just out of the question, copy-truncating that 16GB logfile will take too much time, and by the way, the disk is almost full.

After a hint from the ever helping kjetilho, and a Google search, I ended up with what seems to be a well-known trick, but I did not know about it till now. Welcome to the dark side of gdb.

With the power of gdb at your hand, you can hook into the inner parts of any running program, and change, well, virtually anything. Sounds insanely dangerous for systems in production, right? Yes I agree. Still, this hack is not that ugly. It does just what the doctor ordered: It changes a process’ fds for you while it’s running.

Google found this hack by Robert McKay, and I added a couple of lines for robustness. Here it is: fdswap.sh.

For a simple test, (yes, do test this before smashing your production environment to pieces) make small shell script like this

#!/bin/bash
n=0
while true; do ((n++)); echo $n; sleep 1; done

Run that script in a separate window. Pick up the pid, and find out what vty is connected to that process’ standard output

ls -l /proc/$pid/fd/0

Then run the script, be really afraid of all that scary gdb output, but watch that number pumping magically stop.

fdswap.sh /dev/pts/$vty /dev/null $pid

Have a look at the fds again

ls -l /proc/$pid/fd

You can use strace to check that the process is still happily pumping numbers, but now to /dev/null

strace -p $pid -e write

Magnus Hagander: Robotic moderation duties

Thu, 08 Jul 2010 20:06:00 +0000

Every now and then, the discussion about why it takes too long for messages to get approved when posted to some of the PostgreSQL mailinglists comes up, and it goes around a couple of laps. Maybe we get a new moderator. But eventually it comes back down to not enough people. Personally, I've usually just found myself not having the time to keep up with moderation. So when I needed a project to learn some Android development on, I figured this could be an interesting (probably) and useful (maybe) topic.

So, meet Mailinglist Moderator. A tiny android application that helps with the daily moderation chores for anybody moderating Mailman or Majordomo2 mailinglists (should be easy enough to add more list managers if there is any that people actually use).

The application will simply enumerate all unmoderated items and let you set them to accept or reject either one by one or in batch. Personally, I've found it makes it a *lot* more likely I will do moderation - it's literally down to 30 seconds while waiting for a bus or train, or something like that. And once the hurdle is gone, it's a lot more likely I'll end up actually moderating. I found it useful - hopefully others will as well.

Here are a couple of screenshots showing what the application looks like.

The app is available as an APK for download on my github page. It hasn't been published to the market now, but I'll do that if enough people find it useful and ask me for it...

And of course, this is all BSD licensed open source, so any contributions are welcome!

Magnus Hagander: PostgreSQL Europe Marchandise Store

Sat, 03 Jul 2010 17:02:00 +0000

We've finally opened the merchandise store for PostgreSQL Europe. It's a chance for everybody who haven't had the chance to attend one of the many PostgreSQL events where we've been selling mugs and shirts for a long time, as well as a chance to get some stuff that we haven't previously had available.

There's close to zero earnings for PostgreSQL Europe off these purchases - we're trying to make it as cheap as we can for everybody. You are of course most welcome to donate some extra to the project, should you wish.

Planet Redpill Linpro

Nicolai Langfeldt: It's just too easy

Kristian Lyngstol: Wiki.sh mediawiki for geeks

Presenting wiki.sh

Demo

Cute tricks

Jorge Enrique Barrera: Recommended Firefox addons

Lars Strand: Bubba Two - some performance numbers

Lars Strand: The Bubba Two NAS

Jean-Marc Reymond: How to update Smooks in JBoss ESB

Ingvar Hagelund: RPMS of varnish-2.1.3

Espen Braastad: munincollector-ng

David Jensen: Chroot'd Ubuntu on Samsung Galaxy S (Android)

A couple of prerequisites:

Partition

Create a rootfs with rootstock

Now on your phone

Log in via SSH

A script to help

Kristian Lyngstol: Stripping cookies with VCL

Jorge Enrique Barrera: Introduction to irssi

Getting started

Customizing the settings

Connecting to an IRC-network

Navigation

Adding networks, servers and channels

Logging

Scripts

Minor adjustments

Kacper Wysocki: tune2fs and green disks

Edward Bjarte Fjellskål: snort-2.8.6.1 debian/ubuntu packages

Sigvard Lyth: A vacation in Bornholm

Christopher Grenness: Solving Android Market Download Problems

Kristian Lyngstol: Varnish backend selection through DNS

Enter the DNS director

Availability

Shervin Asgari: Migrating from JODConverter 2 to JODConverter 3

Kacper Wysocki: CPM: Reliable multiuser password management

Kristian Lyngstol: Varnish 2.1.3 Use it!

Varnish 2.0.6 compared to 2.1

What’s next?

Kristian Lyngstol: Smart bans with Varnish

A quick reminder on how Varnish bans objects

Enter the ban-lurker

Writing ban-lurker-friendly bans

Conclusion

Roger Kind Kristiansen: Android: Map tiles not loading – and possible fixes

Missing/wrong Google Maps API key

Wrong placement of permissions in the AndroidManifest.xml

Magnus Hagander: PGDay.EU announced and call for papers

Kristian Lyngstol: A Varnish Crash Course for aspiring sysadmins

1. What Varnish is, and how it fits into the web

2. Configuration, stopping and starting.

3. Quickly getting an overview of Varnish

4. Clear part of the cache

5. Final advice

Roger Kind Kristiansen: Flattr: Mikrobetaling (mikrodonasjon) for nettinnhold

Jan Henning Thorsen: My x200 used to say "beep"

Roger Kind Kristiansen: HDD vs. SSD comparison

Startup times

File operations

Battery lifetime

Conclusion

Jan Henning Thorsen: Xperia X10 mini

Other useful/cool applications

Todo

Sources

Ingvar Hagelund: Changing a process’ file descriptor on the fly

Magnus Hagander: Robotic moderation duties

Magnus Hagander: PostgreSQL Europe Marchandise Store