IRSYNC & Limiting Passwordless SSH Keys

Anyone who has ever used SSH key-pairs to access more than a couple of servers (or hundreds in my case), will tell you they are an invaluable convenience. It is a natural progression and very common usage that SSH key-pairs are coupled with other common tasks or tools, where having a pass phrase attached to the key would be counter-intuitive to the task automation. So, what do we do despite our better judgment? We create key-pairs with absolutely no pass phrase. The implications are abundantly obvious, if the private key ever gets lost or stolen, any accounts that have the key-pair associated to it can be instantly compromised.

In the case of my recently released project Incremental Rsync (IRSYNC), one of the implementation hurdles at work was to have servers backup using a secure medium. This is easily handled with rsync’s -e option to have data transferred over ssh using a key-pair but then the obvious issue comes up that what if a client server ever gets compromised? Then the backup account on the backup server can be compromised (please don’t use root!@#!@#) allowing for backups to be deleted or worse yet data to be stolen for every server that backups to said server/account.

A solution to this is to limit the commands that can be executed over SSH by a specific public key, though this is not a perfect way to mitigate the threat it does go a long way to help. For my backup server implementation I have setup the user ‘irsync’ on the backup server, this account has the usual ‘~irsync/.ssh/authorized_keys’ file where I place the public key. Where things differ is that you prefix a script path in front of the public key that is used to interpret commands sent over ssh, which looks something like this:

command="/data/irsync/validate-ssh.sh" ssh-dss AAAAB3NzaC1kc3MAAAC......87JVNLJ5nhaK1A== irsync@irsync

The ‘validate-ssh.sh’ script is basically a simple interpreter, it looks at the commands being passed over ssh and either allows them or denies them with some logging thrown in for auditing purposes. The script can be downloaded from: http://www.rfxn.com/downloads/validate-ssh.sh. Please take note to edit the scripts ‘log_file=’ value to an appropriate path, usually the base backup path or user homedir.

An example of validate-ssh.sh in play would be as follows, first the client side view then the logs from $log_file:

root@praxis [~]# ssh -i /usr/local/irsync/ssh/id_dsa irsync@buserver3 "rm -rf /some/path"
sshval(13156): ssh command rejected from 192.168.3.33: rm -rf /some/path

root@praxis [~]# ssh -i /usr/local/irsync/ssh/id_dsa irsync@buserver3
sshval(13403): interactive shell rejected from 192.168.3.33

May 04 11:36:15 buserver3 sshval(13156): ssh command rejected from 192.168.3.33: rm -rf /some/path
May 04 11:40:03 buserver3 sshval(13403): interactive shell rejected from 192.168.3.33

On the flip side when a command is authorized, it gets recorded into the $log_file as follows:

May 04 05:29:08 buserver3 sshval(29993): ssh command accepted from 10.10.6.6: rsync --server -lHogDtprx --timeout=600 --delete-excluded --ignore-errors --numeric-ids . /data/irsync/mysql02.mynetwork.com.full

Take note that if you do choose to use validate-ssh.sh with irsync, you will need to create your own script to manage the snapshots as internally irsync uses the find command, piping results to xargs and rm which will not be authorized by validate-ssh.sh (for good reason!). This is actually a very simple task, although all your snapshots will have to use the same rotation age (whatever).

#!/bin/sh
age=14
bkpath=/data/irsync

for i in `ls $bkpath | grep snaps`; do
wd=$bkpath/$i
find $wd -maxdepth 1 -mtime +14 -type d | xargs rm -rf
done

You can save this to /root/irsync_rotate.sh, chmod 750 it and run it as a daily cronjob by linking it into /etc/cron.daily/ (ln -s /root/irsync_rotate.sh /etc/cron.daily/) or you can add an entry into /etc/crontab as follows:

02 4 * * * root /root/irsync_rotate.sh >> /dev/null 2>&1

Although I detailed the use of validate-ssh.sh in the context of backups with irsync, this could easily be adapted to any usage when you want to restrict the commands executed over ssh with key pairs. You could even create your own script in perl or whatever floats your boat and use that instead — if you happen to go that route, please share with me what you created in the comments or by e-mail to ryan <at> rfxn.com.

(ATF) Aggregate Threat Feed

aggregate feed sexyness
ATF sexyness

For my first post back into things in awhile (a long while), I thought I would introduce everyone to the sexyness that i’ve called the Aggregate Threat Feed or ATF for short. This feed is derived from threat data at work, namely our network edge IPS (a custom snort implementation, another post on that later) and aggregated firewall data from 250+ servers, mostly being brute force/invasive scan attack addresses.

There really is nothing terribly fancy about this, the data is presented in a drop list format that is updated every 15 minutes with an optional variable to adjust the amount of addresses returned:

http://asonoc.com/api/atf.php?top=50 (defaults at 100)

The entries in the list are sorted on the database side by highest event count first, you can optionally view the source and event count entries in the list but this is considered strictly for review purposes (it wouldn’t be of much other use). Take note that the maximum value for ‘top’ is 300.

http://asonoc.com/api/atf.php?top=300&fmt=list

The review data looks something like follows:

IP | SOURCE | EVENTS
———————-
187.45.224.5 ips 227
92.48.206.91 ips 202
69.13.196.47 fw 176
210.17.251.159 ips 130
83.170.110.194 fw 125

This is pretty basic to understand, the distinction to note however is that event numbers for IPS source data can be 50 events against 1 or 20 servers whereas the event count for fw sourced data typically reflect unique servers. So an address sourced from fw data with 200 events, actually hit 200 servers.

The next release of APF due in the coming months, will feature many changes and among them will be the inclusion of ATF as part of the new feed subscription feature. Further, users will have the option to enable reporting to the rfxn.com server that allows your own block data to be included in the ATF database. As more installations opt-in on this feature it will allow data aggregation to reflect a more global threat landscape that truly represents the users of APF (currently active installations based on those fetching the rfxn.com reserved.networks list daily: 46,921).

Also on the agenda is a simple ATF landing page that presents statistical data and some fancy graphs/charts (probably use google api cause im lazy like that), that will allow users to better visualize threats included in the feed and details on the actual events that caused an address to end up in it (snort events, firewall triggers etc..).

EOF