When I first introduced you all to the Aggregate Threat Feed back in May, it was a much smaller feed with very simple ambitions — pulling together threat data at work from our network edge and host based firewalls and aggregating the data into a usable feed. The actual intention being that as an attacker exposes themselves more on the network through invasive scans and attacks, they would quickly climb up the threat feed and end up banned proactively. Though this did and still does happen in a way, a problem was introduced when more and more data started to come in from the network edge and it quickly outweighed data from the hosts.
The old way the threat feed was sorted was by number of events. For the network edge IPS the events correlated with actual signature events on the network edge, so these could number from 50 events for an SNMP community scan to thousands of events for an SSH scan. Then you have the host based firewall events (mostly brute force attacks), these events are correlated into the feed by the occurrence of an attackers address across unique servers, so if 188.8.131.52 made a brute force attempt against 11 servers it would show in the feed as 11 events.
The problem that developed here is that the network edge IPS is far more noisy on an exposure level than the host based firewalls, so you would end up with hundreds of IP’s from the network edge with thousands of events each, while the host based firewalls, even though they represent hundreds of attacking IP’s also, the actual event counts relative to unique servers those IP’s attacked, was FAR lower. This meant that often the top 50 or 100 items in the threat feed were all IPS events, though quite valid events the actual host based events had more of a threat significance than some of the IPS events. The host events were simply being washed out of the top 100 on the list from the sheer volume of IPS events (who really wants to import 300 addresses from a threat feed? let alone even 100).
So, what I decided on doing was adding a weighted field into the database that is based on unique targets for each attacking IP. This weighted field is the new sort method for the feed and it works something like this. If the IPS picks up an attacker hammering five servers with an SQL injection exploit, that attacking IP ends up in the threat feed with a weight of 5, if we then have an attacker that runs brute force attacks on 30 servers, that attacking IP ends up in the threat feed with a weight of 30. The end result is that the threat feed gets better populated with the highest weighted attackers at the top, so those attackers who are more aggressive across unique targets, quickly end up at the top of the list. This allows the feed to better protect the devices/hosts it is being used on from a developing attack before the attacker reaches that device/host on the network.
List Format (fields: IP | SERVICE | EVENTS | WEIGHT):
Since I will be busy this coming week with other priorities, I am posting an early month in review blog on signature updates.
In the last 3 weeks we have not seen a whole lot of action on in-the-wild malware, most of what is propagating at the moment are variants of already detected content. That is however not to say there has not been new signatures extracted, allot of this months signatures have come from account level compromises on vulnerable e107, wordpress and joomla installations along with user submissions. There is not a whole lot of ground breaking malware threats, it is more of the usual such as mass mailers, perl/php command shells, irc bots and php socket flooding tools.
In total, the 3 weeks ending Sat July 24th, there has been 128 new signatures in 54 classifications with 65 signatures being added in the last 7 days. This brings us to a total of 2,588 (1002 MD5 / 1586 HEX) signatures, an increase of 117 signatures over the last blog post on signature updates. For those paying attention, there is a discrepancy of -11 signatures between the 128 new signatures and the +117 change since the last update, this is because there has also been 11 signatures removed for poor performance/false positives.
As always new signatures are automatically updated daily or can be manually updated with the -u|–update command line options. The 128 new signatures fall into the following classification groups:
Today I found the time and energy, despite how tedious it was, to go over the last two weeks worth of malware submissions and missed edge IPS data from when I was away. This resulted in a total of 126 new signatures (67 MD5 / 59 HEX) which brings LMD to a total of 2,471 signatures (894 MD5 / 1577 HEX). This now also gives the project a unique distinction among anti-virus and malware detection offerings, as the single largest project, commercial or open source, detecting Linux malware.
To further illustrate the lapse in coverage by other vendors, we can turn to CYMRU analysis of the MD5 hashes in LMD, as discussed on the LMD home page, CRYMRU provides malware data to vendors such as trendmicro, symantec, kaspersky, microsoft, google and more.
KNOWN MALWARE: 301
% AV DETECT (AVG): 57
% AV DETECT (LOW): 58
% AV DETECT (HIGH): 71
UNKNOWN MALWARE: 593
This in short shows that of all the vendors that CYMRU provides data for, only 301 of LMD’s 894 MD5 signatures are detected by competing solutions and of those threats detected, on average, only 57% of vendors detect each threat. This information really has no other significance than to reinforce the validity of this project and the time I am investing into it, chalk one up for stroking own ego!
New signatures in this update are classified into the following groups, you will notice ALLOT of command shells in this update, including an interesting addition, a JSP command shell!
I am back, fresh off a trip home to Montreal, which I must say was an absolutely amazing time. It has left me reflecting on a lot of things, most importantly that there really is no place like home — I miss Montreal more than I can even describe. That said though, time to get back into the mix of things — there is a mountain of malware submissions to review, 91 to be exact. Today I really could not find the energy or time to go through them all but I did process the edge IPS data to extract some in the wild signature data which generated 8 new signatures that are now live. In the coming days, I will work through the malware submissions and get those signatures out as soon as possible.
Yup, nothing to see here except numbers…
2,018: Downloads of the newest project, Linux Malware Detect, month to date.
2,294: Signatures for Linux Malware Detect.
6,207: Downloads for all projects for the month to date.
14,176: Google results with link backs to rfxn.com or related domains (i.e: r-fx.org, rfxn.org etc..).
30,061: Active APF installations relative to unique IP’s fetching the reserved.networks file daily.
70,826: Project downloads for the last 12 months, May 2009 – April 2010.
133,931: Total visitor session to rfxn.com, month to date.
258,154: The number of web sites protected by APF (passed unique install IP’s to domainsbyip.com).
1,231,604: Total hits to rfxn.com, month to date.