R-fx Networks

Tag: projects

BFD 1.5-1 Update: Forged Syslog Data Vulnerability

by on Jan.29, 2014, under Development, My Blog

Current Release:

An updated version of BFD 1.5 has been released, version 1.5-1, which addresses an address scoping issue in the event forged syslog data is encountered on the host system running BFD from a malicious local user or any other sources that may generate forged syslog data. In such situations, BFD can be manipulated to ban addresses that it would otherwise not validly be triggered to do so, with wide scoped CIDR notation at up to a /8.

The 1.5-1 release addresses this by ensuring that addresses BFD passes onto the BAN_COMMAND are fully qualified C class (/32 CIDR) addresses only, as opposed to any CIDR notation address.

Thanks goes to rack911.com for responsibly advising of this issue and awaiting the release of a fix prior to any public disclosure. The responsible disclosure practices of rack911.com are a statement to their professionalism as a managed services provider as well as their dedication to improving the security landscape of the web hosting industry at large.

Leave a Comment :, , , more...

LMD 1.4.1: Delivering on your requests

by on Nov.20, 2011, under Development, My Blog

The release of LMD 1.4.1 is now live and with it comes a few new features. In this small update, I have tried to deliver on on a couple of common feature requests from users which were in-line with my development goals. That said, right to it…

The biggest change has come in the form of what has been dubbed public mode scanning. This is where non-root users can execute malware scans. For this to work, a new quarantine, session and temporary path directory tree needed to be created that users had write access under. This presented some challenges and in the early incarnation of this feature, the pub/ directory tree created for this feature was set world writable. The more I worked with the ideas around this feature the more I hated it, I simply could not impose upon users a world writable path. Then I flirted with the idea of simply creating the directory tree and if users wanted the feature they had to set it mode 777 themselves, though this was a fair trade it still felt like a lazy solution.

In the end, the solution I came up with was to populate the new pub directory tree with user paths based on passwd users and explicitly set ownership to each user for their pub/username path (–mkpubpaths). This meant that something was needed to regularly update the pub directory tree for new users and as such a cronjob was added that runs every 10 minutes to create said paths (cron.d/maldet_pub). This feature is controlled by conf.maldet variable public_scan which is disabled by default and when in a disabled state the cronjob simply does nothing along with user initiated scans exiting with an error that the feature is not currently enabled.

Supplementing the public mode scanning feature is the support for mod_security2 upload scanning which next to user initiated scans was one of the most requested features recently. Although the inotify real-time monitoring still works very well, it is not an option in some environments which makes mod_security2 upload scanning highly desirable. Conveniently, the only obstacle for upload scanning was simply that LMD did not support user initiated scans and with the introduction of public mode scanning there was only a few changes required to fully integrate it. That being the creation of a validation script for mod_security2’s inspectFile hook which returns an approved or denied status for uploaded files based on malware hits. This script was created as modsec.sh and is located in the LMD installation path. Full details on public mode and mod_security2 upload scanning are included in the README file.

Another highly requested feature is the ability to redefine configuration variables on the CLI on a per execution basis. This has been added through the -co|–config-option CLI flags. This was primarily requested by those creating integration interfaces for LMD along with those who create custom scan cronjobs. Likewise, this proved useful in the creation of the mod_security2 validation script. The usage of this feature is straight forward, simply append a comma spaced list of variables you would like to redefine in the format of VAR=VALUE.

For example, to change the email address for a specific scan and enable quarantining of hits:
maldet –config-option email_addr=you@domain.com,quar_hits=1

Effectively, any LMD variable located in conf.maldet or internal.conf can be redefined in this way.

Smaller changes include added support for Plesk in the cron.daily scans, email_ignore_clean conf.maldet variable that allows for reports where all hits are cleaned to be ignored and improved accuracy of (gz)base64 injection signatures to reduce false positives.

That covers the notable changes in this release. Although this isn’t as big or feature packed of an update as the last couple of releases, I am confident it will add to the maturity and utility of the project for all users. Please check the CHANGELOG and README files for further details. This update will push out automatically to LMD installations with the default daily cronjob enabled or you can manually update using the ‘maldet -d’ command.

LMD By The Numbers:
16,036 Downloads month-to-date (includes version updates)
15,261 Malware source URL’s tracked
14,443 Active installations (by unique IP daily signature queries)
11,017 Active 1.4.x installations (by unique IP daily signature queries)
10,192 File submissions pending malware review
9,644 Updates to 1.4.1 (by unique IP signature queries)
8,579 Total malware signatures
7,300 Google references to “linux malware detect”
6,715 MD5 malware signatures
6,374 Unique malware files in the LMD malware repository
3,221 Zombie server nodes seen in the last 30d on IRC C&C networks
1,864 HEX malware signatures
1,338 New signatures since 1.4.0
261 Command & Control IRC networks tracked
226 Signature updates in the last 12 months
196 Unique malware signature classifications
112 Files on average submitted daily through checkout feature
101 GB of bandwidth used per month on average to serve LMD updates
18 Signature updates per month on average
6.3 New signatures per day on average
1.6 Days between signature updates on average

2 Comments :, , more...

LMD 1.4: Little Something For Everyone!

by on Apr.20, 2011, under Development

The much awaited for 1.4 release of Linux Malware Detect is here! In this release there is quite literally something for everyone, from massive performance gains to FreeBSD support and everything in between :). For those who wish to dive straight into it, you can run the -d or –update-ver option to update your install to the latest build and check out the change log for full details.

I will try cover some of the highlights of this release for those with the appetite for it, here goes…

One of the more exciting changes is that Clam Anti-Virus is now supported as an optional scanner engine. When LMD detects that ClamAV is installed on the local system, through detection of the clamscan binary, it will default to using clamscan as the default scanner engine. The use of clamscan as the scanner engine leverages LMD in a couple of ways. First, it allows for ClamAV’s threat database to be used in detecting threats, over 900k strong, in addition to the LMD signatures which are ClamAV compatible. Secondly and more importantly, it improves scan performance greatly, over five times faster. Finally, it also improves the accuracy of threat detection as ClamAV is more efficient at doing hex payload analysis of files using LMD’s hex pattern match signatures. To enable this all you need to do is have ClamAV installed and LMD will detect it all on its own, if you wish to override the detection/usage of clamscan then you can set clamav_scan=0 in conf.maldet.

Another change that I am excited to announce, is that LMD 1.4 is now compatible with FreeBSD, less the inotify real-time monitoring as it is a Linux specific feature that requires me to design a new monitoring subsystem around FreeBSD’s inotify equivalent, kqueue. That said, allot of testing went into ensuring FreeBSD compatibility but it did not end there, I also went to great pains to improve Linux compatibility both with RH variants and non-RH variants alike, the officially supported set of distributions is as follows:
– RHEL/CentOS 5.6
– RHEL 6
– Fedora Core 14
– OpenSuse 11.4
– Suse Linux Enterprise Server 11 SP1
– Ubuntu Desktop/Server 10.10
– Debian 6.0.1a

This supported list is not meant as an exclusive list, it is simply a “test” set of distributions that I work with that give LMD the best expectation of working on an even wider set of Linux distributions. This improved compatibility will open up LMD to a larger community of users and there-in allow the project to grow and prosper in new and exciting ways.

The way LMD updates itself has now been improved, traditionally the daily signature updates only updated the core hex and md5 signature files but that proved to create some gaps in ensuring that all dynamic components for detecting threats are current. As such, now the update feature also pulls down the most current set of cleaner rules and LMD signatures in ClamAV format. In addition, the update process has seen an improvement in error checking; the signature files are now validated for length and missing files, if either validation checks fail then all signatures are forcibly updated.

The hex scanner (internally known as stage2 scanner) has been improved in that it now makes use of a named pipe (FIFO – first in first out) for processing file hex payload data, this allows for greater depth penetration into files and at a much lower cost in overhead. This means more accurate threat detection, fewer false positives and improved scan speeds; although it still pales in comparison to when clamscan is used as the scanner engine but nevertheless it is an improvement and an important one at that.

Further adding to the threat detection capabilities of LMD, is a new statistical analysis component that will see allot of expansion in later releases. The first feature in the statistical analysis component is called the string length test. The string length test is used to identify threats based on the length of the longest uninterrupted string within a file. This is useful as obfuscated code is often stored using encoding methods that produce very long strings without spaces (e.g: base64, gzip etc.. encoded files). This feature is presented in conf.maldet through the string_length variables, it is disabled by default as it can in some situations have a relatively high false-positive rate, especially on .js files. Future releases will see extension and file type based filtering specifically wrapped around the statistical analysis components to reduce false positives, however it is still a very powerful feature in detecting obfuscated/encoded malware.

There is a number of usage changes that have been made, the most notable and important being in ignore files, specifically the ignore_inotify and ignore_file_ext files.

The first, ignore_inotify is a specific file designed for ignoring paths from inotify real time monitoring, previous to LMD 1.4 this file only accepted absolute directory/file paths which was very limiting and created headaches for many people. The ignore_inotify file now fully supports posix extended regular expressions, meaning you can ignore absolute paths still or create regular expressions to cover specific file types or dynamic path/directory structures. An example of this is that temporary sql files may write out to /var/tmp in the format of /var/tmp/#sql_12384_4949.MYD, previously you would have to ignore /var/tmp completely which exposed the system more than it helped. Now, you can add an entry to ignore_inotify such as ^/var/tmp/#sql_.*\.MYD$ and it will properly ignore the temporary SQL files while retaining full monitoring of /var/tmp.

The second, ignore_file_ext was a feature added in the 1.3.x branch that was pulled back due to technical issues. The file speaks for itself, it allows you to ignore files from scan results based on file extensions, this has now been fixed and is working properly. The usage of the file is straight forward, simply add one extension per line to ignore_file_ext and it will be excluded from scan results (e.g: .tar.gz , .rpm , .html , .js etc…), there is no need to use an asterisk (*) in entries in the ignore file.

Further usage changes include that the -c|–checkout flags now supports directories instead of just absolute files, so you can upload threats to rfxn.com from an entire directory (please make sure all threats within the directory are actual malware, I would prefer not to sort through hundreds of html/web files). The -r|–scan-recent and -a|–scan-all flags now support single file scans, previously only directory paths were accepted. A background option has been added in the form of -b|–background that allows scans to be run in the background, the -b|–background option must come before the scan options, such as (see –help for more details):

maldet --background --scan-recent /home/?/public_html 7
maldet --background --scan-all /home/?/public_html
maldet -b -r /home/?/public_html 7
maldet -b -a /home/?/public_html

There have also been a couple of changes to the -e|–report flags allowing for the listing of available reports and emailing of previous scan reports. The usage of these changes is straight forward and is as follows:

maldet --report list
maldet --report SCANID user@domain.com

That about covers things, there have been a number of smaller changes and fixes in LMD 1.4 which are detailed in the change log. To ensure you are running the latest build please run the -d or –update-ver option to have LMD auto-update or visit the project home page and download the latest build.

2 Comments :, , , , , more...

LMD 1.3.9r1: Hexdepth Bug

by on Apr.03, 2011, under Development, My Blog

I have put up a revision to the 1.3.9 release of LMD that fixes a hexdepth bug in which malware greater than 65Kbytes would cause an error in the internal hexstring.pl script and be considered clean on the stage2 hex scanning of malware. This would mean that unless malware had a MD5 signature for it to be caught on stage1 scan, it would not be picked up by a corresponding HEX rule in stage2 scan if its file size was greater than 65Kbyte, due to the bug.

In addition, I have made the decision in this revision to enable release update checks in the default cron.daily entry installed by LMD, this can be found at /etc/cron.daily/maldet line 9 (after update) if you wish to comment it out. I would however encourage users to leave this option enabled as it will greatly improve receiving timely updates for future bugs fixes and release updates. In the past, the decision was made to not enable automatic release updates for many reasons but mostly in the interest of the software still maturing and being in early development, thereby not wanting to rock any boats with large and sweeping release updates to a version they may have got working the way they prefer. Now though, LMD has come a long way, the installer imports most options and ignore files and there are no drastically sweeping changes planned that will cause a great deal of headaches — so it seemed fitting time to enable automatic updates.

You can update your installation using the ‘maldet -d|–update-ver’ flags or download the current build for new installations.

This release update also coincides with passing 7k signatures….. We now sit at 7,106 signatures or +146 signatures added today. This is no small feat, I remember when we had just a couple hundred signatures not so long ago and I thought that was a big deal! The LMD submissions repository stays very active, it is now the source of almost 60% of the weekly signature additions and has contributed greatly to creating a vastly more accurate signature set that is representative of the threats you, the users, face day-to-day.

That said, month ending March stats recorded +1,464 installations of LMD bringing the install count to 7,157 — which puts LMD now ahead of APF in month-to-month new installation growth. Although, APF still beat LMD on raw downloads last month (3,091 vs 2,583), it is reasonable to predict that LMD will soon take the number one spot for downloads as well. It however still has a long way to go for total active installations, which APF sits at a comfortable 24,791 currently.

Till next time, happy malware hunting 🙂

Leave a Comment :, , , more...

LMD: One Year Later

by on Nov.08, 2010, under Development, My Blog

With my move back to Canada behind me and adjusting to some new routines with life, its about time to get back into the mix with the projects. Though things have been slow the last couple of months, it has not stopped me from making sure regular and prompt malware updates are released.

Today, we reflect on the first year of Linux Malware Detect, which was released in a very infantile beta release about a year ago. The project has evolved in allot of ways from its original goals, it has certainly changed in every way for the better. What was originally to be a closed project, relegated to mostly internal work related needs, ended up like most of my projects morphing into a public release. The first release saw the world with less than 200 signatures, no reliable signature update method, manual upgrade options and very flawed scanning and detection methods (v 0.7<). Now, we sit at version 1.3.6, with 4,813 signatures, a scanning method that though still needs some work, is far superior than what was originally in place, a detection routine based on solid md5 hashes and hex signatures. We have cleaner rules that can clean some nasty injected malware, we got a fully functional quarantine and restore system, reporting system, real-time file based monitoring, integrated signature updater and version updater and a vibrant community of users that regularly submit malware for review. Yes, LMD has grown up! The most grown-up part of LMD has to be how signatures are handled and how the processing of them is almost an entirely automated process now, this was detailed a little more in Signature Updates & Threat Database posted in September. The key part here though, is “almost entirely automated”, everyday that the processing scripts run to bring in new malware, there is always a number of files that cant be processed automatically and these are moved to a manual review queue. With how busy life has been the last couple of months, the review queue has slowly risen to 1,097 files pending review. This queue is at the top of my list for tackling over the next couple of days and weeks, its allot of work to review that much malware but it will get done. Many of the files to review are actual user-submissions so if you did submit something and find its still not detected by LMD, this would be why :).

There is still allot on the to-do list for LMD going forward, with the upcoming release of version 2.0 we will see some changes in how LMD does business. The first and to me the biggest will be optional usage statistics, which will allow users to have LMD report anonymized statistics back to rfxn.com. These statistics will show us which malware hits are found on your servers, which in turn contributes towards better focus on what type of malware threats are prioritized in the daily processing queue for hashing & review. The statistics will also help create informative profiles on the soon-to-be-released dailythreats.org web site about how maldet is used and what are the most prevalent threats in the wild.

Other additions to LMD 2.0 will be a refined scanner that will provide greater speed with large file sets (50k – 1M+ files), an ability to fork scans to the background, better and more predictable logging format for 3rd party processing of LMD log data, redesigned reporting system, full BSD support, ability to create custom signatures from the LMD command line, expanded cleaner rules, wildcard support for exclude paths, a number of security and bug refinements and as always, more signatures.

If you have any feature requests for LMD 2.0, go ahead and post them as a comment and I will make sure they get added to the list. Thank you to everyone who continues to support rfxn.com projects through donations, feedback and by just using & spreading the word about the projects. I look forward to another year of LMD and seeing it become the premier malware detection tool for Linux and all Unix variant OS’s.

7 Comments :, , , more...

Signature Updates & Threat Database

by on Sep.16, 2010, under Development, My Blog

It has been a very active month for those that pay attention to the signatures as they are released, you might have noticed a sudden spike about two weeks ago in signatures from 2,500’ish to the now 4,425 mark. The vast majority of these signatures were put up in MD5 format as a great many are variants of “known” malware and were extracted through processing historical threat data for the last 90 days, sorted by unique hashes, from clean-mx.de. I also did some leg work in my processing scripts which has allowed them to handle base64 and gzip decoding of POST payloads from IPS data which is generating a marked increase in new malware and known malware variants. Together, this has added 1806 MD5 and 31 HEX signatures in the last 45 days bringing us to the current mark of 4425 (2808 MD5 / 1617 HEX) total signatures.

In addition to the above, the daily processing scripts have been rewritten and combined into a single task on the processing server, this has brought together what was previously 9 different scripts into a single, streamlined and much more efficient task. The reason that things got to the point where there was 9 different scripts to update various elements of the back end processing server is that the LMD project developed very fluidly over the last year, meaning that every time I had a new idea or added a new feature, I in turn created a new script to support the idea/feature — over time this naturally was not sustainable and now what we have is exactly that — sustainable.

For those interested, here is the output report generated and sent to my inbox at the end of each daily malware update task:

started daily malware update tasks at 2010-09-13 00:09:35
running daily malware fetch... finished in 710s
running daily ftp malware fetch... finished in 6s
regenerating signatures from daily malware HEX hits... finished in 95s
propagating signature files... finished in 2s
generating sqlfeed data... finished in 88s
running mysql inserts for sqlfeed on praxis... finished in 42s
syncing & updating malware source data (master-urls.dat).... finished in 27s
syncing & updating irc c&c nets... finished in 15s
rebuilding maldetect-current... finished in 3s
pushing maldetect-current and signatures to web... finished in 4s
completed daily malware update tasks at 2010-09-13 00:26:05 (990s)
processed 156 malware url's
retrieved 40 malware files
extracted and hashed 16 new signatures
extracted 59 new irc c&c networks
queued 24 unknown files for review

An important part to streamlining the daily update tasks was also in rewriting some of the basic processing scripts to better log and store information on malware sources, such information includes date, source url, file md5, sig name, top level domain, online state, ip, asn, netowner and more. All malware is also now processed through an IRC extraction script that checks for irc server details in malware files and adds it to a irc command & control list with details such as date, source file md5, source file sig name, irc server, irc port, irc chan, online state, ip, asn, netowner and more. The “online state” fields in both the malware source and IRC c&c databases perform active checks, for the malware source this is simply verifies a URL is still active and/or domain still resolves, for the IRC c&c database this is a bot that manually connects to the irc network and verifies the network and channels are online & populated. All irc users, host masks and a sampling period of channel activity is also recorded from each active IRC c&c network, this information at this time is not included in the database as allot of it requires sanitizing as many IRC c&c networks dont mask connecting hosts and the channel activity reveals exceedingly sensitive information about actively vulnerable web sites and servers, this is something I am working on adding but its a difficult task so it will take some time. The malware signatures database has also been populated but requires a little more work, mainly adding meta data to describe each signature in a format that is longer than the single-word descriptions included in the signature naming scheme.

Together, the malware signature database, the malware source database and the IRC C&C networks database will all tie together into a single threat portal to be released in the next couple of weeks (I hope) allowing correlation between data in all 3 databases seamlessly. For example one could query all malware sourced from a specific IP, ASN or Netowner or you could find all the source URL’s for a specific malwares MD5 signature, or you could query the signature database to find more information on a specific signature, etc… there are a great many options that will be available for reviewing, cross referencing and exporting data from the databases.

These databases are all already completed, active and receiving updates, all that is left for me to do is create the front end that will find its home on http://www.dailythreats.com. The signature database, as expected, has 4,526 entries, the malware source database has 7,859 entries and the IRC C&C database has 386 entries. There is currently 511 files pending review in the malware queue, there has been 3,592 malware files reviewed in the last 45 days, of those 1,806 were unique files and the 511 files in queue for review represent files that could not be auto-hashed against a known threat or variant threat from HEX pattern matches.

The biggest pitfall of all these changes has been the explosion in the review queue that I must tend with daily, it has started to back up on me as I am in the middle of moving from Michigan to Montreal but as soon as I am done with my move in a couple of weeks, I plan to get that queue under control and work on some more back end scripts to help streamline its processing slightly.

Well that’s it for now, keep an eye out for details to come on the dailythreats.com site, its going to be exciting 🙂

Leave a Comment :, , more...

Looking for something?

Use the form below to search the site:

Still not finding what you're looking for? Drop a comment on a post or contact us so we can take care of it!

Visit our friends!

A few highly recommended friends...