LMD 1.4: Little Something For Everyone!

The much awaited for 1.4 release of Linux Malware Detect is here! In this release there is quite literally something for everyone, from massive performance gains to FreeBSD support and everything in between :). For those who wish to dive straight into it, you can run the -d or –update-ver option to update your install to the latest build and check out the change log for full details.

I will try cover some of the highlights of this release for those with the appetite for it, here goes…

One of the more exciting changes is that Clam Anti-Virus is now supported as an optional scanner engine. When LMD detects that ClamAV is installed on the local system, through detection of the clamscan binary, it will default to using clamscan as the default scanner engine. The use of clamscan as the scanner engine leverages LMD in a couple of ways. First, it allows for ClamAV’s threat database to be used in detecting threats, over 900k strong, in addition to the LMD signatures which are ClamAV compatible. Secondly and more importantly, it improves scan performance greatly, over five times faster. Finally, it also improves the accuracy of threat detection as ClamAV is more efficient at doing hex payload analysis of files using LMD’s hex pattern match signatures. To enable this all you need to do is have ClamAV installed and LMD will detect it all on its own, if you wish to override the detection/usage of clamscan then you can set clamav_scan=0 in conf.maldet.

Another change that I am excited to announce, is that LMD 1.4 is now compatible with FreeBSD, less the inotify real-time monitoring as it is a Linux specific feature that requires me to design a new monitoring subsystem around FreeBSD’s inotify equivalent, kqueue. That said, allot of testing went into ensuring FreeBSD compatibility but it did not end there, I also went to great pains to improve Linux compatibility both with RH variants and non-RH variants alike, the officially supported set of distributions is as follows:
– FreeBSD 9.0-CURRENT
– RHEL/CentOS 5.6
– RHEL 6
– Fedora Core 14
– OpenSuse 11.4
– Suse Linux Enterprise Server 11 SP1
– Ubuntu Desktop/Server 10.10
– Debian 6.0.1a

This supported list is not meant as an exclusive list, it is simply a “test” set of distributions that I work with that give LMD the best expectation of working on an even wider set of Linux distributions. This improved compatibility will open up LMD to a larger community of users and there-in allow the project to grow and prosper in new and exciting ways.

The way LMD updates itself has now been improved, traditionally the daily signature updates only updated the core hex and md5 signature files but that proved to create some gaps in ensuring that all dynamic components for detecting threats are current. As such, now the update feature also pulls down the most current set of cleaner rules and LMD signatures in ClamAV format. In addition, the update process has seen an improvement in error checking; the signature files are now validated for length and missing files, if either validation checks fail then all signatures are forcibly updated.

The hex scanner (internally known as stage2 scanner) has been improved in that it now makes use of a named pipe (FIFO – first in first out) for processing file hex payload data, this allows for greater depth penetration into files and at a much lower cost in overhead. This means more accurate threat detection, fewer false positives and improved scan speeds; although it still pales in comparison to when clamscan is used as the scanner engine but nevertheless it is an improvement and an important one at that.

Further adding to the threat detection capabilities of LMD, is a new statistical analysis component that will see allot of expansion in later releases. The first feature in the statistical analysis component is called the string length test. The string length test is used to identify threats based on the length of the longest uninterrupted string within a file. This is useful as obfuscated code is often stored using encoding methods that produce very long strings without spaces (e.g: base64, gzip etc.. encoded files). This feature is presented in conf.maldet through the string_length variables, it is disabled by default as it can in some situations have a relatively high false-positive rate, especially on .js files. Future releases will see extension and file type based filtering specifically wrapped around the statistical analysis components to reduce false positives, however it is still a very powerful feature in detecting obfuscated/encoded malware.

There is a number of usage changes that have been made, the most notable and important being in ignore files, specifically the ignore_inotify and ignore_file_ext files.

The first, ignore_inotify is a specific file designed for ignoring paths from inotify real time monitoring, previous to LMD 1.4 this file only accepted absolute directory/file paths which was very limiting and created headaches for many people. The ignore_inotify file now fully supports posix extended regular expressions, meaning you can ignore absolute paths still or create regular expressions to cover specific file types or dynamic path/directory structures. An example of this is that temporary sql files may write out to /var/tmp in the format of /var/tmp/#sql_12384_4949.MYD, previously you would have to ignore /var/tmp completely which exposed the system more than it helped. Now, you can add an entry to ignore_inotify such as ^/var/tmp/#sql_.*\.MYD$ and it will properly ignore the temporary SQL files while retaining full monitoring of /var/tmp.

The second, ignore_file_ext was a feature added in the 1.3.x branch that was pulled back due to technical issues. The file speaks for itself, it allows you to ignore files from scan results based on file extensions, this has now been fixed and is working properly. The usage of the file is straight forward, simply add one extension per line to ignore_file_ext and it will be excluded from scan results (e.g: .tar.gz , .rpm , .html , .js etc…), there is no need to use an asterisk (*) in entries in the ignore file.

Further usage changes include that the -c|–checkout flags now supports directories instead of just absolute files, so you can upload threats to rfxn.com from an entire directory (please make sure all threats within the directory are actual malware, I would prefer not to sort through hundreds of html/web files). The -r|–scan-recent and -a|–scan-all flags now support single file scans, previously only directory paths were accepted. A background option has been added in the form of -b|–background that allows scans to be run in the background, the -b|–background option must come before the scan options, such as (see –help for more details):

maldet --background --scan-recent /home/?/public_html 7
maldet --background --scan-all /home/?/public_html
maldet -b -r /home/?/public_html 7
maldet -b -a /home/?/public_html

There have also been a couple of changes to the -e|–report flags allowing for the listing of available reports and emailing of previous scan reports. The usage of these changes is straight forward and is as follows:

maldet --report list
maldet --report SCANID [email protected]

That about covers things, there have been a number of smaller changes and fixes in LMD 1.4 which are detailed in the change log. To ensure you are running the latest build please run the -d or –update-ver option to have LMD auto-update or visit the project home page and download the latest build.

LMD: One Year Later

With my move back to Canada behind me and adjusting to some new routines with life, its about time to get back into the mix with the projects. Though things have been slow the last couple of months, it has not stopped me from making sure regular and prompt malware updates are released.

Today, we reflect on the first year of Linux Malware Detect, which was released in a very infantile beta release about a year ago. The project has evolved in allot of ways from its original goals, it has certainly changed in every way for the better. What was originally to be a closed project, relegated to mostly internal work related needs, ended up like most of my projects morphing into a public release. The first release saw the world with less than 200 signatures, no reliable signature update method, manual upgrade options and very flawed scanning and detection methods (v 0.7<). Now, we sit at version 1.3.6, with 4,813 signatures, a scanning method that though still needs some work, is far superior than what was originally in place, a detection routine based on solid md5 hashes and hex signatures. We have cleaner rules that can clean some nasty injected malware, we got a fully functional quarantine and restore system, reporting system, real-time file based monitoring, integrated signature updater and version updater and a vibrant community of users that regularly submit malware for review. Yes, LMD has grown up!

The most grown-up part of LMD has to be how signatures are handled and how the processing of them is almost an entirely automated process now, this was detailed a little more in Signature Updates & Threat Database posted in September. The key part here though, is “almost entirely automated”, everyday that the processing scripts run to bring in new malware, there is always a number of files that cant be processed automatically and these are moved to a manual review queue. With how busy life has been the last couple of months, the review queue has slowly risen to 1,097 files pending review. This queue is at the top of my list for tackling over the next couple of days and weeks, its allot of work to review that much malware but it will get done. Many of the files to review are actual user-submissions so if you did submit something and find its still not detected by LMD, this would be why :).

There is still allot on the to-do list for LMD going forward, with the upcoming release of version 2.0 we will see some changes in how LMD does business. The first and to me the biggest will be optional usage statistics, which will allow users to have LMD report anonymized statistics back to rfxn.com. These statistics will show us which malware hits are found on your servers, which in turn contributes towards better focus on what type of malware threats are prioritized in the daily processing queue for hashing & review. The statistics will also help create informative profiles on the soon-to-be-released dailythreats.org web site about how maldet is used and what are the most prevalent threats in the wild.

Other additions to LMD 2.0 will be a refined scanner that will provide greater speed with large file sets (50k – 1M+ files), an ability to fork scans to the background, better and more predictable logging format for 3rd party processing of LMD log data, redesigned reporting system, full BSD support, ability to create custom signatures from the LMD command line, expanded cleaner rules, wildcard support for exclude paths, a number of security and bug refinements and as always, more signatures.

If you have any feature requests for LMD 2.0, go ahead and post them as a comment and I will make sure they get added to the list. Thank you to everyone who continues to support rfxn.com projects through donations, feedback and by just using & spreading the word about the projects. I look forward to another year of LMD and seeing it become the premier malware detection tool for Linux and all Unix variant OS’s.

Let The Rewrites Begin: New Life For PRM

In my last post, I reflected on the last 7-8 years of projects here at rfxn.com, in doing so I also dug up some statistics on project downloads. I not only did this for my own curiosity but to prioritize the mile long to do list I have for the projects, based on downloads. One of the revealing things was just exactly what people are downloading, in particular that projects like LES , PRM & SIM are still very popular download destinations on the site.

Although a new incarnation of APF & BFD are on the agenda, I thought I would work up to those by first knocking off rewrites of some of the smaller projects, starting this off is PRM. This is a project originally written in December of 2003 and although it has stood the test of time by doing exactly what it was intended for and doing it reliably, it was starting to show its age in a number of ways, especially the not-so-intuitive logic and less-than-appealing documentation.

Today I have put out PRM v1.0.6, a ground-up rewrite of just about everything in the project, simplified logic, oodles of new features and one of the biggest problem areas over the years, far better ignore options to control exactly what PRM does along with detailed documentation.

Enough said, check the changelog for for summary of changes and the README for details on the new usage.

Project Page: http://www.rfxn.com/projects/process-resource-monitor/
Current Release:
http://www.rfxn.com/downloads/prm-current.tar.gz
http://www.rfxn.com/appdocs/README.prm
http://www.rfxn.com/appdocs/CHANGELOG.prm