LMD 1.4: Little Something For Everyone!
The much awaited for 1.4 release of Linux Malware Detect is here! In this release there is quite literally something for everyone, from massive performance gains to FreeBSD support and everything in between . For those who wish to dive straight into it, you can run the -d or –update-ver option to update your install to the latest build and check out the change log for full details.
I will try cover some of the highlights of this release for those with the appetite for it, here goes…
One of the more exciting changes is that Clam Anti-Virus is now supported as an optional scanner engine. When LMD detects that ClamAV is installed on the local system, through detection of the clamscan binary, it will default to using clamscan as the default scanner engine. The use of clamscan as the scanner engine leverages LMD in a couple of ways. First, it allows for ClamAV’s threat database to be used in detecting threats, over 900k strong, in addition to the LMD signatures which are ClamAV compatible. Secondly and more importantly, it improves scan performance greatly, over five times faster. Finally, it also improves the accuracy of threat detection as ClamAV is more efficient at doing hex payload analysis of files using LMD’s hex pattern match signatures. To enable this all you need to do is have ClamAV installed and LMD will detect it all on its own, if you wish to override the detection/usage of clamscan then you can set clamav_scan=0 in conf.maldet.
Another change that I am excited to announce, is that LMD 1.4 is now compatible with FreeBSD, less the inotify real-time monitoring as it is a Linux specific feature that requires me to design a new monitoring subsystem around FreeBSD’s inotify equivalent, kqueue. That said, allot of testing went into ensuring FreeBSD compatibility but it did not end there, I also went to great pains to improve Linux compatibility both with RH variants and non-RH variants alike, the officially supported set of distributions is as follows:
– FreeBSD 9.0-CURRENT
– RHEL/CentOS 5.6
– RHEL 6
– Fedora Core 14
– OpenSuse 11.4
– Suse Linux Enterprise Server 11 SP1
– Ubuntu Desktop/Server 10.10
– Debian 6.0.1a
This supported list is not meant as an exclusive list, it is simply a “test” set of distributions that I work with that give LMD the best expectation of working on an even wider set of Linux distributions. This improved compatibility will open up LMD to a larger community of users and there-in allow the project to grow and prosper in new and exciting ways.
The way LMD updates itself has now been improved, traditionally the daily signature updates only updated the core hex and md5 signature files but that proved to create some gaps in ensuring that all dynamic components for detecting threats are current. As such, now the update feature also pulls down the most current set of cleaner rules and LMD signatures in ClamAV format. In addition, the update process has seen an improvement in error checking; the signature files are now validated for length and missing files, if either validation checks fail then all signatures are forcibly updated.
The hex scanner (internally known as stage2 scanner) has been improved in that it now makes use of a named pipe (FIFO – first in first out) for processing file hex payload data, this allows for greater depth penetration into files and at a much lower cost in overhead. This means more accurate threat detection, fewer false positives and improved scan speeds; although it still pales in comparison to when clamscan is used as the scanner engine but nevertheless it is an improvement and an important one at that.
Further adding to the threat detection capabilities of LMD, is a new statistical analysis component that will see allot of expansion in later releases. The first feature in the statistical analysis component is called the string length test. The string length test is used to identify threats based on the length of the longest uninterrupted string within a file. This is useful as obfuscated code is often stored using encoding methods that produce very long strings without spaces (e.g: base64, gzip etc.. encoded files). This feature is presented in conf.maldet through the string_length variables, it is disabled by default as it can in some situations have a relatively high false-positive rate, especially on .js files. Future releases will see extension and file type based filtering specifically wrapped around the statistical analysis components to reduce false positives, however it is still a very powerful feature in detecting obfuscated/encoded malware.
There is a number of usage changes that have been made, the most notable and important being in ignore files, specifically the ignore_inotify and ignore_file_ext files.
The first, ignore_inotify is a specific file designed for ignoring paths from inotify real time monitoring, previous to LMD 1.4 this file only accepted absolute directory/file paths which was very limiting and created headaches for many people. The ignore_inotify file now fully supports posix extended regular expressions, meaning you can ignore absolute paths still or create regular expressions to cover specific file types or dynamic path/directory structures. An example of this is that temporary sql files may write out to /var/tmp in the format of /var/tmp/#sql_12384_4949.MYD, previously you would have to ignore /var/tmp completely which exposed the system more than it helped. Now, you can add an entry to ignore_inotify such as ^/var/tmp/#sql_.*\.MYD$ and it will properly ignore the temporary SQL files while retaining full monitoring of /var/tmp.
The second, ignore_file_ext was a feature added in the 1.3.x branch that was pulled back due to technical issues. The file speaks for itself, it allows you to ignore files from scan results based on file extensions, this has now been fixed and is working properly. The usage of the file is straight forward, simply add one extension per line to ignore_file_ext and it will be excluded from scan results (e.g: .tar.gz , .rpm , .html , .js etc…), there is no need to use an asterisk (*) in entries in the ignore file.
Further usage changes include that the -c|–checkout flags now supports directories instead of just absolute files, so you can upload threats to rfxn.com from an entire directory (please make sure all threats within the directory are actual malware, I would prefer not to sort through hundreds of html/web files). The -r|–scan-recent and -a|–scan-all flags now support single file scans, previously only directory paths were accepted. A background option has been added in the form of -b|–background that allows scans to be run in the background, the -b|–background option must come before the scan options, such as (see –help for more details):
maldet --background --scan-recent /home/?/public_html 7 maldet --background --scan-all /home/?/public_html maldet -b -r /home/?/public_html 7 maldet -b -a /home/?/public_html
There have also been a couple of changes to the -e|–report flags allowing for the listing of available reports and emailing of previous scan reports. The usage of these changes is straight forward and is as follows:
maldet --report list maldet --report SCANID email@example.com
That about covers things, there have been a number of smaller changes and fixes in LMD 1.4 which are detailed in the change log. To ensure you are running the latest build please run the -d or –update-ver option to have LMD auto-update or visit the project home page and download the latest build.
|Print article||This entry was posted by Ryan M. on April 20, 2011 at 1:25 am, and is filed under Development. Follow any responses to this post through RSS 2.0. You can leave a response or trackback from your own site.|
No trackbacks yet.
about 1 year ago - 2 comments
The release of LMD 1.4.1 is now live and with it comes a few new features. In this small update, I have tried to deliver on on a couple of common feature requests from users which were in-line with my development goals. That said, right to it… The biggest change has come in the form More >
about 1 year ago - 10 comments
As cliche as it sounds, where has the time gone? Today we celebrate two years of Linux Malware Detect, open-source (web) malware detection. The project has seen allot of change since the first release. What was initially started as an internal project to deal with a large increase in malware activity at my job, a More >
about 2 years ago - 5 comments
New technologies, new toys — Oh how I love getting my hands dirty with them. Today I am going to have a look at ATA Over Ethernet (AoE) as an alternative solution to NFS in the role of a NAS/SAN implementation. We will look at both the server side vblade setup and the client side More >
about 2 years ago - No comments
I have put up a revision to the 1.3.9 release of LMD that fixes a hexdepth bug in which malware greater than 65Kbytes would cause an error in the internal hexstring.pl script and be considered clean on the stage2 hex scanning of malware. This would mean that unless malware had a MD5 signature for it More >
about 2 years ago - No comments
It has been a busy couple of weeks for the LMD project, lots of late nights and sleepless days behind me and I can say I am a ‘little’ happier with where things are in the project now This release has no major feature changes or additions other than a modification in the default hexdepth More >
about 2 years ago - 3 comments
In today’s hosting environment it is common place for servers to have hardware based raid cards but what is not common place is having a reliable method for checking the status of the raid arrays. Few would question the value to data integrity by making use of raid technology but very few organizations and businesses More >
about 2 years ago - 1 comment
Today marks the release of LMD 1.3.7, which is a minor release update that fixes a few bugs and is also the final 1.x release before version 2.0 as described in the LMD: one year later blog post. The bug list for LMD has remained very small over the last 6 months and this release More >
about 2 years ago - 7 comments
With my move back to Canada behind me and adjusting to some new routines with life, its about time to get back into the mix with the projects. Though things have been slow the last couple of months, it has not stopped me from making sure regular and prompt malware updates are released. Today, we More >
about 2 years ago - No comments
It has been a very active month for those that pay attention to the signatures as they are released, you might have noticed a sudden spike about two weeks ago in signatures from 2,500′ish to the now 4,425 mark. The vast majority of these signatures were put up in MD5 format as a great many More >
about 2 years ago - 1 comment
In a previous blog I discussed how one of the more enjoyable parts of my day-to-day malware rituals also involves the tracking and killing of command and control bot networks. Recently I have begun automating this process a bit; I have created a series of scripts that extract irc servers, port numbers and channels from More >