LMD 1.4: Little Something For Everyone!

The much awaited for 1.4 release of Linux Malware Detect is here! In this release there is quite literally something for everyone, from massive performance gains to FreeBSD support and everything in between :). For those who wish to dive straight into it, you can run the -d or –update-ver option to update your install to the latest build and check out the change log for full details.

I will try cover some of the highlights of this release for those with the appetite for it, here goes…

One of the more exciting changes is that Clam Anti-Virus is now supported as an optional scanner engine. When LMD detects that ClamAV is installed on the local system, through detection of the clamscan binary, it will default to using clamscan as the default scanner engine. The use of clamscan as the scanner engine leverages LMD in a couple of ways. First, it allows for ClamAV’s threat database to be used in detecting threats, over 900k strong, in addition to the LMD signatures which are ClamAV compatible. Secondly and more importantly, it improves scan performance greatly, over five times faster. Finally, it also improves the accuracy of threat detection as ClamAV is more efficient at doing hex payload analysis of files using LMD’s hex pattern match signatures. To enable this all you need to do is have ClamAV installed and LMD will detect it all on its own, if you wish to override the detection/usage of clamscan then you can set clamav_scan=0 in conf.maldet.

Another change that I am excited to announce, is that LMD 1.4 is now compatible with FreeBSD, less the inotify real-time monitoring as it is a Linux specific feature that requires me to design a new monitoring subsystem around FreeBSD’s inotify equivalent, kqueue. That said, allot of testing went into ensuring FreeBSD compatibility but it did not end there, I also went to great pains to improve Linux compatibility both with RH variants and non-RH variants alike, the officially supported set of distributions is as follows:
– RHEL/CentOS 5.6
– RHEL 6
– Fedora Core 14
– OpenSuse 11.4
– Suse Linux Enterprise Server 11 SP1
– Ubuntu Desktop/Server 10.10
– Debian 6.0.1a

This supported list is not meant as an exclusive list, it is simply a “test” set of distributions that I work with that give LMD the best expectation of working on an even wider set of Linux distributions. This improved compatibility will open up LMD to a larger community of users and there-in allow the project to grow and prosper in new and exciting ways.

The way LMD updates itself has now been improved, traditionally the daily signature updates only updated the core hex and md5 signature files but that proved to create some gaps in ensuring that all dynamic components for detecting threats are current. As such, now the update feature also pulls down the most current set of cleaner rules and LMD signatures in ClamAV format. In addition, the update process has seen an improvement in error checking; the signature files are now validated for length and missing files, if either validation checks fail then all signatures are forcibly updated.

The hex scanner (internally known as stage2 scanner) has been improved in that it now makes use of a named pipe (FIFO – first in first out) for processing file hex payload data, this allows for greater depth penetration into files and at a much lower cost in overhead. This means more accurate threat detection, fewer false positives and improved scan speeds; although it still pales in comparison to when clamscan is used as the scanner engine but nevertheless it is an improvement and an important one at that.

Further adding to the threat detection capabilities of LMD, is a new statistical analysis component that will see allot of expansion in later releases. The first feature in the statistical analysis component is called the string length test. The string length test is used to identify threats based on the length of the longest uninterrupted string within a file. This is useful as obfuscated code is often stored using encoding methods that produce very long strings without spaces (e.g: base64, gzip etc.. encoded files). This feature is presented in conf.maldet through the string_length variables, it is disabled by default as it can in some situations have a relatively high false-positive rate, especially on .js files. Future releases will see extension and file type based filtering specifically wrapped around the statistical analysis components to reduce false positives, however it is still a very powerful feature in detecting obfuscated/encoded malware.

There is a number of usage changes that have been made, the most notable and important being in ignore files, specifically the ignore_inotify and ignore_file_ext files.

The first, ignore_inotify is a specific file designed for ignoring paths from inotify real time monitoring, previous to LMD 1.4 this file only accepted absolute directory/file paths which was very limiting and created headaches for many people. The ignore_inotify file now fully supports posix extended regular expressions, meaning you can ignore absolute paths still or create regular expressions to cover specific file types or dynamic path/directory structures. An example of this is that temporary sql files may write out to /var/tmp in the format of /var/tmp/#sql_12384_4949.MYD, previously you would have to ignore /var/tmp completely which exposed the system more than it helped. Now, you can add an entry to ignore_inotify such as ^/var/tmp/#sql_.*\.MYD$ and it will properly ignore the temporary SQL files while retaining full monitoring of /var/tmp.

The second, ignore_file_ext was a feature added in the 1.3.x branch that was pulled back due to technical issues. The file speaks for itself, it allows you to ignore files from scan results based on file extensions, this has now been fixed and is working properly. The usage of the file is straight forward, simply add one extension per line to ignore_file_ext and it will be excluded from scan results (e.g: .tar.gz , .rpm , .html , .js etc…), there is no need to use an asterisk (*) in entries in the ignore file.

Further usage changes include that the -c|–checkout flags now supports directories instead of just absolute files, so you can upload threats to rfxn.com from an entire directory (please make sure all threats within the directory are actual malware, I would prefer not to sort through hundreds of html/web files). The -r|–scan-recent and -a|–scan-all flags now support single file scans, previously only directory paths were accepted. A background option has been added in the form of -b|–background that allows scans to be run in the background, the -b|–background option must come before the scan options, such as (see –help for more details):

maldet --background --scan-recent /home/?/public_html 7
maldet --background --scan-all /home/?/public_html
maldet -b -r /home/?/public_html 7
maldet -b -a /home/?/public_html

There have also been a couple of changes to the -e|–report flags allowing for the listing of available reports and emailing of previous scan reports. The usage of these changes is straight forward and is as follows:

maldet --report list
maldet --report SCANID [email protected]

That about covers things, there have been a number of smaller changes and fixes in LMD 1.4 which are detailed in the change log. To ensure you are running the latest build please run the -d or –update-ver option to have LMD auto-update or visit the project home page and download the latest build.

Linux Malware Detect v1.3.3: Making good things better

This morning I have put out LMD v1.3.3, this is on the back of two other successive releases in recent days that improved LMD in many areas, along with correcting some bugs that were graciously reported by those helping to break-in the project. I have also listened to feedback and revised a number of features along with completely redoing how the inotify monitoring operates, to provide a much more robust model for real-time file monitoring.

I am also happy to say that people are embracing the use of the -c|–checkout option to send me malware that is not currently detected, which is being processed daily with my regular signature maintenance tasks. I have today added 24 new signatures, all of them created from user submissions.

There are a few big changes in this release…

First and foremost is that the configuration file conf.maldet has been completely revised with more granular options provided for quarantine, scan and monitoring, along with better commenting. Adding to the configuration convenience is that the install.sh script will now import config settings from previous install along with migrating session data.

Next up and something I am excited about, is a rule driven – signature based – cleaner function that can remove string based malware injections from files. The cleaner has two default rules created by me that will clean files of base64 and gzinflate injected strings very accurately. Through the next couple of days/weeks, I will be adding more cleaner rules that will allow for a much broader base of signatures that we can clean files for.

Finally, inotify monitoring got some loving with a top-down review of things and I came up with a less-invasive way of spawning the inotifywait processes that no longer requires a process for each path/user monitored. There is now a single master process that will monitor all configured paths, with better dynamic scaling of the sysctl hooks for inotify based on system resources. In addition, I added an option to pass the monitor service a comma spaced paths list or file containing line spaced paths, from the command line. This is in addition to preserving the users monitoring feature which has also been improved but is no longer the default, you must now call -m|–monitor with one of the USERS|FILE|PATHS options, see –help or the README file for more details.

Please be mindful that although LMD is considered stable it is still a relatively new project and as such your mileage may be a little bumpy, if you run into any issues please post comments on the project page, in this post or send me an email to ryan rfxn.com.

Home: http://www.rfxn.com/projects/linux-malware-detect/
Current Release:

v1.3.3 | May 15th 2010:
[Fix] quarantined files were not properly dropping owner
[New] signature based, rule driven, cleaner component added
[New] base64.inject cleaner rule
[New] gzbase64.inject cleaner rule
[New] -n|--clean SCANID option added to batch clean scan all files from a scan
[Fix] made default install file/path permissions more strict (750/640)
[New] install.sh now preserves conf.maldet settings
[New] install.sh now links backups of old installation to INSTALL_PATH.last
[Fix] install.sh now properly imports session data from previous install
[New] -s|--restore can now take a SCANID to batch restore all files from a scan
[Change] improved the layout of conf.maldet; more scan options and commenting
[New] added quar_susp_minuid option for suspend user minimum user id
[Fix] inotify monitor now properly acts on MODIFY,MOVE_TO,MOVE_FROM states
[Change] inotify monitor now can take a list of paths or file for path input
[Change] inotify monitor now has no default use, must specifiy USER|FILE|PATHS
[Change] revised short and long usage output for new options/usage changes
[Change] inotify monitor now spawns only one process for all monitored paths
[Change] inotify monitor sets max_user_instances to processors*2
[Change] inotify monitor sets max_user_watches to inotify_base_watches*users
[Change] migrated all inotify options from internals.conf to conf.maldet
[New] added inotify_base_watches to conf.maldet for max file wathces multiplier
[New] added inotify_nice to conf.maldet for run-time prio of inotifywait
[New] added inotify_webdir to conf.maldet for html/web root only monitoring
[Change] extensive format change to README
[Change] rewrote inotify section of README to reflect the many changes
[Change] -q|--quarantine now calls cleaner if quar_clean=1
[Change] -n|--clean can now do in place cleaning without quarantine

Upgrade CentOS 4.8 to 5.x (32bit)

Traditionally, the dist upgrade path that many were familiar with from the RH8/9->Fedora or similarly Fedora dist upgrades, have applied more or less to RHEL/CentOS but with the release of 4.5 and early releases of 5.0 the actual dist upgrade path was messy or nearly impossible. The early versions of 5.0 (up to 5.2) had excessive dependency issues with versions later than 4.4 for straight dist upgrades that would often result in a box blowing up on you or forcing a messy downgrade attempt of 4.5+ to 4.4 to try get things to dist upgrade. With more recent release updates the gap has closed and now dist upgrades on are far more reasonable to complete with little in the way of problems.

If you are currently running a version of RHEL/CentOS earlier than 4.8 (cat /etc/redhat-release) then please do a proper ‘yum update’ and get yourself on 4.8. Although this is intended for CentOS it “should” (read: at own risk) work on RHEL systems as well, in the unfortunate situation that something does blow up please post a comment and I will try to assist.

The first thing we must do is make sure none of our core binaries, libraries or other content is set immutable as this will cause a package to fail on installation. If you are running an earlier version of LES or you use immutable bits on system paths (sbin/bin/share/include/libexec/etc) then you should run the following:

wget http://www.rfxn.com/downloads/disable.les.rpmpkg
sh disable.les.rpmpkg

Once that is done we should go ahead and have a quick run through of cleaning up yum cache, double check that any pending updates are installed and rebuild the rpmdb:

rpm --rebuilddb
yum clean all
yum update

If for some reason the rpm rebuild hangs for more than a few minutes then you may need to manually clear the rpmdb files:

rm -f /var/lib/rpm/__db.00*
rpm --rebuilddb

If you run into any minor dependency issues for packages that are not essential, such as syslinux and lftp then you can either exclude them or better yet remove them. If you are not sure what a package does, then you should query it for description details and make an educated choice (rpm -qi PACKAGE):

rpm -e lftp syslinux mkbootdisk

OR (but not recommended)

yum update --exclude=syslinux --exclude=lftp --exclude=mkboot

At this point you should be able to run a ‘yum update’ command with optional exclude and receive no errors (again, I recommend you remove conflicts items instead of using exclusions).

# yum update –exclude=nagios-plugins
Setting up Update Process
Setting up repositories
Reading repository metadata in from local files
Excluding Packages in global exclude list
No Packages marked for Update/Obsoletion

Now we are ready to get going, I have put together a small package that contains the needed packages for this upgrade in addition to a few that you might require to resolve dependency conflicts:

wget http://www.rfxn.com/downloads/CentOS-5up.tar.gz
tar xvfz CentOS-5up.tar.gz
cd CentOS-5up

We need to go ahead and setup the centos-release package as follows:

rpm -Uhv centos-release-*

If you see that CentOS-Base.repo was created as /etc/yum.repos.d/CentOS-Base.repo.rpmnew then go ahead and move it into the proper place:

mv /etc/yum.repos.d/CentOS-Base.repo.rpmnew /etc/yum.repos.d/CentOS-Base.repo

Now we are ready to go with kernel changes, this is an important part so pay attention. The key to successful upgrade is that you remove ALL OLD KERNELS as many packages will fail to install during the upgrade if they detect a release 4.x kernel due to minimum kernel version dependency checks. We will start with first installing the new kernel so it preserves grub templating:

rpm -ivh kernel-2.6.18-164.el5.i686.rpm kernel-devel-2.6.18-164.el5.i686.rpm --nodeps

NOTE: release 5.x has smp support integrated into the standard kernel, so no -smp version is required for mp systems

If you are running an older system the chances are you got allot of older kernel packages installed so make sure you get them all out of the way:

rpm -e $(rpm -qa | grep kernel | grep -v 2.6.18 | tr '\n' ' ')

You may end up with a few dependencies coming up such as lm_sensors and net-snmp if the list is fairly small and packages you do not recognize as critical (if unsure always query the package for info ‘rpm -qi PACKAGE’, remember you can reinstall them later):

# rpm -e $(rpm -qa | grep kernel | grep -v 2.6.18)
error: Failed dependencies:
kernel-utils is needed by (installed) lm_sensors-2.8.7-2.40.5.i386

The command the ended up being required on most of my servers to get the kernel packages and related dependencies came out to the following:

rpm -e $(rpm -qa | grep kernel | grep -v 2.6.18 | tr '\n' ' ') lm_sensors net-snmp net-snmp-devel net-snmp-utils

That said and done you should now only have 2 kernel packages installed which are the 2.6.18 release 5.x kernels, DO NOT under any circumstance continue if you still got 2.6.9 release 4.x kernels packages still installed, remove them!

# rpm -qa | grep -i kernel

A cleanup of /etc/grub.conf may be required, though if all went as planned then the rpm command should have done this up for us but review it anyways for good measure. You should find that 2.6.18-164.el5 is the only kernel in the file, if it is not go ahead and clean it by removing all older entries for 2.6.9 kernels.

There is a known bug with python-elementtree package versions which cause yum/rpm to think the release 4.x version is newer than the 5.x version, to get around this without blowing up the entire python installation we need to remove the package from just the rpmdb as follows:

rpm -e --justdb python-elementtree --nodeps

We can now go ahead and use yum to start the upgrade process, this is a dry run and will take a few minutes to compile list of available packages and associated dependency checks. You should carry the exclude options, if any, that you used during the ‘yum update’ process as so to avoid unresolvable dependencies:

yum clean all
yum upgrade --exclude=nagios-plugins

You will end up with a small list of dependency errors, these should be resolved by again evaluating a packages need as a critical system component and either removing it with ‘rpm -e’ or excluding it with ‘–exclude’ (remember to query description with ‘rpm -qi PACKAGE’ if you are unsure what something does). In my case the packages that threw up red flags were stuff I had manually installed over time such as iftop and mrtg in addition to default installed samba, these can all safely be removed or excluded as you prefer (removal always safest to prevent dependency chain issues).

Error: Missing Dependency: libpcap.so.0.8.3 is needed by package iftop
Error: Missing Dependency: perl(Convert::ASN1) is needed by package samba
Error: Missing Dependency: libevent-1.1a.so.1 is needed bypackage nfs-utils
Error: Missing Dependency: perl-Socket6 is needed by package mrtg
Error: Missing Dependency: perl-IO-Socket-INET6 is needed by package mrtg

rpm -e iftop samba nfs-utils mrtg system-config-samba

At this point we should be ready to do a final dry run of with yum and see where we stand on dependencies, rerun the earlier ‘yum upgrade’ while making sure to carry over any exclude options you are using.

yum upgrade --exclude=nagios-plugins

You should now end up with a summary of actions that yum needs to perform, go ahead and kick it off… this will take a bit to complete so go grab some coffee/jolt/redbull and maybe a small snack cause it could be a long night if this blows up on you.

Transaction Summary
Install 183 Package(s)
Update 327 Package(s)
Remove 0 Package(s)
Total download size: 299 M
Is this ok [y/N]:

Once yum has completed (hopefully without major errors) we need to fix a few things, the first is the rpmdb needs a rebuild due to version changes that will cause any rpm commands to fail:

# rpm -qa
rpmdb: Program version 4.3 doesn’t match environment version
error: db4 error(-30974) from dbenv->open: DB_VERSION_MISMATCH: Database environment version mismatch
error: cannot open Packages index using db3 – (-30974)
error: cannot open Packages database in /var/lib/rpm

This can be fixed by running the following to manually rebuild the rpmdb:

rm -f /var/lib/rpm/__db.00*
rpm --rebuilddb
yum clean all

The next issue on the list is python-elementtree and python-sqlite, one or both of these may have ended up in a broken state that will cause all yum commands to break, so we will go ahead and reinstall both of these for good measure:

rpm -e --justdb python-elementtree --nodeps
rpm -ivh python-elementtree-1.2.6-5.el5.i386.rpm
rpm -ivh python-sqlite-1.1.7-1.2.1.i386.rpm --nodeps --force

The yum command should now work, go ahead and run it with no options, if you do not get any errors you are all sorted.

Hopefully the install went well for you, the only thing left to do is go ahead and reboot the system; this is the last point at which you have to make backups (but we all maintain backups right?). For the sake of avoiding a heart attack if the system goes into an fsck, we will reboot with the -f option to skip fsck:

shutdown -rf now

That’s a wrap, I hope you found this HowTo useful, if you did run into any issues then go ahead and post them into the comments field and I will try to assist but when in doubt typically google is the fastest alternative.