Apache Week
   

Copyright 1996-2005
Red Hat, Inc.

First published: 3rd March 2003

Vendor patches to Apache 1.3

We decided to take a look at what custom patches vendors add to the versions of Apache 1.3 they ship. The Apache Software Foundation would rather that vendors of Apache didn't add any third-party modifications to Apache at all - it adds to brand confusion. You might think you are getting a copy of the Apache web server but you're actually getting something that is based on the Apache web server.

There are hundreds of distributions and hundreds of vendors so in order to make this manageable we started out by looking at just Linux vendors that have publicised security updates for Apache in the first few months of 2003 to the bugtraq mailing list. Where a vendor has multiple versions of products we tried to look at the most recent version of Apache 1.3 (since most vendors do not yet ship Apache 2).

Our survey consisted of Conectiva, Debian, EnGarde, Gentoo, Mandrake, OpenPKG, Red Hat, SCO, SuSE, and Trustix.

Versions

At the time of the survey, not all the Linux vendors were shipping Apache 1.3.27. Several shipped older versions for which they had backported security fixes. Mandrake, Debian, and Conectiva included Apache 1.3.26 with backported patches for CAN-2002-0839, CAN-2002-0840, and CAN-2002-0843. SuSE included Apache 1.3.23 with backported security fixes for only CAN-2002-0392 and CAN-2002-0839. SuSE also add a backported patch for mod_proxy (PR#8234)

All the vendors shipped with EAPI, the interface that links Apache to mod_ssl, and most bundled some selection of extra modules.

Configuration files

All the vendors shipped a custom httpd.conf file or made patches to the default file. Examining the configuration file changes was outside the scope of this survey since these are things that can be easily changed by the user.

All the vendors except OpenPKG and SuSE pointed the magic mime types file at the system /etc/mime.types file, with many adding additional types using AddType directives in httpd.conf.

SysV init is a standard process used by Linux distributions to control which software the init command launches or shuts off on a given runlevel. These sometime get confused with the apachectl command which provides similar functionality. All the vendors except OpenPKG included custom init scripts or patches with their Apache packages.

Build Patches

All the vendors provided patches to help build Apache on their particular Linux distribution and to customise it to their environment. Conectiva, Gentoo, and Mandrake added a serverroot configuration option and then used that to help build Apache. Most vendors patched apxs and changed file and directory locations.

Debian, Gentoo, Mandrake, Red Hat, and SuSE added dbm patches to ensure that the files created for dbm-based authentication from Perl tools like dbmmanage are in a format that Apache can understand.

Additional security fixes

Conectiva, Debian, EnGarde, Gentoo, Mandrake, Red Hat, and SCO all included a patch for CAN-2001-0131, a vulnerability in htpasswd and htdigest that could allow local users to overwrite arbitrary files via a symlink attack. This vulnerability is not yet fixed in Apache, as it's tricky to get right cross-platform. The vendors patching this themselves only have to worry about the Linux architecture so can add a specific fix.

Server version strings

Altering the server version string can help users determine that they are running a vendor-modified version of Apache. It can also help the vendor track market share through surveys like those from Netcraft. Four of the distributions had patches to make sure that they added a customised string to the server version string. These distributions were quite well behaved and did not add their customised string if the ServerTokens directive is set to 'product only' or 'minimum'.

  • Debian GNU/arch
  • (Gentoo/Linux)
  • (Red-Hat/Linux)
  • (Trustix Secure Linux/Linux)

Conectiva and SCO were a little more invasive, with Conectiva adding (Conectiva/Linux) to the server version string no matter what the ServerTokens directive was set to. SCO did a similar thing, with their extra string giving the version of an acceleration patch they add.

Finally, Mandrake changed the base product name altogether, renaming from Apache to Apache-AdvancedExtranetServer.

Hard Limits

In Apache 1.3, a compile-time constant defines the maximum possible number of server processes, defaulting to 256. Only three vendors changed this default: Debian set it to 512 processes via a build-time define, EnGarde patch it to 1024, and SuSE set it to 2048 via a define.

Large File Support

Debian, Mandrake, SuSE, and SCO build Apache with Large File support, so that on 32-bit systems Apache can use files larger than 2 gigabytes - this is particularly useful for log files. Enabling LFS does slightly change the Apache 1.3 binary module ABI, which can cause problems if using binary modules built against a different version of Apache.

The remainder

After taking account of all the patches and modifications above, we're left with only four vendors that add additional patches.

SuSE added:

  • A patch to change the ap_set_content_length API function to accept a length of type off_t instead of long, to improve the support for Large Files mentioned above.

Gentoo added:

  • A patch to make the regexp library work with Large File Support on 32-bit systems. This is a modification the affects the ABI.
  • A patch to fix a segmentation fault when using a custom response in a module, (PR#8334)
  • A patch to fix a problem when using server-parsed HTML with suexec where an <--#exec tag with a cmd attribute contains more than one word. (Debian bug 47951)
  • A patch to allow SSL environment variables to be accessible when using mod_ssl and suExec. (similar to PR#9163)
  • A patch to cause Apache to not run if user or group directives are found within a VirtualHost but suExec is not configured correctly. (Debian bug 21525)

Debian added the same patches as Gentoo and additionally:

  • A fix for a htdigest buffer overflow if arguments passed to it are too long. This is only a security issue if htdigest is used setuid
  • Changes to ApacheBench to support round-robin DNS

SCO added:

  • A patch to mod_proxy needed for mod_backhand
  • A patch to add a new API function, ap_call_execute, needed by the old mod-frontpage-VR module
  • the "Accelerating Apache" performance patches from SGI.

The "Accelerating Apache" performance patches were first submitted to the Apache Group by SGI in 1999. We reported that they were designed to improve the performance of Apache when measured specifically by the SPECweb96 benchmark. The patches were named after the ten fold increase in speed they gave over regular Apache on a dual processor SGI IRIX machine. Some of the patches were folded in to Apache in 2000, but other parts were rejected by the Apache developers. The Accelerating Apache project was dropped by SGI in February 2001.

Security Consequences

In March 2003 a vulnerability was found in the Oracle modifications to mod_dav. This was not the first security hole that has been introduced by third party modifications to Apache by vendors. However our own research based on issues listed in the CVE dictionary shows that the majority of these vulnerabilities are due to poor configuration defaults rather than patches for new functionality that went wrong:

CVE Type of Issue Severity Affected
CAN-2002-0842 Remote attacker can run arbitrary commands High Oracle
CAN-2002-0842 Remote attacker can run arbitrary commands High SCO (briefly)
CAN-2000-1168 Remote attacker can run arbitrary commands High IBM
CVE-2000-1016 Remote attacker can see files in /usr/doc Low SuSE Linux
CVE-2000-0883 Remote attacker can see files in /perl Medium Mandrake Linux
CVE-2000-0869 Remote attacker can read and write any file in docroot High SuSE Linux
CVE-2000-0868 Remote attacker can obtain the source to CGI scripts Medium SuSE Linux
CVE-2000-0234 Remote attacker can read .htaccess files Medium Cobalt
CVE-1999-0678 Remote attacker can see files in /usr/doc Low Debian Linux

Conclusion

What we found in our survey was that no two of the ten vendors were alike; some vendors like OpenPKG made only the expected build and configuration changes, whilst others made fairly substantial changes including affecting the ABI. ABI changes mean that you can't reliably take a module precompiled for one distribution and start using it on another. Third party modifications to Apache have been known to cause bugs and security issues. This is often frustrating for the Apache Software Foundation who end up receiving all the bug reports for issues that don't even exist in the official Apache releases. This is one of the reasons why the Apache Software Foundation insists that when vendors make modifications to Apache that they change the name of their version so it is not confused with official Apache releases.

One thing that impressed us was how easy it was to identify the changes that the vendors had made. In almost all cases the vendor's source package contained a pristine copy of Apache along with one or more patch files for the various changes. Working out what those changes did and where they came from was another issue though, vendors could do a much better job of labelling the origin of, and reason for, each of the patches they make.


This feature brought to you by: Mark J Cox
Comments or criticisms? Please email us at editors@apacheweek.com