Build Apache and Bugzilla

Custom-compiling Subversion and Apache is the best way of controlling what you want (and don't want) from your repository. I recently had to set up a subversion repository for a project, and found that custom-compilation was the only way to get exactly what the project required.

The project requirements were:

1. Allow secure repository access from anywhere on the internet.
2. Ideally meet security requirements without having to ask the IT department to change any firewall rules.
3. Allow the repository to be kept on a local or an NFS-mounted filesystem.

Using Apache 2.0 as the front-end for Subversion would be the best way to meet requirements 1 and 2: We could use https, which was already allowed through our firewall. Using Subversion 1.1, and specifically using the new FSFS repository storage method, would allow us to satisfy requirment 3.

Custom-compiling Apache 2.0 would allow me to exclude modules that are typically compiled into most binary Apache distributions, but which seemed at odds with our goals, such as mod_userdir and mod_status. Custom-compiling Subversion would allow me to use the very latest, 1.2.1 (at the time of this writing), and be certain it integrated with my custom Apache install.

My work was done on Fedora Core 2, so this article will have the most relevence to that OS and distribution, but hopefully this article will get you most of the way there with other Unixes as well.

Prerequisites

Fedora Core is a reasonably feature-rich distribution, so it already comes with gcc and openssl, two essentials for getting Apache and Subversion working in the way that satisfied my three goals. If you do not have those packages on your Unix/Linux, you'll need to get them.

(Your distribution media for Fedora Core will have these packages if you haven't already installed them; http://www.sunfreeware.com will have these packages for Solaris; other resources are available for other Unices. Definitely get the binaries for these packages if you can; there's no benefit to be had from custom-compiling these packages.)

The exact versions used were OpenSSL 0.9.7a Feb 19 2003, and gcc (GCC) 3.3.2 20031022 (Red Hat Linux 3.3.2-1).

Please note that I'm assuming you are able to run all of these commands as root --- it's what's required to get a lot of this stuff working.

Compiling Apache

I downloaded the latest Apache 2.0 source code (2.0.54 at the time of this writing) and copied it to /usr/local/src where I always do my custom-compiling. If your system does not already have this directory, I would encourage you to create it.

(Red Hat / Fedora Core note: distribution RPMs never install in /usr/local, which is nice: after all, /usr/local is for stuff that's "local" to your installation! Red Hat and Fedora Core's packages respect the proper use of local and leave it alone, which is particularly nice at upgrade time: your local modifications to your distribution survive upgrades intact. Complimentarily, almost all software that is custom-compiled and installed from source code uses /usr/local as its base install path.)

Here's what I did to prepare Apache for compilation:

[root@localhost etc]# cp /downloads/httpd-2.0.54.tar.gz /usr/local/src
[root@localhost etc]# cd /usr/local/src
[root@localhost src]# tar -xzvf httpd-2.0.54.tar
[root@localhost src]# cd /usr/local/src/httpd-2.0.54

NOTE: Only GNU tar recognises the -z option. If your Unix has a non-gzip-aware tar, I recommend you do this in place of the tar command above:

[root@localhost src]# gunzip httpd-2.0.54.tar.gz
[root@localhost src]# tar -xvf httpd-2.0.54.tar
[root@localhost src]# # re-gzip to save disc space
[root@localhost src]# gzip httpd-2.0.54.tar

At this point, I was ready to configure Apache. I almost always run ./configure with arguments when I custom-compile software, and over the years I've come to make a wrapper script called ./runconfigure.sh so that I've got a record of what arguments I used when I ran ./configure. So here's the wrapper script that I used, below. In it, I asked for all modules to be shared, for most modules to be compiled in (especially ssl and dav, which are needed by subversion), but for mod_status and mod_userdir to be compiled out. Arguably, other superfluous modules like mod_cgi could have been compiled out as well.

#!/bin/sh
# runconfigure.sh -- wrapper script for ./configure
./configure --prefix=/usr/local/apache_ost_svn \
--enable-mods-shared="most ssl dav" \
--disable-status \
--disable-userdir \
--with-port=2002

# Sadly, there is no way to specify the ssl port;
# we will have to manually modify that later.

Here's what I did to compile and install Apache:

[root@localhost httpd-2.0.54]# chmod +x runconfigure.sh
[root@localhost httpd-2.0.54]# ./runconfigure.sh
[root@localhost httpd-2.0.54]# make
[root@localhost httpd-2.0.54]# make install

A Few Fixes to Apache's Configuration and Startup

Interestingly, the Apache install script did not remove the mod_userdir configuration from httpd.conf, even when I chose not to compile that module. So, I had to comment it out manually. There's only one line that needed to be commented out: UserDir public_html/. Here's one way to do so, without having to use an interactive editor:

[root@localhost httpd-2.0.54]# cd /usr/local/apache_ost_svn/conf
[root@localhost conf]# sed -i -e 's/^UserDir public_html/# &/' httpd.conf

There's another small problem I needed to fix with Apache so that SSL would work smoothly. The apachectl control script needs to define the property "SSL" on the command line to make Apache read its SSL configuration from httpd.conf and ssl.conf. Although "SSL" must be defined for both startup and shutdown, it is only defined in the startup clause of the Apache control script. To fix this problem, I opened /usr/local/apache_ost_svn/bin/apachectl in a text editor, looked for the line startssl|sslstart|start-SSL, and added the following lines above that line:

stopssl|sslstop|stop-SSL)
    $HTTPD -k stop -DSSL
    ERROR=$?
    ;;

Once you have installed your own custom-compiled Apache with mod_ssl, don't forget to start and stop Apache with the commands /usr/local/apache_ost_svn/bin/apachectl startssl and /usr/local/apache_ost_svn/bin/apachectl stopssl, not /usr/local/apache_ost_svn/bin/apachectl startand /usr/local/apache_ost_svn/bin/apachectl stop.

Securing Apache Conveniently

To run using https, Apache needs a certificate to give to browsers. This certificate is not generated when Apache is compiled; I had to create one myself.

First, I created the directories ssl.crt and ssl.key in the default locations expected by /usr/local/apache_ost_svn/conf/ssl.conf, (which is included by /usr/local/apache_ost_svn/conf/httpd.conf):

[root@localhost conf]# cd /usr/local/apache_ost_svn/conf
[root@localhost conf]# mkdir ssl.crt
[root@localhost conf]# mkdir ssl.key

My project only required a self-signed certificate, not a certificate from a bona fide certificate generating authority. Interestingly, there's a lot of confusing information on the internet on just how to generate a self-signed certificate for Apache, and, sadly, most of that information is way more complicated than it has to be. It turns out I was able to generate the key and the cert all in a single openssl command (note that, after some output, you will be prompted for a pass phrase twice):

[root@localhost conf]# openssl req -new -x509 -days 3650 \
-keyout ./ssl.key/server.key \
-out ./ssl.crt/server.crt \
-subj '/C=US/ST=Massachusetts/L=Boston/O=Big, Inc./OU=Dev/CN=www.myserver.com'

Please note that for the -subj option, the format of the line is described in RFC2253, and the allowable headings (such as "CN=" for "common name equals") is described in RFC1485 and probably later RFCs.

Because of the certificate, whenever Apache started, it prompted me for the pass phrase. This was not useful, because I planned on using apachectl (or some form of it) in my server's startup and shutdown scripts: I didn't want my server to stop and prompt me for a password every time I rebooted it --- after all, what if I had to reboot it remotely? I wouldn't be at the console to enter the pass phrase. Happily, there's a way to get around this problem (note that the openssl command prompted me for the pass phrase I selected above):

[root@localhost conf]# cp ssl.key/server.key ssl.key/server.key.orig
[root@localhost conf]# openssl rsa -in ssl.key/server.key.orig \
-out ssl.key/server.key
[root@localhost conf]# chmod 400 ssl.key/server.key

Note that the new ssl.key/server.key above is insecure, so that's why I protected it by permissioning the file as restrictively as possible.

Remember, above, how we were not able to specify an ssl port in the runconfigure script? Well, we will fix that now:

[root@localhost conf]# sed -i -e 's/443/2003/g' ssl.conf

At the very end of /usr/local/apache_ost_svn/conf/httpd.conf, add these mod-rewrite lines as a nice way to forward browsers from http to https:

# redirect all port 2002 requests to 2003
RewriteEngine on
RewriteCond "%{SERVER_PORT}"    "^2002$"
RewriteRule "^(.*)$"            "https://%{SERVER_NAME}:2003/$1" [R,L]

Also note that I changed httpd.conf's and ssl.conf's group from #-1 to nobody.

Also note that I changed httpd.conf's and ssl.conf's Listen and ServerName parameters to exact values.

Compiling Subversion

Subversion can be found at http://subversion.tigris.org. The latest version as of this writing was 1.2.1.

[root@localhost conf]# cp /downloads/subversion-1.2.1.tar.gz /usr/local/src
[root@localhost conf]# cd /usr/local/src/
[root@localhost src]# tar -xzvf subversion-1.2.1.tar.gz
[root@localhost src]# cd subversion-1.2.1

(Again, note that if you are not using GNU tar, you will have to gunzip the source code first, and then untar it.)

As you can see with my runconfigure.sh wrapper script for Subversion, I got very specific about which features I did and did not want in my Subversion installation. Because I knew exactly what I wanted, I could afford to do this, and it spared me from pre-requisite hell.

#!/bin/sh
# runconfigure.sh -- wrapper script for ./configure
./configure \
--with-apr=/usr/local/apache_ost_svn \
--with-apr-util=/usr/local/apache_ost_svn \
--with-ssl \
--without-berkeley-db \
--without-zlib \
--without-jdk \
--without-jikes \
--without-swig \
--without-junit

Once I saved runconfigure.sh, I ran it and installed Subversion:

[root@localhost subversion-1.2.1]# chmod +x runconfigure.sh
[root@localhost subversion-1.2.1]# ./runconfigure.sh
[root@localhost subversion-1.2.1]# make
[root@localhost subversion-1.2.1]# make check
[root@localhost subversion-1.2.1]# make install

For the convenience of whoever is allowed shell access to the machine, I wanted to ensure that Subversion was in the path (and, in fact, first in the path, because I wanted it to be run by default over any older version that may have shipped with my operating system).

First, I needed to ensure that Subversion's shared libraries could be found. On Linux, I did this by ensuring /etc/ld.so.conf contained the directory /usr/local/lib. (The /etc/ld.so.conf layout is just a list of directories, one per line --- a refreshingly simple syntax, in this age of XML.) If your copy of /etc/ld.so.conf is missing /usr/local/lib, add it and then run ldconfig with no arguments. On Solaris, ensure $LD_LIBRARY_PATH contains /usr/local/lib. If that directory is not present, add it. In fact, add it in /etc/profile and export LD_LIBRARY_PATH so that all users automatically pick it up when they log in. Also, if you are on Solaris, change your current environment so that you have access to Subversion's libraries right away:

LD_LIBRARY_PATH=/usr/local/lib
export LD_LIBRARY_PATH

On any Linux/Unix system, you will want to add Subversion to your path. On my machine, I placed it first in the path, so that it would override any older Subversion that might be installed in /usr/bin:

PATH=/usr/local/bin:$PATH
export PATH

And, on any Linux/Unix system, you will want to be sure that anyone who logs onto the system has Subversion in the path. Here's a nifty way I did that without having to open a text editor:

[root@localhost src]# cat >> /etc/profile <<EOR
> PATH=/usr/local/bin:$PATH
> export PATH
> EOR

Creating the Repository

Next, I needed to create a Subversion repository. For the purposes of this article, I will domonstrate how to create a new repository. (In real-life, I actually used cvs2svn to port an entire CVS repository to Subversion. Check out http://cvs2svn.tigris.org: as long as you already have Python installed on your system, cvs2svn is fantastic.) Here's how to create a sample repository and check in a one-file project:

[root@localhost src]# mkdir /usr/local/ost_svn_repository
[root@localhost src]# svnadmin create /usr/local/ost_svn_repository

Now, if you have a backup of a repository, you will load that backup, like so:

[root@localhost src]# svnadmin load /usr/local/ost_svn_repository < svn_backup.dump

Finally, as a convenience, here is the command-line you would execute to back up that original svn repository:

[root@localhost src]# svnadmin dump /usr/local/ost_svn_repository > svn_backup.dump

Otherwise, you can create and check in a test project, like so:

[root@localhost src]# cd /tmp
[root@localhost tmp]# mkdir -p test_project/trunk
[root@localhost trunk]# cd test_project/trunk
[root@localhost trunk]# cat > test_file.txt <<EOS
> this
> is
> a
> test
> file
> EOS
[root@localhost trunk]# svn import -m "initial checkin" . file:///usr/local/ost_svn_repository/test_project/trunk

Integrating Subversion with Apache

I wanted to use basic authentication to restrict access to my repository, so the first thing I needed to do was create an htpasswd file. And, what better location than the repository's configuration directory? Please note that for illustrative purposes, I used htpasswd's ability to accept passwords on the command line; you may want to make htpasswd prompt you for the passwords instead. I also used ridiculously insecure example passwords for illustrative purposes. Note how the first invocation of htpasswd used the -c switch to create the password file, whereas subsequent ones did not. Finally, if you don't mind having the passwords briefly shown in clear text, and you have to create a lot of users, consider making this a shell script that you can just run once and then delete when you are done:

[root@localhost trunk]# /usr/local/apache_ost_svn/bin/htpasswd -c -m -b /usr/local/ost_svn_repository/conf/htpasswd user1 password1
[root@localhost trunk]# /usr/local/apache_ost_svn/bin/htpasswd -m -b /usr/local/ost_svn_repository/conf/htpasswd user2 password2
[root@localhost trunk]# /usr/local/apache_ost_svn/bin/htpasswd -m -b /usr/local/ost_svn_repository/conf/htpasswd user3 password3

One step I didn't want to forget was making the repository readable and writeble by the user nobody, which Apache runs as by default. (Forgetting this step would produce sorts of errors from Apache.)

[root@localhost trunk]# chown -R nobody /usr/local/ost_svn_repository

Finally, I needed to change httpd.conf to let Apache know about my repository and the fact that only authorised users are allowed to read and write to it. I was going to use my Apache installation only for Subversion, so we will made the subversion repository the root of Apache's filesystem. In /usr/local/apache_ost_svn/conf/httpd.conf, I changed the line containing DocumentRoot from this:

DocumentRoot /usr/local/apache_ost_svn/htdocs

to this:

DocumentRoot /usr/local/ost_svn_repository

I also changed the Directory directive from this:

<Directory "/usr/local/apache_ost_svn/htdocs">

to this:

<Directory "/usr/local/ost_svn_repository">

Additionally, I had to chage /usr/local/apache_ost_svn/conf/ssl.conf so that this line:

DocumentRoot "/usr/local/apache_ost_svn/htdocs"

became this line:

DocumentRoot "/usr/local/ost_svn_repository"

Next, I added these lines to the end of /usr/local/apache_ost_svn/conf/httpd.conf, to let Apache know that /usr/local/ost_svn_repository is a Subversion repository and that access is restricted to ssl and basic authentication:

<Location />
    DAV svn
    SVNPath /usr/local/ost_svn_repository

    AuthType Basic
    AuthName "Subversion Repository"
    AuthUserFile /usr/local/ost_svn_repository/conf/htpasswd
    Require valid-user
    SSLRequireSSL
</Location>

In both /usr/local/apache_ost_svn/conf/httpd.conf and /usr/local/apache_ost_svn/conf/ssl.conf, I double-checked to be certain ports 80 and 443 were mapped to my custom ports of 2002 and 2003, to ensure that any web server I ran on the same machine would be able to use the standard ports without interfering with my subversion machine.

In both /usr/local/apache_ost_svn/conf/httpd.conf and /usr/local/apache_ost_svn/conf/ssl.conf, I ensured that the ServerName directive had the legitimate name of my web server.

Sadly, there is currently a flaw in the install target of Subversion's makefile, where it is assumed that Apache is installed in /usr/local/apache2, which was not the case when I did my install. Manually copying the libraries works:

[root@localhost conf]# cd /usr/local/src/subversion-1.2.1
[root@demo subversion-1.2.1]# cp subversion/mod_dav_svn/.libs/mod_dav_svn.so /usr/local/apache_ost_svn/modules/
[root@demo subversion-1.2.1]# cp subversion/mod_authz_svn/.libs/mod_authz_svn.so /usr/local/apache_ost_svn/modules/

Now add these two LoadModule directives to the bottom of the LoadModule stanza in httpd.conf (just after the LoadModule directive for mod_rewrite):

LoadModule dav_svn_module     modules/mod_dav_svn.so
LoadModule authz_svn_module   modules/mod_authz_svn.so

Ensuring Apache Starts when the Server Does

Chances are, your Linux/Unix installation shipped with Apache already installed. My Fedora Core system did, so I wanted to disable it to keep it from interfering with my custom Apache. (Although, at least my custom Apache is using different ports in case the original installation is ever accidentally started.)

The method for disabling the automatic startup of Apache is different from system to system, but let me show you how I disabled the Apache that shipped with my Fedora Core system.

Fedora Core uses a spiffy system utility called chkconfig to manage the starting and stopping of system services at various runlevels. I asked chkconfig what runlevels Apache was running at:

[root@localhost etc]# chkconfig --list httpd
httpd           0:off   1:off   2:off   3:on    4:on    5:on    6:off

I told chkconfig not to run Apache at the three runlevels where it was on, and then I manually shut down Apache (seeing as it was currently running).

[root@localhost etc]# chkconfig --level 345 httpd off
[root@localhost etc]# chkconfig --list httpd
httpd           0:off   1:off   2:off   3:off   4:off   5:off   6:off
[root@localhost etc]# /etc/rc.d/init.d/httpd stop
Stopping httpd:                                            [  OK  ]

Now what I needed to do was put a different script in /etc/rc.d/init.d to start and stop my custom Apache. Fortunately, /usr/local/apache_ost_svn/bin/apachectl is a shell script that already mostly follows the conventions of a Unix init script: it already takes "start" and "stop" as arguments. However, the non-standard "startssl" and "stopssl" are used to get Apache to use SSL, so I had to do some editing.

First, I copied Apache's control script into Fedora Core's standard location for init scripts:

[root@localhost etc]# cp /usr/local/apache_ost_svn/bin/apachectl /etc/rc.d/init.d/apache_ost_svn

Next, I borrowed, in altered form, the first few lines of Fedora Core's /etc/rc.d/init.d/httpd script and put them at the top of /etc/rc.d/init.d/apache_ost_svn, so that apache_ost_svn would be usable by Fedora Core's spiffy chkconfig programme. (Note that chkconfig's parameters in init scripts seem only to be comments, but they are not just comments.) Here are the lines I added just below the "#!/bin/sh" in /etc/rc.d/init.d/apache_ost_svn:

#
# apache_ost_svn        Startup script for the Apache HTTP Server
#
# chkconfig: - 85 15
# description: This Apache installation is really a Subversion repository.
# processname: httpd
# config: /usr/local/apache_ost_svn/conf/httpd.conf

Then, I removed these two stanzas:

stopssl|sslstop|stop-SSL)
    $HTTPD -k stop -DSSL
    ERROR=$?
    ;;
startssl|sslstart|start-SSL)
    $HTTPD -k start -DSSL
    ERROR=$?
    ;;

Next, I changed this stanza:

start|stop|restart|graceful)
    $HTTPD -k $ARGV
    ERROR=$?
    ;;

to this:

start|stop|restart|graceful)
    echo -n $"Apache + SVN $ARGV, status: "
    $HTTPD -k $ARGV -DSSL
    ERROR=$?
    [ "$ERROR" -eq 0 ] && echo "OK" || echo "FAILED"
    ;;

so that the property "SSL" was defined for all four targets captured by that stanza, and so that we get some feedback on the command-line (though not as pretty as Fedora Core's scripts).

Next, I told chkconfig to add apache_ost_svn to its setup, and to make it active at runlevels 3, 4, and 5:

[root@localhost init.d]# chkconfig --add apache_ost_svn
[root@localhost init.d]# chkconfig --list apache_ost_svn
apache_ost_svn      0:off   1:off   2:off   3:off   4:off   5:off   6:off
[root@localhost init.d]# chkconfig --level 345 apache_ost_svn on
[root@localhost init.d]# chkconfig --list apache_ost_svn
apache_ost_svn      0:off   1:off   2:off   3:on    4:on    5:on    6:off

Then I started Apache using its new init script.

[root@localhost init.d]# /etc/rc.d/init.d/apache_ost_svn start
Apache + SVN start, status: OK

Testing the Connection

The moment of truth came when I got to test repository access from my web broser. I went to https://localhost/, accepted the cert, entered one of the sample usernames and passwords, and browsed my Subversion repository.

The command-line client worked too:

[root@localhost init.d]# cd /tmp
[root@localhost tmp]# svn co https://localhost:446/test_project/trunk test_project
Error validating server certificate for 'https://localhost:446':
 - The certificate is not issued by a trusted authority. Use the
   fingerprint to validate the certificate manually!
 - The certificate hostname does not match.
Certificate information:
 - Hostname: Test-Only Certificate
 - Valid: from Mar 12 00:48:54 2005 GMT until Mar 12 00:48:54 2006 GMT
 - Issuer: Test-Only Certificate
 - Fingerprint: 56:f8:be:cc:df:69:c4:64:43:ba:d4:0b:5d:65:a2:0e:9f:9f:5d:ee
(R)eject, accept (t)emporarily or accept (p)ermanently? t
Authentication realm: <https://localhost:446> Subversion Repository
Password for 'root':  # there is no user root, so just hit enter here
Authentication realm: <https://svn1.digitas.com:446> Subversion Repository
Username: user1
Password for 'user1':
A  test_project/test_file.txt
Checked out revision 1.

Get What You Want

Custom compiling Apache and Subversion can certainly be involved, but it's the best way to truly get your repository configured the way you want. Hopefully, this article will make it easier for you to get what you want out of Apache and Subversion.