Intro

Recently, I reported CVE-2017-7668 (Apache Server buffer-over-read). This is a cross-post from my personal blog where I explain how to fuzz network programs with AFL by porting techniques learned in honggfuzz into AFL. After a small chat with Dominic he asked me to re-post it here which, for me it’s an honour to do so!

The reported CVE was obtained with code analysis and instrumentation of the right parts of the code (mainly core and parsing) – First, with honggfuzz I got the initial dirty test cases and then, through radamsa generated a few thousands mutations and finally AFL with the technique described here.

Goal

When stumbling upon the great American Fuzzy Lop and trying its awesome deterministic fuzzing capabilities and instrumentation soon we find out that this fuzzer was built to fuzz programs that take input from the command line (or standard input) instead of a network socket. Because of that, I thought I would give it a try and make AFL fuzz against Apache’s httpd server. First the AFL way by adding a new option to Apache’s command line and the second way, by using the persistence fuzzing (afl-clang-fast) by shamelessly copying the way Robert Swiecky fuzzes Apache with honggfuzz.

Takeaways for the reader

• Learn to fuzz network based programs with AFL
• Code to start fuzzing Apache with AFL in no time
• A push in the interwebz fuzzing race

Let’s do it!

Setup part 1

I will be using a Debian GNU/Linux 8 64bit with the kernel 4.9.0-0.bpo.2-rt-amd64. You don’t really need that setup. All that you need is an operating system (under a virtual machine or not) that can compile and run AFL with the afl-clang feature.

But, before getting into any compilations/installations/fuzzing, I encourage you to set an organised folder structure that suits you best but, in case you haven’t got one already, I am sharing mine.
Under the Fuzzing folder I have:

• Victims – For the target programs that we are about to fuzz
• Fuzzers – AFL, honggfuzz, radamsa, etc. go here
• Testcases  – The samples we are going to feed the fuzzer to throw against our Victim
• Sessions – For storing the fuzzing sessions
• Compilers – To store compilers such as clang-4.0 and binaries needed to compile

Getting clang-4.0 and llvm-tools

Getting pre-built binaries for clang-4.0 and the llvm-tools is fairly easy if you have Debian or Ubuntu. You can get these from here http://releases.llvm.org/download.html. In my case the clang+llvm-4.0.0-x86_64-linux-gnu-debian8.tar.xz tarball.
If you are following the structure mentioned above, you can cd into your Compilers folder and drop the tarball there, extract it and then add the binaries folder to your path by adding the following line to the end of your ~/.bashrc file (~/.profile nor /etc/environment worked for me – It seems that you need to logout and login for these changes to take place).

PATH="$HOME/Fuzzing/Compilers/clang+llvm-4.0.0-x86_64-linux-gnu-debian8/bin:$PATH"

Now issuing the which command on a new shell we should have the following output:

$ which clang
$ /home/javier/Fuzzing/Compilers/llvm-clang-binaries/clang+llvm-4.0.0-x86_64-linux-gnu-debian8/bin/clang

Compiling and Installing AFL

Compiling AFL should be pretty straight forward but, for the lazy, you can just copy paste these commands and you should be ready to go:

sudo apt install build-essential
wget http://lcamtuf.coredump.cx/afl/releases/afl-latest.tgz
tar xzf afl-latest.tgz
cd afl*
make && sudo make install && echo-e "\n[+] AFL ready to fuzz at $(which afl-fuzz)"

That’s it, the binary afl-fuzz should be in your path now ready to be unleashed.

Compiling and installing Apache

First move to a folder where we are about to download Apache server and all the dependencies needed. In my case, the folder is at ~/Fuzzing/Victims/apache_afl/.

Before compiling Apache we are going to need the Apache Portable Runtime (APR), APR Utils and support for HTTP/2 through nghttp2.
No lazyness this time, go download:

• APR and APR Util
• nghttp
• The following dependencies:
  • sudo apt install pkg-config
  • sudo apt install libssl-dev
• The latest pcre-8.x (DON’T DOWNLOAD PCRE2!!)

Now we need to get the latest Apache build, which I recommend you do from their subversion repository by doing so:

sudo apt install subversion
svn checkout http://svn.apache.org/repos/asf/httpd/httpd/branches/2.4.x httpd-2.4.x

Now if you downloaded and unpacked everything, you should have a similiar output from ls -l command:

drwxr-xr-x 28 javier javier  4096 Apr  9 23:12 apr-1.5.2
drwxr-xr-x 20 javier javier  4096 Apr  9 23:13 apr-util-1.5.4
-rwxr-xr-x  1 javier javier  1445 May  6 20:56 compile_dependencies_with_flags.sh
drwxr-xr-x 11 javier javier  4096 Apr 29 01:22 httpd-2.4.x
drwxr-xr-x 14 javier javier  4096 Apr  9 23:14 nghttp2-1.21.0
drwxr-xr-x  9 javier javier 12288 Apr  9 22:53 pcre-8.40

I made the following script to compile and link it all since I found myself often changing flags for the compiler and it was too time consuming compiling each dependency one by one with its own flags. Get it with:

wget https://gist.githubusercontent.com/n30m1nd/14418fd425a3b2d14b64650710fae301/raw/e1cff738eb1ffaa55cb8a1a66bb1a2b06ed7f97e/compile_httpd_with_flags.sh

Before editing any files yet lets run the bash script and see that we can compile everything cleanly without any missing dependencies whatsoever:

CC="clang" CXX="clang++" PREFIX="/usr/local/apache_clean_test/" ./compile_dependencies_with_flags.sh

Please see the next asciinema for reference of a nice compilation run (takes a while to load).

Fuzzing Apache with AFL through an input file

As you might know by now, AFL in its basic usage feeds a file into the target program through its “argv” array in the following form:

afl-fuzz -i testcases/ -o session_1/ -- ./victim -v -f @@

The problem with Apache is that it doesn’t have such functionality so we will have to patch it our own way.

Patching Apache

Taking into account the aforementioned problem, we need to write some lines into Apache’s main.c file to make it able to read files from input.
You can patch Apache with the following patch file here. Now apply it by cd‘ing into the base path of Apache httpd’s source code and issuing the following command:

patch -p0 -i apatching_apache_for_AFL_fuzzing.diff

I am not going to cover all the patch in detail but some parts are worth mentioning.

The first and only time that I have seen the following technique was by Robert Swiecky, an information security researcher at Google when fuzzing Apache with honggfuzz. It is pretty clever and pretty obvious once you see the way it is done. It basically consists of launching a new thread inside Apache that will create a connection to the web server itself and send our fuzzed input;  all happening within the same unique process so we can get all the instrumentation data into AFL. Clever! Right?
To achieve this it uses the unshare function that disassociates parts of this thread’s context from the others without the need of creating a new process. Specifically, the network and  mount namespaces are separated. This is done so we can have several processes with the same settings (listening on the same loopback interface and port with the help of netIfaceUp on line 44 of the patch file and writing logs to /tmp on line 75) running at the same time on each process we launch.

We can see that the unshare function is indirectly called on line 188 previous to firing the new thread that will receive the fuzzed input at line 189.

The process of reading a file through the “-F” switch starts on line 156 and when the file is read into a buffer, this buffer is passed onto the function responsible to launch the new thread (189) that will, in turn, send the fuzzed file inside the SENDFILE function on line 119.

Fuzzing Apache

Yes! We are ready now! Let’s compile Apache:

CC="afl-clang" CXX="afl-clang++" PREFIX="/usr/local/apache_afl_blogpost/" ./compile_dependencies_with_flags.sh

If you are familiar with AFL and how it works you probably have your own testcases to feed it with, in case you don’t the following video shows how to launch AFL and create two very simple test cases – remember that we need to be root or use “sudo sysctl -w kernel.unprivileged_userns_clone=1” in order to use the unshare function AND MORE IMPORTANT TO LAUNCH APACHE WITH THE “-X” FLAG AND “-m none -t 5000” FOR AFL SO IT CAN BOOT APACHE:
Well, that was not too fast, was it? 5 execs per second on my laptop… how can we speed things up a bit?

Setup part 2

Compiling afl-clang-fast

Remember we downloaded clang-4.0 and the llvm-tools before and set it in our path? This is where it comes most handy. Inside your AFL folder, navigate to the llvm_mode and run make and sudo make install in the root folder of AFL. What we have just done is compiling an experimental feature of AFL that will run a certain number of fuzzed inputs against a program without having to run the whole program per fuzzing input.

Patching Apache for Persistence

Following the same dance as before, download this patch, patch it and ready to fuzz!

Fuzzing Apache with AFL on Persistent mode

Let the video speak for itself but again remember the previously mentioned “-X” flag for Apache server and the “-m none -t 5000” flags for AFL:

Update:

As pointed by Robert himself, you don’t need to run everything as root as I did in the examples which, obviously imposes security risks. You can make use of the following command line (this one to be run as root :P) to let non-root users make use of the unshare function:

patch -p0 -i apatching_apache_for_AFL_fuzzing.diff

Conclusions

We have learned how to effectively fuzz server programs such as web servers by using Robert’s technique of launching a different thread and different context through the unshare function.
While relatively fast, it is not as fast as honggfuzz, which can go up to 20k iterations per second with 8 processes running. Also after a few days of fuzzing AFL’s stability goes way down 50% because of the multithreading that Apache is implemented on and so, any reported crashes, can be false positives or would either be needing the last iterations launched which AFL lacks of.

It is left as an exercise to the reader to implement into Apache a way to save the last 1k sent inputs into a file and to think which other ways would improve stability and/or speed. Hint: Don’t instrument everything.

Shouts to the Apache Security Team, it was a pleasure to work with such an efficient team.

Happy pwning! x)

For further doubts, ramblings or whatever you can contact me on javier at domain’s name