-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Debugging ModSecurity
This page contains information about how to debug ModSecurity. This is not a complete guide or a developer guide, this contains basic information that may help the bug reporting process. Sometimes reproduce problems are not easy due to platform specific stuff or even configurations. It is even worst when log files does not contains specific reference for what is leading ModSecurity to fail (process is killed before log be saved). In that kind of situation it is recommend the utilization of tools such as GDB, which is able to acquire details on the faulty function.
In this topic it is explained the debugging techniques that are applicable to Nginx and Apache in a Linux environment. Debugging in other platforms are also possible, however, they are not covered by this specific topic. For more information about IIS, visit the IIS Troubleshooting guide.
Debugging ModSecurity means debug the HTTP server. ModSecurity works as a server add-on/plugin/extension, it is part of the same process id (PID) of the server.
Before start the debug process, make sure that ModSecurity is compiled with debug parameter ("-g" for those who are using gcc). Another gcc parameter that is desired is the "-O0", which will disable the compiler optimization, making the GDB's output more friendly. Independent of the target HTTP server, ModSecurity should be compiled with those special flags. Those flags can be placed during the configuration phase, as following:
CFLAGS="-g -O0" ./configure ...normal configure parameters...
It is recommended to keep the debugging process as simple as possible, to do so, eliminate the usage of the multiple threads by the HTTP server is recommended. The instructions on how to disable the threads/workers are dependent on your server, as explained on the sub-sections bellow.
Apache server accepts an special command line parameter: "-X", that starts the server in debug mode and not detach it from console. This flag should be passed straight to apache2 or http binary, along with other options, such as the configuration file that should be used. The parameter should not be passed to the apachectl script, instead, the http/apache2 file should be used directly. If you are using Ubuntu your Apache will probably be at: /usr/sbin/apache2.
This setup may affect the behavior of the HTTP server in a way that makes impossible or more hard to reproduce the bug, if it is the case, ask for help in our mailing list and check out the Apache's debugging instructions at: https://httpd.apache.org/dev/debugging.html.
Different from Apache where a parameter is expected to place the server in debugging mode, Nginx expects debugging information on the configuration files. To not detach Nginx from the console, the user can set daemon to off. Threads can be disabled by having master_process set to off. As demonstrated bellow:
daemon off;
master_process off;
By having this demonstration configuration at Nginx' configuration, it will be executed in one single thread and it won't detach from the console. If the problem is not reproducible under those circumstances other techniques can be used as explained at: http://wiki.nginx.org/Debugging. Besides the debugging parameters that should be passed to gcc, it is also recommend to use the option --with-debug in the Nginx' configure script. The recommend configuration options are illustrated bellow:
CFLAGS="-g -O0" ./configure --with-debug ...normal paramanters...
As defined at GDB's website: "GDB, the GNU Project debugger, allows you to see what is going on `inside' another program while it executes -- or what another program was doing at the moment it crashed." The utilization of GDB is recommended to understand and report ModSecurity crashes. This section are not intend to give an advanced usage guide of GDB, instead, it is just about print the stack trace in a situation where the HTTP server died.
GDB can be attached to process that are already running or it can be attached in the beginning of the execution. In this document the second option will be explained. The way to get GBD "watching" to a process is very simple: the target binary should be passed as a parameter to GDB, for instance:
$ gdb /usr/sbin/apache2
GNU gdb (GDB) 7.6.1-ubuntu
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
...
Notice that this does not started the target software yet. In order to have it running "run" most be typed in the GDB console. Parameters to the target process are expected to be pasts in the GDB console, as parameters of the command: "run", as demonstrated bellow:
(gdb) run -X
Starting program: /usr/sbin/apache2 -X
[Thread debugging using libthread_db enabled]
...
After really start the process (after execute the "run" command), the HTTP server should be working as if there is no GDB. The next step is try to reproduce the bug. Once the bug is reproduced the process will crash and so, the GDB command "bt full" can be used to extract information about the crash. Demonstration of all GDB steps can be found bellow:
(zimmerle@zlinux)-(~/core/spider-modsec/tests)$ gdb /usr/sbin/apache2
GNU gdb (GDB) 7.6.1-ubuntu
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/apache2...(no debugging symbols found)...done.
(gdb) run -d -X -f /home/zimmerle/core/spider-modsec/tests/regression/server_root/conf/httpd.conf -c 'Listen localhost:8088' -c 'Include /home/zimmerle/core/spider-modsec/tests/regression/server_root/conf/config_10-request-directives.t_000003.conf' -k start
Starting program: /usr/sbin/apache2 -d -X -f /home/zimmerle/core/spider-modsec/tests/regression/server_root/conf/httpd.conf -c 'Listen localhost:8088' -c 'Include /home/zimmerle/core/spider-modsec/tests/regression/server_root/conf/config_10-request-directives.t_000003.conf' -k start
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Program received signal SIGABRT, Aborted.
0x00007ffff7196f77 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt full
#0 0x00007ffff7196f77 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
resultvar = 0
pid = 62396
selftid = 62396
#1 0x00007ffff719a5e8 in __GI_abort () at abort.c:90
save_stage = 2
act = {__sigaction_handler = {sa_handler = 0x7ffff7163d18, sa_sigaction = 0x7ffff7163d18}, sa_mask = {__val = {0, 140737353784168, 140737338818848, 140737488346464, 140737339657820, 0, 6, 28, 0, 140737354072104, 140737353784208, 140737353784216, 140737345047185,
1, 140733193388080, 140737488346216}}, sa_flags = -9232, sa_restorer = 0x12}
sigs = {__val = {32, 0 <repeats 15 times>}}
#2 0x00007ffff603eac0 in msc_escape (string=0x7ffff7faa490 "phase:2,deny,chain,id:500219", buff=0x0, buff_length=0, opt=20) at msc_escape.c:133
i = 28
length_only = 1
escaped_string_length = 28
#3 0x00007ffff605250f in _log_escape (mp=0x7ffff7ff0028, input=0x7ffff7faa490 "phase:2,deny,chain,id:500219", input_len=28, escape_quotes=1, escape_colon=0, escape_re=0) at msc_util.c:1335
length = 0
res = 0
options = 20
buffer = 0x0
#4 0x00007ffff605209a in log_escape (mp=0x7ffff7ff0028, text=0x7ffff7faa470 "phase:2,deny,chain,id:500219") at msc_util.c:1210
No locals.
#5 0x00007ffff6064651 in msre_rule_generate_unparsed (pool=0x7ffff7ff0028, rule=0x7ffff7faa088, targets=0x7ffff7fa9b90 "ARGS:a", args=0x7ffff7fa9b98 "@streq 1", actions=0x0) at re.c:2240
unparsed = 0x0
r_targets = 0x7ffff7fa9b90 "ARGS:a"
r_args = 0x7ffff7fa9b98 "@streq 1"
r_actions = 0x7ffff7faa470 "phase:2,deny,chain,id:500219"
#6 0x00007ffff6064b41 in msre_rule_create (ruleset=0x7ffff7fa9bc8, type=0, fn=0x7ffff7facdc0 "/home/zimmerle/core/spider-modsec/tests/regression/server_root/conf/config_10-request-directives.t_000003.conf", line=5, targets=0x7ffff7fa9b90 "ARGS:a",
args=0x7ffff7fa9b98 "@streq 1", actions=0x7ffff7fa9ba8 "phase:2,deny,chain,id:500219", error_msg=0x7fffffffdde8) at re.c:2353
rule = 0x7ffff7faa088
my_error_msg = 0x0
argsp = 0x7ffff7fa9b98 "@streq 1"
rc = 1
#7 0x00007ffff602737c in add_rule (cmd=0x7fffffffe140, dcfg=0x7ffff7fb0810, type=0, p1=0x7ffff7fa9b90 "ARGS:a", p2=0x7ffff7fa9b98 "@streq 1", p3=0x7ffff7fa9ba8 "phase:2,deny,chain,id:500219") at apache2_config.c:759
my_error_msg = 0x0
rid = 0x0
rule = 0x0
offset = 0
#8 0x00007ffff6029ca9 in cmd_rule (cmd=0x7fffffffe140, _dcfg=0x7ffff7fb0810, p1=0x7ffff7fa9b90 "ARGS:a", p2=0x7ffff7fa9b98 "@streq 1", p3=0x7ffff7fa9ba8 "phase:2,deny,chain,id:500219") at apache2_config.c:2043
No locals.
#9 0x00005555555a7083 in ?? ()
No symbol table info available.
#10 0x00005555555a96ff in ap_walk_config ()
No symbol table info available.
#11 0x00005555555aa859 in ap_process_config_tree ()
No symbol table info available.
#12 0x00005555555888d6 in main ()
No symbol table info available.
(gdb)