There are quite a few things that can be done with bash to help debugging your
scripts containing the rulesets. One of the first problems with finding a bug
is to know on which line the problem appears. This can be solved in two
different ways, either using the bash -x flag, or by simply entering some
echo statements to find the place where the problem happens.
Ideally, you would, with the echo statement, add something
like the following echo statement at regular intervals in the code:
...
echo "Debugging message 1."
...
echo "Debugging message 2."
...
In my case, I generally use pretty much worthless messages, as long as they have
something in them that is unique so I can find the error message by a simple
grep or search in the script file. Now, if the error message shows up after the
"Debugging message 1." message, but before "Debugging message 2.", then we know
that the erroneous line of code is somewhere in between the two debugging
messages. As you can understand, bash has the not really bad, but at least
peculiar idea of continuing to execute commands even if there is an error in one
of the commands before. In netfilter, this can cause some very interesting
problems for you. The above idea of simply using echo statements to find the
errors is extremely simple, but it is at the same time very nice since you can
narrow the whole problem down to a single line of code and see what the problem
is directly.
The second possibility to find the above problem is to use the -x variable to
bash, as we spoke of before. This can of course be a little problem, especially
if your script is large, and if your console buffer isn't large enough. What
the -x variable means is quite simple, it tells the script to just echo every
single line of code in the script into the standard output of the shell
(generally your console). What you do is to change your normal start line of
the script from this:
#!/bin/bash
Into the line below:
#!/bin/bash -x
As you will see, this changes your output from perhaps a couple of lines, to
copious amounts of data on the output. The code shows you every single command
line that is executed, and with the values of all the variables et cetera, so
that you don't have to try and figure out exactly what the code is doing.
Simply put, each line that gets executed is output to your screen as well. One
thing that may be nice to see, is that all of the lines that bash outputs are
prefixed by a + sign. This makes it a little bit easier to discern error or
warning messages from the actual script, rather than just one big mesh of
output.
The -x option is also very interesting for debugging a couple of other rather
common problems that you may run into with a little bit more complex rulesets.
The first of them is to find out exactly what happens with what you thought was
a simple loop, such as an for, if or while statement? For example, let's look
at an example.
#!/bin/bash
iptables="/sbin/iptables"
$iptables -N output_int_iface
cat /etc/configs/machines | while read host; do
$iptables -N output-$host
$iptables -A output_int_iface -p tcp -d $host -j output-$host
cat /etc/configs/${host}/ports | while read row2; do
$iptables -A output-$host -p tcp --dport $row2 -d $host -j ACCEPT
done
done
This set of rules may look simple enough, but we continue to run into a problem
with it. We get the following error messages that we know come from the above
code by using the simple echo debugging method.
work3:~# ./test.sh
Bad argument `output-'
Try `iptables -h' or 'iptables --help' for more information.
cat: /etc/configs//ports: No such file or directory
So we turn on the -x option to bash and look at the output. The output is shown
below, and as you can see there is something very weird going on in it. There
are a couple of commands where the $host and $row2 variables are replaced by
nothing. Looking closer, we see that it is only the last iteration of code
that causes the trouble. Either we have done a programmatical error, or there
is something strange with the data. In this case, it is a simple error with the
data, which contains a single extra linebreak at the end of the file. This
causes the loop to iterate one last time, which it shouldn't. Simply remove the
trailing linebreak of the file, and the problem is solved. This may not be a
very elegant solution, but for private work it should be enough. Otherwise, you
could add code that looks to see that there is actually some data in the $host
and $row2 variables.
work3:~# ./test.sh
+ iptables=/sbin/iptables
+ /sbin/iptables -N output_int_iface
+ cat /etc/configs/machines
+ read host
+ /sbin/iptables -N output-sto-as-101
+ /sbin/iptables -A output_int_iface -p tcp -d sto-as-101 -j output-sto-as-101
+ cat /etc/configs/sto-as-101/ports
+ read row2
+ /sbin/iptables -A output-sto-as-101 -p tcp --dport 21 -d sto-as-101 -j ACCEPT
+ read row2
+ /sbin/iptables -A output-sto-as-101 -p tcp --dport 22 -d sto-as-101 -j ACCEPT
+ read row2
+ /sbin/iptables -A output-sto-as-101 -p tcp --dport 23 -d sto-as-101 -j ACCEPT
+ read row2
+ read host
+ /sbin/iptables -N output-sto-as-102
+ /sbin/iptables -A output_int_iface -p tcp -d sto-as-102 -j output-sto-as-102
+ cat /etc/configs/sto-as-102/ports
+ read row2
+ /sbin/iptables -A output-sto-as-102 -p tcp --dport 21 -d sto-as-102 -j ACCEPT
+ read row2
+ /sbin/iptables -A output-sto-as-102 -p tcp --dport 22 -d sto-as-102 -j ACCEPT
+ read row2
+ /sbin/iptables -A output-sto-as-102 -p tcp --dport 23 -d sto-as-102 -j ACCEPT
+ read row2
+ read host
+ /sbin/iptables -N output-sto-as-103
+ /sbin/iptables -A output_int_iface -p tcp -d sto-as-103 -j output-sto-as-103
+ cat /etc/configs/sto-as-103/ports
+ read row2
+ /sbin/iptables -A output-sto-as-103 -p tcp --dport 21 -d sto-as-103 -j ACCEPT
+ read row2
+ /sbin/iptables -A output-sto-as-103 -p tcp --dport 22 -d sto-as-103 -j ACCEPT
+ read row2
+ /sbin/iptables -A output-sto-as-103 -p tcp --dport 23 -d sto-as-103 -j ACCEPT
+ read row2
+ read host
+ /sbin/iptables -N output-
+ /sbin/iptables -A output_int_iface -p tcp -d -j output-
Bad argument `output-'
Try `iptables -h' or 'iptables --help' for more information.
+ cat /etc/configs//ports
cat: /etc/configs//ports: No such file or directory
+ read row2
+ read host
The third and final problem you run into that can be partially solved with the
help of the -x option is if you are executing the firewall script via SSH, and
the console hangs in the middle of executing the script, and the console simply
won't come back, nor are you able to connect via SSH again. In 99.9% of the
cases, this means there is some kind of problem inside the script with a couple
of the rules. By turning on the -x option, you will see exactly at which line
the script locks dead, hopefully at least. There are a couple of circumstances
where this is not true, unfortunately. For example, what if the script sets up
a rule that blocks incoming traffic, but since the ssh/telnet server sends the
echo first as outgoing traffic, netfilter will remember the connection, and
hence allow the incoming traffic anyways if you have a rule above that handles
connection states.
As you can see, it can become quite complex to debug your ruleset to its full
extent in the end. However, it is not impossible at all. You may also have
noticed, if you have worked remotely on your firewalls via SSH, for example, that the firewall may hang when you load bad rulesets. There is one more thing
that can be done to save the day in these circumstances. Cron is an excellent
way of saving your day. For example, say you are working on a firewall 50
kilometers away, you add some rules, delete some others, and then delete and
insert the new updated ruleset. The firewall locks dead, and you can't reach
it. The only way of fixing this is to go to the firewall's physical location
and fix the problem from there, unless you have taken precautions that is!
