25.3.1.1. Two methods for manipulating data

Even though you don't have to work with bucket brigades directly, since you can write filters using the simplified, streaming filter interface (which works with bucket brigades behind the scenes), it's still important to understand bucket brigades. For example, you need to know that an output filter will be invoked as many times as the number of bucket brigades sent from an upstream filter or a content handler, and that the end-of-stream indicator (EOS) is sometimes sent in a separate bucket brigade, so it shouldn't be a surprise if the filter is invoked even though no real data went through.

You will also need to understand how to manipulate bucket brigades if you plan to implement protocol modules, as you have seen earlier in this chapter.

25.3.1.2. HTTP request versus connection filters

HTTP request input filters get invoked on the body of the HTTP request only if the body is consumed by the content handler. HTTP request headers are not passed through the HTTP request input filters.

HTTP response output filters get invoked on the body of the HTTP response, if the content handler has generated one. HTTP response headers are not passed through the HTTP response output filters.

Connection-level filters are applied at the connection level.

A connection may be configured to serve one or more HTTP requests, or handle other protocols. Connection filters see all the incoming and outgoing data. If an HTTP request is served, connection filters can modify the HTTP headers and the body of the request and response. Of course, if a different protocol is served over the connection (e.g., IMAP), the data could have a completely different pattern than the HTTP protocol (headers and body).

Apache supports several other filter types that mod_perl 2.0 may support in the future.

25.3.1.3. Multiple invocations of filter handlers

For example, if a content-generation handler sends a string, and then forces a flush, following with more data:

# assuming buffered STDOUT ($|=  =0)
$r->print("foo");
$r->rflush;
$r->print("bar");

bucket type       data
----------------------
1st    transient   foo
2nd    flush

and send it to the filter chain. Then, assuming that no more data was sent after print("bar"), it will create a last bucket brigade containing data:

bucket type       data
----------------------
1st    transient   bar

and send it to the filter chain. Finally it'll send yet another bucket brigade with the EOS bucket indicating that no more will be data sent:

bucket type       data
----------------------
1st    eos

In our example the filter will be invoked three times. Notice that sometimes the EOS bucket comes attached to the last bucket brigade with data and sometimes in its own bucket brigade. This should be transparent to the filter logic, as we will see shortly.

Figure 25-4. mod_perl 2.0 filter logic

Jumping ahead, we will show some pseudocode that represents all three parts. This is what a typical filter looks like:

sub handler {
    my $filter = shift;

    # runs on first invocation
    unless ($filter->ctx) {
        init($filter);
        $filter->ctx(1);
    }

    # runs on all invocations
    process($filter);

    # runs on the last invocation
    if ($filter->seen_eos) {
        finalize($filter);
    }

    return Apache::OK;
}
sub init     { ... }
sub process  { ... }
sub finalize { ... }

Let's examine the parts of this pseudofilter:

25.3.1.4. Blocking calls

Input and output filters differ in the ways they acquire the bucket brigades (which include the data that they filter). Although the difference can't be seen when a streaming API is used, it's important to understand how things work underneath.

When an input filter is invoked, it first asks the upstream filter for the next bucket brigade (using the get_brigade( ) call). That upstream filter in turn asks for the bucket brigade from the next upstream filter in the chain, and so on, until the last filter that reads from the network (called core_in) is reached. The core_in filter reads, using a socket, a portion of the incoming data from the network, processes it, and sends it to its downstream filter, which processes the data and sends it to its downstream filter, and so on, until it reaches the very first filter that asked for the data. (In reality, some other handler triggers the request for the bucket brigade (e.g., the HTTP response handler or a protocol module), but for our discussion it's good enough to assume that it's the first filter that issues the get_brigade( ) call.)

Figure 25-5. mod_perl 2.0 input filter program control and data flow

As mentioned earlier, the streaming interface hides these details; however, the first call to $filter->read( ) will block, as underneath it performs the get_brigade( ) call.

Figure 25-6. mod_perl 2.0 output filter program control and data flow

25.3.3.1. Bucket brigade-based connection input filter

This example's filter handler looks for data like:

GET /perl/test.pl HTTP/1.1

and turns it into:

HEAD /perl/test.pl HTTP/1.1

• Create a new, empty bucket brigade, $ctx_bb, pass it to the upstream filter via get_brigade( ), and wait for this call to return. When it returns, $ctx_bb is populated with buckets. Now the filter should move the bucket from $ctx_bb to $bb, on the way modifying the buckets if needed. Once the buckets are moved, and the filter returns, the downstream filter will receive the populated bucket brigade.

• Pass $bb to get_brigade( ) to the upstream filter, so it will be populated with buckets. Once get_brigade( ) returns, the filter can go through the buckets and modify them in place, or it can do nothing and just return (in which case, the downstream filter will receive the bucket brigade unmodified).

Both techniques allow addition and removal of buckets, alhough the second technique is more efficient since it doesn't have the overhead of creating the new brigade and moving the bucket from one brigade to another. In this example we have chosen to use the second technique; in the next example we will see the first technique.

Our filter has to perform the substitution of only one HTTP header (which normally resides in one bucket), so we have to make sure that no other data gets mangled (e.g., there could be POST ed data that may match /^GET/ in one of the buckets). We use $filter->ctx as a flag here. When it's undefined, the filter knows that it hasn't done the required substitution; once it completes the job, it sets the context to 1.

To optimize the speed, the filter immediately returns Apache::DECLINED when it's invoked after the substitution job has been done:

return Apache::DECLINED if $filter->ctx;

mod_perl then calls get_brigade( ) internally, which passes the bucket brigade to the downstream filter. Alternatively, the filter could do:

my $rv = $filter->next->get_brigade($bb, $mode, $block, $readbytes);
return $rv unless $rv =  = APR::SUCCESS;
return Apache::OK if $filter->ctx;

but this is a bit less efficient.

If the job hasn't yet been done, the filter calls get_brigade( ), which populates the $bb bucket brigade. Next, the filter steps through the buckets, looking for the bucket that matches the regex /^GET/. If it finds it, a new bucket is created with the modified data s/^GET/HEAD/, and that bucket is inserted in place of the old bucket. In our example, we insert the new bucket after the bucket that we have just modified and immediately remove the bucket that we don't need any more:

$b->insert_after($bn);
$b->remove; # no longer needed

Finally, we set the context to 1, so we know not to apply the substitution on the following data and break from the for loop.

The handler returns Apache::OK, indicating that everything was fine. The downstream filter will receive the bucket brigade with one bucket modified.

This handler returns to the client the request type it has issued. In the case of the HEAD request, Apache will discard the response body, but it will still set the correct Content-Length header, which will be 24 in case of a GET request and 25 for HEAD. Therefore, if this response handler is configured as:

Listen 8005
<VirtualHost _default_:8005>
    <Location />
        SetHandler modperl
        PerlResponseHandler +Book::RequestType
    </Location>
</VirtualHost>

and a GET request is issued to /:

panic% perl -MLWP::UserAgent -le \
'$r = LWP::UserAgent->new( )->get("https://localhost:8005/"); \
print $r->headers->content_length . ": ".  $r->content'
24: the request type was GET

the response's body is:

the request type was GET

and the Content-Length header is set to 24.

However, if we enable the Book::InputFilterGET2HEAD input connection filter:

Listen 8005
<VirtualHost _default_:8005>
    PerlInputFilterHandler +Book::InputFilterGET2HEAD

    <Location />
        SetHandler modperl
        PerlResponseHandler +Book::RequestType
    </Location>
</VirtualHost>

and issue the same GET request, we get only:

25:

which means that the body was discarded by Apache, because our filter turned the GET request into a HEAD request. If Apache wasn't discarding the body of responses to HEAD requests, the response would be:

the request type was HEAD

25.3.3.2. Bucket brigade-based HTTP request input filter

As promised, in this filter handler we have used the first technique of bucket-brigade modification. The handler creates a temporary bucket brigade (ctx_bb), populates it with data using get_brigade( ), and then moves buckets from it to the bucket brigade $bb, which is then retrieved by the downstream filter when our handler returns.

This filter doesn't need to know whether it was invoked for the first time with this request or whether it has already done something. It's a stateless handler, since it has to lowercase everything that passes through it. Notice that this filter can't be used as a connection filter for HTTP requests, since it will invalidate the incoming request headers. For example, the first header line:

GET /perl/TEST.pl HTTP/1.1

will become:

get /perl/test.pl http/1.1

which messes up the request method, the URL, and the protocol.

Now if we use the Book::Dump response handler we developed earlier in this chapter, which dumps the query string and the content body as a response, and configure the server as follows:

<Location /lc_input>
    SetHandler modperl
    PerlResponseHandler    +Book::Dump
    PerlInputFilterHandler +Book::InputRequestFilterLC
</Location>

when issuing a POST request:

panic% echo "mOd_pErl RuLeS" | POST 'https://localhost:8002/lc_input?FoO=1&BAR=2'

we get a response like this:

args:
FoO=1&BAR=2
content:
mod_perl rules

25.3.3.3. Stream-based HTTP request input filter

You've probably asked yourself why we had to go through the bucket-brigade filters when all this can be done so much more easily. The reason is that we wanted you to understand how the filters work underneath, which will help you when you need to debug filters or optimize their speed. Also, in certain cases a bucket-brigade filter may be more efficient than a stream-based one. For example, if the filter applies a transformation to selected buckets, certain buckets may contain open file handles or pipes, rather than real data. When you call read( ) the buckets will be forced to read in that data, but if you don't want to modify these buckets, you can pass them as they are and let Apache use a faster technique for sending data from the file handles or pipes.

The logic is very simple here: the filter reads in a loop and prints the modified data, which at some point (when the internal mod_perl buffer is full or when the filter returns) will be sent to the next filter.

read( ) populates $buffer to a maximum of BUFF_LEN characters (1,024 in our example). Assuming that the current bucket brigade contains 2,050 characters, read( ) will get the first 1,024 characters, then 1,024 characters more, and finally the remaining two characters. Notice that even though the response handler may have sent more than 2,050 characters, every filter invocation operates on a single bucket brigade, so you have to wait for the next invocation to get more input. In one of the earlier examples, we showed that you can force the generation of several bucket brigades in the content handler by using rflush( ). For example:

$r->print("string");
$r->rflush( );
$r->print("another string");

It's possible to get more than one bucket brigade from the same filter handler invocation only if the filter is not using the streaming interface—simply call get_brigade( ) as many times as needed or until the EOS token is received.

The configuration section is pretty much identical:

<Location /lc_input2>
     SetHandler modperl
     PerlResponseHandler    +Book::Dump
     PerlInputFilterHandler +Book::InputRequestFilterLC2
 </Location>

When issuing a POST request:

% echo "mOd_pErl RuLeS" | POST 'https://localhost:8002/lc_input2?FoO=1&BAR=2'

we get a response like this:

args:
FoO=1&BAR=2
content:
mod_perl rules

25.3.4.1. Stream-based HTTP request output filter

The example of a stream-based output filter that we are going to present is simpler than the one that directly manipulates bucket brigades, although internally the stream-based interface is still manipulating the bucket brigades.

Book::FilterROT13 implements the simple Caesar-cypher encryption that replaces each English letter with the one 13 places forward or back along the alphabet, so that "mod_perl 2.0 rules!" becomes "zbq_crey 2.0 ehyrf!". Since the English alphabet consists of 26 letters, the ROT13 encryption is self-inverse, so the same code can be used for encoding and decoding. In our example, Book::FilterROT13 reads portions of the output generated by some previous handler, rotates the characters and sends them downstream.

Let's say that we want to encrypt the output of the registry scripts accessed through a /perl-rot13 location using the ROT13 algorithm. The following configuration section accomplishes that:

PerlModule Book::FilterROT13
Alias /perl-rot13/ /home/httpd/perl/
<Location /perl-rot13>
    SetHandler perl-script
    PerlResponseHandler ModPerl::Registry
    PerlOutputFilterHandler Book::FilterROT13
    Options +ExecCGI
    #PerlOptions +ParseHeaders
</Location>

Now that you know how to write input and output filters, you can write a pair of filters that decode ROT13 input before the request processing starts and then encode the generated response back to ROT13 on the way back to the client.

The request output filter can be used as the connection output filter as well. However, HTTP headers will then look invalid to standard HTTP user agents. The client should expect the data to come encoded as ROT13 and decode it before using it. Writing such a client in Perl should be a trivial task.

25.3.4.2. Another stream-based HTTP request output filter

The purpose of our filter handler is to reverse every line of the response body, preserving the newline characters in their places. Since we want to reverse characters only in the response body, without breaking the HTTP headers, we will use an HTTP request output filter.

Next, we add the following configuration to httpd.conf:

PerlModule Book::FilterReverse1
PerlModule Book::SendAlphaNum
<Location /reverse1>
    SetHandler modperl
    PerlResponseHandler     Book::SendAlphaNum
    PerlOutputFilterHandler Book::FilterReverse1
</Location>

Now when a request to /reverse1 is made, the response handler Book::SendAlphaNum::handler( ) sends:

1234567890
abcdefghijklmnopqrstuvwxyz

as a response and the output filter handler Book::FilterReverse1::handler reverses the lines, so the client gets:

0987654321
zyxwvutsrqponmlkjihgfedcba

The Apache::Filter module loads the read( ) and print( ) methods that encapsulate the stream-based filtering interface.

The reversing filter is quite simple: in the loop it reads the data in the readline( ) mode in chunks up to the buffer length (1,024 in our example), then it prints each line reversed while preserving the newline control characters at the end of each line. Behind the scenes, $filter->read( ) retrieves the incoming brigade and gets the data from it, and $filter->print( ) appends to the new brigade, which is then sent to the next filter in the stack. read( ) breaks the while loop when the brigade is emptied or the EOS token is received.

So as not to distract the reader from the purpose of the example, we've used oversimplified code that won't correctly handle input lines that are longer than 1,024 characters or use a different line-termination token (it could be "\n", "\r", or "\r\n", depending on the platform). Moreover, a single line may be split across two or even more bucket brigades, so we have to store the unprocessed string in the filter context so that it can be used in the following invocations. So here is an example of a more complete handler, which does takes care of these issues:

sub handler {
    my $f = shift;

    my $leftover = $f->ctx;
    while ($f->read(my $buffer, BUFF_LEN)) {
        $buffer = $leftover . $buffer if defined $leftover;
        $leftover = undef;
        while ($buffer =~ /([^\r\n]*)([\r\n]*)/g) {
            $leftover = $1, last unless $2;
            $f->print(scalar(reverse $1), $2);
        }
    }

    if ($f->seen_eos) {
        $f->print(scalar reverse $leftover) if defined $leftover;
    }
    else {
        $f->ctx($leftover) if defined $leftover;
    }

    return Apache::OK;
}

25.3.4.3. Bucket brigade-based HTTP request output filter

Here's the corresponding configuration:

PerlModule Book::FilterReverse2
PerlModule Book::SendAlphaNum
<Location /reverse2>
    SetHandler modperl
    PerlResponseHandler     Book::SendAlphaNum
    PerlOutputFilterHandler Book::FilterReverse2
</Location>

Now when a request to /reverse2 is made, the client gets:

0987654321
zyxwvutsrqponmlkjihgfedcba

as expected.

The bucket brigades output filter version is just a bit more complicated than the stream-based one. The handler receives the incoming bucket brigade $bb as its second argument. Because when it is completed, the handler must pass a brigade to the next filter in the stack, we create a new bucket brigade, into which we put the modified buckets and which eventually we pass to the next filter.

The core of the handler is in removing buckets from the head of the bucket brigade $bb one at a time, reading the data from each bucket, reversing the data, and then putting it into a newly created bucket, which is inserted at the end of the new bucket brigade. If we see a bucket that designates the end of the stream, we insert that bucket at the tail of the new bucket brigade and break the loop. Finally, we pass the created brigade with modified data to the next filter and return.