The following sections describe Postfix queues: their purpose,
what normal behavior looks like, and how to diagnose abnormal
behavior.
Messages that have been submitted via the Postfix
sendmail(1)
command, but not yet brought into the main Postfix queue by the
pickup(8) service, await processing in the "
maildrop" queue. Messages
can be added to the "
maildrop" queue even when the Postfix system
is not running. They will begin to be processed once Postfix is
started.
The "
maildrop" queue is drained by the single threaded
pickup(8)
service scanning the queue directory periodically or when notified
of new message arrival by the
postdrop(1) program. The
postdrop(1)
program is a setgid helper that allows the unprivileged Postfix
sendmail(1) program to inject mail into the "
maildrop" queue and
to notify the
pickup(8) service of its arrival.
All mail that enters the main Postfix queue does so via the
cleanup(8) service. The cleanup service is responsible for envelope
and header rewriting, header and body regular expression checks,
automatic bcc recipient processing and guaranteed insertion of the
message into the Postfix "
incoming" queue.
In the absence of excessive CPU consumption in
cleanup(8) header
or body regular expression checks or other software consuming all
available CPU resources, Postfix performance is disk I/O bound.
The rate at which the
pickup(8) service can inject messages into
the queue is largely determined by disk access times, since the
cleanup(8) service must commit the message to stable storage before
returning success. The same is true of the
postdrop(1) program
writing the message to the "maildrop" directory.
As the pickup service is single threaded, it can only deliver
one message at a time at a rate that does not exceed the reciprocal
disk I/O latency (+ CPU if not negligible) of the cleanup service.
Congestion in this queue is indicative of an excessive local
message submission rate or perhaps excessive CPU consumption in
the
cleanup(8) service due to excessive
body_checks.
Note, that once the
active queue is full, the cleanup service
will attempt to slow down message injection by pausing $
in_flow_delay
for each message. In this case "
maildrop" queue congestion may be
a consequence of congestion downstream, rather than a problem in
its own right.
Note also, that one should not attempt to deliver large volumes
of mail via the
pickup(8) service. High volume sites must avoid
using content filters that reinject scanned mail via Postfix
sendmail(1) and
postdrop(1).
A high arrival rate of locally submitted mail may be an indication
of an uncaught forwarding loop, or a run-away notification program.
Try to keep the volume of local mail injection to a moderate level.
The "postsuper -r" command can place selected messages into
the "
maildrop" queue for reprocessing. This is most useful for
resetting any stale
content_filter settings. Requeuing a large number
of messages using "postsuper -r" can clearly cause a spike in the
size of the "
maildrop" queue.
The administrator can define "smtpd"
access(5) policies, or
cleanup(8) header/body checks that cause messages to be automatically
diverted from normal processing and placed indefinitely in the
"
hold" queue. Messages placed in the "hold" queue stay there until
the administrator intervenes. No periodic delivery attempts are
made for messages in the "
hold" queue. The
postsuper(1) command
can be used to manually release messages into the "
deferred" queue.
Messages can potentially stay in the "
hold" queue for a time
exceeding the normal maximal queue lifetime (after which undelivered
messages are bounced back to the sender). If such "old" messages
need to be released from the "
hold" queue, they should typically
be moved into the "
maildrop" queue, so that the message gets a new
timestamp and is given more than one opportunity to be delivered.
Messages that are "young" can be moved directly into the "deferred"
queue.
The "
hold" queue plays little role in Postfix performance, and
monitoring of the "
hold" queue is typically more closely motivated
by tracking spam and malware, than by performance issues.
All new mail entering the Postfix queue is written by the
cleanup(8) service into the "
incoming" queue. New queue files are
created owned by the "postfix" user with an access bitmask (or
mode) of 0600. Once a queue file is ready for further processing
the
cleanup(8) service changes the queue file mode to 0700 and
notifies the queue manager of new mail arrival. The queue manager
ignores incomplete queue files whose mode is 0600, as these are
still being written by cleanup.
The queue manager scans the
incoming queue bringing any new
mail into the "
active" queue if the active queue resource limits
have not been exceeded. By default, the
active queue accommodates
at most 20000 messages. Once the
active queue message limit is
reached, the queue manager stops scanning the incoming (and deferred,
see below) queue.
Under normal conditions the
incoming queue is nearly empty (has
only mode 0600 files), with the queue manager able to import new
messages into the
active queue as soon as they become available.
The
incoming queue grows when the message input rate spikes
above the rate at which the queue manager can import messages into
the
active queue. The main factor slowing down the queue manager
is transport queries to the trivial-rewrite service. If the queue
manager is routinely not keeping up, consider not using "slow"
lookup services (MySQL, LDAP, ...) for transport lookups or speeding
up the hosts that provide the lookup service.
The
in_flow_delay parameter is used to clamp the input rate
when the queue manager starts to fall behind. The
cleanup(8) service
will pause for $
in_flow_delay seconds before creating a new queue
file if it cannot obtain a "token" from the queue manager.
Since the number of
cleanup(8) processes is limited in most
cases by the SMTP server concurrency, the input rate can exceed
the output rate by at most "SMTP connection count" / $
in_flow_delay
messages per second.
With a default process limit of 100, and an
in_flow_delay of
1s, the coupling is strong enough to limit a single run-away injector
to 1 message per second, but is not strong enough to deflect an
excessive input rate from many sources at the same time.
If a server is being hammered from multiple directions, consider
raising the
in_flow_delay to 10 seconds, but only if the incoming
queue is growing even while the
active queue is not full and the
trivial-rewrite service is using a fast transport lookup mechanism.
The queue manager is a delivery agent scheduler; it works to
ensure fast and fair delivery of mail to all destinations within
designated resource limits.
The
active queue is somewhat analogous to an operating system's
process run queue. Messages in the
active queue are ready to be
sent (runnable), but are not necessarily in the process of being
sent (running).
While most Postfix administrators think of the "
active" queue
as a directory on disk, the real "
active" queue is a set of data
structures in the memory of the queue manager process.
Messages in the "
maildrop", "
hold", "
incoming" and "deferred"
queues (see below) do not occupy memory; they are safely stored on
disk waiting for their turn to be processed. The envelope information
for messages in the "
active" queue is managed in memory, allowing
the queue manager to do global scheduling, allocating available
delivery agent processes to an appropriate message in the active
queue.
Within the
active queue, (multi-recipient) messages are broken
up into groups of recipients that share the same transport/nexthop
combination; the group size is capped by the transport's recipient
concurrency limit.
Multiple recipient groups (from one or more messages) are queued
for delivery via the common transport/nexthop combination. The
destination concurrency limit for the transports caps the number
of simultaneous delivery attempts for each nexthop. Transports with
a recipient concurrency limit of 1 are special: these are grouped
by the actual recipient address rather than the nexthop, thereby
enabling per-recipient concurrency limits rather than per-domain
concurrency limits. Per-recipient limits are appropriate when
performing final delivery to mailboxes rather than when relaying
to a remote server.
Congestion occurs in the
active queue when one or more destinations
drain slower than the corresponding message input rate. If a
destination is down for some time, the queue manager will mark it
dead, and immediately defer all mail for the destination without
trying to assign it to a delivery agent. In this case the messages
will quickly leave the
active queue and end up in the deferred
queue. If the destination is instead simply slow, or there is a
problem causing an excessive arrival rate the
active queue will
grow and will become dominated by mail to the congested destination.
The only way to reduce congestion is to either reduce the input
rate or increase the throughput. Increasing the throughput requires
either increasing the concurrency or reducing the latency of
deliveries.
For high volume sites a key tuning parameter is the number of
"smtp" delivery agents allocated to the "smtp" and "relay" transports.
High volume sites tend to send to many different destinations, many
of which may be down or slow, so a good fraction of the available
delivery agents will be blocked waiting for slow sites. Also mail
destined across the globe will incur large SMTP command-response
latencies, so high message throughput can only be achieved with
more concurrent delivery agents.
The default "smtp" process limit of 100 is good enough for most
sites, and may even need to be lowered for sites with low bandwidth
connections (no use increasing concurrency once the network pipe
is full). When one finds that the queue is growing on an "idle"
system (CPU, disk I/O and network not exhausted) the remaining
reason for congestion is insufficient concurrency in the face of
a high average latency. If the number of outbound SMTP connections
(either ESTABLISHED or SYN_SENT) reaches the process limit, mail
is draining slowly and the system and network are not loaded, raise
the "smtp" and/or "relay" process limits!
Especially for the "relay" transport, consider lower SMTP
connection timeouts (1-5 seconds) and higher than default destination
concurrency limits. Compute the expected latency when 1 out of N
of the MX hosts for a high volume site is down and not responding,
and make sure that the configured concurrency divided by this
latency exceeds the required steady-state message rate. If the
destination is managed by you, consider load balancers in front of
groups of MX hosts. Load balancers have higher uptime and will be
able to hide individual MX host failures.
If necessary, dedicate and tune custom transports for high
volume destinations.
Another common cause of congestion is unwarranted flushing of
the entire
deferred queue. The deferred queue holds messages that
are likely to fail to be delivered and are also likely to be slow
to fail delivery (timeouts). This means that the most common reaction
to a large
deferred queue (flush it!) is more than likely counter-
productive, and is likely to make the problem worse. Do not flush
the
deferred queue unless you expect that most of its content has
recently become deliverable (e.g.
relayhost back up after an outage)!
Note that whenever the queue manager is restarted, there may
already be messages in the
active queue directory, but the "real"
active queue in memory is empty. In order to recover the in-memory
state, the queue manager moves all the
active queue messages
back into the
incoming queue, and then uses its normal incoming
queue scan to refill the
active queue. The process of moving all
the messages back and forth, redoing transport table (
trivial-rewrite(8)
resolve service) lookups, and re-importing the messages back into
memory is expensive. At all costs, avoid frequent restarts of the
queue manager.
When all the deliverable recipients for a message are delivered,
and for some recipients delivery failed for a transient reason (it
might succeed later), the message is placed in the
deferred queue.
The queue manager scans the
deferred queue periodically. The
scan interval is controlled by the
queue_run_delay parameter.
While a
deferred queue scan is in progress, if an
incoming queue
scan is also in progress (ideally these are brief since the incoming
queue should be short), the queue manager alternates between bringing
a new "incoming" message and a new "deferred" message into the
queue. This "round-robin" strategy prevents starvation of either
the
incoming or the
deferred queues.
Each
deferred queue scan only brings a fraction of the deferred
queue back into the
active queue for a retry. This is because each
message in the
deferred queue is assigned a "cool-off" time when
it is deferred. This is done by time-warping the modification
times of the queue file into the future. The queue file is not
eligible for a retry if its modification time is not yet reached.
The "cool-off" time is at least $
minimal_backoff_time and at
most $
maximal_backoff_time. The next retry time is set by doubling
the message's age in the queue, and adjusting up or down to lie
within the limits. This means that young messages are initially
retried more often than old messages.
If a high volume site routinely has large
deferred queues, it
may be useful to adjust the
queue_run_delay,
minimal_backoff_time
and
maximal_backoff_time to provide short enough delays on first
failure, with perhaps longer delays after multiple failures, to
reduce the retransmission rate of old messages and thereby reduce
the quantity of previously deferred mail in the
active queue.
One common cause of large
deferred queues is failure to validate
recipients at the SMTP input stage. Since spammers routinely launch
dictionary attacks from unrepliable sender addresses, the bounces
for invalid recipient addresses clog the
deferred queue (and at
high volumes proportionally clog the
active queue). Recipient
validation is strongly recommended through use of the
local_recipient_maps
and
relay_recipient_maps parameters.
When a host with lots of deferred mail is down for some time,
it is possible for the entire
deferred queue to reach its retry
time simultaneously. This can lead to a very full
active queue once
the host comes back up. The phenomenon can repeat approximately
every
maximal_backoff_time seconds if the messages are again deferred
after a brief burst of congestion. Ideally, in the future Postfix
will add a random offset to the retry time (or use a combination
of strategies) to reduce the chances of repeated complete deferred
queue flushes.
