Queueing Tasks
This section discusses how to use task queues to postpone processing of some tasks
and delegate their execution to another kernel thread.
Introduction to Task Queues
A common operation in kernel programming is to schedule a task to
be performed at a later time, by a different thread. The following examples
give some reasons that you might want a different thread to perform a
task at a later time:
Your current code path is time critical. The additional task you want to perform is not time critical.
The additional task might require grabbing a lock that another thread is currently holding.
You cannot block in your current context. The additional task might need to block, for example to wait for memory.
A condition is preventing your code path from completing, but your current code path cannot sleep or fail. You need to queue the current task to execute after the condition disappears.
You need to launch multiple tasks in parallel.
In each of these cases, a task is executed in a different
context. A different context is usually a different kernel thread with a different
set of locks held and possibly a different priority. Task queues provide a
generic kernel API for scheduling asynchronous tasks.
A task queue is a list of tasks with one or more threads to
service the list. If a task queue has a single service thread, all
tasks are guaranteed to execute in the order in which they are added
to the list. If a task queue has more than one service
thread, the order in which the tasks will execute is not known.
Note - If the task queue has more than one service thread, make sure that
the execution of one task does not depend on the execution of
any other task. Dependencies between tasks can cause a deadlock to occur.
Task Queue Interfaces
The following DDI interfaces manage task queues. These interfaces are defined in the
sys/sunddi.h header file. See the taskq(9F) man page for more information about
these interfaces.
ddi_taskq_t |
Opaque handle |
TASKQ_DEFAULTPRI |
System default priority |
DDI_SLEEP |
Can block for memory |
DDI_NOSLEEP |
Cannot block for memory |
ddi_taskq_create() |
Create a
task queue |
ddi_taskq_destroy() |
Destroy a task queue |
ddi_taskq_dispatch() |
Add a task to a task queue |
ddi_taskq_wait() |
Wait for
pending tasks to complete |
ddi_taskq_suspend() |
Suspend a task queue |
ddi_taskq_suspended() |
Check whether a task queue is
suspended |
ddi_taskq_resume() |
Resume a suspended task queue |
Using Task Queues
The typical usage in drivers is to create task queues at attach(9E).
Most taskq_dispatch() invocations are from interrupt context.
To study task queues used in Solaris drivers, go to https://www.opensolaris.org/os/. In
the left margin menu, click Source Browser. In the Symbol field of the
search area, enter ddi_taskq_create. In the Project list, select onnv. Click the Search
button. In your search results you should see the USB generic serial driver
(usbser.c), the 1394 mass storage HBA FireWire driver (scsa1394/hba.c), and the SCSI HBA
driver for Dell PERC 3DC/4SC/4DC/4Di RAID devices (amr.c).
Click the file name amr.c. The ddi_taskq_create() function is called in the
amr_attach() entry point. The ddi_taskq_destroy() function is called in the amr_detach() entry point
and also in the error handling section of the amr_attach() entry point. The
ddi_taskq_dispatch() function is called in the amr_done() function, which is called in the
amr_intr() function. The amr_intr() function is an interrupt-handling function that is an
argument to the ddi_add_intr(9F) function in the amr_attach() entry point.
Observing Task Queues
This section describes two techniques that you can use to monitor the system
resources that are consumed by a task queue. Task queues export statistics on
the use of system time by task queue threads. Task queues also use
DTrace SDT probes to determine when a task queue starts and finishes execution
of a task.
Task Queue Kernel Statistics Counters
Every task queue has an associated set of kstat counters. Examine the output
of the following kstat(1M) command:
$ kstat -c taskq
module: unix instance: 0
name: ata_nexus_enum_tq class: taskq
crtime 53.877907833
executed 0
maxtasks 0
nactive 1
nalloc 0
priority 60
snaptime 258059.249256749
tasks 0
threads 1
totaltime 0
module: unix instance: 0
name: callout_taskq class: taskq
crtime 0
executed 13956358
maxtasks 4
nactive 4
nalloc 0
priority 99
snaptime 258059.24981709
tasks 13956358
threads 2
totaltime 120247890619The kstat output shown above includes the following information:
The name of the task queue and its instance number
The number of scheduled (tasks) and executed (executed) tasks
The number of kernel threads processing the task queue (threads) and their priority (priority)
The total time (in nanoseconds) spent processing all the tasks (totaltime)
The following example shows how you can use the kstat command to observe
how a counter (number of scheduled tasks) increases over time:
$ kstat -p unix:0:callout_taskq:tasks 1 5
unix:0:callout_taskq:tasks 13994642
unix:0:callout_taskq:tasks 13994711
unix:0:callout_taskq:tasks 13994784
unix:0:callout_taskq:tasks 13994855
unix:0:callout_taskq:tasks 13994926
Task Queue DTrace SDT Probes
Task queues provide several useful SDT probes. All the probes described in this
section have the following two arguments:
You can use these probes to collect precise timing information about individual task
queues and individual tasks being executed through them. For example, the following script
prints the functions that were scheduled through task queues for every 10 seconds:
# !/usr/sbin/dtrace -qs
sdt:genunix::taskq-enqueue
{
this->tq = (taskq_t *)arg0;
this->tqe = (taskq_ent_t *) arg1;
@[this->tq->tq_name,
this->tq->tq_instance,
this->tqe->tqent_func] = count();
}
tick-10s
{
printa ("%s(%d): %a called %@d times\n", @);
trunc(@);
}On a particular machine, the above D script produced the following output:
callout_taskq(1): genunix`callout_execute called 51 times
callout_taskq(0): genunix`callout_execute called 701 times
kmem_taskq(0): genunix`kmem_update_timeout called 1 times
kmem_taskq(0): genunix`kmem_hash_rescale called 4 times
callout_taskq(1): genunix`callout_execute called 40 times
USB_hid_81_pipehndl_tq_1(14): usba`hcdi_cb_thread called 256 times
callout_taskq(0): genunix`callout_execute called 702 times
kmem_taskq(0): genunix`kmem_update_timeout called 1 times
kmem_taskq(0): genunix`kmem_hash_rescale called 4 times
callout_taskq(1): genunix`callout_execute called 28 times
USB_hid_81_pipehndl_tq_1(14): usba`hcdi_cb_thread called 228 times
callout_taskq(0): genunix`callout_execute called 706 times
callout_taskq(1): genunix`callout_execute called 24 times
USB_hid_81_pipehndl_tq_1(14): usba`hcdi_cb_thread called 141 times
callout_taskq(0): genunix`callout_execute called 708 times
