openSolaris 2008 - Dynamically Reconfiguring IB Devices (<tt>cfgadm</tt>) - System Administration Guide: Devices and File Systems

Dynamically Reconfiguring IB Devices (`cfgadm`)

One can configure or unconfigure an IB device from a running system by using the cfgadm CLI only. This command also provides a way to display the IB fabric, manage communication services, and update p_key table databases. For more information, see cfgadm_ib(1M).

The cfgadm CLI manages dynamic reconfiguration, referred to in this guide as DR, of the entire IB fabric as seen by a host. The cfgadm operations are supported on all the IB devices, such as Port, VPPA, HCA_SVC, IOC, and pseudo devices.

The cfgadm command displays information about attachment points (Ap_Ids), which are locations in the system where DR operations can occur. For details on the Ap_Ids that cfgadm supports, see cfgadm_ib.1M. Note that all IB Ap_Ids are shown as connected.

The cfgadm command provides the following IB device status information.

Receptacle State	Description
`connected`/`configured`/`ok`	The device is connected and available. The `devinfo` node is present.
`connected`/`unconfigured`/`unknown`	The device is unavailable and no `devinfo` node or device driver exists for this device. Or, the device was never configured for use by `ib` nexus driver. The device might be known to the IB Device Manager.

The following sections describe how to dynamically reconfigure (DR) IB devices with the cfgadm command. All of the sample IB device information in these sections has been truncated to focus on relevant information.

How to Display IB Device Information

You can use the prtconf command to display general information about IB devices. For example:

$ prtconf
    pci, instance #0
        pci15b3,5a44, instance #0
             ibport, instance #253
             ibport, instance #254
             ibport, instance #255
    .
    .
    .
    ib, instance #0
        ioc, instance #243
        ioc, instance #244
        ioc, instance #245
        ioc, instance #246
        ioc, instance #247
        ioc, instance #248
        ibgen, instance #249

In the preceding example, pci15b3,5a44 refers to an IB HCA.

Use the following steps to display specific IB device information.

Become superuser.

Display IB fabric information.

For example:

# cfgadm -a
Ap_Id                       Type       Receptacle Occupant   Condition
ib                          IB-Fabric  connected  configured   ok
hca:1730000008070           IB-HCA     connected  configured   ok
ib::1730000007F5198         IB-IOC     connected  configured   ok
ib::1730000007F5199         IB-IOC     connected  configured   ok
ib::1730000008070,0,hnfs    IB-HCA_SVC connected  configured   ok
ib::1730000008071,0,sdp     IB-PORT    connected  configured   ok
ib::1730000008072,0,sdp     IB-PORT    connected  configured   ok
ib::1730000008071,8001,ipib IB-VPPA    connected  configured   ok
ib::1730000008072,8001,ipib IB-VPPA    connected  configured   ok
ib::ibgen,0                 IB-PSEUDO  connected  configured   ok
#

In the preceding example output, the components are described as follows:

Ap_Id ib::1730000008072,0,sdp: Identifies an IB Port device that is connected to port 2 and is bound to the sdp service.
Ap_Id ib::1730000008072,8001,ipib: Identifies an IB VPPA device that is connected to port 2, using a p_key value of 0x8001, and is bound to the ibd service.
Ap_Id ib:: 1730000008070,0,hnfs: Identifies an IB HCA_SVC device bound to the hnfs service.
Ap_Id ib::1730000007F5198: Identifies an IOC device.
Ap_Id ib::ibgen,0: Identifies a pseudo device.

Display specific IB device information.

For example, for an IB VPPA device:

# cfgadm -al -s "cols=ap_id:info" ib::1730000008072,8001,ipib
Ap_Id                          Information
ib::1730000008072,8001,ipib    ipib
#

For example, for an IB HCA device:

# cfgadm -al -s "cols=ap_id:info" hca::1730000008070
Ap_Id                          Information
hca::1730000008070             VID: 0x15b3, PID: 0x5a44, #ports: 0x2, 
port1 GUID: 0x1730000008071, port2 GUID: 0x1730000008072
#

The preceding output displays the number of ports and their GUIDs.

How to Unconfigure an IOC Device

You can unconfigure an IB device that is still physically connected to the system, but a driver will never attach to it.

Become superuser.

Unconfigure the IB device.

For example:

# cfgadm -c unconfigure ib::1730000007F5198
Unconfigure the device: /devices/ib:fabric::1730000007F5198
This operation will suspend activity on the IB device
Continue (yes/no)? y
#

Verify that the device is unconfigured.

For example:

# cfgadm -a ib::1730000007F5198
ib::1730000007F5198         IB-IOC     connected  unconfigured unknown
#

How to Configure an IOC Device

Become superuser.

Configure a IB device.

For example:

# cfgadm -yc configure ib::1730000007F5198

Verify that the IB device is configured.

For example:

# cfgadm -al ib::1730000007F5198
Ap_Id                Type    Receptacle  Occupant    Condition
ib::1730000007F5198  IB-IOC  connected   configured  ok

How to Unconfigure an IB Port, HCA_SVC, or a VPPA Device

Use the following steps if you want to remove an IB Port, HCA_SVC, or a VPPA device from the system.

The example below illustrates how to unconfigure a VPPA device, but the same procedure applies to Port and HCA_SVC devices as well.

Become superuser.

Unconfigure the IB VPPA device.

For example:

# cfgadm -c unconfigure ib::1730000007F51,8001,ipib
Unconfigure the device: /devices/ib:fabric::1730000007F51,8001,ipib
This operation will suspend activity on the IB device
Continue (yes/no)? Y
#

Verify that the device is disconnected.

For example:

# cfgadm -a ib::1730000007F51,8001,ipib
Ap_Id                       Type    Receptacle Occupant     Condition
ib::1730000007F51,8001,ipib IB-VPPA connected  unconfigured unknown
#

How to Configure a IB Port, HCA_SVC, or a VPPA Device

Use the following steps if you want to configure an IB Port, HCA_SVC, or a VPPA device on the system.

The example below illustrates how to configure a VPPA device, but similar steps can be used to configure Port and HCA_SVC devices as well.

Become superuser.

Configure the IB VPPA device.

For example:

# cfgadm -c configure ib::1730000007F51,8001,ipib

Verify that the device is connected.
For example:
```
# cfgadm -a ib::1730000007F51,8001,ipib   
Ap_Id                       Type     Receptacle Occupant   Condition
ib::1730000007F51,8001,ipib IB-VPPA  connected  configured ok
```
Note - A cfgadm based configure or unconfigure operation of IB Port and HCA_SVC devices is similar to the preceding examples for an IB VPPA device.

How to Unconfigure an IB Pseudo Device

Use the following steps if you want to remove an IB pseudo device from the system.

Become superuser.

Unconfigure the IB pseudo device.

For example:

# cfgadm -c unconfigure ib::ibgen,0
Unconfigure the device: /devices/ib:fabric::ibgen,0
This operation will suspend activity on the IB device
Continue (yes/no)? Y
#

Verify that the device is disconnected.

# cfgadm -a ib::ibgen,0
Ap_Id                  Type      Receptacle Occupant     Condition
ib::ibgen,0            IB-PSEUDO connected  unconfigured unknown

How to Configure an IB Pseudo Device

Use the following steps to configure an IB pseudo device.

Become superuser.
Configure the IB pseudo device.
For example:
```
# cfgadm -yc configure ib::ibgen,0
```

Verify that the device is connected.

For example:

# cfgadm -a ib::ibgen,0
Ap_Id                  Type       Receptacle Occupant   Condition
ib::ibgen,0            IB-PSEUDO  connected  configured ok

How to Display Kernel IB Clients of an HCA

The following IB cfgadm plugin command can be invoked to list kernel IB clients using this HCA. Note that the last column would show a “yes” if a kernel IB client uses another HCA. IB Managers and kernel clients that do not use the HCA are shown with an Ap_Id of “-”.

Display kernel IB clients of an HCA.

For example:

$ cfgadm -x list_clients hca:173000007F50   
Ap_Id                       IB Client         Alternate HCA
ib::1730000007F51D0            ibgen              no
ib::1730000007F51D1            ibgen              no
ib::1730000007F51,8001,ipib    ibd                no
ib::ibgen,0                    ibgen              no
-                              ibdm               no
-                              ibmf               no
-                              nfs/ib             no
$

How to Unconfigure IB Devices Connected to an HCA

An actual DR of an HCA is beyond the scope of the IB cfgadm plugin. Although DR of an HCA can be achieved by using the plugin of the underlying bus. For example, a PCI based HCA can use the cfgadm_pci command. For more information, see cfgadm_pci(1M).

However, the IB cfgadm plugin assists in the HCA DR by listing its kernel IB clients as illustrated in steps below.

Become superuser.

List the kernel IB clients of the HCA.

For example:

# cfgadm -x list_clients hca:173000007F50   
Ap_Id                       IB Client         Alternate HCA
ib::1730000007F51D0            ibgen              no
ib::1730000007F51D1            ibgen              no
ib::1730000007F51,8001,ipib    ibd                no
ib::ibgen,0                    ibgen              no
-                              ibdm               no
-                              ibmf               no
-                              nfs/ib             no

Unconfigure kernel IB clients, such as Port, VPPA, HCA_SVC, or IOC devices, that do not have alternate HCAs present.

For example:

# cfgadm -x unconfig_clients hca:1730000008070
Unconfigure Clients of HCA /devices/ib:1730000008070
This operation will unconfigure IB clients of this HCA
Continue (yes/no)? y

Verify that the kernel IB clients of the HCA are unconfigured.

# cfgadm -x list_clients hca:173000007F50   
Ap_Id                       IB Client         Alternate HCA
-                              ibdm               no
-                              ibmf               no
-                              nfs/ib             no
#

Configuring an IB HCA

Invoke the bus-specific cfgadm plugin to configure the HCA. The exact details are beyond the scope of this chapter.

How to Update the IB `p_key` Tables

If the p_key table information of an HCA ports changes, for example, additional p_keys are enabled or disabled, InfiniBand Transport Framework (IBTF) and IBDM need to be notified so that their internal p_key databases are updated. The cfgadm command helps update the p_key databases of IBTF and IBDM. For more information, see ibtl(7D) and ibdm(7D).

Become superuser.
Update the p_key tables.
For example:
```
# cfgadm -x update_pkey_tbls -y ib
```

How to Display IB Communication Services

Use the following steps to display the communication services that are currently in use by the IBTF.

Become superuser.

Display IB communication services.

For example:

# cfgadm -x list_services ib
Port communication services:
                srp
VPPA communication services:
                ibd
HCA_SVC communication services:
                hnfs

How to Add a VPPA Communication Service

Use the following steps to add a new VPPA communication service.

Similar steps can be used to add a new HCA_SVC or a port communication service.

Become superuser.

Add a new VPPA communication service.

For example:

# cfgadm -o comm=vppa,service=new -x add_service ib

Verify that the new service has been added.

For example:

# cfgadm -x list_services ib
Port communication services:
                srp
VPPA communication services:
                ibd
                new
HCA_SVC communication services:
                nfs_service
#

How to Remove an Existing IB Port, HCA_SVC, or a VPPA Communication Service

Use the following steps to delete an existing IB Port, HCA_SVC, or a VPPA communication service.

Become superuser.

Remove a VPPA communication service.

For example:

# cfgadm -o comm=vppa,service=new -x delete_service ib

Verify that the communication service has been removed.

For example:

# cfgadm -x list_services ib
Port communication services:
                srp
VPPA communication services:
                ibd
HCA_SVC communication services:
                hnfs
#

How to Update an IOC Configuration

Use the following steps to update properties of all the IOC device nodes or for a particular IOC Ap_Id. The properties that can get updated are as follows:

port-list
port-entries
service-id
service-name

For more information on these properties, see ib(7D).

Note that these properties may not get updated if there is no configuration change. The following example describes how to update a particular IOC's configuration. If you need to update the configuration of all the IOCs, then specify the static ib Ap_Id instead of the particular IOC Ap_Id.

Become superuser.

Update the configuration of an IOC.

For example:

# cfgadm -x update_ioc_conf ib::1730000007F5198
This operation can update properties of IOC devices.
Continue (yes/no)? y
#

Verify that the properties have been updated by running prtconf -v.

Dynamically Reconfiguring IB Devices (cfgadm)

How to Display IB Device Information

How to Unconfigure an IOC Device

How to Configure an IOC Device

How to Unconfigure an IB Port, HCA_SVC, or a VPPA Device

How to Configure a IB Port, HCA_SVC, or a VPPA Device

How to Unconfigure an IB Pseudo Device

How to Configure an IB Pseudo Device

How to Display Kernel IB Clients of an HCA

How to Unconfigure IB Devices Connected to an HCA

Configuring an IB HCA

How to Update the IB p_key Tables

How to Display IB Communication Services

How to Add a VPPA Communication Service

How to Remove an Existing IB Port, HCA_SVC, or a VPPA Communication Service

How to Update an IOC Configuration

Dynamically Reconfiguring IB Devices (`cfgadm`)

How to Update the IB `p_key` Tables