The lspci and setpci commands are available natively in Linux distributions. This command has various levels of output, and provides a very useful point-in-time look at the capabilities and status of the different components trained on the PCI bus.
Most of these capabilities are reflections of the Configuration Space registers required by the PCI-Express Base Specification. As with most commands, usage instructions can be found by running “lspci –help” or “man lspci”, in Linux.
lspci
By default, the lspci command will display all of the device information as shown below.
lspci -tv
This command displays PCI devices in tree format and gives the root port BDF (Bus, Device, Function) number.
In the above log, the Xilinx device is connected to Bus Number ’00’, Device Number ‘01’ and Function Number ‘1’.
lspci -vvv
This is the most verbose command that displays everything. To run this command, root permission is required.
The log below shows only the portion related to the Xilinx PCIe device.
Points to note:
• Both the Link Capability Register and Link Status Register show Gen3x8. Sometimes, due to link issues, it might train down. The trained down link status is reflected in the link status register:
• The Correctable Error Status register shows Non-Fatal Error as set. During the boot, the host probes the non-configured functions as well. Because the design is configured for one function, the probe for other functions is reported as an unsupported request. This unsupported request is reported as an Advisory Non-Fatal error. If Non-Fatal Error, Unsupported Request and Correctable Error are set during the boot, this can be ignored. They can be cleared by doing a config write to the respective bits in the corresponding registers.
• Users must monitor errors in the uncorrectable error status register. If an error is reported in this register, it must be investigated and resolved.
• The PCI Express Capabilities starts at ‘80’.
lspci -vs <BDF>
This command gives verbose output for the selected device as shown below:
lspci -vvvs <BDF>
This is the same command but with more verbose output as shown below:
lspci -nvmms <BDF>
This command displays the PCI device Vendor ID and Device ID as numbers.
lspci -xxx
This command gives a hexadecimal dump of the whole PCI configuration space.
The first two bytes at 0x00 are actually 0x10EE. Within the PCIe specification, all data is defined by an offset, for example the Yellow box containing 0x80 is at offset 0x34 – which makes this the pointer to the first address of the extended capabilities register.
PCI-IDs: In the red box at 0x00 is the Vendor ID (0x10EE), followed in the blue box by the Device ID (0x7038). Down at offset 0x2C and 0x2E in the red and blue boxes respectively are the Sub-Vendor ID (0x10EE) and the Sub-Device ID (0x0700).
Bus Master Enable: Back up at 0x04 in the yellow box there is a 16 bit word “0000 0000 0000 0111”. This is the Command register. Bit 2 is “Bus Master Enable”.
At 0x06 in the green box is a 16 bit word “0000 0000 0000 0001”. This is the status register, which will change over time and is a way of signalling to the root complex that certain conditions have occurred.
BAR and Memory at 0x10, there is a 32-bit word “0000 0000 0000 0000 1111 0111 1010 0000”
- Bit 0 = 0 – A request for memory space
- Bit 2:1 = “00” – Base address which is 32 bits wide
- Bit 3 = “0” – Non-prefetchable
- Bit 31:4 = 0xF7C0 (with the lowest 4 bits assumed to be 0, as these must be assigned on a byte and dword boundary)
Note: if this had been a 64 bit wide request, the next D-Word would contain the upper memory address, and the follow on BAR would be BAR2. - At 0x14 – this would be the BAR1 address, however, since it is all zeros, this device only has one BAR option
Checking the PCIe Link Width
PCIe width determines the number of PCIe lanes. The command below makes it easier to find the PCIe Link Width information in the Link Capabilities Register and the negotiated link width in the Link Status Register.
Checking PCIe Speed
Similar to the command for checking the PCIe link width information, the command below provides information on PCIe speed.
Checking PCIe Max Payload Size (MPS)
The command below provides the Max Payload Size value under the Device Control Register.
Checking PCIe Max Read Request Size
Listing all PCIe Devices
setpci
The setpci command can be used for reading from and writing to configuration registers. See “setpci –help” for detailed information on setpci features.
setpci knows the names of all registers in the standard configuration headers. The “setpci –dumpregs” command shows a list of all PCI registers and capabilities as shown below:
[root@localhost xilinx]# setpci –dumpregs
cap pos w name
00 W VENDOR_ID
02 W DEVICE_ID
04 W COMMAND
06 W STATUS
08 B REVISION
09 B CLASS_PROG
0a W CLASS_DEVICE
0c B CACHE_LINE_SIZE
0d B LATENCY_TIMER
0e B HEADER_TYPE
0f B BIST
10 L BASE_ADDRESS_0
14 L BASE_ADDRESS_1
18 L BASE_ADDRESS_2
1c L BASE_ADDRESS_3
20 L BASE_ADDRESS_4
24 L BASE_ADDRESS_5
28 L CARDBUS_CIS
2c L SUBSYSTEM_VENDOR_ID
2e W SUBSYSTEM_ID
30 L ROM_ADDRESS
3c B INTERRUPT_LINE
3d B INTERRUPT_PIN
3e B MIN_GNT
3f B MAX_LAT
18 B PRIMARY_BUS
19 B SECONDARY_BUS
1a B SUBORDINATE_BUS
1b B SEC_LATENCY_TIMER
1c B IO_BASE
1d B IO_LIMIT
1e W SEC_STATUS
20 W MEMORY_BASE
22 W MEMORY_LIMIT
24 W PREF_MEMORY_BASE
26 W PREF_MEMORY_LIMIT
28 L PREF_BASE_UPPER32
2c L PREF_LIMIT_UPPER32
30 W IO_BASE_UPPER16
32 W IO_LIMIT_UPPER16
38 L BRIDGE_ROM_ADDRESS
3e W BRIDGE_CONTROL
10 L CB_CARDBUS_BASE
14 W CB_CAPABILITIES
16 W CB_SEC_STATUS
18 B CB_BUS_NUMBER
19 B CB_CARDBUS_NUMBER
1a B CB_SUBORDINATE_BUS
1b B CB_CARDBUS_LATENCY
1c L CB_MEMORY_BASE_0
20 L CB_MEMORY_LIMIT_0
24 L CB_MEMORY_BASE_1
28 L CB_MEMORY_LIMIT_1
2c W CB_IO_BASE_0
2e W CB_IO_BASE_0_HI
30 W CB_IO_LIMIT_0
32 W CB_IO_LIMIT_0_HI
34 W CB_IO_BASE_1
36 W CB_IO_BASE_1_HI
38 W CB_IO_LIMIT_1
3a W CB_IO_LIMIT_1_HI
40 W CB_SUBSYSTEM_VENDOR_ID
42 W CB_SUBSYSTEM_ID
44 L CB_LEGACY_MODE_BASE
01 00 – CAP_PM
02 00 – CAP_AGP
03 00 – CAP_VPD
04 00 – CAP_SLOTID
05 00 – CAP_MSI
06 00 – CAP_CHSWP
07 00 – CAP_PCIX
08 00 – CAP_HT
09 00 – CAP_VNDR
0a 00 – CAP_DBG
0b 00 – CAP_CCRC
0c 00 – CAP_HOTPLUG
0d 00 – CAP_SSVID
0e 00 – CAP_AGP3
0f 00 – CAP_SECURE
10 00 – CAP_EXP
11 00 – CAP_MSIX
12 00 – CAP_SATA
13 00 – CAP_AF
0001 00 – ECAP_AER
0002 00 – ECAP_VC
0003 00 – ECAP_DSN
0004 00 – ECAP_PB
0005 00 – ECAP_RCLINK
0006 00 – ECAP_RCILINK
0007 00 – ECAP_RCECOLL
0008 00 – ECAP_MFVC
000a 00 – ECAP_RBCB
000b 00 – ECAP_VNDR
000d 00 – ECAP_ACS
000e 00 – ECAP_ARI
000f 00 – ECAP_ATS
0010 00 – ECAP_SRIOV
Identifying the register in setpci
Below are various ways to identify the register being used in the setpci command.
- Using the Hexadecimal address
- Provide the register name
- For registers which are part of the PCI capability, the first register can be addressed with the name of the capability. In the –dumpregs output. Check for names starting with `CAP_’ or `ECAP_’. This can be followed with +offset to add an offset (a hex number) to the address. This makes it easier to address registers which are part of the respective capability register which is set.
- Width specifiers (b, .w, or .l) are used to choose how many bytes (1, 2, or 4) are to be read or written. The specifier can be dropped if the register is being referred to by its name and the width of the register is already known.
- All names of registers and width specifiers are case-insensitive.
Examples:
COMMAND
- This points to the value in the command register. If this is replaced with 4.w, it will point to the same location.
COMMAND.l
- This points to the value of both the command and the status register.
VENDOR_ID+1.b
- This points to the upper byte of the vendor ID register.
CAP_PM+2.w
- This corresponds to the second word of the power management capability.
ECAP108.l
- This points to the first 32-bit word of the extended capability with ID 0x108.
setpci –s 24:00.0 04.w=6
- For MSI interrupts to work, the Bus Master Enable bit must be set in the PCIe Configuration. The above command can be used to set the ‘Bus Master Enable’ bit in the command register. “24:00.0” is the BDF number in this example. It will be different for different devices and also depends on the system. For the correct BDF of the device to be addressed, see the corresponding lspci log.
- A value of 6 means that we are setting the Memory Enable bit and the Bus Master Enable bit.
setpci –s 24:00.0 4a.w=1
- The MSI register must also be enabled in the PCIe Configuration Space for MSI interrupts to work. In UltraScale+ devices, it is at offset 0x48 (which is also shown in the lspci log as Capabilities: [48]). To do this, issue a PCIe Configuration Write to set bit 16 (MSI Control register bit 0) to 1; the above command does that.
- After executing the above command, run the lspci command. This should show “MSI: Enable +” instead.
setpci -s 01:00.0 82.b
- The above command reads from the Link Status Register of the UltraScale+ PCIe endpoint device
- The address ‘82’ is for UltraScale+ device. See (PG213) for the detailed table.
setpci -s 00:01.0 d0.b
- The above command reads from the Link Control 2 Register of the Root Port
- The link capability base address is a0 as shown below in the corresponding lspci log:
- The Link Control 2 Register offset is ’30’. Adding ‘a0′ to ’30’ results in ‘d0′. The ‘a0’ address is for the root port device shown in the above screen shot. This can vary depending on the device used. For the root port using the UltraScale+ device, the Capabilities address starts at ’70’ as shown below and the Link Control 2 register offset will be the same ‘30’ i.e. 48 in decimal value.
The base address value of ‘70’ can also be read from the lspci log as shown below:
setpci -s 00:01.0 d0.b=42
- The above command writes to the Link Control 2 Register to set the speed to Gen2.
- The value ‘2’ here indicates Gen2 and the other bit is ‘Slot Clock’ which is already enabled and hence not changed. After issuing the above command, if you do lspci, it will show that the speed value in the register has changed to Gen2. However, this is just the speed that it will train to when it goes through the link training again.
- To change the link speed to Gen2, the link must be retrained. This can be done by executing the command shown below:
- setpci -s 00:01.0 b0.b=62
Changing The PCIe Max Read Request Size
- Query the register in order to avoid overriding other properties.
- setpci -s 04:00.0 78.w
- Write the desired value to the register.
- setpci -s 04:00.0 78.w=2936
- The byte offset of the Device Control Register where the Max Read Request resides is 78h for UltraScale+ devices. This can vary for other devices. Users should consult the respective product guides. The table below is from (PG213).
