关于PCIe tree的bus/device的详细architecture，参考LDD3和Mastering Linux Device Driver Development - John Madieu
Root complex (RC): This refers to the PCIe host controller in the SoC. It can access the main memory without CPU intervening, which is a feature used by other devices to access the main memory. They are also known as Host-to-PCI bridges.
Bridges: These provide an interface to other buses, such as PCI or PCI X, or even another PCIe bus. A bridge can also provide an interface to the same bus
Endpoint (EP): Endpoints are PCIe devices, and are represented by type 00h configuration space headers. They never appear on a switch’s internal bus and have no downstream port
我们要确认PCI bridge的 link status register(位于capability register offset 0x12, bit 13 Link active bit) 以知道 PCIe link 是否 active. 如何查看：如下图 0x00 ~ 0x3C 是config space标准空间；0x34 capability pointer是地址，指向capability register空间，其值为0x40，因此capability register空间是从0x40开始；因此 link status regsiter 是 0x40+0x12 = 0x52, 其bit13 即 0x53 的 bit5.
[ 1116.993784] bht-sd 0000:03:00.0: saving config space at offset 0x0 (reading 0x98621217) [ 1116.993790] bht-sd 0000:03:00.0: saving config space at offset 0x4 (reading 0x100406) [ 1116.993793] bht-sd 0000:03:00.0: saving config space at offset 0x8 (reading 0x8050101) [ 1116.993796] bht-sd 0000:03:00.0: saving config space at offset 0xc (reading 0x10) [ 1116.993798] bht-sd 0000:03:00.0: saving config space at offset 0x10 (reading 0x51100000) [ 1116.993801] bht-sd 0000:03:00.0: saving config space at offset 0x14 (reading 0x51101000) [ 1116.993804] bht-sd 0000:03:00.0: saving config space at offset 0x18 (reading 0x0) [ 1116.993806] bht-sd 0000:03:00.0: saving config space at offset 0x1c (reading 0x0) [ 1116.993809] bht-sd 0000:03:00.0: saving config space at offset 0x20 (reading 0x0) [ 1116.993811] bht-sd 0000:03:00.0: saving config space at offset 0x24 (reading 0x0) [ 1116.993813] bht-sd 0000:03:00.0: saving config space at offset 0x28 (reading 0x0) [ 1116.993816] bht-sd 0000:03:00.0: saving config space at offset 0x2c (reading 0x21217) [ 1116.993818] bht-sd 0000:03:00.0: saving config space at offset 0x30 (reading 0x0) [ 1116.993821] bht-sd 0000:03:00.0: saving config space at offset 0x34 (reading 0x6c) [ 1116.993823] bht-sd 0000:03:00.0: saving config space at offset 0x38 (reading 0x0) [ 1116.993826] bht-sd 0000:03:00.0: saving config space at offset 0x3c (reading 0x10b) [ 1118.015083] pcieport 0000:00:1c.7: re-enabling LTR [ 1118.015133] bht-sd 0000:03:00.0: restoring config space at offset 0x3c (was 0x100, writing 0x10b) [ 1118.015170] bht-sd 0000:03:00.0: restoring config space at offset 0x14 (was 0x0, writing 0x51101000) [ 1118.015191] bht-sd 0000:03:00.0: restoring config space at offset 0x10 (was 0x0, writing 0x51100000) [ 1118.015205] bht-sd 0000:03:00.0: restoring config space at offset 0xc (was 0x0, writing 0x10) [ 1118.015227] bht-sd 0000:03:00.0: restoring config space at offset 0x4 (was 0x100000, writing 0x100406)
PCI driver的register access API分析
API在LDD3的PCI driver一章已经有较详细说明：
*Linux offers a standard interface to access the configuration space. As far as the driver is concerned, the configuration space can be accessed through 8- bit, 16-bit, or 32-bit data transfers. The relevant functions are prototyped in <linux/* pci.h>
1 2 3 4 5 6
int pci_read_config_byte(struct pci_dev *dev, int where, u8 *val); int pci_read_config_word(struct pci_dev *dev, int where, u16 *val); int pci_read_config_dword(struct pci_dev *dev, int where, u32 *val); int pci_write_config_byte(struct pci_dev *dev, int where, u8 val); int pci_write_config_word(struct pci_dev *dev, int where, u16 val); int pci_write_config_dword(struct pci_dev *dev, int where, u32 val);
int pci_read_config_word(const struct pci_dev *dev, int where, u16 *val) { if (pci_dev_is_disconnected(dev)) { PCI_SET_ERROR_RESPONSE(val); return PCIBIOS_DEVICE_NOT_FOUND; } return pci_bus_read_config_word(dev->bus, dev->devfn, where, val); } EXPORT_SYMBOL(pci_read_config_word);
pci_bus_read_config_word的内部实现LDD3也说了：
The actual implementation of pci_read_config_byte(dev, where, val), for instance, expands to: dev->bus->ops->read(bus, devfn, where, 8, val);
其中bus->ops使用pci_ops结构：
1 2 3 4 5 6
struct pci_ops { int (*read)(struct pci_bus *bus, unsigned int devfn, int where, int size, u32 *val); int (*write)(struct pci_bus *bus, unsigned int devfn, int where, int size, u32 val); };
pci_read_config_byte的实现是宏函数，使用##连接符动态定义byte, word, dword，因此直接搜索不到，实际还是在access.c#L74，就在EXPORT_SYMBOL(pci_bus_read_config_word); 的前面定义:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
#define PCI_OP_READ(size, type, len) \ int noinline pci_bus_read_config_##size \ (struct pci_bus *bus, unsigned int devfn, int pos, type *value) \ { \ int res; \ unsigned long flags; \ u32 data = 0; \ if (PCI_##size##_BAD) return PCIBIOS_BAD_REGISTER_NUMBER; \ pci_lock_config(flags); \ res = bus->ops->read(bus, devfn, pos, len, &data); \ if (res) \ PCI_SET_ERROR_RESPONSE(value); \ else \ *value = (type)data; \ pci_unlock_config(flags); \ return res; \ }
PCIe协议分析抓包发现一个可疑的vendor defined message，可能是对应上述错误信息：
ACS violation在PCIe spec描述如下：
(4)原因和解决办法
逻辑链：PCIe Link training到L0之后，PCI driver还没发送Config Write操作定义Bus number的时候, SD express卡就发送了一个Vendor Specific defined message，这个是违反spec规定的；而RC side ACS 功能是enable的，会检测收到的包中的bus number是否和自己已扫描的bus相符，如果不符就报错ACS violation，并放弃卡初始化。
解决办法(workaround)：关闭PCIe bridge driver的ACS enable bit (ACS Control register bit0=1’b0):
