thr3ads.net - Virtualization - [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support [Nov 2008]

If this information is useful, please help other people find it:
Share via:

Yu Zhao

2008-Nov-21 18:36 UTC

[PATCH 0/13 v7] PCI: Linux kernel SR-IOV support

Greetings,

Following patches are intended to support SR-IOV capability in the
Linux kernel. With these patches, people can turn a PCI device with
the capability into multiple ones from software perspective, which
will benefit KVM and achieve other purposes such as QoS, security,
and etc.

The Physical Function and Virtual Function drivers using the SR-IOV
APIs will come soon!

Major changes from v6 to v7:
1, remove boot-time resource rebalancing support. (Greg KH)
2, emit uevent upon the PF driver is loaded. (Greg KH)
3, put SR-IOV callback function into the 'pci_driver'. (Matthew Wilcox)
4, register SR-IOV service at the PF loading stage.
5, remove unnecessary APIs (pci_iov_enable/disable).

---

[PATCH 1/13 v7] PCI: enhance pci_ari_enabled()
[PATCH 2/13 v7] PCI: remove unnecessary arg of pci_update_resource()
[PATCH 3/13 v7] PCI: define PCI resource names in an 'enum'
[PATCH 4/13 v7] PCI: remove unnecessary condition check in pci_restore_bars()
[PATCH 5/13 v7] PCI: export __pci_read_base()
[PATCH 6/13 v7] PCI: make pci_alloc_child_bus() be able to handle NULL bridge
[PATCH 7/13 v7] PCI: add a new function to map BAR offset
[PATCH 8/13 v7] PCI: cleanup pci_bus_add_devices()
[PATCH 9/13 v7] PCI: split a new function from pci_bus_add_devices()
[PATCH 10/13 v7] PCI: support the SR-IOV capability
[PATCH 11/13 v7] PCI: reserve bus range for SR-IOV device
[PATCH 12/13 v7] PCI: document the SR-IOV sysfs entries
[PATCH 13/13 v7] PCI: document for SR-IOV user and developer

Cc: Alex Chiang <achiang at hp.com>
Cc: Bjorn Helgaas <bjorn.helgaas at hp.com>
Cc: Grant Grundler <grundler at parisc-linux.org>
Cc: Greg KH <greg at kroah.com>
Cc: Ingo Molnar <mingo at elte.hu>
Cc: Jesse Barnes <jbarnes at virtuousgeek.org>
Cc: Matthew Wilcox <matthew at wil.cx>
Cc: Randy Dunlap <randy.dunlap at oracle.com>
Cc: Roland Dreier <rdreier at cisco.com>
Cc: Simon Horman <horms at verge.net.au>
Cc: Yinghai Lu <yinghai at kernel.org>

---

Single Root I/O Virtualization (SR-IOV) capability defined by PCI-SIG
is intended to enable multiple system software to share PCI hardware
resources.  PCI device that supports this capability can be extended
to one Physical Functions plus multiple Virtual Functions. Physical
Function, which could be considered as the "real" PCI device, reflects
the hardware instance and manages all physical resources. Virtual
Functions are associated with a Physical Function and shares physical
resources with the Physical Function.Software can control allocation of
Virtual Functions via registers encapsulated in the capability structure.

SR-IOV specification can be found at
http://www.pcisig.com/members/downloads/specifications/iov/sr-iov1.0_11Sep07.pdf

Devices that support SR-IOV are available from following vendors:
http://download.intel.com/design/network/ProdBrf/320025.pdf
http://www.netxen.com/products/chipsolutions/NX3031.html
http://www.neterion.com/products/x3100.html

Yu Zhao

2008-Nov-21 18:38 UTC

head link

[PATCH 1/13 v7] PCI: enhance pci_ari_enabled()

Change parameter of pci_ari_enabled() from 'pci_dev' to
'pci_bus'.

ARI forwarding on the bridge mostly concerns the subordinate devices
rather than the bridge itself. So this change will make the function
easier to use.

Signed-off-by: Yu Zhao <yu.zhao at intel.com>

---
 drivers/pci/pci.h |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 9de87e9..1449884 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -162,13 +162,13 @@ struct pci_slot_attribute {
 extern void pci_enable_ari(struct pci_dev *dev);
 /**
  * pci_ari_enabled - query ARI forwarding status
- * @dev: the PCI device
+ * @bus: the PCI bus
  *
  * Returns 1 if ARI forwarding is enabled, or 0 if not enabled;
  */
-static inline int pci_ari_enabled(struct pci_dev *dev)
+static inline int pci_ari_enabled(struct pci_bus *bus)
 {
-	return dev->ari_enabled;
+	return bus->self && bus->self->ari_enabled;
 }
 
 #endif /* DRIVERS_PCI_H */
-- 
1.5.6.4

Yu Zhao

2008-Nov-21 18:38 UTC

head link

[PATCH 2/13 v7] PCI: remove unnecessary arg of pci_update_resource()

This cleanup removes unnecessary argument 'struct resource *res' in
pci_update_resource(), so it takes same arguments as other companion
functions (pci_assign_resource(), etc.).

Signed-off-by: Yu Zhao <yu.zhao at intel.com>

---
 drivers/pci/pci.c       |    4 ++--
 drivers/pci/setup-res.c |    7 ++++---
 include/linux/pci.h     |    2 +-
 3 files changed, 7 insertions(+), 6 deletions(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 21f2ac6..c408be8 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -377,8 +377,8 @@ pci_restore_bars(struct pci_dev *dev)
 		return;
 	}
 
-	for (i = 0; i < numres; i ++)
-		pci_update_resource(dev, &dev->resource[i], i);
+	for (i = 0; i < numres; i++)
+		pci_update_resource(dev, i);
 }
 
 static struct pci_platform_pm_ops *pci_platform_pm;
diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c
index 2dbd96c..b7ca679 100644
--- a/drivers/pci/setup-res.c
+++ b/drivers/pci/setup-res.c
@@ -26,11 +26,12 @@
 #include "pci.h"
 
 
-void pci_update_resource(struct pci_dev *dev, struct resource *res, int resno)
+void pci_update_resource(struct pci_dev *dev, int resno)
 {
 	struct pci_bus_region region;
 	u32 new, check, mask;
 	int reg;
+	struct resource *res = dev->resource + resno;
 
 	/*
 	 * Ignore resources for unimplemented BARs and unused resource slots
@@ -162,7 +163,7 @@ int pci_assign_resource(struct pci_dev *dev, int resno)
 	} else {
 		res->flags &= ~IORESOURCE_STARTALIGN;
 		if (resno < PCI_BRIDGE_RESOURCES)
-			pci_update_resource(dev, res, resno);
+			pci_update_resource(dev, resno);
 	}
 
 	return ret;
@@ -197,7 +198,7 @@ int pci_assign_resource_fixed(struct pci_dev *dev, int
resno)
 		dev_err(&dev->dev, "BAR %d: can't allocate %s resource
%pR\n",
 			resno, res->flags & IORESOURCE_IO ? "I/O" :
"mem", res);
 	} else if (resno < PCI_BRIDGE_RESOURCES) {
-		pci_update_resource(dev, res, resno);
+		pci_update_resource(dev, resno);
 	}
 
 	return ret;
diff --git a/include/linux/pci.h b/include/linux/pci.h
index feb4657..7e7ff03 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -642,7 +642,7 @@ int pcie_get_readrq(struct pci_dev *dev);
 int pcie_set_readrq(struct pci_dev *dev, int rq);
 int pci_reset_function(struct pci_dev *dev);
 int pci_execute_reset_function(struct pci_dev *dev);
-void pci_update_resource(struct pci_dev *dev, struct resource *res, int resno);
+void pci_update_resource(struct pci_dev *dev, int resno);
 int __must_check pci_assign_resource(struct pci_dev *dev, int i);
 int pci_select_bars(struct pci_dev *dev, unsigned long flags);
 
-- 
1.5.6.4

Yu Zhao

2008-Nov-21 18:39 UTC

head link

[PATCH 3/13 v7] PCI: define PCI resource names in an 'enum'

This patch moves all definitions of the PCI resource names to an 'enum',
and also replaces some hard-coded resource variables with symbol
names. This change eases introduction of device specific resources.

Signed-off-by: Yu Zhao <yu.zhao at intel.com>

---
 drivers/pci/pci-sysfs.c |    4 +++-
 drivers/pci/probe.c     |    2 +-
 drivers/pci/proc.c      |    7 ++++---
 include/linux/pci.h     |   37 ++++++++++++++++++++++++-------------
 4 files changed, 32 insertions(+), 18 deletions(-)

diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c
index 5d72866..0d74851 100644
--- a/drivers/pci/pci-sysfs.c
+++ b/drivers/pci/pci-sysfs.c
@@ -101,11 +101,13 @@ resource_show(struct device * dev, struct device_attribute
*attr, char * buf)
 	struct pci_dev * pci_dev = to_pci_dev(dev);
 	char * str = buf;
 	int i;
-	int max = 7;
+	int max;
 	resource_size_t start, end;
 
 	if (pci_dev->subordinate)
 		max = DEVICE_COUNT_RESOURCE;
+	else
+		max = PCI_BRIDGE_RESOURCES;
 
 	for (i = 0; i < max; i++) {
 		struct resource *res =  &pci_dev->resource[i];
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 003a9b3..4c5429f 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -423,7 +423,7 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus
*parent,
 	child->subordinate = 0xff;
 
 	/* Set up default resource pointers and names.. */
-	for (i = 0; i < 4; i++) {
+	for (i = 0; i < PCI_BRIDGE_RESOURCE_NUM; i++) {
 		child->resource[i] = &bridge->resource[PCI_BRIDGE_RESOURCES+i];
 		child->resource[i]->name = child->name;
 	}
diff --git a/drivers/pci/proc.c b/drivers/pci/proc.c
index e1098c3..f6f2a59 100644
--- a/drivers/pci/proc.c
+++ b/drivers/pci/proc.c
@@ -352,15 +352,16 @@ static int show_device(struct seq_file *m, void *v)
 			dev->vendor,
 			dev->device,
 			dev->irq);
-	/* Here should be 7 and not PCI_NUM_RESOURCES as we need to preserve
compatibility */
-	for (i=0; i<7; i++) {
+
+	/* only print standard and ROM resources to preserve compatibility */
+	for (i = 0; i <= PCI_ROM_RESOURCE; i++) {
 		resource_size_t start, end;
 		pci_resource_to_user(dev, i, &dev->resource[i], &start, &end);
 		seq_printf(m, "\t%16llx",
 			(unsigned long long)(start |
 			(dev->resource[i].flags & PCI_REGION_FLAG_MASK)));
 	}
-	for (i=0; i<7; i++) {
+	for (i = 0; i <= PCI_ROM_RESOURCE; i++) {
 		resource_size_t start, end;
 		pci_resource_to_user(dev, i, &dev->resource[i], &start, &end);
 		seq_printf(m, "\t%16llx",
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 7e7ff03..d455ec8 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -82,7 +82,30 @@ enum pci_mmap_state {
 #define PCI_DMA_FROMDEVICE	2
 #define PCI_DMA_NONE		3
 
-#define DEVICE_COUNT_RESOURCE	12
+/*
+ *  For PCI devices, the region numbers are assigned this way:
+ */
+enum {
+	/* #0-5: standard PCI resources */
+	PCI_STD_RESOURCES,
+	PCI_STD_RESOURCE_END = 5,
+
+	/* #6: expansion ROM resource */
+	PCI_ROM_RESOURCE,
+
+	/* resources assigned to buses behind the bridge */
+#define PCI_BRIDGE_RESOURCE_NUM 4
+
+	PCI_BRIDGE_RESOURCES,
+	PCI_BRIDGE_RESOURCE_END = PCI_BRIDGE_RESOURCES +
+				  PCI_BRIDGE_RESOURCE_NUM - 1,
+
+	/* total resources associated with a PCI device */
+	PCI_NUM_RESOURCES,
+
+	/* preserve this for compatibility */
+	DEVICE_COUNT_RESOURCE
+};
 
 typedef int __bitwise pci_power_t;
 
@@ -268,18 +291,6 @@ static inline void pci_add_saved_cap(struct pci_dev
*pci_dev,
 	hlist_add_head(&new_cap->next, &pci_dev->saved_cap_space);
 }
 
-/*
- *  For PCI devices, the region numbers are assigned this way:
- *
- *	0-5	standard PCI regions
- *	6	expansion ROM
- *	7-10	bridges: address space assigned to buses behind the bridge
- */
-
-#define PCI_ROM_RESOURCE	6
-#define PCI_BRIDGE_RESOURCES	7
-#define PCI_NUM_RESOURCES	11
-
 #ifndef PCI_BUS_NUM_RESOURCES
 #define PCI_BUS_NUM_RESOURCES	16
 #endif
-- 
1.5.6.4

Yu Zhao

2008-Nov-21 18:40 UTC

head link

[PATCH 4/13 v7] PCI: remove unnecessary condition check in pci_restore_bars()

Remove the unnecessary number of resources condition checks because
the pci_update_resource() will check availability of the resources.

Signed-off-by: Yu Zhao <yu.zhao at intel.com>

---
 drivers/pci/pci.c |   19 ++-----------------
 1 files changed, 2 insertions(+), 17 deletions(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index c408be8..9d3f793 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -360,24 +360,9 @@ pci_find_parent_resource(const struct pci_dev *dev, struct
resource *res)
 static void
 pci_restore_bars(struct pci_dev *dev)
 {
-	int i, numres;
-
-	switch (dev->hdr_type) {
-	case PCI_HEADER_TYPE_NORMAL:
-		numres = 6;
-		break;
-	case PCI_HEADER_TYPE_BRIDGE:
-		numres = 2;
-		break;
-	case PCI_HEADER_TYPE_CARDBUS:
-		numres = 1;
-		break;
-	default:
-		/* Should never get here, but just in case... */
-		return;
-	}
+	int i;
 
-	for (i = 0; i < numres; i++)
+	for (i = 0; i < PCI_BRIDGE_RESOURCES; i++)
 		pci_update_resource(dev, i);
 }
 
-- 
1.5.6.4

Yu Zhao

2008-Nov-21 18:40 UTC

head link

[PATCH 5/13 v7] PCI: export __pci_read_base()

Export __pci_read_base() so it can be used by whole PCI subsystem.

Signed-off-by: Yu Zhao <yu.zhao at intel.com>

---
 drivers/pci/pci.h   |    9 +++++++++
 drivers/pci/probe.c |   20 +++++++++-----------
 2 files changed, 18 insertions(+), 11 deletions(-)

diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 1449884..fd0d087 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -159,6 +159,15 @@ struct pci_slot_attribute {
 };
 #define to_pci_slot_attr(s) container_of(s, struct pci_slot_attribute, attr)
 
+enum pci_bar_type {
+	pci_bar_unknown,	/* Standard PCI BAR probe */
+	pci_bar_io,		/* An io port BAR */
+	pci_bar_mem32,		/* A 32-bit memory BAR */
+	pci_bar_mem64,		/* A 64-bit memory BAR */
+};
+
+extern int __pci_read_base(struct pci_dev *dev, enum pci_bar_type type,
+				struct resource *res, unsigned int reg);
 extern void pci_enable_ari(struct pci_dev *dev);
 /**
  * pci_ari_enabled - query ARI forwarding status
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 4c5429f..ae5c7fe 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -135,13 +135,6 @@ static u64 pci_size(u64 base, u64 maxbase, u64 mask)
 	return size;
 }
 
-enum pci_bar_type {
-	pci_bar_unknown,	/* Standard PCI BAR probe */
-	pci_bar_io,		/* An io port BAR */
-	pci_bar_mem32,		/* A 32-bit memory BAR */
-	pci_bar_mem64,		/* A 64-bit memory BAR */
-};
-
 static inline enum pci_bar_type decode_bar(struct resource *res, u32 bar)
 {
 	if ((bar & PCI_BASE_ADDRESS_SPACE) == PCI_BASE_ADDRESS_SPACE_IO) {
@@ -156,11 +149,16 @@ static inline enum pci_bar_type decode_bar(struct resource
*res, u32 bar)
 	return pci_bar_mem32;
 }
 
-/*
- * If the type is not unknown, we assume that the lowest bit is
'enable'.
- * Returns 1 if the BAR was 64-bit and 0 if it was 32-bit.
+/**
+ * pci_read_base - read a PCI BAR
+ * @dev: the PCI device
+ * @type: type of the BAR
+ * @res: resource buffer to be filled in
+ * @pos: BAR position in the config space
+ *
+ * Returns 1 if the BAR is 64-bit, or 0 if 32-bit.
  */
-static int __pci_read_base(struct pci_dev *dev, enum pci_bar_type type,
+int __pci_read_base(struct pci_dev *dev, enum pci_bar_type type,
 			struct resource *res, unsigned int pos)
 {
 	u32 l, sz, mask;
-- 
1.5.6.4

Yu Zhao

2008-Nov-21 18:41 UTC

head link

[PATCH 6/13 v7] PCI: make pci_alloc_child_bus() be able to handle NULL bridge

Make pci_alloc_child_bus() be able to allocate buses without bridge
devices. Some SR-IOV devices can occupy more than one bus number,
but there is no explicit bridges because that have internal routing
mechanism.

Signed-off-by: Yu Zhao <yu.zhao at intel.com>

---
 drivers/pci/probe.c |    8 ++++++--
 1 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index ae5c7fe..cd205fd 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -398,12 +398,10 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus
*parent,
 	if (!child)
 		return NULL;
 
-	child->self = bridge;
 	child->parent = parent;
 	child->ops = parent->ops;
 	child->sysdata = parent->sysdata;
 	child->bus_flags = parent->bus_flags;
-	child->bridge = get_device(&bridge->dev);
 
 	/* initialize some portions of the bus device, but don't register it
 	 * now as the parent is not properly set up yet.  This device will get
@@ -420,6 +418,12 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus
*parent,
 	child->primary = parent->secondary;
 	child->subordinate = 0xff;
 
+	if (!bridge)
+		return child;
+
+	child->self = bridge;
+	child->bridge = get_device(&bridge->dev);
+
 	/* Set up default resource pointers and names.. */
 	for (i = 0; i < PCI_BRIDGE_RESOURCE_NUM; i++) {
 		child->resource[i] = &bridge->resource[PCI_BRIDGE_RESOURCES+i];
-- 
1.5.6.4

Yu Zhao

2008-Nov-21 18:41 UTC

head link

[PATCH 7/13 v7] PCI: add a new function to map BAR offset

Add a function to map resource number to corresponding register so
people can get the offset and type of device specific BARs.

Signed-off-by: Yu Zhao <yu.zhao at intel.com>

---
 drivers/pci/pci.c       |   22 ++++++++++++++++++++++
 drivers/pci/pci.h       |    2 ++
 drivers/pci/setup-res.c |   13 +++++--------
 3 files changed, 29 insertions(+), 8 deletions(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 9d3f793..9382b5f 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -2007,6 +2007,28 @@ int pci_select_bars(struct pci_dev *dev, unsigned long
flags)
 	return bars;
 }
 
+/**
+ * pci_resource_bar - get position of the BAR associated with a resource
+ * @dev: the PCI device
+ * @resno: the resource number
+ * @type: the BAR type to be filled in
+ *
+ * Returns BAR position in config space, or 0 if the BAR is invalid.
+ */
+int pci_resource_bar(struct pci_dev *dev, int resno, enum pci_bar_type *type)
+{
+	if (resno < PCI_ROM_RESOURCE) {
+		*type = pci_bar_unknown;
+		return PCI_BASE_ADDRESS_0 + 4 * resno;
+	} else if (resno == PCI_ROM_RESOURCE) {
+		*type = pci_bar_mem32;
+		return dev->rom_base_reg;
+	}
+
+	dev_err(&dev->dev, "BAR: invalid resource #%d\n", resno);
+	return 0;
+}
+
 static void __devinit pci_no_domains(void)
 {
 #ifdef CONFIG_PCI_DOMAINS
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index fd0d087..3de70d7 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -168,6 +168,8 @@ enum pci_bar_type {
 
 extern int __pci_read_base(struct pci_dev *dev, enum pci_bar_type type,
 				struct resource *res, unsigned int reg);
+extern int pci_resource_bar(struct pci_dev *dev, int resno,
+			    enum pci_bar_type *type);
 extern void pci_enable_ari(struct pci_dev *dev);
 /**
  * pci_ari_enabled - query ARI forwarding status
diff --git a/drivers/pci/setup-res.c b/drivers/pci/setup-res.c
index b7ca679..854d43e 100644
--- a/drivers/pci/setup-res.c
+++ b/drivers/pci/setup-res.c
@@ -31,6 +31,7 @@ void pci_update_resource(struct pci_dev *dev, int resno)
 	struct pci_bus_region region;
 	u32 new, check, mask;
 	int reg;
+	enum pci_bar_type type;
 	struct resource *res = dev->resource + resno;
 
 	/*
@@ -62,17 +63,13 @@ void pci_update_resource(struct pci_dev *dev, int resno)
 	else
 		mask = (u32)PCI_BASE_ADDRESS_MEM_MASK;
 
-	if (resno < 6) {
-		reg = PCI_BASE_ADDRESS_0 + 4 * resno;
-	} else if (resno == PCI_ROM_RESOURCE) {
+	reg = pci_resource_bar(dev, resno, &type);
+	if (!reg)
+		return;
+	if (type != pci_bar_unknown) {
 		if (!(res->flags & IORESOURCE_ROM_ENABLE))
 			return;
 		new |= PCI_ROM_ADDRESS_ENABLE;
-		reg = dev->rom_base_reg;
-	} else {
-		/* Hmm, non-standard resource. */
-	
-		return;		/* kill uninitialised var warning */
 	}
 
 	pci_write_config_dword(dev, reg, new);
-- 
1.5.6.4

Yu Zhao

2008-Nov-21 18:41 UTC

head link

[PATCH 8/13 v7] PCI: cleanup pci_bus_add_devices()

This cleanup makes pci_bus_add_devices() easier to read.

Signed-off-by: Yu Zhao <yu.zhao at intel.com>

---
 drivers/pci/bus.c |   55 +++++++++++++++++++++++++++--------------------------
 1 files changed, 28 insertions(+), 27 deletions(-)

diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
index 999cc40..9d800cb 100644
--- a/drivers/pci/bus.c
+++ b/drivers/pci/bus.c
@@ -71,7 +71,7 @@ pci_bus_alloc_resource(struct pci_bus *bus, struct resource
*res,
 }
 
 /**
- * add a single device
+ * pci_bus_add_device - add a single device
  * @dev: device to add
  *
  * This adds a single pci device to the global
@@ -105,7 +105,7 @@ int pci_bus_add_device(struct pci_dev *dev)
 void pci_bus_add_devices(struct pci_bus *bus)
 {
 	struct pci_dev *dev;
-	struct pci_bus *child_bus;
+	struct pci_bus *child;
 	int retval;
 
 	list_for_each_entry(dev, &bus->devices, bus_list) {
@@ -120,39 +120,40 @@ void pci_bus_add_devices(struct pci_bus *bus)
 	list_for_each_entry(dev, &bus->devices, bus_list) {
 		BUG_ON(!dev->is_added);
 
+		child = dev->subordinate;
 		/*
 		 * If there is an unattached subordinate bus, attach
 		 * it and then scan for unattached PCI devices.
 		 */
-		if (dev->subordinate) {
-		       if (list_empty(&dev->subordinate->node)) {
-			       down_write(&pci_bus_sem);
-			       list_add_tail(&dev->subordinate->node,
-					       &dev->bus->children);
-			       up_write(&pci_bus_sem);
-			}
-			pci_bus_add_devices(dev->subordinate);
-
-			/* register the bus with sysfs as the parent is now
-			 * properly registered. */
-			child_bus = dev->subordinate;
-			if (child_bus->is_added)
-				continue;
-			child_bus->dev.parent = child_bus->bridge;
-			retval = device_register(&child_bus->dev);
-			if (retval)
-				dev_err(&dev->dev, "Error registering pci_bus,"
-					" continuing...\n");
-			else {
-				child_bus->is_added = 1;
-				retval = device_create_file(&child_bus->dev,
-							&dev_attr_cpuaffinity);
-			}
+		if (!child)
+			continue;
+		if (list_empty(&child->node)) {
+			down_write(&pci_bus_sem);
+			list_add_tail(&child->node, &dev->bus->children);
+			up_write(&pci_bus_sem);
+		}
+		pci_bus_add_devices(child);
+
+		/*
+		 * register the bus with sysfs as the parent is now
+		 * properly registered.
+		 */
+		if (child->is_added)
+			continue;
+		child->dev.parent = child->bridge;
+		retval = device_register(&child->dev);
+		if (retval)
+			dev_err(&dev->dev, "Error registering pci_bus,"
+				" continuing...\n");
+		else {
+			child->is_added = 1;
+			retval = device_create_file(&child->dev,
+						&dev_attr_cpuaffinity);
 			if (retval)
 				dev_err(&dev->dev, "Error creating cpuaffinity"
 					" file, continuing...\n");
 
-			retval = device_create_file(&child_bus->dev,
+			retval = device_create_file(&child->dev,
 						&dev_attr_cpulistaffinity);
 			if (retval)
 				dev_err(&dev->dev,
-- 
1.5.6.4

Yu Zhao

2008-Nov-21 18:42 UTC

head link

[PATCH 9/13 v7] PCI: split a new function from pci_bus_add_devices()

This patch splits a new function from pci_bus_add_devices(). The new
function can be used to register PCI bus to the device core.

Signed-off-by: Yu Zhao <yu.zhao at intel.com>

---
 drivers/pci/bus.c |   49 ++++++++++++++++++++++++++++++-------------------
 drivers/pci/pci.h |    1 +
 2 files changed, 31 insertions(+), 19 deletions(-)

diff --git a/drivers/pci/bus.c b/drivers/pci/bus.c
index 9d800cb..65f5a6f 100644
--- a/drivers/pci/bus.c
+++ b/drivers/pci/bus.c
@@ -91,6 +91,34 @@ int pci_bus_add_device(struct pci_dev *dev)
 }
 
 /**
+ * pci_bus_add_child - add a child bus
+ * @bus: bus to add
+ *
+ * This adds sysfs entries for a single bus
+ */
+int pci_bus_add_child(struct pci_bus *bus)
+{
+	int retval;
+
+	if (bus->bridge)
+		bus->dev.parent = bus->bridge;
+
+	retval = device_register(&bus->dev);
+	if (retval)
+		return retval;
+
+	bus->is_added = 1;
+
+	retval = device_create_file(&bus->dev, &dev_attr_cpuaffinity);
+	if (retval)
+		return retval;
+
+	retval = device_create_file(&bus->dev, &dev_attr_cpulistaffinity);
+
+	return retval;
+}
+
+/**
  * pci_bus_add_devices - insert newly discovered PCI devices
  * @bus: bus to check for new devices
  *
@@ -140,26 +168,9 @@ void pci_bus_add_devices(struct pci_bus *bus)
 		 */
 		if (child->is_added)
 			continue;
-		child->dev.parent = child->bridge;
-		retval = device_register(&child->dev);
+		retval = pci_bus_add_child(child);
 		if (retval)
-			dev_err(&dev->dev, "Error registering pci_bus,"
-				" continuing...\n");
-		else {
-			child->is_added = 1;
-			retval = device_create_file(&child->dev,
-						&dev_attr_cpuaffinity);
-			if (retval)
-				dev_err(&dev->dev, "Error creating cpuaffinity"
-					" file, continuing...\n");
-
-			retval = device_create_file(&child->dev,
-						&dev_attr_cpulistaffinity);
-			if (retval)
-				dev_err(&dev->dev,
-					"Error creating cpulistaffinity"
-					" file, continuing...\n");
-		}
+			dev_err(&dev->dev, "Error adding bus, continuing\n");
 	}
 }
 
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 3de70d7..315bbe6 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -170,6 +170,7 @@ extern int __pci_read_base(struct pci_dev *dev, enum
pci_bar_type type,
 				struct resource *res, unsigned int reg);
 extern int pci_resource_bar(struct pci_dev *dev, int resno,
 			    enum pci_bar_type *type);
+extern int pci_bus_add_child(struct pci_bus *bus);
 extern void pci_enable_ari(struct pci_dev *dev);
 /**
  * pci_ari_enabled - query ARI forwarding status
-- 
1.5.6.4

Yu Zhao

2008-Nov-21 18:42 UTC

head link

[PATCH 10/13 v7] PCI: support the SR-IOV capability

Support Single Root I/O Virtualization (SR-IOV) capability.

Signed-off-by: Yu Zhao <yu.zhao at intel.com>

---
 drivers/pci/Kconfig      |   13 ++
 drivers/pci/Makefile     |    3 +
 drivers/pci/iov.c        |  491 ++++++++++++++++++++++++++++++++++++++++++++++
 drivers/pci/pci-driver.c |   12 +-
 drivers/pci/pci.c        |    8 +
 drivers/pci/pci.h        |   51 +++++
 drivers/pci/probe.c      |    4 +
 include/linux/pci.h      |    9 +
 include/linux/pci_regs.h |   21 ++
 9 files changed, 610 insertions(+), 2 deletions(-)
 create mode 100644 drivers/pci/iov.c

diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig
index e1ca425..493233e 100644
--- a/drivers/pci/Kconfig
+++ b/drivers/pci/Kconfig
@@ -50,3 +50,16 @@ config HT_IRQ
 	   This allows native hypertransport devices to use interrupts.
 
 	   If unsure say Y.
+
+config PCI_IOV
+	bool "PCI IOV support"
+	depends on PCI
+	select PCI_MSI
+	default n
+	help
+	  PCI-SIG I/O Virtualization (IOV) Specifications support.
+	  Single Root IOV: allows the Physical Function device driver
+	  to enable the hardware capability, so the Virtual Function
+	  is accessible via the PCI configuration space using its own
+	  Bus, Device and Function Number. Each Virtual Function also
+	  has PCI Memory Space to map its own register set.
diff --git a/drivers/pci/Makefile b/drivers/pci/Makefile
index af3bfe2..8c7c12d 100644
--- a/drivers/pci/Makefile
+++ b/drivers/pci/Makefile
@@ -29,6 +29,9 @@ obj-$(CONFIG_DMAR) += dmar.o iova.o intel-iommu.o
 
 obj-$(CONFIG_INTR_REMAP) += dmar.o intr_remapping.o
 
+# PCI IOV support
+obj-$(CONFIG_PCI_IOV) += iov.o
+
 #
 # Some architectures use the generic PCI setup functions
 #
diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
new file mode 100644
index 0000000..03f62ca
--- /dev/null
+++ b/drivers/pci/iov.c
@@ -0,0 +1,491 @@
+/*
+ * drivers/pci/iov.c
+ *
+ * Copyright (C) 2008 Intel Corporation
+ *
+ * PCI Express I/O Virtualization (IOV) support.
+ *   Single Root IOV 1.0
+ */
+
+#include <linux/ctype.h>
+#include <linux/string.h>
+#include <linux/pci.h>
+#include <linux/delay.h>
+#include <asm/page.h>
+#include "pci.h"
+
+
+#define pci_iov_attr(field)					\
+static ssize_t iov_##field##_show(struct device *dev,		\
+		struct device_attribute *attr, char *buf)	\
+{								\
+	struct pci_dev *pdev = to_pci_dev(dev);			\
+	return sprintf(buf, "%d\n", pdev->iov->field);		\
+}
+
+pci_iov_attr(total);
+pci_iov_attr(initial);
+pci_iov_attr(nr_virtfn);
+
+static inline void virtfn_bdf(struct pci_dev *dev, int id, u8 *busnr, u8
*devfn)
+{
+	u16 bdf;
+
+	bdf = (dev->bus->number << 8) + dev->devfn +
+		dev->iov->offset + dev->iov->stride * id;
+	*busnr = bdf >> 8;
+	*devfn = bdf & 0xff;
+}
+
+static int virtfn_add(struct pci_dev *dev, int id)
+{
+	int i;
+	int rc;
+	u8 busnr, devfn;
+	struct pci_dev *virtfn;
+	struct resource *res;
+	resource_size_t size;
+
+	virtfn_bdf(dev, id, &busnr, &devfn);
+
+	virtfn = alloc_pci_dev();
+	if (!virtfn)
+		return -ENOMEM;
+
+	virtfn->bus = pci_find_bus(pci_domain_nr(dev->bus), busnr);
+	BUG_ON(!virtfn->bus);
+	virtfn->sysdata = dev->bus->sysdata;
+	virtfn->dev.parent = dev->dev.parent;
+	virtfn->dev.bus = dev->dev.bus;
+	virtfn->devfn = devfn;
+	virtfn->hdr_type = PCI_HEADER_TYPE_NORMAL;
+	virtfn->multifunction = 0;
+	virtfn->vendor = dev->vendor;
+	pci_read_config_word(dev, dev->iov->cap + PCI_IOV_VF_DID,
+			     &virtfn->device);
+	virtfn->cfg_size = PCI_CFG_SPACE_EXP_SIZE;
+	virtfn->error_state = pci_channel_io_normal;
+	virtfn->is_pcie = 1;
+	virtfn->pcie_type = PCI_EXP_TYPE_ENDPOINT;
+	virtfn->dma_mask = 0xffffffff;
+
+	dev_set_name(&virtfn->dev, "%04x:%02x:%02x.%d",
+		     pci_domain_nr(virtfn->bus), busnr,
+		     PCI_SLOT(devfn), PCI_FUNC(devfn));
+
+	pci_read_config_byte(virtfn, PCI_REVISION_ID, &virtfn->revision);
+	virtfn->class = dev->class;
+	virtfn->current_state = PCI_UNKNOWN;
+	virtfn->irq = 0;
+
+	for (i = 0; i < PCI_IOV_NUM_BAR; i++) {
+		res = dev->resource + PCI_IOV_RESOURCES + i;
+		if (!res->parent)
+			continue;
+		virtfn->resource[i].name = pci_name(virtfn);
+		virtfn->resource[i].flags = res->flags;
+		size = resource_size(res);
+		do_div(size, dev->iov->total);
+		virtfn->resource[i].start = res->start + size * id;
+		virtfn->resource[i].end = virtfn->resource[i].start + size - 1;
+		rc = request_resource(res, &virtfn->resource[i]);
+		BUG_ON(rc);
+	}
+
+	virtfn->subsystem_vendor = dev->subsystem_vendor;
+	pci_read_config_word(virtfn, PCI_SUBSYSTEM_ID,
+			     &virtfn->subsystem_device);
+
+	pci_device_add(virtfn, virtfn->bus);
+	rc = pci_bus_add_device(virtfn);
+
+	return rc;
+}
+
+static void virtfn_remove(struct pci_dev *dev, int id)
+{
+	u8 busnr, devfn;
+	struct pci_bus *bus;
+	struct pci_dev *virtfn;
+
+	virtfn_bdf(dev, id, &busnr, &devfn);
+
+	bus = pci_find_bus(pci_domain_nr(dev->bus), busnr);
+	BUG_ON(!bus);
+	virtfn = pci_get_slot(bus, devfn);
+	BUG_ON(!virtfn);
+	pci_dev_put(virtfn);
+	pci_remove_bus_device(virtfn);
+}
+
+static int iov_add_bus(struct pci_bus *bus, int busnr)
+{
+	int i;
+	int rc;
+	struct pci_bus *child;
+
+	for (i = bus->number + 1; i <= busnr; i++) {
+		child = pci_find_bus(pci_domain_nr(bus), i);
+		if (child)
+			continue;
+		child = pci_add_new_bus(bus, NULL, i);
+		if (!child)
+			return -ENOMEM;
+
+		child->subordinate = i;
+		child->dev.parent = bus->bridge;
+		rc = pci_bus_add_child(child);
+		if (rc)
+			return rc;
+	}
+
+	return 0;
+}
+
+static void iov_remove_bus(struct pci_bus *bus, int busnr)
+{
+	int i;
+	struct pci_bus *child;
+
+	for (i = bus->number + 1; i <= busnr; i++) {
+		child = pci_find_bus(pci_domain_nr(bus), i);
+		BUG_ON(!child);
+		if (list_empty(&child->devices))
+			pci_remove_bus(child);
+	}
+}
+
+static int iov_enable(struct pci_dev *dev, int nr_virtfn)
+{
+	int i, j;
+	int rc;
+	u8 busnr, devfn;
+	u16 ctrl, offset, stride;
+
+	pci_write_config_word(dev, dev->iov->cap + PCI_IOV_NUM_VF, nr_virtfn);
+	pci_read_config_word(dev, dev->iov->cap + PCI_IOV_VF_OFFSET,
&offset);
+	pci_read_config_word(dev, dev->iov->cap + PCI_IOV_VF_STRIDE,
&stride);
+
+	if (!offset || (nr_virtfn > 1 && !stride))
+		return -EIO;
+
+	dev->iov->offset = offset;
+	dev->iov->stride = stride;
+
+	virtfn_bdf(dev, nr_virtfn - 1, &busnr, &devfn);
+	if (busnr > dev->bus->subordinate)
+		return -EIO;
+
+	rc = dev->driver->virtual(dev, nr_virtfn);
+	if (rc)
+		return rc;
+
+	pci_read_config_word(dev, dev->iov->cap + PCI_IOV_CTRL, &ctrl);
+	ctrl |= PCI_IOV_CTRL_VFE | PCI_IOV_CTRL_MSE;
+	pci_write_config_word(dev, dev->iov->cap + PCI_IOV_CTRL, ctrl);
+	ssleep(1);
+
+	iov_add_bus(dev->bus, busnr);
+	for (i = 0; i < nr_virtfn; i++) {
+		rc = virtfn_add(dev, i);
+		if (rc)
+			goto failed;
+	}
+
+	dev->iov->nr_virtfn = nr_virtfn;
+
+	return 0;
+
+failed:
+	for (j = 0; j < i; j++)
+		virtfn_remove(dev, j);
+
+	iov_remove_bus(dev->bus, busnr);
+
+	ctrl &= ~(PCI_IOV_CTRL_VFE | PCI_IOV_CTRL_MSE);
+	pci_write_config_word(dev, dev->iov->cap + PCI_IOV_CTRL, ctrl);
+	ssleep(1);
+
+	return rc;
+}
+
+static void iov_disable(struct pci_dev *dev)
+{
+	int i;
+	int rc;
+	u16 ctrl;
+	u8 busnr, devfn;
+
+	if (!dev->iov->nr_virtfn)
+		return;
+
+	rc = dev->driver->virtual(dev, 0);
+	if (rc)
+		return;
+
+	for (i = 0; i < dev->iov->nr_virtfn; i++)
+		virtfn_remove(dev, i);
+
+	virtfn_bdf(dev, dev->iov->nr_virtfn - 1, &busnr, &devfn);
+	iov_remove_bus(dev->bus, busnr);
+
+	pci_read_config_word(dev, dev->iov->cap + PCI_IOV_CTRL, &ctrl);
+	ctrl &= ~(PCI_IOV_CTRL_VFE | PCI_IOV_CTRL_MSE);
+	pci_write_config_word(dev, dev->iov->cap + PCI_IOV_CTRL, ctrl);
+	ssleep(1);
+
+	dev->iov->nr_virtfn = 0;
+}
+
+static ssize_t iov_set_nr_virtfn(struct device *dev,
+			struct device_attribute *attr,
+			const char *buf, size_t count)
+{
+	int rc;
+	long nr_virtfn;
+	struct pci_dev *pdev = to_pci_dev(dev);
+
+	rc = strict_strtol(buf, 0, &nr_virtfn);
+	if (rc)
+		return rc;
+
+	if (nr_virtfn < 0 || nr_virtfn > pdev->iov->initial)
+		return -EINVAL;
+
+	if (nr_virtfn == pdev->iov->nr_virtfn)
+		return count;
+
+	mutex_lock(&pdev->iov->physfn->iov->lock);
+	iov_disable(pdev);
+
+	if (nr_virtfn)
+		rc = iov_enable(pdev, nr_virtfn);
+	mutex_unlock(&pdev->iov->physfn->iov->lock);
+
+	return rc ? rc : count;
+}
+
+static DEVICE_ATTR(total_virtfn, S_IRUGO, iov_total_show, NULL);
+static DEVICE_ATTR(initial_virtfn, S_IRUGO, iov_initial_show, NULL);
+static DEVICE_ATTR(nr_virtfn, S_IWUSR | S_IRUGO,
+		   iov_nr_virtfn_show, iov_set_nr_virtfn);
+
+static struct attribute *iov_attrs[] = {
+	&dev_attr_total_virtfn.attr,
+	&dev_attr_initial_virtfn.attr,
+	&dev_attr_nr_virtfn.attr,
+	NULL
+};
+
+static struct attribute_group iov_attr_group = {
+	.attrs = iov_attrs,
+	.name = "iov",
+};
+
+/**
+ * pci_iov_init - initialize device's SR-IOV capability
+ * @dev: the PCI device
+ *
+ * Returns 0 on success, or negative on failure.
+ *
+ * The major differences between Virtual Function and PCI device are:
+ * 1) the device with multiple bus numbers uses internal routing, so
+ *    there is no explicit bridge device in this case.
+ * 2) Virtual Function memory spaces are designated by BARs encapsulated
+ *    in the capability structure, and the BARs in Virtual Function PCI
+ *    configuration space are read-only zero.
+ */
+int pci_iov_init(struct pci_dev *dev)
+{
+	int i;
+	int pos;
+	u32 pgsz;
+	u16 ctrl, total, initial, offset, stride;
+	struct pci_iov *iov;
+	struct resource *res;
+	struct pci_dev *physfn;
+
+	if (!dev->is_pcie)
+		return -ENODEV;
+
+	if (dev->pcie_type != PCI_EXP_TYPE_RC_END &&
+	    dev->pcie_type != PCI_EXP_TYPE_ENDPOINT)
+		return -ENODEV;
+
+	pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_IOV);
+	if (!pos)
+		return -ENODEV;
+
+	pci_read_config_word(dev, pos + PCI_IOV_CTRL, &ctrl);
+	if (ctrl & PCI_IOV_CTRL_VFE) {
+		pci_write_config_word(dev, pos + PCI_IOV_CTRL, 0);
+		ssleep(1);
+	}
+
+	physfn = NULL;
+	if (!list_empty(&dev->bus->devices))
+		list_for_each_entry(physfn, &dev->bus->devices, bus_list)
+			if (physfn->iov)
+				break;
+
+	ctrl = 0;
+	if (!(physfn && physfn->iov) &&
pci_ari_enabled(dev->bus))
+		ctrl |= PCI_IOV_CTRL_ARI;
+
+	pci_write_config_word(dev, pos + PCI_IOV_CTRL, ctrl);
+	pci_read_config_word(dev, pos + PCI_IOV_TOTAL_VF, &total);
+	pci_read_config_word(dev, pos + PCI_IOV_INITIAL_VF, &initial);
+	pci_write_config_word(dev, pos + PCI_IOV_NUM_VF, initial);
+	pci_read_config_word(dev, pos + PCI_IOV_VF_OFFSET, &offset);
+	pci_read_config_word(dev, pos + PCI_IOV_VF_STRIDE, &stride);
+
+	if (!total || initial > total || (initial && !offset) ||
+	    (initial > 1 && !stride))
+		return -EIO;
+
+	pci_read_config_dword(dev, pos + PCI_IOV_SUP_PGSIZE, &pgsz);
+	i = PAGE_SHIFT > 12 ? PAGE_SHIFT - 12 : 0;
+	pgsz &= ~((1 << i) - 1);
+	if (!pgsz)
+		return -EIO;
+
+	pgsz &= ~(pgsz - 1);
+	pci_write_config_dword(dev, pos + PCI_IOV_SYS_PGSIZE, pgsz);
+
+	iov = kzalloc(sizeof(*iov), GFP_KERNEL);
+	if (!iov)
+		return -ENOMEM;
+
+	iov->cap = pos;
+	iov->total = total;
+	iov->initial = initial;
+	iov->offset = offset;
+	iov->stride = stride;
+	iov->pgsz = pgsz;
+
+	for (i = 0; i < PCI_IOV_NUM_BAR; i++) {
+		res = dev->resource + PCI_IOV_RESOURCES + i;
+		pos = iov->cap + PCI_IOV_BAR_0 + i * 4;
+		i += __pci_read_base(dev, pci_bar_unknown, res, pos);
+		if (!res->flags)
+			continue;
+		res->end = res->start + resource_size(res) * total - 1;
+	}
+
+	if (physfn && physfn->iov) {
+		pci_dev_get(physfn);
+		iov->physfn = physfn;
+	} else {
+		mutex_init(&iov->lock);
+		iov->physfn = dev;
+	}
+
+	dev->iov = iov;
+
+	return 0;
+}
+
+/**
+ * pci_iov_release - release resources used by the SR-IOV capability
+ * @dev: the PCI device
+ */
+void pci_iov_release(struct pci_dev *dev)
+{
+	if (!dev->iov)
+		return;
+
+	if (dev == dev->iov->physfn)
+		mutex_destroy(&dev->iov->lock);
+	else
+		pci_dev_put(dev->iov->physfn);
+
+	kfree(dev->iov);
+}
+
+/**
+ * pci_iov_resource_bar - get position of the SR-IOV BAR
+ * @dev: the PCI device
+ * @resno: the resource number
+ * @type: the BAR type to be filled in
+ *
+ * Returns position of the BAR encapsulated in the SR-IOV capability.
+ */
+int pci_iov_resource_bar(struct pci_dev *dev, int resno,
+			 enum pci_bar_type *type)
+{
+	if (resno < PCI_IOV_RESOURCES || resno > PCI_IOV_RESOURCE_END)
+		return 0;
+
+	BUG_ON(!dev->iov);
+
+	*type = pci_bar_unknown;
+	return dev->iov->cap + PCI_IOV_BAR_0 +
+		4 * (resno - PCI_IOV_RESOURCES);
+}
+
+/**
+ * pci_restore_iov_state - restore the state of the SR-IOV capability
+ * @dev: the PCI device
+ */
+void pci_restore_iov_state(struct pci_dev *dev)
+{
+	u16 ctrl;
+
+	if (!dev->iov)
+		return;
+
+	pci_read_config_word(dev, dev->iov->cap + PCI_IOV_CTRL, &ctrl);
+	if (ctrl & PCI_IOV_CTRL_VFE)
+		return;
+
+	pci_write_config_dword(dev, dev->iov->cap + PCI_IOV_SYS_PGSIZE,
+			       dev->iov->pgsz);
+	ctrl = 0;
+	if (dev == dev->iov->physfn && pci_ari_enabled(dev->bus))
+		ctrl |= PCI_IOV_CTRL_ARI;
+	pci_write_config_word(dev, dev->iov->cap + PCI_IOV_CTRL, ctrl);
+
+	if (!dev->iov->nr_virtfn)
+		return;
+
+	pci_write_config_word(dev, dev->iov->cap + PCI_IOV_NUM_VF,
+			      dev->iov->nr_virtfn);
+	ctrl |= PCI_IOV_CTRL_VFE | PCI_IOV_CTRL_MSE;
+	pci_write_config_word(dev, dev->iov->cap + PCI_IOV_CTRL, ctrl);
+
+	ssleep(1);
+}
+
+/**
+ * pci_iov_register - register the SR-IOV capability
+ * @dev: the PCI device
+ */
+int pci_iov_register(struct pci_dev *dev)
+{
+	int rc;
+
+	if (!dev->iov)
+		return -ENODEV;
+
+	rc = sysfs_create_group(&dev->dev.kobj, &iov_attr_group);
+	if (rc)
+		return rc;
+
+	rc = kobject_uevent(&dev->dev.kobj, KOBJ_CHANGE);
+
+	return rc;
+}
+
+/**
+ * pci_iov_unregister - unregister the SR-IOV capability
+ * @dev: the PCI device
+ */
+void pci_iov_unregister(struct pci_dev *dev)
+{
+	if (!dev->iov)
+		return;
+
+	sysfs_remove_group(&dev->dev.kobj, &iov_attr_group);
+	iov_disable(dev);
+	kobject_uevent(&dev->dev.kobj, KOBJ_CHANGE);
+}
diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c
index b4cdd69..3d5f3a3 100644
--- a/drivers/pci/pci-driver.c
+++ b/drivers/pci/pci-driver.c
@@ -234,6 +234,8 @@ __pci_device_probe(struct pci_driver *drv, struct pci_dev
*pci_dev)
 			error = pci_call_probe(drv, pci_dev, id);
 		if (error >= 0) {
 			pci_dev->driver = drv;
+			if (drv->virtual)
+				pci_iov_register(pci_dev);
 			error = 0;
 		}
 	}
@@ -262,6 +264,8 @@ static int pci_device_remove(struct device * dev)
 	struct pci_driver * drv = pci_dev->driver;
 
 	if (drv) {
+		if (drv->virtual)
+			pci_iov_unregister(pci_dev);
 		if (drv->remove)
 			drv->remove(pci_dev);
 		pci_dev->driver = NULL;
@@ -292,8 +296,12 @@ static void pci_device_shutdown(struct device *dev)
 	struct pci_dev *pci_dev = to_pci_dev(dev);
 	struct pci_driver *drv = pci_dev->driver;
 
-	if (drv && drv->shutdown)
-		drv->shutdown(pci_dev);
+	if (drv) {
+		if (drv->virtual)
+			pci_iov_unregister(pci_dev);
+		if (drv->shutdown)
+			drv->shutdown(pci_dev);
+	}
 	pci_msi_shutdown(pci_dev);
 	pci_msix_shutdown(pci_dev);
 }
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 9382b5f..ca26e53 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -763,6 +763,7 @@ pci_restore_state(struct pci_dev *dev)
 	}
 	pci_restore_pcix_state(dev);
 	pci_restore_msi_state(dev);
+	pci_restore_iov_state(dev);
 
 	return 0;
 }
@@ -2017,12 +2018,19 @@ int pci_select_bars(struct pci_dev *dev, unsigned long
flags)
  */
 int pci_resource_bar(struct pci_dev *dev, int resno, enum pci_bar_type *type)
 {
+	int reg;
+
 	if (resno < PCI_ROM_RESOURCE) {
 		*type = pci_bar_unknown;
 		return PCI_BASE_ADDRESS_0 + 4 * resno;
 	} else if (resno == PCI_ROM_RESOURCE) {
 		*type = pci_bar_mem32;
 		return dev->rom_base_reg;
+	} else if (resno < PCI_BRIDGE_RESOURCES) {
+		/* device specific resource */
+		reg = pci_iov_resource_bar(dev, resno, type);
+		if (reg)
+			return reg;
 	}
 
 	dev_err(&dev->dev, "BAR: invalid resource #%d\n", resno);
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 315bbe6..3113d11 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -183,4 +183,55 @@ static inline int pci_ari_enabled(struct pci_bus *bus)
 	return bus->self && bus->self->ari_enabled;
 }
 
+/* Single Root I/O Virtualization */
+struct pci_iov {
+	int cap;		/* capability position */
+	int status;		/* status of SR-IOV */
+	u16 total;		/* total VFs associated with the PF */
+	u16 initial;		/* initial VFs associated with the PF */
+	u16 nr_virtfn;		/* number of VFs available */
+	u16 offset;		/* first VF Routing ID offset */
+	u16 stride;		/* following VF stride */
+	u32 pgsz;		/* page size for BAR alignment */
+	struct pci_dev *physfn;	/* lowest numbered PF */
+	struct mutex lock;	/* lock for VF bus */
+};
+
+#ifdef CONFIG_PCI_IOV
+extern int pci_iov_init(struct pci_dev *dev);
+extern void pci_iov_release(struct pci_dev *dev);
+extern int pci_iov_resource_bar(struct pci_dev *dev, int resno,
+				enum pci_bar_type *type);
+extern int pci_iov_register(struct pci_dev *dev);
+extern void pci_iov_unregister(struct pci_dev *dev);
+extern void pci_restore_iov_state(struct pci_dev *dev);
+#else
+static inline int pci_iov_init(struct pci_dev *dev)
+{
+	return -EIO;
+}
+static inline void pci_iov_release(struct pci_dev *dev)
+
+{
+}
+
+static inline int pci_iov_resource_bar(struct pci_dev *dev, int resno,
+				       enum pci_bar_type *type)
+{
+	return 0;
+}
+
+static inline int pci_iov_register(struct pci_dev *dev)
+{
+}
+
+static inline void pci_iov_unregister(struct pci_dev *dev)
+{
+}
+
+static inline void pci_restore_iov_state(struct pci_dev *dev)
+{
+}
+#endif /* CONFIG_PCI_IOV */
+
 #endif /* DRIVERS_PCI_H */
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index cd205fd..cb26e64 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -785,6 +785,7 @@ static int pci_setup_device(struct pci_dev * dev)
 static void pci_release_capabilities(struct pci_dev *dev)
 {
 	pci_vpd_release(dev);
+	pci_iov_release(dev);
 }
 
 /**
@@ -968,6 +969,9 @@ static void pci_init_capabilities(struct pci_dev *dev)
 
 	/* Alternative Routing-ID Forwarding */
 	pci_enable_ari(dev);
+
+	/* Single Root I/O Virtualization */
+	pci_iov_init(dev);
 }
 
 void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
diff --git a/include/linux/pci.h b/include/linux/pci.h
index d455ec8..c9046a3 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -93,6 +93,12 @@ enum {
 	/* #6: expansion ROM resource */
 	PCI_ROM_RESOURCE,
 
+	/* device specific resources */
+#ifdef CONFIG_PCI_IOV
+	PCI_IOV_RESOURCES,
+	PCI_IOV_RESOURCE_END = PCI_IOV_RESOURCES + PCI_IOV_NUM_BAR - 1,
+#endif
+
 	/* resources assigned to buses behind the bridge */
 #define PCI_BRIDGE_RESOURCE_NUM 4
 
@@ -171,6 +177,7 @@ struct pci_cap_saved_state {
 
 struct pcie_link_state;
 struct pci_vpd;
+struct pci_iov;
 
 /*
  * The pci_dev structure is used to describe PCI devices.
@@ -259,6 +266,7 @@ struct pci_dev {
 	struct list_head msi_list;
 #endif
 	struct pci_vpd *vpd;
+	struct pci_iov *iov;
 };
 
 extern struct pci_dev *alloc_pci_dev(void);
@@ -426,6 +434,7 @@ struct pci_driver {
 	int  (*resume_early) (struct pci_dev *dev);
 	int  (*resume) (struct pci_dev *dev);	                /* Device woken up */
 	void (*shutdown) (struct pci_dev *dev);
+	int (*virtual) (struct pci_dev *dev, int nr_virtfn);
 	struct pm_ext_ops *pm;
 	struct pci_error_handlers *err_handler;
 	struct device_driver	driver;
diff --git a/include/linux/pci_regs.h b/include/linux/pci_regs.h
index e5effd4..1d1ade2 100644
--- a/include/linux/pci_regs.h
+++ b/include/linux/pci_regs.h
@@ -363,6 +363,7 @@
 #define  PCI_EXP_TYPE_UPSTREAM	0x5	/* Upstream Port */
 #define  PCI_EXP_TYPE_DOWNSTREAM 0x6	/* Downstream Port */
 #define  PCI_EXP_TYPE_PCI_BRIDGE 0x7	/* PCI/PCI-X Bridge */
+#define  PCI_EXP_TYPE_RC_END	0x9	/* Root Complex Integrated Endpoint */
 #define PCI_EXP_FLAGS_SLOT	0x0100	/* Slot implemented */
 #define PCI_EXP_FLAGS_IRQ	0x3e00	/* Interrupt message number */
 #define PCI_EXP_DEVCAP		4	/* Device capabilities */
@@ -436,6 +437,7 @@
 #define PCI_EXT_CAP_ID_DSN	3
 #define PCI_EXT_CAP_ID_PWR	4
 #define PCI_EXT_CAP_ID_ARI	14
+#define PCI_EXT_CAP_ID_IOV	16
 
 /* Advanced Error Reporting */
 #define PCI_ERR_UNCOR_STATUS	4	/* Uncorrectable Error Status */
@@ -553,4 +555,23 @@
 #define  PCI_ARI_CTRL_ACS	0x0002	/* ACS Function Groups Enable */
 #define  PCI_ARI_CTRL_FG(x)	(((x) >> 4) & 7) /* Function Group */
 
+/* Single Root I/O Virtualization */
+#define PCI_IOV_CAP		0x04	/* SR-IOV Capabilities */
+#define PCI_IOV_CTRL		0x08	/* SR-IOV Control */
+#define  PCI_IOV_CTRL_VFE	0x01	/* VF Enable */
+#define  PCI_IOV_CTRL_MSE	0x08	/* VF Memory Space Enable */
+#define  PCI_IOV_CTRL_ARI	0x10	/* ARI Capable Hierarchy */
+#define PCI_IOV_STATUS		0x0a	/* SR-IOV Status */
+#define PCI_IOV_INITIAL_VF	0x0c	/* Initial VFs */
+#define PCI_IOV_TOTAL_VF	0x0e	/* Total VFs */
+#define PCI_IOV_NUM_VF		0x10	/* Number of VFs */
+#define PCI_IOV_FUNC_LINK	0x12	/* Function Dependency Link */
+#define PCI_IOV_VF_OFFSET	0x14	/* First VF Offset */
+#define PCI_IOV_VF_STRIDE	0x16	/* Following VF Stride */
+#define PCI_IOV_VF_DID		0x1a	/* VF Device ID */
+#define PCI_IOV_SUP_PGSIZE	0x1c	/* Supported Page Sizes */
+#define PCI_IOV_SYS_PGSIZE	0x20	/* System Page Size */
+#define PCI_IOV_BAR_0		0x24	/* VF BAR0 */
+#define PCI_IOV_NUM_BAR		6	/* Number of VF BARs */
+
 #endif /* LINUX_PCI_REGS_H */
-- 
1.5.6.4

Yu Zhao

2008-Nov-21 18:43 UTC

head link

[PATCH 11/13 v7] PCI: reserve bus range for SR-IOV device

Reserve bus range for SR-IOV at device scanning stage when the kernel
boot parameter 'pci=assign-busses' is used .

Signed-off-by: Yu Zhao <yu.zhao at intel.com>

---
 drivers/pci/iov.c   |   24 ++++++++++++++++++++++++
 drivers/pci/pci.h   |    6 ++++++
 drivers/pci/probe.c |    3 +++
 3 files changed, 33 insertions(+), 0 deletions(-)

diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
index 03f62ca..c1a3cea 100644
--- a/drivers/pci/iov.c
+++ b/drivers/pci/iov.c
@@ -489,3 +489,27 @@ void pci_iov_unregister(struct pci_dev *dev)
 	iov_disable(dev);
 	kobject_uevent(&dev->dev.kobj, KOBJ_CHANGE);
 }
+
+/**
+ * pci_iov_bus_range - find bus range used by the Virtual Function
+ * @bus: the PCI bus
+ *
+ * Returns max number of buses (exclude current one) used by Virtual
+ * Functions.
+ */
+int pci_iov_bus_range(struct pci_bus *bus)
+{
+	int max = 0;
+	u8 busnr, devfn;
+	struct pci_dev *dev;
+
+	list_for_each_entry(dev, &bus->devices, bus_list) {
+		if (!dev->iov)
+			continue;
+		virtfn_bdf(dev, dev->iov->total - 1, &busnr, &devfn);
+		if (busnr > max)
+			max = busnr;
+	}
+
+	return max ? max - bus->number : 0;
+}
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 3113d11..574bbc7 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -205,6 +205,7 @@ extern int pci_iov_resource_bar(struct pci_dev *dev, int
resno,
 extern int pci_iov_register(struct pci_dev* dev);
 extern void pci_iov_unregister(struct pci_dev* dev);
 extern void pci_restore_iov_state(struct pci_dev *dev);
+extern int pci_iov_bus_range(struct pci_bus *bus);
 #else
 static inline int pci_iov_init(struct pci_dev *dev)
 {
@@ -232,6 +233,11 @@ static inline void pci_iov_unregister(struct pci_dev* dev)
 static inline void pci_restore_iov_state(struct pci_dev *dev)
 {
 }
+
+static inline int pci_iov_bus_range(struct pci_bus *bus)
+{
+	return 0;
+}
 #endif /* CONFIG_PCI_IOV */
 
 #endif /* DRIVERS_PCI_H */
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index cb26e64..7b591e5 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1074,6 +1074,9 @@ unsigned int __devinit pci_scan_child_bus(struct pci_bus
*bus)
 	for (devfn = 0; devfn < 0x100; devfn += 8)
 		pci_scan_slot(bus, devfn);
 
+	/* Reserve buses for SR-IOV capability. */
+	max += pci_iov_bus_range(bus);
+
 	/*
 	 * After performing arch-dependent fixup of the bus, look behind
 	 * all PCI-to-PCI bridges on this bus.
-- 
1.5.6.4

Yu Zhao

2008-Nov-21 18:43 UTC

head link

[PATCH 12/13 v7] PCI: document the SR-IOV sysfs entries

Document the SR-IOV sysfs entries.

Signed-off-by: Yu Zhao <yu.zhao at intel.com>

---
 Documentation/ABI/testing/sysfs-bus-pci |   26 ++++++++++++++++++++++++++
 1 files changed, 26 insertions(+), 0 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-bus-pci
b/Documentation/ABI/testing/sysfs-bus-pci
index ceddcff..d66d63d 100644
--- a/Documentation/ABI/testing/sysfs-bus-pci
+++ b/Documentation/ABI/testing/sysfs-bus-pci
@@ -9,3 +9,29 @@ Description:
 		that some devices may have malformatted data.  If the
 		underlying VPD has a writable section then the
 		corresponding section of this file will be writable.
+
+What:		/sys/bus/pci/devices/.../iov/total_virtfn
+Date:		November 2008
+Contact:	Yu Zhao <yu.zhao at intel.com>
+Description:
+		This file appears when a device has the SR-IOV capability
+		and the device driver (PF driver) support this operation.
+		It holds the number of total Virtual Functions (read-only).
+
+What:		/sys/bus/pci/devices/.../iov/initial_virtfn
+Date:		November 2008
+Contact:	Yu Zhao <yu.zhao at intel.com>
+Description:
+		This file appears when a device has the SR-IOV capability
+		and the device driver (PF driver) support this operation.
+		It holds the number of initial Virtual Functions (read-only).
+
+What:		/sys/bus/pci/devices/.../iov/nr_virtfn
+Date:		November 2008
+Contact:	Yu Zhao <yu.zhao at intel.com>
+Description:
+		This file appears when a device has the SR-IOV capability
+		and the device driver (PF driver) support this operation.
+		It holds the number of available Virtual Functions, and
+		could be written (0 ~ InitialVFs) to change the number of
+		the Virtual Functions.
-- 
1.5.6.4

Yu Zhao

2008-Nov-21 18:44 UTC

head link

[PATCH 13/13 v7] PCI: document for SR-IOV user and developer

Create how-to for the SR-IOV user and driver developer.

Signed-off-by: Yu Zhao <yu.zhao at intel.com>

---
 Documentation/DocBook/kernel-api.tmpl |    1 +
 Documentation/PCI/pci-iov-howto.txt   |  138 +++++++++++++++++++++++++++++++++
 2 files changed, 139 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/PCI/pci-iov-howto.txt

diff --git a/Documentation/DocBook/kernel-api.tmpl
b/Documentation/DocBook/kernel-api.tmpl
index 5818ff7..506e611 100644
--- a/Documentation/DocBook/kernel-api.tmpl
+++ b/Documentation/DocBook/kernel-api.tmpl
@@ -251,6 +251,7 @@ X!Edrivers/pci/hotplug.c
 -->
 !Edrivers/pci/probe.c
 !Edrivers/pci/rom.c
+!Edrivers/pci/iov.c
      </sect1>
      <sect1><title>PCI Hotplug Support Library</title>
 !Edrivers/pci/hotplug/pci_hotplug_core.c
diff --git a/Documentation/PCI/pci-iov-howto.txt
b/Documentation/PCI/pci-iov-howto.txt
new file mode 100644
index 0000000..216cecc
--- /dev/null
+++ b/Documentation/PCI/pci-iov-howto.txt
@@ -0,0 +1,138 @@
+		PCI Express I/O Virtualization Howto
+		Copyright (C) 2008 Intel Corporation
+
+
+1. Overview
+
+1.1 What is SR-IOV
+
+Single Root I/O Virtualization (SR-IOV) is a PCI Express Extended
+capability which makes one physical device appear as multiple virtual
+devices. The physical device is referred to as Physical Function (PF)
+while the virtual devices are referred to as Virtual Functions (VF).
+Allocation of the VF can be dynamically controlled by the PF via
+registers encapsulated in the capability. By default, this feature is
+not enabled and the PF behaves as traditional PCIe device. Once it's
+turned on, each VF's PCI configuration space can be accessed by its own
+Bus, Device and Function Number (Routing ID). And each VF also has PCI
+Memory Space, which is used to map its register set. VF device driver
+operates on the register set so it can be functional and appear as a
+real existing PCI device.
+
+2. User Guide
+
+2.1 How can I manage the SR-IOV
+
+If a device has the SR-IOV capability and the device driver (PF driver)
+supports this operation, then there should be some entries under the
+PF's sysfs directory:
+	- /sys/bus/pci/devices/NNNN:BB:DD.F/iov/
+	(NNNN:BB:DD:F is the domain, bus, device and function numbers)
+
+To change number of Virtual Functions:
+	- /sys/bus/pci/devices/NNNN:BB:DD.F/iov/nr_virtfn
+	(writing positive integer to this file will change the number of
+	VFs, and 0 means disable the capability)
+
+The total and initial numbers of VFs can get from:
+	- /sys/bus/pci/devices/NNNN:BB:DD.F/iov/total_virtfn
+	- /sys/bus/pci/devices/NNNN:BB:DD.F/iov/initial_virtfn
+
+2.2 How can I use the Virtual Functions
+
+The VF is treated as hot-plugged PCI devices in the kernel, so they
+should be able to work in the same way as real PCI devices. And also
+the VF requires device driver that is same as a normal PCI device's.
+
+3. Developer Guide
+
+3.1 SR-IOV APIs
+
+To use the SR-IOV service, the Physical Function driver needs to declare
+a callback function in its 'struct pci_driver':
+
+	static struct pci_driver dev_driver = {
+		...
+		.virtual =	dev_virtual,
+		...
+	};
+
+	The 'dev_virtual' is a callback function that the SR-IOV service
+	will invoke it when the number of  VFs is changed by the user.
+	The first argument of this callback is PF itself ('struct pci_dev'),
+	and the second argument is the number of VFs requested. The callback
+	should return 0 if the requested number of VFs is supported and all
+	necessary resources are granted to these VFs; otherwise it should
+	return a negative value indicating the error.
+
+3.2 Usage example
+
+Following piece of code illustrates the usage of APIs above.
+
+static int __devinit dev_probe(struct pci_dev *dev,
+				const struct pci_device_id *id)
+{
+	...
+
+	return 0;
+}
+
+static void __devexit dev_remove(struct pci_dev *dev)
+{
+	...
+}
+
+#ifdef CONFIG_PM
+static int dev_suspend(struct pci_dev *dev, pm_message_t state)
+{
+	...
+
+	return 0;
+}
+
+static int dev_resume(struct pci_dev *dev)
+{
+	pci_restore_state(dev);
+
+	...
+
+	return 0;
+}
+#endif
+
+static void dev_shutdown(struct pci_dev *dev)
+{
+	...
+}
+
+static int dev_virtual(struct pci_dev *dev, int nr_virtfn)
+{
+
+	if (nr_virtfn) {
+		/*
+		 * allocate device internal resources for VFs.
+		 * these resources are device-specific (e.g. rx/tx
+		 * queue in the NIC) and necessary to make the VF
+		 * functional.
+		 */
+	} else {
+		/*
+		 * reclaim the VF related resources if any.
+		 */
+	}
+
+	return 0;
+}
+
+static struct pci_driver dev_driver = {
+	.name =		"SR-IOV Physical Function driver",
+	.id_table =	dev_id_table,
+	.probe =	dev_probe,
+	.remove =	__devexit_p(dev_remove),
+#ifdef CONFIG_PM
+	.suspend =	dev_suspend,
+	.resume =	dev_resume,
+#endif
+	.shutdown =	dev_shutdown,
+	.virtual =	dev_virtual
+};
-- 
1.5.6.4

Greg KH

2008-Nov-21 20:57 UTC

head link

[PATCH 0/13 v7] PCI: Linux kernel SR-IOV support

On Sat, Nov 22, 2008 at 02:36:05AM +0800, Yu Zhao wrote:> Greetings,
> 
> Following patches are intended to support SR-IOV capability in the
> Linux kernel. With these patches, people can turn a PCI device with
> the capability into multiple ones from software perspective, which
> will benefit KVM and achieve other purposes such as QoS, security,
> and etc.
> 
> The Physical Function and Virtual Function drivers using the SR-IOV
> APIs will come soon!
Thanks for respining these patches, but I think we really need to see a
driver using this in order to get an idea of how it will be used.

Also, the Xen and KVM people need to agree on the userspace interface
here, perhaps also getting some libvirt involvement as well, as they are
going to be the ones having to use this all the time.

thanks,

greg k-h

Yu Zhao

2008-Nov-26 14:03 UTC

head link

[SR-IOV driver example 0/3] introduction

SR-IOV drivers of Intel 82576 NIC are available. There are two parts
of the drivers: Physical Function driver and Virtual Function driver.
The PF driver is based on the IGB driver and is used to control PF to
allocate hardware specific resources and interface with the SR-IOV core.
The VF driver is a new NIC driver that is same as the traditional PCI
device driver. It works in both the host and the guest (Xen and KVM)
environment.

These two drivers are testing versions and they are *only* intended to
show how to use SR-IOV API.

Intel 82576 NIC specification can be found at:
http://download.intel.com/design/network/datashts/82576_Datasheet_v2p1.pdf

[SR-IOV driver example 1/3] PF driver: allocate hardware specific resource
[SR-IOV driver example 2/3] PF driver: integrate with SR-IOV core
[SR-IOV driver example 3/3] VF driver tar ball

Yu Zhao

2008-Nov-26 14:21 UTC

head link

[SR-IOV driver example 2/3] PF driver: integrate with SR-IOV core

This patch integrates the IGB driver with the SR-IOV core. It shows how
the SR-IOV API is used to support the capability. Obviously people does
not need to put much effort to integrate the PF driver with SR-IOV core.
All SR-IOV standard stuff are handled by SR-IOV core and PF driver only
concerns the device specific resource allocation and deallocation once it
gets the necessary information (i.e. number of Virtual Functions) from
the callback function.

---
 drivers/net/igb/igb_main.c |   30 ++++++++++++++++++++++++++++++
 1 files changed, 30 insertions(+), 0 deletions(-)

diff --git a/drivers/net/igb/igb_main.c b/drivers/net/igb/igb_main.c
index bc063d4..b8c7dc6 100644
--- a/drivers/net/igb/igb_main.c
+++ b/drivers/net/igb/igb_main.c
@@ -139,6 +139,7 @@ void igb_set_mc_list_pools(struct igb_adapter *, struct
e1000_hw *, int, u16);
 static int igb_vmm_control(struct igb_adapter *, bool);
 static int igb_set_vf_mac(struct net_device *, int, u8*);
 static void igb_mbox_handler(struct igb_adapter *);
+static int igb_virtual(struct pci_dev *, int);
 #endif
 
 static int igb_suspend(struct pci_dev *, pm_message_t);
@@ -184,6 +185,9 @@ static struct pci_driver igb_driver = {
 #endif
 	.shutdown = igb_shutdown,
 	.err_handler = &igb_err_handler,
+#ifdef CONFIG_PCI_IOV
+	.virtual = igb_virtual
+#endif
 };
 
 static int global_quad_port_a; /* global quad port a indication */
@@ -5107,6 +5111,32 @@ void igb_set_mc_list_pools(struct igb_adapter *adapter,
 	reg_data |= (1 << 25);
 	wr32(E1000_VMOLR(pool), reg_data);
 }
+
+static	int
+igb_virtual(struct pci_dev *pdev, int nr_virtfn)
+{
+	unsigned char my_mac_addr[6] = {0x00, 0xDE, 0xAD, 0xBE, 0xEF, 0xFF};
+	struct net_device *netdev = pci_get_drvdata(pdev);
+	struct igb_adapter *adapter = netdev_priv(netdev);
+	int i;
+
+	if (nr_virtfn > 7)
+		return -EINVAL;
+
+	if (nr_virtfn) {
+		for (i = 0; i < nr_virtfn; i++) {
+			printk(KERN_INFO "SR-IOV: VF %d is enabled\n", i);
+			my_mac_addr[5] = (unsigned char)i;
+			igb_set_vf_mac(netdev, i, my_mac_addr);
+			igb_set_vf_vmolr(adapter, i);
+		}
+	} else
+		printk(KERN_INFO "SR-IOV is disabled\n");
+
+	adapter->vfs_allocated_count = nr_virtfn;
+
+	return 0;
+}
 #endif
 
 /* igb_main.c */
-- 
1.5.4.4

Yu Zhao

2008-Nov-26 14:40 UTC

head link

[SR-IOV driver example 3/3] VF driver tar ball

The attachment is the VF driver for Intel 82576 NIC. Since the VF
appears as the normal PCI device driver, this VF driver has no
difference from other PCI NIC drivers. It handles interrupt, DMA
operations, etc. to perform packet receiving and transmission.

How the design of the VF internals is up to the hardware vendor.
So the VF may have totally different register set from the PF,
which means the VF driver may have its own logics (rather than
derive from PF driver) to handle the hardware specific stuff.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: igbvf-0.5.2.tar.gz
Type: application/x-tar-gz
Size: 141423 bytes
Desc: not available
Url :
http://lists.linux-foundation.org/pipermail/virtualization/attachments/20081126/054c3b1d/attachment-0001.bin

Jeff Garzik

2008-Nov-26 20:14 UTC

head link

[SR-IOV driver example 0/3] introduction

Yu Zhao wrote:> SR-IOV drivers of Intel 82576 NIC are available. There are two parts
> of the drivers: Physical Function driver and Virtual Function driver.
> The PF driver is based on the IGB driver and is used to control PF to
> allocate hardware specific resources and interface with the SR-IOV core.
> The VF driver is a new NIC driver that is same as the traditional PCI
> device driver. It works in both the host and the guest (Xen and KVM)
> environment.
> 
> These two drivers are testing versions and they are *only* intended to
> show how to use SR-IOV API.
> 
> Intel 82576 NIC specification can be found at:
> http://download.intel.com/design/network/datashts/82576_Datasheet_v2p1.pdf
> 
> [SR-IOV driver example 1/3] PF driver: allocate hardware specific resource
> [SR-IOV driver example 2/3] PF driver: integrate with SR-IOV core
> [SR-IOV driver example 3/3] VF driver tar ball
Please copy netdev at vger.kernel.org on all network-related patches.  This 
is where the network developers live, and all patches on this list are 
automatically archived for review and handling at 
http://patchwork.ozlabs.org/project/netdev/list/

	Jeff

Yu Zhao

2008-Dec-01 16:39 UTC

head link

[SR-IOV driver example 0/3] introduction

On Thu, Nov 27, 2008 at 04:14:48AM +0800, Jeff Garzik
wrote:> Yu Zhao wrote:
> > SR-IOV drivers of Intel 82576 NIC are available. There are two parts
> > of the drivers: Physical Function driver and Virtual Function driver.
> > The PF driver is based on the IGB driver and is used to control PF to
> > allocate hardware specific resources and interface with the SR-IOV
core.
> > The VF driver is a new NIC driver that is same as the traditional PCI
> > device driver. It works in both the host and the guest (Xen and KVM)
> > environment.
> > 
> > These two drivers are testing versions and they are *only* intended to
> > show how to use SR-IOV API.
> > 
> > Intel 82576 NIC specification can be found at:
> >
http://download.intel.com/design/network/datashts/82576_Datasheet_v2p1.pdf
> > 
> > [SR-IOV driver example 1/3] PF driver: allocate hardware specific
resource
> > [SR-IOV driver example 2/3] PF driver: integrate with SR-IOV core
> > [SR-IOV driver example 3/3] VF driver tar ball
> 
> Please copy netdev at vger.kernel.org on all network-related patches.  This
> is where the network developers live, and all patches on this list are 
> automatically archived for review and handling at 
> http://patchwork.ozlabs.org/project/netdev/list/
Will do.

Thanks,
Yu

Yu Zhao

2008-Dec-02 09:27 UTC

head link

[SR-IOV driver example 0/3 resend] introduction

SR-IOV drivers of Intel 82576 NIC are available. There are two parts
of the drivers: Physical Function driver and Virtual Function driver.
The PF driver is based on the IGB driver and is used to control PF to
allocate hardware specific resources and interface with the SR-IOV core.
The VF driver is a new NIC driver that is same as the traditional PCI
device driver. It works in both the host and the guest (Xen and KVM)
environment.

These two drivers are testing versions and they are *only* intended to
show how to use SR-IOV API.

Intel 82576 NIC specification can be found at:
http://download.intel.com/design/network/datashts/82576_Datasheet_v2p1.pdf

[SR-IOV driver example 0/3 resend] introduction
[SR-IOV driver example 1/3 resend] PF driver: hardware specific operations
[SR-IOV driver example 2/3 resend] PF driver: integrate with SR-IOV core
[SR-IOV driver example 3/3 resend] VF driver: an independent PCI NIC driver

Jeff Kirsher

2008-Dec-03 03:12 UTC

head link

[SR-IOV driver example 0/3 resend] introduction

On Tue, Dec 2, 2008 at 1:27 AM, Yu Zhao <yu.zhao at intel.com>
wrote:> SR-IOV drivers of Intel 82576 NIC are available. There are two parts
> of the drivers: Physical Function driver and Virtual Function driver.
> The PF driver is based on the IGB driver and is used to control PF to
> allocate hardware specific resources and interface with the SR-IOV core.
> The VF driver is a new NIC driver that is same as the traditional PCI
> device driver. It works in both the host and the guest (Xen and KVM)
> environment.
>
> These two drivers are testing versions and they are *only* intended to
> show how to use SR-IOV API.
>
> Intel 82576 NIC specification can be found at:
> http://download.intel.com/design/network/datashts/82576_Datasheet_v2p1.pdf
>
> [SR-IOV driver example 0/3 resend] introduction
> [SR-IOV driver example 1/3 resend] PF driver: hardware specific operations
> [SR-IOV driver example 2/3 resend] PF driver: integrate with SR-IOV core
> [SR-IOV driver example 3/3 resend] VF driver: an independent PCI NIC driver
> --
>
First of all, we (e1000-devel) do support the SR-IOV API.

With that said, NAK on the driver changes.  We were not involved in
these changes and are currently working on a version of the drivers
that will make them acceptable for kernel inclusion.

-- 
Cheers,
Jeff

Jesse Barnes

2008-Dec-16 23:23 UTC

head link

[PATCH 0/13 v7] PCI: Linux kernel SR-IOV support

On Friday, November 21, 2008 10:36 am Yu Zhao wrote:> Greetings,
>
> Following patches are intended to support SR-IOV capability in the
> Linux kernel. With these patches, people can turn a PCI device with
> the capability into multiple ones from software perspective, which
> will benefit KVM and achieve other purposes such as QoS, security,
> and etc.
>
> The Physical Function and Virtual Function drivers using the SR-IOV
> APIs will come soon!
>
> Major changes from v6 to v7:
> 1, remove boot-time resource rebalancing support. (Greg KH)
> 2, emit uevent upon the PF driver is loaded. (Greg KH)
> 3, put SR-IOV callback function into the 'pci_driver'. (Matthew
Wilcox)
> 4, register SR-IOV service at the PF loading stage.
> 5, remove unnecessary APIs (pci_iov_enable/disable).
Thanks for your patience with this, Yu, I know it's been a long haul. :)

I applied 1-9 to my linux-next branch; and at least patch #10 needs a respin, 
so can you re-do 10-13 as a new patch set?

On re-reading the last thread, there was a lot of smoke, but very little fire 
afaict.  The main questions I saw were:

  1) do we need SR-IOV at all?  why not just make each subsystem export
     devices to guests?
    This is a bit of a red herring.  Nothing about SR-IOV prevents us from
    making subsystems more v12n friendly.  And since SR-IOV is a hardware
    feature supported by devices these days, we should make Linux support it.

  2) should the PF/VF drivers be the same or not?
    Again, the SR-IOV patchset and PCI spec don't dictate this.  We're
free to
    do what we want here.

  3) should VF devices be represented by pci_dev structs?
    Yes.  (This is an easy one :)

  4) can VF devices be used on the host?
    Yet again, SR-IOV doesn't dictate this.  Developers can make PF/VF combo
    drivers or split them, and export the resulting devices however they want. 
    Some subsystem work may be needed to make this efficient, but SR-IOV
    itself is agnostic about it.

So overall I didn't see many objections to the actual code in the last post,
and the issues above certainly don't merit a NAK IMO...

Given a respin of 10-13 I think it's reasonable to merge this into 2.6.29,
but
I'd be much happier about it if we got some driver code along with it, so as
not to have an unused interface sitting around for who knows how many 
releases.  Is that reasonable?  Do you know if any of the corresponding PF/VF 
driver bits are ready yet?

Thanks,
-- 
Jesse Barnes, Intel Open Source Technology Center

Reasonably Related Threads

Search for more reasonably related threads

Virtualization - Nov 2008 - [PATCH 0/13 v7] PCI: Linux kernel SR-IOV support

[PATCH 0/13 v7] PCI: Linux kernel SR-IOV support

[PATCH 1/13 v7] PCI: enhance pci_ari_enabled()

[PATCH 2/13 v7] PCI: remove unnecessary arg of pci_update_resource()

[PATCH 3/13 v7] PCI: define PCI resource names in an 'enum'

[PATCH 4/13 v7] PCI: remove unnecessary condition check in pci_restore_bars()

[PATCH 5/13 v7] PCI: export __pci_read_base()

[PATCH 6/13 v7] PCI: make pci_alloc_child_bus() be able to handle NULL bridge

[PATCH 7/13 v7] PCI: add a new function to map BAR offset

[PATCH 8/13 v7] PCI: cleanup pci_bus_add_devices()

[PATCH 9/13 v7] PCI: split a new function from pci_bus_add_devices()

[PATCH 10/13 v7] PCI: support the SR-IOV capability

[PATCH 11/13 v7] PCI: reserve bus range for SR-IOV device

[PATCH 12/13 v7] PCI: document the SR-IOV sysfs entries

[PATCH 13/13 v7] PCI: document for SR-IOV user and developer

[PATCH 0/13 v7] PCI: Linux kernel SR-IOV support

[SR-IOV driver example 0/3] introduction

[SR-IOV driver example 2/3] PF driver: integrate with SR-IOV core

[SR-IOV driver example 3/3] VF driver tar ball

[SR-IOV driver example 0/3] introduction

[SR-IOV driver example 0/3] introduction

[SR-IOV driver example 0/3 resend] introduction

[SR-IOV driver example 0/3 resend] introduction

[PATCH 0/13 v7] PCI: Linux kernel SR-IOV support

Reasonably Related Threads