Hi,
I am leaving the older message just for the sake of continuity in case you need
to know any of the history.
We're using Intel Pro/1000 VT with RHEL 5.2 and trying to use IOCTL to send and
receive data. We're
using the igb driver but there's some change we put in to get the packet right
after it's DMAed from the
NIC, and use IOCTL to send the packet to the user space. This is the Rx path.
In the Tx path we're calling
IOCTL from user space to send a bunch of pointers to the driver and use
Scatter-Gather to send the packet
out.
We can't seem to get more than 50MByte/S per port, when we run on all 4 ports.
With single port we can
see 90MByte/S, occasionally reaching 105MBytes/S . With 2 ports it's somewhere
at the rate of 60MB/S,
with 3 ports, it's 50+MB/S, and with 4 ports max. we're able to do 200MB/S
aggregate. (4 ports at wire
speed should give an aggregate of over 440MB/S, counting off TCP/IP overheads.)
We *believe* (not sure) that we're hitting serialization issues. We believe
that even though we make 4
IOCTL calls for packets to 4 different ports all these calls are serialized. We
can provide more info. as
needed.
Please let us know if there's something fundamentally wrong. Any pointers to
other's data would be
helpful. Finally, if you need more data on the driver/kernel/debug
commands/traces we can get that.
Thanks,
-Kumar
----- Original Message ----
From: "Brandeburg, Jesse" <[EMAIL PROTECTED]>
To: Kumar Narayanan <[EMAIL PROTECTED]>; "Ronciak, John" <[EMAIL PROTECTED]>;
Richard Scobie <[EMAIL PROTECTED]>; [email protected]
Sent: Tuesday, May 20, 2008 8:49:14 AM
Subject: RE: [E1000-devel] RSS - Receive-Side Scaling on Intel PRO/1000 NIC
yes, my reference to IGB *only* applies to 82575 based VT
adapter, not to the 82571 based PT adapter.
Thanks for the dell reference!
Jesse
________________________________
From: Kumar Narayanan [mailto:[EMAIL PROTECTED]
Sent: Monday, May 19, 2008 7:13 PM
To: Brandeburg, Jesse;
Ronciak, John; Richard Scobie;
[email protected]
Subject: Re: [E1000-devel] RSS -
Receive-Side Scaling on Intel PRO/1000 NIC
Thanks
for the quick response, Jesse.
I think that I'll use the e1000 on another
server where we don't need RSS. Just that
we spent time, and ended up
probably costing all your time also. Moving on...
I suppose that your
reference to IGB driver, MSI-X, 2.6.23 (or 2.6.25.4) etc. apply to 82575 (Pro
1000 VT), right? Or does it apply to
Pro 1000 PT also?
The NIC is orderable. Dell Part# : 430-2688. Go
to "http://www.dell.com" and
enter 430-2688 in the search box. It'll let you
order from there. I just discovered
it today - wasn't orderable till late
lastweek.
-Kumar
----- Original Message ----
From: "Brandeburg, Jesse"
<[EMAIL PROTECTED]>
To: Kumar Narayanan
<[EMAIL PROTECTED]>; "Ronciak, John" <[EMAIL PROTECTED]>; Richard
Scobie <[EMAIL PROTECTED]>; [email protected]
Sent:
Monday, May 19, 2008 6:15:24 PM
Subject: RE: [E1000-devel] RSS - Receive-Side
Scaling on Intel PRO/1000 NIC
First, you should be able to return your PT adapters, I'm
sorry that you got a bad deal from the marketing
documentation.
Second, the Intel provided IGB driver on sourceforge
supports full tx and rx multiple queues with MSI-X (tx multiple queues requires
a 2.6.23, recommend a 2.6.25.4 kernel) and RSS based steering to the multiple
queues. Right now the hash that steers the flows to a particular queue is
random, but can easily be made static with a simple patch.
I suggest you see if you can find an Intel distributor who
will sample you an 82575 based board. If that doesn't work for you then I
suggest you buy one from dell (you'll probably have to call them, as we haven't
found it on the website yet)
Jesse
________________________________
From: Kumar Narayanan [mailto:[EMAIL PROTECTED]
Sent: Monday, May 19, 2008 4:05 PM
To: Ronciak, John;
Brandeburg, Jesse; Richard Scobie;
[email protected]
Subject: Re: [E1000-devel] RSS -
Receive-Side Scaling on Intel PRO/1000 NIC
Based
on the email below and some more reading I am thinking of getting
Intel
Pro/1000 VT quad-port GbE NIC. Please let me know if any of the
following
assumptions are incorrect.
1. It's based on 85275
2. It
supports MSI-X
3. It has RSS support
4. Need kernel version 2.6.23 or RHEL
5.2
The reason for asking these questions is that looking at Intel's web
site it appeared
that Intel Pro/1000 PT quad-port GbE NIC supports RSS. While
this may be of
technical interest as a user I can't have RSS function with PT
adapters with Linux. I
didn't see *anything* in the Intel's product site that
says that PT adapters has RSS
feature but it's not usable with Linux. I am
trying to prevent myself from falling into
the same/similar trap
now.
Thanks,
-Kumar
-----
Original Message ----
From: "Ronciak, John"
<[EMAIL PROTECTED]>
To: Kumar Narayanan <[EMAIL PROTECTED]>;
"Brandeburg, Jesse" <[EMAIL PROTECTED]>; Richard Scobie
<[EMAIL PROTECTED]>; [email protected]
Sent: Friday,
May 2, 2008 1:35:11 PM
Subject: RE: [E1000-devel] RSS - Receive-Side Scaling
on Intel PRO/1000 NIC
>I am assuming that without RSS the interrupts
are free to go to any of
the cores in
>a multi-core system. Is that a
fair statement?
Yes that is true assuming that user-space level IRQ balancer
is being
used. The in-kernel version doesn't work very well at
all.
>With RSS we thought we might be able to bring in session
stickiness -
I.E. packets belonging to >the same session will find way to
the same
core. Is that a valid assumption?
Well yes that is some what true
but as Auke has said, not having MSI-X
interrupts enabled on the 82571
adapters hurt the performance which
could be gained. You can turn off
the IRQ balancer and the set the
affinity of the connections to a single
processor which might give you
what you are after.
The real way to get
all of this to work is to use the 82575 parts with
the igb driver. It
supports MSI-X which might be better for what you
are trying to do.
On
the I/OAT front, please make sure that you try RHEL5.2 as there are a
number
of bug fixes and added features regarding I/OAT. It matches what
is in
the 2.6.23 kernel regarding I/OAT. It releases later this month
but the
final RCs are available
today.
Cheers,
John
-----------------------------------------------------------
"Those
who would give up essential Liberty, to purchase a little
temporary Safety,
deserve neither Liberty nor Safety.", Benjamin
Franklin 1755
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]
On Behalf Of Kumar
Narayanan
Sent: Friday, May 02, 2008 11:37 AM
To:
Brandeburg, Jesse; Richard Scobie; [email protected]
Subject:
Re: [E1000-devel] RSS - Receive-Side Scaling on Intel PRO/1000
NIC
Hi
Jesse, Thanks for the kind reply... I got a good amount of info. from
your's
and Auke's email.
We enabled IOAT with the hope that what's being said
for WIN (that RSS
is a
necessary condition for IOAT) may be true for
LINUX also, and that by
enabling
IOAT we could get RSS also going
...
I am assuming that without RSS the interrupts are free to go to any
of
the cores in
a multi-core system. Is that a fair statement? With RSS
we thought we
might be
able to bring in session stickiness - I.E packets
belonging to the same
session will
find way to the same core. Is that a
valid assumption?
What would be my options to achieve the session
stickiness as I
mentioned above?
What's my next best
option?
Thanks,
-Kumar
----- Original Message
----
From: "Brandeburg, Jesse" <[EMAIL PROTECTED]>
To:
Kumar Narayanan <[EMAIL PROTECTED]>; Richard
Scobie
<[EMAIL PROTECTED]>; [email protected]
Sent:
Thursday, May 1, 2008 6:32:15 PM
Subject: RE: [E1000-devel] RSS -
Receive-Side Scaling on Intel PRO/1000
NIC
Hi Kumar, maybe look for
"Crystal Beach" in bios for enabling IOAT.
Some vendors left the code name in
there.
yes, IOAT (copy offload) is not needed for supporting
RSS.
while the hardware supports RSS, the Linux drivers don't currently
do
anything for it in e1000/e1000e. all the parts supported by e1000e
have
RSS and at least 2 queue support, but don't have MSI-X, so can
only
generate a ssingle interrupt vector, which kind of makes multiple
queues
not very useful on those parts.
Tell us what you were trying to
achieve with RSS and let us help you get
there, maybe without
RSS.
Jesse
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]
On Behalf Of Kumar
Narayanan
Sent: Thursday, May 01, 2008 6:28 PM
To:
Richard Scobie; [email protected]
Subject:
Re: [E1000-devel] RSS - Receive-Side Scaling on Intel
PRO/1000
NIC
Hello Richard, Thanks for the email.
We don't
need I/O AT per se. Somewhere we read in Intel doc. that RSS is
a
necessary condition for I/OAT to function. However, that seems to
be
relevant
for WIN server OS only.
We want only RSS. We enabled
I/O AT thinking that RSS *might* get
enabled
if I/O AT is also enabled. I
wonder if anyone is using RSS in LINUX with
Intel
PRO/1000 GbE NIC. Intel
technical support came back with no answer.
Hence
I am asking the e1000
developers...
BTW, I checked. I/O AT mod is installed but it's not
enabled. I read the
URL
you gave me. It appears that we need to enable it
in BIOS. I am trying
to understand
what command in BIOS will enable I/O
AT.
----- Original Message ----
From: Richard Scobie
<[EMAIL PROTECTED]>
To: [email protected]
Sent:
Thursday, May 1, 2008 5:15:13 PM
Subject: Re: [E1000-devel] RSS -
Receive-Side Scaling on Intel PRO/1000
NIC
Kumar Narayanan
wrote:
> Hello, I am using Intel Pro/1000 PT Quad Port Server NIC with
RHEL 5.
> I am trying to enable RSS - Receive-Side Scaling. The
documentation
says
> that the above NIC supports LINUX I/O Scaling -
RSS being part of this
> I/O Scaling feature.
If you mean IOAT
support, look here and read the README in the Linux
download.
http://www.intel.com/support/network/adapter/pro100/sb/CS-023725.htm
Also
look at this discussion for details of what you may need to enable
in the
BIOS:
http://sourceforge.net/mailarchive/forum.php?thread_name=480FBB2F.709000
9%40sauce.co.nz&forum_name=e1000-devel
Regards,
Richard
------------------------------------------------------------------------
-
This SF.net email is sponsored
by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event.
There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/j
avaone
_______________________________________________
E1000-devel
mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
________________________________________________________________________
____________
Be
a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it
now.
http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ
________________________________________________________________________
____________
Be
a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it
now.
http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
