                               README Notes
                    Broadcom bnxt_re Linux RoCE Driver
                             Version 20.8.0.1
                                08/02/2017

                            Broadcom Limited
                         5300 California Avenue,
                            Irvine, CA 92617

                   Copyright (c) 2015-2017 Broadcom
                           All rights reserved


Table of Contents
=================

  Introduction
  Limitations
  BNXT_RE Driver Dependencies
  BNXT_RE Driver compilation
  BNXT_RE Dynamic Debug Messages
  BNXT_RE Driver Defaults
  Configuration Tips
  Unloading and Removing Driver
  LLDPAD configuration
  VF Resource Distribution
  Congestion Control
  BNXT_RE Driver Statistics


Introduction
============

This file describes the bnxt_re Linux RoCE driver for the Broadcom NetXtreme-C
and NetXtreme-E 10/20/25/40/50/100/200 Gbps Ethernet Network Controllers.


Limitations
===========

- The current version of the driver will compile on RHEL7.x, RHEL6.7/6.8,
  SLES11 SP4, SLES12 SP2, Ubuntu 16.04 and most 3.x/4.x kernels, and some
  2.6 kernels starting from 2.6.32-573.

- The current version of the driver supports only Ubuntu 14.04.4 and
  14.04.5 (latest) with 4.x kernel

- RoCE V2 is supported only on RHEL 7.3 or 4.7 kernels or latest kernels.

- When remote directories are mounted using NFS-RDMA, unloading bnxt_re shall
  cause system hang and the system needs a reboot for normal operations.
  Always unmount all active NFS mounts over bnxt_re interface, before unloading
  bnxt_re driver.

- Using same MTU on both client and server is recommended.  User can see unexpected
  results if there is a mismatch in MTUs on Client and Server.

- Changing MAC address of the interface while bnxt_re is loaded can trigger failure
  during GID deletion. Unload bnxt_re driver before changing the interface MAC address.

- The krping tool requires an update in order to operate under the RHEL7.2.
  The driver has passed the 4 standard test cases using (get_dma, fast_reg,
  mr, and mw).

- The legacy FMR Pool is not supported yet.

- Resize CQ is not supported yet.

- Raw Ethertype QP is not supported yet.

- Tunnel is not supported yet.

- Linux RoCE driver supports only 127 entries deep GID table per port.
  Thus, effectively only 126 different VLANs could be active in a RoCE V1 only
  mode. If RoCE V2 is supported, driver supports upto 63 VLANs.

- On SLES11 SP4 default kernel(3.0.101-63-default), tc command to map the
  priority to traffic class throws error and hence ETS b/w will not get
  honored when NIC + RoCE traffic is run together.
  This issue is fixed in 3.0.101-91-default. Users are advised to upgrade to
  this kernel while testing ETS.

- On RHEL7.3 kernel (3.10.0-514.el7.x86_64), the following issues are seen
  with NFSoRDMA stack. The issues are fixed in upstream kernel.

  1. In some rare conditions, while recovering from errors due to connection
     loss, the nfs server side could crash with the following message:
	"kernel BUG at drivers/iommu/iova.c:208!"
     This is fixed upstream in 'svcrdma' kernel module by this commit:
	commit ce1ca7d2d140a1f4aaffd297ac487f246963dd2f
	svcrdma: avoid duplicate dma unmapping during error recovery

  2. In some rare conditions, after recovering from errors due to connection
     loss, the mount point could become unresponsive on the nfs client. The
     nfs client logs the following message in this condition:
	"RPC: rpcrdma_buffer_get: out of reply buffers"
     This is fixed upstream in 'xprtrdma' kernel module by this commit:
	commit 05c974669ecec510a85d8534099bb75404e82c41
	xprtrdma: Fix receive buffer accounting

  3. The nfs server crashes with the following message when the IP address
     of a network interface is changed on the server:
	"general protection fault: 0000 [#1] SMP"
     This is fixed upstream in 'sunrpc' kernel module by this commit:
	commit ea08e39230e898844d9de5b60cdbb30067cebfe7
	sunrpc: svc_age_temp_xprts_now should not call setsockopt non-tcp
		transports

- On RHEL 7.4 kernel(3.10.0-693.el7.x86_64), the following things are observed
  with NFSoRDMA stack.
  1. Multiple mount points (using different VLANs) result in only a single QP
     being created for all.
  2. Deleting the vlan before unmounting all the multiple points results in
     delete gid failures.
  Both the issues are due to NFS v4.1 implementation(which is what the
  'mount' cmd defaults to on RHEL 7.4).
  The workaround for both is to mount using NFSv4.0(vers=4.0) in the 'mount' cmd

- For enabling RoCE SR-IOV on RH 6.x, the following driver load sequence should
  be followed. bnxt_re driver load fails otherwise.

	#service NetworkManager stop
	#modprobe bnxt_re
	#modprobe bnxt_en num_vfs=<num_vfs>
	#ifconfig <iface> up
	#modprobe bnxt_en num_vfs=<num_vfs>


BNXT_RE Driver Dependencies
===========================

The RoCE driver has dependencies on the bnxt_en Ethernet counterpart.

  - It also has dependencies on the IB verbs kernel component
    (Details given below).


BNXT_RE Driver compilation
==========================

bnxt_re driver compilation depends on whether IB stack is available along with
the OS distribution or an external OFED is required.

 => Distros that has IB Stack available along with OS distribution:
    RH7.1/7.2/7.3/6.7/6.8, SLES12SP2 and Ubuntu 16.04/14.04

To compile bnxt_re:
	$make

 => Distros that need external OFED to be installed:
    SLES11SP4

Please refer OFED release notes from the following link and install
OFED before compiling bnxt_re driver.
http://downloads.openfabrics.org/downloads/OFED/release_notes/OFED_3.18-2_release_notes

To compile bnxt_re:
	$export OFED_VERSION=OFED-3.18-2
	$make

BNXT_RE Driver Defaults
=======================

Driver load configures the default RoCE and CNP priorities and DSCP values.
It also enable PFC and CC by default. No other configuration required
if the switches are configured with these values.
The default values that are programmed are as follows.

RoCE DSCP 26
ROCE Priority 3
CNP DSCP 48
CNP Priority 7

The users can change these values using bnxt_setupcc.sh or other tools.
PFC settings can be configured with bnxtqos or lldptool and CC settings
can be configured using configfs hooks under /sys/kernel/config/bnxt_re.
While adding new settings, please make sure that the default settings
(say, app TLVs) are removed, before programming the new values.
Refer bnxt_setupcc.sh for usage of lldptool/bnxtqos and configfs.

Note: Unloading bnxt_re driver would remove the DCBx settings on the adapter.
      This might include some of the settings done by users after loading
      bnxt_re. Please make sure that the DCBx settings for L2 traffic are
      correct, after unloading/reloading bnxt_re.
      To confirm the dcbx settings,  use bnxtqos or lldptool as following.

	$lldptool get-tlv -n -i <eth x>
	or
	$ bnxtqos -dev=p6p1 get_qos

Configuration Tips
==================

- It is recommended to use same host OS version on client and server while
  running NFS-RDMA/iSER/NVMEoF tests. Heterogeneous host OS may lead to unexpected
  results. This is due to the incompatible ULP server and Client kernel modules.

- It is recommended to assign at least 3GB RAM to VMs used for memory intensive
  applications like NFSoRDMA, iSER, NVMoF etc.

- When using large number of QPs (close to maximum supported) along with large
  message sizes it is recommended to increase the `max_map_count` kernel parameter
  using sysctl to avoid memory map failures in the application.
  Please refer to https://www.kernel.org/doc/Documentation/sysctl/vm.txt on how to tune
  this kernel parameter.

- When TCP/IP and RoCE traffic are running simultaneously with high work load or
  RoCE traffic with high work load, high CPU utilization can result,
  leading to CPU soft lockup. Hence it is recommended to spread the workload
  across the available CPU cores. This can be achieved by setting the SMP
  affinity of the interrupts and RoCE applications.
  Please refer to OS documentation for setting smp_affinity and specific
  commands like taskset etc.

- To avoid the SQ full reported during iSER stress testing (kernels SLES 12 and later,
  RHEL 7.4 and later), configure the minimum tx depth for the QPs to 4096.
  All connections getting established after setting min_tx_depth uses the user specified values.

  echo 4096 > /sys/kernel/config/bnxt_re/bnxt_re0/ports/1/tunables/min_tx_depth

- At lower link speeds (e.g. 10G or 25G), for heavy RDMA-READ workloads with
  large number of active QPs, a higher ack-timeout value is recommended.
  For example:
   ib_read_bw with -q 4096 would require ack timeout 18. Ack timeout is
   controlled by option "-u".
   ib_read_bw --report_gbits -F -m 4096 -q 4096 -d bnxt_re2 -x 3 -u 18 -D 60 -s 65536

- For use cases where the adapter QP limit is exercised or active qps are
  close to adapter limits, ack timeout needs to be increased to 24 to avoid
  retransmissions and loss of performance.
  For example:
   For multiple instances of ib_send_bw/ib_read_bw/ib_write_bw, which creates
   total of 64K QPs, specify higher ack timeout in each application instance
   using -u 24.
- "restrict_mrs" module parameter is used to limit the number of MRs supported
  by the driver. For now the parameter applies only to 574xx devices.
  Setting this flag to 1 would reduce the number for MRs supported by the device
  to 64K.

BNXT_RE Dynamic Debug Messages
==============================
The bnxt_re driver supports the Linux dynamic debug feature.

All error, warning and info messages are logged by default.
Any debug messages if needed, could be enabled by writing to
the standard <debugfs>/dynamic_debug/control file.
Debug messages can be enabled/disabled at various granularities
like - module, file, function, a range of line numbers or a
specific line number.

The following kernel document describes this in detail with examples:
https://www.kernel.org/doc/Documentation/dynamic-debug-howto.txt

A few examples on how to use this with bnxt_re driver:

1) To check the debug messages that are available in bnxt_re:
# cat /sys/kernel/debug/dynamic_debug/control | grep bnxt_re

2) To enable all debug messages in bnxt_re during load time:
# insmod bnxt_re.ko  dyndbg==p

3) To enable all debug messages in bnxt_re after loading:
# echo "module bnxt_re +p" > /sys/kernel/debug/dynamic_debug/control

4) To disable all debug messages in bnxt_re after loading:
# echo "module bnxt_re -p" > /sys/kernel/debug/dynamic_debug/control

5) To enable a debug message at a specific line number in a file:
# echo -n "file qplib_fp.c line 2554 +p" > /sys/kernel/debug/dynamic_debug/control


BNXT_RE Compiler Switches
=========================

ENABLE_DEBUGFS - Enable debugFS operation

ENABLE_RE_FP_SPINLOCK - Enable spinlocks on the fast path bnxt_re_qp queue
			resources

ENABLE_FP_SPINLOCAK - Enable spinlocks on the fast path bnxt_qplib queue
		      resources

ENABLE_DEBUG_SGE - Enable the dumping of SGE info to the journal log

SR-IOV and VF Resource Distribution
===================================

Note: Before enabling the VFs, both bnxt_en and bnxt_re drivers should be loaded.
      Loading bnxt_re driver after creating VFs is not supported.

      In distros that support auto loading of bnxt_re based on udev rules,
      (ie. having an entry  ENV{ID_NET_DRIVER}=="bnxt_en", RUN{builtin}+="kmod load bnxt_re"
      in /usr/lib/udev/rules.d/90-rdma-hw-modules.rules)
      if the bnxt_re dirver is unloaded before creating VFs, vf creation loads bnxt_re
      driver. This operation throws error in dmesg as this is considered as loading
      driver after creating VFs. Disable RoCE on the adapter if RoCE feature is not
      required.

If SR-IOV is supported on the adapter, QPs, SRQs, CQs and MRs are distributed
across VF by the bnxt_re driver. Driver allocates 64K of QPs, SRQs and CQs
for the PF pool. It creates 256K MRs for the PF pool. If SRIOV is enabled on
the adapter, and if VFs are created (active VFs), 32K of each of these resources
are reserved for PF and remaining is divided equally amongst active number of VFs.

For eg: Active number of VFs can be obtained from the following command.
	$cat /sys/class/net/p6p1/device/sriov_numvfs

If sriov_numvfs is 2, each VF can create resources (QP, SRQ and CQ)
upto 16K (32K divided across 2 VFs). MRs per VF would be 112K ((256K - 32K) divided
across 2 VFs)

Note: Since PF is in privileged mode, it is allowed to use the
entire PF pool resources. But VFs are restricted to create max configured
by the above calculation. User must ensure that total resources created by
PF and its VFs shall be less than Max configured (64K for QPs/SRQs/CQs and 256K for MRs).

Use following command to get the active resource count.
$cat /sys/kernel/debug/bnxt_re/info

DCB Settings
============
To modify the default DCB settings programmed by driver load, users can
use bnxtqos or lldptool. Some example commands given below. Use bnxt_setupcc.sh
for easy setup.

Note: Since bnxt_re driver enables PFC on ROCE Priority during driver load,
      PFC must be disabled using bnxtqos/lldptool before changing any TC
      mapping. This ensures proper mapping between the user traffic class
      to HW Queues.

bnxtqos
-------
#Dump existing settings
 bnxtqos -dev=<ethx> get_qos
#Disable PFC
 bnxtqos -dev=<ethx> set_pfc enabled=none
#Delete the existing app tlvs
 bnxtqos -dev=<ethx> set_apptlv -d app=3,5,26
 bnxtqos -dev=<ethx> set_apptlv -d app=3,3,4791
#Add App TLVs with for RoCE priority 5 and dscp value 28
 bnxtqos -dev=<ethx> set_apptlv app=5,3,4791
 bnxtqos -dev=<ethx> set_apptlv app=5,1,35093
 bnxtqos -dev=<ethx> set_apptlv app=5,5,28
#Set the ETS and Priority to Traffic Class mapping
 bnxtqos -dev=<ethx> set_ets tsa=0:ets,1:ets priority2tc=0:0,1:0,2:0,3:0,4:0,5:1,6:0,7:0 tcbw=20,80
#Enable PFC on priority 5
 bnxtqos -dev=<ethx> set_pfc enabled=5
#To dump the current QOS settings
 bnxtqos -dev=<ethx> get_qos

lldptool
--------
Note: If the switches are capable of handling RoCE TLVs, the following
settings are not required as adapter will override local settings, if any,
with the switch settings.

Following steps are recommended to configure
the local adapter to set DCB parameters, in case switches are not capable
of DCB negotiations.

# Load L2 driver and make sure port and Link are  UP
 service lldpad start
 lldptool -L -i p6p1 adminStatus=rxtx
#Disable PFC
lldptool -T -i <ethx> -V PFC enabled=none
#Delete the existing app TLVs. For eg:
lldptool -T -i <ethx> -V APP -d app=3,5,26
lldptool -T -i <ethx> -V APP -d app=3,3,4791
#For RoCE-V1 protocol with Priority-5
 lldptool -T -i p6p1 -V APP app=5,1,35093
#For RoCE-V2 protocol with Priority-5
 lldptool -T -i p6p1 -V APP app=5,3,4791
 lldptool -T -i p6p1 -V ETS-CFG tsa=0:ets,1:ets,2:strict,3:strict,4:strict,5:strict,6:strict,7:strict \
    up2tc=0:0,1:0,2:0,3:0,4:0,5:1,6:0,7:0  tcbw=10,90,0,0,0,0,0,0
 lldptool -T -i p6p1 -V PFC enabled=5
 service lldpad restart

Note: Please refer man pages of lldptool, lldptool-app,
lldptool-ets, lldptool-pfc, etc. for more details

Note: VF inherits the PFC settings of the PF. VF doesn't have privilege to
set DCB parameters using lldptool. No need of running lldpad service on the VM.

Note: The driver supports only one priority for RoCE traffic. Please use same
DCB priority for both RoCE-v1 and RoCE-v2 traffic.

Note: The driver by default supports Priority VLAN Tagging i.e it adds a NULL
VLAN tag if a priority is configured for RoCE Traffic, without VLANs being
configured. However, for customers who are interested only in PFC via DSCP,
driver provides a knob to disable the auto VLAN 0 tag insertion.

echo 1 > /sys/kernel/config/bnxt_re/bnxt_re0/ports/1/cc/disable_prio_vlan_tx

Congestion Control
===================

Explicit Congestion Notification (ECN) is a congestion avoidance mechanism.
In this protocol a Congestion Notification Packet(CNP) signals the existance
of congestion to the remote transmitter. Reacting to CNP, the transmitter reduces
the tranmit rate on a transmit-flow for a given time quanta. CNP is generated
by the receiver when it detects congestion in the receive processing pipe.

To export the tuning parameters RoCE driver uses configfs support from linux
kernel. Following are the steps to configure congestion control parameters.

	1. Pre-requisites
	   ===============
		1.a Host base lldpad is configured for RoCE-v2 protocol
		    and a valid priority is assigned to RoCE-v2.
			ref: "LLDPAD configuration" section of this
			     document.

	2. Mount per-port-configfs interface
	   ===================================
		2.a Load RoCE driver
		2.b ls /sys/kernel/config should list directory "bnxt_re"
		2.c Create a directory in configfs-path with the RoCE device name.
		    E.g. for bnxt_re1 use following:
			mkdir -p /sys/kernel/config/bnxt_re/bnxt_re1
		2.d ls /sys/kernel/config/bnxt_re/bnxt_re1
			ls /sys/kernel/config/bnxt_re/bnxt_re1/ports/1/cc/
				cnp_dscp  cnp_prio  apply  cc_mode roce_prio
				ecn_enable  g  inact_cp  init_cr  init_tr
				nph_per_state  rtt  tcp_cp  roce_dscp  ecn_marking
		2.e Change the value of a specific parameter
			echo -n 0x64 > rtt
			echo -n 0x05 > roce_dscp
			...
		2.f Apply the changes to hardware
			echo -n 0x01 > apply
		Note: Any changes will not take effect unless step 2.f is
		      carried out.

		2.g Read back a specific parameter
			cat rtt
			cat roce_dscp
			...

	3. Unmount per-port-configfs interface
	   ====================================
		3.a remove all per-port-configfs mounts as following:
			rmdir /sys/kernel/config/bnxt_re/bnxt_re1
			rmdir /sys/kernel/config/bnxt_re/bnxt_re0
			...

		Note: If configfs is mounted rmmod bnxt_re will fail.
		      It is must to perform step 3.a before issuing
		      rmmod bnxt_re.


Link Aggregation
================
Link aggregation is a common technique that is used to provide
additional aggregate bandwidth and high availability for logical
interfaces that aggregate multiple physical interfaces. Additional
aggregate bandwidth can be achieved by balancing the traffic load
across multiple physical interfaces. High availability can be achieved
by reconfiguring the loads across the active links when one of the
physical links fails.
The concepts of link aggregation can be applied to RoCE also.

The current solution allows a link aggregation only if all of the
following conditions are met:

-> The netdev associated with each RDMA interface is a
   slave of an upper level device.
-> The slave interfaces have the same master interface.
-> Two netdevs on the same physical device are slaves.
-> The link aggregate cannot span separate physical devices.
-> The master interface has exactly two slaves.
-> The bond mode of the master interface is one of the
   following modes: round-robin (mode0),
   active-backup (mode 1), xor (mode 2), or 8023ad (mode 4).

Note: mode 0, 2 and 4 will be handled as active-active mode in HW.

When a LAG is created roce device interface is visible
with name bnxt_re_bond0.

Note: RoCE Bond device is not created if SR-IOV/NPAR is enabled or if the
      adpater is a 4 port adapter. There should be exactly two RoCE
      devices from an adapter when bond is created. If L2 bond is enabled
      on this adapter, RoCE doesn't work on the bnxt_re devices created
      for each slaves.
Note: RoCE Bond is created only if there are two slaves for the bond
      and the slave devices are from the same physical adapter. Multiple
      adapters are not supported.

=> Instructions

Enable LAG:
-> Load bnxt_en and bnxt_re driver
-> Run following commands to add the interfaces to the bond
	modprobe bonding mode=active-backup miimon=100
	ifconfig p6p1 down
	ifconfig p6p2 down
	ip link set p6p1 master bond0
	ip link set p6p2 master bond0
	ifconfig bond0 192.168.1.1 netmask 255.255.255.0 up

Disable LAG:
ip link set p6p2 nomaster
ip link set p6p1 nomaster

Known Issues:

-> Supports only on distros RH 7.2 and later, SLES12 and later.
-> bnxt_re and bnxt_en drivers need to be loaded before creating bond interface.
-> Changing bond mode when RoCE driver is in use can cause system hang.
   E.g. changing the bonding mode while running a user application,
   can cause a system hang.
   Please make sure that no reference to bnxt_re is taken while changing the bond mode.
   Use the following command to check the module usage count
	#lsmod|grep bnxt_re
   For proper removal of bnxt_re devices or update the bond state:
   1. Unmount all active NFS RDMA mounts.
   2. Stop the ibacm service (or any similar service) on systems where OFED is
      installed using the command:
	# service ibacm stop
   3. Stop all user space RoCE applications.

-> Only NETDEV_BONDING_INFO events on the slave interfaces
   are handled for creating/modifying/destroying the bond on
   latest distros (RH 7.3 or later/SLES 12 or later and
   other distros that support NETDEV_BONDING_INFO event). So traffic
   failover doesn't happen with ifup/ifdown of the slave
   interfaces. Fail over should work with an actual link up/down event
   from the switch side.


BNXT_RE Driver Statistics
=======================

The bnxt_re driver supports debugFS which allows statistics and debug parameters be accessed.
To access this information, read the /sys/kernel/debug/bnxt_re/bnxt_re<x>/info file. Each port will be
listed with associated state. The available statistics will vary based on hardware capability, eg:

# cat /sys/kernel/debug/bnxt_re/bnxt_re0/info

bnxt_re debug info:
=====[ IBDEV bnxt_re0 ]=============================
	link state: UP
	Max QP: 0xff7f
	Max SRQ: 0xffff
	Max CQ: 0xffff
	Max MR: 0x10000
	Max MW: 0x10000
	Active QP: 0x2
	Active SRQ: 0x0
	Active CQ: 0x21
	Active MR: 0x4
	Active MW: 0x0
...


Field Explanation:

Device resource limits:
Max QP 		Max number of QP limit
Max SRQ		Max number of SRQ limit
Max CQ		Max number of CQs limit
Max MR		Max number of memory region limit
Max MW		Max number of memory window limit

Active Resources:
Active QP	Number of active QPs
Active SRQ	Number of active SRQs
Active CQ	Number of active CQs
Active MR	Number of active Memory Regions
Active MW	Number of active Memory Windows

Byte and Packet Counters:
Rx Pkts	 	Number of RoCE (v1/v2) packets received
Rx Bytes	Number of RoCE (v1/v2) bytes received
Tx Pkts		Number of RoCE (v1/v2) packets transmitted
Tx Bytes	Number of RoCE (v1/v2) bytes transmitted

Congestion Notification Counters:
CNP Tx Pkts	Number of RoCE CNP packets received
CNP Tx Bytes	Number of RoCE CNP bytes received
CNP Rx Pkts	Number of RoCE CNP packets transmitted
CNP Rx Bytes	Number of RoCE CNP bytes transmitted

RDMA operation Counters:
tx_atomic_req	Number of atomic requests transmitted
rx_atomic_req	Number of atomic requests received
tx_read_req	Number of read requests transmitted
tx_read_resp	Number of read responses transmitted
rx_read_req	Number of read requests received
rx_read_resp	Number of read responses received
tx_write_req	Number of write requests transmitted
rx_write_req	Number of write request received
tx_send_req	Number of send requests transmitted
rx_send_req	Number of send requests received

Recoverable Errors:
Recoverable Errors	Number of recoverable errors detected.  Recoverable errors are
	    		detected by the HW.  HW instructs FW to initiate the recovery
			process.  RC connection does not teardown as a result of these errors.
to_retransmits		Number of retransmission requests
rnr_naks_rcvd		Number of RNR (Receiver-Not-Ready) NAKs received.
dup_req			Number of duplicated requests detected.
missing_resp		Number of responses missing
seq_err_naks_rcvd	Number of PSN sequencing error NAKs received
res_oob_drop_count	Number of packets dropped because of no host buffers
res_oos_drop_count	Number of  out of sequence packets received
rx_roce_discard_pkts	Number of discard packets received
rx_roce_error_pkts	Number of error packets received


Fatal Errors:
max_retry_exceeded	Number of retransmission requests exceeded the max
unrecoverable_err	Number of unrecoverable errors detected
bad_resp_err		Number of bad response errors detected
local_qp_op_err		Number of QP local operation errors detected
local_protection_err	Number of local protection errors detected
mem_mgmt_op_err		Number of times HW detected an error because of illegal bind/fast
			register/invalidate attempted by the driver
remote_invalid_req_err	Number of invalid request received from the remote rdma initiator.
remote_access_err	Number of times H/W received a REMOTE ACCESS ERROR NAK from the peer.
remote_op_err		Number of times HW received a REMOTE OPERATIONAL ERROR NAK from the peer.

Responder errors:
res_exceed_max		Number of times HW detected incoming Send, RDMA write or RDMA read
			messages which exceed the maximum transfer length.
res_length_mismatch	Number of times HW detected incoming RDMA write message payload
			size does not match write length in the RETH.
res_exceeds_wqe		Number of times HW detected Send payload exceeds RQ/SRQ RQE buffer capacity.
res_opcode_err		Number of times HW detected First, Only, Middle, Last packets for
			incoming requests are improperly ordered with respect to the previous packet.
res_rx_invalid_rkey	Number of times HW detected a incoming request with an R_KEY that
			did not reference a valid MR/MW.
res_rx_domain_err	Number of times HW detected a incoming request with an R_KEY that
			referenced a MR/MW that was not in the same PD as the QP on which the
			request arrived.
res_rx_no_perm		Number of times HW detected a incoming RDMA write request with an
			R_KEY that referenced a MR/MW which did not have the access permission
			needed for the operation.
res_rx_range_err	Number of times HW detected an incoming RDMA write request that had
			a combination of R_KEY, VA and length that was out of bounds of the
			associated MR/MW.
res_tx_invalid_rkey	Number of times HW detected a R_KEY that did not reference a valid
			MR/MW while processing incoming read request.
res_tx_domain_err	Number of times HW detected a incoming request with an R_KEY that
			referenced a MR/MW that was not in the same PD as the QP on which
			the RDMA read request is received.
res_tx_no_perm		Number of times HW detected a incoming RDMA read request with an R_KEY
			that referenced a MR/MW which did not have the access permission needed
			for the operation.
res_tx_range_err	Number of times HW detected an incoming RDMA read request that had a
			combination of R_KEY, VA and length that was out of bounds of the associated MR/MW.
res_irrq_oflow		Number of times HW detected that peer sent us more RDMA read or atomic
			requests that the negotiated maximum
res_unsup_opcode	Number of times HW detected that peer sent us a request with an opcode
			for a request type that is not supported on this QP.
res_unaligned_atomic	Number of times HW detected that VA of an atomic request is on a memory
			boundary that prevents atomic execution.
res_rem_inv_err		Number of times HW detected a incoming send with invalidate request in
			which the R_KEY to invalidate did not MR/MW which could be invalidated.
res_mem_error64		Number of times HW detected a RQ/SRQ SGE which points to an inaccessible memory.
res_srq_err		Number of times HW detected a QP moving to error state because the associated
			SRQ is in error.
res_cmp_err		Number of time HW detected that there is no CQE space available on CQ or
			CQ is not in valid state.
res_invalid_dup_rkey	Number of times HW detected invalid R_KEY while re-sending responses to
			duplicate read requests.
res_wqe_format_err	Number of times HW detected error in the format of the WQE in the RQ/SRQ.
res_cq_load_err		Number of times HW detected error while attempting to load the CQ context.
res_srq_load_err	Number of times HW detected error while attempting to load the SRQ context.


QP Information in debugfs
=========================
A debugfs entry is created for each QP under /sys/kernel/debug/bnxt_re/<pci_bdf>/qp_info/.
This gives information about type of QP, QP state, QP number and UDP port information.

For eg:
$ cat /sys/kernel/debug/bnxt_re/0000\:04\:00.0/qp_info/0x82
type     = IB_QPT_RC(2)
state    = IB_QPS_RTS(3)
source qpn       = 130
dest qpn         = 135
source port      = 20521
dest port        = 4791
