#CLUS

Nexus 9000
Architecture

Tim Stevenson
Distinguished Engineer, Technical Marketing
BRKARC-3222

#CLUS
Session Abstract
This session presents an in-depth study of the architecture of the
latest generation of Nexus 9000 modular and top-of-rack data
center switches. Topics include forwarding hardware, switching
fabrics, and other physical design elements, as well as a discussion
of key hardware-enabled features and capabilities that combine to
provide high-performance data center network services.

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
What This Session Covers
• Latest generation of Nexus 9000 switches with Cloud Scale ASICs
• Nexus 9500 modular switches with Cloud Scale linecards
• Nexus 9300 Cloud Scale top-of-rack (TOR) switches
• System and hardware architecture, key forwarding functions, packet walks

Not covered:

• First generation Nexus 9000 ASIC/platform

architectures

• Nexus 9500 merchant-silicon based architectures

• Other Nexus platforms

• Catalyst 9000 platform

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Agenda
• Data Center and Silicon Strategy
• Cloud Scale ASIC Architecture
• Cloud Scale Switching Platforms
• Packet Walks
• Key Takeaways

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Cisco Data Center Strategy

Application Centric Capture Intent Multicloud Ready Platform Innovation

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Nexus 9000 Cloud Scale Switching Portfolio
Key Elements of the Data Center Strategy

Nexus 9300-EX and 9300-FX/FX2 Nexus 9500 with

Premier TOR platforms
X9700-EX and X9700-FX Modules
Flagship switching modules for Nexus 9500
modular chassis

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
 Building Data Center Fabrics with Nexus 9000
DCNM
DCNM
L3
RR RR L3

VXLAN / L3
L3
EVPN L3

L2
L3 VPC
L3 L3
Hypervisor Hypervisor

Hypervisor

ACI – Turnkey Fabric Standalone – Standalone – Standalone – Traditional

Programmable Fabric Programmable IP Data Center Network
with VXLAN+EVPN Network

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Why Custom Silicon?
• Cisco competitive advantage – vehicle for differentiating
innovations
• ACI policy model + congestion-aware • In-built encryption technologies
flowlet switching • Intelligent buffers
• Flexible forwarding tiles • Streaming hardware telemetry
• Single-pass tunnel encapsulations

• Tight integration between hardware / software / marketing / sales /

support
• Closely aligns hardware designs with software innovations,
strategic product direction, competitive differentiators, serviceability

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Agenda
• Data Center and Silicon Strategy
• Cloud Scale ASIC Architecture
• Cloud Scale Switching Platforms
• Packet Walks
• Key Takeaways

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Cisco Cloud Scale ASIC Family
• Ultra-high port densities → Reduces equipment footprint, enables
device consolidation, denser fabric designs
• Multi-speed 100M/1/10/25/40/50/100G → Flexibility and future
proofing
• Rich forwarding feature-set → ACI, Segment Routing,
single-pass L2/L3 VXLAN routing
• Flexible forwarding scale → Single platform, multiple scaling
alternatives
• Intelligent buffering → Shared egress buffer with dynamic,
advanced traffic optimization
• In-built analytics and telemetry → Real-time network visibility for
capacity planning, security, and debugging
#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
 Slice 0

Cloud Scale Family Members 900G

Slice Interconnect
Slice 1
LSE 900G
• 1.8T chip – 2 slices of 9 x 100G each
Slice 0 LSE – 18 x
• X9700-EX modular linecards; 9300-EX TORs
1.8T 100G
LS1800FX
• 1.8T chip – 1 slice of 18 x 100G with MACSEC
• X9700-FX modular linecards; 9300-FX TORs LS1800FX – 18 x Slice 0 Slice 1
100G 1.6T 1.6T
S6400
Slice Interconnect
• 6.4T chip – 4 slices of 16 x 100G each
Slice 0
• 9364C/9332C TORs; E2 fabric modules Slice 2 Slice 3
1.8T
1.6T 1.6T
LS3600FX2 Slice Interconnect

• 3.6T chip – 2 slices of 18 x 100G with MACSEC Slice 1 S6400 – 64 x 100G

+ CloudSec 1.8T
9300-FX2 TORs
•
LS3600FX2 – 36 x
100G
#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
What Is a “Slice”?
Slice
Ingress Slice 1 Interconnect
• Self-contained forwarding Egress Slice 1
complex controlling subset of
ports on single ASIC
• Separated into Ingress and Ingress Slice 2
Egress functions
Egress Slice 2
• Ingress of each slice connected
to egress of all slices
• Slice interconnect provides non- Ingress Slice n

blocking any-to-any Egress Slice n

interconnection between slices

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
 Slice Forwarding Path
(S6400 /
LS3600FX2 only)
Slice
Ingress → Ingress Forwarding Controller
SSX

Packet Payload
Ingress Ingress Packet
Packets MAC Parser

Lookup
Lookup Key Result

Lookup Pipeline
Replication Slice
Interconnect

Egress Forwarding Controller

Egress
Egress Egress Packet Egress
Buffering / Queuing
Packets MAC Rewrites Policy
/ Scheduling

← Egress

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Ingress Lookup Pipeline
From
Ingress Ingress Forwarding Controller
MAC
Packet To Egress
Parser Slice

Lookup Pipeline
Flex
TCAM
Tiles TCAM
Lookup Lookup
Key Result
Flush

Load
Forwarding Ingress
Flow Table Balancing,
Lookup Classification
AFD / DPP

LSE / LS1800FX /
LS3600FX2 only

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Flexible Forwarding Tiles Flex Flex Flex
Tile Tile Tile
• Provide fungible pool of table entries for
lookups Flex Flex Flex
Tile Tile Tile
• Number of tiles and number of entries in each
tile varies between ASICs
Flex Flex Flex
• Variety of functions, including: Tile Tile Tile
• IPv4/IPv6 unicast longest-prefix match (LPM)
• IPv4/IPv6 unicast host-route table (HRT)
• IPv4/IPv6 multicast (*,G) and (S,G)
• MAC address/adjacency tables Forwarding
• ECMP tables Lookup
• ACI policy

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Flex Tile Routing Templates Default

• Configurable forwarding templates determine flex tile functions

• “system routing template” syntax

• Templates as of NX-OS 9.2(2):

• Default LPM Heavy
• Dual-stack host scale*†
• Internet peering*
• LPM heavy
• MPLS heavy*
• Multicast heavy
• Multicast extra-heavy Multicast Heavy
• Service provider

• Defined at system initialization – reboot required to change profile

* Template does not support IP multicast
† Template not supported on modular Nexus 9500
** Template not supported on TORs
#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
 IP Unicast Forwarding
• Hardware lookup in flex tiles based on (VRF, IPDA)
• Longest-match from hash-based exact match (HRT) + pivot/trie match (LPM)
• Lookup result returns adjacency – directly or via load-sharing decision (ECMP/DLB)

HRT
Longest
(VRF,IPDA) Hash match N ADJ_PTR
Multipath? ADJ

Y
LPM
Load
Pivot/trie ECMP_PTR Sharing ADJ_PTR
ECMP/DLB

Longest prefix match for Next-hop selection

Destination IP

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
 VXLAN Forwarding
Encapsulation
• VXLAN and other tunnel encapsulation/
Outer
decapsulation performed in single pass L2/L3 Tunnel
MACs/
Lookup Destination
IPs/VNID
• Encapsulation
MAC/LPM/HRT ADJ Rewrite
• L2/L3 lookup drives tunnel destination
• Rewrite block drives outer header fields
(tunnel MACs/IPs/VNID, etc.)

• Decapsulation
• Outer lookup determines if tunnel is transit Decapsulation
or terminated on local TEP
• Inner lookup determines final output port Tunnel Inner L2/L3
Decap
and rewrites Lookup Lookup

TEP Table MAC/LPM/HRT Rewrite

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Load Sharing
Equal-Cost Multipath (ECMP) Dynamic Load-Balancing (DLB)
• Static flow-based load-sharing • Supported on leaf switches in ACI
fabric
• Picks ECMP next-hop based on
hash of packet fields and • Congestion aware, flow-based or
universal ID flowlet-based – rebalances
• Source / destination IPv4 / IPv6 flows/flowlets based on path
address (L3) congestion
• Source / destination TCP / UDP
ports (L4)
• L3 + L4 (default)
• GRE key field

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
ECMP versus DLB Load-Sharing

ECMP DLB
#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Multicast Forwarding
• Hardware performs multicast lookups in HRT
• Additional, secondary table for multicast also provisioned (“MC_INFO”)
from flex tiles – RPF check and MET pointer
• MET in egress slice holds local output interface list (OIL)
• Replication is single copy, multiple reads

MRoute RPF, Replication

(VRF,IPSA,IPDA)
Lookups MET PTR List

HRT MC_INFO MET

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
 256 256

Classification TCAM
256 256
256 256 256 256
256 256 256 256
256 256 256 256
• Dedicated TCAM for packet classification 256 256 256 256

• Capacity varies depending on platform 256 256 256 256

256 256 256 256
• Leveraged by variety of features: 256 256 256 256

• RACL / VACL / PACL 256 256 256 256

Ingress Slice Ingress Slice
• L2/L3 QOS
Egress Slice Egress Slice
• SPAN / SPAN ACL
256 256 256 256
• NAT 256 256
256 256
• COPP 256 256 256 256

• Flow table filter (LS1800FX / LS3600FX2) 256 256 256 256

LSE / S6400 LS1800FX / LS3600FX2

4K ingress ACEs / 5K ingress ACEs /
2K egress ACEs per slice 2K egress ACEs per slice

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
TCAM Region Resizing RACL
RACL
RACL
RACL
RACL RACL

• Default carving allocates 100% of TCAM and enables: RACL RACL

• Ingress / Egress RACL RACL QOS

• Ingress QOS QOS QOS

SPAN SPAN
• SPAN
SACL SACL
• SPAN ACLs
FT FT
• Flow table filter (LS1800FX / LS3600FX2 only)
RSVD RSVD
• Reserved regions
Ingress Slice
• Based on features required, user can resize TCAM regions to adjust scale
Egress Slice
• To increase size of a region, some other region must be sized smaller
RACL RACL
• Region sizes defined at initialization – changing allocation requires system RACL RACL
reboot RACL RACL
• Configure all regions to desired size (“hardware access-list tcam region”), save RACL RSVD
configuration, and reload

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
 Cloud Scale Hardware Telemetry

Flow Table (FT) Flow Table Events (FTE) Streaming Statistics

Export (SSX)
• Captures full data-plane • Triggers notifications
packet flow information, based on thresholds / • Streams ASIC statistics
plus metadata criteria met by data- based on user
plane packet flows configuration

Data-Plane Flow Data ASIC State

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Flow Table
• Collects full flow information plus metadata
• 5-tuple flow info
• Interface/queue info
• Flow start/stop time
• Flow latency

• 32K flow table entries per slice

• Direct hardware export
• Leveraged by Network Insights, Netflow, Tetration
• LSE / LS1800FX / LS3600FX2 platforms support
hardware flow table

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
 Flow Table Operation – Network Insights
• Determine if collection enabled for packet Flow Table
(filter TCAM)
Flow Records
• If so, install FT record
• Flush records, encapsulate in IP/UDP Flow data
collection

• Perform lookup and forward on front-panel Filter

port
FT Export
Lookup
Pipeline
Ingress Packets

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
 Flow Table Operation – Netflow Netflow v9 support:
• 9300-FX TORs: 7.0(3)I7(1)
• 9300-EX TORs: 7.0(3)I7(2)
1. Install FT records as usual • 9300-FX2 TORs: 7.0(3)I7(3)
• 9500-EX Modular: 9.2(2)
2. Flush records to switch CPU
3. CPU builds traditional Netflow cache Flow Table NF Cache
FT Export NDEv9
Flow
4. NDEv9 exported to collector(s) on Flow Records
Records
front-panel port or mgmt0
Switch CPU
Flow data
collection
Filter

Lookup
Ingress Packets
Pipeline

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
Flow Table Events
• Triggers notifications based on criteria / thresholds met by
data-plane packet flows
• Collects full flow information plus metadata
• 5-tuple flow info with timestamp
• Interface/queue info
• Buffer drop indication
• Forwarding drop, ACL drop, policer drop indication
• Latency/burst threshold exceeded indication

• Direct hardware export, with flow-level and global throttling

• LS1800FX / LS3600FX2 platforms support triggered flow
table events

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
 Flow Table Events Operation
• Determine if event(s) enabled for packet Flow Table

(filter TCAM) Flow Table Event

FIFO

• If so, collect flow data in FTE FIFO; Throttle Event

throttling
excess events Flow data
collection

• Flush and encapsulate record in IP/UDP Filter

• Perform lookup and forward on front-panel FTE Export

port Lookup
Pipeline
Ingress Packets

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Streaming Statistics Export (SSX)
• Streams ASIC statistics at rapid cadence based
on user config Periodically
pull values
• Interface counters (RMON counters) ASIC
SSX
Stats
• Ingress/egress queue depth and queue drops
• Egress total buffer depth SSX Export

• Egress queue microbursts

• User defines streaming parameters – which Lookup

statistics, how often, and to which collector Pipeline

• Direct export from ASIC to front-panel port – no

switch CPU involvement
• Hardware support in S6400 / LS3600FX2

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Hardware Telemetry Platform Support

Platform ASIC FT FTE SSX

9300-EX LSE ✓ X X

9300-FX LS1800FX ✓ ✓ X

9364C S6400 X X ✓

9300-FX2 LS3600FX2 ✓ ✓ ✓

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Buffering
• Cloud Scale platforms implement shared-memory egress buffered architecture
• Each ASIC slice has dedicated buffer – only ports on that slice can use that buffer
• Dynamic Buffer Protection adjusts max thresholds based on class and buffer occupancy
• Intelligent buffer options maximize buffer efficiency

Slice 0 Slice 0 Slice 1 Slice 0

18.7MB 10.2MB 10.2MB 20MB
Slice 0
Slice Interconnect Slice Interconnect Slice Interconnect
40.8MB
Slice 1 Slice 2 Slice 3 Slice 1
18.7MB 10.2MB 10.2MB 20MB

LSE LS1800FX S6400 LS3600FX2

18.7MB/slice 40.8MB/slice 10.2MB/slice 20MB/slice
(37.4MB total) (40.8MB total) (40.8MB total) (40MB total)
Shared option (10MB/slice +
20MB shared)
#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
 Intelligent Buffering
Innovative Buffer Management for Cloud Scale switches
• Dynamic Buffer Protection (DBP) – Controls buffer allocation for
congested queues in shared-memory architecture
• Approximate Fair Drop (AFD) – Maintains buffer headroom per
queue to maximize burst absorption
• Dynamic Packet Prioritization (DPP) – Prioritizes short-lived flows to
expedite flow setup and completion

Images courtesy of:

https://www.clker.com/clipart-206333.html
https://www.clker.com/clipart-catroon-mouse.html
Miercom Report: Speeding Applications in Data Center Networks
http://miercom.com/cisco-systems-speeding-applications-in-data-center-networks/
#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Dynamic Buffer Protection (DBP)
• Prevents any output queue from consuming more than its fair share of
buffer in shared-memory architecture
• Defines dynamic max threshold for each queue
• If queue length exceeds threshold, packet is discarded
• Otherwise packet is admitted to queue and scheduled for transmission

α
• Threshold calculated by multiplying free memory by configurable, per-
queue Alpha (α) value (weight)
• Alpha controls how aggressively DBP maintains free buffer pages
during congestion events

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
 Alpha Parameter Examples Default Alpha on
Cloud Scale switches

Alpha (α) = 0.5 Alpha (α) = 1 Alpha (α) = 14

40 40 40

35 35 Buffer per queue == 35

free buffer
30 Buffer per queue == 30 30
½ free buffer Buffer per queue ==
Buffer in MB

Buffer in MB
Buffer in MB
25 25 25
14 x free buffer
20 20 20

15 15 15

10 10 10

5 5 5

0 0 0
1 2 4 8 16 32 64 1 2 4 8 16 32 64 1 2 4 8 16 32 64
Number of Congested Queues Number of Congested Queues Number of Congested Queues

Buffer per queue (MB) Free buffer (MB) Buffer per queue (MB) Free buffer (MB) Buffer per queue (MB) Free buffer (MB)

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Buffering – Ideal versus Reality
Ideal buffer state Actual buffer state
Buffer available for burst absorption

Buffer available for burst absorption

Buffer consumed by sustained-
bandwidth TCP flows

Buffer consumed by sustained-

bandwidth TCP flows

Sustained-bandwidth TCP flows Sustained-bandwidth TCP

back off before all buffer flows consume all available
consumed buffer before backing off

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Approximate Fair Drop (AFD)
Maintain throughput while minimizing buffer consumption by elephant flows – keep buffer
state as close to the ideal as possible
Higher-bandwidth elephants – higher AFD drop
probability
Lower-bandwidth elephants – lower AFD drop probability
Distinguish elephant
flows from other flows
Queue admission

Ideal depth AFD

exceeded? Y set? N

Y
Buffer

Non-elephants – no AFD

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Dynamic Packet Prioritization (DPP)
• Prioritize initial packets of new / short-lived flows
• Up to first 1023 packets of each flow assigned to higher-priority
qos-group
<= 1023 packets
Identify unique flows
Drive new, higher
priority

Track per-flow SP queue

packet count

Maintain original
priority

> 1023 packets Q-default

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
 Queuing and Scheduling

qos-group qos-group qos-group qos-group qos-group qos-group qos-group qos-group SPAN

CPU
0 1 2 3 4 5 6 7

Strict Best
Configurable Weights / Priority
Priority Effort
Egress
Port

Final
Winner
• 8 qos-groups per output port
• Egress queuing policy defines priority and weights
• Dedicated classes for CPU traffic and SPAN traffic

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Agenda
• Data Center and Silicon Strategy
• Cloud Scale ASIC Architecture
• Cloud Scale Switching Platforms
• Packet Walks
• Key Takeaways

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Cloud Scale Platforms
Nexus 9300-EX and 9300-FX/FX2 Nexus 9500 with X9700-EX and
X9700-FX Modules
• Premier TOR platforms
• Switching modules for Nexus
• Full Cloud Scale functionality
9500 modular chassis
• ACI leaf / standalone leaf or spine
• Full Cloud Scale functionality
• FX option with MACSEC using
• ACI spine / standalone
LS1800FX silicon
aggregation or spine
• FX2 option with key
• FX option with MACSEC using
enhancements using LS3600FX2
LS1800FX silicon
silicon

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Nexus 9300-EX Cloud Scale TOR Switches

48-port 10/25G SFP28 + Key Features

6-port 100G QSFP28
N9K-C93180YC-EX – LSE-based
ACI: 1.3(1)
NX-OS: 7.0(3)I4(2) Dual capability – ACI and NX-OS mode
Flexible port configurations –
48-port 1/10GBASE-T + 1/10/25/40/50/100G
6-port 100G QSFP28 Native 25G server access ports
N9K-C93108TC-EX – LSE-based
ACI: 2.0(1) Flow Table for Tetration Analytics, Netflow
NX-OS: 7.0(3)I4(2)
Smart buffer capability (AFD / DPP)
32-port 40G/100G QSFP28
N9K-C93180LC-EX – LSE-based
ACI: 2.2(1)
NX-OS: 7.0(3)I6(1)

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
Nexus 9300-EX Switch Architectures

CPU LSE

1-48 49-54
Front Panel Ports

C93180YC-EX (10/25G + 100G) /

C93108TC-EX (10G + 100G)

CPU LSE

1-28 29-32
Slice 0
Slice 1
Front Panel Ports

C93180LC-EX (40G + 100G)

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Nexus 9300-FX Cloud Scale TOR Switches –
Pervasive MACSEC
48-port 10/25G SFP28 + Key Features
6-port 100G QSFP28
N9K-C93180YC-FX –
LS1800FX-based Dual capability – ACI and NX-OS mode
ACI: 2.2(2e)
NX-OS: 7.0(3)I7(1) Flexible port configurations –
100M/1/10/25/40/50/100G
48-port 1/10GBASE-T + Line-rate 256-bit encryption on all ports
6-port 100G QSFP28
N9K-C93108TC-FX –
32G FC support on all SFP ports
LS1800FX-based
ACI: 2.2(2e)
25G distances beyond 3m (RS-FEC)
NX-OS: 7.0(3)I7(1)
Flow Table for Tetration Analytics,
Network Insights, Netflow
48-port 100M/1GBASE-T +
Smart buffer capability (AFD / DPP)
4-port 10G/25G + 2-port
100G QSFP28
N9K-C9348GC-FXP –
LS1800FX-based
ACI: 3.0(1)
NX-OS: 7.0(3)I7(1)

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Nexus 9300-FX Switch Architectures

CPU LS1800FX

1-48 49-54
Front Panel Ports

C93180YC-FX (10/25G + 100G) /

C93108TC-FX (10G + 100G)

CPU LS1800FX

1-48 49-52 53-54

Slice 0
Front Panel Ports

C9348GC-FXP (100M/1G + 10/25G + 100G)

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Nexus 9364C / 9332C 100G Cloud Scale Switches

Key Features

Dual capability – ACI and NX-OS mode

64-port 100G QSFP28
+ 2-port 10G SFP+ Compact, high-performance fixed ACI spine
N9K-C9364C – S6400-based
ACI: 3.0(1) 100G/50G/40G/10G (single port mode – no
NX-OS: 7.0(3)I7(2)
breakout)
2 x 100M/1G/10G SFP+ ports
MACSEC/CloudSec on 16 / 8 ports
Streaming Statistics Export (SSX)
Smart buffer capability (AFD / DPP)
32-port 100G QSFP28
+ 2-port 10G SFP+
N9K-C9332C – S6400-based
ACI: 4.0(1)
NX-OS: 9.2(3)
#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
 Nexus 9364C / 9332C Switch Architecture
CPU S6400

CloudSec CloudSec CloudSec CloudSec CloudSec CloudSec CloudSec CloudSec

1 2 3 4 5 6 7 8

65-66
1-48 49-64 (10G)
Front Panel Ports
C9364C (100G + 10G)

CPU S6400

CloudSec CloudSec CloudSec CloudSec

1 2 3 4

Slice 0 Slice 2 33-34

Slice 1 Slice 3 1-24 25-32 (10G)
Front Panel Ports
C9332C (100G + 10G)
#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
Nexus 9300-FX2 Cloud Scale TOR Switches

Key Features

Dual capability – ACI and NX-OS mode

36-port 100G QSFP28

Versatile standalone 100G switch
N9K-C9336C-FX2 – LS3600FX2-based High-performance 100G ACI leaf switch
ACI: 3.1(2)
NX-OS: 7.0(3)I7(3) (9336C)
100G/50G/40G/10G with breakout capability
Flow Table for Tetration Analytics, Network
Insights, Netflow
Streaming Statistics Export (SSX)
48-port 10/25G SFP28 + MACSEC/CloudSec on all ports
12-port 100G QSFP28 Smart buffer capability (AFD / DPP)
N9K-C93240YC-FX2 – LS3600FX2-based
ACI: 4.0(1)
NX-OS: 7.0(3)I7(3)

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
 Nexus 9300-FX2 Switch Architecture
CPU LS3600FX2

1-36
Front Panel Ports

C9336C-FX2 (100G)

CPU LS3600FX2

1-48 49-60
Slice 0
Front Panel Ports
Slice 1

C93240YC-FX2 (10/25G + 100G)

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
 Nexus 9500 Modular Cloud Scale Switches

+ +
Nexus 9504 Nexus 9508 Nexus 9516

Common Equipment E/E2-Series Fabric

Modules
EX / FX Series Line Cards

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
 X9700-EX 100G Cloud Scale Modules
N9K-X9732C-EX / N9K-X9736C-EX
Key Features

9732C-EX – Dual capability ACI and NX-OS

9736C-EX – NX-OS only
32-port 100G QSFP28 Line-rate performance up to 3.2Tbps capacity
X9732C-EX – LSE-based
ACI: 1.3(1)
NX-OS: 7.0(3)I4(2)
Flexible port configurations –
10/25/40/50/100G with breakout
Flow Table for Netflow, Tetration
Smart buffer capability (AFD / DPP)

36-port 100G QSFP28

X9736C-EX – LSE-based
ACI: Not supported
NX-OS: 7.0(3)I6(1)

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
 N9K-X9732C-EX / N9K-X9736C-EX Architecture
8 x 100G
with speedup

EOBC To Fabric Modules EOBC To Fabric Modules

LC LSE 1 LSE 2 LSE 3 LSE 4 LC LSE 1 LSE 2 LSE 3 LSE 4

CPU CPU

1-8 9-16 17-24 25-32 1-9 10-18 19-27 28-36

Front Panel Ports Front Panel Ports

X9732C-EX X9736C-EX
8 x 100G front- 9 x 100G front-
panel ports per LSE panel ports per LSE
Slice 0
Slice 1

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
 X9700-FX 100G Cloud Scale Modules
N9K-X9732C-FX / N9K-X9736C-FX
Key Features

9732C-FX – NX-OS only

9736C-FX – Dual capability ACI and NX-OS
3.2Tbps capacity line-rate performance at 170-byte
32-port 100G QSFP28 frames
X9732C-FX – LS1800FX-based N+1 fabric redundancy on 9732C-FX
ACI: Not supported
NX-OS: 9.2(1) 3.6Tbps capacity with optional 5 th fabric module on
9736C-FX
Flexible port configurations – 10/25/40/50/100G with
breakout
Line-rate MACSEC on all ports
CloudSec encryption (8 ports) on 9736C-FX
36p 100G QSFP28
X9736-FX – LS1800FX-based Flow Table for Network Insights, NetFlow (roadmap),
ACI: 13.0(1) Tetration
NX-OS: NX-OS: 7.0(3)I7(3)
Smart buffer capability (AFD / DPP)
#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
 N9K-X9732C-FX / N9K-X9736C-FX Architecture
Optional 9th & 10th
EOBC To Fabric Modules 100G FM links

8 x 100G with
LC LS1800FX 1 LS1800FX 2 LS1800FX 3 LS1800FX 4 speedup
Optional 9th
CPU
100G FM link

1-8 9-16 17-24 25-32 8 x 100G with

Front Panel Ports speedup

X9732C-FX EOBC To Fabric Modules

8 x 100G MACSEC
capable ports per
LS1800FX LC LS1800FX 1 LS1800FX 2 LS1800FX 3 LS1800FX 4
CPU
CloudSec CloudSec CloudSec CloudSec
1 2 3 4

1-7 29-30 8-14 31-32 15-21 33-34 22-28 35-36

Front Panel Ports

X9736C-FX
Slice 0 9 x 100G front- 7 x 100G MACSEC
panel ports per 2 x 100G MACSEC /
capable ports per CloudSec capable
LS1800FX LS1800FX ports per LS1800FX
#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
9732C-EX / 9736C-EX / 9732C-FX / 9736C-FX
Fabric Connectivity

FM 2 FM 3 FM 4 FM 6

800G 800G 800G 800G 800G 800G 800G 800G

3200G (3.2T) 3200G (3.2T) 3.2T per LC slot

9732C-EX/FX / 9732C-EX/FX /
9736C-EX/FX 9736C-EX/FX

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
9732C-FX Fabric Connectivity – 5 FMs
Optional

FM 2 FM 3 FM 4 FM 5 FM 6

800G 800G 800G 800G 800G 800G 800G 800G 800G 800G

4.0T per LC slot

(3.2T max front
4000G (4.0T) 4000G (4.0T)
panel bandwidth)

9732C-FX 9732C-FX

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
9736C-FX Fabric Connectivity – 5 FMs
Optional

FM 2 FM 3 FM 4 FM 5 FM 6

800G 800G 800G 800G 800G 800G 400G 400G 800G 800G

3600G (3.6T) 3600G (3.6T) 3.6T per LC slot

9736C-FX 9736C-FX

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Fabric Redundancy – 5 FMs
Limitations and Notes
• All modules installed in chassis must be either 9732C-FX or
9736C-FX to use 5 FMs
• If other module type installed, 5th FM powered off automatically
• 9732C-FX:
• 5 FMs required for N+1 fabric module redundancy
• 9736C-FX:
• 5 FMs required for full bandwidth
• Bandwidth reduction on FM failure varies depending on which FM failed
Note: 5 x FMs supported on all chassis types in standalone from 7.0(3)I7(2). 5 x FMs with 9736C-
FX supported from in ACI 13.2(2).

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
 X9700-EX/FX EOR/MOR Cloud Scale Modules
N9K-X97160YC-EX / N9K-X9788TC-FX
Key Features

NX-OS mode only

Flow Table for Tetration Analytics, NetFlow
Smart buffer capability (AFD / DPP)
48p 10/25G SFP+ and 4p 97160-EX:
100G QSFP28 1.6Tbps capacity with line-rate performance
X97160YC-EX – LSE-based
ACI: Not supported Flexible port configurations –1/10/25G SFP28
NX-OS: 7.0(3)I5(2)
ports, 1/10/25/40/50/100G QSFP28 ports
9788-FX:
880Gbps capacity with line-rate performance
48p 1/10GBASE-T and 4p Flexible port configurations –1/10GBASE-T
100G QSFP28 ports, 1/10/25/40/50/100G QSFP28 ports
X9788TC-FX – LS1800FX-based
ACI: Not supported Line-rate MACSEC on all ports
NX-OS: 7.0(3)I7(3)

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
 N9K-X97160YC-EX /
N9K-X9788TC-FX Architecture
8 x 100G
with speedup
To Fabric Modules
EOBC

LC LSE 1 LSE 2 8 x 100G with

CPU speedup
To Fabric Modules
EOBC
1-12 25-36 49 50 13-24 37-48 51 52
Front Panel Ports
LC LS1800FX 1 LS1800FX 2
X97160YC-EX CPU

1-24 49 50 25-48 51 52
24 x 10/25G and 2 x 100G Front Panel Ports
front-panel ports per LSE
24 x 1/10G and 2 x
X9788TC-FX
100G front-panel
ports per LS1800FX Slice 0
Slice 1

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
MACSEC Hardware Encryption
• Provides link-level hop-by-hop MACSEC Frame Format
encryption
DMAC
• IEEE 802.1AE 128-bit and 256-bit AES SMAC
encryption with MKA Key Exchange DMAC SEC-Tag

• Native hardware support available on: SMAC EType

EType Payload
• All ports on X9736C-FX linecard
Payload ICV
• All ports on Nexus 93180YC-FX / 93108TC-
FX switches FCS FCS
Original Frame Encrypted Frame
• 16 x 100G ports on Nexus 9364C switch
• All ports on Nexus 9336C-FX2 / N9K-
C93240YC-FX2 switches

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
 MACSEC
Original Frame Encrypted Frame Original Frame Encrypted Frame Original Frame
DMAC DMAC DMAC DMAC DMAC
SMAC SMAC SMAC SMAC SMAC
EType SEC-Tag EType SEC-Tag EType
Payload EType Payload EType Payload
Original Frame Original Frame
FCS Payload FCS Payload FCS
DMAC ICV ICV DMAC
SMAC FCS FCS SMAC
EType EType
Payload Payload
FCS MACSEC
256-bit
MACSEC
256-bit
MACSEC
256-bit
MACSEC
256-bit FCS

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
 CloudSec Hardware Encryption
• Provides VTEP-to-VTEP encryption CloudSec Frame Format

DMAC
• Encrypts VXLAN header and payload
SMAC
for transport over arbitrary IP network
DMAC EType
• Hardware support available on: SMAC IP
• 8 x 100G ports on X9736C-FX linecard EType UDP*
• 16 x 100G ports on Nexus 9364C IP SEC-Tag
• All ports on 9300-FX2 TORs UDP VXLAN
VXLAN Payload
• No support on other TOR switches
Payload ICV
FCS FCS
Original VXLAN Packet Encrypted Packet

* CloudSec UDP dest port

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
 CloudSec
Encrypted Packet
DMAC
Original Original
VXLAN Packet SMAC VXLAN Packet
DMAC EType DMAC
SMAC IP SMAC
EType UDP* EType
IP SEC-Tag IP
UDP VXLAN UDP
VXLAN Payload VXLAN

Original Frame
Payload ICV Payload Original Frame
FCS FCS FCS
DMAC DMAC
SMAC SMAC
EType EType
Payload Payload
VTEP VTEP
FCS CLOUDSEC
256-bit
CLOUDSEC
256-bit FCS

* CloudSec UDP dest port

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
Agenda
• Data Center and Silicon Strategy
• Cloud Scale ASIC Architecture
• Cloud Scale Switching Platforms
• Packet Walks
• Key Takeaways

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
 Packet Walk (TOR) – IP Unicast Receive from Rewrite SMAC/DMAC
Slice 0, transmit TTL decrement
to Slice 1 Write QOS fields

93180YC-EX

LSE

Slice Interconnect

IFC EFC
Buffering /
Packet Lookup Egress
MAC Queuing / Rewrites MAC
Parser Pipeline Policy
Scheduling
Slice 0 Slice 1

Receive frame MTU checks Longest-match prefix lookup Receive from slice interconnect Egress policy FCS generation
FCS checking Extract header fields Adjacency/ECMP pointer Buffer packet in queue (DBP) enforcement Transmit frame
VLAN checking Generate lookup key DLB/ECMP decision AFD drops
Ingress policy enforcement Scheduling
AFD/DPP
Flow table

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
 Policy and rewrites for

Packet Walk (TOR) – Multicast

MET lookup
each copy
Replication to local OIL
9364C
S6400
Slice 2 Slice 3
Replication

Slice with no receivers

drops packet
Buffering / Buffering /
Egress
Queuing / Queuing / Rewrites MAC
Policy
Scheduling Scheduling
EFC

Replication to all
slices Slice Interconnect

(*,G) or (S,G) lookup

RPF check EFC
MET pointer Buffering /
IFC Egress
Queuing / Rewrites MAC
Packet Lookup Policy
MAC Scheduling
Parser Pipeline

Replication
Slice 0 Slice 1

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
 Packet Walk (Modular) – Multicast
Multicast replication to
Replication to all FM-E2 local OIFs (fabric links)
slices S6400
(*,G) or (S,G) lookup
Slice Interconnect
MET pointer

(*,G) or (S,G) lookup

MET lookup Slice 0 Slice 1 Slice 2 Slice 3 MET pointer
Replication to local OIL
(front panel and/or
fabric)

LSE 1 LSE 2 LSE 3 LSE 4 LSE 1 LSE 2

Slice 1 Slice 1 Slice 1

Replication to all
Slice Interconnect Slice Interconnect slices Slice Interconnect

Slice 0 Slice 0 Slice 0

X9732C-EX / X97160YC-EX
Replication to all (*,G) or (S,G) lookup X9736C-EX
slices RPF check
MET pointer Multicast
replication to local
OIFs

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
 Packet Walk – VXLAN Encapsulation Add L2 / IP / UDP / VXLAN header

9336C-FX2

LS3600FX2

Slice Interconnect

IFC EFC
Buffering /
Packet Lookup Egress
MAC Queuing / Rewrites MAC
Parser Pipeline Policy
Scheduling
Slice 0 Slice 1

L2/L3 lookup
Adjacency pointer
Remote tunnel endpoint
ECMP decision

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
 Packet Walk – VXLAN Decapsulation

9336C-FX2

LS3600FX2

Slice Interconnect

EFC IFC
Buffering /
Egress Lookup Packet
MAC Rewrites Queuing / MAC
Policy Pipeline Parser
Scheduling
Slice 0 Slice 1

Destination TEP lookup

Remove outer headers Inner L2/L3 lookups
Rewrite SMAC/DMAC Adjacency/ECMP pointer Tunnel check
Decrement TTL ECMP decision Extract header fields
Generate lookup key

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Agenda
• Data Center and Silicon Strategy
• Cloud Scale ASIC Architecture
• Cloud Scale Switching Platforms
• Packet Walks
• Key Takeaways

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
 Building Data Center Fabrics with Nexus 9000
DCNM
DCNM
L3
RR RR L3

VXLAN / L3
L3
EVPN L3

L2
L3 VPC
L3 L3
Hypervisor Hypervisor

Hypervisor

ACI – Turnkey Fabric Standalone – Standalone – Standalone – Traditional

Programmable Fabric Programmable IP Data Center Network
with VXLAN+EVPN Network

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
Key Takeaways
• You should now have a thorough
understanding of the Nexus 9000
Cloud Scale switching platform
architecture
• Feature-rich, innovative switching
platform addresses virtually every
deployment scenario
• Nexus 9000 Cloud Scale platform
forms foundation of Cisco Data
Center strategy

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
Complete your
online session • Please complete your session survey
evaluation after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live water bottle.
• All surveys can be taken in the Cisco Live
Mobile App or by logging in to the Session
Catalog on ciscolive.cisco.com/us.
Cisco Live sessions will be available for viewing
on demand after the event at ciscolive.cisco.com.

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session

How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space

Webex Teams will be moderated cs.co/ciscolivebot#BRKARC-3222

by the speaker until June 16, 2019.

#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Continue your education

Demos in the
Walk-in labs
Cisco campus

Meet the engineer

Related sessions
1:1 meetings

#CLUS BRKARC-3222 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
Thank you

#CLUS
#CLUS