Despite all the cool product launches, the swanky floor models of next gen gear, the interesting tech talks, and the inspiring amount of general Cisco enthusiasm at Cisco Live, my favorite part of this (and most) events is the Network Operations Center (NOC). I absolutely adore the concept of lighting up and supporting a large network in a short amount of time coupled with the intermittent cadence that gives the NOC team time to revel, lament, and plan.

The Cisco Live NOC is particularly special given that the expectations here are absolutely unreal. Attendees expect the network to not only function flawlessly, but to be world class. Cisco enthusiasts and fair-weather fans alike expect the NOC to be driven by the latest and greatest platforms, a testament to Cisco’s quality and an enticing example of what could be possible back in their own IT reality. An obvious double edged sword, the usage of latest platforms and integrations brings with it tempting advantages, but comes with a lack of proven maturity and field-tested stability.

Beyond all of these reasons, I just really appreciate a chance to ogle a beautiful cross-architecture setup, and with Cisco Live I have the delightful treat of having intimate experience with most of the solutions on display. This year, I had the privilege of attending Cisco Live US 2019 as a liaison for my customer base. It was a truly awesome experience, but with the dust settled I really want pay homage to the Cisco Live NOC team for their amazing work as the literal backbone to this monumental event.

I only snapped a few pictures of the primary NOC at the San Diego Convention Center, but thankfully they put on an amazing NOC team panel at the end of the week. I did not have the pleasure of attending in person, but will be scoping it out ASAP when the video goes live on the Cisco Live On-Demand Library (and will post a link here). For now, you can check out the Cisco Live NOC panel recordings from the recent events:

Cisco Live 2018 Orlando // Cisco Live Network and NOC Panel - PNLGEN-1001 
https://www.ciscolive.com/global/on-demand-library.html?search=noc#/video/1533846618354002DMvM 

Cisco Live 2019 Barcelona // Cisco Live Network and NOC: Panel Discussion - PNLNMS-1035 
https://www.ciscolive.com/global/on-demand-library.html?search=noc#/video/1549556564761001mUKG

Now, for the good stuff, let’s check out the build! First up, a wide shot of the network on display in the San Diego Convention Center, comprised of most of the important bits of the network including the wireless LAN controllers, the compute/storage infra, and of course, the core routing and switching:

A gorgeous setup, without question! Here are a few closeup shots of each of the racks, starting with the two racks on the far left featuring the campus infrastructure. These racks house the core switches, a pair of Nexus 7706 (which I’m sure will soon be replaced by the Catalyst 9600) along with the wireless LAN controllers. Interesting to note here is the use of both the traditional AireOS controllers (two pairs of the 8540 WLC) along with the new IOS-XE based WLC, the Catalyst 9800-40 (the white box, top left). Awesome to note, there were (120) 3802 APs powered by the C9800 WLCs and SDA, making this the first Cisco Live to leverage the C9800 controller.

On the security front, you’ll spy a number of Firepower security appliances and a Firepower Management Center to rule them all. I really appreciate the modular UCS design of the Firepower 9300…it’s just badass. If you look closely you’ll also see the sleek little pair of Nexus 7702 switches servicing just the WLCs…neato!

Also tucked away in here are the DNA Centers….but wait, why is that one so huge? It must be the new DNA Center Large appliance! In both the 2019 Barcelona event and this iteration in San Diego, they opted to use a total of 3 DNA Center appliances: one dedicated to SDA, one to Assurance, and one as a cold standby.

Where’s the exit? Right smack in the middle, of course, with a beautiful pair of ASR 1009 routers! The (4) ASR 1002-HX (one pair at the top, one barely connected pair at the bottom), are particularly interesting. In the NOC panel discussion they shared that a pair of these routers were used to provide an Encrypted Traffic Analytics (ETA) stream to Stealthwatch. ETA is a Cisco capability that provides enables detection of malicious traffic in encrypted feeds without the need for decryption, using a combination of enhanced NetFlow data and Cisco secret sauce on the backend to identify the malicious traffic patterns. Tucked away neatly at the bottom of this rack is a pair of Stealthwatch appliances which I believe are both Flow Sensors.

Rounding out this dream network is set of redundant data centers, each rocking Cisco HyperFlex and Nexus 9500 switches. Freaking beautiful, these two setups are so clean and extensible it gives me chills. Clearly a bit overpowered for the needs of Cisco Live, but awe inspiring all the same. This data center is powered mainly by VMWare vCenter and ESXi hosts, providing the virtual infra for all the apps and services required to make Cisco Live a reality. Some notable third party VMs, as called out by the NOC team:

• N3N VMs used for facial recognition (4 VMs, each with 32 cores).
• LIPI / Find My Friends (indoor wayfinding from MazeMap)
• Grafana + InfluxDB (graphy analytic goodness)
• VSFTP + HAProxy (FTP services w/ load balancing)

But what about the Cisco VMs and apps? There are far too many to list (check out the NOC panel vid for full details), including the usual suspects like ISE, CMX, Prime Infrastructure, Data Center Network Manager (DCNM), UCSM, Intersight. Here are others that caught my eye:

• Stadium Vision Director (digital signage)
• Video Surveillance Operations Manager
• Cisco Workload Optimization Manager (CWOM) (Monitor Workoads within the DC)
• Cisco Action Orchestrator (Create Automation Workflows, switch configurations)
• Cisco Prime Process Orchestrator (Automation Tasks)

Ok, so maybe this data center setup isn’t as overpowered as I initially thought. It’s really cool to see so many Cisco technologies working together, especially this design which utilizes both converged and hyperconverged clusters in the same rack.

I know I’m not the only one that drools over this stuff, it’s just breathtaking…but what good is all the hardware without the experience and skill to design and execute the actual implementation? Let’s check out the high level network topology:

As is made abundantly obvious by this diagram, the five racks shown here are just the tip of the ISEber; the real bulk of devices used for this event are access switches (mostly Catalyst 9300 switches) and wireless APs (mostly 3700/3800 series with gillaroo antennas in the SDCC). In addition, this centralized infrastructure supported an SDA fabric in the Marriott next door (via BGP peering from the event core N7700 pair and the Marriot core C9500 pair, functioning as fusion devices). There was also a large install base of Meraki access switches (and likely APs) in the Hyatt. More conventional setups can be seen in places like the Hilton, which looks to be rocking a standard collapsed core with the ever reliable combination of Catalyst 9500 and 9300 for core and access, respectively.

My favorite part of any NOC is the automation and monitoring, as this is where the real excitement happens. Cisco Live obviously benefits from having an incredible team of experienced veterans at the helm and, appropriately, has an astounding level of automation and monitoring depth. Here’s a quick snapshot of some of the choice monitoring screens used by the NOC team:

Under the hood, this event was brought up and maintained by a massive level of automation using both Cisco tools, such as DNA Center and Cisco Action Orchestrator, along with open source tools. The big thing to note about this workflow is the extensive use of APIs. It’s no longer sufficient to treat APIs as an afterthought, they must be a pivotal aspect of a solution from the beginning and, ideally, the way in which a product GUI is created (as is the case with products like Cisco’s Viptela). This sentiment was reinforced at the CLUS2019 keynote with the announcement of DevNet certifications!

Truly impressive, stuff! Massive congrats to the Cisco Live US 2019 NOC team for a superb event and a humble thank you from me for flawless, high performance connectivity throughout my entire week. As a token of my appreciation, I decided to create an ode to the Cisco Live NOC team:

An Ode to the Cisco Live NOC

The weight of the world resting on your shoulders
A week that simply cannot fail
The latest and greatest, the sophisticated stories
So little time before they must be unveiled

All eyes watching, expecting perfection
Guarding a legacy of being the best
Science mandates that solutions be proven
What better way to put Cisco to the test