Ever had to stand up a Security Operations Center (SOC) in two days? This is the monumental challenge faced by Cisco engineers at various events and conferences around the globe throughout the year. You may ask, “How is it possible to deploy a full-fledged SOC with just two days of preparation?” The key to being able to make the nearly impossible happen is our custom “SOC in a Box”. It’s essentially a roadshow case, racked with the required hardware for a SOC, that can be packed and shipped to any location. In this blog, I’ll go through the phases of preparing the kit from ideation in San Jose to implementation at the RSA Conference in San Francisco.
Phase 1: Dusting off the cobwebs
Arriving at the Cisco campus in San Jose, California, and walking into the lab on Monday morning one week before RSAC was so nostalgic. It reminded me of my days as a TAC (Cisco Support team) engineer doing customer “recreates” (recreate issues reported by customers) in the lab. What a sight to behold, a multi-story office building entirely dedicated to lab space!
When we finally found our gear, the case looked dusty… like it hadn’t been touched in a year (because it hadn’t). Really the case just needed a little tender loving care. We started with a drawing of what we wanted to build: In the depiction the internet cloud is actually the Moscone Center network and is not managed/secured by RSA
Most of this phase involved cleaning out the case, removing any unnecessary hardware, securing the remaining hardware with proper rackmounts and screws, and adding zip ties for power cable management.
Next, we needed to reimage the UCS C220 M5 and install the ESXi 8.0, a robust, bare-metal hypervisor that installs directly onto your physical server. Here is where the hurdles begin! After creating a bootable USB thumb drive, we faced an issue with the server not recognizing the drive. Shout out to Robert Harris for setting up CIMC and using the browser based KVM to upload the ISO file.
With the server sorted, it was time to move on to the switch. After a “write erase” of the config, we noticed the switch only had two 10G interfaces, another hurdle as we needed a minimum of four 10G interfaces. After lunch, we made a quick stop at the Cisco “repot depot” storefront in Building 9 to pick up a “nm-4-10g” network module for the Catalyst 3850. After a bit of networking Layer 1 troubleshooting, we realized the switch was not recognizing the network module. We also tried to reimage the switch from rommon and install the latest software but that didn’t resolve the issue either. Shout out to Matt Vander Horst, who helped us clear this hurdle by looking up the spec sheet and discovering that the 24 port Catalyst 3850 does not support the 4x10G network module and that we would need a 48 port Catalyst 3850.
With the switch on pause, we moved on to the Cisco Firepower 4125 Firewall. In the RSAC SOC, we typically like to run the latest and greatest software releases so we can showcase the new features and put our Cisco security tools to the test in a complex, real-world environment. This firewall needed an FXOS upgrade to run FTD 7.4.1. Although FXOS 2.14 installed successfully, we came to the next hurdle when we noticed a fault with one of disks in the chassis. Dinkar Sharma helped us with the disk fault but, even after opening a TAC case and getting support from Ravi Kiran Nagaraja, the issue persisted. Shout out to Justin Murphy and Shannon Wellington for delivering an 800 GB SSD drive from their lab on short notice as our last-ditch effort. With the new disk installed we crossed our fingers but to no avail. Again, the same error regarding a failure to format the disk which indicates an issue with the chassis itself.
At this point, our “SOC in a Box” could have been a failure. The shipping deadline was approaching fast, and we didn’t have the necessary switch or a working Firewall. Talk about a major hurdle!
Phase 2: Beg, borrow, and steal (not really, because we asked nicely)
After a simple exchange on Webex teams, Zohreh Kehzri came to the rescue with a 48 port Catalyst 3850 with eight 10G ports! We walked over to building 17 (getting our steps in around the San Jose campus) to pick up the 3850 and, one more reimage later, we had a functioning switch, finally getting us over this hurdle. After the struggles of phase 1, we were glad to take a quick win. With the new switch racked in the case, it was time to drop our homegrown unit off for shipping before we headed over to the Security Summit. Here is what our “SOC in a Box” looked like right before we shipped it.
At the Security Summit, I spotted Eric Kostlan, the resident firewall guru. Knowing that we were in desperate need of a hardware firewall, I went back to the “beg, borrow, and steal” approach, asking Erik if he could help. In not-so-shocking fashion, he checked his lab environment and sourced a spare firewall. After hearing of the issues we faced with the other chassis, he even made the extra effort to ensure FXOS 2.14 was installed successfully and the security engine came up healthy, getting us over one more hurdle.
Once the sessions at the Security Summit were over around 6:30 pm, we went to Eric’s lab and borrowed the firewall out of his racks before heading to dinner. The next day, I hoisted the new FTD 4115 into an Uber XL and headed to San Francisco to get ready for the conference. (A network engineer’s dream to Uber a firewall from city to city!)
Now that we have acquired all the components of the puzzle, it’s time to put the pieces together.
Phase 3: Power it up, wire it up
On Saturday morning, May 4, Moscone Center in San Francisco was buzzing with conference preparation. It’s truly mindboggling to see the show floor transform from bare concrete to a completed showcase in 48 hours. I picked up my badge and wheeled the case over to the South Expo. Here is what the case looked like next to the 10G fiber drop before any set up was started.
This phase is mostly powering up the hardware and wiring it with internet access, management access, and the SPAN (Switched Port Analyzer is a dedicated port on a switch that takes a mirrored copy of network traffic from within the switch to be sent to a destination) from Moscone Network Operations Center. Shout out to Ryan Maclennan for working with the on-site technicians to ensure Layer 1 on the 10G SPAN was working correctly. The 24 port Catalyst 3850 was used for the SOC management network, a subnet provided by the Moscone Center. After re-IP-addressing the management interfaces of all our devices, the basis of the network was online.
In these situations, it is imperative to be flexible. Since we were uncertain on how to change the IP addresses of the Cisco Telemetry Broker (CTB) manager and CTB broker node, we quickly pivoted the Observable Network Appliance (ONA), which would accomplish the same goal of converting the SPAN to IPFIX (Internet Protocol Flow Information Export) to pump up to Cisco XDR.
Additionally, we finished the Firewall logical device installation and connected the SPAN to a passive interface and completed the rest of the basic configuration from the Cisco Secure Firewall Management Center (FMC). Next, we installed Splunk Enterprise Security (ES) on an Ubuntu machine and configured the Splunk Technical Add-ons (TAs) for Cisco XDR integration, eStreamer log ingestion, and Firewall dashboarding. Shout out to Seyed Khadem-Djahaghi for the custom dark mode dashboard he created in the Splunk console.
Here is what our custom “SOC in the Box” looked like wired up and fully operational, connected to the Moscone NOC and NetWitness Platform. We have room for NetWitness appliances and their 140TB of storage for those network packets.
Phase 4 – Big time on the big screens
With our “SOC in a Box” operational and all our tools online, it was time for the finishing touches of putting up the pretty dashboards on the big. On Sunday afternoon, we were able to login to the Cisco Security tools and showcase them on the “SOC Dashboard” on public display between North and South Expo. At this point, it felt like we had successfully finished the race and cleared all the hurdles. Here’s what it looked like before the show opened; Cisco Secure Cloud Analytics, Cisco XDR, Splunk ES, and FMC were on the big screens.
We had a lot of visitors during show hours examining the SOC Dashboard.
On Tuesday morning when we came into the SOC, we ran into that unexpected final hurdle – the Splunk was down! After checking on the command line interface, we found that the disk was full – the 2TB we had originally allocated had been used. Luckily, we had a spare UCS C240 M4 with 18TB of storage in our “SOC in a Box”, we borrowed a VGA monitor and USB keyboard from the RSA A/V team so we could spin up the server on the fly and allocate more storage to Splunk ES. Hurdle cleared, and we coasted to a successful finish.
During our SOC tours, we explained to the conference attendees (including our very own Engineering SVP, Shaila Shankar) how we are using our tools for threat hunting and incident response! (Above is one of many selfies I have taken with Shaila.)
Components Used:
- Switch: Catalyst 3850 (24 port)
- Switch: Catalyst 3850 with 10G SFP+ (48 port)
- Firewall: Secure Firewall 4115
- Server: UCS C220 M5
- Server: UCS C240 M4
In the topology shown above, the purple box encompasses our on-premises “SOC in a Box” infrastructure. Starting in the bottom right, the Umbrella Virtual appliances are deployed within the Moscone Network Operations Center. By assigning the virtual appliances as the DNS servers in the DHCP scope all DNS queries on the network are visible to Cisco Umbrella – User Protection Suite.
Next, the SPAN of all conference network traffic is plugged into the Catalyst 3850, which is essentially being used as a SPAN replicator. From the switch, the SPAN traffic is sent to a Secure Firewall 4115 in Intrusion Detection mode for deep packet inspection, an On-premises network appliance (ONA) to get IPFIX (Internet Protocol Flow Information Export) data to XDR, and to NetWitness, where the full pcap (packet capture) is stored.
Firewall Management Center (FMC) uses eStreamer to send detection and connection data to Splunk and NetWitness. Files are sent to Malware Analytics from both FMC and Netwitness. Cisco XDR integrates with Umbrella, Secure Firewall, Malware Analytics, NetWitness, Splunk, and numerous threat intel sources for threat hunting and incident response.
A new addition to our SOC this year was Cisco Secure Access. By deploying the resource connector in our ESXi, the on-premises gear is accessible from anywhere provided proper authentication has taken place. Our custom “SOC in a Box” was one of the highlights of the SOC tours and generated quite a bit of excitement around Cisco Security!
So long RSAC 2024!! We’ll be back again next year!
To learn more:
Thanks to:
- Robert Harris
- Matt Vander Horst
- Dinkar Sharma
- Eric Kostlan
- Ryan Maclennan
- Seyed Khadem-Djahaghi
- The RSA Conference staff
- The Moscone Network Operations Center
- And the entire Cisco and NetWitness RSAC SOC team members
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Security on social!
Cisco Security Social Channels
Instagram
Facebook
Twitter
LinkedIn
Share:
