2018-Jan-31

Did you know? You can share this story using the social media icons on the left. Please include the hashtag #WeAreCisco. You can also rate and comment on the story below.

Security Doesn’t Just Happen

The Computer Security Incident Response Team, aka CSIRT, identifies, investigates, engages and prevents security incidents.  These can be violations of policy, law or unacceptable and malicious activity that affect the running of Cisco. That means phishing attacks that compromise employee laptops, malware attacks on data center servers, threats targeting sensitive data stored in the cloud and more.

Remember the WannaCry ransomware attack in May 2017? It’s a prime example of what CSIRT exists to thwart. WannaCry crippled over 400,000 computers running Microsoft Windows and caused billions of dollars in damage globally. Its impacts on Cisco could have been much worse without the efforts of CSIRT.

“A lot of different groups within Cisco had to pitch in,” recalls Tammy Nguyen, an information security analyst and one of the team’s newer members. “Internally, CSIRT helped with detection and remediation, as well as working with IT and our partners to make sure their systems were patched.”

CSIRT forms part of the investigative branch of our Information Security (InfoSec) organization. Today, CSIRT has about 100 team members in eight countries who are on call 24 x 7, offering follow-the-sun support.

Multiple Teams

There’s a common misconception that having an IPS (intrusion prevention system) and collecting system alerts makes a CSIRT, however—it’s a combination of technologies, people and process. “You’ve got to have those three components in order to have an incident response capability,” says Chris Fry, CSIRT team lead.

With that in mind, CSIRT consists of three teams:

The analysis team monitors hundreds of thousands of alerts every day, looking for attacks. The efficiency of this process is greatly enhanced by automated workflows that are codified in a playbook. (In fact, three CSIRT team members—Jeff Bollinger, Brandon Enright and Matthew Valites—have written a book on this approach titled “Crafting the InfoSec Playbook.”) Once the team identifies a problem, they partner with Operations or IT to alleviate it.

The investigations team deals with new and more specialized incidents. “If we have been attacked, that’s when our investigators get involved, which includes briefing directors and VPs,” Chris says.

The engineering team is responsible for the deployment and operations of most of Cisco’s equipment and data as well as the development of in-house solutions. There’s also a threat intelligence team that works with different teams both internal and external, collecting and sharing threat intel with the likes of Talos (our Security Intelligence Team) as well as industry peers and others.

Team members run the gamut in experience. Tammy joined CSIRT’s threat analysis team as an intern out of university, learning the skills on the job. At the other end of the spectrum are people like Chris and Christopher Benson—both 20-year Cisco security veterans who joined the company when the entire IT organization consisted of just 200 people. Chris remembers when CSIRT consisted of just four people when it was born in 2003.

“We used to wear all the hats,” says Chris, a founding member of CSIRT who today leads the team’s threat intelligence function. “You did analysis, investigations and engineering. You deployed all these systems and looked at alerts—you did everything.”

But as Cisco grew and acquired other companies and infrastructures, CSIRT’s structure had to change. That’s when the decision was made to split CSIRT into three primary functions. At the same time, the security world has seen dramatic changes. New technologies, regulations and legislation shape the team’s day-to-day work. These days, a lot of malware comes out of the former USSR and from nation states that are interested in Cisco’s technology and the information we have on customers.

Rapid Onboarding of Acquisitions

A key achievement of CSIRT is a capability called C-Bridge monitoring, which reduces the on-ramp time for newly acquired companies to integrate with Cisco. In the past, it would take months for an acquired company to get onto Cisco’s networks. This would force new teams to switch back and forth between Cisco networks and their own infrastructure, connecting and disconnecting VPN each time.

C-Bridge is a portable security monitoring rack that is sent to the acquired company premises, where it provides a conduit that allows the new teams to connect to Cisco and their local infrastructure.

“They can start collaborating with their new teams and integrating into Cisco culture literally on day one,” says Christopher, operations team lead on the CSIRT engineering team.

Perhaps surprisingly, one of the very things employees like about Cisco is also one of the factors that’s most challenging for CSIRT. Compared to other companies, Cisco’s network is relatively open. This open network policy allows employees to administer their own systems, load up with software and be creative.

“Our employees can do a lot more with their systems than employees at most other companies,” Chris explains. “We don’t lock our network down like, say, a bank would.”

From a security perspective, that translates to additional exposure and risk from malware and compromised systems. Add to that the sheer size of Cisco and its network, plus the fact that it has some 2,700 labs, both internal and Internet facing.

“It keeps us busy,” says Chris.

CSIRT is spread across the world, but its members are team players willing to try new things and pull together when needed. They typically communicate in encrypted form using IRC (yes, Internet Relay Chat—from the early 90s). Chris says protecting Cisco from attack, or minimizing the impact of an attack, is very rewarding work. It’s one reason the team has a low attrition rate, with many longtime members graduating into leadership roles.

“We have a really strong team that loves to work together,” Chris says. “It’s one of the reasons I’ve stuck around for so long.”

 

Related Links:

• Infosec Playbook
• Security Monitoring
• Find jobs at Cisco
• Follow @WeAreCisco to see more about the Cisco Life!

 

Do you have a personal experiences story to tell? Share with the #WeAreCisco

---

Connect everything. Innovate everywhere. Benefit everyone.