Originally published Oct. 31, 2017

Lesley McClurg is trudging through the KQED newsroom, oozing distress.

McClurg, who in the course of her career as a radio reporter has conducted interviews amid raging fires and torrential floods, has finally met her match: an office printer. She has just spent an entire hour trying to get her radio script to print, a task that took the cumulative analytical skills of three nearby co-workers plus a guy from IT.

“And now I need to go on the radio and sound like I’m calm and relaxed and NPR-friendly,” McClurg tells me. “But, really, I need to scream.”

And so went Day 33 of the Great KQED Ransomware Attack.

This summer, criminals somewhere in Russia, Ukraine or maybe down the block, hacked into KQED’s computer system, installed malicious code that encrypted the station’s files, software and servers, and demanded money for their safe return. The perpetrators, as is routine in ransomware attacks, demanded payment in the form of the cryptocurrency bitcoin—if the victims ever wanted to see their data alive again. The message, rife with grammatical and spelling errors (cold comfort to no one but the copy editors), included directions on how to recover the data in “3 Easy Step” (sic), with step one being the transfer of funds.

The chaos commenced on June 15, as KQED’s employees began suddenly reporting computer crashes en masse. The IT department at the San Francisco-based NPR and PBS station decided to play it safe—they shut down the entire network to stop the malware’s spread.

The results were difficult to face: Phones went dead; the internet vanished; reporters’ audio interviews disappeared. And it meant the only way the head of IT, John Reilly, could get a message out to each of the 503 full- and part-time employees to keep their hands off their computers was to post handwritten signs at every entrance and on every desk.

KQED, one of the largest public media companies in America, has been hobbling its way to recovery ever since. What was it like? Think of a really boring episode of “The Twilight Zone”—or better yet, “Black Mirror.” Or getting stuck in an absurdist satire about human dependence on technology. Everyone had their particular breaking point, triggered by one Rube Goldbergian workaround or another.

One afternoon, a KQED marketing employee attempted to send out a meeting agenda in a single email to some 60 people. Our temporary email system wouldn’t allow grouping of multiple addresses into one alias, so each destination had to be entered one at a time. Any tiny problem with the syntax—an extra period here, an omitted semicolon there—grounded the message until you could figure it out. Four hours, that job took.

It’s a Problem

Ransomware—it’s a growth industry. Last week, a new strain called "Bad Rabbit" hit computer systems in Russia, Eastern Europe and beyond. This year, a posse of federal agencies, including the Justice Department and the CIA, issued a report describing ransomware as the “fastest growing malware threat, targeting users of all types—from the home user to the corporate network.”

And in another study, anti-virus software maker Symantec estimated 463,000 ransomware attacks worldwide in 2016, a 36 percent increase from the previous year.

Entrepreneurial hackers also have innovated “ransomware as a service,” in which they peddle kits to criminals who are less technically adept.

“It’s not some teenager sitting in his garage,” USA Today tech reporter Elizabeth Weise told KQED. “These are effectively organized crime; they have offices, they have office workers who come in.”

Over the last two years, researchers from New York University and Google tracked about $25 million in victim payments. Not bad, but not the global drug trade either. Estimates of monetary gain for perpetrators of the WannaCry ransomware scourge, a worldwide attack last May on computers running the Microsoft Windows operating system, were remarkably low, even though the virus disabled more than 200,000 computers, including those of FedEx and England’s National Health Service.

That does not, however, take into account the cost in lost sales and business interruptions, which can be considerably higher. The U.K. company Reckitt Benckiser, maker of Clearasil, Lysol and Airborne, estimated a $117 million loss resulting from the Petya ransomware attack in June.

In KQED’s case, the attackers demanded 1.7 bitcoins per computer (roughly $2,500 at the exchange rate of the time) or—act now!—just $27,000 for the lot.

KQED briefly considered paying up, said Chief Technology Officer Dan Mansergh, but abandoned the idea after the FBI laid out the risks. They were: being seen as an easy mark; inclusion of code in the decryption key that would set the stage for future intrusions; and failure of the perps to keep their part of the bargain to set your data free.

KQED has yet to fully calculate the attack’s cost, but much of it will be in IT. In fiscal year 2018, the company is allocating $475,000 for new network protections. Those funds will pay for four new IT employees, plus more sophisticated anti-virus software.

As for the cost in employee productivity and stress? A lot.

‘Worst Stress of My Life’

The attack took place on a Thursday. The next day, news reporters and editors at KQED, the most-listened-to radio station in the Bay Area, could access the internet only through personal hot spots on their cellphones. Reporters read stories into their recording decks to get on the air and didn’t use sound bites; no one crumpled cellophane to emulate the sound of a fire, but it was still old school.

The station's weekly current affairs television program, KQED Newsroom, had to abandon its original production plans and run a previously aired special show. And when the television studios were still out the following week, the staff took production out of the building, to a makeshift studio at UC Hastings College of the Law and another at the Electronic Frontier Foundation for a segment on ransomware.

Meanwhile, IT collected and wiped clean every computer that the malware, called Samas, had so much as breathed on. But that was nowhere near the end of it. To stop the infection from spreading, IT had taken the network down. No phones, one rickety internet router, no access to documents or audio, and the only way to print was to walk your laptop to a printer and plug in the cable. And after the malware was eradicated, the network did not come back up.

“We could not just put it back the way it was, because there were all kinds of security holes,” said Michael Kadel, KQED’s network systems engineer.

So the staff went to work creating temporary systems to keep things functioning until a more secure infrastructure could be built. They worked long hours, long being a euphemism for 12-, 16- and 20-hour shifts at a time.

“Worst stress of my life,” said IT director Reilly.

Kadel agreed. “A solid two months of just terrible, terrible stress.”

Kadel said he’d once suffered through cancer. “This was way worse.”

Those views are not necessarily unheard of in these types of situations, said Jonah Silas Sheridan, a Bay Area computer security consultant for nonprofits.

“It’s a big deal for security practitioners,” Sheridan said. “How can you secure someone’s systems when you are in a place of sleeplessness and arrhythmia yourself?”

Meanwhile, the radio news team and their engineers went full-on MacGyver to rig a system that kept the news on the air. In normal times, reporters and editors were fully networked, with universal real-time access to every story, revision and audio clip. Now, every story had to pass through three or four computers. Audio had to be uploaded to one site, then downloaded somewhere else. Cellphones and the instant messaging app Slack were conscripted into the workflow. The motto: Whatever works.

As weeks turned into months, the newsfolk reported on the congressional health care debate, the move to end DACA, the total solar eclipse and the eruption of far-right rallies in Charlottesville and other cities, all using a system that took three to four times the usual amount of time.

The audience most likely didn’t notice. But the staff found the relentless workarounds, jury-rigging and duct-taping exhausting.

“Initially, we all had this attitude, ‘The show must go on,’ ” said Mina Kim, the afternoon drive-time news anchor. “After that surge of adrenaline and can-do attitude subsides, you’re just exhausted. It’s workaround fatigue.”

I asked John Reilly, who’s done a lot of consulting in his career, if he’d ever seen an organization experience the level of disruption KQED had. “No, not through an attack,” he said.

How could it have happened? And how much at risk are other companies?

Convenience vs. Security

Security experts like to recite an aphorism to describe the basic dilemma of their profession.

The safest computer, they say, is one sitting in a metal box in a locked room, switched off and unconnected to anything. Such a computer, goeth the Parable of the Completely Safe Workstation, is impenetrable.

And utterly useless.

“You do have to make trade-offs of usability and convenience for security,” said Paul Guthrie, a vice president for PSC, the security firm hired by KQED after the attack. “And sometimes the pendulum swings too far.”

Guthrie is one of the tech and security people both inside and outside KQED who agree the company’s pendulum over the years had strayed quite a way over into convenience territory.

Kadel and Reilly, who had been the head of IT for a year and a half when the attack hit, were frank about the vulnerabilities the ransomware exposed.

“Honestly, I’m surprised this didn’t happen long ago,” Reilly said.

Kadel put it this way: The culture of KQED, a multifaceted media organization that produces a huge amount of radio, TV, educational materials and websites, tilted toward easier collaboration. If someone wanted to put the video from a news interview into the video editing system, for instance, and then share the audio with radio reporters, they could. And they could do that from the same computer they used for daily email.

But security people know this type of one-big-network-for-all is risky. Their job, after all, is to keep malicious hackers from turning the place into “The Day the Earth Stood Still.” And when systems connect to each other like they could at KQED, malicious actors can more easily infect the entire infrastructure.

Another problem: The station’s IT department allowed employees to download and install free software from the web. IT folks call this the “granting of local admin rights.” Many in the industry also call it “A Catastrophe Waiting to Happen.” Free software is often laden with viruses, worms and other nasty extras.

Kadel said one reason IT gave the staff free rein was due to a lack of IT employees to oversee the process.

“You can’t visit 600 people to do a Flash update,” he said. “You have to let people do their own Flash update.”

Lots of consultants say companies routinely shortchange IT. But Kadel may have a point. At the time of the attack, KQED employed nine IT employees to service roughly 503 workers. That means IT represented less than 2 percent of the organization’s workforce.

A 2016 survey of media and entertainment companies by the research firm Gartner found that IT workers made up an average of 6.5 percent of employees, more than three times the level at KQED. Across all industries, the number is 5 percent.

Finally, KQED used standard anti-virus programs to protect its system. But hackers know how to get around these by altering their code. In a test run by KQED’s security consultant, just 10 out of 60 off-the-shelf anti-virus programs recognized the variant of Samas that hit the company.

“If somebody wishes to put out a new variant, it takes a little bit for the anti-virus vendors to catch up,” said Guthrie.

KQED is now plugging all these holes, redesigning its network and all of its media systems. It’s also deploying much more sophisticated anti-virus software that looks for processes that are outside the normal behavior, shutting them down. Among the new IT employees coming on board: a full-time security pro.

What Kept Working

When everything went down, the safest place for data turned out to be the cloud. KQED lost no member or donor information, because it wasn’t stored in the network. And as the attack short-circuited system after system, KQED’s network of websites kept on keeping on. How so?

“We’ve moved most of KQED.org to be hosted in the cloud, so it’s sitting on [Amazon] servers,” said Tim Olson, KQED’s chief digital officer. “It’s more disaster recovery, more efficient for everybody.”

The cloud is “certainly a strategy that a lot of smaller organizations employ, because it’s a lot of work to keep your systems patched,” said IT consultant Sheridan. And big cloud services like those of Amazon, Microsoft and Google spend tons of money to pay for cybersecurity. “That being said, you’re now in a very deep trust relationship with someone else, Sheridan said.

But in terms of maintaining access to data after the attack, the cloud was the silver lining. When everything went down and all of KQED’s storage systems became inaccessible, there sat my audio interviews, my budget numbers, my everything, safe and snug on Google Drive.

Talking About It

Below the hard truths about security vs. convenience now adorning whiteboards at KQED are a few about survival, too.

“We've learned a lot,” said Reilly. “People have realized that we weren't prepared in a lot of ways. But then again, in some ways I view this as a dress rehearsal. Because we’re in an earthquake zone.”

What’s being done to fight back against the spreading threat of ransomware across the world?

Companies and individuals dealing with a ransomware attack may find help at NoMoreRansom.org, an initiative of the Netherlands’ police, Europol and the computer security companies Kaspersky Lab and McAfee. The website includes decryption tools for dozens of known ransomware infections, as well as advice on how to prevent an attack in the first place. (Update Nov. 15: The U.S. Dept. of Homeland Security has banned federal agencies from using Kaspersky products due to alleged ties to the Russian government and intelligence services.)

One thing every expert recommends: Back up like crazy.

“The number one thing to do is back up your data,” said Alexander Garcia-Tobar, co-founder of ValiMail, a San Francisco company that authenticates email in order to prevent intrusions like ransomware.

“If you’re backing up on a daily basis or even on a weekly basis, you back up to before the known infection and you cross your fingers.”

Guthrie said high-end tools designed to keep things running in the event of a catastrophe can be a lifesaver. But they are expensive.

“There’s network backup and restore systems that are very good and very fast and very expensive,” said Guthrie. “Smaller companies tend to do what they can with the resources they’re given.”

KQED decided to go public with our tale as a kind of public service, so that other organizations can learn not only from our mistakes but also from the way we kept functioning with minimal disruption to the public, despite maximum disruption internally.

Sheridan said opening up can only help.

“You might be surprised at how many different kinds of breaches, how many kinds of incidents, have happened in other organizations, and continue to happen,” he said. “We would be well-advised as a community to figure out a way to talk about these things without shame. We need to learn from them collectively.”