Podcasts

Last week I dashed off a very quick post about #CensusFail that went stupid viral. I think it was retweeted about 1200 times and it sort of became "the story" of what happened.

As far as I know the information I posted is accurate, but I wanted to write this to add a bit more context and look at where it's shaky. I literally wrote that thing up in about 10 minutes while I was working on last week's show. I was doubly under the pump because The Project had a camera guy coming to my house that evening to record an interview about the whole debacle.

I'd also just arrived back in Australia after spending six days in Las Vegas attending Black Hat, B-Sides and Defcon. Prior to that I was in Brazil. So yes, long story short, I was exhausted, jet lagged, slammed with work and I didn't really have much time to write a decent post. I certainly wasn't expecting what I did write to be spread so widely. So, now that I've had a minute to breathe, let's look back through the bullet points in original post to see where it's solid and where it isn't.

The information I put together came from multiple sources, some closer to the action that others.

• IBM and the ABS were offered DDoS prevention services from their upstream provider, NextGen Networks, and said they didn't need it.

I'm pretty firm on this one. They may have worked with their upstream provider on a contingency plan (geoblocking) but I've got pretty solid information that they opted not to have DDoS gear installed at the edge of the census network. That was a mistake. The edge gear can detect certain types of DDoS activity and send a signal to the upstream provider for its filtering/blocking to begin. If you don't have it, you're basically running naked if your geoblocking isn't effective. Oops.

• Their plan was to just ask NextGen to geoblock all traffic outside of Australia in the event of an attack.

Again, as far as I know this is solid and supported by statements made by officials since.

• This plan was activated when there was a small-scale attack against the census website.

As far as I know this is also solid. There was a DDoS attack targeting the Census website and they asked NextGen to block all non-Australia packets. This worked, for a time.

• Unfortunately another attack hit them from inside Australia. This was a straight up DNS reflection attack with a bit of ICMP thrown in for good measure. It filled up their firewall's state tables. Their solution was to reboot their firewall, which was operating in a pair.

This is the part I suspect *could* be wrong. Whether this attack actually happened or not I can't be sure. One source told me there was attack traffic hitting the Census website from within Australia, but the more I think of it the more I realise this could have just been legit traffic mischaracterised as DDoS traffic. That's the thing with stories like these. It's like reporting on a battle: The fog of war kicks in and details get lost or smudged.

I am very firm on the census website firewall being rebooted at some point and the secondary not being synced. I'm not 100% on whether this was because of Australia-based DDoS traffic hitting the census website or it was a result of straight-up shitty capacity planning. So was it an attack or their connection filling up? I can't be 100% sure. I doubt they are either.

• They hadn't synced the ruleset when they rebooted the firewall so the secondary was essentially operating as a very expensive paperweight. This resulted in a short outage.

Again, very solid on this having happened. Just not sure on the why.

• Some time later IBM's monitoring equipment spat out some alerts that were interpreted by the people receiving them as data exfiltration. Already jittery from the DDoS disaster and wonky firewalls, they became convinced they'd been owned and the DDoS attack was a distraction to draw their focus away from the exfil.

I am absolutely, 100% rock solid on this one. We even saw the relevant minister and senior bureaucrats support this one in statements made to the media. The bit they left out is the traffic that triggered the alarm was entirely normal and should never have resulted in a false positive.

• They pulled the pin and ASD was called in.

Public statements support this.

• The IBM alerts were false positives incorrectly characterising offshore-bound system information/logs as exfil.

This is the part that's most hilarious. I'm told it was bog-stock traffic behvaiour that set off the alerts. I am confident there was no valid reason behind those alerts triggering.

I'm actually pretty sympathetic here and it's hard to say the person who decided to unplug made the wrong call. If you suspect you've been owned and all your data is being siphoned off, it's probably the right thing to do.

It's the people who set up such shitty monitoring that are to blame for this part of the disaster, not the people who pulled the pin.

• ASD still needs to roll incident response before they can send the website live again. Even though it was false positives that triggered the investigation, there still needs to be an investigation.

This is just standard. Once you call an IR team they need to investigate.

So. That's where I stand on what I wrote last week. I'm sure about most of it, but the timeline and details around whether there was Australian attack traffic? I can't 100% substantiate that.

I'm highly confident the firewall thing happened. They did reboot without a synced secondary. But that's just sort of funny, and if it happened in isolation no one would think it's a big deal.

There's other stuff I haven't mentioned, too, like routes changing on the night to send traffic around the primary connectivity provider. This might be due to the "geoblocking falling over," something our fearless leaders have mentioned once or twice in interviews and at press conferences. If I had to guess, they tried to route around NextGen and get Telstra to pull together some last-minute DDoS filtering. That's just speculation, but if I had to guess, that's how it went down.

Either way it was amateur hour. The next question becomes: Who's responsible?

Predictably, the government is trying to shift blame for the debacle on to ABS bureaucrats and IBM. That's mostly fair enough. Telling a company like IBM that they should prepare for DDoS attacks is sort of like telling your babysitter not to put the kids in the oven while you're out for the night. It's just so weird that they didn't adequately prepare for it. That said, we don't know who made the final decision. It could have been an IBMer telling the ABS that they absolutely had it under control, or it could have been an executive-level public servant trying to shave a few bucks off the budget. We just don't know.

The thing I'd really like to know is why the ASD wasn't given authority to actually look at this set up before it went live. If its only involvement was asking high-level, compliance-like questions ("Do you have a DDoS mitigation plan? Y/N") then honestly that's not good enough. I suspect that's what's happened in this instance and this is where you'd go looking for ministerial accountability if you were so inclined.

If you're interested in infosec stuff beyond CensusFail, do check out my podcast, Risky Business. RSS feed here. iTunes subscription link here.

Or follow me on Twitter here.

I have been able to cobble together the following by talking to my sources. Sorry this post is so brief, but I'm still trying to get this week's show out and I'm massively under the pump. So here it is: Set your faces to stunned.

• IBM and the ABS were offered DDoS prevention services from their upstream provider, NextGen Networks, and said they didn't need it.
• Their plan was to just ask NextGen to geoblock all traffic outside of Australia in the event of an attack.
• This plan was activated when there was a small-scale attack against the census website.
• Unfortunately another attack hit them from inside Australia. This was a straight up DNS reflection attack with a bit of ICMP thrown in for good measure. It filled up their firewall's state tables. Their solution was to reboot their firewall, which was operating in a pair.
• They hadn't synced the ruleset when they rebooted the firewall so the secondary was essentially operating as a very expensive paperweight. This resulted in a short outage.
• Some time later IBM's monitoring equipment spat out some alerts that were interpreted by the people receiving them as data exfiltration. Already jittery from the DDoS disaster and wonky firewalls, they became convinced they'd been owned and the DDoS attack was a distraction to draw their focus away from the exfil.
• They pulled the pin and ASD was called in.
• The IBM alerts were false positives incorrectly characterising offshore-bound system information/logs as exfil.
• ASD still needs to roll incident response before they can send the website live again. Even though it was false positives that triggered the investigation, there still needs to be an investigation.

At least IBM got to bump their margins up a bit by not paying for the DDoS prevention though... amirite?!

On this week's show we talk about the week's security news with Adam Boileau and I spill on what my sources have told me about #censusfail.

This week's show is brought to you by Canary.tools. Canary is a fantastic bit of kit -- it's essentially an easily configurable, compact honeypot you can just drop on your network like a dropbox to detect attacks. No begging the data centre people for rack space, just drop it and go. We'll be talking to Canary.tools head honcho Haroon Meer this week about the disconnect between what some startups are pitching to venture capitalists versus what users actually need.

Oh, and do add Patrick and Adam on Twitter if that's your thing.

On this week's show we speak with Signal Sciences' co-founder Zane Lackey about hackers building defensive tools and software companies. Dan Guido and Andy Greenberg talk about car hacking and the week's security news, and Wade Woolwine of Rapid7 is in the sponsor slot talking about EDR/IDR software.

On this week's show we're taking a look at the DNC leaks, but don't worry, we won't be getting bogged down in the same old angles. Instead, we're going to chat to Lorenzo Franceschi-Bicchierai from VICE motherboard about his experience in interviewing the Guccifer 2 persona.

Then we'll hear from Kevin Poulsen about what these latest developments mean for Wikileaks. It's a topic you're probably sick of hearing about this week, but stick with us, we've got some new angles, and they're relevant.

This week's sponsor interview is an absolute, certified, 24-carat cracker. Bromium is this week's sponsor and its CTO and co-founder, Simon Crosby, pops along to talk about his experience in dealing with the wrath of Tavis Ormandy. Tavis actually managed to dig a custom build of Bromium's software out of VirusTotal and find a really cool bug in it. But there's actually a fair bit more to that story and Simon fills us in.

Adam Boileau, as usual, joins us to discuss the week's security news headlines.

Oh, and do add Patrick and Adam on Twitter if that's your thing.

On this week's show we're catching up with Brian Krebs of Krebs On Security. He'll be talking to us about recent trends in cybercrime, and he's got a warning for security teams in the banking sector. He says things are going to get pretty sticky, and he's usually right on this stuff.

This week's show is brought to you by Bugcrowd, big thanks to them. And in the sponsor slot we're speaking with HD Moore, who recently joined the company's advisory board. I know HD well and I can tell you he was initially quite sceptical of bounties. So he joins us to talk about why he changed his mind and how he plans on helping Bugcrowd do stuff better.

Adam Boileau, as usual, joins us to discuss the week's security news headlines.

Oh, and do add Patrick and Adam on Twitter if that's your thing.

On this week's show we're chatting with The Grugq about secure messaging. Facebook has announced it's rolling out an end-to-end encryption feature and the reaction to this wonderful announcement has been somewhat bizarre. We'll be talking to Grugq about why crypto absolutists are hating on companies that are rolling out non-default e2e features. We'll also talk about a couple of interesting case studies in which e2e encryption did absolutely nothing for the people using it.

This week's show is brought to you by Sensepost, an absolutely fantastic security firm that operates in England and South Africa. Sensepost has been an academy for security luminaries over the years. Haroon Meer of Thinkst was an early stage employee, Maltego creator Roelof Temmingh was a co-founder.

So, they're smart. And one of the things SensePost does is security training at BlackHat in Las Vegas. They've been doing this for 15 years and Sensepost's Daniel Cuthbert will be joining us in this week's sponsor interview to talk about what courses they're offering and who winds up actually taking them. The really interesting part is it's not always security professionals in those courses.

Adam Boileau, as usual, joins us to discuss the week's security news headlines.

Oh, and do add Patrick and Adam on Twitter if that's your thing.

In this week's feature interview we're chatting with Stanford's very own Jennifer Granick about a recent ruling in a Virginia court that appears to give the FBI permission to hack into any computer it wants, sans warrant. Well that's what the headlines are screaming, anyway. But as you'll hear, it's not quite that black and white.

This week's edition of the show is brought to you by Senetas, big thanks to them. We'll of course be hearing from Senetas founder and CTO Julian Fay later on in this week's sponsor segment. He's joining us to talk about the latest guidance from NIST with regard to moving towards quantum resistant encryption. You've heard Julian and I discuss why NIST thinks the industry should do this, now we're going to talk about the how.

Adam Boileau, as usual, joins us to discuss the week's security news headlines.

Oh, and do add Patrick and Adam on Twitter if that's your thing.

On this week's show we'll be catching up on the news of the last few weeks with Adam Boileau, then it's straight into the sponsor segment.

And we're really lucky this week to have Dan Guido joining us from the sponsor's chair. Dan is a semi regular feature guest on Risky Business. He is of course the head honcho over at Trail of Bits, a very interesting security problem solving organisation. He'll be along to talk about some developer tools they've just open sourced for iOS, to preview DARPA's Cyber Grand challenge final at DEFCON and to discuss an investment hack/secure has made into a company building serious host based protection agents out of osquery, the endpoint visibility tool created by Facebook.

One of Trail of Bits current gigs is actually developing osquery for Facebook, and Dan is pretty excited about it. Find out why after the news...

Oh, and do add Patrick and Adam on Twitter if that's your thing.

On this week's show we're chatting with Lauri Love. Lauri is an activist facing extradition to the United States where prosecutors hope to charge him with a raft of offences relating to attacks against US government computers... he'll tell us about what's going on with his extradition hearing and why he really, really doesn't want to go to the USA.

This week's show is sponsored by Tenable Network Security, big thanks to them. If you're looking for some vulnerability scanning and management software you really should talk to Tenable!

In this week's sponsor interview we're chatting with Chris Cleary. He's a director of business development at Tenable. His focus is on the US federal government, so we'll be chatting to him about the fallout from the OPM breach, one year on. There have been some significant changes to the way things are done, Chris says, but it's too soon to see if they'll pan out as intended.

Adam Boileau, as always, stops by to discuss the week's news headlines.

Oh, and do add Patrick and Adam on Twitter if that's your thing.