Building a dynamic security infrastructure [in a box; with SDN]

Years ago I wrote about building a secure network in a box. Over a weekend I decided to revisit this concept thanks to a colleague at work wanting to do something similar. It got me thinking “a lot has changed since I last did this” and it felt like time to revisit it. Well, disappointment wasn’t in the cards because it’s easier, smarter, and more flexible now that it was back then.

DNS – the treasure trove of information your ISP can see

In recent years, the nature of privacy on the internet has become a very important topic amongst those concerned with the now lack of net neutrality. The de-facto mechanism for dealing with privacy has been to "SSL all the things", which I am very much in favor of. What many do not realize, though, is that simply using SSL for the traffic that transits a given ISP still leaves a wealth of thick, rich, delicious personal data still easily available to your ISP to harvest, sell, and do with as they please.

Faucet: Enterprise OpenFlow in production

Remember OpenFlow? It was the media and marketing darling for the better part of 5 years as “the machine” conflated OpenFlow with SDN and SDN with - almost literally - everything. “Still Does Nothing” was a common phrase uttered around those of us that had run large scale, complex networks for a long time. Quietly, and mostly, out of the fickle media and blogosphere eye, a scrappy little SDN project called faucet has been diligently plugging away– making easy to use, production quality, well documented, and very stable code that runs OpenFlow networks quite happily in production and at scale.

DDoS mitigation podcast, black hole routing

As an often-security-engineer and an individual that has been working on large networks for quite a while, dealing with DDoS, or the threat of DDoS is a well traveled path. Recently I was invited to discuss some of the basics of DDoS mitigation on the Network Collective Podcast. This was a really fun and insightful chat with a wealth of great information for engineers and operators of any skill level.   Ep38 - DDoS Mitigation from Network Collective on Vimeo.

Trouble with Tribbles …..errr NAT

As a follow up to my last post, I wanted to dive a little deeper into the world of address translation and to suss out some of the more compelling details. As I’ve said on many occasions, it pains me to see NAT referenced as a security mechanism. That said, where PNAT can be beneficial is in an overall privacy strategy, however, even that is comparatively low value and given the current state of global IPv4 allocations, arguably a detriment to usability - we’ll get to that - before we do, it is important to understand what ’NAT” as we call it today actually is, and to do that we need to explain all of the types of address translation (yes, there are several).

NAT discussions, fact checking, and debate

I’ve been very vocal about the misinterpretation of NAT for many, many years. Since it’s inception, NAT has been slowly perverted into what many now believe to be a security mechanism. While I do see a reasonable use of IP masquerading in a larger security strategy, this is not the original intent (or implementation) of NAT. What mosts network engineers call “NAT” is actually one to many network port address translation - or taking one public address and “hiding” a number of private (likely RFC1918) addresses “behind” it, using ports to translate traffic and keeping the state of those connections.

Strategy Series: What is your netflow strategy?

You have one, right? Even if your entire strategy is “collect some flow data”, there is absolutely NO reason not to have a netflow implementation, and frankly, it will save you time and money over time if you make the effort to do it. I love network data and analytics and I have waxed poetic about how important they are at every opportunity. There are a myriad of options for analytics and flow data.

No privacy may be the new privacy.

Taking politics and putting them aside, what the new administration has been attempting to change with regard to internet privacy is something we should all be informed about. Wether you have a tin foil hat or don’t care, “knowing is half the battle”. The other half is doing - which I will also lend some brief insight to (sorta). What’s changing? Nothing yet (as of the time of this writing). What will likely change?

Creating an internal span port inside proxmox OVS

In the last few years I have moved all of my virtualization to proxmox and docker. Seeing as I like to look at packets because I am a closet security guy, and being as I have been working off-and-on on a security project in recent times, I wanted to be able to span a port not only from a hardware switch, but also within my software switches. I had been using linux bridge, which I am not a fan of, so when I started down this path I did not look hard to find a way to do so under that platform.

BGP RPKI – why aren't we using it more?

I was recently at a meeting where BGP RPKI was the topic de jour. While this has been a topic that I have visited on occasion of the last few years and something I wanted to spend significant time on, I have found that setting aside the time has been difficult and sparse, much like the deployment of BGP RPKI. In order to better understand the options available, it's important to break down the pieces and terminology involved; BGP is daunting enough to those unfamiliar with it and adding PKI on top of that can be even more so.

OpenDNS acquires BGPMon and the future of route monitoring

For those that run BGP networks, BGPmon is often a tool they turn to for some really unique and hard to find information. Remember back in February 2008 when Pakistan Telecom "blocked" Youtube? That one was a really, really public example of something that BGPMon caught. BGPmon has been around for a long, long time. Quietly watching prefixes. Silently noting changes and reporting them to the ones lucky enough to know of its existence.

NIX4NetEng #5 – GeoIP tools

Sometimes in networking and security it becomes necessary to do lookups of location data on IP addresses and prefixes. On my Mac I use homebrew to manage packages, but most of these tools are available with thetypocal apt, yum and port package management systems. For this post, I’m going to shift gears and show the install on my mac: sliver:~ buraglio$ brew install geoip ==> Downloading https://downloads.sf.net/project/machomebrew/Bottles/geoip-1.6.3.mavericks.bottle.tar.gz ######################################################################## 100.0% ==> Pouring geoip-1.

NIX4Neteng #4: POODLE and SSLv3, scanning and updating

With the recent release of the POODLE SSLv3 vulnerability, folks are scrambling around trying to figure out what runs what and where. Running a handful of things that do SSL, I was obligated, both personally and professionally, to figure out an easy way to drill down and figure out what does what and then fix the vulnerable services. When there are a lot of devices, this can seem like a daunting task, and it is if you’re trying to do it manually.

sonrancid: A [very] basic RANCID module for sonicwall

I know, I know, I’m always saying that you don’t need a firewall. That’s mostly to get your attention to push my agenda of sane security architecture, I do actually believe that firewalls are appropriate in a great many use cases and I’ve managed them big and small ranging from Juniper SRX 5800 clusters to tiny purpose built BSD distros on custom hardware. I even managed Checkpoint and gauntlet firewall back in the 1990s.

Why you don't need a firewall [how to secure an open perimeter network]

I admit that the title was meant to be inflammatory. However, there are use cases that aren’t terribly uncommon where an in-line security appliance is just not the correct tool for the job. Someone once told me “a firewall protects a network like a fuse protects an electrical circuit”, and it’s mostly a correct statement. Firewall vendors will probably argue this and enterprise folks may discount this as heresy and call for burning me at the stake.

NIX4NetEng #2 IPv4/6 address investigation tools – whois + dig

I don’t care what your vendor alignment of choice is, Cisco, Juniper, Brocade, Alcatel….it doesn’t matter. At one point or another you’re going to need to bird dog an address to see where it’s coming from, who owns it, what it’s DNS name is or what path you’re taking to get to it. We’ve already talked about BGP tools, they’re a great choice for checking routes across the internet. Hunting down addresses is an interesting one, though, as address management and lookups can bleed into other aspects of networking like path selection, latency, jitter and many other things.

Install nfsen and nfdump on CentOS 6.5 for netflow and or sflow collection

I am an absolutely huge fan of statistical and instrumentation data, especially when it comes to traffic analysis, visualization and baselining. I’ve rambled on about the importance of it at every opportunity. As a result of that, I have been doing work with netflow and netflow-like data for a fairly long time. My first collector was the OSU Flow tools based stuff back around 13 years ago. From there I played with all kinds of netflow tools, both commercial and open source, finally settling most of my focus on nfdump and nfsen.

Speculation and soapboxing about the leaked NSA spy catalog.

The buzz as of late around the security and networking communities has been about the NSA and their catalog or spy toolkit. I’ve spent time in my career thinking about and doing infosec and I did a brief stint working for the FBI in a project called NCDIR. I like to think that I can provide at least a peripherally competent commentary about it [take it with a grain of salt].

Sonicwall revisited – Now with IPv6 – and way too much licensing cruft.

About a year ago I did a brief review of the “new Sonicwall”, specifically a smaller branch office device that was said should have had all of the features of the larger devices. I proposed that it had some significant limitations (much to the disagreement of a great deal of folks). However, I stand by my statements. If you ignore the fact that firewalls often cause more problems than they solve, that NAT is a nightmarish kludge (and not a security mechanism), andwill likely be phased out for better options eventually, the SonicOS I tested was pretty limited as far as what I believe should be features.

Inline-jflow on MX series Juniper

One of the things that I’ve always lamented about using non-Cisco hardware is the lack of true 1:1 netflow support. Say what you will about jflow, cflow, sflow….there is no substitute for netflow, with sflow being the exception to that since it is a protocol that inherently supports ipv6 and can transport far more than simple network information if configured in certain ways on certain devices. On newer MX series Juniper routers the game has changed.

Building a secured network in a box

In many environments, the move to virtualization is a path well traveled. My home and lab networks are no exception to this and I’m sure nearly everyone who reads these pages has at least been exposed to it in one way or another. I have played with nearly all of the virtualization platforms and am firmly in the camp that there will be a large segment of networking that will move to a virtualized platform especially in the data center and campus segments.

Why hardware firewalls are the walking dead

OK, maybe they’re not totally dead, but they’re being demoted. To the mail room. During the course of my career I’ve always had at least some responsibility for firewall and security devices. In those ~15 years, how these boxes are built and function has shifted. From the perspective of my career, there were IOS ACLs (yes, I know, not a firewall), there was the IOS firewall versions and there were software packages such as gauntlet, checkpoint.

IPv6 Toolkit 1.3 fun – scan6

Recently SI6 released the IPv6 Toolkit 1.3  This release is on the heels of this IETF draft  on IPv6 host scanning. It was long thought that scanning an IPv6 network was impossible. The address space was too large and reliably ascertaining the hosts from it would be too time consuming to even attempt. However, as Dr. Hans Zarkov says in the 1980 classic cult film of my youth, Flash Gordon, “You can’t beat the human spirit!

Watch out, Gigamon (and others), Arista is bringing their A game

It’s no secret that I’m a fan of the model Arista Networks is using to make gear and provide innovative services and products. In my opinion, they’re changing the landscape of campus and data center networking gear. I’m always a fan of the little guy trying to change the world and this falls under that category. For those that don’t know, Arista Networks is a “hardware” networking company that is using merchant silicon wrapped in their custom linux based operating system (which is very much like IOS).

Identify and remedy problem IKE and eventd processes on Juniper SRX

Recently we encountered a very strange behavior on an SRX 5800 cluster. The cluster, which is in active/active mode, started dropping OSPF adjacencies to it’s neighboring routing equipment, in this case, Juniper MX480 and Brocade/Foundry MLX8. Strange behavior indeed, since for us, these had been rock solid for around 2 years and we’d never seen this odd behavior before. Honestly, we started looking at the routers first since this was something the SRX has never done before.

Sonicwall – Old dog learns [some] new tricks

~12 years ago I had a drinking buddy that worked with me at the regional ISP. We had a lot in common, he had been an icon back in the didjits era of punk rock in Champaign Urbana and we had briefly been in a terrible band together. He introduced me to a dude that to this day I just knew as “Ravi Sonicwall”. He had apparently been recruited from the U of I, written a lot of the low level pieces of the original sonicwall and retired to enjoy life and buy beers (he actually scolded me at a bar for buying him a beer saying “when I’m in town, I buy the beers”).

Using a Brocade MLXe as a replicator to an IDS

Have you ever needed to replicate a lot of data transparently to an IDS without the use of a rack of optical taps? Not enough budget for a Gigamon or cPacket? Have a spare MLXe laying around? you’re in luck, we were in that boat too. Let me first preface this by saying that this would be fairly trivial using OpenFlow / SDN. That being said, we didn’t have the time to set that up, so this is what we came with.

ftps (secure ftp) from the command line

I have recently enabledDuo Security for many of my personal services, and I can’t recommend them enough. Personal two factor authentication is very useful and really powerful. It works on my iPhone and I have yet to run into any real issues….except for one. I can’t use automation to scp or sftp anything anymore and keep my two factor auth working in a way I’m comfortable with. Enter ftps. FTP is a terrible, yet immensely useful protocol.

Black Hole routing

It’s no secret or ground breaking area to do black hole routing. ISPs and NSPs have been doing it forever to allow for a very low cost, very scriptable and very effective way to wholesale block a layer3 address. However, it can seem like a bit of a black box to anyone who has never done it. I recently did some work spinning this up in a good sized network that it didn’t currently exist, and remembered how monumentally useful (and simple) it actually is.

Upgrading a new SRX 3600

We are putting a few new SRX 3600 clusters into production soon, and we’ve had them for about 6 months in boxes. This presented a fairly significant issue, one that I didn’t think about until it smacked me in the face. The code on these boxes was old. Very old. JunOS 9.2 old. No problem, lets just upgrade them to 10.4R something. Wrong. the code that shipped on these boxes was so old, and we waited so long to upgrade them that I was unable to upgrade them straight to anything modern.

Moving JunOS code between SRX cluster nodes

Regardless of the fact that there is now a good ISSU-like service for the SRX (named Low-Impact Cluster Upgrade; LICU for short), if you’re upgrading your Active/Active cluster from something that isn’t 10.4, or if you just aren’t comfortable with how baked LICU actually is, you’ll need to know how to move the junos code around. This is easy if you have physical access to both nodes, but for those that have.

Enabling IPv6 on Juniper SRX 5800 cluster

After enabling the IPv6 Flow based processing, we decided to get rolling with making our IPv6 path congruent with everything else (IPv4 unicast and multicast). With all of the other things we had going on, we thought this would be a low hanging fruit that would be easily plucked from the routing tree. Well, a minor oversight on our part caught us by surprise. According to this handy dandy matrix for JunOS 10.

Multicast through Juniper SRX 5800

We’ve been working toward a more simplified model for our network path, and in doing so, we desired a congruent path for IPv6, IPv4 Multicast and IPv4 Unicast. However, this is actually pretty hard when dealing with the link speeds, amounts of traffic and flows that we do, in conjunction with Firewall…..and IDP/IPS… Lots of research, reading and testing was done. Juniper SRX series has full support for 90% of this, with IPv6 IDP coming in Q2 of 2011.

SRX IPv6 flow based processing

One of our plans is to consolidate as many of the egress trafic paths as possible. To facilitate this, we had to do some things like buy carrier grade equipment. Enter the SRX 5800. No one really does IPS/IDP+Firewall quite like the SRX. After extensive research and exhaustive hands on testing with quite a bit of equipment, that is what we settled on. Even the IBM “technical evangelist” guy that came to talk to us said “No one really does it like they do” when referring to Juniper and 10G firewall/IPS.

Updating SRX IDP signatures

IDP signatures need to be updated often. On the SRX platform, there is also the notion of a “detector”. This also meeds to be updated on a regular basis. it seems. Over the past few weeks, we’ve needed to update the IDP signatures and detector on our SRX 5800 cluster several times, and the results have normally been fine. Updating the IDP signatures has never been that big of a problem (see postings about updating stuff on cluster nodes).

Juniper SRX Cluster

I have had the opportunity to work pretty extensively on the Juniper SRX firewall/IDS platform over the last few months. In doing so, I’ve found many “gotchas” the hard way. Here are a few that I’ve found so far: Clustering is a beast in and of itself. I think it needs a bit more polishing, but it could be that we just need to refine our design. On the SRX 650 it works, but you must be on the right code version (I got it to work under 9.