May. 9, 2023
I will preface this with that I always say: do not implement this without IPv6 unless you literally have no other choice. IPv6 will allow for a significant resource offload because most eyeball services (Netflix, Youtube, Google, Facebook, etc.) will prefer IPv6, thus removing your requirement for more IPv4 NAT state and overload / port utilization. Because I found no simple how-to for using NETMAP for CGN on a Mikrotik, here is one.
Aug. 3, 2020
It is all too common that smaller shops do not have the resources for a proper test lab. Even with the cost of grey market hardware, and the ease of virtualization, the gap is definitely there - be it time, financial limits, manpower, or even a general malaise toward even asking for something which may get denied. This presents a problem for multiple reasons:
Changes are not staged in a safe environment first - i.
May. 4, 2020
One of the most common questions I hear from small and even medium sized ISPs is “why should I run my own DNS resolver(s)?” The perception that DNS is hard, complicated, or even unnecessary is often cited as a reason to just farm it out to one of the “free” anycast resolver services available across the internet. Now, there are many reasons to be wary of DNS, both from the professional and the consumer side - it is a huge treasure trove of personal information about behavior, and is easily monitized by entities large enough to consume and process it.
Sep. 8, 2019
Flow data is a critical piece of understanding how your network works what what it is actively doing. It also provides a great baseline and capacity planning tool. However, some of the more feature rich NetFlow and/or sFlow collectors can be quite daunting in their cost and/or complexity to install. ElastiFlow is a great alternative for flow analytics and is built on the well traveled and robust ElasticStack, meaning, its back end is well documented, well supported, and scales exceptionally well.
Jul. 29, 2019
Small to medium ISPs are an interesting phenomenon. Early in my career I was pretty heavily involved in that space, so much of my current thought processes and methodologies are heavily informed by that experience. Something that never ceases to amaze me today is that the practice of scripting and “automating” things seems to have become somewhat of a lost art, or at the very least it is not part of an initial deployment plan.
Apr. 29, 2019
There is no shortage of network telemetry data that can be collected, recorded, graphed, and stored for cross reference and triage. Not one to be underestimated, latency at a can be incredibly powerful when leveraged for baseline and deviation notification. As I have eluded to in the past, there are many tools in this space.I have written about a few of them in detail and touched on others in passing. Regardless of the tool, the data is powerful and the instrumentation they provide will only serve to make your network more robust and easier to work on.
Dec. 10, 2018
In recent years, the nature of privacy on the internet has become a very important topic amongst those concerned with the now lack of net neutrality. The de-facto mechanism for dealing with privacy has been to "SSL all the things", which I am very much in favor of. What many do not realize, though, is that simply using SSL for the traffic that transits a given ISP still leaves a wealth of thick, rich, delicious personal data still easily available to your ISP to harvest, sell, and do with as they please.
Nov. 5, 2018
Remember OpenFlow? It was the media and marketing darling for the better part of 5 years as “the machine” conflated OpenFlow with SDN and SDN with - almost literally - everything. “Still Does Nothing” was a common phrase uttered around those of us that had run large scale, complex networks for a long time. Quietly, and mostly, out of the fickle media and blogosphere eye, a scrappy little SDN project called faucet has been diligently plugging away– making easy to use, production quality, well documented, and very stable code that runs OpenFlow networks quite happily in production and at scale.
Dec. 18, 2017
You have one, right? Even if your entire strategy is “collect some flow data”, there is absolutely NO reason not to have a netflow implementation, and frankly, it will save you time and money over time if you make the effort to do it. I love network data and analytics and I have waxed poetic about how important they are at every opportunity. There are a myriad of options for analytics and flow data.
Mar. 20, 2017
In the last few years I have moved all of my virtualization to proxmox and docker. Seeing as I like to look at packets because I am a closet security guy, and being as I have been working off-and-on on a security project in recent times, I wanted to be able to span a port not only from a hardware switch, but also within my software switches. I had been using linux bridge, which I am not a fan of, so when I started down this path I did not look hard to find a way to do so under that platform.
Jun. 20, 2015
I recently had a need to test OpenFlow on the brocade ICX 7450 for a fairly good sized, high visibility project. The basic goal is pretty simple, Layer2 path provisioning. Straightforward and fairly well supported in OpenFlow, even from the early days. To do this, the idea was to use a turnkey platform, that way there is one throat to choke if there are issues. I landed on the Brocade Vyatta controller (which is essentially ODL), and the ICX.
Jan. 19, 2015
VMWare is a powerful tool, and monitoring is a critical service. How does one monitor such an integral piece of infrastructure, and what do they monitor it with? There are powerful commercial ways of monitoring VMware, however, for those with existing SNMP based systems in place, specifically cacti, there are options. To that end, I'll set aside my strong distaste for SNMP [yet again], because those are for a larger, less useful series of posts.
Oct. 15, 2014
With the recent release of the POODLE SSLv3 vulnerability, folks are scrambling around trying to figure out what runs what and where. Running a handful of things that do SSL, I was obligated, both personally and professionally, to figure out an easy way to drill down and figure out what does what and then fix the vulnerable services. When there are a lot of devices, this can seem like a daunting task, and it is if you’re trying to do it manually.
Sep. 15, 2014
I know, I know, I’m always saying that you don’t need a firewall. That’s mostly to get your attention to push my agenda of sane security architecture, I do actually believe that firewalls are appropriate in a great many use cases and I’ve managed them big and small ranging from Juniper SRX 5800 clusters to tiny purpose built BSD distros on custom hardware. I even managed Checkpoint and gauntlet firewall back in the 1990s.
Jul. 26, 2014
IP addressing and subnetting is a common interview subject. I assert that memorizing these things is useful for learning the concepts but ultimately futile in that it is time consuming and inefficient use of engineering time when tools can be utilized to accomplish the same goals in less time with fewer errors. Honestly, I gave up doing this kind of work manually around 10 years ago and have never regretted it, and in actuality, I’d probably struggle to do it at this point because it’s a repetitive process better suited by code.
Apr. 30, 2014
Many network engineers are also tasked with maintaining systems that provide network services, those things that make the network easier to use such as DNS and DHCP or management systems that perform useful things like monitor the network, collect flow data or bestow access to the equipment by acting as bastion or jump hosts. In many instances, robust and high availability services run on UNIX, Linux or BSD systems for stability and reliability, so those that manage these systems need to be well versed system admins as well as whatever their other job functions are.
Mar. 20, 2014
Time to rewind from the new and shiny and get back to roots of networking. BGP is one of those odd protocols that is foundational to the functioning of the internet but yet somewhat hard to get experience with. Say what you will about this venerable protocol, it’s been here a while and it is not going anywhere any time soon. I’ve been doing BGP since around late 1999, and I completely fell into it by accident, having only the Cisco Internet Routing Architectures book (which I literally read cover to cover) and theUlysses Black Routing Protocols Book and whatever I could find on a random search engine to guide me, and that was only after having to learn on the CLI for the first 6-7 months.
Mar. 10, 2014
I recently had the displeasure of dealing with a series of failed disks in my newly created ZFS based NAS. I had cobbled together roughly 12TB of disk space and jammed them into an old PC, stretching the limits of the platform when I decided to go with ZFS. I broke all of the rules, underpowered, single core PC, only a handful of GIG of non-ECC RAM, etc. I’m sure storage guys are having a coronary after reading that, but it works for me and has minimal issues since I just relatively redundant need bulk storage and it doesn’t need to be fast (the ethernet connection is only 100M).
Jan. 11, 2014
I am an absolutely huge fan of statistical and instrumentation data, especially when it comes to traffic analysis, visualization and baselining. I’ve rambled on about the importance of it at every opportunity. As a result of that, I have been doing work with netflow and netflow-like data for a fairly long time. My first collector was the OSU Flow tools based stuff back around 13 years ago. From there I played with all kinds of netflow tools, both commercial and open source, finally settling most of my focus on nfdump and nfsen.
Nov. 29, 2013
As part of a larger fun project I’m working on (OVS for the ALIX platform; more to come on that once I have it 100% working), I have been playing a lot with OVS. It’s a great platform, andas others have mentioned, it’s as close to an SDN reference data plane implementation as we have. I’d be surprised if many if not all commercial implementations of OpenFlow aren’t based on OVS.
Sep. 1, 2013
One of the things that I’ve always lamented about using non-Cisco hardware is the lack of true 1:1 netflow support. Say what you will about jflow, cflow, sflow….there is no substitute for netflow, with sflow being the exception to that since it is a protocol that inherently supports ipv6 and can transport far more than simple network information if configured in certain ways on certain devices. On newer MX series Juniper routers the game has changed.
Jul. 5, 2013
I had the need to build a FlowVisor instance under CentOS. Since nearly all of the docs I could find were for debian, I threw this together. I utilized this GENI doc and the github docs as a simple reference. This is the quick and dirty method I used: Install the prerequisites: sudo yum -y install ant eclipse java-1.6.0-openjdk.x86_64 git sudo yum -y groupinstall “Development Tools” Create my standard directories: mkdir /services cd /services git clone git://github.
Jun. 30, 2013
One of my biggest complaints about VMware is that it is an enterprise application. It has historically catered to the masses, which I completely understand, but those of us that aren’t a fortune 500 company are figuratively and operationally shoved into a corner and forced to find hackish ways of doing things to work around the enterprise nature. One really, really good example of this is OS dependency. I hated architecture dependencies back in the old days (x86, SPARC, PPC) and I absolutely despise things that are OS platform dependent now.
Jun. 22, 2013
As much as I like to think I automate everything, I’m pretty bad at writing code to make my life easier since it tends to take me longer to write the code and it tends to make be a bit grumpy (this is eomthing I’m fixing by learning as much code dev as I can during my limited spare time). However, I like to think I can be fairly smart about working around my limited programming skills (think boba fett rather than jedi) by using the tools available to common folk.
Jun. 16, 2013
I recently had the need to debug a run away ip_rx process on an older Brocade MLX. For anyone that has had to do any type of low level debugging on the Brocade (Foundry) platform, you know that there many somewhat deep level diagnostics that are possible. The debug (like cisco debug) is a bit lacking, but the dm, LP and MP commands are very useful (and a tad scary).
Jun. 7, 2013
I’ve been doing a lot of MPLS in the last 45 or so days (which is one of the reasons I have been absentee in the OpenFlow world lately). Having had almost no real world MPLS experience aside from a handful of pseudo-wires and a very small LDP signaled network, I had to spend some time reading, hacking at routers and essentially learning. In doing so, I found a few things.
Apr. 27, 2013
Let me preface this post by saying that I am absolutely not an enterprise IT or systems guy, take everything that I write here on out with that as a side dish. I’m also very, very cheap. That said, one of the things I really like about KVM is the ability to easily view the console of a guest system using free, non-windows software like VNC. However, much like everything in life, there are reasons to do one thing or another.
Mar. 9, 2013
I started working on Juniper equipment around 2002. At my employer, we had an M40 with the serial number 256. We did Layer3 only. I had no idea if the Juniper even did layer2. It certainly wasn’t a layer3 switch like a 6500 like I was used to. It was like a deliciously robust version of any Layer 3 router I’d worked on previously. Over the years Juniper has added a switching line utilizing their FreeBSD based OS, JunOS.
Mar. 2, 2013
I’ve recently run into a situation where there was no longer enough space in the FIB to handle both the full IPv4 global table and the full IPv6 global table. We prefer to run a default-free network within this particular SP network, but in this case, until a hardware refresh can happen, we’ll need to adjust that. Given what we knew about the size of both tables, it made more sense to take a default IPv6 route from one transit provider and filter the rest.
Mar. 1, 2013
I am a network engineer by profession, but with the proliferation of SDN and OpenFlow, I have had to spend a lot of time re-learning a lot of system admin skills that I’d shelved years ago. Now, I’ve been a virtualization user forever. From VMware (Fusion, ESX), VirtualBox, to Parallels, I’ve used them at least in testing if not in production environments. I’d not really spent any mentionable amount of time with XEN, qEMU or KVM, but some projects I was working on suggested it for the virtualization mechanism, so I figured I’d try to pick it up.
Jan. 31, 2013
Starting from a base CentOS system with nothing configured, and referencing the CentOS wiki, here is how I like to set up a headless virtualbox environment: Disable selinux. It’s overly cumbersome and is enabled by default in CentOS. I like to permanently disable it even though the default is permissive. I ride the edge, I know. vi /etc/selinux/config and change SELINUX=enabled to SELINUX=disabled Then reboot. Using the methodology I originally found found here, I like to install the epel repo using this method: cat <<EOM >/etc/yum.
Jan. 24, 2013
If you are running a network and aren’t using RANCID, you should give it a serious look. RANCID is a cross platform configuration management toolkit for backing up router configurations and certain environmental and hardware information into version control. It’s been around for as long as I can remember and supports nearly every platform I can think of, including a few modules that I cobbled together myself. There is are a few nice web based front ends for CVS and SVN, I prefer to use ViewVC because I have a lot of experience with it, however, there may be cases where a web server isn’t a good option, unavailable or just too much work.
Nov. 24, 2012
Recently, there was a thread over at Packet Pushers about what folks use for their daily workflow. I quickly realized that my setup is pretty simple (as I like it) and relied on a large amount of terminal based tools, which makes sense since I have been a UNIX (or UNIX based) OS user since my migration from the original MacOS back in the 1990s. Anyway, Since I wrote most of this up already, I thought I’d post it here:
Oct. 22, 2011
Recently I was poking around Mail.app, setting up my new machine. I like to keep redundant copies of everything, email being no exception. I have backups of all of my email dating back to 1998, for the most part. It has come in handy from time to time and I like it for reference reasons. It’s a small amount of actual data as far as space goes, and it’s easy to do.
Aug. 6, 2011
I've recently decided that even though I love the BSD style MacPorts system, it can be too clunky to maintain and doesn't handle dependancies as well as I'd like (much like the actual BSD ports collection). So, in doing a little looking I found that Fink is still out of date, but Homebrew is very simple and also really elegant comparatively speaking. Since homebrew doesn't wrk well with other packge systems installed, and I already I'd like to know what I had installed since this system has been in use for 2+ years, so I do a list and send it to a txt file: touch ~/Documents/installed.
Feb. 18, 2011
I have huge iPhoto and iTunes catalogs. This can present a problem for both loading the applications and for backup. I have learned to deal with the Application load times, but backups are very important to me.
I'd gone through the iPhoto backup process and restore more than once, and I didn't like the fact that I didn't have an offsite backup, so I paid for a flickr pro account ($24/yr, supports iPhoto export and RAW format).
Feb. 4, 2011
I was recently helping my brother-in-law out with the new Seagate FreeAgent GoFlex Desk 3 TB USB 3.0 External Hard Drivehe had purchased to do time machine backups on his mac. I personally have the 2t version and have been pretty happy with it, save for one small incident that I think was my fault that required some basic data recovery.
Since the drive comes in a file system that is not HFS+ Journaled, it needed to be reformatted to support time machine backups.
Jan. 7, 2011
Regardless of the fact that there is now a good ISSU-like service for the SRX (named Low-Impact Cluster Upgrade; LICU for short), if you’re upgrading your Active/Active cluster from something that isn’t 10.4, or if you just aren’t comfortable with how baked LICU actually is, you’ll need to know how to move the junos code around. This is easy if you have physical access to both nodes, but for those that have.
Dec. 29, 2010
I recently needed to upgrade a few MX480 routers and decided that it would be a good opportunity to get some experience with Juniper’s in service software upgrade.
I’d read a bit about it but I’d not had the chance to really use it. It’s pretty straightforward and it does what it claims. The following are my notes from rolling through this on my test lab MX480.
A few things are necessary to get going with ISSU, first and foremost, you need to have a box with two routing engines.
Sep. 6, 2010
I know this is documented elsewhere, but this was a pain for me, so I wanted to take some notes. I have several Snow Leopard (MAcOS 10.6) Macs and a Netgear DNS-323. I want to mount the drive using NFS and any good UNIX admin would. Unlike older versions of the Mac OS, NFS mounts are now handled under the Disk Utility application (which seems odd to me, but whatever). So, to make this work right I had to do the following:
Sep. 3, 2010
I knew a tool like this had to exist, but I had never needed to look in the past. While debugging a RA problem, I come upon the need to view IPv6 router advertisements. How can one do this? tcpdump? Yeah, I guess that could work. It’s almost like using a bulldozer when a wheelbarrow is all you need, though. I could use ndpmon, I suppose, but that, too seems like overkill.
Sep. 2, 2010
IDP signatures need to be updated often. On the SRX platform, there is also the notion of a “detector”. This also meeds to be updated on a regular basis. it seems. Over the past few weeks, we’ve needed to update the IDP signatures and detector on our SRX 5800 cluster several times, and the results have normally been fine. Updating the IDP signatures has never been that big of a problem (see postings about updating stuff on cluster nodes).