I am an absolutely huge fan of statistical and instrumentation data, especially when it comes to traffic analysis, visualization and baselining. I’ve rambled on about the importance of it at every opportunity. As a result of that, I have been doing work with netflow and netflow-like data for a fairly long time. My first collector was the OSU Flow tools based stuff back around 13 years ago. From there I played with all kinds of netflow tools, both commercial and open source, finally settling most of my focus on nfdump and nfsen.
A bit of history: nfdump was born out of a research network, requiring it to be able to consume huge amounts of flows efficiently. This makes it very powerful and very useful for nearly anyone, from the small technology tinkerer to to the enterprise network engineer up through the service provider architect. nfsen is really just a php wrapper for nfdump, however, the really nice thing about it (other then being free, opensource software) is that it is extendable via plugins. This is really what makes it a serious player from all angles. From botnet detection to displaying IP geo-data on a map, there is likely a plugin for it. Not finding what you are looking for? Write it! The architecture to use it is already there and documented.
For ease of install, I chose CentOS 6.5. Once you have a system up and running, to get nfsen/nfdump working, here is what you need to do:
yum install -y httpd php wget gcc make rrdtool-devel \ rrdtool-perl perl-MailTools perl-Socket6 flex byacc vi /etc/selinux/config set SELINUX=disabled reboot
You’ll need to make an iptables rule (I generally make one for ip6tables as well, in case I want to also enable ipv6 transport)
sudo iptables -I INPUT -p udp -m state --state NEW -m udp --dport 9995 -j ACCEPT sudo ipt6ables -I INPUT -p udp -m state --state NEW -m udp --dport 9995 -j ACCEPT
Also allow for access to the web server you just installed.
sudo ip6tables -I INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT sudo ip6tables -I INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT sudo iptables -I INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT sudo iptables -I INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT service iptables save service ip6tables save
Once you enable https you can safely remove the table rules for port 80.
Start HTTPd
sudo service httpd start
Enable HTTPd at boot
chkconfig httpd on
Now you need the actual code. I like to grab the latest from sourceforge. (nfdump-1.6.11 and nfsen-1.3.6p1 at the time of this writing)
wget http://downloads.sourceforge.net/project/nfdump/stable/nfdump-1.6.11/nfdump-1.6.11.tar.gz wget http://downloads.sourceforge.net/project/nfsen/stable/nfsen-1.3.6p1/nfsen-1.3.6p1.tar.gz
tar -zxvf nfdump-1.6.11.tar.gz ./configure --enable-nfprofile --enable-nftrack --enable-sflow make && sudo make install
By default 1.6.11 enables v9 and ipfix =)
adduser netflow vi /etc/group
Add user netflow to group apache
vi etc/nfsen.conf
change www user to apache
Add your host to the file to allow for collection, my %sources looks like this:
%sources = ( 'home' => { 'port' => '9995', 'col' => '#0000ff', 'type' => 'netflow' }, 'internal' => { 'port' => '9996', 'col' => '#FF0000', 'type' => 'netflow' }, # 'gw' => { 'port' => '9995', 'col' => '#0000ff', 'type' => 'netflow' }, # 'peer1' => { 'port' => '9996', 'IP' => '172.16.17.18' }, # 'peer2' => { 'port' => '9996', 'IP' => '172.16.17.19' }, );
As you can see, I have two valid sources with different ports and different colors. You can make all netflow, all sflow, or any combination of protocol.
change directory to /home/netflow
./install.pl etc/nfsen.conf cd /home/netflow/bin/ ./nfsen start
Make it start at boot (referenced from this post).
vi /etc/init.d/nfsen
Add this into the file:
#!/bin/bash # # chkconfig: - 50 50 # description: nfsen DAEMON=/home/netflow/bin/nfsen case "$1" in start) $DAEMON start ;; stop) $DAEMON stop ;; status) $DAEMON status ;; restart) $DAEMON stop sleep 1 $DAEMON start ;; *) echo "Usage: $0 {start|stop|status|restart}" exit 1 ;; esac exit 0
Then chkconfig it on to start it at boot:
chmod 755 nfsen && chkconfig --add nfsen && chkconfig nfsen on
That’s pretty much it. Once you configure your netflow or sflow source, you should start seeing data in ~5-10 minutes. Point your browser at your web server and see: Mine is set as https://netmon/nfsen/nfsen.php (you’ll need to include the “nfsen.php” uness you edit your apache config to recognize “nfsen.php” as in index.
Common issues:
I see this one every time: “ERROR: nfsend connect() error: Permission denied!” It’s always a permissions issue, as documented here. You need to make sure that the nfsen package can read the nfsen.comm socket file. I fixed mine by doing
chmod g+rwx ~netflow/
My nfsen.conf file is using /home/netflow as the $BASEDIR.
You’ll likely see “Frontend – Backend version mismatch!”, this is a known issue. There is a patch to fix it here, I never bothered since it did not cause any issues for me.
Disk full. Depending on your setup, you may generate a firehose worth of data. I have filled disks in less than a day in the past on a good sized regional WAN. I generally keep a month of data, but you can store as much data as disk you want to buy. I have a script run from cron to prune data, if you want to do the same:
vi /usr/local/etc/rmflowdata.sh
Paste this in:
#!/bin/bash # prune flow data # Usage: # +30 is the number of days, adjust accordingly. /bin/find /home/netflow/profiles-data/live -name "nfcapd.*" -type f -mtime +30 -delete
Add this to your crontab:
@daily /usr/local/bin/rmflowdata.sh
Make it executable
chmod 755 /usr/local/bin/rmflowdata.sh
There are probably more elegant ways to do it but this works just fine, is lightweight and can be run manually if needed.
There are a lot of great use cases for this. If you’re looking for an SDN tie-in, guess what, there is one. OpenVSwitch supports sflow export and low-and-behold, nfsen and nfdump can easily consume and display sflow data. Want flow statistics on your all VM, OVS based SDN lab? Guess what, you can!
There are some other great things you can do with flow data, too, specifically sflow. It’s not just for network statistics, there is a host based sflow implementation that track any number of interesting metrics. blog.sflow.com is a great resource for all things sflow (also, it does IPv6 by default, as it should).
Ok, now you have absolutely no good reason not to be collecting flow data. It’s easy, it’s useful and almost everything (hosts, routers, virtual switches) supports exporting some kind of flow information. You can even generate it from an inline linux box or a box off of an optical tap or a span port running softflowd or nprobe. Both of which I can confirm work wonderfully (the above collector is gathering flows from softflowd running on my security onion box as well as exported flows from pfflowd on a pfsense router).
Pingback: Internets of Interest for 20th January 2014 — EtherealMind
Pingback: Technology Short Take #38 - blog.scottlowe.org - The weblog of an IT pro specializing in virtualization, networking, storage, and servers
Pingback: Setting up Network flow monitoring using Nfsen on CentOS | Pradyumna joshi's Blog
Hi, thanks for this post. I’m a little stuck at
“vi etc/nfsen.conf” as I cannot find that file. Can you explain which directory this is in?
Thanks
Better late than never: but /data/nfsen/etc/nfsen.conf ?
And also, next time. sudo find / -name nfsen.conf
🙂
Hi,
Thanks for the post. I’m getting stuck at “vi etc/nfsen.conf” – as i cannot find that file. Can you please explain a bit further?
Thanks