The talk covers some of the operational issues which NTP has faced over the past 30 years, RFC drafts for future changes to the protocol, and Best Current Practices.
I recently wanted to look at some packet captures on my NTP pool servers and find out if any NTP clients hitting my servers use extension fields or legacy MACs. Because the overall number of NTP packets is quite large, I didn’t want to spool all NTP packets to disk then later filter with a Wireshark display filter – I wanted to filter at the capture stage.
I started searching and found that not many quick guides exist to do this in the capture filter. However, the capability is there in both tcpdump and tshark, using either indexing into the UDP header, or using the overall captured frame length. Here’s an example of tcpdump doing the former (displaying it to the terminal), and tshark doing the latter (writing it to a file):
tcpdump -i eth0 -n -s 0 -vv 'udp port 123 and udp[4:2] > 56' tshark -i eth0 -n -f 'udp port 123 and greater 91' -w file.pcap
Both of the above filters are designed to capture NTP packets greater than the most common 48-byte UDP payload. In the case of udp[4:2], we’re using the UDP header’s 16-bit length field, which includes the header itself. In the case of greater, it uses the overall captured frame length, and actually means greater-than-or-equal-to (i.e. the same as the >= operator); see the pcap-filter(7) man page for more details.
This will be a shorter, less-polished post than usual. It’s really just a way to start to bring a bit of structure to my thoughts. Feel free to add some comments or weigh in on the linked Twitter thread.
I came away from a recent PacketPushers episode, “Don’t Believe The Programming Hype” a bit disappointed that the discussion didn’t get to the heart of the matter. Then there was this tweet from Jon Langemak which kicked off some interesting discussion about the place of programming:
I find it curious that people are looking for devs and are willing to teach them networking but not networkers willing to teach them dev
— Jon Langemak (@blinken_lichten) April 23, 2017
Why shouldn’t we believe the non-programming crowd?
So here are my propositions (and some corresponding anti-propositions):
- Programming isn’t hype; programming is a fundamental IT skill. If you don’t understand the basics of computer architecture (e.g. CPU instruction pointers, registers, RAM, stacks, cache, etc.) and how to create instructions which make computers do useful things (i.e. coding), you’re not an IT professional.
- This doesn’t mean you must understand exactly how every computer architecture works, or know how to code in assembly language (or directly in hexadecimal like Mel the Real Programmer). But it does mean that if you’re confronted with a problem, you know enough fundamentals to peel back the layers, and work through the elements in each layer methodically (with the help of the documentation) to determine the component in error.
- There is no fundamental distinction between scripting and programming. Both are designed to accomplish basically the same things, at different levels of abstraction. Suggesting that someone who writes scripts to automate his or her infrastructure is not a programmer confuses the issue and does not progress the discussion.
- This doesn’t mean that vendor-originated code isn’t better than in-house scripts (sometimes it is; sometimes it isn’t), but it does mean that the value of the local process and infrastructure knowledge contained within in-house scripts mustn’t be sidelined. No off-the-shelf software “just works” without configuration or customisation of some sort, so in-house scripts are fundamentally necessary.
- The above distinction is probably just a specific instance of the generalisation that there’s a difference between builders and non-builders. This is also a non-real distinction. Every enterprise IT organisation is building something, whether they use code they’ve written themselves or rely solely on vendors. You can’t compile a data centre contract into existence or code up a WAN link; working IT systems are built both by those who create the individual components, and those who build on those components. It’s turtles all the way down; not many software vendors assemble their own motherboards or run their own chip fabs…
How did we get here?
So how did we get to this point where the concept of learning programming seems so intolerable to networking folks?
In my mind the main factor is the normalisation of proprietary software – that is, the widespread acceptance that it’s OK for software to be closed to scrutiny and not fully controlled by those who depend upon it. (On the contrary, I hold that Open Source is essential for the operational safety of functioning IT systems.) The dominance in their respective markets of Microsoft Windows on the corporate desktop & server and Apple iOS on consumer devices are no doubt the most common examples of this, but other, more egregious examples pop up routinely.
This has led to a generation of IT folks for whom the fundamentals were something akin to magic. “If I press here it does this…” The pushback against programming is only one example of this. (Another is the cargo-culting/guessing troubleshooting culture that many of us in operational roles have encountered.)
My plea to those who are pushing back against the programming trend is simply this: have a go! Programming is just building things – like you already do – just at a different level of abstraction. You might find that you like it, and it opens up a whole new range of possibilities for you.
I recognise that I write this from a biased perspective:
- When I started in IT, there were no “IT” qualifications. When I went to university, the course was still called Applied Science with a major in computing.
- My first job (while still in uni) was a programmer. After that, I was a Unix sysadmin. Back then, Unix sysadmins were expected to know C in addition to the usual scripting tools. Network engineering was a relatively late addition in my career; I didn’t start being the primary networking resource in my job until 2003, and didn’t get my CCNA until 2012. So I always approached networking from a slightly different perspective to the “average” network engineer who did his/her CCNA straight out of school.
- I’m a Free Software “true believer”; I first installed Linux from floppy disks with a distro which used kernel version 0.97. I’ve been using Linux on the desktop for almost 20 years. You can’t survive that long in the Open Source world without being able to compile (and sometimes fix!) software on a semi-regular basis.
Feel free to discuss! Either in the comments below, or shoot me some thoughts on Twitter.
Vyatta Core (VC) is one of my top fanboy loves. It provides a firewall/router based on Debian Linux but with an elegant configuration system modeled on Junos. Vyatta previously offered VC as a community edition of their commercial router (with somewhat reduced features and a lag time). Their strategy seems to have changed when Brocade took them over, and unfortunately, the project has withered on the vine. I think I can safely pronounce it dead:
- Around 21 June 2013, public git commits were stopped and moved to internal servers. Bug tracking seems to have been moved to internal servers as well.
- Since then there has been complete silence about roadmap of VC, even to the point that the roadmap page now gives an access denied error. Previously-available patches and discussions also now show access denied.
- The forums are full of spam and documentation has been neglected.
- Off-the-record sources in Brocade/Vyatta have confirmed the above, so the assumption must be that all future development will be on Vyatta Subscription Edition (VSE), with no further progress for VC.
The community has already mourned this loss and started seeking alternatives. And yet, a vibrant community feel still exists in a few places:
- Vyatta’s tweet stream still seems consist of approximately 50% retweets of users raving about how great the config system is and how the product solves their real world problems.
- When Ubiquiti‘s EdgeRouter Lite using EdgeOS (a fork of VC6.3 ported to 64-bit embedded MIPS) was released, its forums exploded with user activity. They grew so much in the first few weeks after release that I unsubscribed due to not being able to keep up with the message volume. Unfortunately, Ubiquiti have always done their development internally, releasing source code as tarballs rather than the preferred method of hosting public git repositories.
- Plenty of people (including me) are still deploying it in new environments. When I first encountered VC, I liked it so much that I recommended to a client to use it for all their virtual routing. We bought VSE and haven’t even used it, because VC does everything we need. I would be happy to keep paying for VSE just as a way to support the future development of VC.
- There are still some interesting threads cropping up in the forums.
- Contributors to the unofficial Vyatta wiki have already started plans for a fork.
Why has this happened to such a great product? I think it simply boils down to Brocade (and Vyatta prior to the acquisition) not understanding FLOSS (Free/Libre and Open Source Software) culture and how to run a successful FLOSS project. Simon Phipps just wrote about how hard it is for companies to migrate to FLOSS and do it well, and VC is the latest casualty in this. Red Hat is one of the few companies doing it consistently.
I think it’s several months too late for Brocade to fix this situation. As the above links show, many people have lost faith in Brocade to do the right thing with VC. But hypothetically speaking, what would be the best possible outcome?
- Brocade employs a full-time staff member as community advocate/liaison. (I hereby volunteer for this job. Contact me for rates. 🙂 They reverse their decision about public git repositories. They convince Ubiquiti to do likewise and work towards a common shared code base in the public git repos. Together they make a kick-butt Vyatta Core, which is used as the core of future VSE and EdgeOS releases. It becomes the Vyatta equivalent of Fedora to Red Hat’s RHEL, the OpenSUSE to SLES. Brocade accepts contributions from the community, and begins providing per-incident or support contract-based services for VC as well as VSE.
- Next best thing: Brocade cultivates some good contacts in the community and begins regular source code drops to enable community respins, forming a relationship akin to that between RHEL and CentOS. The community takes up the mantle of providing support on whatever the Open Source respin is called.
- The community forks from the latest available Vyatta Core, integrating Brocade and Ubiquiti code whenever possible. This would be the least desirable outcome, but I see it as the most likely, given how hard it is to change networking vendors from the community side.
Outcome 1 is obviously a bit pie in the sky from me. But there are sound reasons why Brocade should do it. Most significant of these is that the virtual routing space is a hot area at the moment, and making VC the go-to choice for virtual routing would give Brocade the jump on Juniper’s vSRX and Cisco’s CSR1000v (by undercutting them on price). Vyatta is modelled on the Junos style of configuration, so the learning curve for Juniper engineers would be trivial. And if people go to Brocade for their virtual routing, they’re more likely to go to them for their physical network and storage switching.
Furthermore, VC has huge mind share amongst the people who deploy it. The people I’ve met who use Vyatta love working with Vyatta. I love working with it. It has the potential to gain Brocade many extra customers and avocates, both paid and unpaid. (Hint to Brocade: reasonably-priced support for VC would get you a lot of customers that wouldn’t have otherwise paid for VSE.) Embracing the community is a great way to ensure that network/systems engineers who have the opportunity to influence decision makers will influence them towards Brocade.
Let’s hope it’s not too late, and that Brocade proves me wrong. It would be a shame to see such a great project die. (And I am serious about the offer to become their community advocate. Call me. 🙂
This post is the story of my first practical look at Junos on Juniper EX-series switches.
One day last December, Skeeve Stevens from eintellego opened a can of worms by offering a deal on Juniper equipment to all network engineers on the AusNOG mailing list. I had been looking for an excuse to try out Juniper switches for a while; my office switch, an HP ProCurve 3400cl (J4905A), was very loud, and the EX2200-C was the only affordable fully-managed Gigabit fanless switch which i could find from a major networking vendor. So i ordered one and waited patiently. I expected it to take 6 weeks or so to arrive (since apparently all the cool kids ordered one too), so i was pleasantly surprised when it arrived the day after Boxing Day. (For those non-Commonwealth inhabitants reading this page, Boxing Day is the day after Christmas Day; it’s always a holiday here in Australia.) Over the Christmas/New Year break, i spent three days learning and playing with Junos. I don’t claim this as a fair, balanced, or reasonable review – it’s simply a recounting of my experiences.
I wanted to get a good general idea of the Junos platform and set it up as my main office switch, moving the HP to my test lab (which is only turned on when i’m using it). So the basic task i set myself was to learn enough Junos to replicate the functions of my ProCurve as completely as possible. I knew ahead of time that it wasn’t going to be a 100% fit – the 3400cl was a great value switch for its day (and still is, in many ways), and had a lot of features included out of the box (like OSPF and PIM-DM multicast routing). Junos requires an additional license to implement OSPF, and it wasn’t discounted (buying it would nearly have tripled the cost of the switch), so i knew that i’d be eliminating that part of my config. My other OSPF devices are VMs running Debian Linux with Quagga, and Cisco 1700 & 1800 series routers running IOS, so i simply connected them to the same VLANs and phased out the ProCurve as a router.
My first few impressions of the hardware were positive:
- The switch has a solid metal case with a nice heavy feel to it.
- It was well packed, and came with a very well-packed rack mount kit (perhaps a little too well-packed):
- It also came with a rather longer-than-expected, but solid, power cord retainer:
- It sports wall-mounting holes as well as a rack-mount kit (hat tip to Chris Lathem for pointing this out):
The biggest turn-off for me on the hardware side was the realisation that Juniper switches use zero-based port numbers. Seriously? Being an old C programmer, i think of this as “proper” from a programming perspective, but try telling an end-user (troubleshooting an issue in a branch office over the phone with tech support) that port 1 is the second port not the first. I can’t think of any reason other than compatibility with “the way they’ve always done it” on their routing platforms, and i consider this self-indulgent behaviour on an equipment manufacturer’s part. Other vendors manage to start unit or chassis numbers at zero without being bloody-minded about starting port numbers at zero (although HP’s Comware switches start their port numbers at the bottom left rather than the top left, which is almost as frustrating).
After my inspection of the hardware, i moved on to plugging it in and inspecting the software. Bootup took a bit longer than I expected (the system uses a 1.2 GHz Marvell ARM CPU – the same as in my QNAP TS-219P), but it looked reassuringly Unixy: Junos uses a FreeBSD-based management plane. After seeing the switch booted through to a login prompt, i logged in as root and had a poke around. Typical Unix commands like ls, df, and ps all had their usual functions, and one runs “cli” to get into the Junos configuration system. I worked for the first day or so from the serial console.
I noticed some disconcerting dates in the initial copyright notice:
“This product includes FreeBSD software developed by the University of California, Berkeley, … 1979, … 1994. “
Does this mean that 1994 is the most recent date they synchronised with the FreeBSD project? I hope this is not true, and that Juniper are actively engaging with the Open Source project upon which their code is based.
“GateD software copyright © 1995, the Regents of the University. All rights reserved. Gate Daemon was originated and developed through release 3.0 by Cornell University and its collaborators. Gated is based on Kirton’s EGP, UC Berkeley’s routing daemon (routed), and DCN’s HELLO routing protocol. Development of Gated has been supported in part by the National Science Foundation. Portions of the GateD software copyright © 1988, Regents of the University of California. All rights reserved. Portions of the GateD software copyright © 1991, D. L. S. Associates.”
It has been a very long time since i’ve seen a Unix distribution old enough to contain a gated implementation, and i sincerely hope that no vestiges of this code remain.
During boot an alarm light appeared on the front panel. I was afraid i had a hardware fault and would have to return my shiny new switch! After some investigation in the relevant manuals, i found the command to view the active alarms (“show chassis alarms”) and found that it was due to the management interface being down. I already had a management VLAN in my network, and i was expecting this switch to host it. I investigated the management LAN (at this point a shout-out to the helpful folks in #juniper is due) and found that it was not VLAN-capable at all. It’s more like a NIC for the switch’s management plane. So it’s a true out-of-band network, and can’t be bridged onto any part of the data plane. I quickly decided not to use it (and disabled the alarm with “set chassis alarm management-ethernet link-down ignore”). I can understand why the management port is provided, but i’d much prefer that it could be bridged (in software, if necessary) onto the relevant VLAN for installations that aren’t large enough to have a dedicated physical out-of-band management network. I also think enabling alarms for the management LAN being disconnected in the factory default configuration violates the principle of least surprise.
Software Updates and Support
When commissioning a new switch, router, server, storage array, etc., one of the first things i do is look for the latest firmware/software to ensure i’m up-to-date with security patches. This proved to be a bit of an issue for Junos. As a newly-arrived customer, i didn’t know what to expect; HP offer all software updates for their switches on their public web and FTP sites; with Cisco it’s a bit more involved, requiring a login to their site, and with routers you need to know the right things to say to TAC if you don’t have a support contract (with switches they’re more generous – could this have anything to do with the existence of GNS3?).
With Juniper, it seems one must have a login on their site that is specifically granted access to register devices. I had already registered a user account on their site in order to download the documentation and view their certification materials, and i was somewhat perplexed and frustrated to find that i couldn’t even register my switch until Juniper had changed my account to have “real customer” status. Their support staff were first class, though, and got it sorted out pretty quickly.
My reading of documentation began with Juniper’s “Day One” guides, some of which i had downloaded prior to receiving the switch. These guides range from the 66-page “Exploring the Junos CLI” to the massive “Complete Software Guide for Junos OS for EX Series Ethernet Switches, Release 11.1”, weighing in at 4628 pages. Suffice it to say i only used the latter as a reference. If you’ve used Vyatta at all, you’ll find the former guide a fairly dull read. A quick skim to see the differences between Junos and Vyatta was all that was necessary for me. (Vyatta is a deliberate clone of the applicable parts of Junos’ interface.) “Configuring Junos Basics” was a useful guide to the initial setup of the switch, some parts of which have made it into my Network Engineer’s Rosetta Stone – initial setup wiki page. At 84 pages, it’s pretty digestible in a short CLI session. I used the free PDF format downloads, but most of the Day One guides are available for purchase in print format also. (Curiously, they cost US$0.99 to purchase in Kindle format, even though Kindle can read PDFs just fine.) The Day One guides (and the follow-on series “This Week”) are generally excellent in quality, and formatted for easy comprehension, although i thought that the headings could have been clearer in some cases. (“This Week: Hardening Junos Devices” devices is the one that i made mention of in my notes.)
My checklist for features to convert from ProCurve to Juniper looked like this:
- Initial setup: user logins, hostname, management IP address, default gateway, DNS, NTP, time zone
- Essentials: VLANs, Rapid Spanning Tree Protocol (RSTP), LACP trunk to VM server, port labels, syslog, SNMP, ssh public keys
- Hardening: login inactivity timer, login banner, disable unused services (including telnet and web management), management ACLs, eliminate LLDP data leakage on Internet interfaces, Spanning Tree BPDU protection/BPDU filtering
- Nice-to-have: QoS on VLANs, port mirroring, IGMP querier, DHCP relay
Initial setup was pretty straightforward, especially given the aforementioned Day One guide. Having worked with Vyatta for a few months, i found it easy to adjust to the Junos CLI style. It’s a simple and elegant configuration system which made instant sense to me the first time i used it, and Junos worked very much as i expected. When logged in via ssh, responsiveness of the EX2200-C at the CLI (in terms of character echo delay) is noticeably snappier than the ProCurve 3400cl it replaced (and many other similar devices).
Junos has a number of powerful configuration features. The ones i came to appreciate most were configuration groups (i used this to set up SNMP) and interface ranges (which i used to set up my VLAN assignments). The latter is very powerful: a range’s name can be used in place of an interface name almost anywhere, and it lives in the configuration as a real entity, as opposed to the temporary nature of the equivalent on IOS. (Recent HP Comware versions allow a port list to be permanently associated with a name, but store the configuration in the individual ports rather than the interface range itself.)
Configuration is very slow to commit on the EX2200-C compared with Vyatta on an Intel server (understandable, given the low-end CPU); “commit | display detail” set my mind at rest that it was doing useful things, so i just got used to waiting for my commits to complete. When working in configuration mode, using “run <operational command>” works similarly to IOS’ “do <operational command>” with the added bonus that tab and ? completion works. This was very helpful when i found that “run show interfaces brief” is not very brief, and “run show interfaces terse” is probably more like what i expected of briefness.
There were a number of features in the configuration process that struck me as sub-optimal, or at least unpleasantly surprising:
- There are separate NTP servers during boot and normal operation, with separate configurations for each. (Presumably under the covers the former are used as arguments to ntpdate during boot whereas the latter are used to configure ntpd.) This strikes me as a rather obscure feature which probably resulted from a large government department insisting on its inclusion in a large RFP. The very strange thing about this is that only one boot NTP server is allowed, whereas multiple NTP servers are allowed during normal operation.
- The Junos shell interprets the pipe character (vertical bar) in a counter-intuitive fashion: the pipe is a well-established mechanism on Unix/Linux, Windows, IOS and Comware (amongst others) that sends the output of the first command to the input of the second command, which is a filter. Junos seems to use it as a generic way to affect the display of the output of the first command, which would more regularly be done by flags or additional arguments on other operating systems (e.g. “ls -l” on Unix/Linux, “dir /b” in DOS/Windows, “show ip interfaces brief” on IOS). This is another departure from the principle of least surprise, and i’m glad that Vyatta didn’t follow Junos in this respect (thus “show | compare” in Junos is simply “compare” in Vyatta).
- The Junos best practice (i can’t remember which guide i read this in) is to set the IP address of lo0.0 to 127.0.0.1/32 rather than 127.0.0.1/8, as it is on Linux and Windows. This is more likely an issue when using Junos for routing rather than switching, but i still couldn’t see a reason for it.
- Entering multi-line banners is annoying: one must use n within the string instead of just pressing enter and finishing the string on the next line.
- VLAN setup is a little tricky. I initially tweeted: “
#Juniper #EX2200-C impression #9: #Junos makes more sense as a router OS than a switch OS. VLAN setup is almost as painful as #Mikrotik.” But after spending more time setting up VLANs on Mikrotik recently, i can now formally rescind that remark. 🙂 VLANs are configured in two different ways: under the interface (or interface range) and under the VLAN definition. Juniper recommends using the former for trunk (tagged) interfaces, and the latter for access (untagged only) interfaces. Because i often play around with my VLAN assignments, i ignored this advice and defined them all as if they were trunks.
- Junos includes a concept of an “edge” port, but i couldn’t find a clear definition of it in the manuals i was working with, and as soon as i tried to apply BPDU filtering using “bpdu-block-on-edge” in interface context, i discovered its inflexibility, and discarded it in favour of “set ethernet-switching-options bpdu-block interface NAME”.
Almost everything i had on my checklist to convert from my 3400cl was done at the end of my three-day lab. The exact mechanisms for each feature (on both the Juniper EX2200-C and the HP 3400cl it replaced) can be found in the config files attached to this post. A few features seemed to be missing or deficient (although this could be due to my poor research skills rather than a feature gap):
- There is seemingly no equivalent of IOS’ “switchport trunk allowed vlan remove ID”. I couldn’t find a way to set up a VLAN trunk such that all VLANs except a particular list were allowed.
- There’s also seemingly no equivalent of ProCurve’s “lldp admin-status PORTNUMBER RxOnly” (for allowing receive but not send of LLDP info); Julien Goodwin suggested that ACLs could be used to fix this; i simply chose to disable LLDP entirely on my ISP uplinks.
- More perplexingly still, there’s seemingly no equivalent of IOS’ “no ip routing”. Seems very strange that there is a Junos kernel option to drop all TCP packets (i can’t think why anyone would want to turn on this option, since it would prevent remote management via ssh), but no option to turn off IP routing. Workarounds suggested by the IRC crowd include:
- blocking traffic between vlans by policy
- using multiple routing instances and putting each VLAN in its own instance
- create only L2 VLANs with no L3 interface (which is the option i chose in this case)
- There’s no way to enable all interfaces for sFlow – it appears one must add each interface individually. And further, there’s no SNMP write access at all, so i couldn’t just configure authentication and leave the heavy lifting to sFlowTrend.
- Probably the feature i spent the most time investigating and discussing on IRC (hat tip especially to Kurt Bales) was VLAN QoS priorities. Junos’ QoS capabilities are extensive, but configuring them is not as straightforward (some would say “simplistic”) a matter as ProCurve makes it (vlan X; qos priority N). Because my home office switch is never congested, i decided it wasn’t worth spending the time to learn this part of Junos until later.
Junos is a mostly-logical network OS with an elegant configuration system and excellent freely-available documentation. The quality of the EX2200-C hardware is first rate, and i suspect this only increases in subsequent models in the series. I’m glad i took the time to get familiar with the basics of Junos, and i’ll probably try to be certified to their entry-level standard if time permits. I don’t think Juniper’s offering is unique enough to cause me to choose them ahead of Cisco or HP (the latter’s Comware switches are still my go-to choice), but they’re certainly an excellent choice for quality managed switches, especially for those organisations with an investment in Juniper’s routing platform.