Making NTP best practices easy with Juju charms

NTP: a behind-the-scenes protocol

As I’ve mentioned before, Network Time Protocol is one of those oft-ignored-but-nonetheless-essential subsystems which is largely unknown, except to a select few.  Those who know it well generally fall into the following categories:

  1. time geeks who work on protocol standardisation and implementation,
  2. enthusiasts who tinker with GPS receivers or run servers in the NTP pool, or
  3. sysadmins who have dealt with the consequences of inaccurate time in operational systems. (I fall mostly into this third category, with some brief forays into the others.)

One of the consequences of NTP’s low profile is that many important best practices aren’t widely known and implemented, and in some cases, myths are perpetuated.

Fortunately, Ubuntu & other major Linux distributions come out of the box with a best-practice-informed NTP configuration which works pretty well.  So sometimes taking a hands-off approach to NTP is justified, because it mostly “just works” without any special care and attention.  However, some environments require tuning the NTP configuration to meet operational requirements.

When best practices require more

One such environment is Canonical’s managed OpenStack service, BootStack.  A primary service provided in BootStack is the distributed storage system, Ceph.  Ceph’s distributed architecture requires the system time on all nodes to be synchronised to within 50 milliseconds of each other.  Ordinarily NTP has no problem achieving synchronisation an order of magnitude better than this, but some of our customers run their private clouds in far-flung parts of the world, where reliable Internet bandwidth is limited, and high-quality local time sources are not available.  This has sometimes resulted in time offsets larger than Ceph will tolerate.

A technique for dealing with this problem is to select several local hosts to act as a service stratum between the global NTP pool and the other hosts in the environment.  The Juju ntp charms have supported this configuration for some time, and historically in BootStack we’ve achieved this by configuring two NTP services: one containing the manually-selected service stratum hosts, and one for all the remaining hosts.

We select hosts for the service stratum using a combination of the following factors:

  • Reasonable upstream Internet connectivity is needed.  It doesn’t have to be perfect – NTP can achieve less than 5 milliseconds offset over an ADSL line, and most of our customer private clouds have better than that.
  • Bare metal systems are preferred over VMs (but the latter are still workable).  Containers are not viable as NTP servers because the system clock is not virtualised; time synchronisation for containers should be provided by their host.
  • There should be no “choke points” in the NTP strata – these are bad for both accuracy and availability.  A minimum of 3 (but preferably 4-6) servers should be included in each stratum, and these should point to a similar number of higher-stratum NTP servers.
  • Because consistent time for Ceph is our primary goal, the Ceph hosts themselves should be clients rather than part of the service stratum, so that they get a consistent set of servers offering reliable response at local LAN latencies.

A manual service stratum deployment

Here’s a diagram depicting what a typical NTP deployment with a manual service stratum might look like (click for a larger image).

To deploy this in an existing BootStack environment, the sequence of commands might look something like this (application names are examples only):

# Create the two ntp applications:
$ juju deploy cs:ntp ntp-service
    # ntp-service will use the default pools configuration
$ juju deploy cs:ntp ntp-client
$ juju add-relation ntp-service:ntpmaster ntp-client:master
    # ntp-client uses ntp-service as its upstream stratum

# Deploy them to the cloud nodes:
$ juju add-relation infra-node ntp-service
    # deploys ntp-service to the existing infra-node service
$ juju add-relation compute-node ntp-client
    # deploys ntp-client to the existing compute-node service

Updating the ntp charm

It’s been my desire for some time to see this process made easier, more accurate, and less manual.  Our customers come to us wanting their private clouds to “just work”, and we can’t expect them to provide the ideal environment for Ceph.

One of my co-workers, Stuart Bishop, started me thinking with this quote:

“[O]ne of the original goals of charms [was to] encode best practice so software can be deployed by non-experts.”

That seemed like a worthy goal, so I set out to update the ntp charm to automate the service stratum host selection process.

Design criteria

My goals for this update to the charm were to:

  • provide a stable NTP service for the local cloud and avoid constantly changing upstream servers,
  • ensure that we don’t impact the NTP pool adversely, even if the charm is widely deployed to very large environments,
  • provide useful feedback in juju status which is sufficient to explain its choices,
  • use only functionality available in stock Ubuntu, Juju, and charm helpers, and
  • improve testability of the charm code and increase test suite coverage.

What it does

  • This functionality is enabled using the auto_peers configuration option; this option was previously deprecated, because it could be better achieved through juju relations.
  • On initial configuration of auto_peers, each host tests its latency to the configured time sources.
  • The charm inspects the machine type and the software running on the system, using this knowledge to reduce the likelihood of a Ceph, Swift, or Nova compute host being selected, and to increase the likelihood that bare metal hosts are used.  (This usually means that the Neutron gateways and infrastructure/monitoring hosts are more likely to be selected.)
  • The above factors are then combined into an overall suitability score for the host.  Each host compares its score to the other hosts in the same juju service to determine whether it should be part of the service stratum.
  • The results of the scoring process are used to provide feedback in the charm status message, visible in the output of juju status.
  • if the charm detects that it’s running in a container, it sets the charm state to blocked and adds a status message indicating that NTP should be configured on the host rather than in the container.
  • The charm makes every effort to restrict load on the configured NTP servers by testing connectivity a maximum of once per day if configuration changes are made, or once a month if running from the update-status hook.

All this means that you can deploy a single ntp charm across a large number of OpenStack hosts, and be confident that the most appropriate hosts will be selected as the NTP service stratum.

Here’s a diagram showing the resulting architecture:

How it works

  • The new code uses ntpdate in test mode to test the latency to each configured source.  This results in a delay in seconds for each IP address responding to the configured DNS name.
  • The delays for responses are combined using a root mean square, then converted to a score using the negative of the natural logarithm, so that delays approaching zero result in a higher score, and larger delays result in a lower score.
  • The scores for all host names are added together.  If the charm is running on a bare metal machine, the overall score given a 25% increase in weighting.  If the charm is running in a VM, no weight adjustment is made.  If the charm is running in a container, the above scoring is skipped entirely and the weighting is set to zero.
  • The weight is then reduced by between 10% and 25% based on the presence of the following running processes: ceph, ceph-osd, nova-compute, or swift.
  • Each unit sends its calculated scores to its peer units on the peer relation.  When the peer relation is updated, each unit calculates its position in the overall scoring results, and determines whether it is in the top 6 hosts (by default – this value is tunable).  If so, it updates /etc/ntp.conf to use the configured NTP servers and flags itself as connecting to the upstream stratum.  If the host is not in the top 6, it configures those 6 hosts as its own servers and flags itself as a client.

How to use it

This updated ntp charm has been tested successfully with production customer workloads.  It’s available now in the charm store.  Those interested in the details of the code change can review the merge proposal – if you’d like to test and comment on your experiences with this feature, that would be the best place to do so.

Here’s how to deploy it:

# Create a single ntp service:
$ juju deploy --channel=candidate cs:ntp ntp
    # ntp service still uses default pools configuration
$ juju config ntp auto_peers=true

# Deploy to existing nodes:
$ juju add-relation infra-node ntp
$ juju add-relation compute-node ntp

You can see an abbreviated example of the juju status output for the above deployment at http://pastebin.ubuntu.com/25901069/.

Disclosure statement

I wrote most of this post as part of my day job as a Site Reliability Engineer for Canonical.  We’re hiring.

Don’t believe the non-programming hype

Background

This will be a shorter, less-polished post than usual.  It’s really just a way to start to bring a bit of structure to my thoughts.  Feel free to add some comments or weigh in on the linked Twitter thread.

I came away from a recent PacketPushers episode, “Don’t Believe The Programming Hype” a bit disappointed that the discussion didn’t get to the heart of the matter.  Then there was this tweet from  which kicked off some interesting discussion about the place of programming:

Why shouldn’t we believe the non-programming crowd?

So here are my propositions (and some corresponding anti-propositions):

  1. Programming isn’t hype; programming is a fundamental IT skill.  If you don’t understand the basics of computer architecture (e.g. CPU instruction pointers, registers, RAM, stacks, cache, etc.) and how to create instructions which make computers do useful things (i.e. coding), you’re not an IT professional.
    • This doesn’t mean you must understand exactly how every computer architecture works, or know how to code in assembly language (or directly in hexadecimal like Mel the Real Programmer).  But it does mean that if you’re confronted with a problem, you know enough fundamentals to peel back the layers, and work through the elements in each layer methodically (with the help of the documentation) to determine the component in error.
  2. There is no fundamental distinction between scripting and programming.  Both are designed to accomplish basically the same things, at different levels of abstraction.  Suggesting that someone who writes scripts to automate his or her infrastructure is not a programmer confuses the issue and does not progress the discussion.
    • This doesn’t mean that vendor-originated code isn’t better than in-house scripts (sometimes it is; sometimes it isn’t), but it does mean that the value of the local process and infrastructure knowledge contained within in-house scripts mustn’t be sidelined.  No off-the-shelf software “just works” without configuration or customisation of some sort, so in-house scripts are fundamentally necessary.
  3. The above distinction is probably just a specific instance of the generalisation that there’s a difference between builders and non-builders.  This is also a non-real distinction.  Every enterprise IT organisation is building something, whether they use code they’ve written themselves or rely solely on vendors.  You can’t compile a data centre contract into existence or code up a WAN link; working IT systems are built both by those who create the individual components, and those who build on those components.  It’s turtles all the way down; not many software vendors assemble their own motherboards or run their own chip fabs…

How did we get here?

So how did we get to this point where the concept of learning programming seems so intolerable to networking folks?

In my mind the main factor is the normalisation of proprietary software – that is, the widespread acceptance that it’s OK for software to be closed to scrutiny and not fully controlled by those who depend upon it.  (On the contrary, I hold that Open Source is essential for the operational safety of functioning IT systems.)  The dominance in their respective markets of Microsoft Windows on the corporate desktop & server and Apple iOS on consumer devices are no doubt the most common examples of this, but other, more egregious examples pop up routinely.

This has led to a generation of IT folks for whom the fundamentals were something akin to magic.  “If I press here it does this…”  The pushback against programming is only one example of this.  (Another is the cargo-culting/guessing troubleshooting culture that many of us in operational roles have encountered.)

My plea to those who are pushing back against the programming trend is simply this: have a go!  Programming is just building things – like you already do – just at a different level of abstraction.  You might find that you like it, and it opens up a whole new range of possibilities for you.

Caveats

I recognise that I write this from a biased perspective:

  1. When I started in IT, there were no “IT” qualifications.  When I went to university, the course was still called Applied Science with a major in computing.
  2. My first job (while still in uni) was a programmer.  After that, I was a Unix sysadmin.  Back then, Unix sysadmins were expected to know C in addition to the usual scripting tools.  Network engineering was a relatively late addition in my career; I didn’t start being the primary networking resource in my job until 2003, and didn’t get my CCNA until 2012.  So I always approached networking from a slightly different perspective to the “average” network engineer who did his/her CCNA straight out of school.
  3. I’m a Free Software “true believer”; I first installed Linux from floppy disks with a distro which used kernel version 0.97.  I’ve been using Linux on the desktop for almost 20 years.  You can’t survive that long in the Open Source world without being able to compile (and sometimes fix!) software on a semi-regular basis.

Invitation

Feel free to discuss!  Either in the comments below, or shoot me some thoughts on Twitter.

An update to "What's in my Podcast Roll?"

I blogged previously about the podcasts I regularly listen to, and the topic came up again this week on the SAGE-AU mailing list, so I thought I’d update it with my current thoughts.

Regular listens from the previous list

  • SANS Internet Storm Centre daily podcast [feed] – Still my “must listen” podcast.
  • Risky Business [feeds] – They have done a little cleaning up on the NSFW content, but they could still do with a little more.  The RB2 feed has been expanded to include “Serious Business”, a light-hearted look at general current affairs with Dan Ilic, an Australian comedian based in the U.S.
  • Packet Pushers [feeds] – Getting a bit too frequent for a full-length (60+ minutes) podcast, but still interesting; they’ve diversified content and now include “Network Break”, a shorter, business-/news-focused show, and “Datanauts”, a “silo-busting” show on data centre topics in general.  One disappointment is that Michele “Mrs. Y” Churbirka’s “Healthy Paranoia” is inactive at the moment.
  • Linux Voice [feed] – Desktop/mobile/freedom-focused Linux podcast from the ex-Linux Format/Tuxradar team.  Sometimes not as technical as I would like.
  • DevOps Cafe [feed] – good interviews, not too frequent.
  • The Cloudcast [feed] – Broad coverage of cloud topics, from both business and technical perspectives.  Some of their guest spots are a bit light on content, but overall still pretty good. Sometimes it’s hard to keep up with the amount of content.
  • Andy Stanley – A profound Bible scholar disguised as a catchy communicator. Multiple podcasts:

Added recently

  • Quirks and Quarks [feed] – weekly science show which features interviews with (mostly) doctorate-qualified scientists talking about their studies.  Compelling stuff – I never miss an episode.  They take a break over the Northern Hemisphere summer, which almost gives me withdrawal symptoms.
  • Software Gone Wild [feed] – Software-Defined Networking from Ivan Pepelnjak, a long-time networking expert.  Discusses interesting projects; not afraid to call out traditional vendors for their hype and vapourware.
  • Arrested DevOps [feed] – Deals a lot with of cultural and organisational issues relevant to the DevOps movement, as well as technical topics.

Podcasts I’ve dropped


Source: libertysys.com.au

Default permit still winning the security battle

I was stoked when Patrick Gray took up my suggestion to ask Marcus Ranum to reflect on “The Six Dumbest Ideas in Computer Security“.  I encourage you to listen to the interview for yourself, but my summary of it is that Marcus was mostly discouraged that very little progress has been made in computer security, while Patrick was of the opinion that a number of good lessons had been learned in certain key areas.

Patrick pointed particularly to Apple’s iOS as a commercially-successful example of default-deny execution policy.  Whilst iOS, Windows Vista and later, and even Android (to a lesser extent) have implemented varying levels of default-deny when it comes to execution of programs, I think default-permit policy is still the dominant mindset in our industry.  As I was listening to the interview, a few areas came to mind where it still seems to be true:

  • Outbound connections from client devices.  Despite the fact that client-based exploits have become the dominant method of compromising organisations (the so-called “Advanced Persistent Threat” which compromised RSA was started with a phishing campaign and an Excel-delivered Flash exploit) and security practitioners generally assume that client devices (whether PCs or phones) are routinely compromised, many (most?) networks provide allow outbound connections from client devices by default, often to any destination and sometimes on any protocol.  This is exacerbated by the appalling lack of proxy server support in most iOS and Android applications, which means that administrators of BYOD networks rarely have any choice in the matter if they want to provide a functional service.
  • Compounding the problem is the fact that generally when users browse or client-side apps make connections, all web sites are allowed.  In this area, enumerating badness (Marcus’ stupid idea #2) is still dominant; many (most?) web filtering solutions which attempt to protect clients from malware use a blacklist of known-bad sites.
    I’ve worked in K-12 school IT management, support, and consulting for a number of years, and every now and then the suggestion of whitelisting web sites comes up.  That’s usually all that happens.  Other fields (perhaps banking, industrial control systems, or medical applications?) might also consider it, but I suspect that they end up with similar conclusions (i.e. that it’s impractical to implement).  (I’d love to hear from anyone who has actually tried this in a real network.)
  • Scripting languages are a common exception to the default-deny execution policies of operating systems.  To my knowledge, Windows PowerShell is the only common scripting system which allows for script signing policies.  However, scripts can request that Windows simply turn this feature off, which defeats the purpose.  To my knowledge, no signing system or default deny policy has ever been implemented for Unix/Linux systems (other than the default protection provided by Mandatory Access Control systems like SELinux and AppArmor).
  • The Android application permissions system is one of my pet peeves. Android applications must inform the Google Play store about which security- and privacy-related features they intend to use.  This is good; however, permissions are approved when the application is installed, and users only have the choice of installing or not installing.  Many applications require permissions that are not obviously critical to their operation, but because users typically try to install an application because they want to use it, an informed evaluation of an application’s permissions is rarely performed at installation time.  Most applications are installed regardless of what permissions they request.  So effectively, this becomes a default-permit situation.  (Moxie Marlinspike‘s WhisperSystems seemed to be making progress on this before they were acquired by Twitter, and I hope that Open WhisperSystems takes up this work again in the near future.)

All of this says to me that we’re still living very much in a default-permit world, and there’s a lot of work to be done before we can confidently say that progress has been made in this department.

I’d love to hear any further thoughts on this; you can reach me through the feedback page, or on Twitter.


Source: libertysys.com.au

The tragedy of Vyatta Core's demise

Vyatta Core (VC) is one of my top fanboy loves.  It provides a firewall/router based on Debian Linux but with an elegant configuration system modeled on Junos.  Vyatta previously offered VC as a community edition of their commercial router (with somewhat reduced features and a lag time).  Their strategy seems to have changed when Brocade took them over, and unfortunately, the project has withered on the vine.  I think I can safely pronounce it dead:

The community has already mourned this loss and started seeking alternatives.  And yet, a vibrant community feel still exists in a few places:

  • Vyatta’s tweet stream still seems consist of approximately 50% retweets of users raving about how great the config system is and how the product solves their real world problems.
  • When Ubiquiti‘s EdgeRouter Lite using EdgeOS (a fork of VC6.3 ported to 64-bit embedded MIPS) was released, its forums exploded with user activity.  They grew so much in the first few weeks after release that I unsubscribed due to not being able to keep up with the message volume.  Unfortunately, Ubiquiti have always done their development internally, releasing source code as tarballs rather than the preferred method of hosting public git repositories.
  • Plenty of people (including me) are still deploying it in new environments.  When I first encountered VC, I liked it so much that I recommended to a client to use it for all their virtual routing.  We bought VSE and haven’t even used it, because VC does everything we need.  I would be happy to keep paying for VSE just as a way to support the future development of VC.
  • There are still some interesting threads cropping up in the forums.
  • Contributors to the unofficial Vyatta wiki have already started plans for a fork.

Why has this happened to such a great product?  I think it simply boils down to Brocade (and Vyatta prior to the acquisition) not understanding FLOSS (Free/Libre and Open Source Software) culture and how to run a successful FLOSS project.  Simon Phipps just wrote about how hard it is for companies to migrate to FLOSS and do it well, and VC is the latest casualty in this.  Red Hat is one of the few companies doing it consistently.

I think it’s several months too late for Brocade to fix this situation.  As the above links show, many people have lost faith in Brocade to do the right thing with VC.  But hypothetically speaking, what would be the best possible outcome?

  1. Brocade employs a full-time staff member as community advocate/liaison.  (I hereby volunteer for this job. Contact me for rates. 🙂  They reverse their decision about public git repositories.  They convince Ubiquiti to do likewise and work towards a common shared code base in the public git repos.  Together they make a kick-butt Vyatta Core, which is used as the core of future VSE and EdgeOS releases.  It becomes the Vyatta equivalent of Fedora to Red Hat’s RHEL, the OpenSUSE to SLES.  Brocade accepts contributions from the community, and begins providing per-incident or support contract-based services for VC as well as VSE.
  2. Next best thing: Brocade cultivates some good contacts in the community and begins regular source code drops to enable community respins, forming a relationship akin to that between RHEL and CentOS.  The community takes up the mantle of providing support on whatever the Open Source respin is called.
  3. The community forks from the latest available Vyatta Core, integrating Brocade and Ubiquiti code whenever possible.  This would be the least desirable outcome, but I see it as the most likely, given how hard it is to change networking vendors from the community side.

Outcome 1 is obviously a bit pie in the sky from me.  But there are sound reasons why Brocade should do it.  Most significant of these is that the virtual routing space is a hot area at the moment, and making VC the go-to choice for virtual routing would give Brocade the jump on Juniper’s vSRX and Cisco’s CSR1000v (by undercutting them on price).  Vyatta is modelled on the Junos style of configuration, so the learning curve for Juniper engineers would be trivial.  And if people go to Brocade for their virtual routing, they’re more likely to go to them for their physical network and storage switching.

Furthermore, VC has huge mind share amongst the people who deploy it.  The people I’ve met who use Vyatta love working with Vyatta.  I love working with it.  It has the potential to gain Brocade many extra customers and avocates, both paid and unpaid.  (Hint to Brocade: reasonably-priced support for VC would get you a lot of customers that wouldn’t have otherwise paid for VSE.)  Embracing the community is a great way to ensure that network/systems engineers who have the opportunity to influence decision makers will influence them towards Brocade.

Let’s hope it’s not too late, and that Brocade proves me wrong.  It would be a shame to see such a great project die.  (And I am serious about the offer to become their community advocate.  Call me.  🙂


Source: libertysys.com.au