The School for Sysadmins Who Can’t Timesync Good and Wanna Learn To Do Other Stuff Good Too, part 1 – the problem with NTP

(With apologies to Derek Zoolander and Justin Steven.  And to whoever had to touch the HP-UX NTP setup at Queensland Police after I left. And to anyone who prefers the American spelling “synchronization”.)

(This is the first of a series on NTP.  Part 2 is an overview of how NTP works.)

The problem with NTP

In my experience, Network Time Protocol (NTP) is one of the least well-understood of the fundamental Internet application-layer protocols, and very few IT professionals operate it effectively.  Part of the reason for this is that the documentation for NTP is highly technical and assumes a certain level of background knowledge.

I first encountered NTP more than 20 years ago, and my first efforts with it were an unmitigated disaster due to my ignorance of how the protocol was designed to function.  Since then virtually every IT environment I’ve encountered has had a less-than-optimal NTP setup.

I am still far from an expert on NTP, but I’ve learned quite a lot about operating it since my early days.  I hope this series of posts will help you develop a working knowledge of NTP faster and get the basics of NTP configuration right in your environment.

Why learn NTP?

Why bother learning this rather obscure corner of Internet lore?  I mean, the Internet mostly works, despite this alleged widespread lack of expertise in time sync, right?

Here are some of the reasons you might want to learn more about NTP:

  1. You run Ceph, Mongodb, Kerberos, or a similar distributed system, and you want it to actually work.
  2. You want your logs to match up across multiple systems, potentially on multiple continents.
  3. You like learning about new things and tinkering with embedded systems.
  4. You think bandwidth-efficient, high-precision time synchronisation is just a fun, nerdy problem.
  5. You think this is cool:

    A scenario where the latter behavior [the PPS driver disciplining the local clock in the absence of external sources] can be most useful is a planetary orbiter fleet, for instance in the vicinity of Mars, where contact between orbiters and Earth only one or two times per Sol (Mars day). These orbiters have a precise timing reference based on an Ultra Stable Oscillator (USO) with accuracy in the order of a Cesium oscillator. A PPS signal is derived from the USO and can be disciplined from Earth on rare occasion or from another orbiter via NTP. In the above scenario the PPS signal disciplines the spacecraft clock between NTP updates.

    (Personally, they had me at “planetary orbiter fleet”. 🙂 )

Caveats

In this series, I’ll describe a few best practices for setting up NTP in a standard 64-bit Ubuntu Linux 16.04 LTS environment.  Bear in mind this quite limited scope; this advice will not apply in all circumstances and intentionally ignores the less common use cases.  Further caveats:

    1. I have no looks.
    2. I am not an expert.   My descriptions of the algorithms are based on the documentation and operational experience.  I’m not a member of the NTP project; I’ve never submitted a patch; I’ve never compiled ntpd from source (I hate reading & writing C/C++).
    3. I’ve only worked with the reference implementation of NTP, and only on Linux, with only one reference clock driver (NMEA), and a limited range of configuration options.
    4. I will be glossing over a lot of detail.  Sometimes it’s because I don’t think it’s necessary in order to work with NTP successfully; sometimes it’s because I haven’t looked into that particular corner and so I don’t understand it; sometimes it’s because I have looked into that particular corner and I still don’t understand it. 🙂  But mostly it’s because I’m attempting to keep this series accessible for those who are newcomers.  If you’re an experienced NTP operator, you probably won’t find much of interest (if anything) until later in the series.
    5. We won’t cover much history or theory of time sync in this series.  If you’d like to know a little more about that, check out Julien Goodwin‘s previous LCA & SLUG talks:

https://youtu.be/GhKKWdGn5qg

https://youtu.be/OwpHFm6xVFg

A bad runner’s journey into bad running, part 2 – how I run

(You can read part 1 first, if you want.)

When I first started running, I naturally fell into what I considered to be a proper running pace, but I found that I was not able to run as far as I wanted, and it wasn’t until I made a conscious effort to slow down and concentrate on distance that I was able to achieve 5 km without stopping (the goal of Couch to 5K).  They advise that to start you should run as slowly as possible; the slowest thing you can actually call running.  After a while I realised this was really good advice for me.

I haven’t really concentrated on my technique very much.  The main thing I tried was swinging my arms a little more than I thought was natural.  This seems to give the lungs more room and make diaphragm cramping less likely.  A lot of people I see running keep their elbows bent sharply with their forearms held high, but that hasn’t worked for me so far.

I sometimes find that my lungs start to protest as I get to about the 1-2 km mark in a run.  My brain immediately starts to make plans to stop and thinks up appropriate excuses – “You’ll probably feel sick if you keep going”, “you may not make it to 5 km if you don’t stop for a walk now”, and similar things.  I found that if I just ignore those excuses and keep running (and maybe ease off the pace a bit), after a little while the tight feeling in the lungs passes and I finish my 5 km very successfully.

I run in a place I love: at the beach, around low tide, in the early morning or late afternoon.  Our beaches here aren’t usually crowded, and early or late in the day is the time when many native birds are active, especially raptors.  (My favourite to watch is the white-bellied sea eagle – sometimes they’re amazing enough to make me stop running and just watch as they glide along.).

I run on the moist part of the sand – between the thick, dry sand above the high tide mark, and the hard, wet part that the water has recently been on.  This seems to give the right combination of stability of surface and underfoot cushioning.  I often have a swim or surf before (in the hotter part of the year) or after (in the cooler part of the year).

I run with as few accessories as possible: sunscreen (when necessary), a swimming rash vest, running skins (compression shorts), my Garmin fitness tracker and GPS (I currently use older models that don’t have the functions combined), and my car key on a shoelace around my neck.  In winter I have worn a light fitness jacket, but in this climate it’s not really necessary in all but a few weeks of the year, and causes me to overheat at other times.

I don’t wear shoes.  This means I need to keep a watch out for any sharp shells on the beach, but I haven’t found that particularly tricky, and I find I heat up less.  Until I started training on the road, I hadn’t really experienced any pain in my feet or legs due to running in bare feet (more on this later).

I don’t carry a phone or MP3 player.  I love music, but I don’t like things being in my ears when I’m sweating.  I prefer to listen to the sound of the waves and the birds.  Sometimes I sing (or some rough facsimile thereof), or recite music in my head, especially some of Neal Morse‘s longer pieces.

A bad runner’s journey into bad running, part 1

(I’m trying to get motivated to write a little more. This post is a little off-topic from my usual fare; hopefully it will be of interest to some.  There will probably be 3 or 4 parts.)

Disclaimer

This is my story about running.  I don’t claim it should be the norm for anyone.  If I lapse into 2nd person as I write, please be assured that it’s not intentional, and is not intended to be advice to anyone.  If you decide to follow my example, please see your doctor first to make sure it’s advisable for you to do so.

Why running?

The short version: I want to grow old well, and the only representative male in my recent family history died too young of a heart attack.

The longer version: Both of my grandfathers were sugar cane farmers.  My maternal grandfather ate two heavily-salted eggs every morning, was almost never sick, and lived to 95, still in full possession of his faculties.  Unfortunately, because my mother was adopted, that grandfather is only of assistance in gauging lifestyle and diet in my family, not genetic disposition to heart disease.

My father had rheumatic fever three times as a child, which wrecked his heart so badly that he’s had five open-heart surgeries for artificial valve work, starting at age 37.  So his experience wasn’t really useful in thinking about my health risks.  His father, however, was apparently healthy until one day he just dropped dead of a heart attack without warning.  He was 57; my father was 26; I was less than one year old.

So, I hit 40 knowing that my closest male relative died suddenly only 17 years older than I was, and realised that that wasn’t the way I wanted to go.  We had recently moved from Brisbane to the Sunshine Coast for a change of pace, and I was enjoying a more active lifestyle, but I decided that running was the next step in putting in place some better habits for the future.

First steps

I decided to ask friends and family for advice.  My younger brother has run marathons, and several of my work colleagues were runners at various levels.  So I asked them for tips on getting started and found out about Couch to 5K, a program designed to get couch potatoes fit enough to run 5 km (or 30 minutes) without stopping.  I decided to give it a shot.

I like to do things at my own pace, so I decided from the outset that I wasn’t going to be held to their 9-week schedule.  When we first moved to the Sunshine Coast, I couldn’t run from one end of Kings Beach to the other without stopping.  So I knew it would be a slow process.  But I kept plugging away little by little and eventually started achieving distances in the 2-3 km range.

Edit: part 2 is here.