The Little Network That Could: Which IPv6 Address?

Paul Gear |

An IPv6 addressing detour

My next post about The Little Network That Could (TLNTC) was going to be about IPv6 addressing plans - and this post is definitely about IPv6 addressing - but before we can lay out a plan, there's something more fundamental that needs to be decided: which IPv6 address range should I be using in my network plan? This seems like a very simple matter, and shouldn't need to be the subject of a lengthy blog post, but - like many things - this is more complex than it initially appears.

This complexity stems from mostly from my requirements of TLNTC: because it shares certain characteristics of enterprise networks and other characteristics of SOHO networks, it falls into a network category which IPv6's designers didn't really cater for when many of the protocol features were initially specified.

[Edit: Adding a few comments and making some wording changes after feedback from the folks in #ipv6 on Libera.chat IRC. Of course, they express it as "You don't understand IPv6 or DNS", and "your list is terrible", but that's IRC life. At least it didn't degenerate into name-calling.

Some of the comments were genuinely useful (I've updated the comments on ULA + NPTv6), but others were just repetition of RFC 1900, mandating the cookie-cutter edge network architecture mentioned after the RFC review, and assuming that by having IPv6 addresses in so many different places I'm doing it wrong. In some cases the software packages in question don't support DNS (I'm open to looking at more options); in other cases it's not possible: one's firewall software might be able to look up the (dynamically renumbered) address of a web server in DNS, but it can't look up firewall zone definition of 2001:db8:10:20::/60 there.]

A nested detour: network renumbering

During the 1990s, I worked for a time at the Queensland Police Service in their Unix systems division, and I distinctly remember one particular technical decision which was widely discussed: if the role of a Unix host changes, should its name be changed to match? I can't remember which side of this discussion I favoured at the time, but I distinctly remember the outcome and the reason for it: we decided never to change a host's name, because the process was too labour-intensive. The data in this decision's favour was compelling: we identified 27 places where a host's name was recorded in our systems, and only three of them were on the host itself.

This decision has sat in the back of my mind for the last 25 years, and has popped into the foreground each time I've come to think about large configuration changes in any complex system; network renumbering is a case in point. Most of the considerations around my choice of IPv6 address ranges revolve around one simple goal: avoiding renumbering. Renumbering has a long history of being poorly handled in Internet standards; let's take another detour to review some of those.

February 1996

Renumbering has been known to be a problem since very early on in the Internet's commercialisation, with RFC 1900: Renumbering Needs Work making these recommendations (my summary):

  1. Don't store IP addresses in places maintained by humans; always use fully-qualified domain names (FQDNs). Hard-coding IP addresses into any application should be deprecated.
  2. If you have a legacy application which breaks rule 1, generate its configuration file from another one which uses FQDNs and substitutes their values using DNS lookups.
  3. Don't implement software licensing restrictions which make use of IP addresses.
  4. Adopt and further develop tools which make it easy to renumber hosts (e.g. DHCP, DDNS, and service location protocols).

It's easy to criticise from this distance in history, but despite those recommendations being excellent advice, it was never likely that they would be any more than a Utopian ideal. One out of four ain't bad.

September 2005

RFC 4192: Procedures for Renumbering an IPv6 Network without a Flag Day has a highly informative section where it lists what needs to change when a network is renumbered (lightly paraphrased here for brevity - please see the original RFC for exact quotes):

  1. Link prefixes assigned to links
  2. IPv6 addresses assigned to interfaces on switches and routers
  3. Routing information propagated by switches and routers
  4. Link prefixes advertised by switches and routers
  5. Ingress/egress filters
  6. ACLs and other embedded addresses on switches and routers
  7. IPv6 addresses assigned to interfaces on hosts (use of SLAAC or DHCPv6 can mitigate this)
  8. DNS entries
  9. IPv6 addresses and other configuration information provided by DHCP
  10. IPv6 addresses embedded in configuration files, applications, and elsewhere.

Notice how many of these points apply only to switches, routers, firewalls, and other network equipment (answer: 8/10) and how many apply to every other device connected to a network (answer: 2/10). Point 10 ends with the comment:

Finding everything that must be updated and automating the process may require significant effort, which is discussed in more detail in Section 3. This process must be tailored to the needs of each network.

The promised "more detail" in section 3 is, in my judgement, very disappointing. It consists of:

  • an acknowledgement that it's easy to shoot oneself in the foot in a renumbering exercise (because of all the devices which aren't in the control of the network administrator and could contain hard-coded IP addresses),
  • a short discussion of the evils of applications ignoring DNS TTLs,
  • a call for "application designers, equipment vendors, and the Open Source community" to take note, and
  • advice that network operators "develop or purchase appropriate tools" for automation of renumbering.

May 2010

The issues with renumbering were again acknowledged in RFC 5887: Renumbering Still Needs Work:

Despite these efforts, renumbering, especially for medium to large sites and networks, is widely viewed as an expensive, painful, and error-prone process, and is therefore avoided by network managers as much as possible. Some would argue that the very design of IP addressing and routing makes automatic renumbering intrinsically impossible. In fact, managers have an economic incentive to avoid having to renumber, and many have resorted to private addressing and Network Address Translation (NAT) as a result. This has the highly unfortunate consequence that any mechanisms for managing the scaling problems of wide-area (BGP4) routing that require occasional or frequent site renumbering have been consistently dismissed as unacceptable.

And yet RFC 5887's authors' goals were very much in the opposite direction - encouraging renumbering:

It is certainly to be expected that as the pressure on IPv4 address space intensifies in the next few years, there will be many attempts to consolidate usage of addresses so as to avoid wastage, as part of the "end game" for IPv4, which necessarily requires renumbering of the sites involved. However, strategically, it is more important to implement and deploy techniques for IPv6 renumbering, so that as IPv6 becomes universally deployed, renumbering becomes viewed as a relatively routine event. In particular, some mechanisms being considered to allow indefinite scaling of the wide-area routing system might assume site renumbering to be a straightforward matter.

I think this is an equally Utopian view to that expressed in RFC 1900, because of the same things that RFC 4192 failed to adequately address. On the brighter side, at least RFC 5887 contained a substantial section acknowledging that management issues were a serious concern for renumbering.

July 2010

RFC 5902: IAB Thoughts on IPv6 Network Address Translation has some useful discussion of renumbering avoidance as one of the potential motivations for NAT under IPv6, and notes that provider-independent address space (PI-space) is the tool of choice for enterprises, but that the proliferation of PI-space for smaller sites/organisations would pose long-term scalability limits on the Internet overall.

March 2014

RFC 7157: IPv6 Multihoming without Network Address Translation did not address the issue of renumbering, but repeated what seems to be a Shibboleth in the IAB/IETF community: transparent end-to-end connectivity, which seems to trump all other concerns. It reads:

We conclude that DHCPv6-based solutions are suitable to solve the multihoming issues described in this document, but NPTv6 may be required as an intermediate solution.

This may solve multihoming at the network layer, but doesn't deal with the problems of inconsistent client implementations of source address selection or the other problems caused by renumbering/multinumbering. (At this point I'd also highlight that I haven't heard a great defence of transparent end-to-end connectivity, as opposed to end-to-end connectivity without the "transparent" part. At the moment it seems to me that stateless NPTv6 combined with either a reclassification of ULA's precedence above IPv4 or a GUA allocation made available for internal-only use is likely the only long-term viable solution for smaller site multihoming in IPv6.)

November 2016

RFC 8028: First-Hop Router Selection by Hosts in a Multi-Prefix Network doubled-down on the idea of multiple active prefixes being the solution for multihoming IPv6 networks.

October 2018

RFC 8475: Using Conditional Router Advertisements for Enterprise Multihoming proposed the idea of rapid renumbering of multihomed networks using Router Advertisements (RAs) with dynamically changing valid lifetimes, with no discussion of the problems which this might introduce in the end nodes.

December 2019

RFC 8678: Enterprise Multihoming Using Provider-Assigned IPv6 Addresses without Network Prefix Translation: Requirements and Solutions builds on RFC 8475's idea of rapid renumbering, and concludes that RAs are a more suitable mechanism than RFC 7157's suggestion of DHCPv6. Again, there's little acknowledgement of renumbering's impact on end nodes in the network.

TLNTC's place in this world

My impression of the overarching design scenario for the Internet (both IPv4 and IPv6) held by the above RFCs is that of small, flat customer edge ("eyeball") networks whose addressing and topology are largely telco-controlled, which connect to large data centres owned by telcos, content providers, and other enterprises relatively nearby (in terms of network topology).

The customers who own these edge networks are generally expected to be relatively unskilled - or even technologically illiterate, whilst the data centre owners are expected to have skilled network design and operations staff. The obvious implications of this are that eyeball networks must work with no intervention, based only on the information available on the wire from their ISP, and that any complexity in requirements or topology should be left to the professionals running the data centre networks.

If that were the end of the story there would be no place for networks like TLNTC. And, since the published RFCs are the standard by which IPv6 implementations are judged, by running an edge network with a complex topology I may rightly be accused of trying to use IPv6 for a purpose for which it was not designed.

However, using technologies for purposes other than those for which they were designed is not really an uncommon practice, and it seems that I'm not entirely alone in wanting my edge network to provide some services and share some data centre characteristics. Nick Buraglio, et. al. have been working on a draft informational RFC currently titled IPv6 Site connection to many Carriers, which covers the IPv6 multihoming options currently available to small-to-medium sites. Nick and some of his friends run The Modem Show podcast, where they recently discussed this draft (hereafter called FBNVV, after the initials of its authors).

FBNVV classifies various options for IPv6 connectivity based on a set of requirements, namely:

  • Carrier resiliency
  • End-to-end connectivity
  • Internal connectivity
  • Convergence speed
  • Complex topology support
  • Subscriber-only services
  • Traffic steering

Of these, end-to-end connectivity, internal connectivity, and complex topology support are highly important in TLNTC, whilst the others are less so, because it is not currently a multihomed network, largely for the reasons FBNVV outlines, and the difficulty of running networks using the techniques described in RFC 7157 and later. (I don't think I would be unreasonable in using the word 'absurd' to describe some of those.) However, multihoming would be a much-appreciated feature, given that my current plan for failure of my primary ISP consists of:

  1. turn on my 4G/5G phone's mobile hotspot (which uses a different service provider)
  2. switch my work laptop to use the hotspot, and continue working
  3. wait for my ISP to fix the problem, and wear the outage on the public-facing services hosted in TLNTC

The real problem with renumbering

One weakness of the current FBNVV draft is that it doesn't state strongly enough why renumbering is a problem; it's not because renumbering can't work, but because even if it does, it creates an untenable situation for any non-trivial network (what FBNVV calls "complex topologies").

Probably the best explanation of this which I have read is Johannes Weber's blog post IPv6 renumbering, a pain in the .... Weber sums it up this way (emphasis in original):

A few months ago my lab moved to another ISP which required [me] to change all IP addresses... While it was almost no problem to change the legacy IPv4 addresses (only a few NATs), it was a huge pain in the ... to change the complete infrastructure with its global unicast IPv6 addresses. It turned out that changing the interface IPv6 addresses was merely the first step, while many modifications at different services were the actual problem. And this was only my lab and not a complex company or the like.

...

TL;DR: Changing the public IPv4 addresses was done in a few minutes, while it took me a couple of weeks until all services were running with the new IPv6 addresses...

For a concrete explanation of why there was such a big administrative overhead in renumbering, check out Weber's table detailing the touch points and their scale. He is at pains to point out that his network is a simple lab, and that it would be worse in a more complex network.

In IPv6 Buzz episode 055: The Good, Bad, And Ugly Of IPv6 With Geoff Huston, Huston also points out (at around 39:40) that not every device renumbers in the same way, so some devices become non-functional for inconsistent amounts of time during a renumbering event. Host Ed Horley adds, "This doesn't help enterprises at all either, because they're hard-coding addresses on boxes left and right."

Where are IPv6 addresses used?

As I've been gradually modernising and rebuilding TLNTC over the last couple of years, I've been keeping track of the same types of things as Johannes Weber, building up a list of all the places in my network where IPv6 addresses are recorded. Here's the list as it currently stands:

  1. Listen and query source addresses of all UDP-based services. (In order to traverse stateful firewalls, UDP reply traffic must originate from the same source address as the destination of the original request.) This includes at least the following services:
    • DNS servers:
      • bind9 (listen-on-v6, notify-source-v6, etc. directives in /etc/bind/named.conf.options)
      • dnsmasq (/etc/dnsmasq.d/*.conf)
    • Net-SNMP snmpd (/etc/snmp/snmpd.conf)
    • NTP servers:
      • chronyd (/etc/chrony/chrony.conf)
      • ntpd (/etc/ntp.conf)
  2. Configuration files, databases, etc. for automation tools such as Ansible.
  3. DNS ACLs to allow queries, zone transfers, etc. (allow-query, etc. directives in /etc/bind/named.conf.options)
  4. DNS forwarder configuration (/etc/bind/named.conf.options or /etc/dnsmasq.d/*.conf)
  5. DNS primary (master) servers addresses in the configuration of all secondary (slave) servers (/etc/bind/named.conf.local)
  6. DNS resolver configuration (/etc/resolv.conf) for those hosts which do not use RDNSS (including local resolver configurations on DNS resolvers themselves)
  7. DNS zone file contents (/var/lib/bind/master/*)
  8. DNS64 configuration (/etc/bind/named.conf.options)
  9. Docker virtual network definitions (/etc/docker/daemon.json)
  10. Firewall configuration files (/etc/shorewall6/* and /etc/nftables.conf)
  11. Filters and query parameters for log file monitoring and reporting tools, SIEMs, etc. (e.g. /etc/fail2ban/*.local)
  12. KVM virtual network definitions, should these need IPv6 connectivity
  13. Mail server listen and send addresses, and relay ACLs (/etc/postfix/main.cf)
  14. NAT64 configuration (/etc/tayga.conf)
  15. Network configuration files of all hosts where static addresses are needed (/etc/network/interfaces in the old ifupdown scheme, or /etc/netplan/*.yaml in current Ubuntu/Debian). This is needed for all hosts which act as routers, because received DHCPv6-PD prefixes are for a single level only and don't transfer to downstream routers.
  16. Network diagrams (generated from source by graphviz)
  17. Network inventory and IPAM spreadsheets/databases (just a spreadsheet in my case, because the scale of TLNTC doesn't warrant anything more sophisticated)
  18. Proxy server configuration, including ACLs (/etc/squid/squid.conf)
  19. Router advertisement daemon configuration files (/etc/radvd.conf), not because the addresses to advertise need to be specified (radvd works this out from the interface if ::/64 is specified as the interface address), but because the RDNSS address needs to be included manually.
  20. Routing configuration for all hosts involved in dynamic or static routing (/etc/frr/*.conf)
  21. Web server ACLs (e.g. mod-status for Apache status monitoring, or any virtual host which must be source address restricted)

I'm almost certain that this list is incomplete, but it contains everything I've documented so far in my efforts. In the majority of the above cases, I've extracted these into variables templated into place using Ansible, but almost none of the applications in question are RFC 1900-compliant; they do not allow DNS names in place of addresses or address ranges.

It should be noted that all items in the above list which work in a dual-stack environment will contain IPv4 addresses as well as IPv6. However, none of the circumstances which would necessitate renumbering under IPv6 would do so under IPv4, because my network uses RFC 1918 addresses internally. Only those services which are dependent on public IPv4 addresses would be affected, and many of these can be handled by simple DNAT rules on the outside firewall.

I hope that the above is convincing enough evidence that renumbering is untenable for anything but the most significant network events (such as a company being acquired).

Which IPv6 address?

That detour aside, where does that leave me with respect to my original question: which IPv6 address range should I choose for my network? Here are the options I considered:

  1. PI space from my RIR (APNIC)
  2. PI space sub-allocation from my professional association, the ITPA
  3. ULA internally and NPTv6 at edge
  4. Hurricane Electric tunnel
  5. PA space from my ISP
Option Cost Size Pros Cons Evaluation
1 $1180/yr + $500 for the first year /48 (or more?) Never have to renumber (as long as I'm prepared to pay); works with any (good) ISP Requires a slightly more complex edge router setup (BGP required); contributes to the overall BGP scaling problem This would be my preferred option, were cost not a consideration. In the future I hope it will be a viable option.
2 $249/yr (my membership fee, which I pay anyway) /48 Never have to renumber (as long as I'm a member); works with any (good) ISP ITPA has been through some rocky times recently and its future is a little uncertain; has the same BGP considerations as obtaining my own PI space. This would probably be my first preference, had I not already started using my ISP's PA-space. The lower cost, same size allocation, and portability make it an easier option than obtaining my own PI-space.
3 $0 /48 Never, EVER, have to renumber. Zero cost. Needs split DNS to avoid traffic tromboning at my edge firewall. May not work with some protocols? (It seems like this might be an overblown concern - people made the same claim about IPv4 NAT, and it just isn't true; ref: Huston, IPv6 buzz ep. 055, around 34:00.) Is lower precendence than IPv4, so if I have any networks which need IPv4, it becomes unusable as an external IPv6 connectivity option. Block size mismatch with my ISP's allocation, therefore needs NAT66 at the edge for any ranges past the first /56. ULA + NPTv6 is non-viable for external connectivity. Recommended listening: IPv6 Buzz episode 90 where they conclude that there aren't that many good use cases for ULA. One of the options that they (legitimately!) discussed was squatting on some unused portion of the IANA IPv6 allocation, e.g. the 3000::/4 space. I would probably do that ahead of using ULA, but would never recommend it to anyone.
4 $0 /48 Never have to renumber (as long as HE keeps offering its tunnel service); HE have a tunnel broker in Sydney now, so performance is greatly improved over previous times when all traffic would have been tunnelled via the US. Works with any ISP. Traffic tromboning to some domestic destinations; HE peer with all of the smaller ISPs and most cloud providers in Australia, but for some of the larger ones we might need hit Los Angeles or Singapore to get to a domestic resource. Would lower my already-lowered-due-to-PPP MTU. This seems nearly worth doing as a backup plan. However, given that my ISP supports IPv6 natively, it seems like a retrograde step.
5 $120/year (the cost of my business support plan, which I pay anyway) /56 This is the default option, because I already had started using this before I went down the path of exploring other addressing options - so any change from this would require at least some renumbering regardless. Native IPv6 performance (8 ms pings to Brisbane from my place). Might have to renumber later if my ISP decides they can't/won't maintain my current block. Tied to my current ISP (not really a concern because I have no plans to change). Smaller block size than other options (not really a concern because I'm not using all of the /56 as it is). Whilst I probably wouldn't choose this again, I'm sticking with it for now and documenting all the places IPv6 addresses are stored (in this blog post!) so that any future renumbering will require less effort.

[Another option which I initially assumed that I would follow was that of using both GUA and ULA. But I quickly found that needing to distinguish between internal only and external facing wasn't always as cut & dried as I hoped. Also, one of the main drivers for me to start going down the IPv6-only path was that I don't want to maintain two sets of firewall rules, and running both GUA and ULA ends up looking a lot like that, although at least it would be in one firewall configuration.]

Where to next?

The options for avoiding renumbering whilst simultaneously allowing for future multihoming of TLNTC leave me with a pretty bad case of grumpy old man syndrome. It's 2023 and we definitely can't have nice things. On the one hand, we have the IAB/IETF goals of transparent end-to-end connectivity and global routing scalability, and on the other we have small sites with the need for reliable connectivity and avoiding costly renumbering. I hope that some compromise can be found in upcoming standardisation efforts which bridges the gap between these competing needs.