32 Bit Real Estate

Two obvious costs of running Internet apps for users on your own hardware: hardware and bandwidth. We buy big servers and run them in racks at network providers that charge us to route large volumes of traffic using BGP4 Anycast. You probably have at least a hazy intuition for what those costs look like.

But there’s a non-obvious cost to what we do: we need routable IPv4 addresses. To make Anycast work, so users in Singapore are routed to Singapore instances, particularly for non-HTTP applications, like UDP DNS servers or TCP game servers, we assign distinct public IPv4 addresses to each app running on Fly.io.

You don’t have to think about any of this stuff when you deploy on Fly.io. We just assign apps addresses and get out of the way. But we have to think a lot about this stuff, because IPv4 addresses cost money.

You can’t, like, own a number, man!

Let’s take a second to reflect on what it means to acquire an IPv4 address. To own an IP address is to control what it routes to. The rock-bottom source of truth for IPv4 routing is BGP4. You announce IP address prefixes over IPv4 to your peers, your peers relay those announcements, and the world learns that traffic for your addresses needs to be sent your way.

Lots of providers will accept a BGP4 peering session with a customer. But no reputable provider will take a random announcement over that session. You’ll need to demonstrate authority over the prefixes you plan to announce.

Technically, nobody owns an IPv4 address. They’re administered, ostensibly as a public benefit, by the 5 Regional Internet Registries — ARIN, RIPE, APNIC, AFRINIC, and LACNIC. In the long long ago, it was the job the RIRs to allocate blocks of addresses to providers. Today, their IPv4 hives are void of honey, and their new job is just to keep track of who’s claiming which long-allocated blocks as they get shuffled around between their new, er, stewards.

All this is to say, if you want to “own” an IPv4 address, you find its previous owner and get them to arrange a transfer with an RIR. There is paperwork involved, and it’s sometimes whispered, a usage justification.

With the transfer completed, your newly “acquired” IPv4 addresses will be associated with your BGP4 ASN (oh, you’ll need to shell out a couple hundred bucks for the ASN, too), and you can hope that an upstream provider will accept announcements for them. The addresses are now “yours”.

For illustration: here’s a block of IPv4 addresses we control in IRR Explorer.

Certain Plans Require Additional Gribbles

So, Anycast. Easy. You have a routable IPv4 address. You have machines in a bunch of places. In each of those places, you peer BGP4 with your upstream and advertise that exact same IPv4 address. Global BGP4 routing ensures that when people try to connect to to that address, they’ll be sent to the “closest” machine. That’s how Fly.io’s Anycast network functions: we have blocks of addresses, machines in a bunch of regions, BGP4 peering in each of them, and a Rust-based proxy that picks up connections for those addresses.

You can think of ways to work around needing routable IPv4 addresses for Anycast applications. None of them work for us.

Here’s the first thing that doesn’t work: IPv6. Let’s get it out of the way first. Yes, there is such a thing as IPv6. Yes, it has enough address space to give distinct addresses to each electron in every atom of every living human. And, yes, we do a lot with IPv6, including allocating public IPv6 prefixes to all of our applications. We’re IPv6 fans.

But if I had to place a bet, it’d be that you aren’t using IPv6. And you’re nerdy enough to be reading a blog post on how much IPv4 addresses cost! The silent majority of non-nerd users are nowhere close to being reliably IPv6-connected.

The next thing that doesn’t work is name-based hosting. Applications can share IPv4 addresses using TLS SNI. We could park most apps on a single address and just let browsers tell our proxies which apps people are looking for. So far so good: the people who want their own IPv4 addresses would be flying business class, and we could charge them 2x as much.

We like money as much as anyone else. But SNI breaks down for apps that don’t use TLS, and we want to support all kinds of apps, not just web apps. Now, most Fly.io apps are web apps. And we know which apps are which. So we could use SNI for the majority of our apps and IPv4 addresses for the exceptions. But that feels weird, and worse, means we’d be maintaining multiple provisioning paths for customers. It’s not worth the hassle, for us or our customers, so IPv4 addresses are just bundled in for everyone.

IPv4: The Internet’s Real Estate Market

There are lot of organizations holding addresses they don’t need. Internet routing used to be real, real dumb, and to give a big company enough addresses to number their machines, you often had to give them way more addresses than they really needed. Apple has a /8 – that’s 256 /16s, each of which has 256 /24s, each of which is 256 addresses. So does Prudential Insurance(?!), and so does Ford. Lots of other organizations had “smaller” blocks that are still unfathomably large to modern sensibilities. So, like a crappy cryptocurrency, IPv4 is pre-mined.

If you invested in IP addresses 15 years ago, you’re doing pretty well. If you invested in IP addresses 15 months ago, you’re also doing well.

When we wrote the first draft of this post, the going rate for smaller IPv4 blocks was $45 per IP address. was the rate in September of 2021. We looked just now, and the spot price was $50. We give up quoting them; by the time you get to this sentence, they’ll have gotten more expensive. Here’s some recent transactions.

Did you know big cloud providers publish their IP ranges? They do this because it’s useful for automating firewalls. It's also useful if you want to guess how much IPv4 wealth they’re sitting on. You could even write a Ruby script to tell you. Which I did. (Dan Goldin did the same thing 6 months before me).

Here’s what I saw in March of 2021 when IPv4s cost $25 each:

And here’s what it looked like in September of 2021:

These estimates are low: Amazon announces more than 100 million IP addresses. That’s 2.4% of all possible IPv4 addresses!

IP prices vary by block size. It’s helpful to think of a 256-address /24 as the irreducible block size. That /24 is 10-15% less expensive than a /18, which is 64 contiguous /24s. There are reasons for the price difference, and we don’t know all of them. One big reason is that if you’re doing classical network engineering and you need to number 10,000 hosts, the routing is easier to do with a /18 than with 40 /24s. Another reason is probably vanity.

Anyways, AWS’s average block comprises 18,000 IPv4s. Bonkers. Giant booksellers who sell cloud services as a side hustle don’t even bother with IPv6. Here is a complete list of the IPv6 addresses dig aaaa amazon.com returns:

dig aaaa amazon.com +short                                                           

Financial Network Engineering

Imagine for a moment that you’re us. Take a drink, eat an Italian beef sandwich, pet your dog. Now get apps working for developers in, say, twenty different cities. You can buy a beast of a server for $20k USD, chop it up into 500+ virtual machines, and build an API for turning those on and off very quickly. People are pretty happy to pay for virtual machine time; you’re almost there.

VMs are mostly useful when they can talk to the Internet. Your developers need IP addresses. Say it costs $45 for a single IP address. That means it costs $23,040 to handle 512 virtual machines with IPv4 addresses. We’ve now doubled our costs.

But hold up for a second.

Servers depreciate; that’s their job. That $20k server depreciates to $500 relatively quickly. But IP blocks are, at least for now, appreciating. Alarmingly.

This is a hard market to time. Nobody believes the Internet will be IPv6-only within the next few years. There are credible people who believe IPv4 addresses will be scarce and useful indefinitely. We might put money on you getting away with an IPv6-only app 20 years from now. But we’d have said that 20 years ago, too.

So, unfortunately, the smart thing to do if you own $1B worth of IPv4 addresses is to buy $1B more IPv4 addresses. Fortunately, everyone is starting to recognize this, so you have some flexibility in financing.

If you’re a farmer and you need a new tractor, you probably don’t just go buy it in cash. You go to a bank and get a loan, collateralized by the value of the tractor. If you’re a company with $500,000 in receivables, you don’t necessarily have to wait for your customers to remit payments; financial institutions will loan you money collateralized by your invoices. And, of course, when you buy a house, you live in it while you pay back a mortgage to the bank.

Your local credit union won’t collateralize a loan on IPv4 blocks like it would for a car or tractor. IPv4 addresses are just numbers, man. Logan Paul can convince unsophisticated people to put a dollar value on completely useless numbers. But your bank isn’t so forward thinking, and the conversation putting a $50 value to a single 32-bit number isn’t going to be productive.

But startups have another option: venture debt. Banks will happily lend startups money based almost entirely on the credibility of a startup’s investors, and, transitively, their blog posts. When a company raises a $12MM round, they’ll spend the next few weeks raising $3MM of debt from banks friendly with that firm.

Venture debt is surprisingly simple. It has an interest rate, which can be pretty close to prime. It usually includes an option grant – the bank gets a small amount of equity (much less than employees get). And venture debt typically requires that you do all your banking with the bank that gave you the loan. Unlike raising a funding round, once you’ve got investors, there isn’t much pitching involved in raising debt. Banks win or lose venture debt deals based on how much certain founders like their web interface. It’s not not the most important factor when choosing a bank.

All this is to say, you can in a sense take out a mortgage on a block of IP addresses.

How much sense does this make long-term?

IPv6 will be viable – most people will be able to run apps with no IPv4 addresses – someday. Could happen in 2 years. Far more likely to happen in 20.

The kinds of companies that benefit the most from a decisive transition to IPv6 tend to look a lot like us: new, small, minimal legacy infrastructure. Not coincidentally, those companies also tend not to have much market power. Strong incentive, weak market power: not exactly a recipe for change.

There are other reasons to think IPv4 addresses will retain value. Path dependence is one of them; .COM names are still pricey despite the universal acceptance of new TLDs. Switching costs are another; lots of very large networks are IPv4-only, some with significant numbers of nodes that can’t easily be dual-stack. It’s easy to imagine a world in which IPv6-only is viable, but IPv4 addresses remain scarce.

Still, IPv4 will be obsolete some day. Addresses will increase in value and then one night, while we’re all sleeping, they’ll become worthless.

This could be a knock against using a bunch of debt to pay for IP assignments, or not.

An alternative is leasing: companies of varying levels of sketchiness will happily lease IP blocks. We’ve had a bunch of offers for two bucks per month per address. Maybe we could negotiate that down to one bucks. A three year lease for 65,000 IP addresses at $1/mo each works out to $2.34 million. Four years is $3.12 million.

We’d almost certainly get more addresses by leasing. And we wouldn’t have the inventory risk of holding thousands of IPv4 addresses that could conceivably be worth $0 if IPv6 takes over. But leases have fixed terms, and running apps don’t. Meanwhile, we do know empirically that people hate renumbering. So do the IPv4 lessors. We’d be in no position to negotiate a good rate on lease renewal – in fact, we’d be in a bad position even if IPv4 addresses got much cheaper. We want to know that when we assign an address to an app, the people running that app will get to keep using that address indefinitely. Leasing isn’t worth the hassle for us.

IPv4 is exhausting

At scale, routable IPv4 addresses are surprisingly expensive. But they’re appreciating. Like an office building or a house, we have tools for financing them that we don’t for machines or salaries. So, part of our business is acquiring addresses outright, and attaching them to apps people run here.

For most ordinary apps, this doesn’t matter much; HTTPS web applications would run just as well with SNI and shared (or temporary) addresses as they do with permanent IPv4 allocations. But not all apps are ordinary, and we want the oddball apps to work well on Fly.io without anyone having to think much about them.

Mostly, we just believe that people who run apps on Fly.io now will still be running them in 10 years, and Fly.io is better if we can keep that IP for them the entire time.

Really, though, we’re nerds, and we think it’s funny that for all the talk of NFTs and ICOs, the Internet itself has been running a high-dollar token auction for the last 20 years.