Workshop

AWell, you operate under a fundamental assumption, it is a heuristic you have likely never articulated out loud, yet it governs every single diagnostic decision you make regarding network infrastructure.

BOh, absolutely. I mean, I guarantee that's exactly how you listening, visualize it too.

AYou believe, quite simply, that a single IP address must physically reside in only one location. You treat an IP address like the exact coordinates of a house on a street.

BRight, because, you know, that is how we are taught from day one. If I open a terminal and ping an IP address, my brain immediately pictures a very specific physical piece of silicon.

AYes, sitting in a specific rack. Exactly. In a specific data center somewhere, it is incredibly comforting. It's like the foundational bedrock of how we think the internet works. One address, one physical destination.

BPrecision is required here because that comfort is built entirely on a functional illusion.

AA very persistent illusion, yeah. Indeed. So today's source material is a highly illuminating article by Ian McCutcheon, titled "A Stitch in Time, Anycast Workshop", and our primary mission for this deep dive is to, well, surgically correct this deeply ingrained, fundamentally flawed assumption in your mental map of network routing.

BYes. We are going to dismantle the idea that our route lives in one place. And honestly, prepare to have your mind completely blown by this. This concept is going to completely shatter how you view internet infrastructure.

ARight. And more importantly, how you design your own networks.

BIt is a significant paradigm shift. It totally is. It's one of those things where once you see the mechanics behind it, you can never unsee it. Your entire mental map is going to get a massive upgrade today.

ALet us begin with a systematic examination of the empirical data to demonstrate this geographic anomaly.

BOkay. Lay out the data.

AConsider a standard single-homed server located in a single data center in Virginia. If we execute a trace route from a machine in Singapore to that specific Virginia server, we can track the latency packet by packet.

BA long trip.

AVery. It takes roughly 233 milliseconds for the packet to arrive. We can literally read the router host names along the path like a boarding pass stub.

BOh, I love doing that. You see Tokyo, then LA.

APrecisely. Tokyo at 75 milliseconds, Los Angeles at 185 milliseconds, finally arriving in Ashburn, Virginia at 231 milliseconds. The packet demonstrably crosses the Pacific Ocean and the North American continent.

BWhich, you know, makes perfect sense. It's a massive physical distance from Singapore to Virginia. The data has to physically travel through fiber optic cables sitting at the bottom of the ocean and the speed of light and glass has a hard limit.

ACorrect. The physics are undeniable. And if we execute a trace route to that exact same Virginia server from a machine in London, it takes 77 milliseconds.

BRight.

AThe path goes from London to Newark to Ashburn across the Atlantic Ocean. This reinforces the routing reality you currently accept a packet has a single destination and reaching it incurs a latency cost directly proportional to physical distance.

BYeah, you pay for the distance you travel. Simple enough.

ANow observe the behavior of the IP address 1.1.1.1.

BOkay, Cloudflare's DNS.

AYes. When we ping 1.1.1.1 from Singapore, the response time is a mere 1.4 milliseconds.

BWait 1.4?

AYes. And when we ping that exact same IP address 1.1.1.1 from London, the response time is 0.9 milliseconds.

BOh, wow.

AA 200-fold difference compared to our Virginia server. Furthermore, the trace route reveals an even more fascinating detail. The packet originating in Singapore never actually leaves the city.

BIn the city of Singapore?

AYou're kidding.

BI am not and the packet originating in London never leaves London.

AOkay, wait. Let's pause right there. If the server is in one place and I'm pinging it from two opposite sides of the planet, getting sub two millisecond response times in both locations just defies physics.

BIt appears to, yes.

AUnless these packets are somehow traveling faster than light, what is happening? Wait, are they using copies that magically sync or is this a longest prefix match trick?

BExplain your hypothesis.

AWell, normally a router obeys specificity, right? If a router has two paths in its table, say, a broad slash 24 subnet and a highly specific slash 25 subnet, the slash 25 wins because it's a smaller, more precise target.

BThat is standard routing logic, yes.

ASo is Cloudflare just injecting hyperspecific routes in local regions to override a global route?

BWell, a perfectly reasoned hypothesis based on standard routing behavior, but incorrect in this instance.

AOh, man, really specificity is not doing the work here.

BThere is no slash 25 beating a slash 24.

ACloudflare is advertising the exact same slash 24 prefix one dot one dot one dot zero slash 24 from more than 300 different cities simultaneously 300.

BYes, there are 300 separate geographic origins for one single route 300 physical locations, all raising their hands to the global routing table and saying, Hey, I am one dot one dot one dot one, all at the exact same time.

AIndeed, that sounds like it should create a massive collision.

BThat is the illusion of the single destination.

AThis is where your mental map must adjust.

BWhen a router in Singapore receives a packet destined for one dot one dot one dot one, it simply consults its border gateway protocol or BGP routing table for the lowest cost shortest path.

ARight, because Cloudflare is advertising the route from a data center physically located in Singapore.

BThe router chooses that path. It is merely a few hops away and the router in London does the exact same calculation independently and finds a path that stays within London.

ABGP is doing exactly what it was engineered to do routing based on cost and distance.

BOh, wow. So the router in Singapore doesn't care and probably doesn't even know that there are 299 other locations claiming the exact same address.

AIt is entirely ignorant of them. Yes. It just looks at its table, sees a route that is two milliseconds away and say, Yep, that's the cheapest path. I'm handing the packet off.

BIt's entirely oblivious to the global redundancy.

APrecisely in standard enterprise networking, you are conditioned to think regionally.

BHopcount is treated as basic plumbing within a confined finite shape.

ARight, point a point B. But when you scatter 300 origins for one exact prefix across the entire globe, Hopcount stops being basic plumbing.

BIt becomes a literal geographic map. A geographic map. I love that analogy.

AEvery router on Earth simply hands the packet to the nearest origin because it is the cheapest route cost equals distance. The nearest copy wins.

BThat, strictly speaking, is the fundamental definition of anycast.

AThat is absolutely fascinating. And I can see why the latency is basically zero, but I can already hear the objection from anyone listening who manages a corporate network.

BOh, yeah, people usually file anycast away under the category of hyperscaler magic.

ALike, sure, Cloudflare can do that because they have a billion dollar budget in their own global backbone.

BCommon misconception. Right. Or sure, that's how the DNS root servers at eight dot eight dot eight or nine dot nine dot nine dot nine work.

ABut if I am just a network engineer running a standard enterprise environment, I don't have hyperscaler money. Does this actually apply to my work?

BIt applies profoundly. This is the most crucial practical application of the knowledge within McCutcheon's article.

AOh, really? Anycast is perfectly executable within a private enterprise network using standard RFC 1918 space.

BWait, just regular private IPs. Exactly. That is the private IP space you already own and control your 10 dot networks. You're 192 dot networks.

AYou do not need to be a global hyperscaler to leverage the fundamental geometry of the network.

BReally? You can just build this internally. I'm guessing it just requires your internal routers to act the same way the Internet backbone does.

AIndeed. The fundamental recipe requires merely a single IP address and two or more origins.

BOkay, walk me through it. Let us examine a precise enterprise application. Imagine standing up an internal DNS resolver.

AYou need a highly available IP for your client machines to query. You assign it the IP address 10 dot 10 dot 10 dot 53.

BOh, wait, I have to jump in here using 10 dot 10 dot 10 dot 53 for DNS server is brilliant because DNS operates on UDP port 53.

AExactly. It is a deliberate and highly efficient choice.

BI love that. By integrating the required port number into the IP address itself, network engineers never have to consult documentation or a wiki to remember what service that IP provides.

AThe function of the address is permanently embedded in the identifier. It's so incredibly clean. You see the dot 53 and your brain immediately registers DNS.

BOh, okay. So we have our internal DNS designated at 10 dot 10 dot 53. What is the next step to make it anycast?

AYou configure your network to advertise that exact same address 10 dot 10 dot 10 dot 53 from a router in every single one of your data centers.

BOkay. Now every client device across your entire corporate network points its DNS configuration to that one single private IP.

AWhen a laptop in your New York office queries 10 dot 10 dot 10 dot 53, the network naturally routes the packet to the New York data center because it's closest.

BYes. And when a server in Tokyo queries, the exact same IP, the network routes it to the Tokyo data center because the corporate network is just following the shortest path on your internal routing map.

AThe exact same way BGP did it globally. The traffic naturally falls into the closest bucket.

BCorrect. And this becomes exceptionally powerful for services that strictly mandate an IP address configuration rather than a host name.

ALike what? Consider logging infrastructure like syslog or timing protocols like NTP.

BIn modern networks, we frequently discuss global server load balancing or GSLB for steering traffic to the nearest healthy site.

ARight. GSLB is the standard playbook. You have a DNS name like logs dash.company.com and if the East Coast data center goes down, the GSLB dynamically updates DNS to point that host name to the West Coast IP address.

BThat is the standard playbook. Yes. But it has a fatal flaw for specific infrastructure.

AGSLB relies entirely on DNS resolution. When you configure a core network switch or a firewall to forward its syslog data, the configuration field almost always requires a raw IP address.

BIt will not accept a fully qualified domain name. It cannot perform a DNS lookup, which is a massive operational pain.

ABecause if that single syslog collector dies, DNS can't save you. You literally have to SSH into hundreds of core switches and manually update the IP address to point to the backup server.

BYes. Manual intervention. During an outage, that is the last thing you want to be doing.

AExactly. It is highly inefficient operational burden. But if you make your syslog collector's IP in anycast address advertising the same raw IP from multiple data centers, every router in your fleet ships its logs to one single unchanging address.

BOh, that is so smart. The underlying network geometry automatically ensures the packets land at the nearest healthy collector. Zero client reconfiguration is required during a site failure. The IP simply moves.

AOkay. I am totally tracking with the benefits here. But let's talk about the elephant in the room. Any network engineer listening right now is having their alarm bells go off regarding asymmetric routing.

BAh, yes. The inevitable panic. Right. If you're advertising the same IP from two places, what happens when an internal routing path recalculates or flaps mid session?

AThe traffic suddenly lands at the alternative data center and the stateful firewall sitting in front of that backup data center has never seen the initial TCP SYN packet.

BIt has no idea what this connection is. So it just drops the packet cold as a security violation. Panic ensues.

AWell, that panic is an entirely emotional reaction to a mathematical reality that in many cases does not apply.

BEmotional. All come on. It is. We must dismiss this objection by strictly separating a layer three network phenomenon from a layer four application reality.

AA layer three objection to a layer four reality. Okay. That's a sharp way to phrase it. Walk me through that distinction.

BLook at the specific protocols we have been discussing. This log is transported over UDP. Network time protocol or NTP is UDP DNS queries are predominantly UDP.

AThese are by definition, stateless protocols. Right. There is no TCP handshake. There is no SYN, SYN-ACK, ACK.

BPrecisely. There is no session state to break. A UDP transmission is just a stream of independent datagrams.

AIf a routing path shifts dynamically in the middle of a transmission, the next datagrams simply lands at the alternative site.

BAnd the server doesn't care. The receiving syslog collector does not care that it did not see the previous datagrams.

AThe firewall does not need to maintain a complex state table. There is absolutely no downside for stateless services.

BFor UDP, anycast is a structurally perfect solution. Okay. Stateless makes total sense. UDP over anycast is a home run.

ABut what about TCP? What if I am running a stateful application? Surely that breaks the second the route flaps, right?

BOnce again, we must rely on data rather than anxiety. Even stateful TCP connections ride. Anycast perfectly fine provided you understand the environment.

AWait, really? Yes. The TCP session only breaks if the path physically moves underneath it midstream.

BThe hyperscalers operating on the public internet have to perform incredibly complex engineering feats to maintain state across anycast,

Abecause public internet routes are highly volatile. They flap constantly. Right. But our internal enterprise networks aren't the chaotic public internet.

BExactly. In a properly designed stable enterprise core network, routes do not randomly flap for no reason. The topology is controlled.

AThat's true. If a path changes, it is almost always because a physical failure occurred, or a planned maintenance failover was executed.

BIn those scenarios, the original destination is gone anyway. Oh, right. The server is dead anyway.

AExactly. The TCP sessions are going to reset regardless. And resetting and reestablishing the connection to the new, healthy site is exactly the desired behavior.

BInside a stable enterprise, anycast for stateful services is fundamentally sound.

AThat is a huge paradigm shift. We've been so conditioned to be terrified of asymmetric routing issues that we avoid anycast entirely for internal infrastructure.

BBut you're saying the internal network is stable enough that the boogeyman just isn't real.

AIt is an exaggerated fear that prevents elegant design. However, there is one critical architectural requirement we must address to make this functional.

BOkay. What's the catch? If you simply advertise an IP from two data centers using basic static routes, you introduce a severe systemic risk to your environment.

AI was actually just thinking about this. Let's map this out. Let's say I can figure my core routers to advertise my syslog IP from both my East Coast and West Coast data centers using static routes.

BA dangerous proposition. Right. Because what happens if the physical server hosting the syslog collector on the East Coast completely crashes, but the router in front of it is still powered on?

ADoesn't the network just keep blindly sending traffic straight into a dead end because geographically it's still technically the nearest advertised route?

BYes. You have essentially built a highly efficient mechanism for discarding your own data.

AYes. Utilizing static routes for anycast is a critical error. A static route is a rigid, unyielding promise to the network that a condition will be true forever.

BWhich is never true in IT. Never. If you configure a static route, the router will happily continue advertising that path to the rest of the company regardless of the application's actual health.

AAs McCutcheon astutely notes in the workshop material, if you do this, you did not build redundancy. You simply built a faster way to lose your logs. It will politely hold the door open for traffic long after the data center is on fire.

BWhich completely defeats the purpose. We want the redundancy to be intelligent. We want the route to know when to quit and get out of the way. So how do we fix that? What is the missing mechanism?

AThe solution is what the author terms the second stitch of anycast design. Is a mechanism known as route health injection, or RHI.

BRoute health injection to get breakdown the mechanics of how that works in a real deployment.

AInstead of pointing a static route directly at a server, you place your anycast IP, often referred to as a virtual IP, or VIP behind a load balancer at each site.

BOkay, a load balancer. This is crucial because a load balancer is explicitly designed to monitor the layer 4 and layer 7 health of the underlying services it manages.

AWith route health injection, the load balancer is configured to dynamically inject the route into your core routing protocol, such as OSPF or BGP, but only while the service is actively passing health checks.

BOh, I see the architecture now. The load balancer acts as a like a real time translator between the application's actual health and the network's routing table.

APrecisely, if the syslog service at the East Coast site crashes, the load balancer detects the failure within milliseconds. Because of the route health injection protocol, the load balancer immediately withdraws the route advertisement from the local router.

BWow, so the dead site literally removes itself from the map. It ceases to exist routing-wise.

AThe local router stops advertising the anycast IP to the rest of the company, which means the surrounding network instantly recalculates, realizes the West Coast data center is now the only valid path to that IP, and seamlessly redirects all the traffic across the country.

BYes. And the moment the East Coast server reboots and passes its health check, the load balancer injects the route back in and traffic naturally flows back to the closest point.

AYou have correctly deduced the entire lifecycle. If we synthesize these mechanisms, what is the net result?

BIt's beautiful, is what it is. And anycast prefix combined with route health injection ensures nearest and alive redundancy.

AIt provides automated failover for free, leveraging the native protocols of the network, and just as importantly, it requires zero cross-functional friction.

BRight, because you don't need to involve the server team.

AYou do not need to force your syslog administrators or your DNS teams to learn the intricacies of BGP routing.

BThat is the true beauty of this design. The application teams just get the one single highly available IP address that they wanted in the first place.

AThe load balancer and the network layer handle all the heavy lifting of making sure that IP always points to a living breathing server.

BIt completely decouples the complexity.

AIt abstracts the complexity entirely into the network layer where the mechanisms to handle distance and availability already natively exist.

BI have to say, looking at it through this lens, anycast is an absolute revelation for enterprise network design.

AIt is remarkably efficient. We spend so much time building complex clustering software, configuring heartbeat links across data centers, and wrestling with VRRP setups

Bwhen we could just be using the fundamental geometry of the network to do the exact same thing, but cleaner.

AAnycast isn't just a party trick for hyperscalers. It's an elegant, clean solution that every enterprise engineers should be using to make their infrastructure more resilient.

BI am overwhelmingly positive about applying this.

AA sound and well-supported conclusion. To summarize the core thesis of our deep dive today, your mental map must evolve.

BA route does not have to live in one fixed location.

ANo.

BFurthermore, a route that does not know when to withdraw itself is not true redundancy.

ASame prefix, many origins, nearest healthy one wins.

BIt's just brilliant in its simplicity.

AIt is structurally sound, and while before we conclude our analysis, I will leave you listening with one final, provocative thought to process regarding the future of this architecture.

BOkay. What do you have?

AWe have thoroughly discussed how anycast allows us to seamlessly abstract geographic failure at the network routing layer for data streams like syslog and DNS.

BBut consider the trajectory of our industry. If we can abstract data destinations so cleanly, what other traditionally fixed IT assets might we eventually abstract purely through network layer geometry?

AOh, that's an interesting angle. Like what specifically? Imagine edge compute instances or highly specialized, incredibly expensive AI inference nodes.

BOkay.

AWhat if, instead of relying on heavy, complex application layer load balancing software to find an available GPU for an AI workload, we simply request inference from a single anycast IP?

BWait.

AThe network routing layer utilizing route health injection based not just on up or down status, but on GPU utilization metrics automatically routes your request to the nearest idle compute node bypassing the application layer entirely.

BOh my gosh, network native AI load balancing using BGP to find an available GPU.

AThat is wild to think about, but it makes total sense based on everything we discovered.

BIt is merely the mathematical extension of the principles we have discussed. Your previous mental map was flawed because you believed an x-ray of your network showed a single physical destination for every IP address.

ARight.

BBut as we've proven today, that x-ray machine was broken. The map is not a set of fixed points on a board. It is a fluid geometry of cost, distance, and health. Adjust your map accordingly.