Everything in networking has a cost. Sometimes it’s bandwidth. Sometimes it’s processing overhead. With iBGP, the cost is session count, and it gets ugly fast.
If you’ve spent any time with BGP documentation, you’ve heard the phrase “full mesh.” If you haven’t, you’re about to. And once you understand why it’s required and what it actually means at scale, you’ll understand immediately why route reflectors exist and why confederations exist and why people who build large networks have opinions about both.
Why Full Mesh Is Required
Start with the iBGP split horizon rule from the last post. Routes learned from an iBGP peer are never re-advertised to another iBGP peer. This exists to prevent routing loops. It’s a good rule. It also means that if Router A learns a route from Router B via iBGP, Router A will not pass that route along to Router C. Router C will never learn that route unless it has its own direct iBGP session with Router B.
The conclusion is unavoidable. For every router in your AS to have complete routing information, every router needs to peer directly with every other router. That’s a full mesh.
The Math
The formula for the number of iBGP sessions required in a full mesh is n(n-1)/2, where n is the number of iBGP-speaking routers.
Let’s run some numbers.
- 3 routers: 3 sessions. That’s fine.
- 5 routers: 10 sessions. Starting to notice it.
- 10 routers: 45 sessions. You’re configuring this by hand and you hate yourself.
- 20 routers: 190 sessions. You’ve started questioning your career choices.
- 50 routers: 1,225 sessions. You need medication.
- 100 routers: 4,950 sessions. This is not a network, it’s a cry for help.
And this is before you factor in that every one of those sessions needs to be configured on both ends. 1,225 sessions means 2,450 neighbor statements, and you have to get every single one right.
The Operational Reality
Full mesh isn’t just painful to configure. It’s painful to maintain. Every time you add a new router to your AS, you have to add a neighbor statement to every existing iBGP router. And every existing router needs to add a neighbor statement pointing to the new one.
In a 10-router mesh you need to touch all 10 routers to add one new peer. In a 50-router mesh, you’re touching 50 routers. That’s not a deployment procedure, that’s a punishment.
Route changes and session flaps in a large full mesh also create significant churn. Every session that bounces generates update messages to all other peers. With a full mesh, this scales as the square of your router count. At some point the overhead of managing the mesh starts affecting the performance of the network itself.
What Happens When the Mesh Isn’t Actually Full
This is where things get interesting, and by interesting I mean dangerous.
If you set up iBGP between routers A, B, and C, but you only create the sessions A-B and B-C and forget A-C, you have an incomplete mesh. Router A and Router C each have sessions with B, but not with each other. B learns routes from both A and C but due to split horizon it won’t re-advertise A’s routes to C or C’s routes to A.
The result is that A can’t reach destinations known only to C and vice versa. But the sessions are all up. BGP looks healthy. The problem is silent until someone reports broken connectivity and you spend an embarrassing amount of time troubleshooting before you notice the missing session.
Always verify your full mesh is actually full. On Cisco: show bgp summary will show you all your neighbors. Count them and check against your expected mesh.
The Solutions
Two solutions exist for the full mesh scaling problem.
Route reflectors are the more common approach. You designate one or two routers as route reflectors. All other routers peer with the route reflector instead of with each other. The route reflector is allowed to re-advertise routes learned from one iBGP client to other iBGP clients, bypassing the split horizon rule in a controlled way.
Instead of n(n-1)/2 sessions, each client just needs a session to the route reflector. A 50-router network goes from 1,225 sessions down to 50. The route reflector design does introduce its own complexity around cluster IDs, route reflector placement, and redundancy, but it’s manageable.
Confederations split a single AS into sub-autonomous systems. Within the confederation, the sub-ASes run eBGP-like between each other (which does allow re-advertisement), and the full mesh requirement within each sub-AS is limited to just the routers in that sub-AS. Externally, the whole thing still appears as a single AS.
Confederations are used less commonly than route reflectors because they’re harder to design and harder to explain to the next person who has to support the network. But they work and some large carriers use them.
Both solutions are covered in detail in upcoming episodes. For now, the takeaway is simple. Know that full mesh is required, know the math, and know why the alternative designs exist.
The Takeaway
iBGP full mesh is fine for small deployments and lab environments. The moment you start adding routers and it stops being small, you need a scaling strategy before you start building. Add route reflectors from the beginning in anything you expect to grow.
The engineers who get in trouble start with full mesh because it’s simple, let the network grow without a plan, and eventually find themselves redesigning the iBGP topology on a live network. That’s a genuinely unpleasant project.
Design for scale before you need it. Redesigning iBGP on a live network is not a fun afternoon.