A network engineer recently emailed me asking if I knew of any good “complete beginners guides” to MPLS. It was a good email, but not as good as the email I received earlier that morning from a Mr Genie, who got in touch claiming to be able to grant me three wishes, in exchange merely for “rubbing” his “magic lamp”. No need to go into details; let’s just say it ended very successfully for both of us.
Anyway, forget I wrote that: the other mail was from an enterprise engineer who had a good understanding of networking, but didn’t know where to start with understanding what MPLS looks like or when it’s used. I started writing some paragraphs for him, but it quickly dawned on me that my mail could be a blog post. And you know me: as soon as I have an idea, I have to spent 60 straight hours working on it, because I’m smart and normal and healthy and not at all broken.
With that in mind, I’m delighted to answer his email in public, in the hope that perhaps it will be useful to you too. If you’ve heard the phrase MPLS but don’t really know what it is or why you should care – this post is for you. We’re going to start by talking about what a label is, how it’s advertised, and how routers process them. Once you understand that, we’ll then move on to some use cases for labels, to give you a bit of context. Sound good? Then let’s do it!
WHAT IS MPLS?
You might know already that MPLS stands for Michael Peterson’s Lacklustre Sandwiches: a protocol named after a man so uninterested in his craft that you have to stop and wonder why he opened his baguette shop in the first place.
…no, sorry, I’ve just checked my notes, and that’s all lies. Let me try again:
You might know already that MPLS stands for Multi-Protocol Label Switching, and that MPLS is a way of adding “labels” to a packet. But what does that actually mean? Why would we want to add labels? What even is a label? When you really start thinking about it, there’s a lot to unpack!
Let’s start with what an MPLS label looks like. If you know what a VLAN tag looks like then you’re not a million miles away from knowing what an MPLS label is. The labels themselves are basically just numbers, as you can see in the packet capture below. This picture is quite busy, but on line 4 you can see that it says “Label: 299776”.
Notice that in this packet capture, the label is inserted in between the “layer 2” Ethernet header, and the “layer 3” IP header. This is why some people say MPLS works at “layer two and a half”.
These numbers, aka these labels, have meaning to the router receiving the packet. Essentially, the label is like an instruction. The label might tell the router to “send the packet towards a specific router”, or that “this packet is destined for yourself, but it’s part of a specific VPN”. Put a pin in that concept of VPNs, because we’ll come back to it in a bit. In fact, in a moment we’re going to go through a big long list of the reasons why we might add labels to packets. But first:
HOW DO ROUTERS LEARN THESE LABELS?
Well, in the same way that IP is an addressing scheme which needs other protocols to actually advertise the IPs around the place, you can think of MPLS in a similar way: MPLS is a labelling scheme, but we need other protocols to actually advertise these labels around the place.
There are four common ways that we can do this, and you might have heard of some of them. These four protocols work in fairly different ways, and have different use cases. Here’s the ultra-high-level overview of each of them:
LDP stands for Label Distribution Protocol. It’s a very basic protocol, and easy to configure. Traffic will always follow the path that OSPF/IS-IS says is best.
RSVP is the Resource Reservation Protocol. RSVP is much more manual than LDP to configure, but the advantage is that it gives you a phenomenal amount of traffic engineering capabilities – the ability to move your traffic in a different way to what your routing protocol says, for example routing around bandwidth problems, or explicitly avoiding certain links in the network.
Segment Routing, aka SPRING (Source Packet Routing in Networking) is a way of advertising labels within OSPF or IS-IS itself. This is great, because it means you don’t need to run an extra protocol to advertise the labels. It’s an especially popular choice to use with some kind of central controller, which can tell routers in real time which labels to use to get to which destinations. By putting the decision making in a central controller, you can get a truly global view of the network, even between OSPF areas or IS-IS levels.
BGP-Labelled Unicast, as the name suggests, uses BGP! This tends to only be used in some very specific, more advanced scenarios. I wrote about it here, but if you’re reading this post then you’ll probably want to bookmark that post for the future once you’ve got the basics down.
If you choose to progress in your studies further then you’ll learn much more about each of these protocols. And if you check out older posts on my blog, you’ll find lots and lots of posts about RSVP.
By the way, it’s worth saying that we generally don’t care about the specific label that’s used: we let our routers take care of that.
Now that we’ve seen how labels are advertised, let’s go back to the label header again and break down what’s in it. Then we’ll talk about how routers process these labels, and then we’ll be ready to talk about some use cases for them.
THE MPLS HEADER
Let’s look again at that packet capture:
The MPLS header is 4 bytes long (that’s 32 bits, for those of you counting along at home), and contains four pieces of information:
- The label itself (which is 20 bits long, giving us a huge number of potential labels to use)
- A time-to-live (which I prefer to call the “time to die”, but that’s just the goth in me), which does exactly the same as the TTL in an IP packet: it goes down by one after each hop, to make sure that packets don’t loop forever in the network
- The so-called “Experimental Bits” which were indeed experimental at one point in history, but which nowadays are used strictly for Quality of Service markings
- And an extra bit, the “bottom of stack bit“. If this is set to 0 then it means that there’s more labels underneath this one. By contrast, if it’s set to 1 then this label is the bottom of the label stack. It’s very common to have stacks of labels on a packet, and soon we’ll see some reasons for this.
HOW DO ROUTERS PROCESS THESE LABELS?
Whatever protocol we choose to advertise these labels, we can imagine a conversation between two routers where Router B might say to Router A “if you want me to forward a packet to Router Z, attach label 123456 to the packet when you send it to me”. Router B will then program itself to know that if a packet comes in with label 123456 then the packet should always be sent out of a specific interface.
From Router B’s perspective, this outgoing interface may or may not be the “best” path to Router Z. In fact, there’s a chance that Router A worked out the exact hop-by-hop path in advance. What matters is that when Router B sends this packet to Router C, this Router C will also know what to do with the packet based on the label – and so will Router D, and Router E, and so on, all the way to the end destination. As long as there’s a label-switched-path from end to end, then the routers along the way will be able to successfully get the packet from A to Z.
Interestingly, Router B might even change the label to something else as it sends it out. This is actually extremely common: in LDP and RSVP, the labels generally only mean something to the router itself, and each router along the way makes its own decision about what label it wants the previous-hop to use. When the labels change along the way, we call this “swapping” the label. For example, it’s very common for Router A to send a packet to Router B with label 123456, only for Router B to “swap” this for label 345678 before it sends it on to Router C.
(By contrast, some of the labels in Segment Routing are indeed global, so it isn’t a universal rule about MPLS in general!)
One final thing I want you to know before we move on: these labels we’ve talked about so far, the labels that are used to transport a packet across the network, are called transport labels. Can you guess why?
That might seem like a silly thing to point out, but we often talk about transport labels, in contrast to service labels or VPN labels, which are labels that specifically indicate that the packet is part of a VPN, or perhaps an IPv6 packet going across an IPV4 network. These are words that we use to describe the purpose that the labels serve, but just to be clear, the labels all look the same on the wire. There’s nothing in packet captures to indicate if a label is a transport label or a service label. It’s just a cool word we use to describe the function the label is performing.
The absolutely crucial thing to understand in that example above is that when Router B received the packet and forwarded it to Router C, Router B didn’t look at the destination IP at all. Instead, it’s the transport label that told Router B what to do with the packet.
Indeed, you could say that the packet was switched, based on the label. Label-switched, you could say. And because we’re not looking at the underlying destination address, the protocol could be IP, IPv6, or anything else. Multi-protocol, you could say. Multi-Protocol Label-Switching. Mmm! Suddenly it makes sense!
WHAT IS THE POINT IN USING LABELS?
Now, if you’re still new to the world of MPLS, you might be thinking at this point that everything we’ve said so far sounds an awful lot like regular IP routing. If we didn’t have MPLS, then the router would just look at the destination IP address, and send the packet to the next hop. So what’s the point of all this MPLS malarkey? After all, if you’re running a routing protocol like OSPF or IS-IS, or perhaps BGP, then surely the whole point of running these protocols is so that all the routers in your network know how to get to everything?
Indeed, it is. If we’d set up our network in such a way that every router knew about every single IP address in the entire network, then Router B could for sure receive a packet from Router A, look at the destination address, check its own routing table, find the “best” route, and send the packet on its merry way.
But here’s a question: what happens if you don’t want to send everything down the supposedly “best” path?
For example, if everything is being sent down the one “best” path, there’s a chance that link might start to become heavily used, perhaps even maxed out. And this would be a shame if you have other potential paths available that weren’t being used at all. It would be kind of nice to send some selective traffic down the less-best path, to get the most out of our network.
Let’s pause there and think about how we might solve this problem of having one link saturated, when another less-good path exists with bandwidth available.
If you have two or more paths to the destination which are equally “good” from the point of view of your routing protocol, then you can take advantage of something called ECMP, or Equal-Cost Multipath. This allows you to load-balance traffic, and get the most out of your links.
What if the links aren’t equal cost? Well, you could adjust the metrics to make them equal-cost. But this isn’t ideal: they probably have different costs for a reason, after all.
Nowadays we have SD-WAN, which among other things can selectively route traffic down different links based on qualities other than the destination IP address – perhaps instead based on the application itself, or maybe even the source IP address. This is great for branch offices, but doesn’t really scale to Service Provider sized networks with many many gigabits of traffic a second, with tens of thousands of users going to millions of destinations.
You could use a protocol that allows for unequal cost load balancing. The problem here though is that you can’t control which traffic takes the less good path: you’re putting your faith in the hash algorithm of your router. Pity the poor souls who, through no fault of their own, end up randomly having all their traffic going down a slower link than everyone else. Pity the poor application developers who spend weeks trying to work out why sometimes their application suddenly slows down over the network by a factor of ten.
UCMP seems like a nice idea, and it definitely has use cases, but it isn’t as attractive as it might initially seem. If you’re going to send different traffic down paths of different speed, 100% you want to control what goes where. For example, you could take customers that pay more, and send them down the better path. You could send voice down the better path. You could then send best-effort traffic down the less preferred path. It’s one thing to do this using SD-WAN, where you can be very precise about what traffic goes down the less-good link. It’s another to let the hashing algorithm of your router pick unlucky victims seemingly at random. If you’re going to use unequal cost paths, you really need a way of being in control of exactly what traffic goes down which path.
There are other options available within IP, but none of them grow very well. What we need is some kind of scalable, feature-rich protocol that allows us to send traffic any way we choose, rather than always following the “best” path.
This very question was one of the problems that faced service providers, and larger enterprises, a few decades ago. A solution was needed – and if you’re smart, you might have worked out that MPLS was the answer. Give yourself ten points if you worked it out.
SOME USE CASES FOR MPLS
In fact, the more the service provider world thought about it, the more they realised there was other functionality that regular old “destination IP routing” couldn’t give them. As you read this list, you’ll be able to imagine that every single one of these requirements is solved by adding a pre-arranged label to the packet, in such a way that the routers along the way only need to look at the label to decide how to forward the packet, without even looking once at the destination IP address:
- The ability to forward certain traffic along a less optimal path, with the specific intention of freeing up the optimal path for priority traffic.
- The ability to forward based on elements other than either source or destination IP address, for example QoS markings.
- The ability to offer a variety of VPN applications, where routers in the core of the network need to be able to send traffic to a destination even though those core routers don’t know anything about the private IPs in the VPN.
- The ability to connect layer 2 islands which are separated by physical distance, and perhaps separated by a layer 3 network.
- The ability to pre-signal a variety of backup paths that can immediately be used, guaranteeing that the traffic won’t loop because it’s following a set pre-agreed path.
- The ability to engineer traffic to avoid certain links or routers, perhaps in order to perform maintenance on a router in such a way that traffic can gracefully be moved away from it.
- The ability to change the path traffic takes based on how much bandwidth the flows in that path are using.
- The ability to run BGP between our routers at the edge of the network, but not in the core, thus allowing a service provider to place specialist boxes in the core which aren’t running BGP. These boxes can have a smaller routing table, but much faster forwarding capabilities.
Now that you know what a label is, it isn’t difficult to imagine how it would be possible to do all of these things by using labels instead of plain old regular IP. As new paths are created, as failures happen, as bandwidth requirements changes, as new VPNs come online, our label advertising protocol of choice can send new advertisements about new labelled paths. When it comes to automatic backup paths and routing around bandwidth problems, the network almost becomes self-driving!
We mentioned earlier that the great thing about this system of forwarding packets based on a label is that the routers in between the source and destination routers – the transit routers – don’t need to know, or even care, about the IPv4 or IPv6 address on the packet. All they care about is that a packet came into an interface with label X, and it should therefore be sent out of a different interface with label Y, so that the next router along the path will know what to do with it.
We mentioned as well that we have a name for this end-to-end labelled path: we call it an LSP, or Label-Switched Path.
Let’s take a look at such a label-switched path, by bringing up my Famous Ten Router Lab. You’ve surely heard of it: it’s famous, after all. You’ve probably seen it in the new Chanel adverts, or perhaps you saw my Famous Ten Router Lab being “papped” by the press at 2am as it drunkenly fell out of the exclusive Bachelor’s Club. Yes sir: my lab sure does get around on the celebrity circuit.
Check out the fact that an LSP is starting at Router 1, and going all the way to Router 10. In a real network there’d be LSPs all over the place, but here we’re focusing in on just one. The protocol that created it doesn’t massively matter, but for the sake of it, let’s imagine that it was created with RSVP. In Juniper land, creating a new RSVP LSP is one single command:
set protocols mpls label-switched-path R1_TO_R10 to 192.168.1.10
(There’s a few extra commands to get RSVP and MPLS going in general, but you just do them once and then forget about it.)
We give our RSVP LSP a name, we give a destination, and that’s it! There’s much, much more that we can do, but for now let’s keep it simple. The result will be something like this:
It’s worth pointing out that LSPs are only unidirectional, so for traffic to properly work in a real network (or your lab at home) you’d need a second LSP in the reverse direction. But like I say, we’re just focusing on a single LSP for now to keep it simple.
We have some names for the moving parts on this diagram:
- Router 1 and Router 10 are PE routers, or Provider Edge.
- Routers 2, 3, 4 and 5 are all transit routers, and we call these simply P routers, for Provider.
- Router 1 is sometimes called the head-end, and sometimes called the ingress router. Imagine the LSP as being like a tunnel that the traffic is entering, or ingressing.
- Similarly, Router 10 is sometimes called the tail-end, and sometimes called the egress router.
Imagine that R1 and R10 are talking BGP, and that R10 says to R1 “Hey friend! If you want to get to 220.127.116.11/24, send the traffic to me.” Now, if you’re good at BGP then you’ll know that R10 will advertise its loopback as the next-hop for this prefix. And hey, guess what: our LSP goes to Router 10’s loopback!
In that respect, it’s hopefully easy to see why the default behaviour on a Juniper router is to use label-switched-paths to get to stuff that you learn by BGP. All we had to do was configure the LSP to R10’s loopback, and the router itself takes care of actually getting BGP to use this labelled path. Nice!
Routers 2, 3, 4 and 5 don’t need to run BGP. They’re not looking at the IP address at all. They’re just looking at whatever label is on the packet. They receive the packet, look at the label, work out what interface it needs to go out of, and swap the label accordingly.
Congratulations: you now know the very basics of MPLS! Having said that, there’s two things I want to round this post off with. First:
I HEARD THAT MPLS WAS MADE TO SPEED UP ROUTING LOOKUPS?
It’s worth saying that there’s a received wisdom about the history of MPLS which is actually slightly misleading.
The story goes that back in the day, routers were very inefficient at looking up IP addresses. Every packet that came in had to be compared against a large table of IPs, which also happens now of course – but back in those days, these lookups were extremely inefficient, because those devices didn’t have any of the quick hardware lookups and clever indexing that we have today.
This became a bigger and bigger problem as the internet’s routing table grew and grew. And in a way, this was kind of a silly problem considering that most core routers probably only had a very small number of potential physical next hops. Why bother performing all these long and complicated lookups, only to find that the packet can only exit out of one of like five potential interfaces?
The history books tell us that this is why MPLS was invented. By using labels, the lookup process could be drastically sped up, so that you’re just looking at a small label base instead of a huge IP routing table. Excellent! Fantastic! Except: it turns out this isn’t true at all.
As you advance more in your studies, give this excellent episode the Network Collective a watch, where they interview one of the original architects of MPLS who basically says that although early routers certainly suffered from that problem, the issue was mostly fixed by the time MPLS came around because lookups in hardware were starting to become a thing, and indexing was becoming more mature.
MPLS VPNs, AKA MPLS APPLICATIONS
The other thing I want to talk briefly about is the idea of “MPLS VPNs”, because it’s a phrase that sometimes confuses folks, especially our colleagues from the Enterprise world. Unfortunately, the marketing folks have muddied the water here somewhat, and it’s worth ending on this.
Bearing in mind that our transit routers – the routers in between the start and the end of the labelled path – aren’t looking at the destination IP address, it might now be easier to see how we could use MPLS to create VPNs. I’ll explain how this works, but it’s quickly worth saying that if you’re coming from the enterprise or security world, you probably associate a VPN with some kind of encryption, an end-to-end encrypted tunnel that possibly goes over the public internet.
In this case there’s no encryption happening. We don’t really need it most of the time, because the traffic isn’t leaving the service provider network. We can choose to encrypt it, but we leave that to the application itself to make that choice.
Instead, a customer can go to a service provider and say to them “We want to buy 100 WAN links from you for our various offices. But instead of putting our WAN links on the public internet, can you please put them in our own private network, and run it for us”. And the ISP will say: sure thing!
In fact, they’ll take the WAN link and configure a special routing table just for the customer. In the same way that we can take one PC and run many virtual machines on it, you can almost imagine that we’re taking one router and running many virtual routers within it. We still manage it as one single router in terms of configuring it, but we create multiple “virtual” routing tables inside it that don’t talk to each other, one for each customer. The customer has their own unique routing table, which we call a VRF, which stands for Virtual Routing and Forward.
This Virtual Private Network is “private” in that no-one other than the customer and the service provider can “see” the traffic. Customers can’t talk to each other, because they’re in completely separate routing tables!
Now, in the same way that in the example above we had a “BGP free core”, where we only ran BGP at the edge, similarly we have a situation where we want the VPN to be on our edge PE routers, but we don’t want to burden the boxes in the core with being aware of the VPN. So first of all, we’re definitely going to use these labelled paths again. But here’s a question: if R1 sends a packet destined to 10.0.6.9 (as a completely random example), how does R10 know which of the potentially hundreds of VPN customers this prefix belongs to?
That’s where the idea of the label stack comes in: we can add a second label inside the packet that will tell R10 which VPN the packet is a part of. The packet capture would looks like this:
- Ethernet header
- Outer transport label
- Inner VPN label
- IP header
- TCP/UDP etc header
Each hop along the way is just looking at the outer label, because it’s how we transport the packet. Then, when the packet reaches the egress PE, that PE can look at the inner VPN label and work out who it’s really destined to. Very nice indeed!
Here’s the interesting thing: this MPLS label stays in the service provider core. The actual WAN link, from the PE to the customer’s router on site, will be pure IP. You only need labels when you’re going across the service provider network: the traffic isn’t labelled from the service provider to the customer.
It’s kind of ironic that we talk about a customer having an “MPLS link”, when actually there isn’t a single bit of actual MPLS happening on it! Rather, this line is going into an MPLS VPN. And when it’s working right, the inner-workings of this MPLS VPN should be completely abstracted from the customer. When they draw their network diagrams, they can draw the ISP core as one big mysterious cloud.
If someone says they have an MPLS line, what they really mean is that they have line that goes into an MPLS VPN. There’s a few different kinds of MPLS VPN – you might have heard of Layer 3 VPNs, or VPLS (Virtual Private LAN Service), which is a kind of layer 2 VPN. Each of those are topics in themselves, so we’ll end it there – but if you scroll through my website, you’ll find plenty of posts that talk about them in more detail, like this one for example. I hope you have fun browsing through my archive!
[EDIT: And one other thing. Sometimes you’ll hear people say that “SD-WAN is an MPLS killer”. You will have your own opinions on whether this is true or just marketing hype, but it’s important to clarify that when someone says this, they don’t mean MPLS. They mean that SD-WAN is a Layer 3 VPN killer. MPLS itself isn’t going anywhere any time soon. It’s very much the technology that powers pretty much every single service provider in the world, regardless of whether they choose to advertise labels via LDP, RSVP or Segment Routing. If you see people making this mistake, you can now feel very smug indeed as you correct them.]
I hope that helped you to get your head around a few new concepts. Comment if it did, and if you’ve got any questions! And have a browse through my blog if you want to find out more about the wonderful magical world of MPLS.
If you did enjoy this post, you’d make my day if you shared it on your favourite social media of choice. And if you want to keep up to date with my posts, follow me on Twitter or follow me on LinkedIn. They’re both terrible websites, so I recommend unfollowing absolutely everyone apart from me. Yes, I highly recommend this strategy. You will surely achieve great success in life if you unfollow absolutely everyone on social media apart from me, the protagonist of life.
Thank you for reading!