The Double-back: A Danger of RSVP Fast Failover

Figure 1 - the link bypass LSP

The Bypass LSP

RSVP spawns bypass LSPs to protect against either a link or a node failure. Figure 1 shows a sample topology snippet. RSVP is configured for link protection. LSP A-to-B transits the path R1-R2-R3 along its path from its source to destination. Since this sample network uses link-protection, router R1 spawns a link-bypass LSP along the path R1-R4-R3-R2 in order to protect any LSPs traversing the R1-R2 link. For the sake of symmetry, we'll also note that R2 spawns a link-bypass LSP along the path R2-R3-R4-R1 to protect LSPs on the R2-R1 link direction.

Traffic

Traffic on each LSP/group of LSPs is as shown on Figure 1. In addition to the A-to-B LSP, there is also traffic in the opposite direction: LSP B-to-A as well as traffic on various LSPs ingressing R2 from the East (right). Some of the traffic ingressing R2 from the East is heading south thru R1, some heading north thru R3.

Figure 2 - protected link failure

R1-R2 Link Fails

Yikes! The R1-R2 link failed. As expected, R1, the point of local repair (PLR), diverts the A-to-B LSP onto the R1-R2 bypass LSP, which takes the traffic on the path thru R1-R4-R3-R2, successfully bypassing the R1-R2 link break (see figure 2), delivering the A-to-B LSP to R2. Once the traffic reaches R2, it continues on its path from R2 to R3, and so on, towards the A-to-B destination. The effective path of the LSP A-to-B traffic in our sample topology segment during the failure is R1-R4-R3-R2-R3. It will remain on that effective path until the A-to-B LSP can resignal and converge around the failure.

At this point you probably see the non-intuitive behavior that sometimes happens when using RSVP bypass LSPs: the bypass LSP's ONLY purpose is to protect the R1-R2 link. It has no understanding of the full path of the LSPs it is protecting, and so has no understanding that it is tunneling the A-to-B LSP to R2 via R3, only to have it egress R2 and double back to R3 as it continues its journey. Again, this is expected, but non-intuitive, behavior. It is natural at this point to perhaps argue that an LSP will detect loops. An LSP will detect loops. The caveat here is that the traffic is taking two LSPs during the outage. From the point of view of the A-to-B LSP (and any other LSPs protected by the R1-R2 bypass LSP), there IS no loop: the A-to-B LSP itself is being tunneled thru the bypass LSP. The next section, along with figures 3 and 4, will examine the label mechanics that allow this LSP tunneling behavior.

Figure 3 - RSVP label mechanics during LSP steady state (pre-failure)

Figure 4 - RSVP label mechanics during time bypass LSP carries traffic

A Quick Review of RSVP LSP Label Mechanics

Before we look at the exact fallout of the protected link failure, here's a quick review of RSVP label mechanics. Figure 3 shows the RSVP label mechanics for LSP A-to-B during steady state over this section of the network, with the label switched path transiting R1, R2, and R3, respectively. Each of those routers performs a label swap, sending the packet along its way. This is why it's referred to a label-switched-path: the path thru the network is defined by a series of hops, with each hop consisting of a label operation or series of label operations that happens at that hop. Take note that the hop at R2 consists of a label swap operation: the label switching router (LSR) reads the 532908 label and swaps it for the 698126 label and sends the label on its way to R3, the next hop.

Figure 4 shows what happens during the failure state for LSP A-to-B, but before RSVP has converged around the failed R1-R2 link. The LSP's traffic is using the bypass LSP to get to R2 since the R1-R2 link is down. R1 is the point of local repair (PLR) because it is the closest router to the failure. As such, it performs the label swap 465987-->532908 operation. Because of the failover of traffic to the bypass LSP, R1 (the PLR) performs an additional, subsequent label operation (push 810945) to get the packet on to the bypass LSP. R1 then sends the packet toward R4. Figure 4 shows that the label stack on the packet on the bypass LSP has a label depth of 2 until it reaches R3. R3 is the penultimate hop for the bypass LSP. As such, R3 performs a label pop operation, removing the top label in the stack (the label for the bypass LSP), and sends the packet to R2 with the label that R1 swapped to: 532908.

Here's why RSVP bypass LSPs are cool: since R3 simply popped the bypass LSP label, the packet arrives at R2 with the exact same label from R1 as it did before the failure. R2 does not care which interface a packet with the 532908 label arrives on. All it cares about is the label value. R2 performs a lookup in its mpls.0 table and sees that the 532908 label is for the A-to-B LSP, performs a swap operation 532908-->698126 and sends the label on its way toward R3.

Figure 5 - LSP mayhem during RSVP convergenge

The Fallout

This condition of bypass LSPs doubling back traffic can persist for as long as it takes for the network to fully converge around the R1-R2 failure. Take a look at figure 5, which shows the effective path that traffic for each LSP takes during the failure: these are not LSP paths, but the effective paths that the traffic on each LSP takes during the failure. Although we've been focusing on the A-to-B LSP, also note what happens when the traffic on the B-to-A LSP (shown in figure 5) in the opposite direction reaches R2, which is the PLR in the opposite direction. R2 performs a similar swap then push label operation as R1 did, putting traffic on the bypass LSP protecting the interface on R2 facing R1, further stressing the R2-R3 link, as traffic doubles back from R2 to R3 en route to R1. Note that the R2 interface facing R3 is now attempting to put 13G of traffic on the wire. The R2-R3 interface is 10G, so it's 130% utilized.

If the network can converge quickly enough, this condition may only last a matter of seconds or less. However, if the network is large/complex enough, convergence may take several minutes. During this time, it is quite understandable that, in the moment, operators would be confused as to why there would be congestion on R2-R3. This doubling back of traffic is not an intuitive behavior of RSVP (although it IS expected). It could be especially impactful if the traffic ingressing R2 from the East is high-value traffic such as Trans-Atlantic traffic, a lot of which would be dropped until the network converges around the failure.

This doubling back of traffic when using bypass LSPs will not always happen: it is typically a product of the network topology. Depending on the overall network architecture, it may be possible to mitigate this problem. Since bypass LSPs don't honor LSP admin-groups/link-coloring or reserve bandwidth, the only options may be to:

• Engineer the path for the bypass LSP by adding strict and/or loose hops to the bypass LSPs for the high-impact areas where this may happen. For an example of how to do this configuration on a bypass LSP on a Junos device, see This Week: Deploying MPLS, page 38
• Design the network topology to prevent this type of doubling back behavior during failure convergence
• Ensure the network can converge quickly around any given failure; with RSVP and the extra state it puts on the network, this is sometimes tough to manage
• Properly plan network capacity to absorb this additional traffic during failover (often easier said than done)
• Accept that this will happen

Conclusion

RSVP fast failover is a useful tool for preserving service during an outage. However, there are some non-intuitive behaviors that can happen during the failure, such as the double-back of traffic described here. Although we have examined this behavior using link-bypass LSPs as the example, this behavior can also happen with node-bypass LSPs. For example, imagine an additional node between R1 and R2.

This behavior with bypass LSPs can cause service disruption, which can defeat the purpose of having bypass LSPs in the first place. Depending on the network planning criteria, commitments to customers, and how long the network takes to resignal the LSP, it may be acceptable for traffic drops during the time the network is converging around the failure. Being aware of this possible behavior can at a minimum help operators understand what is happening during an outage and act accordingly as appropriate for their situation.