Junos: IS-IS Study Notes, Part 5 – Troubleshooting (For JNCIS-SP and JNCIS-ENT Students)

Welcome to this, the fifth and final part in my JNCIS-level series of posts teaching you the wonderful world of the IS-IS routing protocol.

Oh my, you’ve come so far on this IS-IS journey! To think that an hour ago you barely knew what IS-IS stands for. And yet now, you’re a master!

You’ve already read Part 1 where I introduced you to the basic mechanics and configuration, and where you learned about IS-IS levels.

Then you read Part 2 where we took a deeper dive into IS-IS messages, including the all-important Hello and LSP message types.

And then you read Part 3, where you became a world leading expert in LAN interfaces in IS-IS, including the Designated Intermediate System (DIS), and the pseudonode. You also looked at metrics and wide-metrics.

I bet you even remember reading Part 4, where you learned about IS-IS areas, default routes, and using policies to leak from L2 to L1. You’re so smart now!

This fifth and final part shows you how to verify and troubleshoot your work. Some of it will be new to you. Other bits you will have seen before, but this time we dig into a bit more detail.

Before we begin though, a quick word about safe troubleshooting. Please remember to wear gloves and goggles whenever you attempt to troubleshoot a routing protocol, just in case prefixes fly out of the screen and scold your hands or eyes or face. Also, never forget to take a backup of the entire internet before you begin, just in case you delete the world wide web by mistake. It’s happened before, and it’ll happen again. Finally, always – I repeat, ALWAYS – phone your friends and family to tell them that you love them. That’s not specific to troubleshooting; that’s just general life advice. You’re welcome!

 

THE TOPOLOGY FOR THESE TESTS

This topology is very similar to ones you’ve seen before. The hostnames are subtly different, but there are still ten routers, with everything set to be in the same area of 49.0001.

We have a “core” Level 2 network, with two Level 1 networks connecting to it. Do you remember what happens when routers are in different levels, but you have the same area everywhere? If not, go back and read Part 4, you cheeky scamp!

Notice that vMX1 and vMX6 on the left are entirely in Level 1.

vMX2 and vMX7 have two interfaces in Level 1, and one interface in Level 2.

vMX4 and vMX 9 are interesting, because the link connecting them is in both Level 1 and Level 2 at the same time! This is nice: we have a bit of overlap between the Level 1 and Level 2 networks on the right, which will let us see some cool stuff.

 

SHOW ISIS ADJACENCY

In Part 1 you saw how to configure IS-IS: you just add the interfaces you want to run it on.

You can (and should) additionally turn off Level 1 and Level 2 either on a per-interface basis, or globally. Perhaps you’ll set interfaces to something other than the default broadcast, too.

Let’s remind ourselves of the configuration on vMX_9 (to pick a random example), in set format. I’ve gone ahead and added some of the enhancements you learned about in Part 3, like the reference bandwidth, and only using wide-metrics:

root@vMX_9> show configuration protocols isis | display set
set protocols isis reference-bandwidth 100g
set protocols isis level 2 wide-metrics-only
set protocols isis level 1 wide-metrics-only
set protocols isis interface ge-0/0/0.0 point-to-point
set protocols isis interface ge-0/0/0.0 level 1 disable
set protocols isis interface ge-0/0/1.0 point-to-point
set protocols isis interface ge-0/0/2.0 point-to-point
set protocols isis interface ge-0/0/2.0 level 2 disable
set protocols isis interface lo0.0

There’s nothing new here to you, but notice that one interface explicitly turns off Level 1, another interface explicitly turns off Level 2, and a third interface doesn’t turn off either, which means that it runs both levels at once. That’s the interface facing vMX_4.

All interfaces are also configured to be point-to-point, which means there won’t be a DIS election any more. It’s a very good practice to configure this when it’s appropriate, because why bother creating a pseudonode when you don’t need one?

You also saw the command below in Part 2, which checks that adjacencies are up:

root@vMX_9> show isis adjacency
Interface             System         L State        Hold (secs) SNPA
ge-0/0/0.0            vMX_8          2  Up                   20
ge-0/0/1.0            vMX_4          3  Up                   23
ge-0/0/2.0            vMX_10         1  Up                   24

Interface ge-0/0/0.0 is Level 2, while ge-0/0/2.0 is Level 1.

You’ll remember that we left ge-0/0/1.0 as the default behaviour, which is to try to take part in both levels. Interestingly, that interface is showing as “Level 3”. This immediately tells me that it’s a point to point interface.

How do I know that? Check this out:

  • On broadcast L1/L2 interfaces, IS-IS will form two adjacencies – one for Level 1, and one for Level 2.
  • However, on point-to-point L1/L2 interfaces, both levels are done in a single Hello message, which uses “Level 3” to really mean both levels.

To be clear, there isn’t a special secret Level 3; the number just represents both levels. I’m sick and tired of people claiming that I know about a secret Level 3 that is only open to the shadow government of the world. I’m not a member of it! Please cease your investigations.

To show you what I mean, let’s delete the point-to-point configuration on either end of this interface on vMX_4 and vMX_9:

[edit]
root@vMX_9# delete protocols isis interface ge-0/0/1.0 point-to-point

The result: is that two separate adjacencies are formed. We also now see the “SNPA” of the neighbor, the Sub-Network Point of Attachment (ie the MAC address).

root@vMX_9> show isis adjacency
Interface             System         L State        Hold (secs) SNPA
ge-0/0/0.0            vMX_8          2  Up                   24
ge-0/0/1.0            vMX_4          1  Up                    8  50:0:0:50:0:3
ge-0/0/1.0            vMX_4          2  Up                    7  50:0:0:50:0:3
ge-0/0/2.0            vMX_10         1  Up                   20

 

SHOW ISIS ADJACENCY DETAIL/EXTENSIVE

There are also detailed and extensive versions of that command. Let’s take a look at the detailed version. You’ve seen this before, but you’ve learned more things since then, so it’s worth revisiting.

root@vMX_9> show isis adjacency detail vMX_8
vMX_8
  Interface: ge-0/0/0.0, Level: 2, State: Up, Expires in 23 secs
  Priority: 0, Up/Down transitions: 1, Last transition: 00:16:35 ago
  Circuit type: 2, Speaks: IP, IPv6
  Topologies: Unicast
  Restart capable: Yes, Adjacency advertisement: Advertise
  IP addresses: 10.8.9.8
  IPv6 addresses: fe80::5200:ff:fe4b:2

Notice that I was able to refer to the neighboring router by its hostname. It’s so helpful I could almost cry!

The first line shows you what you already knew: the interface, the level, and the state.

After that you see the “Last transition”, which is very helpful for troubleshooting, along with the “Up/Down transitions”, which is also clearly useful. If it’s flapped recently, or if the up/down count is quickly going up, you might have a problem! You can also see the address families it speaks, and its local addresses.

The extensive version of this command is the same, but it shows you the up/down transitions in more detail, with timestamps.

That output was from a point-to-point interface. A broadcast interface shows the same information, but additionally tells you who has been elected to generate the DIS for the broadcast network:

root@vMX_9> show isis adjacency detail vMX_4
vMX_4
  Interface: ge-0/0/1.0, Level: 1, State: Up, Expires in 8 secs
  Priority: 64, Up/Down transitions: 1, Last transition: 00:09:00 ago
  Circuit type: 3, Speaks: IP, IPv6, MAC address: 50:0:0:50:0:3
  Topologies: Unicast
  Restart capable: Yes, Adjacency advertisement: Advertise
  LAN id: vMX_4.02, IP addresses: 10.4.9.4
  IP6 addresses: fe80::5200:ff:fe50:3

vMX_4
  Interface: ge-0/0/1.0, Level: 2, State: Up, Expires in 7 secs
  Priority: 64, Up/Down transitions: 1, Last transition: 00:09:00 ago
  Circuit type: 3, Speaks: IP, IPv6, MAC address: 50:0:0:50:0:3
  Topologies: Unicast
  Restart capable: Yes, Adjacency advertisement: Advertise
  LAN id: vMX_4.02, IP addresses: 10.4.9.4
  IPv6 addresses: fe80::5200:ff:fe50:3

 

CLEAR ISIS ADJACENCY

This command does what it says! Add the hostname at the end of the command, or alternatively add the word “all” if you’re bloodthirsty.

 

SHOW ISIS INTERFACE

Here’s another useful one: “show isis interface” will show you whether an interface is point-to-point, what levels are enabled on the interface, and what the metric is for each level. See if you can spot which levels are disabled on which interfaces.

root@vMX_9> show isis interface
IS-IS interface database:
Interface             L CirID Level 1 DR        Level 2 DR        L1/L2 Metric
ge-0/0/0.0            2   0x1 Disabled          Point to Point        100/100
ge-0/0/1.0            3   0x1 vMX_4.02          vMX_4.02              100/100
ge-0/0/2.0            1   0x1 Point to Point    Disabled              100/100
lo0.0                 3   0x1 Passive           Passive                 0/0

In this output, every interface has a metric of 100. That’s because they’re 1Gb interfaces, and we configured a reference-bandwidth of 100Gb.

We also set “wide-metrics-only”. Remember that without this, the biggest metric that any one interface can advertise is 63. Do you remember why? If not, Part 3 has your answer my guy!

You can see that Level 1 is explicitly disabled on ge-0/0/0, and Level 2 is explicitly configured on ge-0/0/2. You can also see that both of them are “Point to Point”.

Interface ge-0/0/1 doesn’t explicitly say that it’s a broadcast interface, but it does tell you the name of the DIS on the interface, which implies that it’s broadcast. The fact that there’s a separate DIS for Level 1 and Level 2 tells you that this is an L1/L2 interface.

The CirID is the Circuit ID. Each interface on the router gets given one, to make it unique in the link-state database. 0x01 is shared between the point-to-point interfaces. Broadcast interfaces start at 0x02 and go up from there. It’s not too important; I’m just telling you for the sake of completion.

Cool cool cool! Now, let’s look at the extensive version of the interface output.

You can see below that it tells you pretty much everything you’d ever want to know, including:

  • The MTU (“IIH max size” – remember that the MTU is found by padding the IIH, the Hello message)
  • The CSNP interval (how often the Complete Sequence Number PDU is sent out)
  • The DIS election priority
  • The metric
  • The Hello interval
  • And who has won the DIS election (which is actually listed as the Designated Router, although that’s not strictly it’s name).

I’m deploying this command on the broadcast interface. Read it slowly, and try to spot all those details I just mentioned.

root@vMX_9> show isis interface extensive ge-0/0/1.0
IS-IS interface database:
ge-0/0/1.0
  Index: 334, State: 0x6, Circuit id: 0x1, Circuit type: 3
  LSP interval: 100 ms, CSNP interval: 10 s, Loose Hello padding, IIH max size: 1492
  Adjacency advertisement: Advertise, Layer2-map: Disabled
  Interface Group Holddown Delay: 20 s, remaining: 0 s
  Level 1
    Adjacencies: 1, Priority: 64, Metric: 100
    Hello Interval: 9.000 s, Hold Time: 27 s
    Designated Router: vMX_4.02 (not us)
  Level 2
    Adjacencies: 1, Priority: 64, Metric: 100
    Hello Interval: 9.000 s, Hold Time: 27 s
    Designated Router: vMX_4.02 (not us)

 

SHOW ISIS DATABASE

This is where the fun stuff happens!

It’s in the database that you get to see the LSPs themselves, just like in the screenshots you saw in Part 2.

If you just do “show isis database” then you’ll see a list of every router in each level that your router is taking part in. You can then add “detail” or “extensive” to see the exact IPs and adjacencies that the router is advertising in its LSPs.

Let’s take a look at vMX_9 again, which has a presence in both the Level 2 backbone and in the Level 1 network on the right of the topology:

root@vMX_9> show isis database
IS-IS level 1 link-state database:
LSP ID                      Sequence Checksum Lifetime Attributes
vMX_4.00-00                      0x3   0x1e16      383 L1 L2
vMX_4.02-00                      0x1   0x1aa2      383 L1 L2
vMX_5.00-00                      0x4   0xc2d2     1137 L1 L2
vMX_9.00-00                      0x3   0xaf29      389 L1 L2
vMX_10.00-00                     0x4   0xf724     1175 L1 L2
  5 LSPs

IS-IS level 2 link-state database:
LSP ID                      Sequence Checksum Lifetime Attributes
vMX_2.00-00                      0x5   0x1226     1073 L1 L2
vMX_3.00-00                      0x5   0xe2e8     1156 L1 L2
vMX_4.00-00                      0x6   0x66ab     1077 L1 L2
vMX_4.02-00                      0x2   0x18a3     1169 L1 L2
vMX_7.00-00                      0x5   0x539e     1131 L1 L2
vMX_8.00-00                      0x6   0xa3ae     1158 L1 L2
vMX_9.00-00                      0x6   0xf8cd      389 L1 L2
  7 LSPs

You can easily see that Routers 2, 3, 4, 7, 8 and 9 are all in the backbone (Level 2), while routers 4, 5, 9 and 10 are in the Level 1 network. Another way of saying it is that vMX_4, 5, 9 and 10 have each generated a Level 1 LSP.

In addition, vMX_4 is responsible for generating the pseudonode for the broadcast interface between itself and vMX_9, because we removed the “point-to-point” configuration on those interfaces. This happens separately for Level 1 and for Level 2, which makes sense when you remember that they’re separate topologies. This is what the “vMX_4.02-00” LSP is for: it’s the pseudonode for that broadcast segment.

Let’s compare that output with a router that is only in Level 1. What does its database look like?

root@vMX_1> show isis database
IS-IS level 1 link-state database:
LSP ID                      Sequence Checksum Lifetime Attributes
vMX_1.00-00                      0x3   0xfafc      829 L1 L2
vMX_2.00-00                      0x3   0xe701      803 L1 L2
vMX_6.00-00                      0x3   0x4b61      827 L1 L2
vMX_7.00-00                      0x3   0x8d10      799 L1 L2
  4 LSPs

IS-IS level 2 link-state database:
LSP ID                      Sequence Checksum Lifetime Attributes
vMX_1.00-00                      0x4   0xf044      829 L1 L2
  1 LSPs

That’s interesting. It has a Level 1 database of four routers, which is correct – but it’s also generated an LSP for itself in Level 2, even though it doesn’t have any Level 2 adjacencies! This is because we didn’t turn off Level 2 globally on that router.

If a device is entirely in Level 1, it’s worth typing “set protocols isis level 2 disable” to turn the level off completely. This will also stop accidental Level 2 adjacencies forming, and is a lot easier than doing it on an interface-by-interface basis. The same goes for Level 2 routers, by turning off Level 1 globally.

 

SHOW ISIS DATABASE DETAIL

If you add the “detail” flag, you’ll see all the prefixes within that one LSP, and all of the routers that the router is connected to.

Staying on vMX_9, let’s look at vMX_8’s Level 2 LSP. You can do this on any router in the level, because each router should have exactly the same information about every other router that it knows about. Notice we don’t specify Level 2 in the CLI command, because vMX_8 is only in Level 2. But we could add it on if we wanted to be specific.

root@vMX_9> show isis database vMX_8 detail
IS-IS level 1 link-state database:

IS-IS level 2 link-state database:
vMX_8.00-00 Sequence: 0x6, Checksum: 0xa3ae, Lifetime: 907 secs
   IS neighbor: vMX_3.00                      Metric:      100
   IS neighbor: vMX_7.00                      Metric:      100
   IS neighbor: vMX_9.00                      Metric:      100
   IP prefix: 10.3.8.0/24                     Metric:      100 Internal Up
   IP prefix: 10.7.8.0/24                     Metric:      100 Internal Up
   IP prefix: 10.8.9.0/24                     Metric:      100 Internal Up
   IP prefix: 192.168.1.8/32                  Metric:        0 Internal Up
   V6 prefix: 2001:db8::8/128                 Metric:        0 Internal Up

If you’re only used to reading the OSPF database, then your jaw will be at the floor right now from how easy this is to read! You can immediately see the three routers that vMX_8 has a connection to, listed by name. You can also see the four IPv4 and the one IPv6 address connected to it.

There’s also an extensive version of this command, which I won’t show here because it’s massive! But in short, the extensive version of the command will show you everything in that router’s LSP, including what areas it takes part in, its ISO address, timers and lifetimes and checksums. Also, if you’re running MPLS traffic engineering then the extensive command will show you all the RSVP bandwidth reservations at each priority level.

By the way, you can also see that all five prefixes are “Up”. What does that mean? You might be tempted to think that it means they’re working and operational, but in fact it means something else – it’s used to stop routing loops that can occur when IPs are advertised between levels.

 

A QUICK NOTE ON THE DOWN BIT

Each advertisement comes with something called the Distribution bit, otherwise known as the Up/Down bit, or the Down bit for short.

If ever a prefix is advertised from Level 2 into Level 1, this bit is set as Down. To quote the RFC: “If a prefix is advertised from a higher level to a lower level (e.g., level 2 to level 1), the bit MUST be set to 1, indicating that the prefix has traveled down the hierarchy.”

In Part 3 I added a default route on vMX_9, and advertised it into Level 1. Why didn’t vMX_4 redistribute it into Level 2? You may be tempted to think that it’s because the Down bit was set. Interestingly though, in my labbing I found that the down bit was NOT set!

I wanted to make sure I wasn’t going mad, so I made a second aggregate route for 192.0.0.0/8 on vMX_9, and also leaked this just into Level 1. I was surprised to see that the down bit WAS set for that route! So it seems like there’s some special rule that says “Don’t advertise a default route from Level 1 to Level 2.” I couldn’t find any documentation on it – but the labbing doesn’t lie!

On vMX_10 I am looking at vMX_9’s LSP. Notice that the 0/0 route is “Up”, while the 192/8 route is “Down”.

root@vMX_10> show isis database vMX_9 detail
IS-IS level 1 link-state database:

vMX_9.00-00 Sequence: 0x33, Checksum: 0x47a8, Lifetime: 1192 secs
   IS neighbor: vMX_4.02                      Metric:      100
   IS neighbor: vMX_10.00                     Metric:      100
   IP prefix: 0.0.0.0/0                       Metric:       10 Internal Up
   IP prefix: 10.4.9.0/24                     Metric:      100 Internal Up
   IP prefix: 10.9.10.0/24                    Metric:      100 Internal Up
   IP prefix: 192.0.0.0/8                     Metric:      210 Internal Down
   IP prefix: 192.168.1.9/32                  Metric:        0 Internal Up
   V6 prefix: 2001:db8::9/128                 Metric:        0 Internal Up

IS-IS level 2 link-state database:

Quirky bit of fun there! In any case, it’s the Down bit that makes sure that L2-to-L1 stuff doesn’t get re-advertised from L1-to-L2.

 

SHOW ISIS STATISTICS

You’ll only need to know this one if something has seriously gone wrong – it gives you a breakdown of each PDU the router has sent and received, and whether it was processed or dropped. You’ll also see the number of times that SPF has been run. You’ll even see the number of times an LSP has been regenerated, due to nearing the end of its lifetime.

root@vMX_9> show isis statistics
IS-IS statistics for vMX_9:
PDU type       Received    Processed        Drops         Sent       Rexmit
LSP                  43           43            0           44            0
IIH                1941           38            0          986            0
CSNP                769          769            0          374            0
PSNP                 18           18            0           15            0
Unknown               0            0            0            0            0
Totals             2771          868            0         1419            0

Total packets received: 2771 Sent: 1419

SNP queue length: 0 Drops: 0
LSP queue length: 0 Drops: 0
SPF runs: 23
Fragments rebuilt: 13
LSP regenerations: 4
Purges initiated: 0

 

SHOW ISIS OVERVIEW

And one final chill command which does exactly what it says on the tin. You’ll find the System ID, area, SPF timer info, address families, and so much more here.

root@vMX_9> show isis overview
Instance: master
  Router ID: 192.168.1.9
  Hostname: vMX_9
  Sysid: 0000.0000.0009
  Areaid: 49.0001
  Adjacency holddown: enabled
  Maximum Areas: 3
  LSP life time: 1200
  Reference bandwidth: 100000000000
  Attached bit evaluation: enabled
  SPF delay: 200 msec, SPF holddown: 5000 msec, SPF rapid runs: 3
  IPv4 is enabled, IPv6 is enabled
  Traffic engineering: enabled
  Restart: Disabled
    Helper mode: Enabled
  Layer2-map: Disabled
  Source Packet Routing (SPRING): Disabled
  Post Convergence Backup: Disabled
  Level 1
    Internal route preference: 15
    External route preference: 160
    Prefix export count: 0
    Wide metrics are enabled
  Level 2
    Internal route preference: 18
    External route preference: 165
    Prefix export count: 0
    Wide metrics are enabled

 

THAT’S IT – FOR NOW!

If you’ve read all five of these posts, then let me say: CONGRATULATIONS! You have truly levelled up over the past… however long it took you. Seriously, well done.

Did you like it? What do you think of IS-IS as a protocol now? Do you prefer it to OSPF? Or do you like both the same? Let me know in the comments. I’d love to hear your thoughts.

You and me have come a LONG way over these past five posts. We’ve learned a lot together, been through so much. I dare say that in the process we’ve becomes best friends, brothers, sisters, perhaps even lovers. Well, sad news: our love is now over, because you’ve come to the end of the series!

There’s plenty more to cover in IS-IS. We’ve not touched Graceful Restart, or authentication, or overloading, or the unique properties of redistributing external prefixes into a Level 1 network. Hopefully though, these four posts have given you enough confidence to go Googling for more information, and to start adding IS-IS into your labs. I hope they’ve given you a fun introduction to this great protocol.

If you read these posts in order to go for the JNCIS-ENT or JNCIS-SP, the let me wish you the very best of luck with your studies towards this awesome certification!

If you’re on Mastodon, follow me to find out when I make new posts. (2024 edit: I’m also on BlueSky nowadays too. I was once on Twitter, but I’ve given up on it, on account of… well, I don’t need to finish that sentence, do I.)

And if you fancy some more learning, take a look through my other posts. I’ve got plenty of cool new networking knowledge for you on this website, especially covering Juniper tech and service provider goodness.

It’s all free for you, although I’ll never say no to a donation. This website is 100% a non-profit endeavour, in fact it costs me money to run. I don’t mind that one bit, but it would be cool if I could break even on the web hosting, and the licenses I buy to bring you this sweet sweet content.

 

PS: As it happens, I have written a few more posts on IS-IS, which you might like if you’re still hungry for more. Give this post a read where I explain how leaving everything to the L1/L2 default can lead to some seriously disastrous consequences! Then, how about this one where I show you how to use routing policy to remove point-to-point IP addresses from your routing table, to make things more scalable. I hope you enjoy them!

3 thoughts on “Junos: IS-IS Study Notes, Part 5 – Troubleshooting (For JNCIS-SP and JNCIS-ENT Students)

  • May 24, 2023 at 12:28 pm
    Permalink

    The crash course on IS-IS was fun and informative, Thank you I can work networks running with IS-IS protocols with bit more confidence.

    Reply
    • May 26, 2023 at 12:11 pm
      Permalink

      Thank you very much for your kind words Nirmal! Really happy to hear that I was able to help you. 🙂

      Reply
  • June 2, 2023 at 5:45 pm
    Permalink

    Thank you for this IS-IS series. It has helped me immensely and comparing OSPF and writing this helped me understand IS-IS in a better way. Loved the way you conveyed it. Cheers!

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *