Juniper ex4200 trouble

September 11, 2009

The other day I encountered a problem where the Juniper switch I was working on would no longer allow servers to connect the SAN disk attached to it through the switch. Soft-rebooting of the switch didn’t correct the problem. Instead what it did was allow the servers to see the disk and then it would disappear only to come back again in a yo-yo pattern. 

Opening a ticket with tech support they pulled the revision of code running on the switch which was ver 9.2.R2.15. The tech noted there were some problems with this version of code and needed to confirm. He ran the following console commands through an ssh connection.

- cli

-start shell

-vty fpc0  — that is a zero on the end

- show shim register 0×03000040

This produced the following results:

bit number—->  |3 3 2 2 |2 2 2 2 |2 2 2 2 |1 1 1 1 |1 1 1 1 |1 1 0 0 |0 0 0 0 |0 0 0 0

dev     reg: value   |1 0 9 8 |7 6 5 4 |3 2 1 0 |9 8 7 6 |5 4 3 2 |1 0 9 8 |7 6 5 4 |3 2 1 0

0   3000040: 0       |0 0 0 0 |0 0 0 0 |1 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0
1   3000040: 0       |0 0 0 0 |0 0 0 0 |0 0 1 0 |0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0
2   3000040: 0       |0 0 0 0 |0 0 0 0 |1 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0

The results of this register confirmed his suspicions. He said in the register all items in section 0,1,2 should be zero, which means it is properly handling data passing through it. In this case you have a few "1"’s which means there is corruption and the switch will not work correctly until the code is upgraded to a minimal of version 9.3R4. He also pointed to PSN-2008-12-116 which outlines some of the issues. This PSN is posted below:

After patching was complete all the systems became stable and able to see the disk correctly. We reran the previous command confirming the patch was successful by see all zeros in the register. See the results below.

bit number—->  |3 3 2 2 |2 2 2 2 |2 2 2 2 |1 1 1 1 |1 1 1 1 |1 1 0 0 |0 0 0 0 |0 0 0 0 dev     reg: value   |1 0 9 8 |7 6 5 4 |3 2 1 0 |9 8 7 6 |5 4 3 2 |1 0 9 8 |7 6 5 4 |3 2 1 0

0   3000040: 0       |0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0
1   3000040: 0       |0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0
2   3000040: 0       |0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0

Title
Service Impacting Issues in JUNOS 9.2 Software for EX-Series Switches

Products Affected
EX-Series Switch Platforms Running JUNOS 9.2R2.15 and older are affected

Platforms Affected

EX-series

Revision Number
1

PSN Issue :

On-going testing and customer feedback has identified critical issues with JUNOS 9.2 Software for EX-Series Switches that may impede normal operation of the switch. Solution:

All JUNOS software built on or after November 26, 2008, have been modified to correct the above issues.

• PR 314088  Slot 0 interfaces disappeared on virtual chassis due to internal vcp port stuck in TXQ.
• PR 388285  VC Member looses Membership in the VC under certain conditions.
• PR 390667  (VRRP) Virtual IP next-hop not being added to PFE.
• PR 393895  When a trunk interface is configured as tagged as well as native-vlan for a particular vlan and then native-vlan is deleted, the interface continues to send as untagged.
• PR 397035  eswd core@task_quit, task_terminate, task_signal_kevent after gres and related testing.
• PR 397054  Stack split after rebooting a member of virtual chassis.
• PR 399213  Invalid Next Hop entries in Ethernet-switching table.
• PR 399887  Performing a reboot on a linecard in a virtual chassis may result in a 12-15 second packet loss.

Filed under: Junos, Switches | Comments (0)

How does tracert work (Windows)

August 13, 2009

tracert,ping,hops,ttl,time to live exceeded,routers

Have you wondered how tracert actually works and what all the extra data means it reports back. When using the tracert utility, if you use a host name such as tracert microsoft.com the machine will do a dns lookup first to find the address. In the example below I did a tracert to 4.2.2.1.

What are the first column of numbers

The first row of number represents the hop count and the number of routed networks between you and the host.

What are the three numbers to the right of each hop.

These numbers are the round-trip times to that device to response to your request.  What Windows does is use the TTL field in the IP Header to measure the hops between to the client and server.

Windows starts out sending a ping echo request packet with a TTL of 1 to the destination, when a router encounters the packet and begins to process the request it decriments the TTL by 1. When the TTL reaches 0 the router sends back a Time to live exceeded in transit error message. Windows will then two more packets with the same TTL and records the round trip time for that router.  Next the Windows client will increment the TTL by 1 reaching each router until the packet finds it final destination and the Windows client recievies the echo reply.

Example: tracert in action

 

Example: Packet capture showing the echo requests and replies from the routers

Source Destination Protocol Info
192.168.255.3 4.2.2.1 ICMP Echo (ping) request

First Router
192.168.255.1 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
192.168.255.1 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
192.168.255.1 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)

Second Router
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
96.252.170.1 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
96.252.170.1 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
96.252.170.1 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)

Third Router
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
130.81.105.142 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
130.81.105.142 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
130.81.105.142 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)

Fourth Router
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
130.81.28.233 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
130.81.28.233 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
130.81.28.233 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)

Fifth Router
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
130.81.19.30 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
130.81.19.30 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
130.81.19.30 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)

Sixth Router
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
152.63.86.69 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
152.63.86.69 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
152.63.86.69 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)

Seventh Router
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
152.63.81.145 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
152.63.81.145 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
152.63.81.145 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)

Eighth Router
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
4.68.127.189 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
4.68.127.189 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
4.68.127.189 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)

Nineth Router
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
4.68.103.34 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
4.68.103.2 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
4.68.103.2 192.168.255.3 ICMP Time-to-live exceeded (Time to live exceeded in transit)

Final Destination
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
4.2.2.1 192.168.255.3 ICMP Echo (ping) reply
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
4.2.2.1 192.168.255.3 ICMP Echo (ping) reply
192.168.255.3 4.2.2.1 ICMP Echo (ping) request
4.2.2.1 192.168.255.3 ICMP Echo (ping) reply

What are the names and number at the end of each line.

These are the names and ip addresses of the inside interface of the router the packet is trying to travese. This information is part of a two step process in which the Windows client does a reverse-lookup of the ip address returned during the Time to live exceeded in transit message. Once the client has this information the name and the ip address is displayed on the screen and the client goes on to the next hop.

* What happend I got a “*” and nothing displays after that

If you encounter a router that is configured to silently discard packets with TTLs of 0, the * is displayed in it place. This router may be configured to also filter packets by discarding all ICMP error messages passing through it making the display show the * for all subsiquent hops. If this filter is not in place the display may be fully populated in later hops.

Filed under: General News, TCP/IP | Comments (0)

What is Perl and how do I get started

June 18, 2009

Perl is short for “Practical Extraction and Report Language” and it merges the gap between scripting languages of SED and AWK and languages of C and C++. It has alot of the power of the formal languages but the ability to quickly get started like the scripting languages.

Where do I get Perl

So you have decided to give Perl a try and you need to get it. Here are a few resources to try.
www.activestate.com – This is where you Windows users can get Perl, you can also get it from www.perl.com
If you are using a Linux distro it usually comes packaged on the machine and it is already there. Otherwise you can use you package installer such as Yum ot Yast to install it. Lastly www.perl.com can get you a copy of the tar ball to install.

Now what

You have Perl installed at least you think so, so lets test it out. Go to a prompt or terminal sessions and type “perl –version” you should get some like this below:

This is perl, v5.10.0 built for MSWin32-x86-multi-thread
(with 5 registered patches, see perl -V for more detail)

Copyright 1987-2007, Larry Wall

Binary build 1004 [287188] provided by ActiveState http://www.ActiveState.com
Built Sep  3 2008 13:16:37

If you do not get anything back it just means Perl is not in you path and you will need to find the location of Perl on your machine and execute it from there.

Hello World

Where would we be if we didn’t start off with the hello world program. So, open you favorite text editor and type

print “Hello World!\n”;

Save the file as helloworld.pl next open a prompt or terminal session and find your new file and execute it by typing perl helloworld.pl

Now you have installed and successfully written your first Perl program.

Filed under: Perl, Programming | Comments (1)