Skip to content

Do you know how to spot a failing service?

May 20, 2013

sparc_t5_chip This past weekend we moved one of our offices to a new facility a few miles away.  In the process two of our Solaris SPARC servers that run in a mini-cluster came down for the first time in a l-o-n-g time.  After situating them into their new home and powering them up the primary server wasn’t accepting SSH connections.

It was late Friday night, and after the initial panic that the server wasn’t booting at all (it was – just no SSH), and a few expletives, we realized that we could telnet to it locally.  A connection to the console also showed that the boot screen wasn’t reporting any errors on boot.  So what could it be?  I was checking the logs while another staff member thought to run the

svcs -xv

command and saw several errors with both NFS and SSH. We quickly realized that the NFS service had been mis-configured and gone unnoticed.  And upon the next reboot (many moons later during our move) the errors with NFS kept the SSH daemon from starting too.  Once we cleared the NFS issue we could start SSHd without issue.  Shew!

So the takeaway?

a) We rely on SSH as the primary tool for connecting to most of our Solaris and Linux servers – but don’t forget to try older tools like telnet (and that you can pick the port you want to telnet to)

b) Know how to quickly check for services with an issue.  The ‘svcs -xv’ command quickly helped us find those services that failed to start.   For a really handy cheat sheet on svcs commands see this little PDF Oracle gives out: Oracle Solaris 11 Administrator’s Cheat Sheet for Service Management Facility (SMF)

c) One service, even seemingly unrelated, can keep others from starting properly.  (can anyone explain to me the relationship between NFS and SSH?)

Advertisements

From → smf, Solaris, SPARC, svcs

Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: