Skip to content

My next adventure . . .

Facebook Forest City Data Center

Facebook’s Forest City Data Center

Advertisements

Do you know how to spot a failing service?

sparc_t5_chip This past weekend we moved one of our offices to a new facility a few miles away.  In the process two of our Solaris SPARC servers that run in a mini-cluster came down for the first time in a l-o-n-g time.  After situating them into their new home and powering them up the primary server wasn’t accepting SSH connections.

It was late Friday night, and after the initial panic that the server wasn’t booting at all (it was – just no SSH), and a few expletives, we realized that we could telnet to it locally.  A connection to the console also showed that the boot screen wasn’t reporting any errors on boot.  So what could it be?  I was checking the logs while another staff member thought to run the

svcs -xv

command and saw several errors with both NFS and SSH. We quickly realized that the NFS service had been mis-configured and gone unnoticed.  And upon the next reboot (many moons later during our move) the errors with NFS kept the SSH daemon from starting too.  Once we cleared the NFS issue we could start SSHd without issue.  Shew!

So the takeaway?

a) We rely on SSH as the primary tool for connecting to most of our Solaris and Linux servers – but don’t forget to try older tools like telnet (and that you can pick the port you want to telnet to)

b) Know how to quickly check for services with an issue.  The ‘svcs -xv’ command quickly helped us find those services that failed to start.   For a really handy cheat sheet on svcs commands see this little PDF Oracle gives out: Oracle Solaris 11 Administrator’s Cheat Sheet for Service Management Facility (SMF)

c) One service, even seemingly unrelated, can keep others from starting properly.  (can anyone explain to me the relationship between NFS and SSH?)

RHEL 6 Installation Options

red hat

Okay – Configuring another RHEL 6 KVM Host to throw in the rack alongside all the VMware ESXi hosts. While I was about to start with the basic core server and only install the packages I wanted afterwards, I thought I’d see what all the options were from the RHEL 6 boot DVD.

Took me a bit of searching, but here’s what I found.  These are the options when first booting from the RHEL 6 ISO or DVD and the packages that get installed with each choice:

Desktop: base, basic-desktop, core, debugging, desktop-debugging, desktop-platform, directory-client, fonts, general-desktop, graphical-admin-tools, input-methods, internet-applications, internet-browser, java-platform, legacy-x, network-file-system-client, office-suite, print-client, remote-desktop-clients, server-platform, x11.

Minimal Desktop: base, basic-desktop, core, debugging, desktop-debugging, desktop-platform, directory-client, fonts, input-methods, internet-browser, java-platform, legacy-x, network-file-system-client, print-client, remote-desktop-clients, server-platform, x11.

Minimal: core.

Basic Server: base, console-internet, core, debugging, directory-client, hardware-monitoring, java-platform, large-systems, network-file-system-client, performance, perl-runtime, server-platform.

Database Server: base, console-internet, core, debugging, directory-client, hardware-monitoring, java-platform, large-systems, network-file-system-client, performance, perl-runtime, server-platform, mysql-client, mysql, postgresql-client, postgresql, system-admin-tools.

Web Server: base, console-internet, core, debugging, directory-client, java-platform, mysql-client, network-file-system-client, performance, perl-runtime, php, postgresql-client, server-platform, turbogears, web-server, web-servlet.

Virtual Host: base, console-internet, core, debugging, directory-client, hardware-monitoring, java-platform, large-systems, network-file-system-client, performance, perl-runtime, server-platform, virtualization, virtualization-client, virtualization-platform.

Software Development Workstation: additional-devel, base, basic-desktop, core, debugging, desktop-debugging, desktop-platform, desktop-platform-devel, development, directory-client, eclipse, emacs, fonts, general-desktop, graphical-admin-tools, graphics, input-methods, internet-browser, java-platform, legacy-x, network-file-system-client, performance, perl-runtime, print-client, remote-desktop-clients, server-platform, server-platform-devel, technical-writing, tex, virtualization, virtualization-client, virtualization-platform, x11.

After finding this I think most of the packages I need are indeed in the RHEL Virtual Host installation option (side note: CentOS 6 installation options are identical) is what I’ll start with.  🙂

Alright – back to server config!  Hope this helps someone.

Hmmmmm   .  .  .  next I think I’ll write up how to configure a Kickstart repository   .  .  .

WAN Fail Over & Load Balancing

I love it when things just work.

We have a remote location that runs on three T1 circuits.  It rarely goes down but rarely is not synonymous with “never”.  So a while back I configured a backup cellular data connection.  Not much bandwidth, mind you, but enough of a band-aid to hopefully keep folks working should it ever be needed.  It’s a pretty basic concept – you specify targets to probe over the WAN and rules for what happens when a target fails.  In my case I have two targets set – if they both fail in a row then the router switches to the secondary WAN interface (the Cellular modem).

WAN Probe Failure Alert

WAN Probe Failure Alert

Got the alert one evening at dinner that things seemed a little slow, but were working.  Upon checking the logs on the router I saw that the primary WAN / T1 was failing when trying to probe both targets and that the secondary WAN was now in use.

The cool thing was that once the T1 was fixed 30 minutes later, the router realized it’s probes were succeeding once again and it switched back to the primary WAN.  The rules I set said that the primary targets had to both have successful probes for 3 rounds before switching back.

Nice.

Linux Dual Boot Fedora 18 & Ubuntu 13.04

I wanted to create a dual boot system with Fedora 18 and Ubuntu 13.04 for use as a development desktop (and ya’ know – just because you can).  So I took a Sun Ultra 20 M2 that was collecting dust and crammed it full of RAM and a 1.5TB HDD.  I started with Fedora – install went smoothly, partitioned about half the disk and left the other half for Ubuntu 13.04.

Then it was Ubuntu’s turn.  13.04 went with a snap, easy as pie, just like I expected.  The Ubuntu install was nice and polite – asking if I wanted to install Ubuntu alongside the Fedora 18 Spherical Cow install it found.  “You’ll be able to choose between the operating systems upon startup.”  Great! I thought.  One nice thing is being that these are both Linux systems I can mount the other’s partitions and have access to all my files, no matter what OS is currently running.

But then the issue – once rebooted GRUB 2.0 only presented me with Ubuntu.  Hmmmmmm  :-/  Was Fedora still there?  Dang it, I’m gong to have to go edit GRUB manually I thought.  So I booted into Ubuntu – saw that my Fedora partitions were all still in tact.   Went to refresh myself on editing GRUB and quickly found this command:

sudu update-grub

Really?  Is it that easy?  Last time I created a dual-boot Windows / Ubuntu laptop I had to edit GRUB myself (or so I thought at that time).  So time to give it a try:

update-grub dual boot fedora and ubuntu

update-grub command finding Ubuntu and Fedora!

And presto!  Just like “magic” GRUB was updated to include both Ubuntu and Fedora.  Yeah!

Sun Ray Wed Administration CSS Issue

PROBLEM: You fire up Chrome or Firefox to go to the GUI Admin page on your Sun Ray Server but no servers are listed!  You can just about notice that they get listed really fast and then quickly disappear! Gak!  But if you use Internet Explorer it works just fine.

Sun Ray Servers Missing

Look Ma! No servers!

SOLUTION: A quick little script updates a CSS file and restarts the Sun Ray web admin.  This is for a typical Solaris install – if you are running RHEL, Linux, etc. you may have to update some of the paths:

#!/bin/sh
cd /tmp
unzip /opt/SUNWut/webadmin/webapps/ut/WEB-INF/lib/suntheme.jar com/sun/web/ui/suntheme/css/css_master.css
echo "#f1\3Ap1\3At1 { display:block;}" >> com/sun/web/ui/suntheme/css/css_master.css
zip -r /opt/SUNWut/webadmin/webapps/ut/WEB-INF/lib/suntheme.jar com/sun/web/ui/suntheme/css/css_master.css
rm -rf com
/etc/init.d/utwadmin stop
/etc/init.d/utwadmin start
. . .   and we're back!

. . . and we’re back!

I can’t take credit for the solution – that goes to the Sun Ray User’s mail list.  But I wanted to post it on the web, really for my own reference and also in hopes that someone else finds it more easily.  Every time I need it I have to dig through the mail list archives until I find it again.

Best,
Mike

Solaris Package Management

I needed to install GhostScript on a Solaris 10 SPARC server to support a project where we are moving a customer from printing and snail-mailing paper invoices to e-mail a PDF of the invoice each night.   Since package management in Solaris is a bit different from Linux systems using managers like Yum or Apt I thought I’d throw the output of my efforts to the clipboard and paste them up here with a few notes.  Maybe someone will actually find this helpful.  😉

The  best place to download your packages is Unix Packages – a nice one-stop shop for Solaris.  I put all mine in the directory ~/Downloads/sol10sparc-pkgs.  It’s also worth noting that you’ll need to ‘su’ to root (or have root privileges in order to install most packages.
Once you have your package on your Solaris system it’s time to unzip it:
cd Downloads/sol10sparc-pkgs
gunzip ghostscript-9.05-sol10-sparc-local.gz
Now time to run the pkgadd command to install the software:
pkgadd -d ./ghostscript-9.05-sol10-sparc-local
Terminal output during install:
The following packages are available:
1 SMCghosts ghostscript
(sparc) 9.05

Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [?,??,q]:
Here you can just press <ENTER> and it will default to ‘all’ packages.
Processing package instance <SMCghosts> from 

ghostscript(sparc) 9.05
The Ghostscript Group
Using  as the package base directory.
## Processing package information.
## Processing system information.
5 package pathnames are already properly installed.
## Verifying disk space requirements.
## Checking for conflicts with packages already installed.
## Checking for setuid/setgid programs.

Installing ghostscript as 

## Installing part 1 of 1.
/usr/local/bin/bdftops
----snip----
/usr/local/share/man/man1/wftopfa.1
[ verifying class  ]

Installation of  was successful.

Now let’s just say later on you kinda remember installing Ghost-something on this Solaris server. Ugh! What was it called again? Was it SMCghost . . .  or maybe SUNWghost . . .  dang it. How to find it and see what version it is?!

First, let’s grep for any package with ‘ghost’ in the name using the pkginfo command:

pkginfo | grep ghost

Terminal output:

application SMCghosts ghostscript

Ah ha!  Now we have a name (SMCghosts) and we can see details of it using the -l switch:

pkginfo -l SMCghosts

Terminal output:

PKGINST: SMCghosts
NAME: ghostscript
CATEGORY: application
ARCH: sparc
VERSION: 9.05
BASEDIR: /usr/local
VENDOR: The Ghostscript Group
PSTAMP: Steve Christensen
INSTDATE: Mar 19 2013 15:58
EMAIL: steve@smc.vnet.net
STATUS: completely installed
FILES: 621 installed pathnames
5 shared pathnames
34 directories
36 executables
90499 blocks used (approx)

Now what if we want to remove it to install the latest version?  This calls for the pkgrm command:

NOTE: In Solaris it’s best to remove a package before installing a new version.  Solaris doesn’t like to ‘upgrade’, instead it tries to add the package a second time and will produce a warning.

pkgrm SMCghosts

Terminal output:

The following package is currently installed:
SMCghosts ghostscript
(sparc) 9.05

Do you want to remove this package? [y,n,?,q] y

Press <y> and <ENTER> to begin removal:

## Removing installed package instance 
## Verifying package  dependencies in global zone
## Processing package information.
## Removing pathnames in class 
/usr/local/share/man/man1/wftopfa.1
----snip----
/usr/local/bin/bdftops
/usr/local/bin 
## Updating system information.Removal of  was successful.

DONE!  Please leave a comment if you have any input or questions I can help with!

Cheers,
Mike

Solaris Uptime

Why does it just tickle me to see some outrageous uptime on L’Unix servers?  I was just logged in to a remote Solaris server to help a user.  I didn’t know their username so just hit the “w(ho)” command real quick.  Then noticed the uptime – wow.  (and yes – double checked the date on the machine, just to be sure)

In this specific case it’s a Solaris 10 x86 box:

Solaris 10 x86 server named "gwinnett" uptime

Solaris 10 x86 server named “gwinnett” uptime

621 Days!  I’ve actually seen 4 years before on Solaris 10.  It speaks volumes that you have the ability to do patches and package management while the server stays in production.  Throw in some reliable HVAC and power and they just keep going    . . .    and going   . . .   and going.

To find out information about the system you are logged onto:

uname -a

To check uptime you can just type

uptime

Simple!  Cheers,
Mike

And Then There Were Two

OS X and Chromium OS the only two left standing after the latest Zero Day Initiative competition

Chromium's new logo

Recently, HP’s Zero Day Initiative (ZDI) held the annual Pwn2Own competition as a part of  CanSecWest security conference (March 6-8) in Vancouver, BC.  In addition Google offered up its own prize money to the tune of $3+ million dollars being up for grabs.

In the end, every browser on the Windows platform was compromised, including IE10 on Windows 8, IE9 on Windows 7, along with exploits being found for Chrome on Windows as well as Firefox.  Meanwhile the Unix based OS X with Safari and the Linux-based Chromium OS held strong.  Firefox and Google quickly turned around security patches for their products in about 24 hours (amazingly fast).  Of course there was nothing exploited on the OS X or Chromium operating systems so no bug to patch there.  In the end VUPEN Security took home the largest payout of $250,000 for compromising IE10, Firefox, Adobe Flash, and Java:

 

Raspberry Pi first look

UPS was kind enough to delivery my first Raspberry Pi on Thursday!  Why do I think this little SoC (System on a Chip) computer is going to be like a potato chip?  You can’t have just one.

raspberry Pi

Raspberry Pi ~ one very small and cheap PC!

First up – downloading the latest version of Raspian (Debian 6 for the Raspberry Pi’s ARM1176 processor) and putting the image on an SDHC card.  Download is here: http://www.raspberrypi.org/downloads – I went with the standard “Wheezy” image.

Now to unzip it and dd it over the SD HC card I had selected.  This class 10 16 GB Card seemed to offer the most bang for the buck.  (There are other ways besides dd to get the image onto the SD card).  Assumptions here are:

  1. You know the device address of your SD card.  Instructions here if you do not.
  2. Your Raspian version is 2013-02-09-wheezy.  Adjust as needed.
  3. The file is downloaded into your user’s home folder under ‘Downloads’.  (default for many browsers)
unzip ~/Downloads/2013-02-09-wheezy-raspbian.zip
dd bs=4M if=~/2013-02-09-wheezy-raspbian.img of=/dev/sdb

NOTE: My SD card was located at /dev/sdb.  I’ve seen many located at /dev/sdd.  Make sure to strip off the partition number at the end.  i.e. if you see it a /dev/sdd1 then just use /dev/sdd

It should only take a few minutes to image the SD card.  dd doesn’t give any feedback until it’s done, so be patient.  Once imaged it’s time fire it up!  On first boot the Raspberry Pi will run raspi-config – a nice little utility to configure some settings.

raspi-config runs at first boot. You can run any time later too – just type ‘sudo raspi-config’ whenever you like.

Some suggested step here:

  1. Expand the root file system to fill the SD card.
  2. Set the keyboard layout.
  3. Change the password for the pi user.
  4. Change the locale and timezone.
  5. You can keep the memory split at 64MB if you like.  This is how much memory to take away from the CPU and give to the GPU.
  6. Enable SSH so you can get to it remotely.
  7. Disable the desktop (GUI / X Window) on boot.  You can start this easily from the command line by typing “startx”.
  8. If connected to the internet (ethernet cable and DHCP) let it update.

Once logged in you can also run the update this way:

sudu apt-get update
sudu apt-get upgrade

And all installed software will get updated from the Raspberry Pi repository.  Very cool stuff.

NOTE: To avoid corrupting the Raspian OS on the SD card you should always stop the little guy the right way:

sudo halt

Okay – now . . .  let’s try using the X11 system in OS X (MacBook) to forward the Raspian / Debian desktop over the Mac so we can play without having one too many monitors and keyboard on the desktop:

  1. First step is to enable internet sharing on the MacBook from the WiFi to the Ethernet port.
  2. Configure x11 on the Mac to use full-screen mode
  3. Then ssh over to the Raspberry Pi from the Mac’s terminal:
    ssh -X pi@192.168.2.2

    (You’ll be prompted for the password you set on the RaspPi when you first booted it)

  4. Once on the Raspberry Pi just type:
    lxsession

    And your Raspian desktop should appear on the screen!  w00t!  No extra monitor or keyboard needed!

Raspberry Pi x11 forawrdsing to mac

Raspberry Pi with X11 forwarding to the MacBook

Dr. Chuck at the University of Michigan gives an awesome little tutorial on getting this working (and troubleshooting if it doesn’t work) via YouTube: http://youtu.be/MXi-Tk1Wbpc

Okay – that’s it for today!  Will update once I get more seat time with the little guy   . . .

(Hey!  Idea!  Use QEMU on Ubuntu to boot Raspian)

%d bloggers like this: