February 27, 2012

Compilation and Packaging of Apache 2.4 on Solaris 10

Wow, it had been an age since I posted here. Ooops. This isn’t because I’ve not been doing interesting things, more just a matter of failing to find the time. No excuses, this is bad. I resolve to blog more often (which isn’t going to be hard given the recent record).

Anyway, on with the plot, this blog details my recent adventures compiling Apache 2.4 on Solaris 10. I also include steps for compiling OpenSSL, rather than relying on the libraries shipped with Solaris 10. The Update 9 release of Solaris 10 I’m using include OpenSSL version 0.9.7d, but by deciding to compile my own I move up to version 1.0.0g.

Following the succesful compilation, I cover the creation of an SVR4 package or two for the new software. I hope to follow this article with a version for Solaris 11 and IPS.

The first step, is of course to download the source. The Apache 2.4 source is available here:

http://httpd.apache.org/download.cgi#apache24

Once downloaded, unpack into a working directory of choice, mine was /var/tmp/httpd-2.4.1. There are many options for configuring and compiling Apache and I won’t regurgitate the README or the Documentation here, suffice to say you do need to do your own background reading and choose the options most applicable to your needs.

Further, I should mention prerequisites; the compiler itself, of course. I’m using the Sun Studio compilers, installed in my environment into /opt/studio, This means, as a minimum I need to include /opt/studio/bin and /usr/ccs/bin in my PATH:

export PATH=/usr/sbin:/usr/bin:/opt/studio/bin:/usr/ccs/bin

This should result in access to the ‘cc’ and ‘make’ commands as follows;

bash-3.00# which cc
/opt/studio/bin/cc
bash-3.00# which make
/usr/ccs/bin/make

Once you are set with the compiler you need to get back to thinking about your configure options. There are many;

bash-3.00# ./configure --help 
`configure' configures this package to adapt to many kinds of systems.

Usage: ./configure [OPTION]... [VAR=VALUE]...

To assign environment variables (e.g., CC, CFLAGS...), specify them as
VAR=VALUE.  See below for descriptions of some of the useful variables.

Defaults for the options are specified in brackets.

Configuration:
  -h, --help              display this help and exit
      --help=short        display options specific to this package
      --help=recursive    display the short help of all the included packages
  -V, --version           display version information and exit
  -q, --quiet, --silent   do not print `checking ...' messages
      --cache-file=FILE   cache test results in FILE [disabled]
  -C, --config-cache      alias for `--cache-file=config.cache'
  -n, --no-create         do not create output files
      --srcdir=DIR        find the sources in DIR [configure dir or `..']

Installation directories:
  --prefix=PREFIX         install architecture-independent files in PREFIX
                          [/usr/local/apache2]
  --exec-prefix=EPREFIX   install architecture-dependent files in EPREFIX
                          [PREFIX]

By default, `make install' will install all the files in
`/usr/local/apache2/bin', `/usr/local/apache2/lib' etc.  You can specify
an installation prefix other than `/usr/local/apache2' using `--prefix',
for instance `--prefix=$HOME'.

For better control, use the options below.

Fine tuning of the installation directories:
  --bindir=DIR            user executables [EPREFIX/bin]
  --sbindir=DIR           system admin executables [EPREFIX/sbin]
  --libexecdir=DIR        program executables [EPREFIX/libexec]
  --sysconfdir=DIR        read-only single-machine data [PREFIX/etc]
  --sharedstatedir=DIR    modifiable architecture-independent data [PREFIX/com]
  --localstatedir=DIR     modifiable single-machine data [PREFIX/var]
  --libdir=DIR            object code libraries [EPREFIX/lib]
  --includedir=DIR        C header files [PREFIX/include]
  --oldincludedir=DIR     C header files for non-gcc [/usr/include]
  --datarootdir=DIR       read-only arch.-independent data root [PREFIX/share]
  --datadir=DIR           read-only architecture-independent data [DATAROOTDIR]
  --infodir=DIR           info documentation [DATAROOTDIR/info]
  --localedir=DIR         locale-dependent data [DATAROOTDIR/locale]
  --mandir=DIR            man documentation [DATAROOTDIR/man]
  --docdir=DIR            documentation root [DATAROOTDIR/doc/PACKAGE]
  --htmldir=DIR           html documentation [DOCDIR]
  --dvidir=DIR            dvi documentation [DOCDIR]
  --pdfdir=DIR            pdf documentation [DOCDIR]
  --psdir=DIR             ps documentation [DOCDIR]

System types:
  --build=BUILD     configure for building on BUILD [guessed]
  --host=HOST       cross-compile to build programs to run on HOST [BUILD]
  --target=TARGET   configure for building compilers for TARGET [HOST]

Optional Features:
  --disable-option-checking  ignore unrecognized --enable/--with options
  --disable-FEATURE       do not include FEATURE (same as --enable-FEATURE=no)
  --enable-FEATURE[=ARG]  include FEATURE [ARG=yes]
  --enable-layout=LAYOUT
  --enable-dtrace         Enable DTrace probes
  --enable-hook-probes    Enable APR hook probes
  --enable-exception-hook Enable fatal exception hook
  --enable-load-all-modules
                          Load all modules
  --enable-maintainer-mode
                          Turn on debugging and compile time warnings and load
                          all compiled modules
  --enable-debugger-mode  Turn on debugging and compile time warnings and turn
                          off optimization
  --enable-pie            Build httpd as a Position Independent Executable
  --enable-modules=MODULE-LIST
                          Space-separated list of modules to enable | "all" |
                          "most" | "few" | "none" | "reallyall" 
  --enable-mods-shared=MODULE-LIST
                          Space-separated list of shared modules to enable |
                          "all" | "most" | "few" | "reallyall" 
  --enable-mods-static=MODULE-LIST
                          Space-separated list of static modules to enable |
                          "all" | "most" | "few" | "reallyall" 
  --disable-authn-file    file-based authentication control
  --enable-authn-dbm      DBM-based authentication control
  --enable-authn-anon     anonymous user authentication control
  --enable-authn-dbd      SQL-based authentication control
  --enable-authn-socache  Cached authentication control
  --disable-authn-core    core authentication module
  --disable-authz-host    host-based authorization control
  --disable-authz-groupfile
                          'require group' authorization control
  --disable-authz-user    'require user' authorization control
  --enable-authz-dbm      DBM-based authorization control
  --enable-authz-owner    'require file-owner' authorization control
  --enable-authz-dbd      SQL based authorization and Login/Session support
  --disable-authz-core    core authorization provider vector module
  --enable-authnz-ldap    LDAP based authentication
  --disable-access-compat mod_access compatibility
  --disable-auth-basic    basic authentication
  --enable-auth-form      form authentication
  --enable-auth-digest    RFC2617 Digest authentication
  --enable-allowmethods   restrict allowed HTTP methods
  --enable-isapi          isapi extension support
  --enable-file-cache     File cache
  --enable-cache          dynamic file caching. At least one storage
                          management module (e.g. mod_cache_disk) is also
                          necessary.
  --enable-cache-disk     disk caching module
  --enable-socache-shmcb  shmcb small object cache provider
  --enable-socache-dbm    dbm small object cache provider
  --enable-socache-memcache
                          memcache small object cache provider
  --enable-socache-dc     distcache small object cache provider
  --enable-so             DSO capability. This module will be automatically
                          enabled unless you build all modules statically.
  --enable-watchdog       Watchdog module
  --enable-dbd            Apache DBD Framework
  --enable-bucketeer      buckets manipulation filter. Useful only for
                          developers and testing purposes.
  --enable-dumpio         I/O dump filter
  --enable-echo           ECHO server
  --enable-example-hooks  Example hook callback handler module
  --enable-case-filter    Example uppercase conversion filter
  --enable-case-filter-in Example uppercase conversion input filter
  --enable-example-ipc    Example of shared memory and mutex usage
  --enable-buffer         Filter Buffering
  --enable-data           RFC2397 data encoder
  --enable-ratelimit      Output Bandwidth Limiting
  --disable-reqtimeout    Limit time waiting for request from client
  --enable-ext-filter     external filter module
  --enable-request        Request Body Filtering
  --enable-include        Server Side Includes
  --disable-filter        Smart Filtering
  --enable-reflector      Reflect request through the output filter stack
  --enable-substitute     response content rewrite-like filtering
  --enable-sed            filter request and/or response bodies through sed
  --disable-charset-lite  character set translation. Enabled by default only
                          on EBCDIC systems.
  --enable-charset-lite   character set translation. Enabled by default only
                          on EBCDIC systems.
  --enable-deflate        Deflate transfer encoding support
  --enable-xml2enc        i18n support for markup filters
  --enable-proxy-html     Fix HTML Links in a Reverse Proxy
  --enable-http           HTTP protocol handling. The http module is a basic
                          one that enables the server to function as an HTTP
                          server. It is only useful to disable it if you want
                          to use another protocol module instead. Don't
                          disable this module unless you are really sure what
                          you are doing. Note: This module will always be
                          linked statically.
  --disable-mime          mapping of file-extension to MIME. Disabling this
                          module is normally not recommended.
  --enable-ldap           LDAP caching and connection pooling services
  --disable-log-config    logging configuration. You won't be able to log
                          requests to the server without this module.
  --enable-log-debug      configurable debug logging
  --enable-log-forensic   forensic logging
  --enable-logio          input and output logging
  --enable-lua            Apache Lua Framework
  --enable-luajit         Enable LuaJit Support
  --disable-env           clearing/setting of ENV vars
  --enable-mime-magic     automagically determining MIME type
  --enable-cern-meta      CERN-type meta files
  --enable-expires        Expires header control
  --disable-headers       HTTP header control
  --enable-ident          RFC 1413 identity check
  --enable-usertrack      user-session tracking
  --enable-unique-id      per-request unique ids
  --disable-setenvif      basing ENV vars on headers
  --disable-version       determining httpd version in config files
  --enable-remoteip       translate header contents to an apparent client
                          remote_ip
  --enable-proxy          Apache proxy module
  --enable-proxy-connect  Apache proxy CONNECT module. Requires and is enabled
                          by --enable-proxy.
  --enable-proxy-ftp      Apache proxy FTP module. Requires and is enabled by
                          --enable-proxy.
  --enable-proxy-http     Apache proxy HTTP module. Requires and is enabled by
                          --enable-proxy.
  --enable-proxy-fcgi     Apache proxy FastCGI module. Requires and is enabled
                          by --enable-proxy.
  --enable-proxy-scgi     Apache proxy SCGI module. Requires and is enabled by
                          --enable-proxy.
  --enable-proxy-fdpass   Apache proxy to Unix Daemon Socket module. Requires
                          --enable-proxy.
  --enable-proxy-ajp      Apache proxy AJP module. Requires and is enabled by
                          --enable-proxy.
  --enable-proxy-balancer Apache proxy BALANCER module. Requires and is
                          enabled by --enable-proxy.
  --enable-proxy-express  mass reverse-proxy module. Requires --enable-proxy.
  --enable-session        session module
  --enable-session-cookie session cookie module
  --enable-session-crypto session crypto module
  --enable-session-dbd    session dbd module
  --enable-slotmem-shm    slotmem provider that uses shared memory
  --enable-slotmem-plain  slotmem provider that uses plain memory
  --enable-ssl            SSL/TLS support (mod_ssl)
  --enable-optional-hook-export
                          example optional hook exporter
  --enable-optional-hook-import
                          example optional hook importer
  --enable-optional-fn-import
                          example optional function importer
  --enable-optional-fn-export
                          example optional function exporter
  --enable-dialup         rate limits static files to dialup modem speeds
  --enable-static-support Build a statically linked version of the support
                          binaries
  --enable-static-htpasswd
                          Build a statically linked version of htpasswd
  --enable-static-htdigest
                          Build a statically linked version of htdigest
  --enable-static-rotatelogs
                          Build a statically linked version of rotatelogs
  --enable-static-logresolve
                          Build a statically linked version of logresolve
  --enable-static-htdbm   Build a statically linked version of htdbm
  --enable-static-ab      Build a statically linked version of ab
  --enable-static-checkgid
                          Build a statically linked version of checkgid
  --enable-static-htcacheclean
                          Build a statically linked version of htcacheclean
  --enable-static-httxt2dbm
                          Build a statically linked version of httxt2dbm
  --enable-static-fcgistarter
                          Build a statically linked version of fcgistarter
  --enable-lbmethod-byrequests
                          Apache proxy Load balancing by request counting
  --enable-lbmethod-bytraffic
                          Apache proxy Load balancing by traffic counting
  --enable-lbmethod-bybusyness
                          Apache proxy Load balancing by busyness
  --enable-lbmethod-heartbeat
                          Apache proxy Load balancing from Heartbeats
  --enable-mpms-shared=MPM-LIST
                          Space-separated list of MPM modules to enable for
                          dynamic loading. MPM-LIST=list | "all" 
  --enable-unixd          unix specific support
  --enable-privileges     Per-virtualhost Unix UserIDs and enhanced security
                          for Solaris
  --enable-heartbeat      Generates Heartbeats
  --enable-heartmonitor   Collects Heartbeats
  --enable-dav            WebDAV protocol handling. --enable-dav also enables
                          mod_dav_fs
  --disable-status        process/thread monitoring
  --disable-autoindex     directory listing
  --enable-asis           as-is filetypes
  --enable-info           server information
  --enable-suexec         set uid and gid for spawned processes
  --enable-cgid           CGI scripts. Enabled by default with threaded MPMs
  --enable-cgi            CGI scripts. Enabled by default with non-threaded
                          MPMs
  --enable-dav-fs         DAV provider for the filesystem. --enable-dav also
                          enables mod_dav_fs.
  --enable-dav-lock       DAV provider for generic locking
  --enable-vhost-alias    mass virtual hosting module
  --enable-negotiation    content negotiation
  --disable-dir           directory request handling
  --enable-imagemap       server-side imagemaps
  --enable-actions        Action triggering on requests
  --enable-speling        correct common URL misspellings
  --enable-userdir        mapping of requests to user-specific directories
  --disable-alias         mapping of requests to different filesystem parts
  --enable-rewrite        rule based URL manipulation
  --enable-v4-mapped      Allow IPv6 sockets to handle IPv4 connections

Optional Packages:
  --with-PACKAGE[=ARG]    use PACKAGE [ARG=yes]
  --without-PACKAGE       do not use PACKAGE (same as --with-PACKAGE=no)
  --with-included-apr     Use bundled copies of APR/APR-Util
  --with-apr=PATH         prefix for installed APR or the full path to
                             apr-config
  --with-apr-util=PATH    prefix for installed APU or the full path to
                             apu-config
  --with-pcre=PATH        Use external PCRE library
  --with-port=PORT        Port on which to listen (default is 80)
  --with-sslport=SSLPORT  Port on which to securelisten (default is 443)
  --with-distcache=DIR    Distcache installation directory
  --with-z=DIR            use a specific zlib library
  --with-libxml2          location for libxml2
  --with-lua=PATH         Path to the Lua 5.1 prefix
  --with-ssl=DIR          OpenSSL base directory
  --with-mpm=MPM          Choose the process model for Apache to use by
                          default. MPM={event|worker|prefork|winnt} This will
                          be statically linked as the only available MPM
                          unless --enable-mpms-shared is also specified.
  --with-module=module-type:module-file
                          Enable module-file in the modules/<module-type>
                          directory.
  --with-program-name     alternate executable name
  --with-suexec-bin       Path to suexec binary
  --with-suexec-caller    User allowed to call SuExec
  --with-suexec-userdir   User subdirectory
  --with-suexec-docroot   SuExec root directory
  --with-suexec-uidmin    Minimal allowed UID
  --with-suexec-gidmin    Minimal allowed GID
  --with-suexec-logfile   Set the logfile
  --with-suexec-safepath  Set the safepath
  --with-suexec-umask     umask for suexec'd process

Some influential environment variables:
  CC          C compiler command
  CFLAGS      C compiler flags
  LDFLAGS     linker flags, e.g. -L<lib dir> if you have libraries in a
              nonstandard directory <lib dir>
  LIBS        libraries to pass to the linker, e.g. -l<library>
  CPPFLAGS    (Objective) C/C++ preprocessor flags, e.g. -I<include dir> if
              you have headers in a nonstandard directory <include dir>
  CPP         C preprocessor

Use these variables to override the choices made by `configure' or to help
it to find libraries and programs with nonstandard names/locations.

Report bugs to the package provider.

Phew! Have a read through and jot down some candidates. This may lead to other things, for example I quickly realised that I would need to use options to tell configure where to find my OpenSSL and PCRE.

PCRE :  "--with-pcre=/opt/warwick/pcre/bin/pcre-config" 
and  :  "--prefix=/opt/warwick/apache2/2.4" 
and  :  "--with-ssl=/opt/warwick/openssl"

And, of course, it would make sense to put these in place first. So, I downloaded OpenSSL from http://www.openssl.org/source and compiled with:

It was super-easy

# ./Configure solaris-x86-cc --shared --prefix=/opt/warwick/openssl

Then ‘make’, ‘make install’ and OpenSSL is ready.

PCRE was equally as straight forward, available from http://www.pcre.org/

Now I have OpenSSL under /opt/warwick/openssl and PCRE under /opt/warwick/pcre, back to the plot with Apache. The final configure options I used were:

"./configure" \
"--prefix=/opt/warwick/apache2/2.4" \
"--enable-so" \
"--enable-mods-shared=all" \
"--enable-cache" \
"--enable-deflate" \
"--enable-disk-cache" \
"--enable-file-cache" \
"--enable-cgi" \
"--enable-proxy" \
"--enable-proxy-ajp" \
"--enable-proxy-balancer" \
"--enable-proxy-connect" \
"--enable-proxy-http" \
"--enable-ssl" \
"--enable-suexec" \
"--with-suexec-caller=webservd" \
"--enable-exception-hook" \
"--with-ssl=/opt/warwick/openssl" \
"--with-included-apr" \
"--with-mpm=prefork" \
"--with-pcre=/opt/warwick/pcre/bin/pcre-config" \
"CC=cc -m32" \
"CFLAGS=-xarch=386 -xchip=pentium -xspace -xs -xO4 -Xa -xildoff -xc99=all -xnorunpath -m32 -DSSL_ENGINE -DSSL_EXPERIMENTAL" \
"LDFLAGS= -s" \
"$@"

What I found after compilation is that the configure script doesn’t correctly pick up the change of location for SSL. Setting the LD_RUN_PATH before the make solves this:

  # export LD_RUN_PATH=/opt/warwick/openssl/lib

After ‘make’ and ‘make install’, I checked my new libraries and modules with:

bash-3.00# ldd /opt/warwick/apache2/2.4/modules/mod_ssl.so
    libssl.so.1.0.0 =>     /opt/warwick/openssl/lib/libssl.so.1.0.0
    libcrypto.so.1.0.0 =>     /opt/warwick/openssl/lib/libcrypto.so.1.0.0

The PCRE linking can be checked with:

bash-3.00# ldd /opt/warwick/apache2/2.4/bin/httpd 
    libpcre.so.0 =>     /opt/warwick/pcre/lib/libpcre.so.0

After creation of some self-signed certs, the installation is tested and seems to work fine:

bash-3.00# /opt/warwick/apache2/2.4/bin/apachectl start
bash-3.00# tail /opt/warwick/apache2/2.4/logs/error_log 
[Fri Feb 24 13:15:27.869212 2012] [mpm_prefork:notice] [pid 16668] AH00169: caught SIGTERM, shutting down
[Fri Feb 24 13:15:28.315640 2012] [mpm_prefork:notice] [pid 16735] AH00163: Apache/2.4.1 (Unix) OpenSSL/1.0.0g configured 
   -- resuming normal operations
[Fri Feb 24 13:15:28.315777 2012] [core:notice] [pid 16735] AH00094: Command line: '/opt/warwick/apache2/2.4/bin/httpd'

bash-3.00# 
bash-3.00# netstat -an | grep 80
      *.51680                             Idle
      *.80                 *.*                0      0 49152      0 LISTEN
      *.80                              *.*                             0      0 49152      0 LISTEN      
bash-3.00# netstat -an | grep 443
      *.443                *.*                0      0 49152      0 LISTEN
      *.443                             *.*                             0      0 49152      0 LISTEN      
bash-3.00# telnet localhost 80
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET /
<html><h1>It works!</h1></html>
Connection to localhost closed by foreign host.
bash-3.00# 

SSL works too – tested in a browser.

Soon I’ll package this up, first a custom manifest for SMF. The reader should be noted that currently, because of the need to bind to a privileged port, apache normally needs to start as the root user;

root  9911 18276   0 14:29:48 ?           0:00 /opt/warwick/apache2/2.4/bin/httpd -k start

The server will hand off children processes as the webservd user, but there will always be the single parent running as root. Clearly the process doesn’t need full root privileges, only the ability to bind to ports lower than 1024. There is room here then for security improvements / paranoia relief because we can use the least priv model in Solaris to run everything as webservd and give webservd just the enough privs (net_privaddr). The manifest I’ve used is fundamentally the same as the one found here

setprop start/privileges = astring: (basic,!proc_session,!proc_info,!file_link_any,net_privaddr)
setprop start/user = astring: (webservd)

OK, create the manifest and method script with these entries and import…

   #  cp war-apache2.xml /var/svc/manifest/network/
   #  cp war-apache2 /lib/svc/method/

bash-3.00# svccfg import /var/svc/manifest/network/war-apache2.xml 
bash-3.00# svcs war-apache2
STATE          STIME    FMRI
disabled       14:20:08 svc:/network/war-apache2:default

bash-3.00# /opt/warwick/apache2/2.4/bin/apachectl stop

bash-3.00# svcadm enable war-apache2 ; tail -f /opt/warwick/apache2/2.4/logs/error_log 

[Fri Feb 24 13:02:21.804521 2012] [mpm_prefork:notice] [pid 14926] AH00163: Apache/2.4.1 (Unix) OpenSSL/1.0.0g configured -- resuming normal operations
[Fri Feb 24 13:02:21.804585 2012] [core:notice] [pid 14926] AH00094: Command line: '/opt/warwick/apache2/2.4/bin/httpd'

After this, we can see that it all started ok:

bash-3.00# svcs war-apache2
STATE          STIME    FMRI
online         13:02:21 svc:/network/war-apache2:default
bash-3.00# 

Notice, not a single root process…

bash-3.00# ps -ef | grep http | grep -v grep    
webservd 14930 14926   0 13:02:22 ?           0:00 /opt/warwick/apache2/2.4/bin/httpd -k start
webservd 14928 14926   0 13:02:22 ?           0:00 /opt/warwick/apache2/2.4/bin/httpd -k start
webservd 14931 14926   0 13:02:22 ?           0:00 /opt/warwick/apache2/2.4/bin/httpd -k start
webservd 14927 14926   0 13:02:22 ?           0:00 /opt/warwick/apache2/2.4/bin/httpd -k start
webservd 14929 14926   0 13:02:22 ?           0:00 /opt/warwick/apache2/2.4/bin/httpd -k start
webservd 14926 18276   0 13:02:22 ?           0:00 /opt/warwick/apache2/2.4/bin/httpd -k start

Which is neat. very neat. Now I’ll attempt to package this all up in an SVR4 pkg.

# find /opt/warwick/apache2 -print > /var/tmp/war-apache-pkg/file-list 
# cat /var/tmp/war-apache-pkg/file-list | pkgproto > /var/tmp/war-apache-pkg/Prototype

bash-3.00# cat pkginfo
PKG="WARapache" 
NAME="Apache 2.4" 
VERSION="2.4.1" 
ARCH="x86" 
CLASSES="none" 
CATEGORY="utility" 
VENDOR="Warwick" 
PSTAMP="ro-ora01-sol10-201202241534" 
EMAIL="paul.eggleton@warwick.ac.uk" 
ISTATES="S s 1 2 3" 
RSTATES="S s 1 2 3" 
BASEDIR="/"

I also created a basic postinstall and checkinstall script. The checkinstall is optional and I used this simply to check the architecture was correct. The postinstall installs the SMF service;

bash-3.00# cat postinstall
#!/bin/sh

if [ -f /var/svc/manifest/network/war-apache2.xml ]
then
    /usr/sbin/svccfg import /var/svc/manifest/network/war-apache2.xml
else
    echo "Error with installation - /var/svc/manifest/network/war-apache2.xml not present, aborting" 
    exit 1
fi

Next the SVR4 packaging voodoo;

# pkgmk -o -r / -d /var/tmp/war-apache-pkg/ -f Prototype 

# pkgtrans -s /var/tmp/war-apache-pkg WARapache24.pkg WARapache

And we have a package:

bash-3.00# ls WAR*
WARapache24.pkg

WARapache:
install  pkginfo  pkgmap   root
bash-3.00# 

bash-3.00# pkginfo -d WARapache24.pkg 
utility     WARapache Apache 2.4
bash-3.00# 

bash-3.00# pkgadd -d WARapache24.pkg 

The following packages are available:
  1  WARapache     Apache 2.4
                   (x86) 2.4.1

Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [?,??,q]: 

My OpenSSL package was built almost identically. I think this is almost the finished product. I should note that I also included the PCRE files in the final WARapache package to avoided an extra package dependency.

Next up, Solaris 11 and IPS.


February 25, 2011

Disks; cfgadm, fcinfo and LUN mapping on Solaris

OK, so you have a Solaris 10 host with SAN connected storage – how do you make sense of the LUNs you can see? What tools can be used to interrogate the storage and build a mental image of what you have been presented with? This article is intended as a brief introduction to some of the commands in Solaris that will help you achieve your tasks.

Firstly, in order to allow your storage admin to map you some LUNs, you’ll need to provide him with the WWNs of the HBA ports in your server. This is so he can map the LUNs you’ve asked for to the WWNs of your server. These can be found using the fcinfo command. Start with ‘fcinfo hba-port’, Note that the output below shows all 4 of my ports, only 2 of which are occupied and online (c3 and c5).

...
        OS Device Name: /dev/cfg/c3
...
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb

...
        OS Device Name: /dev/cfg/c5
...
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb

The full output;

bash-3.00# fcinfo hba-port
HBA Port WWN: 21000024ff295a34
        OS Device Name: /dev/cfg/c2
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402R00-1023835637
        Driver Name: qlc
        Driver Version: 20100301-3.00
        Type: unknown
        State: offline
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: not established
        Node WWN: 20000024ff295a34
HBA Port WWN: 21000024ff295a35
        OS Device Name: /dev/cfg/c3
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402R00-1023835637
        Driver Name: qlc
        Driver Version: 20100301-3.00
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 20000024ff295a35
HBA Port WWN: 21000024ff295a36
        OS Device Name: /dev/cfg/c4
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402R00-1023835638
        Driver Name: qlc
        Driver Version: 20100301-3.00
        Type: unknown
        State: offline
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: not established
        Node WWN: 20000024ff295a36
HBA Port WWN: 21000024ff295a37
        OS Device Name: /dev/cfg/c5
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402R00-1023835638
        Driver Name: qlc
        Driver Version: 20100301-3.00
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 20000024ff295a37
bash-3.00# 

It is the ‘HBA Port WWN’ that you need to give to your storage admin. He may appreciate the full output, that will confirm a few other items for him such as the link speed and your HBA manufacturer and driver version numbers.

Using the -l (minus ell) flag shows additional information such as link statistics…

bash-3.00# fcinfo hba-port -l 21000024ff295a37
HBA Port WWN: 21000024ff295a37
        OS Device Name: /dev/cfg/c5
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402R00-1023835638
        Driver Name: qlc
        Driver Version: 20100301-3.00
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 20000024ff295a37
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 0
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 0
                Invalid CRC Count: 0
bash-3.00# 

The details (WWN) of the remote ports can be viewed using ‘fcinfo remote-port -p

bash-3.00# fcinfo remote-port -p 21000024ff295a37
Remote Port WWN: 24540002ac0009e2
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac0009e2
Remote Port WWN: 25540002ac0009e2
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac0009e2
Remote Port WWN: 22120002ac000928
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac000928
Remote Port WWN: 23120002ac000928
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac000928

The -l option can still be used in conjunction with this to show link statistics, and the -s option will show you the LUNs; This is very handy as it shows the LUN number / device name mappings.

...
        LUN: 4
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BC0928d0s2

...

The full output is below;

bash-3.00# fcinfo remote-port -l -p 21000024ff295a37
Remote Port WWN: 24540002ac0009e2
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac0009e2
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 2
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 2
                Invalid CRC Count: 0
Remote Port WWN: 25540002ac0009e2
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac0009e2
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 2
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 2
                Invalid CRC Count: 0
Remote Port WWN: 22120002ac000928
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac000928
        Link Error Statistics:
                Link Failure Count: 2
                Loss of Sync Count: 1
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 2
                Invalid CRC Count: 0
Remote Port WWN: 23120002ac000928
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac000928
        Link Error Statistics:
                Link Failure Count: 2
                Loss of Sync Count: 1
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 0
                Invalid CRC Count: 0
bash-3.00# fcinfo remote-port -ls -p 21000024ff295a37
Remote Port WWN: 24540002ac0009e2
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac0009e2
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 2
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 2
                Invalid CRC Count: 0
        LUN: 254
          Vendor: 3PARdata
          Product: SES            
          OS Device Name: /dev/es/ses8
Remote Port WWN: 25540002ac0009e2
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac0009e2
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 2
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 2
                Invalid CRC Count: 0
        LUN: 254
          Vendor: 3PARdata
          Product: SES            
          OS Device Name: /dev/es/ses8
Remote Port WWN: 22120002ac000928
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac000928
        Link Error Statistics:
                Link Failure Count: 2
                Loss of Sync Count: 1
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 2
                Invalid CRC Count: 0
        LUN: 0
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001B70928d0s2
        LUN: 1
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001B90928d0s2
        LUN: 2
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BA0928d0s2
        LUN: 3
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BB0928d0s2
        LUN: 4
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BC0928d0s2
        LUN: 5
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BD0928d0s2
        LUN: 6
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BE0928d0s2
        LUN: 7
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BF0928d0s2
        LUN: 8
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C00928d0s2
        LUN: 9
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C10928d0s2
        LUN: 10
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C20928d0s2
        LUN: 11
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C30928d0s2
        LUN: 12
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C40928d0s2
        LUN: 13
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C50928d0s2
        LUN: 14
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001B80928d0s2
        LUN: 254
          Vendor: 3PARdata
          Product: SES            
          OS Device Name: /dev/es/ses9
Remote Port WWN: 23120002ac000928
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac000928
        Link Error Statistics:
                Link Failure Count: 2
                Loss of Sync Count: 1
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 0
                Invalid CRC Count: 0
        LUN: 0
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001B70928d0s2
        LUN: 1
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001B90928d0s2
        LUN: 2
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BA0928d0s2
        LUN: 3
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BB0928d0s2
        LUN: 4
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BC0928d0s2
        LUN: 5
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BD0928d0s2
        LUN: 6
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BE0928d0s2
        LUN: 7
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BF0928d0s2
        LUN: 8
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C00928d0s2
        LUN: 9
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C10928d0s2
        LUN: 10
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C20928d0s2
        LUN: 11
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C30928d0s2
        LUN: 12
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C40928d0s2
        LUN: 13
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C50928d0s2
        LUN: 14
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001B80928d0s2
        LUN: 254
          Vendor: 3PARdata
          Product: SES            
          OS Device Name: /dev/es/ses9

The cfgadm command can be used to view the system ‘attachment points’ which are broadly defined as the location of certain hardware resources visible to Solaris. Running cfgadm with the -al option will list these attachment points. This will include SAN HBAs and LUNs, USB devices, internal disks such as SATA etc.

The type field gives an indication of the type of the device, you will see some self explanatory entries such as ‘scsi-bus’ (which I think about as a controller), disk – very self explanatory. A fibre channel SAN HBA is usually seen as fc, if it is connected to the fabric/SAN it will show as fc-fabric. If you see an entry labelled ESI then know that you are seeing the Enclosure through the Enclosure Services Interface.

The output below is partially truncated.

bash-3.00# cfgadm -al
Ap_Id                          Type         Receptacle   Occupant     Condition
c0                             scsi-bus     connected    configured   unknown
c0::dsk/c0t0d0                 disk         connected    configured   unknown
c0::dsk/c0t1d0                 disk         connected    configured   unknown
c0::dsk/c0t2d0                 disk         connected    configured   unknown
c0::dsk/c0t3d0                 disk         connected    configured   unknown
c0::dsk/c0t4d0                 disk         connected    configured   unknown
c2                             fc           connected    unconfigured unknown
c3                             fc-fabric    connected    configured   unknown
c3::22110002ac000928           disk         connected    configured   unknown
c3::23110002ac000928           disk         connected    configured   unknown
c3::24530002ac0009e2           ESI          connected    configured   unknown
c3::25530002ac0009e2           ESI          connected    configured   unknown
c4                             fc           connected    unconfigured unknown
c5                             fc-fabric    connected    configured   unknown
c5::22120002ac000928           disk         connected    configured   unknown
c5::23120002ac000928           disk         connected    configured   unknown
c5::24540002ac0009e2           ESI          connected    configured   unknown
c5::25540002ac0009e2           ESI          connected    configured   unknown
sata0/0                        sata-port    empty        unconfigured ok
sata0/1                        sata-port    empty        unconfigured ok

Use the ‘-o show_FCP_dev’ option to get cfgadm to show not only the controllers and the enclosures, but also any fibre channel disks that may be visible on the channel:

Ap_Id                          Type         Receptacle   Occupant     Condition
c2                             fc           connected    unconfigured unknown
c3                             fc-fabric    connected    configured   unknown
c3::22110002ac000928,0         disk         connected    configured   unknown
c3::22110002ac000928,1         disk         connected    configured   unknown
c3::22110002ac000928,2         disk         connected    configured   unknown
c3::22110002ac000928,3         disk         connected    configured   unknown
c3::22110002ac000928,4         disk         connected    configured   unknown
c3::22110002ac000928,5         disk         connected    configured   unknown
c3::22110002ac000928,6         disk         connected    configured   unknown
...
c3::22110002ac000928,12        disk         connected    configured   unknown
c3::22110002ac000928,13        disk         connected    configured   unknown
c3::22110002ac000928,14        disk         connected    configured   unknown
c3::22110002ac000928,254       ESI          connected    configured   unknown
.....
c3::23110002ac000928,13        disk         connected    configured   unknown
c3::23110002ac000928,14        disk         connected    configured   unknown
c3::23110002ac000928,254       ESI          connected    configured   unknown
c3::24530002ac0009e2,254       ESI          connected    configured   unknown
c3::25530002ac0009e2,254       ESI          connected    configured   unknown
c4                             fc           connected    unconfigured unknown
c5                             fc-fabric    connected    configured   unknown
c5::22120002ac000928,0         disk         connected    configured   unknown
c5::22120002ac000928,1         disk         connected    configured   unknown
c5::22120002ac000928,2         disk         connected    configured   unknown
c5::22120002ac000928,3         disk         connected    configured   unknown
c5::22120002ac000928,4         disk         connected    configured   unknown
c5::22120002ac000928,5         disk         connected    configured   unknown
....
c5::22120002ac000928,14        disk         connected    configured   unknown
c5::22120002ac000928,254       ESI          connected    configured   unknown
c5::23120002ac000928,0         disk         connected    configured   unknown
....
c5::23120002ac000928,12        disk         connected    configured   unknown
c5::23120002ac000928,13        disk         connected    configured   unknown
c5::23120002ac000928,14        disk         connected    configured   unknown
c5::23120002ac000928,254       ESI          connected    configured   unknown
c5::24540002ac0009e2,254       ESI          connected    configured   unknown
c5::25540002ac0009e2,254       ESI          connected    configured   unknown
bash-3.00# 

Throughout the outputs above you’ll notice that these disks are multipathed and visible through 2 separate controllers (c3 and c5). Enable STMS (see stmsboot1m)) to aggregate those 2 paths to a single controller. You will gain a pseudo controller when you do this. In this case, the controller becomes c6 (aggregate of c3 and c5). The new disk targets created by STMS then are visible in format:

      18. c6t50002AC001C40928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD13
          /scsi_vhci/disk@g50002ac001c40928
      19. c6t50002AC001C50928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD14
          /scsi_vhci/disk@g50002ac001c50928

Once we have these multiple paths, luxadm can be used to interrogate the controller and view the subpaths (and their state). First, run a ‘luxadm probe’ which will scan the devices and present a list.

bash-3.00# luxadm probe             

Found Fibre Channel device(s):
  Node WWN:2ff70002ac0009e2  Device Type:SES device
    Logical Path:/dev/es/ses8
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001B80928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001C50928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001C40928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001C30928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001C20928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001C10928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001C00928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001BF0928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001BE0928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001BD0928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001BC0928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001BB0928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001BA0928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001B90928d0s2
  Node WWN:2ff70002ac000928  Device Type:SES device
    Logical Path:/dev/es/ses9
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001B70928d0s2
bash-3.00#

Now you can select a logical path and use with ‘luxadm display ’ to view the individual paths;

bash-3.00# luxadm display /dev/rdsk/c6t50002AC001B80928d0s2DEVICE PROPERTIES for disk: /dev/rdsk/c6t50002AC001B80928d0s2
  Vendor:               3PARdata
  Product ID:           VV             
  Revision:             0000
  Serial Num:           01B80928
  Unformatted capacity: 40960.000 MBytes
  Write Cache:          Enabled
  Read Cache:           Enabled
    Minimum prefetch:   0x0
    Maximum prefetch:   0xffff
  Device Type:          Disk device
  Path(s):

  /dev/rdsk/c6t50002AC001B80928d0s2
  /devices/scsi_vhci/disk@g50002ac001b80928:c,raw
   Controller           /dev/cfg/c3
    Device Address              22110002ac000928,e
    Host controller port WWN    21000024ff295a35
    Class                       primary
    State                       ONLINE
   Controller           /dev/cfg/c3
    Device Address              23110002ac000928,e
    Host controller port WWN    21000024ff295a35
    Class                       primary
    State                       ONLINE
   Controller           /dev/cfg/c5
    Device Address              22120002ac000928,e
    Host controller port WWN    21000024ff295a37
    Class                       primary
    State                       ONLINE
   Controller           /dev/cfg/c5
    Device Address              23120002ac000928,e
    Host controller port WWN    21000024ff295a37
    Class                       primary
    State                       ONLINE

bash-3.00#

Things to note are that the size, write cache, read cache and path state are all shown and that the ’,e’ after the Device Address is the LUN number in hex.

The unique part of the LUN indentifier (the LUN id) is embedded halfway through the mpxio / STMS name. To extract from format, for example;

bash-3.00# echo | format |grep c6 | cut -c22-25
01B7
01B8
01B9
01BA
01BB
01BC
01BD
01BE
01BF
01C0
01C1
01C2
01C3
01C4
01C5

This information, in conjunction with luxadm can be used to correlate disks to LUN numbers and LUN IDs. Although the fcinfo commands shown above are generally the easier way to obtain this information. Notice that the LUN number is appended to the Device Address in hex.

bash-3.00# for disk in $( ls /dev/rdsk/c6t50002AC00*s2 )
> do
> echo $disk
> echo $disk | cut -c22-26
> luxadm display $disk |grep 'Device Address'
> done
/dev/rdsk/c6t50002AC001B70928d0s2
01B70
    Device Address              22110002ac000928,0
    Device Address              23110002ac000928,0
    Device Address              22120002ac000928,0
    Device Address              23120002ac000928,0

/dev/rdsk/c6t50002AC001B90928d0s2
01B90
    Device Address              22110002ac000928,1
    Device Address              23110002ac000928,1
    Device Address              22120002ac000928,1
    Device Address              23120002ac000928,1
/dev/rdsk/c6t50002AC001BA0928d0s2
01BA0
    Device Address              22110002ac000928,2
    Device Address              23110002ac000928,2
    Device Address              22120002ac000928,2
    Device Address              23120002ac000928,2
/dev/rdsk/c6t50002AC001BB0928d0s2
01BB0
    Device Address              22110002ac000928,3
    Device Address              23110002ac000928,3
    Device Address              22120002ac000928,3
    Device Address              23120002ac000928,3
/dev/rdsk/c6t50002AC001BC0928d0s2
01BC0
    Device Address              22110002ac000928,4
    Device Address              23110002ac000928,4
    Device Address              22120002ac000928,4
    Device Address              23120002ac000928,4
/dev/rdsk/c6t50002AC001BD0928d0s2
01BD0
    Device Address              22110002ac000928,5
    Device Address              23110002ac000928,5
    Device Address              22120002ac000928,5
    Device Address              23120002ac000928,5
/dev/rdsk/c6t50002AC001BE0928d0s2
01BE0
    Device Address              22110002ac000928,6
    Device Address              23110002ac000928,6
    Device Address              22120002ac000928,6
    Device Address              23120002ac000928,6
/dev/rdsk/c6t50002AC001BF0928d0s2
01BF0
    Device Address              22110002ac000928,7
    Device Address              23110002ac000928,7
    Device Address              22120002ac000928,7
    Device Address              23120002ac000928,7
/dev/rdsk/c6t50002AC001C00928d0s2
01C00
    Device Address              22110002ac000928,8
    Device Address              23110002ac000928,8
    Device Address              22120002ac000928,8
    Device Address              23120002ac000928,8
/dev/rdsk/c6t50002AC001C10928d0s2
01C10
    Device Address              22110002ac000928,9
    Device Address              23110002ac000928,9
    Device Address              22120002ac000928,9
    Device Address              23120002ac000928,9
/dev/rdsk/c6t50002AC001C20928d0s2
01C20
    Device Address              22110002ac000928,a
    Device Address              23110002ac000928,a
    Device Address              22120002ac000928,a
    Device Address              23120002ac000928,a
/dev/rdsk/c6t50002AC001C30928d0s2
01C30
    Device Address              22110002ac000928,b
    Device Address              23110002ac000928,b
    Device Address              22120002ac000928,b
    Device Address              23120002ac000928,b
/dev/rdsk/c6t50002AC001C40928d0s2
01C40
    Device Address              22110002ac000928,c
    Device Address              23110002ac000928,c
    Device Address              22120002ac000928,c
    Device Address              23120002ac000928,c
/dev/rdsk/c6t50002AC001C50928d0s2
01C50
    Device Address              22110002ac000928,d
    Device Address              23110002ac000928,d
    Device Address              22120002ac000928,d
    Device Address              23120002ac000928,d
/dev/rdsk/c6t50002AC001B80928d0s2
01B80
    Device Address              22110002ac000928,e
    Device Address              23110002ac000928,e
    Device Address              22120002ac000928,e
    Device Address              23120002ac000928,e

Labelling the disks with a volume name, which is completely optional, but quite a useful feature relies on a reference to the LUN numbers to ensure the correct labels are assigned to the correct disks. I like to use the following which will print the LUN number for me just before presenting me with a format dialog in which to assign the appropriate volname;

for disk in $( ls /dev/rdsk/c6t50002AC00*s2 )
do 
   echo $disk
   echo $disk | cut -c22-26
   luxadm display $disk |grep 'Device Address' 
   format $disk 
done

A truncated output example (without the actual format screens) will be something like;

/dev/rdsk/c6t50002AC001BD0928d0s2
01BD0
    Device Address              22110002ac000928,5
    Device Address              23110002ac000928,5
    Device Address              22120002ac000928,5
    Device Address              23120002ac000928,5
format /dev/rdsk/c6t50002AC001BD0928d0s2
/dev/rdsk/c6t50002AC001BE0928d0s2
01BE0
    Device Address              22110002ac000928,6
    Device Address              23110002ac000928,6
    Device Address              22120002ac000928,6
    Device Address              23120002ac000928,6
format /dev/rdsk/c6t50002AC001BE0928d0s2

Your format command with named and labelled LUNs will look something like:

       5. c6t50002AC001B70928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD01
          /scsi_vhci/disk@g50002ac001b70928
       6. c6t50002AC001B80928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD15
          /scsi_vhci/disk@g50002ac001b80928
       7. c6t50002AC001B90928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD02
          /scsi_vhci/disk@g50002ac001b90928
       8. c6t50002AC001BA0928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD03
          /scsi_vhci/disk@g50002ac001ba0928
       9. c6t50002AC001BB0928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD04
          /scsi_vhci/disk@g50002ac001bb0928
      10. c6t50002AC001BC0928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD05
          /scsi_vhci/disk@g50002ac001bc0928
      11. c6t50002AC001BD0928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD06
          /scsi_vhci/disk@g50002ac001bd0928
      12. c6t50002AC001BE0928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD07
          /scsi_vhci/disk@g50002ac001be0928
      13. c6t50002AC001BF0928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD08
          /scsi_vhci/disk@g50002ac001bf0928
      14. c6t50002AC001C00928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD09
          /scsi_vhci/disk@g50002ac001c00928
      15. c6t50002AC001C10928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD10
          /scsi_vhci/disk@g50002ac001c10928
      16. c6t50002AC001C20928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD11
          /scsi_vhci/disk@g50002ac001c20928
      17. c6t50002AC001C30928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD12
          /scsi_vhci/disk@g50002ac001c30928
      18. c6t50002AC001C40928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD13
          /scsi_vhci/disk@g50002ac001c40928
      19. c6t50002AC001C50928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD14
          /scsi_vhci/disk@g50002ac001c50928
Specify disk (enter its number): 

The next steps are to configure your LUNs as required. This will depend on the intended usage, perhaps they will be used as RAW or ASM volumes for database usage, perhaps as ZFS zpools for filesystem usage.

Paul.


February 24, 2011

OpsCenter and ISC DHCP server

We have recently received a new OpsCentre (center?) installation. I must say, I am suitably (sadly, predictably) underwhelmed. It is slow, sluggish and fairly non-intuitive. It often fails on seemingly the most simple task (such as server discovery) for the most unfathomable of reasons. The error messages within the GUI for the tasks it fails are also meaningless.

One of the most recent failures I experienced with OpsCentre was during the import of a pre-cooked jet template and flash archive. I managed to create the profile and plan (although it didn’t seem to like my compressed flar archive and I had to switch to an uncompressed one). I was hoping, given that I know both the flash archive and the template are good that this would be as simple as pointing OpsCenter at the necessary files and it would do the rest. However, when I attempted to apply this new deployment plan and profile to a client I received the following errors:

The DeploymentPlan execution job failed because the DeploymentProvider OSDeployment
 for Step Provision OS failed to generate tasks for the job: 
com.sun.xvm.services.osdeployment.OSDeploymentException: Cannot prepare OS profile: 
com.sun.xvm.services.osdeployment.OSDeploymentException: Failed to retrieve Profile 
Information from database. (32557). Contact My Oracle Support if the problem persists. (10445)

At this point I asked myself why I was wasting time using a non-functional GUI when I could quite easily use the command line and configure the local ISC DHCP server on the OpsCentre controller to be my local DHCP server for a local client and then instruct the client to install from our existing, but ‘subnet-remote’ Jet jumpstart server using a flash archive over NFS.

Before I continue, I should probably concede that OpsCentre does seem to be quite good at patching. It will keep track of baselines, let you select baselines for application to clients, check dependencies and download/install all the dependent patches and install them for you in a hands-off manner. Apparently this is the pre-existing portion of the product that Sun/Oracle obtained when they bought the company…. anyway, enough of that.

The ISC DHCP server

For a while now I’ve heard talk about the ISC (Internet Software Consortium) DHCP server as an alternative to the Sun standard DHCP server without actually taking/finding the time to investigate it. Although I used to loath the Sun DHCP server, I have grown to love, like, ok, I’ve just about become accustomed to it. The fact that this is bundled with OpsCentre presented me with a perfect opportunity to test drive the ISC DHCP server. I will admit, based on first impressions, it is far easier to configure and seems more predictable in behaviour than the Sun DHCP server.

First, (and this will be different for a ISC DHCP server that you down load) I note that the service is managed by the following SMF service:

-bash-3.00$ svcs dhcpd 
STATE          STIME    FMRI
online         13:36:26 svc:/application/scn/dhcpd:default
-bash-3.00$ 

One of the first thing that you need to know is that on start/stop it logs to

/var/svc/log/application-scn-dhcpd:default.log

A successful startup will end with the lines:

Starting dhcpd: nxge0Internet Software Consortium DHCP Server V3.0pl2
Copyright 1995-2003 Internet Software Consortium.
All rights reserved.
For info, please visit http://www.isc.org/products/DHCP
Wrote 0 deleted host decls to leases file.
Wrote 0 new dynamic host decls to leases file.
Wrote 0 leases to leases file.
Listening on DLPI/nxge0/00:14:4f:f3:43:f2/10.20.30.0/24
Sending on   DLPI/nxge0/00:14:4f:f3:43:f2/10.20.30.0/24
Sending on   Socket/fallback/fallback-net
[ Feb  2 13:36:26 Method "start" exited with status 0 ]

The configuration file for the ISC DHCP server is located in /etc and is named dhcpd.conf

-bash-3.00$ ls -l /etc/dhcpd.conf
-rw-r--r--   1 root     sys         7100 Feb  2 12:51 /etc/dhcpd.conf

One of the first changes I made was to enable syslog logging for the service. This is acheived by adding a line similar to the following to the config file:

log-facility local6;

You’ll then need something like the following added to the /etc/syslog.conf file – don’t forget any whitespace must be tab only.

local6.info;local6.notice;local6.warn;local6.err;local6.debug   /var/log/dhcpd.log

The /var/log/dhcpd.log file will be your friend during any troubleshooting of client boot problems.

Then, rather than edit the file directly any further, I opted for an include file in which I place all the configuration options for the client(s) I wis to DHCP boot. The very last line of my dhcpd.conf, therefore, reads:

include "/export/dhcp/sol10_x86_dhcpd.conf";

The sol10_x86_dhcpd.conf file contains the following to setup the required Sun macros:

group {
  use-host-decl-names on;
  filename "pxegrub.I86PC.Solaris_10-1";
  next-server 10.20.30.40;
  vendor-option-space SUNW;
  option SUNW.JumpStart-server "192.168.10.1:/opt/SUNWjet";
  option SUNW.root-server-hostname "jumpstart-svr";
  option SUNW.root-server-ip-address 192.168.10.1;
  option SUNW.root-path-name "/export/install/media/x86/solaris10u9/boot";
  option SUNW.install-path "/export/install/media/x86/solaris10u9";

host utah {
        hardware ethernet 0:21:28:5f:32:16;
        fixed-address 10.20.30.41;
        option SUNW.sysid-config-file-server = "192.168.10.1:/opt/SUNWjet/Clients/utah";
        }
}

This configuration sets the required dhcp options which enable the client to reach a remote jumpstart install and sysidcfg server across the network. The client will also need instructions in the grub boot menu it receives from the dhcp server; this file is usually found in /tftpboot on the server; I edited this to include the following

-bash-3.00$ more /tftpboot/menu.lst.010021285F3216
default=0
timeout=2
min_mem64 1024
title Solaris_10 Jumpstart
        kernel$ /I86PC.Solaris_10-1/multiboot kernel/$ISADIR/unix  - install dhcp -B \ 
console=ttya install_config=192.168.10.1:/opt/SUNWjet,sysid_config=192.168.10.1: \
/opt/SUNWjet/Clients/utah,install_media=192.168.10.1:/export/install/media \
/x86/solaris10u9,install_boot=192.168.10.1:/export/install/media/x86/solaris10u9/boot
        module$ /I86PC.Solaris_10-1/$ISADIR/x86.miniroot

The jet / jumpstart server will of course need to be configured appropriately for the client. The /etc/SUNWjet/etc/dhcp.conf should set the following option to NOOP because the jet / jumpstart server itself is not going to be used as the DHCP server;

# DHCP_FORMAT   Which dhcp server type is supported. SUN is currently supported,
#               however, ISC is still in development. If REMOTE_DHCP is set,
#               you must set DHCP_FORMAT to SUN.  Valid entries are "SUN" 
#               and "NOOP". NOOP can be used if you are manually setting
#               up your own DHCP server with the required parameters.
#
DHCP_FORMAT="NOOP"

Once all this is straight, you should be able to jumpstart install your client. A useful command for troubleshooting the dhcp conversation between the client and the dhcp server is snoop. You should see DHCPDISCOVER broadcast from the client and DHCPOFFER sent back. The client should then return an accept pdu so that the server knows the client is accepting the IP address and the lease with a DHCPREQUEST, finally a DHCPACK will be sent from the server to the client. To see this in detail (including a display of the DHCP options sent by the server, use the following;

bash-3.00# snoop -vvv -d nxge0 port bootpc or port bootps
Using device nxge0 (promiscuous mode)

This output will be useful, but quite verbose – a shorter summary will be seen in the dhcpd.log file configured in syslog;

Feb  2 15:58:43 client1 dhcpd: [ID 702911 local6.info] DHCPDISCOVER from 0:2:8:5:3:16 via nxge0
Feb  2 15:58:43 turkana dhcpd: [ID 702911 local6.info] DHCPOFFER on 10.20.30.41 to 0:2:8:5:3:16 via nxge0
Feb  2 15:58:46 client1 dhcpd: [ID 702911 local6.info] DHCPREQUEST for 10.20.30.41 (10.20.30.40) from 0:2:8:5:3:16 via nxge0
Feb  2 15:58:46 client1 dhcpd: [ID 702911 local6.info] DHCPACK on 10.20.30.41 to 0:2:8:5:3:16 via nxge0

Paul.


February 07, 2011

Silliest environment variable ever?

It seems that under Solaris 11, if you can’t be bothered to type ’-s’ everytime you want a continuous ping response from a peer, you can instead set the environment variable MACHINE_THAT_GOES_PING to something non-null and voila…. huh?

sol-esx01:~$ ping sol-esx03
sol-esx03 is alive
sol-esx01:~$ 
sol-esx01:~$ export MACHINE_THAT_GOES_PING="true" 
sol-esx01:~$ 
sol-esx01:~$ ping sol-esx03
PING sol-esx03: 56 data bytes
64 bytes from sol-esx03 (192.168.100.6): icmp_seq=0. time=0.552 ms
64 bytes from sol-esx03 (192.168.100.6): icmp_seq=1. time=0.257 ms
64 bytes from sol-esx03 (192.168.100.6): icmp_seq=2. time=0.222 ms
64 bytes from sol-esx03 (192.168.100.6): icmp_seq=3. time=0.246 ms
64 bytes from sol-esx03 (192.168.100.6): icmp_seq=4. time=0.329 ms
^C
----sol-esx03 PING Statistics----
5 packets transmitted, 5 packets received, 0% packet loss
round-trip (ms)  min/avg/max/stddev = 0.222/0.321/0.552/0.135
sol-esx01:~$ 

Seriously, I really do wonder what the case for implementing this was. I can only imagine this is to make Solaris mimic the behaviour of linux.

Maybe I’m missing something… Please tell me I am.

Paul.


February 04, 2011

IPFilter changes in Solaris 11 Express

Configuration of IP Filter in Solaris 11 Express has changed. It used to be fairly simple; decide on some rules and put them in /etc/ipf/ipf.conf, enable the service with ‘svcadm enable ipfilter’ and check with ipfstat / ipfstat -io. This is optional, but I like to customise my logging by adding ‘log first level local0.info’ to rules which I would like information about when they are fired. This also leads to the following line in syslog.conf:

#
# logging for ipfilter
#
local0.info;local0.notice;local0.warn;local0.err;local0.debug   /var/log/ipfilter.log

However, in Solaris 11 Express, by default the rules for your firewall are not expected to be found in a configuration file. Instead they have become SMF properties for the IP Filter service. If you attempt to put an ipf.conf file in place and start up IP Filter, everything will seem to start, but you will have no matches as intended because the rule base will be empty:

root@sol-esx01:/var/log# ipfstat -io
empty list for ipfilter(out)
empty list for ipfilter(in)

The answer to this lies in the man page to svc.ipfd(1m);

 firewall_config_default/policy

         Global Default policy, firewall_config property group in
         svc:/network/ipfilter:default,  can  also be set to cus-
         tom. Users can set policy to custom to use  prepopulated
         IP  Filter  configuration,  for  example, an existing IP
         Filter configuration or custom configurations that  can-
         not  be  provided by the framework. This Global Default-
         only policy mode allows users to supply a text file con-
         taining  the complete set of IPF rules. When custom mode
         is selected, the specified set of IPF rules is  complete
         and  the framework will not generate IPF rules from con-
         figured firewall policies.

     firewall_config_default/custom_policy_file

         A file path to be used when Global Default policy is set
         to  custom.  The  file  contains a set of IPF rules that
         provide the desired IP Filter configuration.  For  exam-
         ple,  users with existing IPF rules in /etc/ipf/ipf.conf
         can execute the following commands to use  the  existing
         rules:

             1.   Set custom policy:

                    # svccfg -s ipfilter:default setprop \
                    firewall_config_default/policy = astring: "custom" 

             2.   Specify custom file:

                    # svccfg -s ipfilter:default setprop \
                    firewall_config_default/custom_policy_file = astring: \
                    "/etc/ipf/ipf.conf" 

             3.   Refresh configuration:

                    # svcadm refresh ipfilter:default

So, following the instructions;

# svccfg -s ipfilter:default setprop firewall_config_default/policy = astring: "custom" 

# svccfg -s ipfilter:default listprop firewall_config_default/policy
firewall_config_default/policy  astring  custom

# svccfg -s ipfilter:default setprop firewall_config_default/custom_policy_file = astring: "/etc/ipf/ipf.conf" 

# svccfg -s ipfilter:default listprop firewall_config_default/custom_policy_file
firewall_config_default/custom_policy_file  astring  /etc/ipf/ipf.conf

# svcadm refresh ipfilter:default

Seems to do the trick….

# svcs -xv ipfiltersvc:/network/ipfilter:default (IP Filter)
 State: online since February  4, 2011 02:05:01 PM GMT
   See: man -M /usr/share/man -s 5 ipfilter
   See: /var/svc/log/network-ipfilter:default.log
Impact: None.

#
# ipfstat  |grep blocked
 input packets:         blocked 989 passed 1766 nomatch 1753 counted 0 short 0
output packets:         blocked 0 passed 1390 nomatch 1377 counted 0 short 0
 input packets logged:  blocked 403 passed 13
output packets logged:  blocked 0 passed 0

# ipfstat -io |head

pass out quick on lo0 all
pass out quick on e1000g0 proto udp from any to any
pass out quick on e1000g0 proto tcp from any to any flags S/FSRPAU keep state
block in quick from any to any with short
block in quick on e1000g0 from any to any port = 137
block in quick on e1000g0 from any to any port = 138
block in quick on e1000g0 from any to any port = 1434
pass in quick on lo0 all
pass in quick on e1000g0 from 12.34.56.78/32 to any keep state
pass in log first level local0.info quick on e1000g0 proto tcp from 12.34.56.78/16 to any port = ssh flags S/FSRPAU keep state

I’m not sure I’ll ever use anything other than a configuration file for the firewall configs. The configs I need to use are quite complex and lengthy and management/distribution of this seems easiest through a standard text file.

Incidentally, there does seem to be a function to do this for you in /lib/svc/method/ipfilter;

upgrade_config()
{
    old_ipfconf="/etc/ipf/ipf.conf" 

    if [ -f ${old_ipfconf} ]; then
        grep '^[ \t]*[^# \t]' ${old_ipfconf} >/dev/null 2>&1
        if [ $? -eq 0 ]; then
            svccfg -s $SMF_FMRI setprop \
                $FW_CONFIG_DEF_PG/$POLICY_PROP = astring: \
                "custom" >/dev/null 2>&1
            svccfg -s $SMF_FMRI setprop \
                $FW_CONFIG_DEF_PG/$CUSTOM_FILE_PROP = astring: \
                "$old_ipfconf" >/dev/null 2>&1
        fi
    fi

    svccfg -s $SMF_FMRI setprop $FW_CONFIG_DEF_PG/version = count: \
        "$CURRENT_VERSION" >/dev/null 2>&1
    svcadm refresh $SMF_FMRI >/dev/null 2>&1
}

But is isn’t clear to me how this gets called. There is no reference to it from the case statement – none of these options results in it being called (echo “(start|stop|reload|reipf|reipnat|pause|resume)”).

Paul.


January 25, 2011

Automated Installer 'How To'

This article will take you through the steps I followed recently to create my own Automated Installer for Solaris 11 Express. The new IPS packaging system and the Automated Installer are central features of Solaris 11 express that are destined to replace both Jumpstart and SVR4 packaging found in Solaris 10. Knowing this technology is a must for all Solaris 11 would-be System Admins.

Each Automated Installer client needs access to a repository (where all the software is kept and catalogued), an AI boot image and manifests which describe the client configuration and a DHCP server to begin the boot process.

AI Overview Diagram

Creating a local repository

Actually, this is optional, it is possible to simply use the default Oracle repository at http://pkg.oracle.com/solaris/release . In the real world, though, most organisations will want to keep their own local repository for performance or security reasons.

First, you need to create a ZFS filesystem to hold the spooled repository;

root@sol-esx01:~# zpool create repopool c8t1d0s2
root@sol-esx01:~# zfs set mountpoint=/repo repopool

The following 2 commands are optional – they enable ZFS deduplication and compression on the new pool;

root@sol-esx01:~# zfs set dedup=on repopool
root@sol-esx01:~# zfs set compression=on repopool

Next, create a lofi device from the .iso (you need to download Solaris 11 Express Repository Image from here ) and mount it in a convenient location:

root@sol-esx01:~#  lofiadm -a /var/spool/pkg/sol-11-exp-201011-repo-full.iso
root@sol-esx01:~#  mount -F hsfs /dev/lofi/1 /repocd

You can use rsync, as suggested in the AI documentation to copy the contents of this DVD to your local repository filesystem;

root@sol-esx01:~# rsync -aP /repocd/ /repo

Next, setup the parameters for your service and enable it;

root@sol-esx01:~# svccfg -s application/pkg/server setprop pkg/inst_root=/repo/repo
root@sol-esx01:~# svccfg -s application/pkg/server setprop pkg/readonly=true

root@sol-esx01:~# svcadm refresh application/pkg/server
root@sol-esx01:~# svcadm enable application/pkg/server

Set your publisher to the local host;

root@sol-esx01:~# pkg set-publisher -O http://localhost solaris

You will need to index/refresh the repository if you want the search to work….

root@sol-esx01:~# pkgrepo refresh -s /repo/repo

After that completes, search (and install) should work;

root@sol-esx01:~# pkg search xclock
INDEX           ACTION VALUE                                 PACKAGE
basename        file   usr/share/X11/app-defaults/XClock     pkg:/x11/xclock@1.0.4-0.151
basename        file   usr/bin/xclock                        pkg:/x11/xclock@1.0.4-0.151
basename        link   usr/X11/bin/xclock                    pkg:/x11/xclock@1.0.4-0.151
pkg.description set    xclock is the classic X Window Syst.. pkg:/x11/xclock@1.0.4-0.151
pkg.fmri        set    solaris/x11/xclock                    pkg:/x11/xclock@1.0.4-0.151
pkg.summary     set    xclock - analog / digital clock for X pkg:/x11/xclock@1.0.4-0.151

root@sol-esx01:~# pkg install xclock
               Packages to install:     1
           Create boot environment:    No
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                  1/1         6/6      0.0/0.0

PHASE                                        ACTIONS
Install Phase                                  43/43

PHASE                                          ITEMS
Package State Update Phase                       1/1
Image State Update Phase                         2/2
root@sol-esx01:~#

By default the publisher will listen on port 80;

root@sol-esx01:~# netstat -an| grep '\.80'
      *.80                 *.*                0      0 128000      0 LISTEN
root@sol-esx01:~# 

You should be able to check your work with ‘pkg publisher’

root@sol-esx01:~# pkg publisher
PUBLISHER                             TYPE     STATUS   URI
solaris                  (preferred)  origin   online   http://localhost/

That’s it, you are now running your own local publisher from your own local pkg server repository. Now onto the configuration of the automated installer.

Setting up the AI installer

This section moves step by step through the configuration of the Automated Installer.

First, create a zfs filesystem for the AI storage area;

root@sol-esx01:~# zfs create repopool/ai

If desired, turn on dedup, compression (again, both optional) and set your mountpoint;

root@sol-esx01:~# zfs get -r dedup repopool
NAME         PROPERTY  VALUE          SOURCE
repopool     dedup     on             local
repopool/ai  dedup     on             inherited from repopool

root@sol-esx01:~# zfs get -r compression repopool
NAME         PROPERTY     VALUE     SOURCE
repopool     compression  on        local
repopool/ai  compression  on        inherited from repopool

root@sol-esx01:~# zfs get mountpoint repopool/ai
NAME         PROPERTY    VALUE       SOURCE
repopool/ai  mountpoint  /repo/ai    inherited from repopool
root@sol-esx01:~# zfs set mountpoint=/ai repopool/ai

Copy the AI .iso into this filesystem, do not unpack it, the installadm command will do that for you.

root@sol-esx01:~# cp /var/spool/pkg/sol-11-exp-201011-ai-x86.iso /ai/
root@sol-esx01:~# ls /ai
sol-11-exp-201011-ai-x86.iso

Check your netmasks file contains the necessary, this is for DHCP to operate properly.

root@sol-esx01:~# vi /etc/netmasks
root@sol-esx01:~# tail -2 /etc/netmasks
#
192.168.100.0   255.255.255.0

The DNS multicast service is apparently required, although to be honest, I’ve never tried without it – so make sure this is enabled;

root@sol-esx01:~# svcs -a|grep dns
disabled       16:34:08 svc:/network/dns/install:default
disabled       16:34:10 svc:/network/dns/multicast:default
disabled       16:34:11 svc:/network/dns/server:default
online         16:34:20 svc:/network/dns/client:default

root@sol-esx01:~# svcadm enable dns/multicast
root@sol-esx01:~# svcs -a|grep dns

disabled       16:34:08 svc:/network/dns/install:default
disabled       16:34:11 svc:/network/dns/server:default
online         16:34:20 svc:/network/dns/client:default
online         21:26:48 svc:/network/dns/multicast:default

Running installadm

You can now run installadm to create your install service. Installadm has a really nice feature in that it will create the DHCP service for you if you want it to. It is possible to use a replacement DHCP service on the same server, or even a local DHCP service somewhere else on the subnet. The -i option tells installadm the starting address for your DHCP addresses and the -c option tells installadm how many addresses will be available for lease. The following example will offer addresses in the range 192.168.100.10 – 192.168.100.20.

root@sol-esx01:~# installadm create-service -n sol-11-exp-x86 -i 192.168.100.10 -c 10 \
    -s /ai/sol-11-exp-201011-ai-x86.iso /ai/sol-11-exp-x86-target
Setting up the target image at /ai/sol-11-exp-x86-target ...
Registering the service sol-11-exp-x86._OSInstall._tcp.local
Creating DHCP Server
Created DHCP configuration file.
Created dhcptab.
Added "Locale" macro to dhcptab.
Added server macro to dhcptab - sol-esx01.
DHCP server started.
Unable to determine the proper default router
or gateway for the 192.168.100.0 subnet. The default
router or gateway for this subnet will need to
be provided later using the following command:
   /usr/sbin/dhtadm -M -m 192.168.100.0 -e  Router=<address> -g
Added network macro to dhcptab - 192.168.100.0.
Created network table.
adding tftp to /etc/inetd.conf
Converting /etc/inetd.conf
copying boot file to /tftpboot/pxegrub.I86PC.Solaris-1
Service discovery fallback mechanism set up
Service discovery fallback mechanism set up

I forgot the default router in this example, but the router can easily be added as recommended in the output above;

root@sol-esx01:~# /usr/sbin/dhtadm -M -m 192.168.100.0 -e  Router=192.168.100.1 -g

Check the results;

root@sol-esx01:~# installadm list
Service Name   Status       Arch  Port  Image Path
------------   ------       ----  ----  ----------
sol-11-exp-x86 on           x86   46501 /ai/sol-11-exp-x86-target

Check DHCP too; I really like the fact that installadm setups up DHCP for you, sooo much pain saved.

root@sol-esx01:~# pntadm -P 192.168.100.0

Client ID       Flags   Client IP       Server IP       Lease Expiration  Macro           Comment

00              00      192.168.100.19  192.168.100.1   Zero              dhcp_macro_sol-11-exp-x86       
00              00      192.168.100.18  192.168.100.1   Zero              dhcp_macro_sol-11-exp-x86       
00              00      192.168.100.17  192.168.100.1   Zero              dhcp_macro_sol-11-exp-x86       
00              00      192.168.100.16  192.168.100.1   Zero              dhcp_macro_sol-11-exp-x86       
00              00      192.168.100.15  192.168.100.1   Zero              dhcp_macro_sol-11-exp-x86       
00              00      192.168.100.14  192.168.100.1   Zero              dhcp_macro_sol-11-exp-x86       
00              00      192.168.100.13  192.168.100.1   Zero              dhcp_macro_sol-11-exp-x86       
00              00      192.168.100.12  192.168.100.1   Zero              dhcp_macro_sol-11-exp-x86       
00              00      192.168.100.11  192.168.100.1   Zero              dhcp_macro_sol-11-exp-x86       
00              00      192.168.100.10  192.168.100.1   Zero              dhcp_macro_sol-11-exp-x86    

Creating a client

Creating a client is also achieved using the installadm command;

root@sol-esx01:~# installadm create-client -e 00:0c:29:05:31:75 -n sol-11-exp-x86
Service discovery fallback mechanism set up
Service discovery fallback mechanism set up

root@sol-esx01:~# installadm list
Service Name   Status       Arch  Port  Image Path
------------   ------       ----  ----  ----------
sol-11-exp-x86 on           x86   46501 /ai/sol-11-exp-x86-target

The -c option will list the client details;

root@sol-esx01:~# installadm list -c
Service Name   Client Address    Arch  Image Path
------------   --------------    ----  ----------
sol-11-exp-x86 00:0C:29:05:31:75 x86   /ai/sol-11-exp-x86-target

The installadm service comes with a default.xml that will serve any client that attempts to boot / install from this server. It is likely, of course that you will want to change this. There is also an example ‘static_network.xml’ in net_install_image_path/auto_install/sc_profiles that suggests a starting point for statically addressed clients and this can be copied and change as required. I copied this to sc_manifest1.xml (as described in the AI admin guide) and which is sourced as a follow up from ai.xml.

root@sol-esx01:/ai/sol-11-exp-x86-target/auto_install/sc_profiles# ls -lrth
total 7.0K
-r--r--r-- 1 root sys  4.7K 2010-11-05 14:13 static_network.xml
-rw-r--r-- 1 root root  788 2011-01-14 14:02 ai.xml
-r--r--r-- 1 root root 4.7K 2011-01-14 14:25 sc_manifest1.xml

Below is a summary of the changes I made, there are not really that many, I changed the default non-root user (because I don’t know jack, but I do recall that root is a role in Solaris 11 express, so you can’t login if you don’t), I took password hashes from an existing system to store as the root passwords, changed the hostname, network details and that was about it.

You should now create similar XML files for your clients, making sure you change the parameters as required. Below are the differences between my attempts and the default;

root@sol-esx01:/ai/sol-11-exp-x86-target/auto_install/sc_profiles# diff sc_manifest1.xml static_network.xml29,31c29,31
<                 <propval name="login" type="astring" value="cuspdx"/>
<                 <propval name="password" type="astring" value="$5$4X$wATzoqQD8pAxPErqESs3z0r9ypHkHeVsprsBgjmR3sD"/>
<                 <propval name="description" type="astring" value="Paul Eggleton"/>
---
>                 <propval name="login" type="astring" value="jack"/>
>                 <propval name="password" type="astring" value="9Nd/cwBcNWFZg"/>
>                 <propval name="description" type="astring" value="default_user"/>
40c40
<                 <propval name="password" type="astring" value="$5$Z7$GdeY1gSCF........jcuFI4"/>
---
>                 <propval name="password" type="astring" value="$5$VgppCOxA$ycFmY.......niNCouC"/>
46c46
<                 <propval name="hostname" type="astring" value="sol-esx02"/>
---
>                 <propval name="hostname" type="astring" value="solaris"/>
73c73
<                 <propval name='name' type='astring' value='e1000g0/v4'/>
---
>                 <propval name='name' type='astring' value='net0/v4'/>
75,76c75,76
<                 <propval name='static_address' type='net_address_v4' value='192.168.100.5/24'/>
<                 <propval name='default_route' type='net_address_v4' value='192.168.100.1'/>
---
>                 <propval name='static_address' type='net_address_v4' value='x.x.x.x/n'/>
>                 <propval name='default_route' type='net_address_v4' value='x.x.x.x'/>
80c80
<                 <propval name='name' type='astring' value='e1000g0/v6'/>
---
>                 <propval name='name' type='astring' value='net0/v6'/>
89c89
<         <instance name='default' enabled='false'>
---
>         <instance name='default' enabled='true'>
93c93
<                         <value_node value='192.168.100.1'/>
---
>                         <value_node value='x.x.x.x'/>

Following this, create the initial ai.xml. This specifies the boot device, a target name and publisher details (uses the local repository, not the Oracle one). Finally, it sources the sc_manifest1.xml mentioned above;

root@sol-esx01:/ai/sol-11-exp-x86-target/auto_install/sc_profiles# more ai.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE auto_install SYSTEM "file:///usr/share/auto_install/ai.dtd">
<auto_install>
  <ai_instance name="sol-esx02">
    <target>
      <target_device>
      <disk>
          <disk_keyword key="boot_disk"/>
      </disk>
      </target_device>
    </target>
 <software>
   <source>
   <publisher name="solaris">
     <origin name="http://192.168.100.1"/>
   </publisher>
   </source>
  <software_data action="install" type="IPS">
     <name>pkg:/server_install</name>
  </software_data>
  <software_data action="uninstall" type="IPS">
     <name>pkg:/server_install</name>
  </software_data>
 </software>
 <add_drivers>
    <search_all/>
 </add_drivers>
 <sc_manifest_file name="AI" URI="./sc_manifest1.xml"/>
 </ai_instance>
</auto_install>

To add a new manifest file, use installadm add-manifest as shown below. The -n specifies the service name and the -c allows you to specify criteria for matching clients that should use this manifest. Any clients that do not match the criteria are still catered for by the default.xml manifest which resides in net_install_image_path/auto_install/default.xml, in this case: /ai/sol-11-exp-x86-target/auto_install/default.xml

root@sol-esx01:~# installadm add-manifest -m ai.xml -n sol-11-exp-x86 -c MAC="00:0C:29:05:31:75"

The criteria above uses the client mac address, but there are other valid criteria with which to distinguish your clients. Criteria options include in addition to MAC address, the platform type (as per the platform type delivered from uname -i), the IPv4 address, the CPU architecture (such as ARCH=i86pc, think uname -p), or even system memory range values (think particular, cut-down installs for low memory machines). There is also -C option which expects an xml file containing the description of the criteria.

Example;

# installadm add-manifest -m ai.xml -n sol-11-exp-x86 -C /tmp/criteria.xml

Where criteria.xml may contain:

     <ai_criteria_manifest>

         <ai_criteria name=MAC>
             <value>00:0C:29:05:31:75</value>
         </ai_criteria>
     </ai_criteria_manifest>

The current criteria and manifests can be seen using installadm list -c and -m:

root@sol-esx01:/ai/sol-11-exp-x86-target/auto_install/sc_profiles# installadm list -c -m
Service Name   Client Address    Arch  Image Path
------------   --------------    ----  ----------
sol-11-exp-x86 00:0C:29:05:31:75 x86   /ai/sol-11-exp-x86-target

Service Name   Manifest
------------   --------
sol-11-exp-x86 target1.xml

Client boot

You should now be good to go, you can attempt a network boot from your client. Out of interest, the contents of the intended client grub menu will (should) be viewable in /tftpboot/menu.lst. ,

example;

root@sol-esx01:~# more /tftpboot/menu.lst.sol-11-exp-x86
default=0
timeout=30
min_mem64=1000
title Oracle Solaris 11 Express snv_151a boot image
      kernel$ /I86PC.Solaris-1/platform/i86pc/kernel/$ISADIR/unix -B install_media=http://192.168.100.1:5555/ai/sol-11-exp-x86-
target,install_service=sol-11-exp-x86,install_svc_address=192.168.100.1:46501
      module$ /I86PC.Solaris-1/platform/i86pc/$ISADIR/boot_archive
title Oracle Solaris 11 Express snv_151a Automated Install
      kernel$ /I86PC.Solaris-1/platform/i86pc/kernel/$ISADIR/unix -B install=true,install_media=http://192.168.100.1:5555/ai/so
l-11-exp-x86-target,install_service=sol-11-exp-x86,install_svc_address=192.168.100.1:46501
      module$ /I86PC.Solaris-1/platform/i86pc/$ISADIR/boot_archive
root@sol-esx01:~# 

The client should fairly quickly receive a PXE boot offer and display the grub boot menu with an option for Automated Install. The client managed to transfer and boot the miniroot.

The lease / DHCP details can be viewed with pntadm to confirm the address offered and the lease duration;

root@sol-esx01:~# pntadm -P 192.168.100.0

Client ID       Flags   Client IP       Server IP       Lease Expiration   Macro           Comment

01000C29053175  00      192.168.100.19  192.168.100.1   01/15/2011         dhcp_macro_sol-11-exp-x86       
00              00      192.168.100.18  192.168.100.1   Zero               dhcp_macro_sol-11-exp-x86       
00              00      192.168.100.17  192.168.100.1   Zero               dhcp_macro_sol-11-exp-x86       
00              00      192.168.100.16  192.168.100.1   Zero               dhcp_macro_sol-11-exp-x86       
00              00      192.168.100.15  192.168.100.1   Zero               dhcp_macro_sol-11-exp-x86       
00              00      192.168.100.14  192.168.100.1   Zero               dhcp_macro_sol-11-exp-x86       
00              00      192.168.100.13  192.168.100.1   Zero               dhcp_macro_sol-11-exp-x86       
00              00      192.168.100.12  192.168.100.1   Zero               dhcp_macro_sol-11-exp-x86       
00              00      192.168.100.11  192.168.100.1   Zero               dhcp_macro_sol-11-exp-x86       
00              00      192.168.100.10  192.168.100.1   Zero               dhcp_macro_sol-11-exp-x86 

Choose the Automated Install option when you see the grub menu. You should see the client dhcp boot, download the miniroot boot image and begin installing. Snoop will, of course show you the HTTP traffic during install.

root@sol-esx01:~# snoop -d e1000g1
Using device e1000g1 (promiscuous mode)
   sol-esx01 -> 192.168.100.19 HTTP HTTP/1.1 200 OK
   sol-esx01 -> 192.168.100.19 HTTP (body)
   sol-esx01 -> 192.168.100.19 HTTP (body)
192.168.100.19 -> sol-esx01    HTTP C port=54454
   sol-esx01 -> 192.168.100.19 HTTP HTTP/1.1 200 OK

After a fairly short period of time, and ZERO interaction, you should have your newly installed client available to use

cuspdx@sol-esx02:~$ hostname
sol-esx02
cuspdx@sol-esx02:~$ uname -a
SunOS sol-esx02 5.11 snv_151a i86pc i386 i86pc Solaris
cuspdx@sol-esx02:~$
cuspdx@sol-esx02:~$
cuspdx@sol-esx02:~$ uptime
 22:11pm  up 5:02,  1 user,  load average: 0.00, 0.00, 0.00
cuspdx@sol-esx02:~$
cuspdx@sol-esx02:~$
cuspdx@sol-esx02:~$ df -h
Filesystem            Size  Used Avail Use% Mounted on
rpool/ROOT/solaris     17G  2.6G   14G  17% /
swap                  1.4G  396K  1.4G   1% /etc/svc/volatile
/usr/lib/libc/libc_hwcap1.so.1
                       17G  2.6G   14G  17% /lib/libc.so.1
swap                  1.4G  4.0K  1.4G   1% /tmp
swap                  1.4G   40K  1.4G   1% /var/run
rpool/export           14G   32K   14G   1% /export
rpool/export/home      14G   32K   14G   1% /export/home
rpool/export/home/cuspdx
                       14G   34K   14G   1% /export/home/cuspdx
rpool                  14G   93K   14G   1% /rpool

Something that is worthy of note, is the fact that the software distribution is far from the burgeoning megapack that was included in Solaris 10. For example, the SUNWCXall distribution set for Solaris 10 update 9 consisted of around 6gb of software when installed.

-bash-3.00$ df -h
Filesystem             size   used  avail capacity  Mounted on
rpool/ROOT/s10u9       134G   6.0G    72G     8%    /

Whereas the Solaris 11 Express install contains just 2.6gb

Filesystem            Size  Used Avail Use% Mounted on
rpool/ROOT/solaris     17G  2.6G   14G  17% /

This is a good thing, because the majority of software installed under Solaris 10 would have never been used, and is simply there because of the coarse grained package distribution model offered by the installer. Minimisation in Solaris 10 with SVR4 packaging was difficult to achieve or to implement with confidence knowing that everything will work as expected and be patchable with expected results. Less software means less scope for security vulnerabilities to be introduced. Less software is also likely to mean fewer unnecessary daemons running, less memory consumed and better overall performance. In addition, a dataset this size will be much easier to maintain, backup and replicate, consuming both less space and time.

In addition, if you find you need some of the missing software, you can simply use the pkg framework to search the repository and install the software you need AND any required dependencies in one fell swoop. For example, gcc3 can be installed and a couple of prerequisite packages such as lint by simply typing ‘pkg install gcc-3’ ;

root@sol-esx01:~# pkg install gcc-3               Packages to install:     3
           Create boot environment:    No
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                  3/3     779/779    30.1/30.1

PHASE                                        ACTIONS
Install Phase                              1214/1214

PHASE                                          ITEMS
Package State Update Phase                       3/3
Image State Update Phase                         2/2
root@sol-esx01:~#
 

In summary then, AI is not all that difficult to configure, it is incredibly easy to use once setup and the benefits for package and system management are huge. In this article I’ve shown you how to

  • Create your own local repository area (standard ZFS commands)
  • Create a spool of the repo media (lofiadm) and set up the pkg/server SMF service (using svcadm and svccfg properties).
  • Set the default publisher to use this repository (pkg set-publisher)
  • Create a ZFS area for the auto installer (standard ZFS commands)
  • Create an install service (installadm create-service)
  • Creat a manifest and apply the XML manifest (installadm add-manifest) to a client
  • Install that client in a hands-off, automated but custom manner straight out of the box.
  • Described some of the benefits over SVR4 of the IPS / AI model

The real power and flexibility will come in the customisation of clients using multiple xml manifests, and matching these in pretty much any way you please to clients within your environment. I think it is fair to say though, it is easy to see the features and benefits that Auto Install and the new IPS repository based packaging tools will bring to Solaris 11. What we all need to work on in the short term, to get the most out of this technology is the transition from Solaris 10, Jet, Jumpstart, flash archives and other familiar technologies to IPS and AI.

I hope you found this useful, comments welcome.

Paul.


January 19, 2011

Failed Disk in x4500 and hdtool

Writing about web page http://download.oracle.com/docs/cd/E19962-01/820-1120-22/chapter2.html#d0e1462

Today we had a disk failure in one of the x4500s. These machines hold 48 500gb SATA disk drives, and the units are what Oracle term a CRU – Customer Replaceable Units. Which, of course means there has to be a straightforward and reliable way to identify the disk location of any given disk from the information given in the messages file. There is a very useful tool on the ‘x4500 Tools and Drivers CD” called ‘hd’ or hdtool.

The Tools and Drivers CD can be downloaded from My Oracle Support (MOS), under Patches and Updates, search using “Product or Family”, set the product to seach for to ‘x4500’ and you should find the Tools and Drivers CD in the list.

Once downloaded, unzip into /var/spool/pkg and find the .iso contained within;

bash-3.00# unzip p10335199_160_Generic.zip
Archive:  p10335199_160_Generic.zip
   creating: Tools_and_Drivers/
  inflating: Tools_and_Drivers/X4500_Tools_And_Drivers_common_47001.zip 
  inflating: Tools_and_Drivers/license_agreement1.html 
  inflating: Tools_and_Drivers/MD5SUM-SoftwareAndDocumentation.txt 
  inflating: Tools_and_Drivers/readme.html 
  inflating: Tools_and_Drivers/X4500_Tools_And_Drivers_linux_47001.tar.bz2 
  inflating: Tools_and_Drivers/X4500_Tools_And_Drivers_windows_47001.zip 
  inflating: Tools_and_Drivers/X4500_Tools_And_Drivers_read_me.html 
  inflating: Tools_and_Drivers/X4500_Tools_And_Drivers_solaris_47001.tar.bz2 
  inflating: Tools_and_Drivers/X4500_Tools_And_Drivers_CD_47001.iso  

Lofi mount it;

bash-3.00# lofiadm -a /var/spool/pkg/Tools_and_Drivers/X4500_Tools_And_Drivers_CD_47001.iso
/dev/lofi/1
bash-3.00# mount -F hsfs /dev/lofi/1 /mnt
bash-3.00# cd /mnt

Then pkgadd from mnt/solaris/tools/hdtool/SUNWhd-1.07.pkg

bash-3.00# pkgadd -d SUNWhd-1.07.pkg
The following packages are available:
  1  SUNWhd     Sun Fire X4500/X4540 Hard Disk Suite
                (i386) 1.07

Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [?,??,q]:

Processing package instance <SUNWhd> from </mnt/solaris/tools/hdtool/SUNWhd-1.07.pkg>

Sun Fire X4500/X4540 Hard Disk Suite(i386) 1.07
Copyright 2007 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Using </opt> as the package base directory.
## Processing package information.
## Processing system information.
## Verifying package dependencies.
## Verifying disk space requirements.
## Checking for conflicts with packages already installed.
## Checking for setuid/setgid programs.

This package contains scripts which will be executed with super-user
permission during the process of installing this package.

Do you want to continue with the installation of <SUNWhd> [y,n,?] y

Installing Sun Fire X4500/X4540 Hard Disk Suite as <SUNWhd>

## Installing part 1 of 1.
/opt/SUNWhd/hd/bin/hd
/opt/SUNWhd/hd/bin/hd.html
/opt/SUNWhd/hd/bin/hdadm
/opt/SUNWhd/hd/bin/hdadm.html
/opt/SUNWhd/hd/bin/read_cache
/opt/SUNWhd/hd/bin/write_cache
[ verifying class <none> ]
## Executing postinstall script.

Installation of <SUNWhd> was successful.
bash-3.00#

You can now use this tool to view information regarding the disks in your system;

bash-3.00# ./hd -?./hd: illegal option -- ?
 Usage: hd [ -c(olor mode) ] [ -s(ummary) ] [ -p(latform) ]
 [ -b(ypass) to print SunFireX4500 map ]
 [ -d(iagnose)] [-f { syslog_file } ]
 [ -m { adjacent | cross | front2back | diagonal } Mapping pairs ]
 [ -w { <pci_disk_device_path> } ]
 [ -a (fdisk pArtition type) ]
 [ -q (list drive slot number in seQuential list) ]
 [ -g (list drive slot number in seQuential list with temperature ) ]
 [ -l (List SunFireX4500/X4540 available disk in physical orders) ]
 [ -r (List SMART data for all disks in drive slot number) ]
 [ -R (List SMART data's indivdual id in landscape view for all disks) ]
 [ -e <cXtY> (List SMART data for specified disk) ]
 [ -E <cXtY> (List raw hex SMART data for specified disk) ]
 [ -j (List SunFireX4500/X4540 HBA controller numbers and pci nodes) ]
 [ -T (List vtoc for all drives for SunFireX4500/X4540 platform) ]
 [ -t (List vtoc for specified drives) ]
 [ -i (List cXtY, sd# and PCI path) ]
 [ -o (List LSI HBA#, Drive Target# and cXtY) ]
 [ -x (Generate hd_map.html) ]

The HD map it generates, includes, amongst other detail a simple way of showing the status of each drive:

++: Device is present and accessible.
Red: Device not enumerated or no drive in physical slot/location.
--: Device is not accessible, absent/empty or down.

For example, the following shows the failure of c4t4;

---------------------SunFireX4500------Rear----------------------------
36:   37:   38:   39:   40:   41:   42:   43:   44:   45:   46:   47:  
c3t3  c3t7  c2t3  c2t7  c5t3  c5t7  c4t3  c4t7  c1t3  c1t7  c0t3  c0t7 
^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++  
24:   25:   26:   27:   28:   29:   30:   31:   32:   33:   34:   35:  
c3t2  c3t6  c2t2  c2t6  c5t2  c5t6  c4t2  c4t6  c1t2  c1t6  c0t2  c0t6 
^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++  
12:   13:   14:   15:   16:   17:   18:   19:   20:   21:   22:   23:  
c3t1  c3t5  c2t1  c2t5  c5t1  c5t5  c4t1  c4t5  c1t1  c1t5  c0t1  c0t5 
^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++  
 0:    1:    2:    3:    4:    5:    6:    7:    8:    9:   10:   11:  
c3t0  c3t4  c2t0  c2t4  c5t0  c5t4  c4t0  c4t4  c1t0  c1t4  c0t0  c0t4 
^b+   ^b+   ^++   ^++   ^++   ^++   ^++   ^--   ^++   ^++   ^++   ^++  
-------*-----------*-SunFireX4500--*---Front-----*-----------*----------

More complete documentation can be found here

bash-3.00# cd /opt/SUNWhd/hd/bin/
bash-3.00# ./hd

platform = Sun Fire X4500

Device    Serial        Vendor   Model             Rev  Temperature    
------    ------        ------   -----             ---- -----------    
c0t4d0p0  F400P6G4ES5F  ATA      HITACHI HUA7250S  A90A 26 C (78 F)
c0t3d0p0  F400P6G4XGYF  ATA      HITACHI HUA7250S  A90A 32 C (89 F)
c5t6d0p0  F400P6G4VU4F  ATA      HITACHI HUA7250S  A90A 30 C (86 F)
c5t1d0p0  F400P6G4LH8F  ATA      HITACHI HUA7250S  A90A 28 C (82 F)
c1t4d0p0  F400P6G50N6F  ATA      HITACHI HUA7250S  A90A 26 C (78 F)
c1t3d0p0  A570P6G4LSBF  ATA      HITACHI HUA7250S  A90A 31 C (87 F)
c6t7d0p0  F400P6G4WKDF  ATA      HITACHI HUA7250S  A90A 31 C (87 F)
c6t0d0p0  F500P6G4VJDF  ATA      HITACHI HUA7250S  A90A 25 C (77 F)
c4t6d0p0  F400P6G4XX2F  ATA      HITACHI HUA7250S  A90A 32 C (89 F)
c4t1d0p0  F400P6G4ZH6F  ATA      HITACHI HUA7250S  A90A 30 C (86 F)
c3t5d0p0  F400P6G4RBWF  ATA      HITACHI HUA7250S  A90A 29 C (84 F)
c3t2d0p0  F400P6G4X16F  ATA      HITACHI HUA7250S  A90A 31 C (87 F)
c5t0d0p0  F400P6G0JWTF  ATA      HITACHI HUA7250S  A90A 26 C (78 F)
c5t7d0p0  A570P6G4G25F  ATA      HITACHI HUA7250S  A90A 31 C (87 F)
c0t2d0p0  F400P6G4UY0F  ATA      HITACHI HUA7250S  A90A 30 C (86 F)
c0t5d0p0  F400P6G4KVDF  ATA      HITACHI HUA7250S  A90A 30 C (86 F)
c3t3d0p0  A570P6G4LA4F  ATA      HITACHI HUA7250S  A90A 32 C (89 F)
c3t4d0p0  F500P6G4XWYF  ATA      HITACHI HUA7250S  A90A 26 C (78 F)
c4t0d0p0  F500P6G531WF  ATA      HITACHI HUA7250S  A90A 27 C (80 F)
c4t7d0p0  A570P6G4R5VF  ATA      HITACHI HUA7250S  A90A 32 C (89 F)
c6t1d0p0  F400P6G4WW1F  ATA      HITACHI HUA7250S  A90A 28 C (82 F)
c6t6d0p0  F400P6G4XEHF  ATA      HITACHI HUA7250S  A90A 30 C (86 F)
c1t2d0p0  F400P6G4Z54F  ATA      HITACHI HUA7250S  A90A 30 C (86 F)
c1t5d0p0  F400P6G4SL0F  ATA      HITACHI HUA7250S  A90A 28 C (82 F)
c3t7d0p0  F400P6G4MAZF  ATA      HITACHI HUA7250S  A90A 32 C (89 F)
c3t0d0p0  F500P6G5538F  ATA      HITACHI HUA7250S  A90A 27 C (80 F)
c4t4d0p0  F400P6G0JSGF  ATA      HITACHI HUA7250S  A90A 28 C (82 F)
c4t3d0p0  F400P6G4YUUF  ATA      HITACHI HUA7250S  A90A 34 C (93 F)
c6t5d0p0  F400P6G4SLEF  ATA      HITACHI HUA7250S  A90A 27 C (80 F)
c6t2d0p0  F400P6G4Z4KF  ATA      HITACHI HUA7250S  A90A 30 C (86 F)
c1t6d0p0  F400P6G4X5BF  ATA      HITACHI HUA7250S  A90A 30 C (86 F)
c1t1d0p0  F400P6G0JYUF  ATA      HITACHI HUA7250S  A90A 29 C (84 F)
c5t4d0p0  F400P6G42U3F  ATA      HITACHI HUA7250S  A90A 26 C (78 F)
c5t3d0p0  F400P6G4X8KF  ATA      HITACHI HUA7250S  A90A 31 C (87 F)
c0t6d0p0  F400P6G4X9XF  ATA      HITACHI HUA7250S  A90A 32 C (89 F)
c0t1d0p0  F400P6G4KGYF  ATA      HITACHI HUA7250S  A90A 29 C (84 F)
c1t0d0p0  F400P6G4XVSF  ATA      HITACHI HUA7250S  A90A 25 C (77 F)
c1t7d0p0  F400P6G49X1F  ATA      HITACHI HUA7250S  A90A 32 C (89 F)
c6t3d0p0  A570P6G4G5LF  ATA      HITACHI HUA7250S  A90A 31 C (87 F)
c6t4d0p0  F400P6G4W3TF  ATA      HITACHI HUA7250S  A90A 25 C (77 F)
c4t2d0p0  F400P6G4WK6F  ATA      HITACHI HUA7250S  A90A 32 C (89 F)
c4t5d0p0  F400P6G4UHMF  ATA      HITACHI HUA7250S  A90A 29 C (84 F)
c3t1d0p0  F400P6G0K0HF  ATA      HITACHI HUA7250S  A90A 29 C (84 F)
c3t6d0p0  F400P6G4U62F  ATA      HITACHI HUA7250S  A90A 30 C (86 F)
c0t0d0p0  F400P6G0JVZF  ATA      HITACHI HUA7250S  A90A 26 C (78 F)
c0t7d0p0  F400P6G4X75F  ATA      HITACHI HUA7250S  A90A 32 C (89 F)
c5t2d0p0  F400P6G4U47F  ATA      HITACHI HUA7250S  A90A 30 C (86 F)
c5t5d0p0  F400P6G4N7LF  ATA      HITACHI HUA7250S  A90A 27 C (80 F)

---------------------SunFireX4500------Rear----------------------------

36:   37:   38:   39:   40:   41:   42:   43:   44:   45:   46:   47:  
c4t3  c4t7  c3t3  c3t7  c6t3  c6t7  c5t3  c5t7  c1t3  c1t7  c0t3  c0t7 
^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++  
24:   25:   26:   27:   28:   29:   30:   31:   32:   33:   34:   35:  
c4t2  c4t6  c3t2  c3t6  c6t2  c6t6  c5t2  c5t6  c1t2  c1t6  c0t2  c0t6 
^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++  
12:   13:   14:   15:   16:   17:   18:   19:   20:   21:   22:   23:  
c4t1  c4t5  c3t1  c3t5  c6t1  c6t5  c5t1  c5t5  c1t1  c1t5  c0t1  c0t5 
^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++  
 0:    1:    2:    3:    4:    5:    6:    7:    8:    9:   10:   11:  
c4t0  c4t4  c3t0  c3t4  c6t0  c6t4  c5t0  c5t4  c1t0  c1t4  c0t0  c0t4 
^b+   ^b+   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++   ^++  
-------*-----------*-SunFireX4500--*---Front-----*-----------*----------

As for the disk failure, I have logged a call with Oracle and am waiting to hear back from dispatch. Fortunately, the disk that failed was being used as a SPARE drive in the main datapool. We have 4 other SPARES at the moment, so replacement isn’t urgent, although we are running with slightly lower resilience to failure in the meantime.

Paul.


November 25, 2010

IPMP and IP Filter


So here is the problem.  You need to configure IP Filter on Solaris to meet the needs of your security policy but you also need to configure IPMP to provide resiliency against network port failure.  The default configuration for IP Filter on Solaris refers to the ingress or egress interface by device name in the ipf.conf file.   You may have, for example an entry like this:

pass in log first level local0.info quick on bge0 proto tcp from 10.13.1.2/32 to any flags S keep state


The problem is, of course that when using IPMP the system will, when required, fail to a different interface. Say, e1000g0.  Of course the ipmp group may *contain* bge0, but we need to be sure we are going to cater successfully for the case where ipmp fails over to the partner interface.  One option would be to duplicate all your IP Filter rules for every interface in the IPMP group.  Clearly not sensible, nor necessary; the solution (post Solaris 10u2) is to use the ipmp_hook_emulation ndd setting for the /dev/ip driver.  For versions less than or equal to S10u2 go and research the /dev/pfil setting qif_ipmp_set.

First, we need to set ipmp_hook_emulation for the /dev/ip driver to 1.

bash-3.00# /usr/sbin/ndd -get /dev/ip ipmp_hook_emulation
0
bash-3.00# /usr/sbin/ndd -set /dev/ip ipmp_hook_emulation 1
bash-3.00# /usr/sbin/ndd -get /dev/ip ipmp_hook_emulation
1

Once that is one, we change the interface name in ipf.conf to the ipmp groupname;

Check what it is with ifconfig:

bge0: flags=89000842<BROADCAST,RUNNING,MULTICAST,IPv4,NOFAILOVER,OFFLINE> mtu 0 index 2
groupname app
e1000g0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
groupname app
e1000g0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3

Set appropriately in ipf.conf:

pass in log first level local0.info quick on app proto tcp from 10.13.1.2/32 to any flags S keep state

Now, restart ipfilter and monitor the logfile.  We see the firewall acting on entries for the IPMP group, rather than by specific interface.  Now, if mpathd fails over the interface, the firewall continues to operate as expected.


Nov 24 14:49:00 test-serva ipmon[1160]: [ID 702911 local0.info] 14:48:59.955694 app0 @0:9 p 10.13.1.2,123 -> 10.13.1.5,123 PR udp len 20 76 K-S IN

The second part to this solution is to make the changes persistent.

We can make sure ipfilter will be restarted persistently on boot by enabling the SMF service 'network/ipfilter'. The ndd setting however, will not persist.  How do we set the ndd setting on boot?  One option is a legacy RC script, but this does not leverage the dependency framework of SMF to ensure that the setting is applied before the ipfilter service is started. 

Creating an SMF service to do this that sets itself as a dependent to ipfilter is quite straight forward.  The nice thing about SMF is that you can make ipfilter depend on the new tuning service *without* changing the setup of ipfilter.  To do this, create a new service that configures the ndd parameter(s) you are interested in and set the 'network/ipfilter' service and a *dependent* in the XML manifest;

First, a manifest is required, so create /var/svc/manifest/site/ndd-nettune.xml; notice the dependent ipfilter entry.

<?xml version="1.0"?>

<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">

<!--

ident    "@(#)ndd-nettune.xml    1.0    04/09/21 SMI"

-->


<service_bundle type='manifest' name='SUNWcsr:ndd'>


<service

name='network/ndd-nettune'

type='service'

version='1'>


                       <create_default_instance enabled='true' />


                       <single_instance />


           <dependency

   name='fs-minimal'

   type='service'

   grouping='require_all'

   restart_on='none'>

                   <service_fmri value='svc:/system/filesystem/minimal' />

           </dependency>


               <dependency

   name='loopback-network'

   grouping='require_any'

   restart_on='none'

   type='service'>

                   <service_fmri value='svc:/network/loopback' />

           </dependency>


               <dependency

   name='physical-network'

   grouping='optional_all'

   restart_on='none'

   type='service'>

                   <service_fmri value='svc:/network/physical' />

           </dependency>

           

         <dependent

   name='ipfilter-dep'

   grouping='require_all'

   restart_on='none'>

                   <service_fmri value='svc:/network/ipfilter' />

           </dependent>


               <exec_method

   type='method'

   name='start'

   exec='/lib/svc/method/ndd-nettune'

   timeout_seconds='3' >

           </exec_method>


               <exec_method

          type='method'

          name='stop'

          exec=':true'

          timeout_seconds='3' >

           </exec_method>


               <property_group name='startd' type='framework'>

                   <propval name='duration' type='astring'  value='transient' />

           </property_group>


               <stability value='Unstable' />


               <template>

                   <common_name>

                           <loctext xml:lang='C'>

                                   ndd network tuning

                           </loctext>

                   </common_name>

                   <documentation>

                           <manpage title='ndd' section='1M'

           manpath='/usr/share/man' />

                   </documentation>

           </template>


</service>


</service_bundle>



Run xmllint on the file to check it out:

xmllint /var/svc/manifest/site/ndd-nettune.xml 


Validate it with svccfg

svccfg validate /var/svc/manifest/site/ndd-nettune.xml


Create the script that it will run

bash-3.00# vi /lib/svc/method/ndd-nettune


"/lib/svc/method/ndd-nettune" 15 lines, 453 characters
#!/sbin/sh
#

. /lib/svc/share/smf_include.sh
. /lib/svc/share/net_include.sh

# Make sure that the libraries essential to this stage of booting  can be found.
LD_LIBRARY_PATH=/lib; export LD_LIBRARY_PATH
echo "Performing IP Tuning..." >> /tmp/smf.out
/usr/sbin/ndd -set /dev/ip ipmp_hook_emulation 1


# Reset the library path now that we are past the critical stage
unset LD_LIBRARY_PATH
~
~
~


Import the new service:

svccfg import /var/svc/manifest/site/ndd-nettune.xml 


Enable if necessary:

svcadm enable ndd-nettune


Check again that the ipmp_hook_emulation is set:

ndd -get /dev/ip ipmp_hook_emulation


Do some brief testing;


Check the value of ipmp_hook_emulation;

bash-3.00# ndd -get /dev/ip ipmp_hook_emulation
1
bash-3.00#

Check IP Filter and the ndd service are running;

bash-3.00# svcs ipfilter
online         14:45:45 svc:/network/ipfilter:default
bash-3.00# svcs ndd-nettune
online         14:45:45 svc:/network/ndd-nettune:default

Check the new service restarts ok:

bash-3.00# svcadm restart ndd-nettune
bash-3.00# svcs -a |grep nett
online         14:47:48 svc:/network/ndd-nettune:default

And that the ipfilter service can restart:


bash-3.00# svcadm restart ipfilter   

Disable ndd-nettune;

bash-3.00# svcadm disable ndd-nettune

And attempt to restart ipfilter;

bash-3.00# svcadm restart ipfilter

Excellent, it won't start because the dependent isn't met.

offline        14:48:03 svc:/network/ipfilter:default
bash-3.00# svcs -x
svc:/network/ndd-nettune:default (ndd network tuning)
State: disabled since 24 November 2010 14:48:02 GMT
Reason: Disabled by an administrator.
See: http://sun.com/msg/SMF-8000-05
See: ndd(1M)
See: /var/svc/log/network-ndd-nettune:default.log
Impact: 1 dependent service is not running.  (Use -v for list.)
bash-3.00#

Finally, enable ndd-nettune and ensure ipfilter now starts.

bash-3.00# svcadm enable ndd-nettune
bash-3.00# svcadm restart ipfilter
bash-3.00#
bash-3.00# svcs -x
bash-3.00#


So, we have a method to run both IPMP for network resiliency and IP filter for network security.  The method is clean, easy and persistent across reboots.  This functionality was introduced in Solaris 10 update 3, but I should say that it is changing again.  The next release(s) of Solaris will see the introduction of the Clearview project and the IPMP implementation will create virtual interfaces, such as ipmp0, ipmp1 which can be treated just like any other interfaces for the purposes of technologies such as ipfilter.


Paul.


November 19, 2010

Installing Perl Modules (perl DBI / DBD::Oracle) on Solaris

If you have ever tried to install perl modules on Solaris, then you’ll have shared my pain. Solaris, rather helpfully, comes with its own installation of perl, straight off the DVD. You get an install of something like 5.8.4, depending on your version of Solaris.

bash-3.00# which perl
/usr/bin/perl
bash-3.00# perl -v

This is perl, v5.8.4 built for sun4-solaris-64int
(with 32 registered patches, see perl -V for more detail)

Copyright 1987-2004, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using `man perl' or `perldoc perl'.  If you have access to the
Internet, point your browser at http://www.perl.com/, the Perl Home Page.

bash-3.00# 

Marvellous. However, there is a snag. If you use perl in anger you will want to update some of the perl modules. Or even install some new ones that aren’t included by default. I had to climb this particular hill recently when I needed the perl DBI module to access a database, more specifically, the DBD::Oracle module to access an Oracle database.

The most straight forward and popular method for installing perl modules is to use CPAN. Whether you use CPAN or compile by hand, the first, not so small issuette you will hit is that the perl in Solaris is compiled with the Sun Studio compilers.

There is a solution for this that means you can use the GNU compiling tools under /usr/sfw, but it does take a little tweaking to get things straight. The solution is to use /usr/perl5/bin/perlgcc -MCPAN -e shell. This will fire up a CPAN shell for you to use. Below is an account of the configuration tweaks I had to do to get this CPAN shell to download, compile and install my modules.

The first time you run it, you’ll be asked a bewilering array of questions. Do not let it autoconfigure, instead answer [yes] to being ready for manual configuration. You are ready. You just don’t know it yet. Fortunately for the most part, the defaults will work, except for the paths / settings below. If a path to a binary is missing (lynx for example), leave it blank, you won’t need it.

Where is your tar program? [/usr/sbin/tar] /usr/sfw/bin/gtar

Warning: make not found in PATH
Where is your make program? [] /usr/sfw/bin/gmake

Warning: wget not found in PATH
Where is your wget program? [] /usr/sfw/bin/wget
Warning: ncftpget not found in PATH
Where is your ncftpget program? [] 
Warning: ncftp not found in PATH

The configuration tool will ask you to input a URL for downloads, for some reason it only shows ftp URLs even though you set the wget location. You are forced to accept one and remove it later (and add an HTTP one) even if you don’t want to use ftp.

(1) ftp://cpan.etla.org/pub/CPAN
(2) ftp://ftp.demon.co.uk/pub/CPAN/
(3) ftp://ftp.mirror.8086.net/sites/CPAN/
(4) ftp://ftp.mirror.anlx.net/CPAN/
(5) ftp://ftp.mirrorservice.org/sites/ftp.funet.fi/pub/languages/perl/CPAN/
(6) ftp://ftp.plig.net/pub/CPAN/
(7) ftp://ftp.ticklers.org/pub/CPAN/
(8) ftp://mirror.bytemark.co.uk/CPAN/
(9) ftp://mirror.ox.ac.uk/sites/www.cpan.org/
(10) ftp://mirror.sov.uk.goscomb.net/pub/CPAN/
(11) ftp://mirror.tje.me.uk/pub/mirrors/ftp.cpan.org/
(12) ftp://mirrors.uk2.net/pub/CPAN/
Select as many URLs as you like (by number),
put them on one line, separated by blanks, e.g. '1 4 5' [] 

After that, use ‘o conf’ to check your settings:

cpan> o conf     
CPAN::Config options from /usr/perl5/5.8.4/lib/CPAN/Config.pm:
    commit             Commit changes to disk
    defaults           Reload defaults from disk
    init               Interactive setting of all options

    build_cache        10
    build_dir          /.cpan/build
    cache_metadata     1
    cpan_home          /.cpan
    dontload_hash     
    ftp                "" 
    ftp_proxy          
    getcwd             cwd
    gpg                
    gzip               /usr/bin/gzip
    histfile           /.cpan/histfile
    histsize           100
    http_proxy         
    inactivity_timeout 0
    index_expire       1
    inhibit_startup_message 0
    keep_source_where  /.cpan/sources
    lynx               
    make               /usr/sfw/bin/gmake
    make_arg           
    make_install_arg   
    makepl_arg         
    ncftp              
    ncftpget           
    no_proxy           
    pager              /usr/bin/less
    prerequisites_policy ask
    scan_cache         atstart
    shell              /sbin/sh
    tar                /usr/sfw/bin/gtar
    term_is_latin      1
    unzip              /usr/bin/unzip
    urllist           
        ftp://mirror.ox.ac.uk/sites/www.cpan.org/
    wget               /usr/sfw/bin/wget

Notice that there is no entry for cc/gcc; you have to configure this through the shell environment, so make sure that a suitable compiler is in your PATH (/usr/sfw/bin/gcc);

To fix that url entry, use ‘o conf urllist shift’ to remove the bad entry, and o conf urllist unshift [new url] to add the one you need:

cpan> o conf urllist shift

cpan> o conf urllist
    urllist           
Type 'o conf' to view configuration edit options

cpan> o conf urllist unshift http://mirror.ox.ac.uk/sites/www.cpan.org/
    urllist           
Type 'o conf' to view configuration edit options

cpan> o conf urllist
    urllist           
        http://mirror.ox.ac.uk/sites/www.cpan.org/
Type 'o conf' to view configuration edit options

The basic rule for the install of the DBD::Oracle module install is that you have to be able to connect to a test database using a test user from the shell where you launch the CPAN tool. This is because CPAN will use your current environment to connect to a database when it runs ‘make test’. You really MUST read the README that is supplied with the module because some elements may not be correct for your build, but here is my experience (I didn’t really use scott/tiger, but you get the idea):

Set environment variables, you also need to make sure something sensible is set in your tnsnames.ora. You can also use the TWO_TASK variable, the README suggests this and it seems to work. This doesn’t remove the need for a valid tnsnames.ora, of course.

bash-3.00# export LD_LIBRARY_PATH=/app/oracle/product/10.2.0/db_1/lib:/usr/dt/lib:/app/oracle/product/10.2.0/db_1/jdbc/lib:/app/oracle/product/10.2.0/db_1/lib32
bash-3.00# export ORACLE_USERID=scott/tiger
bash-3.00# export PATH=/usr/sbin:/usr/bin:/usr/local/oracle/product/10.2.0/db_1/bin:/usr/sfw/bin
bash-3.00# 
bash-3.00# export ORACLE_HOME=/app/oracle/product/10.2.0/db_1

Make sure your tools look good… ahem.

bash-3.00# which gmake
/usr/sfw/bin/gmake
bash-3.00# which gcc
/usr/sfw/bin/gcc

Run the CPAN shell again and install your module;

bash-3.00# /usr/perl5/bin/perlgcc -MCPAN -e shell
Terminal does not support AddHistory.

cpan shell -- CPAN exploration and modules installation (v1.7601)
ReadLine support available (try 'install Bundle::CPAN')

cpan> install DBD::Oracle
CPAN: Storable loaded ok
LWP not available

....
....

If you get this error:

t/10general.............Can't locate object method "no_diag" via package "Test::Builder" at t/nchar_test_lib.pl line 53.

You probably don’t have the correct version of Test::Builder. You can check this in the perldoc Test::Builder on your system; look for no_diag. The fix, of course is to install the new version using your CPAN Shell:

cpan> install Test::Builder
Running install for module Test::Builder
Running make for M/MS/MSCHWERN/Test-Simple-0.96.tar.gz

....

Installing /usr/perl5/5.8.4/man/man3/Test::Builder::Tester.3
Installing /usr/perl5/5.8.4/man/man3/Test::Builder::Module.3
Installing /usr/perl5/5.8.4/man/man3/Test::Tutorial.3
Writing /usr/perl5/5.8.4/lib/sun4-solaris-64int/auto/Test/Simple/.packlist
Appending installation info to /usr/perl5/5.8.4/lib/sun4-solaris-64int/perllocal.pod
  /usr/sfw/bin/gmake install  -- OK

After this, your DBD::Oracle install should work.

cpan> install DBD::Oracle
Running install for module DBD::Oracle
Running make for T/TI/TIMB/DBD-Oracle-1.26.tar.gz
...
Writing /usr/perl5/site_perl/5.8.4/sun4-solaris-64int/auto/DBD/Oracle/.packlist
Appending installation info to /usr/perl5/5.8.4/lib/sun4-solaris-64int/perllocal.pod
  /usr/sfw/bin/gmake install  -- OK

cpan> quit

Finally you can test with:

#!/usr/bin/perl -w      

use DBI ;

my $db_handle = DBI->connect("dbi:Oracle:host=DBserver;sid=sitst;port=1624", "username", "password", {AutoCommit => 0})
  or die "Cannot connect to Oracle on beta: $DBI::errstr\n";

Or, on the command line;

# perl -e 'use DBD::Oracle'
# echo $?
0
# 

Hopefully, this may save some pain.

Paul.


October 27, 2010

Finding which processes are swapped / MDB tutorial

Today I encountered a server that had previously experienced some serious memory pressure, evidence of the page scanner running, a serious amount of anonymous paging and swapping of light weight processes out of memory. I’m sure some of you have had this experience before and wonder just which processes those are that are sat in the vmstat ‘w’ column.

The vmstat man page simply says:

                     w        the number of  swapped  out  light-
                              weight  processes  (LWPs)  that are
                              waiting for processing resources to
                              finish.

Which is vague, at best. Perhaps it’s just me. Anyway, this is what I saw when I ran vmstat:

bash-3.00# vmstat 1
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr vc vc -- --   in   sy   cs us sy id
 0 0 4 12857400 5644584 13 15 166 144 148 0 285 14 0 0 0 1619 1762 1574 2 1 98
 1 0 94 14134296 7000240 7 14 0 0  0  0  0  0  0  0  0  308  779   83  0  0 100
 0 0 94 14134040 6999984 4 5 0  0  0  0  0  0  0  0  0  306  766   79  0  0 100
 1 0 94 14134040 6999984 4 4 0  0  0  0  0  0  0  0  0  342  755  107  0  0 100
 1 0 94 14134040 6999984 4 4 0  0  0  0  0  0  0  0  0  310  776   83  0  0 100
 1 0 94 14134040 6999984 4 4 0  0  0  0  0  0  0  0  0  310  763   89  0  0 100
 1 0 94 14134040 6999984 4 4 0  0  0  0  0  0  0  0  0  301  757   75  0  0 100
 0 0 94 14134040 6999984 4 4 0  0  0  0  0  0  0  0  0  296  751   69  0  0 100
 1 0 94 14134040 6999984 4 4 0  0  0  0  0  0  0  0  0  352  766  111  0  0 100
 0 0 94 14134040 6999984 4 4 0  0  0  0  0  0  0  0  0  310  787   93  0  0 100
 1 0 94 14134040 6999984 4 4 0  0  0  0  0  0  0  0  0  316  772   87  0  0 100

Wow, 94 LWPs swapped out, that’s quite a lot. This generally won’t cause any issues on your system, as soon as an LWP is asked to do work it will be swapped back in, on demand. In fact, you could force them back in by tracing the process (p-commands such as pfiles etc) or sending the process a signal but in order to do that you would need to know ‘which’ threads are swapped out.

There are a number of ways to do this, and one of them is with mdb. So, I set myself the task of finding which processes are swapped out.

First, access to the kernel threads in the live system is possible through the ‘thread’ walker in mdb. To access the live running kernel, you need to fire up mdb with the ‘-k’ option.

The ::help dcmd in mdb shows us a brief description of each dcmd. Below, the walker and output is shown as an example.

# mdb -k
Loading modules: [ unix genunix specfs dtrace zfs sd pcisch ip hook neti sctp arp usba fcp fctl nca lofs md cpc random crypto wrsmd fcip logindmux ptm ufs sppp nfs ipc ]
> ::help walk       

NAME
  walk - walk data structure

SYNOPSIS
  [ addr ] ::walk name [variable]

ATTRIBUTES

  Target: kvm
  Module: mdb
  Interface Stability: Evolving

> 

So, lets start with the kthreads:

> ::walk thread thr
180e000
2a10001fca0
2a100017ca0
2a10000fca0
2a100007ca0
2a10003fca0

What we get back is a bunch of memory addresses that allow us access to the threads, one per thread, of course. Pipes are powerful constructs in mdb which allow us, much like unix commands, to pipe the output of one cmd (dcmd) into the input of another, in the first instance, we can pipe the output of the walker above straight into ::print to print the kthread_t schedflag.

I should mention that the addresses listed in step one will point to a data structure of type kthread_t which you can look up in /usr/include/sys/thread.h. This is important because it tells you how to read the structure and details what elements within we may be interested in. Take a look and read up on the schedflag.

> ::walk thread thr |::print kthread_t t_schedflag                              
t_schedflag = 0x3e03
t_schedflag = 0x3
t_schedflag = 0x3
t_schedflag = 0x3
t_schedflag = 0x3
t_schedflag = 0x3
....

OK, so I’ve truncated the output, but we can see that this gives us a list of all the schedflags for all the threads on the system. The list is long, so we need to work out which schedflag values we are interested in and ‘grep’ for that value in the output. Fortunately, mdb provides us with just what we need in the ::grep dcmd. Note that the ::grep dcmd uses the . notation which refers to the previous command output. The mdb documentation is suitably confusing on this one, I don’t know about you but I had to just ‘play’ with the dcmd to work out what this was trying to tell me:

     ::grep command

         Evaluate the specified command string,  and  then  print
         the  old  value  of  dot if the new value of dot is non-
         zero. If the command contains whitespace or  metacharac-
         ters,  it must be quoted. The ::grep dcmd can be used in
         pipelines to filter a list of addresses.

Again, output below truncated.

> ::walk thread thr |::print -x kthread_t t_schedflag|::grep .==0x03
3
3
3

As for my quest, I initially expected to be able to match/search for the value 0×0008, due to the following in thread.h:

#define TS_ON_SWAPQ     0x0008  /* thread is on the swap queue */

I will jump ahead here and show you that you can run mdb non-interactively and combine it with standard unix commands.

To my dismay, there are no threads with the 0×0008 schedflag value on my system:

bash-3.00# echo "::walk thread thr |::print kthread_t t_schedflag|::grep .==0008" | mdb -k |wc -l
       0

When there clearly are still threads on the run queue, 91 in fact:

bash-3.00# vmstat 1 3
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr vc vc -- --   in   sy   cs us sy id
 0 0 6 12892280 5681600 13 15 162 140 144 0 278 14 0 0 0 1584 1718 1534 2 1 98
 1 0 91 14172696 7055272 10 17 0 0 0  0  0  0  0  0  0  318  645   87  0  0 100

I came across this by trial and error, but it can’t be a coincidence that there are 91 threads in my system that have TS_RUNQMATCH set. I need to read through the code to work out why, but it would appear that setbackdq() has set/changed the value since the threads were swapped out. If anyone cares to explain in the comments section, I’d be very grateful :)

bash-3.00# grep RUNQ /usr/include/sys/thread.h
#define TS_RUNQMATCH    0x4000  /* exact run queue balancing by setbackdq() */

bash-3.00# echo "::walk thread thr |::print kthread_t t_schedflag|::grep .==4000" | mdb -k |wc -l
      91

So, going with a rather large assumption for now, lets look at the threads with that value set. First, print out the address of the threads in question, and recall that after the ::grep we have the value of the schedflag where the schedflag was equal to TS_RUNQMATCH. We can use ::eval to convert this back to an address.

> ::walk thread thr |::print kthread_t t_schedflag|::grep .==4000 |::eval <thr=K
                30007f36760     
                30007f21a80     
                30007f36aa0     
                30007f514e0     
                3000803af20     
                30007f2f780     
                30007f1d3e0     
                300076366a0     

Then, (finally), pipe each one back to ::print in order to interrogate the proc structure and in turn the p_user user structure (see sys/proc.h and sys/user.h) and in turn the value in p.user.u_comm. Output truncated.

> ::walk thread thr |::print kthread_t t_schedflag|::grep .==4000 |::eval <thr=K |::print -d kthread_t t_procp->p_user.u_comm
t_procp->p_user.u_comm = [ "svc.configd" ]
t_procp->p_user.u_comm = [ "svc.configd" ]
t_procp->p_user.u_comm = [ "svc.configd" ]
t_procp->p_user.u_comm = [ "svc.configd" ]
t_procp->p_user.u_comm = [ "svc.configd" ]
t_procp->p_user.u_comm = [ "svc.configd" ]
t_procp->p_user.u_comm = [ "svc.configd" ]
t_procp->p_user.u_comm = [ "svc.configd" ]
t_procp->p_user.u_comm = [ "svc.configd" ]
t_procp->p_user.u_comm = [ "svc.configd" ]
t_procp->p_user.u_comm = [ "nscd" ]
t_procp->p_user.u_comm = [ "svc.configd" ]
t_procp->p_user.u_comm = [ "svc.configd" ]
t_procp->p_user.u_comm = [ "svc.configd" ]
t_procp->p_user.u_comm = [ "svc.configd" ]
t_procp->p_user.u_comm = [ "svc.configd" ]
t_procp->p_user.u_comm = [ "svc.configd" ]
t_procp->p_user.u_comm = [ "svc.configd" ]
t_procp->p_user.u_comm = [ "svc.startd" ]
t_procp->p_user.u_comm = [ "svc.startd" ]
...

Alternatively, the proc structure contains a pointer to a pid structure, p_pidp which gives us access to the process id via pid_id:

::walk thread thr |::print kthread_t t_schedflag|::grep .==4000 |::eval <thr=K |::print -d kthread_t t_procp->p_pidp->pid_id
t_procp->p_pidp->pid_id = 0t10199
t_procp->p_pidp->pid_id = 0t10213
t_procp->p_pidp->pid_id = 0t10661
t_procp->p_pidp->pid_id = 0t10661
t_procp->p_pidp->pid_id = 0t10661
t_procp->p_pidp->pid_id = 0t10661
t_procp->p_pidp->pid_id = 0t10661

OK, so we got there, command names and process ids for the swapped out processes, if we go with the assumption around TS_RUNQMATCH. However, since drafting my notes for this blog entry I have found that a cleaner and possibly more reliable way of identifying the processes/threads that are swapped out is by looking for threads that are NOT in memory. The schedflag value TS_LOAD would be most helpful here, so the best option would be to search for threads whose schedflag is NOT equal to TS_LOAD.

#define TS_LOAD         0x0001  /* thread is in memory */

How? By using the bitwise AND operator in mdb (&) and AND’ing the schedflag with 1. If the result is zero, then the value in schedflag must have been zero also (TS_LOAD not set).

> ::walk thread thr |::print -d kthread_t t_schedflag | ::grep '(.&1)==0' |::eval <thr=K |::print -d kthread_t t_procp->p_user.u_comm
t_procp->p_user.u_comm = [ "svc.configd" ]
t_procp->p_user.u_comm = [ "svc.configd" ]
t_procp->p_user.u_comm = [ "svc.startd" ]
t_procp->p_user.u_comm = [ "drd" ]
t_procp->p_user.u_comm = [ "syseventd" ]
t_procp->p_user.u_comm = [ "syseventd" ]
t_procp->p_user.u_comm = [ "syseventd" ]

This appears to give the same answer as my previous attempt, and gives the same number of swapped out threads (91);

bash-3.00# echo "::walk thread thr |::print -d kthread_t t_schedflag | ::grep '(.&1)==0' |::eval <thr=K |::print -d kthread_t t_procp->p_user.u_comm" | mdb -k |wc -l
      91

bash-3.00# echo "::walk thread thr |::print kthread_t t_schedflag|::grep .==4000" | mdb -k |wc -l
      91

I really should write a wrapper around this, but for now the blog entry is enough. Hopefully this has been useful, if somewhat verbose. It has been as much about my education as anything else. I need to apologise too, this is even more dull than my last post.

Paul.


October 2023

Mo Tu We Th Fr Sa Su
Sep |  Today  |
                  1
2 3 4 5 6 7 8
9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30 31               

Search this blog

Tags

Galleries

Most recent comments

  • Started sorting out new vers for sparc: http://blogs.warwick.ac.uk/mariamaccallum/entry/apache_249_i… by Maria MacCallum on this entry
  • Solaris 11.1 is slightly different, I only had to do this before starting ipfilter: svccfg –s setpro… by Maria MacCallum on this entry
  • Really useful information, thanks a lot! I do a NAT using IPFILTER and all was working good, until I… by Nilton on this entry
  • Paul, Thanks for your information. It got me started quickly. I have discovered , thought I've not s… by Tom C on this entry
  • Are you familiar with the Monty Python sketch? by Ian Eiloart on this entry

Blog archive

Loading…
RSS2.0 Atom
Not signed in
Sign in

Powered by BlogBuilder
© MMXXIII