All entries for November 2010

November 25, 2010

IPMP and IP Filter

So here is the problem.  You need to configure IP Filter on Solaris to meet the needs of your security policy but you also need to configure IPMP to provide resiliency against network port failure.  The default configuration for IP Filter on Solaris refers to the ingress or egress interface by device name in the ipf.conf file.   You may have, for example an entry like this:

pass in log first level quick on bge0 proto tcp from to any flags S keep state

The problem is, of course that when using IPMP the system will, when required, fail to a different interface. Say, e1000g0.  Of course the ipmp group may *contain* bge0, but we need to be sure we are going to cater successfully for the case where ipmp fails over to the partner interface.  One option would be to duplicate all your IP Filter rules for every interface in the IPMP group.  Clearly not sensible, nor necessary; the solution (post Solaris 10u2) is to use the ipmp_hook_emulation ndd setting for the /dev/ip driver.  For versions less than or equal to S10u2 go and research the /dev/pfil setting qif_ipmp_set.

First, we need to set ipmp_hook_emulation for the /dev/ip driver to 1.

bash-3.00# /usr/sbin/ndd -get /dev/ip ipmp_hook_emulation
bash-3.00# /usr/sbin/ndd -set /dev/ip ipmp_hook_emulation 1
bash-3.00# /usr/sbin/ndd -get /dev/ip ipmp_hook_emulation

Once that is one, we change the interface name in ipf.conf to the ipmp groupname;

Check what it is with ifconfig:

groupname app
e1000g0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
groupname app
e1000g0:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3

Set appropriately in ipf.conf:

pass in log first level quick on app proto tcp from to any flags S keep state

Now, restart ipfilter and monitor the logfile.  We see the firewall acting on entries for the IPMP group, rather than by specific interface.  Now, if mpathd fails over the interface, the firewall continues to operate as expected.

Nov 24 14:49:00 test-serva ipmon[1160]: [ID 702911] 14:48:59.955694 app0 @0:9 p,123 ->,123 PR udp len 20 76 K-S IN

The second part to this solution is to make the changes persistent.

We can make sure ipfilter will be restarted persistently on boot by enabling the SMF service 'network/ipfilter'. The ndd setting however, will not persist.  How do we set the ndd setting on boot?  One option is a legacy RC script, but this does not leverage the dependency framework of SMF to ensure that the setting is applied before the ipfilter service is started. 

Creating an SMF service to do this that sets itself as a dependent to ipfilter is quite straight forward.  The nice thing about SMF is that you can make ipfilter depend on the new tuning service *without* changing the setup of ipfilter.  To do this, create a new service that configures the ndd parameter(s) you are interested in and set the 'network/ipfilter' service and a *dependent* in the XML manifest;

First, a manifest is required, so create /var/svc/manifest/site/ndd-nettune.xml; notice the dependent ipfilter entry.

<?xml version="1.0"?>

<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">


ident    "@(#)ndd-nettune.xml    1.0    04/09/21 SMI"


<service_bundle type='manifest' name='SUNWcsr:ndd'>





                       <create_default_instance enabled='true' />

                       <single_instance />






                   <service_fmri value='svc:/system/filesystem/minimal' />







                   <service_fmri value='svc:/network/loopback' />







                   <service_fmri value='svc:/network/physical' />







                   <service_fmri value='svc:/network/ipfilter' />






   timeout_seconds='3' >






          timeout_seconds='3' >


               <property_group name='startd' type='framework'>

                   <propval name='duration' type='astring'  value='transient' />


               <stability value='Unstable' />



                           <loctext xml:lang='C'>

                                   ndd network tuning




                           <manpage title='ndd' section='1M'

           manpath='/usr/share/man' />





Run xmllint on the file to check it out:

xmllint /var/svc/manifest/site/ndd-nettune.xml 

Validate it with svccfg

svccfg validate /var/svc/manifest/site/ndd-nettune.xml

Create the script that it will run

bash-3.00# vi /lib/svc/method/ndd-nettune

"/lib/svc/method/ndd-nettune" 15 lines, 453 characters

. /lib/svc/share/
. /lib/svc/share/

# Make sure that the libraries essential to this stage of booting  can be found.
echo "Performing IP Tuning..." >> /tmp/smf.out
/usr/sbin/ndd -set /dev/ip ipmp_hook_emulation 1

# Reset the library path now that we are past the critical stage

Import the new service:

svccfg import /var/svc/manifest/site/ndd-nettune.xml 

Enable if necessary:

svcadm enable ndd-nettune

Check again that the ipmp_hook_emulation is set:

ndd -get /dev/ip ipmp_hook_emulation

Do some brief testing;

Check the value of ipmp_hook_emulation;

bash-3.00# ndd -get /dev/ip ipmp_hook_emulation

Check IP Filter and the ndd service are running;

bash-3.00# svcs ipfilter
online         14:45:45 svc:/network/ipfilter:default
bash-3.00# svcs ndd-nettune
online         14:45:45 svc:/network/ndd-nettune:default

Check the new service restarts ok:

bash-3.00# svcadm restart ndd-nettune
bash-3.00# svcs -a |grep nett
online         14:47:48 svc:/network/ndd-nettune:default

And that the ipfilter service can restart:

bash-3.00# svcadm restart ipfilter   

Disable ndd-nettune;

bash-3.00# svcadm disable ndd-nettune

And attempt to restart ipfilter;

bash-3.00# svcadm restart ipfilter

Excellent, it won't start because the dependent isn't met.

offline        14:48:03 svc:/network/ipfilter:default
bash-3.00# svcs -x
svc:/network/ndd-nettune:default (ndd network tuning)
State: disabled since 24 November 2010 14:48:02 GMT
Reason: Disabled by an administrator.
See: ndd(1M)
See: /var/svc/log/network-ndd-nettune:default.log
Impact: 1 dependent service is not running.  (Use -v for list.)

Finally, enable ndd-nettune and ensure ipfilter now starts.

bash-3.00# svcadm enable ndd-nettune
bash-3.00# svcadm restart ipfilter
bash-3.00# svcs -x

So, we have a method to run both IPMP for network resiliency and IP filter for network security.  The method is clean, easy and persistent across reboots.  This functionality was introduced in Solaris 10 update 3, but I should say that it is changing again.  The next release(s) of Solaris will see the introduction of the Clearview project and the IPMP implementation will create virtual interfaces, such as ipmp0, ipmp1 which can be treated just like any other interfaces for the purposes of technologies such as ipfilter.


November 19, 2010

Installing Perl Modules (perl DBI / DBD::Oracle) on Solaris

If you have ever tried to install perl modules on Solaris, then you’ll have shared my pain. Solaris, rather helpfully, comes with its own installation of perl, straight off the DVD. You get an install of something like 5.8.4, depending on your version of Solaris.

bash-3.00# which perl
bash-3.00# perl -v

This is perl, v5.8.4 built for sun4-solaris-64int
(with 32 registered patches, see perl -V for more detail)

Copyright 1987-2004, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using `man perl' or `perldoc perl'.  If you have access to the
Internet, point your browser at, the Perl Home Page.


Marvellous. However, there is a snag. If you use perl in anger you will want to update some of the perl modules. Or even install some new ones that aren’t included by default. I had to climb this particular hill recently when I needed the perl DBI module to access a database, more specifically, the DBD::Oracle module to access an Oracle database.

The most straight forward and popular method for installing perl modules is to use CPAN. Whether you use CPAN or compile by hand, the first, not so small issuette you will hit is that the perl in Solaris is compiled with the Sun Studio compilers.

There is a solution for this that means you can use the GNU compiling tools under /usr/sfw, but it does take a little tweaking to get things straight. The solution is to use /usr/perl5/bin/perlgcc -MCPAN -e shell. This will fire up a CPAN shell for you to use. Below is an account of the configuration tweaks I had to do to get this CPAN shell to download, compile and install my modules.

The first time you run it, you’ll be asked a bewilering array of questions. Do not let it autoconfigure, instead answer [yes] to being ready for manual configuration. You are ready. You just don’t know it yet. Fortunately for the most part, the defaults will work, except for the paths / settings below. If a path to a binary is missing (lynx for example), leave it blank, you won’t need it.

Where is your tar program? [/usr/sbin/tar] /usr/sfw/bin/gtar

Warning: make not found in PATH
Where is your make program? [] /usr/sfw/bin/gmake

Warning: wget not found in PATH
Where is your wget program? [] /usr/sfw/bin/wget
Warning: ncftpget not found in PATH
Where is your ncftpget program? [] 
Warning: ncftp not found in PATH

The configuration tool will ask you to input a URL for downloads, for some reason it only shows ftp URLs even though you set the wget location. You are forced to accept one and remove it later (and add an HTTP one) even if you don’t want to use ftp.

Select as many URLs as you like (by number),
put them on one line, separated by blanks, e.g. '1 4 5' [] 

After that, use ‘o conf’ to check your settings:

cpan> o conf     
CPAN::Config options from /usr/perl5/5.8.4/lib/CPAN/
    commit             Commit changes to disk
    defaults           Reload defaults from disk
    init               Interactive setting of all options

    build_cache        10
    build_dir          /.cpan/build
    cache_metadata     1
    cpan_home          /.cpan
    ftp                "" 
    getcwd             cwd
    gzip               /usr/bin/gzip
    histfile           /.cpan/histfile
    histsize           100
    inactivity_timeout 0
    index_expire       1
    inhibit_startup_message 0
    keep_source_where  /.cpan/sources
    make               /usr/sfw/bin/gmake
    pager              /usr/bin/less
    prerequisites_policy ask
    scan_cache         atstart
    shell              /sbin/sh
    tar                /usr/sfw/bin/gtar
    term_is_latin      1
    unzip              /usr/bin/unzip
    wget               /usr/sfw/bin/wget

Notice that there is no entry for cc/gcc; you have to configure this through the shell environment, so make sure that a suitable compiler is in your PATH (/usr/sfw/bin/gcc);

To fix that url entry, use ‘o conf urllist shift’ to remove the bad entry, and o conf urllist unshift [new url] to add the one you need:

cpan> o conf urllist shift

cpan> o conf urllist
Type 'o conf' to view configuration edit options

cpan> o conf urllist unshift
Type 'o conf' to view configuration edit options

cpan> o conf urllist
Type 'o conf' to view configuration edit options

The basic rule for the install of the DBD::Oracle module install is that you have to be able to connect to a test database using a test user from the shell where you launch the CPAN tool. This is because CPAN will use your current environment to connect to a database when it runs ‘make test’. You really MUST read the README that is supplied with the module because some elements may not be correct for your build, but here is my experience (I didn’t really use scott/tiger, but you get the idea):

Set environment variables, you also need to make sure something sensible is set in your tnsnames.ora. You can also use the TWO_TASK variable, the README suggests this and it seems to work. This doesn’t remove the need for a valid tnsnames.ora, of course.

bash-3.00# export LD_LIBRARY_PATH=/app/oracle/product/10.2.0/db_1/lib:/usr/dt/lib:/app/oracle/product/10.2.0/db_1/jdbc/lib:/app/oracle/product/10.2.0/db_1/lib32
bash-3.00# export ORACLE_USERID=scott/tiger
bash-3.00# export PATH=/usr/sbin:/usr/bin:/usr/local/oracle/product/10.2.0/db_1/bin:/usr/sfw/bin
bash-3.00# export ORACLE_HOME=/app/oracle/product/10.2.0/db_1

Make sure your tools look good… ahem.

bash-3.00# which gmake
bash-3.00# which gcc

Run the CPAN shell again and install your module;

bash-3.00# /usr/perl5/bin/perlgcc -MCPAN -e shell
Terminal does not support AddHistory.

cpan shell -- CPAN exploration and modules installation (v1.7601)
ReadLine support available (try 'install Bundle::CPAN')

cpan> install DBD::Oracle
CPAN: Storable loaded ok
LWP not available


If you get this error:

t/10general.............Can't locate object method "no_diag" via package "Test::Builder" at t/ line 53.

You probably don’t have the correct version of Test::Builder. You can check this in the perldoc Test::Builder on your system; look for no_diag. The fix, of course is to install the new version using your CPAN Shell:

cpan> install Test::Builder
Running install for module Test::Builder
Running make for M/MS/MSCHWERN/Test-Simple-0.96.tar.gz


Installing /usr/perl5/5.8.4/man/man3/Test::Builder::Tester.3
Installing /usr/perl5/5.8.4/man/man3/Test::Builder::Module.3
Installing /usr/perl5/5.8.4/man/man3/Test::Tutorial.3
Writing /usr/perl5/5.8.4/lib/sun4-solaris-64int/auto/Test/Simple/.packlist
Appending installation info to /usr/perl5/5.8.4/lib/sun4-solaris-64int/perllocal.pod
  /usr/sfw/bin/gmake install  -- OK

After this, your DBD::Oracle install should work.

cpan> install DBD::Oracle
Running install for module DBD::Oracle
Running make for T/TI/TIMB/DBD-Oracle-1.26.tar.gz
Writing /usr/perl5/site_perl/5.8.4/sun4-solaris-64int/auto/DBD/Oracle/.packlist
Appending installation info to /usr/perl5/5.8.4/lib/sun4-solaris-64int/perllocal.pod
  /usr/sfw/bin/gmake install  -- OK

cpan> quit

Finally you can test with:

#!/usr/bin/perl -w      

use DBI ;

my $db_handle = DBI->connect("dbi:Oracle:host=DBserver;sid=sitst;port=1624", "username", "password", {AutoCommit => 0})
  or die "Cannot connect to Oracle on beta: $DBI::errstr\n";

Or, on the command line;

# perl -e 'use DBD::Oracle'
# echo $?

Hopefully, this may save some pain.


November 2010

Mo Tu We Th Fr Sa Su
Oct |  Today  | Dec
1 2 3 4 5 6 7
8 9 10 11 12 13 14
15 16 17 18 19 20 21
22 23 24 25 26 27 28
29 30               

Search this blog



Most recent comments

  • Started sorting out new vers for sparc:… by Maria MacCallum on this entry
  • Solaris 11.1 is slightly different, I only had to do this before starting ipfilter: svccfg –s setpro… by Maria MacCallum on this entry
  • Really useful information, thanks a lot! I do a NAT using IPFILTER and all was working good, until I… by Nilton on this entry
  • Paul, Thanks for your information. It got me started quickly. I have discovered , thought I've not s… by Tom C on this entry
  • Are you familiar with the Monty Python sketch? by Ian Eiloart on this entry

Blog archive

Not signed in
Sign in

Powered by BlogBuilder