All entries for February 2011

February 25, 2011

Disks; cfgadm, fcinfo and LUN mapping on Solaris

OK, so you have a Solaris 10 host with SAN connected storage – how do you make sense of the LUNs you can see? What tools can be used to interrogate the storage and build a mental image of what you have been presented with? This article is intended as a brief introduction to some of the commands in Solaris that will help you achieve your tasks.

Firstly, in order to allow your storage admin to map you some LUNs, you’ll need to provide him with the WWNs of the HBA ports in your server. This is so he can map the LUNs you’ve asked for to the WWNs of your server. These can be found using the fcinfo command. Start with ‘fcinfo hba-port’, Note that the output below shows all 4 of my ports, only 2 of which are occupied and online (c3 and c5).

...
        OS Device Name: /dev/cfg/c3
...
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb

...
        OS Device Name: /dev/cfg/c5
...
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb

The full output;

bash-3.00# fcinfo hba-port
HBA Port WWN: 21000024ff295a34
        OS Device Name: /dev/cfg/c2
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402R00-1023835637
        Driver Name: qlc
        Driver Version: 20100301-3.00
        Type: unknown
        State: offline
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: not established
        Node WWN: 20000024ff295a34
HBA Port WWN: 21000024ff295a35
        OS Device Name: /dev/cfg/c3
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402R00-1023835637
        Driver Name: qlc
        Driver Version: 20100301-3.00
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 20000024ff295a35
HBA Port WWN: 21000024ff295a36
        OS Device Name: /dev/cfg/c4
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402R00-1023835638
        Driver Name: qlc
        Driver Version: 20100301-3.00
        Type: unknown
        State: offline
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: not established
        Node WWN: 20000024ff295a36
HBA Port WWN: 21000024ff295a37
        OS Device Name: /dev/cfg/c5
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402R00-1023835638
        Driver Name: qlc
        Driver Version: 20100301-3.00
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 20000024ff295a37
bash-3.00# 

It is the ‘HBA Port WWN’ that you need to give to your storage admin. He may appreciate the full output, that will confirm a few other items for him such as the link speed and your HBA manufacturer and driver version numbers.

Using the -l (minus ell) flag shows additional information such as link statistics…

bash-3.00# fcinfo hba-port -l 21000024ff295a37
HBA Port WWN: 21000024ff295a37
        OS Device Name: /dev/cfg/c5
        Manufacturer: QLogic Corp.
        Model: 375-3356-02
        Firmware Version: 05.03.02
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.01; EFI: 2.00;
        Serial Number: 0402R00-1023835638
        Driver Name: qlc
        Driver Version: 20100301-3.00
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 4Gb
        Node WWN: 20000024ff295a37
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 0
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 0
                Invalid CRC Count: 0
bash-3.00# 

The details (WWN) of the remote ports can be viewed using ‘fcinfo remote-port -p

bash-3.00# fcinfo remote-port -p 21000024ff295a37
Remote Port WWN: 24540002ac0009e2
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac0009e2
Remote Port WWN: 25540002ac0009e2
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac0009e2
Remote Port WWN: 22120002ac000928
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac000928
Remote Port WWN: 23120002ac000928
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac000928

The -l option can still be used in conjunction with this to show link statistics, and the -s option will show you the LUNs; This is very handy as it shows the LUN number / device name mappings.

...
        LUN: 4
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BC0928d0s2

...

The full output is below;

bash-3.00# fcinfo remote-port -l -p 21000024ff295a37
Remote Port WWN: 24540002ac0009e2
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac0009e2
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 2
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 2
                Invalid CRC Count: 0
Remote Port WWN: 25540002ac0009e2
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac0009e2
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 2
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 2
                Invalid CRC Count: 0
Remote Port WWN: 22120002ac000928
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac000928
        Link Error Statistics:
                Link Failure Count: 2
                Loss of Sync Count: 1
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 2
                Invalid CRC Count: 0
Remote Port WWN: 23120002ac000928
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac000928
        Link Error Statistics:
                Link Failure Count: 2
                Loss of Sync Count: 1
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 0
                Invalid CRC Count: 0
bash-3.00# fcinfo remote-port -ls -p 21000024ff295a37
Remote Port WWN: 24540002ac0009e2
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac0009e2
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 2
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 2
                Invalid CRC Count: 0
        LUN: 254
          Vendor: 3PARdata
          Product: SES            
          OS Device Name: /dev/es/ses8
Remote Port WWN: 25540002ac0009e2
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac0009e2
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 2
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 2
                Invalid CRC Count: 0
        LUN: 254
          Vendor: 3PARdata
          Product: SES            
          OS Device Name: /dev/es/ses8
Remote Port WWN: 22120002ac000928
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac000928
        Link Error Statistics:
                Link Failure Count: 2
                Loss of Sync Count: 1
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 2
                Invalid CRC Count: 0
        LUN: 0
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001B70928d0s2
        LUN: 1
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001B90928d0s2
        LUN: 2
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BA0928d0s2
        LUN: 3
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BB0928d0s2
        LUN: 4
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BC0928d0s2
        LUN: 5
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BD0928d0s2
        LUN: 6
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BE0928d0s2
        LUN: 7
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BF0928d0s2
        LUN: 8
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C00928d0s2
        LUN: 9
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C10928d0s2
        LUN: 10
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C20928d0s2
        LUN: 11
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C30928d0s2
        LUN: 12
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C40928d0s2
        LUN: 13
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C50928d0s2
        LUN: 14
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001B80928d0s2
        LUN: 254
          Vendor: 3PARdata
          Product: SES            
          OS Device Name: /dev/es/ses9
Remote Port WWN: 23120002ac000928
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 2ff70002ac000928
        Link Error Statistics:
                Link Failure Count: 2
                Loss of Sync Count: 1
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 0
                Invalid CRC Count: 0
        LUN: 0
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001B70928d0s2
        LUN: 1
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001B90928d0s2
        LUN: 2
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BA0928d0s2
        LUN: 3
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BB0928d0s2
        LUN: 4
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BC0928d0s2
        LUN: 5
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BD0928d0s2
        LUN: 6
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BE0928d0s2
        LUN: 7
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001BF0928d0s2
        LUN: 8
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C00928d0s2
        LUN: 9
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C10928d0s2
        LUN: 10
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C20928d0s2
        LUN: 11
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C30928d0s2
        LUN: 12
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C40928d0s2
        LUN: 13
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001C50928d0s2
        LUN: 14
          Vendor: 3PARdata
          Product: VV             
          OS Device Name: /dev/rdsk/c6t50002AC001B80928d0s2
        LUN: 254
          Vendor: 3PARdata
          Product: SES            
          OS Device Name: /dev/es/ses9

The cfgadm command can be used to view the system ‘attachment points’ which are broadly defined as the location of certain hardware resources visible to Solaris. Running cfgadm with the -al option will list these attachment points. This will include SAN HBAs and LUNs, USB devices, internal disks such as SATA etc.

The type field gives an indication of the type of the device, you will see some self explanatory entries such as ‘scsi-bus’ (which I think about as a controller), disk – very self explanatory. A fibre channel SAN HBA is usually seen as fc, if it is connected to the fabric/SAN it will show as fc-fabric. If you see an entry labelled ESI then know that you are seeing the Enclosure through the Enclosure Services Interface.

The output below is partially truncated.

bash-3.00# cfgadm -al
Ap_Id                          Type         Receptacle   Occupant     Condition
c0                             scsi-bus     connected    configured   unknown
c0::dsk/c0t0d0                 disk         connected    configured   unknown
c0::dsk/c0t1d0                 disk         connected    configured   unknown
c0::dsk/c0t2d0                 disk         connected    configured   unknown
c0::dsk/c0t3d0                 disk         connected    configured   unknown
c0::dsk/c0t4d0                 disk         connected    configured   unknown
c2                             fc           connected    unconfigured unknown
c3                             fc-fabric    connected    configured   unknown
c3::22110002ac000928           disk         connected    configured   unknown
c3::23110002ac000928           disk         connected    configured   unknown
c3::24530002ac0009e2           ESI          connected    configured   unknown
c3::25530002ac0009e2           ESI          connected    configured   unknown
c4                             fc           connected    unconfigured unknown
c5                             fc-fabric    connected    configured   unknown
c5::22120002ac000928           disk         connected    configured   unknown
c5::23120002ac000928           disk         connected    configured   unknown
c5::24540002ac0009e2           ESI          connected    configured   unknown
c5::25540002ac0009e2           ESI          connected    configured   unknown
sata0/0                        sata-port    empty        unconfigured ok
sata0/1                        sata-port    empty        unconfigured ok

Use the ‘-o show_FCP_dev’ option to get cfgadm to show not only the controllers and the enclosures, but also any fibre channel disks that may be visible on the channel:

Ap_Id                          Type         Receptacle   Occupant     Condition
c2                             fc           connected    unconfigured unknown
c3                             fc-fabric    connected    configured   unknown
c3::22110002ac000928,0         disk         connected    configured   unknown
c3::22110002ac000928,1         disk         connected    configured   unknown
c3::22110002ac000928,2         disk         connected    configured   unknown
c3::22110002ac000928,3         disk         connected    configured   unknown
c3::22110002ac000928,4         disk         connected    configured   unknown
c3::22110002ac000928,5         disk         connected    configured   unknown
c3::22110002ac000928,6         disk         connected    configured   unknown
...
c3::22110002ac000928,12        disk         connected    configured   unknown
c3::22110002ac000928,13        disk         connected    configured   unknown
c3::22110002ac000928,14        disk         connected    configured   unknown
c3::22110002ac000928,254       ESI          connected    configured   unknown
.....
c3::23110002ac000928,13        disk         connected    configured   unknown
c3::23110002ac000928,14        disk         connected    configured   unknown
c3::23110002ac000928,254       ESI          connected    configured   unknown
c3::24530002ac0009e2,254       ESI          connected    configured   unknown
c3::25530002ac0009e2,254       ESI          connected    configured   unknown
c4                             fc           connected    unconfigured unknown
c5                             fc-fabric    connected    configured   unknown
c5::22120002ac000928,0         disk         connected    configured   unknown
c5::22120002ac000928,1         disk         connected    configured   unknown
c5::22120002ac000928,2         disk         connected    configured   unknown
c5::22120002ac000928,3         disk         connected    configured   unknown
c5::22120002ac000928,4         disk         connected    configured   unknown
c5::22120002ac000928,5         disk         connected    configured   unknown
....
c5::22120002ac000928,14        disk         connected    configured   unknown
c5::22120002ac000928,254       ESI          connected    configured   unknown
c5::23120002ac000928,0         disk         connected    configured   unknown
....
c5::23120002ac000928,12        disk         connected    configured   unknown
c5::23120002ac000928,13        disk         connected    configured   unknown
c5::23120002ac000928,14        disk         connected    configured   unknown
c5::23120002ac000928,254       ESI          connected    configured   unknown
c5::24540002ac0009e2,254       ESI          connected    configured   unknown
c5::25540002ac0009e2,254       ESI          connected    configured   unknown
bash-3.00# 

Throughout the outputs above you’ll notice that these disks are multipathed and visible through 2 separate controllers (c3 and c5). Enable STMS (see stmsboot1m)) to aggregate those 2 paths to a single controller. You will gain a pseudo controller when you do this. In this case, the controller becomes c6 (aggregate of c3 and c5). The new disk targets created by STMS then are visible in format:

      18. c6t50002AC001C40928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD13
          /scsi_vhci/disk@g50002ac001c40928
      19. c6t50002AC001C50928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD14
          /scsi_vhci/disk@g50002ac001c50928

Once we have these multiple paths, luxadm can be used to interrogate the controller and view the subpaths (and their state). First, run a ‘luxadm probe’ which will scan the devices and present a list.

bash-3.00# luxadm probe             

Found Fibre Channel device(s):
  Node WWN:2ff70002ac0009e2  Device Type:SES device
    Logical Path:/dev/es/ses8
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001B80928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001C50928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001C40928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001C30928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001C20928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001C10928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001C00928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001BF0928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001BE0928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001BD0928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001BC0928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001BB0928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001BA0928d0s2
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001B90928d0s2
  Node WWN:2ff70002ac000928  Device Type:SES device
    Logical Path:/dev/es/ses9
  Node WWN:2ff70002ac000928  Device Type:Disk device
    Logical Path:/dev/rdsk/c6t50002AC001B70928d0s2
bash-3.00#

Now you can select a logical path and use with ‘luxadm display ’ to view the individual paths;

bash-3.00# luxadm display /dev/rdsk/c6t50002AC001B80928d0s2DEVICE PROPERTIES for disk: /dev/rdsk/c6t50002AC001B80928d0s2
  Vendor:               3PARdata
  Product ID:           VV             
  Revision:             0000
  Serial Num:           01B80928
  Unformatted capacity: 40960.000 MBytes
  Write Cache:          Enabled
  Read Cache:           Enabled
    Minimum prefetch:   0x0
    Maximum prefetch:   0xffff
  Device Type:          Disk device
  Path(s):

  /dev/rdsk/c6t50002AC001B80928d0s2
  /devices/scsi_vhci/disk@g50002ac001b80928:c,raw
   Controller           /dev/cfg/c3
    Device Address              22110002ac000928,e
    Host controller port WWN    21000024ff295a35
    Class                       primary
    State                       ONLINE
   Controller           /dev/cfg/c3
    Device Address              23110002ac000928,e
    Host controller port WWN    21000024ff295a35
    Class                       primary
    State                       ONLINE
   Controller           /dev/cfg/c5
    Device Address              22120002ac000928,e
    Host controller port WWN    21000024ff295a37
    Class                       primary
    State                       ONLINE
   Controller           /dev/cfg/c5
    Device Address              23120002ac000928,e
    Host controller port WWN    21000024ff295a37
    Class                       primary
    State                       ONLINE

bash-3.00#

Things to note are that the size, write cache, read cache and path state are all shown and that the ’,e’ after the Device Address is the LUN number in hex.

The unique part of the LUN indentifier (the LUN id) is embedded halfway through the mpxio / STMS name. To extract from format, for example;

bash-3.00# echo | format |grep c6 | cut -c22-25
01B7
01B8
01B9
01BA
01BB
01BC
01BD
01BE
01BF
01C0
01C1
01C2
01C3
01C4
01C5

This information, in conjunction with luxadm can be used to correlate disks to LUN numbers and LUN IDs. Although the fcinfo commands shown above are generally the easier way to obtain this information. Notice that the LUN number is appended to the Device Address in hex.

bash-3.00# for disk in $( ls /dev/rdsk/c6t50002AC00*s2 )
> do
> echo $disk
> echo $disk | cut -c22-26
> luxadm display $disk |grep 'Device Address'
> done
/dev/rdsk/c6t50002AC001B70928d0s2
01B70
    Device Address              22110002ac000928,0
    Device Address              23110002ac000928,0
    Device Address              22120002ac000928,0
    Device Address              23120002ac000928,0

/dev/rdsk/c6t50002AC001B90928d0s2
01B90
    Device Address              22110002ac000928,1
    Device Address              23110002ac000928,1
    Device Address              22120002ac000928,1
    Device Address              23120002ac000928,1
/dev/rdsk/c6t50002AC001BA0928d0s2
01BA0
    Device Address              22110002ac000928,2
    Device Address              23110002ac000928,2
    Device Address              22120002ac000928,2
    Device Address              23120002ac000928,2
/dev/rdsk/c6t50002AC001BB0928d0s2
01BB0
    Device Address              22110002ac000928,3
    Device Address              23110002ac000928,3
    Device Address              22120002ac000928,3
    Device Address              23120002ac000928,3
/dev/rdsk/c6t50002AC001BC0928d0s2
01BC0
    Device Address              22110002ac000928,4
    Device Address              23110002ac000928,4
    Device Address              22120002ac000928,4
    Device Address              23120002ac000928,4
/dev/rdsk/c6t50002AC001BD0928d0s2
01BD0
    Device Address              22110002ac000928,5
    Device Address              23110002ac000928,5
    Device Address              22120002ac000928,5
    Device Address              23120002ac000928,5
/dev/rdsk/c6t50002AC001BE0928d0s2
01BE0
    Device Address              22110002ac000928,6
    Device Address              23110002ac000928,6
    Device Address              22120002ac000928,6
    Device Address              23120002ac000928,6
/dev/rdsk/c6t50002AC001BF0928d0s2
01BF0
    Device Address              22110002ac000928,7
    Device Address              23110002ac000928,7
    Device Address              22120002ac000928,7
    Device Address              23120002ac000928,7
/dev/rdsk/c6t50002AC001C00928d0s2
01C00
    Device Address              22110002ac000928,8
    Device Address              23110002ac000928,8
    Device Address              22120002ac000928,8
    Device Address              23120002ac000928,8
/dev/rdsk/c6t50002AC001C10928d0s2
01C10
    Device Address              22110002ac000928,9
    Device Address              23110002ac000928,9
    Device Address              22120002ac000928,9
    Device Address              23120002ac000928,9
/dev/rdsk/c6t50002AC001C20928d0s2
01C20
    Device Address              22110002ac000928,a
    Device Address              23110002ac000928,a
    Device Address              22120002ac000928,a
    Device Address              23120002ac000928,a
/dev/rdsk/c6t50002AC001C30928d0s2
01C30
    Device Address              22110002ac000928,b
    Device Address              23110002ac000928,b
    Device Address              22120002ac000928,b
    Device Address              23120002ac000928,b
/dev/rdsk/c6t50002AC001C40928d0s2
01C40
    Device Address              22110002ac000928,c
    Device Address              23110002ac000928,c
    Device Address              22120002ac000928,c
    Device Address              23120002ac000928,c
/dev/rdsk/c6t50002AC001C50928d0s2
01C50
    Device Address              22110002ac000928,d
    Device Address              23110002ac000928,d
    Device Address              22120002ac000928,d
    Device Address              23120002ac000928,d
/dev/rdsk/c6t50002AC001B80928d0s2
01B80
    Device Address              22110002ac000928,e
    Device Address              23110002ac000928,e
    Device Address              22120002ac000928,e
    Device Address              23120002ac000928,e

Labelling the disks with a volume name, which is completely optional, but quite a useful feature relies on a reference to the LUN numbers to ensure the correct labels are assigned to the correct disks. I like to use the following which will print the LUN number for me just before presenting me with a format dialog in which to assign the appropriate volname;

for disk in $( ls /dev/rdsk/c6t50002AC00*s2 )
do 
   echo $disk
   echo $disk | cut -c22-26
   luxadm display $disk |grep 'Device Address' 
   format $disk 
done

A truncated output example (without the actual format screens) will be something like;

/dev/rdsk/c6t50002AC001BD0928d0s2
01BD0
    Device Address              22110002ac000928,5
    Device Address              23110002ac000928,5
    Device Address              22120002ac000928,5
    Device Address              23120002ac000928,5
format /dev/rdsk/c6t50002AC001BD0928d0s2
/dev/rdsk/c6t50002AC001BE0928d0s2
01BE0
    Device Address              22110002ac000928,6
    Device Address              23110002ac000928,6
    Device Address              22120002ac000928,6
    Device Address              23120002ac000928,6
format /dev/rdsk/c6t50002AC001BE0928d0s2

Your format command with named and labelled LUNs will look something like:

       5. c6t50002AC001B70928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD01
          /scsi_vhci/disk@g50002ac001b70928
       6. c6t50002AC001B80928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD15
          /scsi_vhci/disk@g50002ac001b80928
       7. c6t50002AC001B90928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD02
          /scsi_vhci/disk@g50002ac001b90928
       8. c6t50002AC001BA0928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD03
          /scsi_vhci/disk@g50002ac001ba0928
       9. c6t50002AC001BB0928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD04
          /scsi_vhci/disk@g50002ac001bb0928
      10. c6t50002AC001BC0928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD05
          /scsi_vhci/disk@g50002ac001bc0928
      11. c6t50002AC001BD0928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD06
          /scsi_vhci/disk@g50002ac001bd0928
      12. c6t50002AC001BE0928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD07
          /scsi_vhci/disk@g50002ac001be0928
      13. c6t50002AC001BF0928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD08
          /scsi_vhci/disk@g50002ac001bf0928
      14. c6t50002AC001C00928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD09
          /scsi_vhci/disk@g50002ac001c00928
      15. c6t50002AC001C10928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD10
          /scsi_vhci/disk@g50002ac001c10928
      16. c6t50002AC001C20928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD11
          /scsi_vhci/disk@g50002ac001c20928
      17. c6t50002AC001C30928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD12
          /scsi_vhci/disk@g50002ac001c30928
      18. c6t50002AC001C40928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD13
          /scsi_vhci/disk@g50002ac001c40928
      19. c6t50002AC001C50928d0 <DEFAULT cyl 5218 alt 2 hd 255 sec 63>  MKCHAD14
          /scsi_vhci/disk@g50002ac001c50928
Specify disk (enter its number): 

The next steps are to configure your LUNs as required. This will depend on the intended usage, perhaps they will be used as RAW or ASM volumes for database usage, perhaps as ZFS zpools for filesystem usage.

Paul.


February 24, 2011

OpsCenter and ISC DHCP server

We have recently received a new OpsCentre (center?) installation. I must say, I am suitably (sadly, predictably) underwhelmed. It is slow, sluggish and fairly non-intuitive. It often fails on seemingly the most simple task (such as server discovery) for the most unfathomable of reasons. The error messages within the GUI for the tasks it fails are also meaningless.

One of the most recent failures I experienced with OpsCentre was during the import of a pre-cooked jet template and flash archive. I managed to create the profile and plan (although it didn’t seem to like my compressed flar archive and I had to switch to an uncompressed one). I was hoping, given that I know both the flash archive and the template are good that this would be as simple as pointing OpsCenter at the necessary files and it would do the rest. However, when I attempted to apply this new deployment plan and profile to a client I received the following errors:

The DeploymentPlan execution job failed because the DeploymentProvider OSDeployment
 for Step Provision OS failed to generate tasks for the job: 
com.sun.xvm.services.osdeployment.OSDeploymentException: Cannot prepare OS profile: 
com.sun.xvm.services.osdeployment.OSDeploymentException: Failed to retrieve Profile 
Information from database. (32557). Contact My Oracle Support if the problem persists. (10445)

At this point I asked myself why I was wasting time using a non-functional GUI when I could quite easily use the command line and configure the local ISC DHCP server on the OpsCentre controller to be my local DHCP server for a local client and then instruct the client to install from our existing, but ‘subnet-remote’ Jet jumpstart server using a flash archive over NFS.

Before I continue, I should probably concede that OpsCentre does seem to be quite good at patching. It will keep track of baselines, let you select baselines for application to clients, check dependencies and download/install all the dependent patches and install them for you in a hands-off manner. Apparently this is the pre-existing portion of the product that Sun/Oracle obtained when they bought the company…. anyway, enough of that.

The ISC DHCP server

For a while now I’ve heard talk about the ISC (Internet Software Consortium) DHCP server as an alternative to the Sun standard DHCP server without actually taking/finding the time to investigate it. Although I used to loath the Sun DHCP server, I have grown to love, like, ok, I’ve just about become accustomed to it. The fact that this is bundled with OpsCentre presented me with a perfect opportunity to test drive the ISC DHCP server. I will admit, based on first impressions, it is far easier to configure and seems more predictable in behaviour than the Sun DHCP server.

First, (and this will be different for a ISC DHCP server that you down load) I note that the service is managed by the following SMF service:

-bash-3.00$ svcs dhcpd 
STATE          STIME    FMRI
online         13:36:26 svc:/application/scn/dhcpd:default
-bash-3.00$ 

One of the first thing that you need to know is that on start/stop it logs to

/var/svc/log/application-scn-dhcpd:default.log

A successful startup will end with the lines:

Starting dhcpd: nxge0Internet Software Consortium DHCP Server V3.0pl2
Copyright 1995-2003 Internet Software Consortium.
All rights reserved.
For info, please visit http://www.isc.org/products/DHCP
Wrote 0 deleted host decls to leases file.
Wrote 0 new dynamic host decls to leases file.
Wrote 0 leases to leases file.
Listening on DLPI/nxge0/00:14:4f:f3:43:f2/10.20.30.0/24
Sending on   DLPI/nxge0/00:14:4f:f3:43:f2/10.20.30.0/24
Sending on   Socket/fallback/fallback-net
[ Feb  2 13:36:26 Method "start" exited with status 0 ]

The configuration file for the ISC DHCP server is located in /etc and is named dhcpd.conf

-bash-3.00$ ls -l /etc/dhcpd.conf
-rw-r--r--   1 root     sys         7100 Feb  2 12:51 /etc/dhcpd.conf

One of the first changes I made was to enable syslog logging for the service. This is acheived by adding a line similar to the following to the config file:

log-facility local6;

You’ll then need something like the following added to the /etc/syslog.conf file – don’t forget any whitespace must be tab only.

local6.info;local6.notice;local6.warn;local6.err;local6.debug   /var/log/dhcpd.log

The /var/log/dhcpd.log file will be your friend during any troubleshooting of client boot problems.

Then, rather than edit the file directly any further, I opted for an include file in which I place all the configuration options for the client(s) I wis to DHCP boot. The very last line of my dhcpd.conf, therefore, reads:

include "/export/dhcp/sol10_x86_dhcpd.conf";

The sol10_x86_dhcpd.conf file contains the following to setup the required Sun macros:

group {
  use-host-decl-names on;
  filename "pxegrub.I86PC.Solaris_10-1";
  next-server 10.20.30.40;
  vendor-option-space SUNW;
  option SUNW.JumpStart-server "192.168.10.1:/opt/SUNWjet";
  option SUNW.root-server-hostname "jumpstart-svr";
  option SUNW.root-server-ip-address 192.168.10.1;
  option SUNW.root-path-name "/export/install/media/x86/solaris10u9/boot";
  option SUNW.install-path "/export/install/media/x86/solaris10u9";

host utah {
        hardware ethernet 0:21:28:5f:32:16;
        fixed-address 10.20.30.41;
        option SUNW.sysid-config-file-server = "192.168.10.1:/opt/SUNWjet/Clients/utah";
        }
}

This configuration sets the required dhcp options which enable the client to reach a remote jumpstart install and sysidcfg server across the network. The client will also need instructions in the grub boot menu it receives from the dhcp server; this file is usually found in /tftpboot on the server; I edited this to include the following

-bash-3.00$ more /tftpboot/menu.lst.010021285F3216
default=0
timeout=2
min_mem64 1024
title Solaris_10 Jumpstart
        kernel$ /I86PC.Solaris_10-1/multiboot kernel/$ISADIR/unix  - install dhcp -B \ 
console=ttya install_config=192.168.10.1:/opt/SUNWjet,sysid_config=192.168.10.1: \
/opt/SUNWjet/Clients/utah,install_media=192.168.10.1:/export/install/media \
/x86/solaris10u9,install_boot=192.168.10.1:/export/install/media/x86/solaris10u9/boot
        module$ /I86PC.Solaris_10-1/$ISADIR/x86.miniroot

The jet / jumpstart server will of course need to be configured appropriately for the client. The /etc/SUNWjet/etc/dhcp.conf should set the following option to NOOP because the jet / jumpstart server itself is not going to be used as the DHCP server;

# DHCP_FORMAT   Which dhcp server type is supported. SUN is currently supported,
#               however, ISC is still in development. If REMOTE_DHCP is set,
#               you must set DHCP_FORMAT to SUN.  Valid entries are "SUN" 
#               and "NOOP". NOOP can be used if you are manually setting
#               up your own DHCP server with the required parameters.
#
DHCP_FORMAT="NOOP"

Once all this is straight, you should be able to jumpstart install your client. A useful command for troubleshooting the dhcp conversation between the client and the dhcp server is snoop. You should see DHCPDISCOVER broadcast from the client and DHCPOFFER sent back. The client should then return an accept pdu so that the server knows the client is accepting the IP address and the lease with a DHCPREQUEST, finally a DHCPACK will be sent from the server to the client. To see this in detail (including a display of the DHCP options sent by the server, use the following;

bash-3.00# snoop -vvv -d nxge0 port bootpc or port bootps
Using device nxge0 (promiscuous mode)

This output will be useful, but quite verbose – a shorter summary will be seen in the dhcpd.log file configured in syslog;

Feb  2 15:58:43 client1 dhcpd: [ID 702911 local6.info] DHCPDISCOVER from 0:2:8:5:3:16 via nxge0
Feb  2 15:58:43 turkana dhcpd: [ID 702911 local6.info] DHCPOFFER on 10.20.30.41 to 0:2:8:5:3:16 via nxge0
Feb  2 15:58:46 client1 dhcpd: [ID 702911 local6.info] DHCPREQUEST for 10.20.30.41 (10.20.30.40) from 0:2:8:5:3:16 via nxge0
Feb  2 15:58:46 client1 dhcpd: [ID 702911 local6.info] DHCPACK on 10.20.30.41 to 0:2:8:5:3:16 via nxge0

Paul.


February 07, 2011

Silliest environment variable ever?

It seems that under Solaris 11, if you can’t be bothered to type ’-s’ everytime you want a continuous ping response from a peer, you can instead set the environment variable MACHINE_THAT_GOES_PING to something non-null and voila…. huh?

sol-esx01:~$ ping sol-esx03
sol-esx03 is alive
sol-esx01:~$ 
sol-esx01:~$ export MACHINE_THAT_GOES_PING="true" 
sol-esx01:~$ 
sol-esx01:~$ ping sol-esx03
PING sol-esx03: 56 data bytes
64 bytes from sol-esx03 (192.168.100.6): icmp_seq=0. time=0.552 ms
64 bytes from sol-esx03 (192.168.100.6): icmp_seq=1. time=0.257 ms
64 bytes from sol-esx03 (192.168.100.6): icmp_seq=2. time=0.222 ms
64 bytes from sol-esx03 (192.168.100.6): icmp_seq=3. time=0.246 ms
64 bytes from sol-esx03 (192.168.100.6): icmp_seq=4. time=0.329 ms
^C
----sol-esx03 PING Statistics----
5 packets transmitted, 5 packets received, 0% packet loss
round-trip (ms)  min/avg/max/stddev = 0.222/0.321/0.552/0.135
sol-esx01:~$ 

Seriously, I really do wonder what the case for implementing this was. I can only imagine this is to make Solaris mimic the behaviour of linux.

Maybe I’m missing something… Please tell me I am.

Paul.


February 04, 2011

IPFilter changes in Solaris 11 Express

Configuration of IP Filter in Solaris 11 Express has changed. It used to be fairly simple; decide on some rules and put them in /etc/ipf/ipf.conf, enable the service with ‘svcadm enable ipfilter’ and check with ipfstat / ipfstat -io. This is optional, but I like to customise my logging by adding ‘log first level local0.info’ to rules which I would like information about when they are fired. This also leads to the following line in syslog.conf:

#
# logging for ipfilter
#
local0.info;local0.notice;local0.warn;local0.err;local0.debug   /var/log/ipfilter.log

However, in Solaris 11 Express, by default the rules for your firewall are not expected to be found in a configuration file. Instead they have become SMF properties for the IP Filter service. If you attempt to put an ipf.conf file in place and start up IP Filter, everything will seem to start, but you will have no matches as intended because the rule base will be empty:

root@sol-esx01:/var/log# ipfstat -io
empty list for ipfilter(out)
empty list for ipfilter(in)

The answer to this lies in the man page to svc.ipfd(1m);

 firewall_config_default/policy

         Global Default policy, firewall_config property group in
         svc:/network/ipfilter:default,  can  also be set to cus-
         tom. Users can set policy to custom to use  prepopulated
         IP  Filter  configuration,  for  example, an existing IP
         Filter configuration or custom configurations that  can-
         not  be  provided by the framework. This Global Default-
         only policy mode allows users to supply a text file con-
         taining  the complete set of IPF rules. When custom mode
         is selected, the specified set of IPF rules is  complete
         and  the framework will not generate IPF rules from con-
         figured firewall policies.

     firewall_config_default/custom_policy_file

         A file path to be used when Global Default policy is set
         to  custom.  The  file  contains a set of IPF rules that
         provide the desired IP Filter configuration.  For  exam-
         ple,  users with existing IPF rules in /etc/ipf/ipf.conf
         can execute the following commands to use  the  existing
         rules:

             1.   Set custom policy:

                    # svccfg -s ipfilter:default setprop \
                    firewall_config_default/policy = astring: "custom" 

             2.   Specify custom file:

                    # svccfg -s ipfilter:default setprop \
                    firewall_config_default/custom_policy_file = astring: \
                    "/etc/ipf/ipf.conf" 

             3.   Refresh configuration:

                    # svcadm refresh ipfilter:default

So, following the instructions;

# svccfg -s ipfilter:default setprop firewall_config_default/policy = astring: "custom" 

# svccfg -s ipfilter:default listprop firewall_config_default/policy
firewall_config_default/policy  astring  custom

# svccfg -s ipfilter:default setprop firewall_config_default/custom_policy_file = astring: "/etc/ipf/ipf.conf" 

# svccfg -s ipfilter:default listprop firewall_config_default/custom_policy_file
firewall_config_default/custom_policy_file  astring  /etc/ipf/ipf.conf

# svcadm refresh ipfilter:default

Seems to do the trick….

# svcs -xv ipfiltersvc:/network/ipfilter:default (IP Filter)
 State: online since February  4, 2011 02:05:01 PM GMT
   See: man -M /usr/share/man -s 5 ipfilter
   See: /var/svc/log/network-ipfilter:default.log
Impact: None.

#
# ipfstat  |grep blocked
 input packets:         blocked 989 passed 1766 nomatch 1753 counted 0 short 0
output packets:         blocked 0 passed 1390 nomatch 1377 counted 0 short 0
 input packets logged:  blocked 403 passed 13
output packets logged:  blocked 0 passed 0

# ipfstat -io |head

pass out quick on lo0 all
pass out quick on e1000g0 proto udp from any to any
pass out quick on e1000g0 proto tcp from any to any flags S/FSRPAU keep state
block in quick from any to any with short
block in quick on e1000g0 from any to any port = 137
block in quick on e1000g0 from any to any port = 138
block in quick on e1000g0 from any to any port = 1434
pass in quick on lo0 all
pass in quick on e1000g0 from 12.34.56.78/32 to any keep state
pass in log first level local0.info quick on e1000g0 proto tcp from 12.34.56.78/16 to any port = ssh flags S/FSRPAU keep state

I’m not sure I’ll ever use anything other than a configuration file for the firewall configs. The configs I need to use are quite complex and lengthy and management/distribution of this seems easiest through a standard text file.

Incidentally, there does seem to be a function to do this for you in /lib/svc/method/ipfilter;

upgrade_config()
{
    old_ipfconf="/etc/ipf/ipf.conf" 

    if [ -f ${old_ipfconf} ]; then
        grep '^[ \t]*[^# \t]' ${old_ipfconf} >/dev/null 2>&1
        if [ $? -eq 0 ]; then
            svccfg -s $SMF_FMRI setprop \
                $FW_CONFIG_DEF_PG/$POLICY_PROP = astring: \
                "custom" >/dev/null 2>&1
            svccfg -s $SMF_FMRI setprop \
                $FW_CONFIG_DEF_PG/$CUSTOM_FILE_PROP = astring: \
                "$old_ipfconf" >/dev/null 2>&1
        fi
    fi

    svccfg -s $SMF_FMRI setprop $FW_CONFIG_DEF_PG/version = count: \
        "$CURRENT_VERSION" >/dev/null 2>&1
    svcadm refresh $SMF_FMRI >/dev/null 2>&1
}

But is isn’t clear to me how this gets called. There is no reference to it from the case statement – none of these options results in it being called (echo “(start|stop|reload|reipf|reipnat|pause|resume)”).

Paul.


February 2011

Mo Tu We Th Fr Sa Su
Jan |  Today  | Mar
   1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28                  

Search this blog

Tags

Galleries

Most recent comments

  • Started sorting out new vers for sparc: http://blogs.warwick.ac.uk/mariamaccallum/entry/apache_249_i… by Maria MacCallum on this entry
  • Solaris 11.1 is slightly different, I only had to do this before starting ipfilter: svccfg –s setpro… by Maria MacCallum on this entry
  • Really useful information, thanks a lot! I do a NAT using IPFILTER and all was working good, until I… by Nilton on this entry
  • Paul, Thanks for your information. It got me started quickly. I have discovered , thought I've not s… by Tom C on this entry
  • Are you familiar with the Monty Python sketch? by Ian Eiloart on this entry

Blog archive

Loading…
Not signed in
Sign in

Powered by BlogBuilder
© MMXIX