XCAT_system_x_support_for_IBM_Flex

There is a newer version of this page. You can find it here.

Introduction

IBM Flex combines networking, storage, and compute nodes in a single offering. It's consist of an IBM Flex Chassis, one or two Chassis Management Modules(CMM) and compute nodes. The compute nodes include the IBM Flex System™ p260, p460, and 24L Power 7 servers as well as the IBM Flex System™ x240 x86 server. In this document only the management of x240 blade server will be covered.

Terminology

The following terms will be used in this document:

Chassis Management Module(CMM) - this term is used to reflect the pair of management modules installed in the rear of the chassis and connected by ethernet to the MN. The CMM is used to discover the compute nodes within the chassis and for some data collection regarding the nodes and chassis.

Compute node: This term is used to refer to the servers in an IBM Flex system. Compute nodes can be either Power 7 servers or x86 Intel based servers.

blade node: blade node refers to a node with the hwtype set to blade and represents the whole compute node server.

Compute blade: refers to a compute server with a blade hwtype.

Overview of Cluster Setup Process

Here is a summary of the steps required to set up the cluster and what this document will take you through:

  1. Prepare the management node - doing these things before installing the xCAT software helps the process to go more smoothly.
  2. Install the xCAT software on the management node.
  3. Configure some cluster wide information
  4. Define a little bit of information in the xCAT database about the ethernet switches and nodes - this is necessary to direct the node discovery process.
  5. Have xCAT configure and start several network daemons - this is necessary for both node discovery and node installation.
  6. Discovery the nodes - during this phase, xCAT configures the BMC's and collects many attributes about each node and stores them in the database.
  7. Set up the OS images and install the nodes.

Distro-specific Steps

  • [RH] indicates that step only needs to be done for RHEL and Red Hat based distros (CentOS, Scientific Linux, and in most cases Fedora).
  • [SLES] indicates that step only needs to be done for SLES.

Command Man Pages and Database Attribute Descriptions

Prepare the Management Node for xCAT Installation

{{:Prepare the Management Node for xCAT Installation}}

Note: for Flex hardware, the switch configuration is only needed to discover (really to locate) the CMMs. The location of each blade is determined by the CMMs.

Install xCAT on the Management Node

{{:Install xCAT on the Management Node}}

Create group entries for common attributes

There are several attributes that are common for a given node group. This section will give examples of setting common attributes for both the cmm and blade groups.

CMM common attributes 

mkdef -t group cmm groups=cmm,all hwtype=cmm mgt=blade

blade common attributes 

mkdef -t group blade cons=blade groups=blade,all mgt=ipmi nodetype=osi,blade arch=x86_64 installnic=mac profile=compute 
netboot=xnba getmac=blade serialport=0 serialspeed=115200

This section will describe the commands needed to define both the CMM node definitions as well as the blade definitions.

Define the CMM with IP setting

You can define the CMM one definition at a time or using a pattern you can define several CMM at once using mkdef. Below are examples of creating a single entry and also creating multiple entries with a pattern.

This command will define a single CMM entry. 

# mkdef cmm7 mpa=cmm07 ip=70.0.0.7

Listing the new entry will look like this: 

# lsdef cmm7
Object name: cmm7
    groups=all,cmm
    hwtype=cmm
    ip=70.0.0.7
    mgt=blade
    mpa=cmm7
    postbootscripts=otherpkgs
    postscripts=syslog,remoteshell,syncfiles

This mkdef command will use a pattern to define two CMM entries and will also use the CMM number at the end of the name to help define the mpa and the ip so that they are using this value: 

# mkdef cmm[8-9] mpa='|cmm(\d+)|cmm($1)|' ip='|cmm(\d+)|70.0.0.($1)|'

Listing these new entries will look like this: 

# lsdef cmm
Object name: cmm8
   groups=all,cmm
   hwtype=cmm
   ip=70.0.0.8
   mgt=blade
   mpa=cmm8
   postbootscripts=otherpkgs
   postscripts=syslog,remoteshell,syncfiles
Object name: cmm9
   groups=all,cmm
   hwtype=cmm
   ip=70.0.0.9
   mgt=blade
   mpa=cmm9
   postbootscripts=otherpkgs
   postscripts=syslog,remoteshell,syncfiles

Define the blades for each CMM

You can define the blades one at a time or in a group using a patter.

This command will define one blade entry: 

#  mkdef cmm04node01 conserver=9.114.34.219 monserver=9.114.34.219
nfsserver=9.114.34.219 tftpserver=9.114.34.219 xcatmaster=9.114.34.219 
1 object definitions have been created or modified.

Listing the new blade node entry: 

# lsdef cmm04node01
Object name: cmm04node01
   arch=x86_64
   bmc=10.0.4.1
   bmcpassword=passw0rd
   bmcusername=USERID
   cons=blade
   conserver=9.114.34.219
   getmac=blade
   groups=blade,all
   id=1
   installnic=mac
   mgt=ipmi
   monserver=9.114.34.219
   mpa=cmm04
   netboot=xnba
   nfsserver=9.114.34.219
   nodetype=osi,blade
   postbootscripts=otherpkgs
   postscripts=syslog,remoteshell,syncfiles
   profile=compute
   tftpserver=9.114.34.219
   xcatmaster=9.114.34.219

This command will define all of the blades for cmm1 and cmm2. Note that it will create 14 blades per chassis. If this is not your configuration you can use this method and then delete unwanted blade definitions or use a more manual method. 

#  mkdef cmm[01-02]node[01-14] conserver=9.114.34.219 monserver=9.114.34.219 
nfsserver=9.114.34.219 tftpserver=9.114.34.219 xcatmaster=9.114.34.219
28 object definitions have been created or modified.

listing the blade node entries: 

# nodels blade
cmm01node01
cmm01node02
cmm01node03
cmm01node04
cmm01node05
cmm01node06
cmm01node07
cmm01node08
cmm01node09
cmm01node10
cmm01node11
cmm01node12
cmm01node13
cmm01node14
cmm02node01
cmm02node02
cmm02node03
cmm02node04
cmm02node05
cmm02node06
cmm02node07
cmm02node08
cmm02node09
cmm02node10
cmm02node11
cmm02node12
cmm02node13
cmm02node14

Listing one of th eblade definitions looks like this: 

[root@x3850n02 ~]# lsdef cmm02node14
Object name: cmm02node14
   arch=x86_64
   bmc=cmm02node14
   bmcpassword=passw0rd
   bmcusername=USERID
   cons=blade
   conserver=9.114.34.219
   getmac=blade
   groups=blade,all
   id=cmm02node14
   installnic=mac
   mgt=ipmi
   monserver=9.114.34.219
   mpa=cmm02
   netboot=xnba
   nfsserver=9.114.34.219
   nodetype=osi,blade
   postbootscripts=otherpkgs
   postscripts=syslog,remoteshell,syncfiles
   profile=compute
   tftpserver=9.114.34.219
   xcatmaster=9.114.34.219

Define the mpa and slot id for each blade in the mp table

This section provides an example of how to modify the mp table to define the blade and cmm configuration. 

# tabdump mp
#node,mpa,id,nodetype,comments,disable
"blade","|cmm(\d+)node(\d+)|cmm($1)|","|cmm(\d+)node(\d+)|($2+0)|",,,
"cmm01","cmm01",,cmm,,
"cmm02","cmm02",,cmm,,

For example the cmm mpa will be defined as follows: 

# lsdef cmm01
Object name: cmm01
    groups=all,cmm
    ip=70.0.0.7
    mac=5c:f3:fc:25:da:99
    mgt=blade 
    mpa=cmm01
    postbootscripts=otherpkgs
    postscripts=syslog,remoteshell,syncfiles

Update the password table for blade, ipmi, and system

There are several passwords required for management.

blade - The userid and password for the blade management plugin which needs to access the CMM to control the blade. This is the login ID for the CMM. ipmi - The ipmi interface to the blade BMC. This is the same ID as the CMM USERID. system - The root id and password which will be set on the node OS during installation and used later for the administrator to login to the node OS.

Update the passwd table with the following: 

tabdump passwd
#key,username,password,cryptmethod,comments,disable
"blade","USERID","Passw0rd",,,
"ipmi","USERID","Passw0rd",,,
"system","root","cluster",,,

Prepare the CMM discovery

xCAT supports two methods to do the CMM matching, one is matching CMM by the CMM switch port information; the other way is matching CMM by the known CMM MAC address (This requires that the user has already collected the MAC of CMM).

Note: xCAT Flex discovery now does not support the CMM with both primary and standby port.

Mapping the CMMs to the switch port information

In large clusters the best automated method for discovering and mapping the CMM and blades to the objects defined is to allow xCAT to poll data from the site switch to which each chassis CMM ethernet port is connected. This will allow xcat to map the CMM to the definition in the switch table and then use the CMM to map the blades to the defined nodes. You need add the switch/port information below to the switch and switches tables to allow xCAT to access and discover the CMMs and blades. 

# tabdump switch
#node,switch,port,vlan,interface,comments,disable
"cmm01","switch","0/1",,,,
"cmm02","switch","0/2",,,,

where: node is the cmm node object name switch is the hostname of the switch port is the switch port id. Note that xCAT does not need the complete port name. Preceeding non numeric characters are ignored.

If you configured your switches to use SNMP V3, then you need to define several attributes in the switches table. Assuming all of your switches use the same values, you can set these attributes at the group level: 

tabch switch=switch switches.snmpversion=3 switches.username=xcatadmin switches.password=passw0rd switches.auth=SHA


# tabdump switches
#switch,snmpversion,username,password,privacy,auth,linkports,sshusername,sshpassword,switchtype,comments,disable
"switch","3","xcatadmin","passw0rd",,"SHA",,,,,,

Note: It might also be necessary to allow authentication at the VLAN level 

snmp-server group xcatadmin v3 auth context vlan-230

Matching by the CMM MAC address

If you use MAC information to do the CMM matching then you can get the mac manually using the either the arp command.

You can also collect the CMM mac addresses automatically by discovering the CMM via lsslp 

# lsslp -m -z -s CMM
cmm01:
        objtype=node
        mpa=cmm01
        nodetype=mp
        mtm=789392X
        serial=100037A
        side=2
        groups=cmm,all
        mgt=blade
        mac=5c:f3:fc:25:da:99
        hidden=0
        otherinterfaces=70.0.0.7
        hwtype=cmm

run slpdiscover (xCAT 2.7.x)/ lsslp --flexdiscover (xCAT 2.8)

The slpdiscover command has been replaced by lsslp --flexdiscover in xCAT 2.8. If you are using the lower version ( xCAT 2.7.x ), the command should be slpdiscover. It will perform the last steps in discovery and configuration needed to complete the hardware configuration for both the CMM and the blades for both the switch port information and CMM MAC discovery methods.

Note: If you have run slpdiscover (xCAT 2.7.x)/ lsslp --flexdiscover (xCAT 2.8) before please delete the ipmi.bmcid.

slpdiscover (xCAT 2.7.x)/ lsslp --flexdiscover (xCAT 2.8) performs the following functions:

  1. locate the CMM
  2. configure the CMM
  3. locate the blades by slot id
  4. configure the IMM IP/USER ID/PW for each nodes that has been defined in MN.

Note: xCAT only supports the CMM connecting to the vlan 1 of the switch setting.  

# lsslp --flexdiscover
cmm01: Found service:management-hardware.IBM:chassis-management-module at address 70.0.0.7
cmm01: Ignoring target in bay 7, no node found with mp.mpa/mp.id matching
cmm01: Ignoring target in bay 8, no node found with mp.mpa/mp.id matching
Configuration of cmm01node05[70.0.0.16] commencing, configuration may take a few minutes to take effect

Note: 

  1. "cmm01: Ignoring target in bay 7, no node found with mp.mpa/mp.id matching" This message means the bay 7 does not have a node defined on MN for this CMM and slot id. 
  2. the IMM password will be same as the CMM password setting.

Checking the result of the slpdiscover (xCAT 2.7.x)/ lsslp --flexdiscover (xCAT 2.8) command

After the slpdiscover/lsslp --flexdiscover the blades and their interfaces should be defined and configured properly.

List the blade node defintions

User can list the blade definitions with nodels: 

# nodels blade
cmm01node05
cmm01node06
cmm01node11
cmm01node12

List a blade node attributes

# lsdef cmm01node11
Object name: cmm01node11
    bmc=70.0.0.9
    bmcpassword=Passw0rd1
    bmcusername=USERID
    cons=ipmi
    getmac=blade
    groups=all,blade
    id=11
    mac=34:40:b5:be:7e:f8
    mgt=ipmi
    mpa=cmm01
    postbootscripts=otherpkgs
    postscripts=syslog,remoteshell,syncfiles

Check the ipmi table

After the slpdiscover (xCAT 2.7.x)/ lsslp --flexdiscover (xCAT 2.8), user also can list the ipmi table to verify that the bmcid has been updated with blade BMC mac. 

# tabdump ipmi
#node,bmc,bmcport,taggedvlan,bmcid,username,password,comments,disable
"blade","|cmm(\d+)node(\d+)|10.0.($1+0).($2+0)|",,,,"USERID","passw0rd",,
"cmm01node05","70.0.0.16",,,"5c:f3:fc:6e:00:41",,,,
"cmm01node06","70.0.0.15",,,"5c:f3:fc:6e:03:94",,,,

Verify that hardware control is working

User can ping the bmc ip and try the

hardware control commands to verify that the bmc ip and userid/password have been defined and configured correctly. 

# rpower cmm01node11 stat
cmm01node11: on


# rinv cmm01node11 vpd
cmm01node11: System Description: IBM Flex System x240+10Gb Fabric
cmm01node11: System Model/MTM: 8737AC1
cmm01node11: System Serial Number: 23FFP63
cmm01node11: Chassis Serial Number: 23FFP63
cmm01node11: Device ID: 32
cmm01node11: Manufacturer ID: IBM (20301)
cmm01node11: BMC Firmware: 1.34 (1AOO27Q 2012/05/04 22:00:54)
cmm01node11: Product ID: 321


# rvitals cmm01node11 leds
cmm01node11: No active error LEDs detected

Using ASU to Update CMOS, uEFI, or BIOS Settings on the Nodes

There are a few ASU settings that need to be changed from the defaults. This section will discuss what needs to be changed and how to run asu update.

Download ASU

If you need to update CMOS/uEFI/BIOS settings on your nodes, download ASU (Advanced Settings Utility) from the IBM Fix Central web site:

Once you have the ASU RPM on your MN (management node), you have several choices of how to run it:

Run ASU Out-of-Band

ASU can be run on the management node (MN) and told to connect to the IMM of a node. First install ASU on the MN: 

rpm -i ibm_utl_asu_asut78c-9.21_linux_x86-64.rpm
cd /opt/ibm/toolscenter/asu

Determine the IP address, username, and password of the IMM (BMC): 

lsdef node1 -i bmc,bmcusername,bmcpasswd
tabdump passwd | grep ipmi      # the default if username and password are not set for the node

Run ASU: 

./asu64 show all --host <ip> --user <username> --password <pw>
./asu64 show uEFI.ProcessorHyperThreading --host <ip> --user <username> --password <pw>
./asu64 set uEFI.RemoteConsoleRedirection Enable --host <ip> --user <username> --password <pw>  # a common setting that needs to be set

If you want to set a lot of settings, you can put them in a file and run: 

./asu64 batch <settingsfile> --host <ip> --user <username> --password <pw>

These are the settings needed to enable the serial console:

TBD

These are other settings which are needed. 

UEFI Boot/Physical Serial:


loaddefault BootOrder
loaddefault uEFI
set Processors.Hyper-Threading Disable
set BootOrder.BootOrder "PXE Network=Hard Disk 0"

Collect and set the MAC address in preparation for deployment

In order to successfully deploy the OS you will need to associate the blade thernet MAC with the blade node object.

Listing the MAC 

# rinv cmm01node11 mac
cmm01node11: MAC Address 1: 34:40:b5:be:c0:08
cmm01node11: MAC Address 2: 34:40:b5:be:c0:0c

Setting the MAC 

# chdef cmm01node11 mac=34:40:b5:be:c0:08

Listing the MAC in the node object 

# lsdef cmm01node11 -i mac
Object name: cmm01node11
   mac=34:40:b5:be:c0:08

Blade node provisioning

Documentation for system x blade node provisioning is in another document which describes the steps necesary to properly provision the nodes. When reading this document keep in mind that there are differences in the attributes between the document which is desribing IDataPlex nodes and these balde nodes. Here is the link to the provisioning document. 

https://sourceforge.net/apps/mediawiki/xcat/index.php?title=XCAT_iDataPlex_Advanced_Setup

Update the CMM firmware

The CMM firmware can be updated by loading the latest cmefs.uxp firmware file using the CMM update command working with the http interface. The administrator needs to download firmware from IBM Fix Central. The compressed tar file will need to be uncompressed and unzipped to extract the firmware update files. Place the cmefs.uxp file in a specified directory on the xCAT MN.

Once the firmware is unzipped and the cmefs.uxp is placed in the directory on the xCAT MN you can use the CMM update command to update the new firmware on one chassis at a time or on all chassis managed by xCAT MN. More details on the CMM update command can be found at: http://publib.boulder.ibm.com/infocenter/flexsys/information/index.jsp?topic=%2Fcom.ibm.acc.cmm.doc%2Fcli_command_update.html

The format of the update command is: flash (-u) the file and reboot (-r) afterwards 

update -T system:mm[1] -r -u http://<server>/<MN directory>/<update file>

flash (-u), show progress (-v), and reboot (-r) afterwards 

update -T system:mm[1] -v -r -u http://<server>/<MN directory>/<update file>

To update firmware and restart a single CMM cmm01 from xCAT MN 70.0.0.1 use: 

ssh USERID@cmm01 update -T system:mm[1] -v -r -u http://70.0.0.1/firmware/cmefs.uxp

If unprompted password is setup on all CMMs then you can use xCAT psh to update all CMMs in the cluster at once. 

psh -l USERID cmm update -T system:mm[1] -v -u http://70.0.0.1/firmware/cmefs.uxp

If you are experiencing a "Unsupported security level" message after the CMM firmware was updated then you should run the following command to overcome this issue. 

rspconfig cmm sshcfg=enable snmpcfg=enable

Blade node firmware updates

Documentation for the system x firmware updates in included in the link below. As with the provisioning document keep in mind that there are some differences in attribues between IDataPlex and blade nodes in the documentation. 

https://sourceforge.net/apps/mediawiki/xcat/index.php?title=XCAT_iDataPlex_Advanced_Setup#Updating_Node_Firmware
MongoDB Logo MongoDB