XCAT_AIX_RTE_Diskfull_Nodes

There is a newer version of this page. You can find it here.

**Installing AIX standalone nodes ** (using standard NIM rte method) 11/09/2010, AM 10:35:04

Overview

The process uses xCAT features to automatically run the necessary NIM commands.

  1. NIM is an AIX tool that enables a cluster administrator to centrally manage the installation and configuration of AIX and optional software on machines within a networked environment. This document assumes you are familiar with NIM. For more information about NIM, see the IBM AIX Installation Guide and Reference. (<http://www-03.ibm.com/servers/aix/library/index.html>)

The process described below is one basic set of steps that may be used to install an AIX standalone node using the NIM "rte" installation method and is not meant to be a comprehensive guide of all the available NIM options.

Before starting this process it is assumed you have completed the following.

  • An AIX system has been installed to use as an xCAT management node.
  • The cluster network is configured. (The Ethernet network that will be used to perform the network boot of the nodes.)
  • xCAT and prerequisite software has been installed and configured on the management node.
  • Any logical partitions that will be used have already been created using the HMC interfaces.

Installing AIX nodes (using standard NIM rte method)

Create an operating system image

Use the xCAT mknimimage command to create an xCAT osimage definition as well as the required NIM installation resources.

An xCAT osimage definition is used to keep track of a unique operating system image and how it will be deployed.

In order to use NIM to perform a remote network boot of a cluster node the NIM software must be installed, NIM must be configured, and some basic NIM resources must be created.

The mknimimage comnand will handle all the NIM setup as well as the creation of the xCAT osimage definition. It will not attempt to reinstall or reconfigure NIM if that process has already been completed. See the mknimimage man page for additional details.

Note: If you wish to install and configure NIM manually you can run the AIX nim_master_setup command (Ex. "nim_master_setup -a mk_resource=no -a device=<source directory>").

Note: For various reasons it is recommended that you make sure that the primary hostname of the management node is the interface that you will be using to install the nodes. If you do this before you configure NIM then NIM will automatically use it to define the NIM primary network. This will mean that you will not have to create any additional NIM network definitions and could avoid additional complications.

By default, the mknimimage command will create the NIM resources in subdirectories of /install. Some of the NIM resources are quite large (1-2G) so it may be necessary to increase the files size limit.

For example, to set the file size limit to "unlimited" for the user "root" you could run the following command. 

_**/usr/bin/chuser fsize=-1 root**_

When you run the command you must provide a source for the installable images. This could be the AIX product media, a directory containing the AIX images, or the name of an existing NIM lpp_source resource. You must also provide a name for the osimage you wish to create. This name will be used for the NIM SPOT resource that is created as well as the name of the xCAT osimage definition. The naming convention for the other NIM resources that are created is the osimage name followed by the NIM resource type, (ex. " 61cosi_lpp_source").

In this example we need resources for installing a NIM "standalone" type machine using the NIM "rte" install method. (This type and method are the defaults for the mknimimage command but you can specify other values on the command line.)

For example, to create an osimage named "610image" using the images contained in the /myimages directory you could run the following command. 

_**mknimimage -s /myimages 610image**_

(Creating the NIM resources could take a while!)

Note: To populate the /myimages directory you could copy the software from the AIX product media using the AIX gencopy command. For example you could run "gencopy -U -X -d /dev/cd0 -t /myimages all".

By default the command will create NIM lpp_source, spot, and bosinst_data resources. You can also specify alternate or additional resources on the command line using the "attr=value" option, ("<nim resource type>=<resource name>").

For example: 

_mknimimage -s /myimages 610image resolv_conf=my_resolv_conf_

Any additional NIM resources specified on the command line must be previously created using NIM interfaces. (Which means NIM must have been configured previously. )

Note: Another alternative is to run mknimimage without the additional resources and then simply add them to the xCAT osimage definition later. You can add or change the osimage definition at any time. When you initialize and install the nodes xCAT will use whatever resources are specified in the osimage definition.

When the command completes it will display the osimage definition which will contain the names of all the NIM resources that were created. The naming convention for the NIM resources that are created is the osimage name followed by the NIM resource type, (ex. " 610image_lpp_source"), except for the SPOT name. The default name for the SPOT resource will be the same as the osimage name.

For example: 

_Object name: 610image_
_bosinst_data=610image_bosinst_data_
_imagetype=NIM_
_lpp_source=610image_lpp_source_
_nimmethod=rte_
_nimtype=standalone_
_osname=AIX_
_resolv_conf=610image_resolv_conf_
_spot= 610image_

Once the initial osimage definition is created you can change it by using the chdef command. For example, you may need to create additional NIM resources to use when installing the nodes, such as script or installp_bundle resources.

To add an installp_bundle resource to the osimage definition created in the previous example you could run the chdef command as follows. 

_chdef -t osimage -o 610image installp_bundle=mybundlename _

The xCAT osimage definition can be listed using the lsdef command and removed using the rmnimimage command. See the man pages for details.

In some cases you may also want to modify the contents of the NIM resources. For example, you may want to change the bosinst_data file or add to the resolv_conf file etc. For details concerning the NIM resources refer to the NIM documentation.

You can list NIM resource definitions using the AIX lsnim command. For example, if the name of your SPOT resource is "610image" then you could get the details by running: 

_**lsnim -l 610image**_

To see the actual contents of a NIM resource use "nim -o showres <resource name>". For example, to get a list of the software installed in your SPOT you could run: 

_**nim -o showres 610image**_

Note: The mknimimage command will take care of the NIM master installation and configuration automatically, however, you can also do this using the standard AIX support. See the AIX documentation for details on using the nim_master_setup command or the SMIT "eznim" interface.

Create an image_data resource (optional)

Starting with xCAT 2.5 support has been added for NIM image_data resources.

A NIM image_data resource is a file that contains stanzas of information that is used when creating file systems on the node. To use this support you must create the file , define it as a NIM resource, and add it to the xCAT osimage definition.

To help simplify this process xCAT ships a sample image_data file called _/opt/xcat/share/xcat/image_data/xCATsnData. _This file assumes you will have at least 70G of disk space available. It also sets the physical partition size to 128M.

This sample image_data file is intended to be used when installing xCAT service nodes but it may also be used for basic standalone compute nodes.

It sets the following default file system sizes. 

_/var -&gt; 5G _
_/opt -&gt; 10G _
_/ -&gt; 30G _
_/usr -&gt; 4G _
_/tmp -&gt; 3G_
_/home -&gt; 0.12G_
_/admin -&gt; 0.12 G _
_/livedump -&gt; 0.25G_

If you need to change any of these be aware that you must change two stanzas for each file system. One is the fs_data and the other is the corresponding vg_data.

Once you have settled on a final version of the image_data file you can copy it to the location that will be used when defining NIM resources. (ex. /install/nim/image_data/myimage_data)

To define the NIM resource you could use the SMIT interfaces or run a command similer to the following. 

_**nim -o define -t image_data -a server=master -a location= /install/nim/image_data/myimage_data myimage_data**_

To add these bundle resources to your xCAT osimage definition run: 

_**chdef -t osimage -o 610SNimage image_data=myimage_data**_

For more information on using the image_data resource refer the the AIX/NIM documentation.

Add additional software

On xCAT cluster nodes you will have to install openssl and openssh as well as some prerequisite RPM packages that were included in the xCAT dependency tar file.

Note: If you have other additional software that you wish to have installed you can use the same basic method that is described below.

The software required by xCAT is specified in the sample bundle file discussed below. The installp filesets should be available from the AIX product media. The prerequisite rpms are available in the dep-aix-<version>.tar.gz tar file that you downloaded from the xCAT download page.

Copy the software to the lpp_source resource

The required software must be copied to the NIM lpp_source that is being used for this OS image. The easiest way to do this is to use the "nim -o update" command.

For example, assume all the required software has been copied and unwrapped in the /tmp/images directory.

Note: In newer versions of the dep-aix-<version>.tar.gz tar file the packges are found in subdirectories corresponding to the AIX OS version (ex. 6.1, 71. etc).

To add all the packages to your lpp_source resource, you can run the following: 

_**nim -o update -a packages=all -a source=/tmp/images/dep-aix/6.1 610image_lpp_source**_

The NIM command will find the correct directories and update the lpp_source resource.

Create NIM installp_bundle resources

To get this additional software installed we need a way to tell NIM to include it in the installation. To facilitate this you can use the sample bundle files that are provided by xCAT.

To use the bundle files you need to define them as NIM resources and add them to the xCAT osimage definition.

Starting with xCAT 2.4.3 there are two sets of sample bundle files provided. One set can be used when installing a service node and one set is used when install a compute node. For each set there is a version number in the name corresponding to the different AIX OS levels. (xCATaixCN53.bnd, xCATaixCN61.bnd etc.) Just use the one that corresponds to the version of AIX you are running. These sample files are installed in "/opt/xcat/share/xcat/installp_bundles". If you are using an older version of xCAT you can find sample bundle files included in the core-aix-<version>.tar.gz tar file.

Important note: Always make sure that the contents of the bundle files you use match the software that was copied to the lpp_source directory. (See the step below named "Check the osimage".)

Copy the bundle file ( say xCATaixCN.6.1.bnd) to a location where it can be defined as a NIM resource, for example "/install/nim/installp_bundle".

To define the NIM resources you can run the following commands. 

_**nim -o define -t installp_bundle -a server=master -a location= /install/nim/installp_bundle/xCATaixCN61.bnd xCATaixCN61**_

To add the bundle resources to your xCAT osimage definition run: 

_**chdef -t osimage -o 610image installp_bundle=xCATaixCN61**_

This bundle file will be included in the underlying NIM commands that are used and NIM will include this additional software when installing the nodes.

Check the osimage (optional)

To avoid potential problems when installing a node it is adviseable to verify that all the software that you wish to install has been copied to the appropriate NIM lpp_source directory.

Any software that is specified in the "otherpkgs" or the "installp_bundle" attributes of the osimage definition must be available in the lpp_source directories.

Also, if your bundle files include rpm entries that use a wildcard (*) you must make sure the lpp_source directory does not contain multiple packages that will match that entry. (NIM will attempt to install multiple version of the same package and produce an error!)

To find the location of the lpp_source directories run the "lsnim -l <lpp_source_name>" command. For example: 

_**lsnim -l 610image_lpp_source**_

If the location of your lpp_source resource is "/install/nim/lpp_source/610image_lpp_source/" then you would find rpm packages in "/install/nim/lpp_source/610image_lpp_source/RPMS/ppc" and you would find your installp and emgr packages in "/install/nim/lpp_source/610image_lpp_source/installp/ppc".

To find the location of the installp_bundle resource files you can use the NIM "lsnim -l" command. For example, 

_**lsnim -l xCATaixSSH**_

Starting with xCAT version 2.4.3 you can use the xCAT chkosimage command to do this checking. For example: 

_**chkosimage -V 61cosi**_

See the chkosimage man page for details.

Define xCAT networks

Create an xCAT network definition for each network that contains cluster nodes. You will need a name for the network and values for the following attributes. 

**net** The network address.
**mask** The network mask.
**gateway** The network gateway.

You can use the xCAT makenetworks command to gather cluster network information and create xCAT network definitions. See the makenetworks man page for details.

In our example we will assume that all the cluster node management interfaces and the xCAT management node interface are on the same network. You can use the xCAT mkdef command to define the network.

For example: 

_**mkdef -t network -o net1 net=9.114.113.224 mask=255.255.255.224 gateway=9.114.113.254**_
  • Note: The xCAT definition should correspond to the NIM network definition. If multiple cluster subnets are needed then you will need an xCAT and NIM network definition for each one.

Create additional NIM network definitions (optional)

For the processs described in this document we are assuming that the xCAT management node and the LPARs are all on the same network.

However, depending on your specific situation, you may need to create additional NIM network and route definitions.

NIM network definitions represent the networks used in the NIM environment. When you configure NIM, the primary network associated with the NIM master is automatically defined. You need to define additional networks only if there are nodes that reside on other local area networks or subnets. If the physical network is changed in any way, the NIM network definitions need to be modified.

To create the NIM network definitions corresponding to the xCAT network definitions you can use the xCAT xcat2nim command.

For example, to create the NIM definitions corresponding to the xCAT "clstr_net" network you could run the following command. 

_**xcat2nim -V -t network -o clstr_net**_

Manual method

The following is an example of how to define a new NIM network using the NIM command line interface.

Step 1

Create a NIM network definition. Assume the NIM name for the new network is "clstr_net", the network address is "10.0.0.0", the network mask is "255.0.0.0", and the default gateway is "10.0.0.247".

nim -o define -t ent -a net_addr=10.0.0.0 -a snm=255.0.0.0 -a routing1='default 10.0.0.247' clstr_net

Step 2

Create a new interface entry for the NIM "master" definition. Assume that the next available interface index is "2" and the hostname of the NIM master is "xcataixmn". This must be the hostname of the management node interface that is connected to the "clstr_net" network. 

_nim -o change -a if2='clstr_net xcataixmn 0' -a cable_type2=N/A master_

Step 3

Create routing information so that NIM knows how to get from one network to the other. Assume the next available routing index is "2", and the IP address of the NIM master on the "master_net" network is "8.124.37.24". Assume the IP address on the NIM master on the "clstr_net" network is " 10.0.0.241". This command will set the route from "master_net" to "clstr_net" to be " 10.0.0.241" and it will set the route from "clstr_net" to "master_net" to be "8.124.37.24". 

_nim -o change -a routing2='master_net 10.0.0.241 8.124.37.24' clstr_net_

Step 4

Verify the definitions by running the following commands. 

_lsnim -l master_
_lsnim -l master_net_
_lsnim -l clstr_net_

See the NIM documentation for details on creating additional network and route definitions. (IBM AIX Installation Guide and Reference. <http://www-03.ibm.com/servers/aix/library/index.html>)

Define the HMC as an xCAT node

The xCAT hardware control support requires that the hardware control point for the nodes also be defined as a cluster node.

The following command will create an xCAT node definition for an HMC with a host name of "hmc01". The groups, nodetype, mgt, username, and password attributes must be set. 

_**mkdef -t node -o hmc01 groups="hmc,all" nodetype=hmc mgt=hmc username=hscroot password=abc123**_

If xCAT Management Node is in the same service network with HMC, you will be able to discover the HMC and create an xCAT node definition for the HMC automatically. 

_**lsslp -w -s HMC**_

The above xCAT command lsslp discovers and writes the HMCs into xCAT database, but we still need to set HMCs' username and password. 

_**chdef -t node -o hmc01 username=hscroot password=abc123**_

For more details with hardware discovery feature in xCAT, please refer to document:

[XCAT_AIX_Cluster_Overview_and_Mgmt_Node]

[XCAT_System_p_Hardware_Management]

Discover the LPARs managed by the HMC

This step assumes that the partitions are already created using the standard HMC interfaces.

Use the rscan command to gather the LPAR information. This command can be used to display the LPAR information in several formats and can also write the LPAR information directly to the xCAT database. In this example we will use the "-z" option to create a stanza file that contains the information gathered by rscan as well as some default values that could be used for the node definitions.

To write the stanza format output of rscan to a file called "mystanzafile" run the following command. 

_**rscan -z hmc01 &gt; mystanzafile**_

This file can then be checked and modified as needed. For example you may need to add a different name for the node definition or add additional attributes and values.

Note: The stanza file will contain stanzas for things other than the LPARs. This information must also be defined in the xCAT database. It is not necessary to modify the non-LPAR stanzas in any way.

The updated stanza file might look something like the following. 

_Server-9117-MMA-SN10F6F3D:_
_objtype=node_
_nodetype=fsp_
_id=5_
_model=9118-575_
_serial=02013EB_
_hcp=hmc01_
_pprofile=_
_parent=Server-9458-10099201WM_A_
_groups=fsp,all_  
_mgt=hmc_
_node01:_
_objtype=node_
_nodetype=lpar,osi_
_id=9_
_hcp=hmc01_
_pprofile=lpar9_
_parent=Server-9117-MMA-SN10F6F3D_
_groups=lpar,all_
_mgt=hmc_
_node02:_
_objtype=node_
_nodetype=lpar,osi_
_id=7_
_hcp=hmc01_
_pprofile=lpar6_
_parent=Server-9117-MMA-SN10F6F3D_
_groups=lpar,all_
_mgt=hmc_

Note: The rscan command supports an option to automatically create node definitions in the xCAT database. To do this the LPAR name gathered by rscan is used as the node name and the command sets several default values. If you use the "-w" option make sure the LPAR name you defined will be the name you want used as your node name.

Define xCAT cluster nodes

The information gathered by the rscan command can be used to create xCAT node definitions.

Since we have put all the node information in a stanza file we can now pass the contents of the file to the mkdef command to add the definitions to the database. 

_**cat mystanzafile | mkdef -z**_

You can use the xCAT lsdef command to check the definitions (ex. "lsdef -l node01"). After the node has been defined, you can use the chdef command to make any additional updates to the definitions, if needed.

Add IP addresses and hostnames to /etc/hosts

Make sure all node hostnames are added to /etc/hosts. Refer to the section titled "Add cluster nodes to the /etc/hosts file" in the following document for details. (http://xcat.svn.sourceforge.net/viewvc/xcat/xcat-core/trunk/xCAT-client/share/doc/xCAT2onAIX.pdf )

Define xCAT groups (optional)

XCAT supports both static and dynamic node groups. See the section titled "xCAT node group support" in the "xCAT2 Top Doc" document for details on using xCAT groups. ( <http://xcat.svn.sourceforge.net/viewvc/xcat/xcat-core/trunk/xCAT-client/share/doc/xCAT2top.pdf>

Set up customization scripts (optional)

xCAT supports the running of customization scripts on the nodes when they are installed.

This support includes:

  • The running of a set of default customization scripts that are required by xCAT.
    You can see what scripts xCAT will run by default by looking at the "xcatdefaults" entry in the xCAT "postscripts" database table. ( I.e. Run "tabdump postscripts".). You can change the default setting by using the xCAT chtab or tabedit command. The scripts are contained in the /install/postscripts directory on the xCAT management node.
  • The optional running of customization scripts provided by xCAT.
    There is a set of xCAT customization scripts provided in the /install/postscripts directory that can be used to perform optional tasks such as additional adapter configuration.
  • The optional running of user-provided customization scripts.

To have your script run on the nodes:

  1. Put a copy of your script in /install/postscripts on the xCAT management node. (Make sure it is executable.)
  2. Set the "postscripts" attribute of the node definition to include the comma separated list of the scripts that you want to be executed on the nodes. The order of the scripts in the list determines the order in which they will be run. For example, if you want to have your two scripts called "foo" and "bar" run on node "node01" you could use the chdef command as follows.

chdef -t node -o node01 -p postscripts=foo,bar

(The "-p" means to add these to whatever is already set.)

Note: The customization scripts are run during the boot process (out of /etc/inittab).

Add NTP setup script

To have xCAT automatically set up ntp on the cluster nodes you must add the setupntp script to the list of postscripts that are run on the nodes.

To do this you can either modify the "postscripts" attribute for each node individually or you can just modify the definition of a group that all the nodes belong to.

For example, if all your nodes belong to the group "compute" then you could add setupntp to the group definition by running the following command.

chdef -p -t group -o compute postscripts=setupntp

Add secondary adapter configuration script

It is possible to have additional adapter interfaces automatically configured when the nodes are booted. XCAT provides sample configuration scripts for both Ethernet and IB adapters. These scripts can be used as-is or they can be modified to suit you particular environment. The Ethernet sample is /install/postscript/configeth. When you have the configuration script that you want you can add it to the "postscripts" attribute as mentioned above. Make sure your script is in the /install/postscripts directory and that it is executable.

If you wish to configure IB interfaces please refer to: "xCAT 2 InfiniBand Support**" <http://xcat.svn.sourceforge.net/viewvc/xcat/xcat-core/trunk/xCAT-client/share/doc/xCAT2IBsupport.pdf> **

Note: Do not forget that the new adapter interface hostnames must be resolvable on the node. To do this you can use the NIM resolve.conf resource to automatically create a resolv.conf file on the nodes when they are installed.

Configure NIM to use nimsh and SSL.

The NIM service handler (nimsh), is provided as an optional feature of NIM to be used in cluster environments where the standard rsh protocols are not secure enough.

Although nimsh eliminates the need for rsh, in the default configuration it does not provide trusted authentication based on key encryption. To use cryptographic authentication with NIMSH, you can configure NIMSH to use OpenSSL in the NIM environment. When you install OpenSSL on a NIM client, SSL socket connections are established during NIMSH service authentication. Enabling OpenSSL provides SSL key generation and includes all cipher suites supported in SSL version 3.

In order to facilitate the setup of nimsh, xCAT provides a sample customization called "confignimsh" that can be used to configure nimsh on the cluster nodes.

This script will also configure nimsh to use SSL and will remove the /.rhosts file from the node. If you do not wish to have the .rhosts file removed from the node you must remove those lines from the confignimsh script before using it.

This script should only be run on AIX standalone (diskfull) cluster compute nodes. It should NOT be run on the xCAT management node, service nodes or diskless nodes.

The basic processes is:

  • Make sure the AIX openssl fileset gets installed on the management node and all the other cluster nodes. (Which should be done in any case.)
  • On the xCAT management node run the following command.nimconfig -c

You must also run this command on any service nodes that are being used.

  • Add "confignimsh" to the list of scripts you want run on the nodes

For example, if all your nodes belong to the group "compute" then you could add confignimsh to the group definition by running the following command.

chdef -p -t group -o compute postscripts=confignimsh

After the nodes boot up you can verify that nimsh was set up correctly by running a NIM command such as: "nim -o lslpp <nodename>".

To be sure that nimsh is actually using SSL you can run the command:

"nimquery -a host=<nodename>".

Example:

> nimquery -a host=xcatn11

host:xcatn11.cluster.com:addr:10.2.0.104:mask:255.255.0.0:gtwy: 10.2.0.200:_pif:en0:_ssl:yes:_psh:no:_res:no:asyn:no:mac:163D0DDAE202:_sslver:OpenSSL 0.9.8k 25 Mar 2009:

The "_ssl:yes" indicates that nimsh is using SSL.

Note: You could also set up nimsh at any time using the xCAT updatenode command to run the confignimsh script on the nodes.

Create prescripts (optional)

Starting with xCAT 2.5, prescript support is provided to run user-provided scripts during the node initialization process. These scripts can be used to help set up specific environments on the servers that handle the cluster node deployment. The scripts will run on the install server for the nodes. (Either the management node or a service node.) A different set of scripts may be specified for each node if desired.

One or more user-provided prescripts may be specified to be run either at the beginning or the end of node initialization. The node initialization on AIX is done either by the nimnodeset command (for diskfull nodes) or the mkdsklsnode command (for diskless nodes.)

You can specify a script to be run at the beginning of the nimnodeset or mkdsklsnode command by setting the prescripts-begin node attribute.

You can specify a script to be run at the end of the commands using the prescripts-end node attribute.

The format of the entry is:

[action1]:s1,s2...[|action2:s3,s4,s5...]...

where:

_action* _is either "standalone" or "diskless"

_s1,s2.._are the prescripts to run for this action

The attributes may be set using the chdef command.

For example, if you wish to run the foo and bar prescripts at the beginning of the nimnodeset command you would run a command similar to the following.

chdef -t node -o node01 prescripts-begin="standalone:foo,bar"

When you run the nimnodeset command it will start by checking each node definition and will run any scripts that are specified by the _prescripts-begin _attributes.

Similarly, the last thing the command will do is run any scripts that were specified by the _prescripts-end _attributes.

For more information about using the xCAT prescript support refer to the "xCAT2 Top Doc", ( <http://xcat.svn.sourceforge.net/viewvc/xcat/xcat-core/trunk/xCAT-client/share/doc/xCAT2top.pdf>)

Gather MAC information for the install adapters.

Use the xCAT getmacs command to gather adapter information from the nodes. This command will return the MAC information for each Ethernet adapter available on the target node. The command can be used to either display the results or write the information directly to the database. If there are multiple adapters the first one will be written to the database.

The command can also be used to do a ping test on the adapter interfaces to determine which ones could be used to perform the network boot. In this case the first adapter that can be successfully used to ping the server will be written to the database.

Before running getmacs you must first run the makeconservercf command. You need to run makeconservercf any time you add new nodes to the cluster. 

_**makeconservercf**_

To retrieve the MAC address for all the nodes in the group "aixnodes" and write the first adapter MAC to the xCAT database you could issue the following command.

getmacs aixnodes

To display all adapter information but not write anything to the database.

getmacs -d aixnodes

To retrieve the MAC address and do a ping test to determine which adapter MAC to use for the node, or the lpar is just created and there is no active profile for that lpar, or the lpar is on a P5 system and there is no lhea/sea ethernet adapters, you could issue the following command. (The ping operation will reboot the lpar and may take a while to complete.)

getmacs -d aixnodes -S 10.14.0.2 -G 10.14.0.2 -C 10.14.0.4

The output would be similar to the following.

# Type Location Code MAC Address Full Path Name Ping Result Device Type

ent U9125.F2A.024C362-V6-C2-T1 fef9dfb7c602 /vdevice/l-lan@30000002 successful virtual

ent U9125.F2A.024C362-V6-C3-T1 fef9dfb7c603 /vdevice/l-lan@30000003 unsuccessful virtual

From this result you can see that " fef9dfb7c602" should be used for this nodes MAC address.

For more information on using the getmacs command see the man page.

Create NIM client & group definitions

You can use the xCAT xcat2nim command to automatically create NIM machine and group definitions based on the information contained in the xCAT database. By doing this you synchronize the NIM and xCAT names so that you can use the same target names when running either an xCAT or NIM command.

To create NIM machine definitions you could run the following command.

xcat2nim -t node aixnodes

To create a NIM group definition called "aixgrp" you could run the following command.

xcat2nim –t group –o aixgrp

To check the NIM definitions you could use the NIM lsnim command or the xCAT xcat2nim command. For example, the following command will display the NIM definitions of the nodes: node01, node02, and node03 (from data stored in the NIM database).

xcat2nim -t node -l –o node01-node03

Initialize the AIX/NIM nodes

You can use the xCAT nimnodeset command to initialize the AIX standalone nodes. This command uses information from the xCAT osimage definition and default values to run the appropriate NIM commands.

For example, to set up all the nodes in the group "aixnodes" to install using the osimage named "610image" you could issue the following command.

nimnodeset -i 610image aixnodes

To verify that you have allocated all the NIM resources that you need you can run the "lsnim –l" command. For example, to check node "node01" you could run the following command.

lsnim -l node01

The nimnodeset command will also set the "profile" attribute in the xCAT node definitions to "610image ". Once this attribute is set you can run the nimnodeset command without the "-i" option.

DHCP configuration(Optional)

For AIX clusters, there is a bootp service daemon on the xCAT MN working with AIX node installations. If the xCAT administrator wants to use the DHCP service instead of bootp for xCAT node cluster node installations, they will need to disable the bootp service. They will need to enable the dhcpsd in rc.tcpip to start dhcp service during system bootup. The AIX NIM installation will enable bootp automatically, so the administrator may need to stop the bootp daemon, to properly enable dhcp server.

The followings are the steps used by the xCAT administrator to disable the bootp and enable dhcpsd daemon for NIM node installation. The xCAT administrator can toggle between bootp and dhcpsd if they want to use bootp as part of the AIX node installation.

You just need to be aware that dhcp server daemon for node installation will not be active whenever the bootp daemon is active

Disable the bootp service(Optional)

a) Stop bootp from rebootting by commenting bootps line in /etc/inetd.conf file:

#bootps dgram udp wait root /usr/sbin/bootpd bootpd /etc/bootptab

b) Stop and restart the inetd subsystem:

refresh -s inetd

c) Stop bootp deamon:

ps -ef | grep bootp

kill the bootp process

d) Start dhcpsd server from rebootting by uncomment dhcpsd lines in /etc/rc.tcpip

# Start up the DHCP Server

start /usr/sbin/dhcpsd "$src_running"

e) Stop and restart the tcpip group

stopsrc -g tcpip

startsrc -g tcpip

After stopped bootp daemon and start DHCP daemon, you need to write the IP and MAC mapping into DHCP configuration file and restart DHCP daemon.

To configure DHCP server run:

makedhcp node01

Open a remote console (optional)

You can open a remote console to monitor the boot progress using the xCAT rcons command. This command requires that you have conserver installed and configured.

If you wish to monitor a network installation you must run rcons before initiating a network boot.

To configure conserver run:

_makeconservercf _

To start a console:

rcons node01

Note: You must always run makeconservercf after you define new cluster nodes.

Initiate a network boot

Initiate a remote network boot request using the xCAT rnetboot command. For example, to initiate a network boot of all nodes in the group "aixnodes" you could issue the following command.

rnetboot aixnodes

Note: If you receive timeout errors from the rnetboot command, you may need to increase the default 60-second timeout to a larger value by setting ppctimeout in the site table:

chdef -t site -o clustersite ppctimeout=180

Verify the deployment

  • You can use the AIX lsnim command to see the state of the NIM installation for a particular node, by running the following command on the NIM master:
    lsnim -l <clientname>
  • Retry and troubleshooting tips:
  • For p6 lpars, it may be helpful to bring up the HMC web interface in a browser and watch the lpar status and reference codes as the node boots.
  • Verify network connections
  • If the rnetboot returns "unsuccessful" for a node, verify that bootp/dhcp and tftp is configured and running properly.
  • For bootp, view /etc/bootptab to make sure an entry exists for the node.
    For dhcp, view /etc/dhcpsd.cnf to make sure an entry exists for the node.
  • Verify that the information in /tftpboot/<node>.info is correct.
  • Stop and restart inetd:

stopsrc -s inetd

startsrc -s inetd

  • Stop and restart tftp:

stopsrc -s tftp

startsrc -s tftp

    • Verify NFS is running properly and mounts can be performed with this NFS server:
  • View /etc/exports for correct mount information.
  • Run the showmount and exportfs commands.
  • Stop and restart the NFS and related daemons:

stopsrc -g nfs

startsrc -g nfs

  • Attempt to mount a filesystem from another system on the network.

Cleanup

The NIM definitions and resources that are created by xCAT commands are not automatically removed. It is therefore up to the system administrator to do some clean up of unused NIM definitions and resources from time to time. (The NIM lpp_source and SPOT resources are quite large.) There are xCAT commands that can be used to assist in this process.

Removing NIM machine definitions

Use the xCAT xcat2nim command to remove all NIM machine definitions that were created for the specified xCAT nodes. This command will not remove the xCAT node definitions.

For example, to remove the NIM machine definition corresponding to the xCAT node named "node01" you could run the command as follows.

xcat2nim -t node -r node01

The xcat2nim command is intended to make it easier to clean up NIM machine definitions that were created by xCAT. You can also use the AIX nim command directly. See the AIX/NIM documentation for details.

Removing NIM resources

Use the xCAT rmnimimage command to remove all the NIM resources associated with a given xCAT osimage definition. The command will only remove a NIM resource if it is not allocated to a node. You should always clean up the NIM node definitions before attempting to remove the NIM resources. The command will also remove the xCAT osimage definition that is specified on the command line.

For example, to remove the "610image" osimage definition along with all the associated NIM resources run the following command.

rmnimimage -x 610image

If necessary, you can also remove the NIM definitions directly by using NIM commands. See the AIX/NIM documentation for details.

Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.