Ubuntu_Quick_Start

• xCAT on Ubuntu Server
• Preparing the Management Node
  • Install the Operating System
  • Configure the network interface
  • Configure domain name resolution
  • Configure the hosts file
• Installing xCAT
  • Obtaining xCAT Software
    • Option 1: Configure the xCAT Software Internet-Hosted Repository
    • Option 2: Download the xCAT Software
  • Configure the xCAT apt-key
  • Configure Ubuntu Package repositories
  • Install xCAT
  • Verify xCAT installation
  • Update xCAT
• Configure xCAT
  • networks Table
  • passwd Table
  • DHCP
• Deploying Nodes
  • Create the Operating System Repository
  • Creating node definitions in xCAT
    • Adding nodes
    • Configure DNS
    • Configure DHCP
  • Installing Stateful/Diskful Nodes
    • Begin Installation
  • Installing Stateless/Diskless Nodes
    • Prepare Images
    • Begin Installation
    • Update Images
  • Monitor installation
• Advanced Topics
  • Customizing System Packages
  • Customizing additional packages
  • Installing other packages with Ubuntu official mirrors
• Known Issues
  • P Servers
  • X Servers

xCAT on Ubuntu Server

This documentation walk you through a very basic install of xCAT.

The following example will configure a single xCAT management node with two compute nodes.

• mn01 is the xCAT management node
• cn01 is a compute node, installed stateful/diskful
• cn02 is a compute node, installed stateless/diskless

The Operating System used is Ubuntu Server 14.04 LTS (Trusty Tahr).

Preparing the Management Node

Pre-install Considerations:
xCAT utilizes the /install directory on the management node to store various configuration files, OS images, post install scripts, etc. It is recommended to create a separate filesystem, or partition, at least 30GB large for the /install directory.

 

Install the Operating System

Install Ubuntu server on the management node (mn01).

It is recommended that the management node have connectivity to the internet in order to access various external repositories for package management. (Refer to Ubuntu's Repositories/CommandLine page for more information).

 

Configure the network interface

In this document, the networking scheme is kept simple and will only consist of a single active network interface. This interface will be used by xCAT as the provisioning network. Verify that provisioning interface in /etc/network/interfaces is configured with a static IP address.

auto eth0
iface eth0 inet static
    address 192.168.1.1
    netmask 255.255.255.0

 

Configure domain name resolution

The management node is the DNS server for the xCAT managed cluster. Configure the /etc/resolv.conf file to use the IP address of the management node for the nameserver attribute.

search cluster.com
nameserver 192.168.1.1

 

Configure the hosts file

The management node performs hostname to IP address resolution. Add IP address and hostname entries to the /etc/hosts file for all the machines in your xCAT managed cluster.

127.0.0.1       localhost.localdomain localhost

192.168.1.1     mn01.cluster.com mn01

192.168.1.10    cn01.cluster.com cn01
192.168.1.11    cn02.cluster.com cn02

 

Installing xCAT

Obtaining xCAT Software

Option 1: Configure the xCAT Software Internet-Hosted Repository

xCAT provides internet hosted software repositories for xcat-core and xcat-dep (dependencies) software packages.

Create the xcat-core.list and xcat-deps.list files in /etc/apt/sources.list.d/ on your management node in the following format:

xcat-core.list:
    deb [arch=<arch>] http://sourceforge.net/projects/xcat/files/ubuntu/<xcat-version>/core-snap <ubuntu-release-name> main

xcat-dep.list:
    deb [arch=<arch>] http://sourceforge.net/projects/xcat/files/ubuntu/xcat-dep <ubuntu-release-name> main

where: 
    <arch> is "amd64" or "ppc64el"
    <xcat-version> is the xCAT numerical major release version or "devel" for development builds.
    <ubuntu-release-name> is the ubuntu release name  (14.04 LTS would be "trusty"). 

 
For Example, Ubuntu Server 14.04 LTS:

Add the following line into /etc/apt/sources.list.d/xcat-core.list:

deb [arch=amd64] http://sourceforge.net/projects/xcat/files/ubuntu/devel/core-snap trusty main

 
Add the following line into /etc/apt/sources.list.d/xcat-dep.list:

deb [arch=amd64] http://sourceforge.net/projects/xcat/files/ubuntu/xcat-dep trusty main

 

Option 2: Download the xCAT Software

Download the xCAT core (Linux - Deb Package) and xCAT Dependency package from the xCAT Download page and copy the tarballs to the management node.

mkdir -p /root/xcat2
cd /root/xcat2

# ls -1 
xcat-core-2.8.5.tar.bz2
xcat-dep-ubuntu.tar.bz

tar jxvf xcat-core-*.tar.bz2     # or core-debs-snap.tar.bz2 (devel or snapshot build)
tar jxvf xcat-dep-ubuntu*.tar.bz

 
Run the mklocalrepo.sh script in EACH of the xcat-core and xcat-dep directories to automatically add configuration files to /etc/apd/sources/list.d.

cd /root/xcat2/xcat-core
./mklocalrepo.sh

cd /root/xcat2/xcat-dep
./mklocalrepo.sh

 

Configure the xCAT apt-key

Regardless of the option chosen above, apt-get is used to easily install xCAT. The xCAT GPG public key must be added for apt to verify the xCAT packages.

Without the key, running 'apt-get update' will display the following error message: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 20E475A8DA736C68

wget -O - "http://sourceforge.net/projects/xcat/files/ubuntu/apt.key/download" | apt-key add -

 

Configure Ubuntu Package repositories

The xCAT software has dependencies on various Ubuntu operating system packages and using apt-get will automatically pull in those dependencies. The main and universe repositories must be configured for this to work well. The following commands will add the necessary apt-repositories to the management node:

# install the add-apt-repository command
apt-get install software-properties-common

# For x86_64:
add-apt-repository "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc) main"
add-apt-repository "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-updates main"
add-apt-repository "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc) universe"
add-apt-repository "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-updates universe"

# For ppc64el:
add-apt-repository "deb http://ports.ubuntu.com/ubuntu-ports $(lsb_release -sc) main"
add-apt-repository "deb http://ports.ubuntu.com/ubuntu-ports $(lsb_release -sc)-updates main"
add-apt-repository "deb http://ports.ubuntu.com/ubuntu-ports $(lsb_release -sc) universe"
add-apt-repository "deb http://ports.ubuntu.com/ubuntu-ports $(lsb_release -sc)-updates universe"

 

Install xCAT

Use the following commands to install xCAT.

apt-get clean all
apt-get update
apt-get install xcat

 

Verify xCAT installation

At this point, xCAT should have successfully installed. Verify the installation by running a few simple commands.

#
# Add xCAT commands into your path 
#
source /etc/profile.d/xcat.sh

#
# display the installed version of xcat 
#
lsxcatd -a 

#
# view the site table contents
#
tabdump site
#key,value,comments,disable
"blademaxp","64",,
"domain","ppd.pok.ibm.com",,
...
...

 

Update xCAT

To update the xCAT software, point the xcat-core and xcat-dep repositories to a later version, then run the following commands:

apt-get update
apt-get install xcat 

 

Configure xCAT

 

networks Table

During xCAT installation, the 'makenetworks' command runs and creates entries into the xCAT networks table for each network that is detected on the management node.

If there are additional networks that need to be configured, reference the following document: Appendix_A:_Network_Table_Setup_Example

 

passwd Table

The xCAT passwd table needs to contain an entry for the "system" key. The system key specifies the password that will be used for the root userid on and nodes that is installed by xCAT. You will also need to add the userid/password for the hardware control being used. (i.e "hmc" for Power Servers; "ipmi" for X Servers)

tabedit passwd
key,username,password,cryptmethod,comments,disable
"system","root","cluster",,,
"ipmi","USERID","PASSW0RD",,,

 

DHCP

Run makedhcp -n to create a new DHCP configuration file and add the network definitions.

makedhcp -n

 

Deploying Nodes

Create the Operating System Repository

Download the ISO image for the operating system onto your management node and run copycds.

Copycds will copy the contents of the iso image to /install/<os>/<arch> on the management node.
In addition, default osimages will be added to the osimage table. </arch></os>

#
# run copycds on the image, located in /tmp
#
/opt/xcat/bin/copycds /tmp/ubuntu-14.04.1-server-amd64.iso

#
# list out the osimages created
#
lsdef -t osimage 
ubuntu14.04.1-x86_64-install-compute  (osimage)
ubuntu14.04.1-x86_64-install-hpc  (osimage)
ubuntu14.04.1-x86_64-install-kvm  (osimage)
ubuntu14.04.1-x86_64-netboot-compute  (osimage)
ubuntu14.04.1-x86_64-statelite-compute  (osimage)

Repeat the above for each operating system you want to provision using xCAT.

 

Creating node definitions in xCAT

 

Adding nodes

Use the mkdef command to create the initial node definition for the two compute nodes.

#
# See 'noderange' man page for various ways to specify a large set of nodes
# i.e cn01-c99, cn[001-100], etc
#
mkdef -t node -o cn01,cn02 groups=compute,all

 
To view the nodes defined:

#
# Either of the following commands can be used to display defined nodes    
#
nodels
lsdef -t node 

 
To view specific information about the node object:

lsdef -t node -o cn01

 

Configure DNS

Ensure that ROOTDIR is not set in /etc/bind/named.conf

Configure the forwarders attribute in the xCAT site table to your site wide DNS servers.
The management node will forward any request that it cannot answer to these servers.

chdef -t site forwarders=1.2.3.4,1.2.3.5

 
Run the makedns command to set up domain name services on the management node.

makedns -n

 

Configure DHCP

xCAT can use the physical MAC address on the network interface devices to assign IP addresses. The MAC address can be located either from the back panel of the servers, or from the physical network adapter.

Locate the MAC addresses and add entries to the compute node definitions.

chdef -t node -o cn01 mac="xx:xx:xx:xx:xx:xx"
chdef -t node -o cn02 mac="yy:yy:yy:yy:yy:yy"

#
# set primarynic and installnic to use MAC address
#
chdef -t node -o cn01,cn02 primarynic=mac installnic=mac

#
# Run makedhcp to add the nodes to dhcp 
#
makedhcp cn01,cn02

 

Installing Stateful/Diskful Nodes

A stateful, or diskful, node is a node where the operating system is installed onto the physical disk (hard drive). The state of the operating system is saved onto the disks and will persist on subsequent reboots.

A few default osimages are created when the copycds command is run and the names are in the form:

#
# xCAT created stateful/diskful osimages have <provmethod>=install
#
<releasename>-<arch>-<provmethod>-<profile>

 

Begin Installation

To install a stateful/diskful install of Ubuntu Server 14.04.1, we will use osimage=ubuntu14.04.1-x86_64-install-compute.

Use the rsetboot command to force the compute node to boot from network ("net") on the next reboot.

rsetboot cn01 net

 
Use the nodeset command to set the next boot state for the node. This tells xCAT what do to on the next boot of the node.

nodeset cn01 osimage=ubuntu14.04.1-x86_64-install-compute

 
Reboot the node to start the install

rpower cn01 boot

 

Installing Stateless/Diskless Nodes

A stateless, or diskless, node is a node where the operating system is installed into memory. The state of the machine is held in memory (RAM) and will not persist on subsequent reboots of the node. The state will return to what has been set in the master image.

A few default osimages are created when the copycds command is run and the names are in the form:

#
# xCAT created stateless/diskless osimages have <provmethod>=netboot
#
<releasename>-<arch>-<provmethod>-<profile>

 

Prepare Images

To install a stateless/diskless install of Ubuntu Server 14.04.1, we will use osimage=ubuntu14.04.1-x86_64-netboot-compute.

Take a look at the default osimage created for netboot-compute:

# lsdef -t osimage -o ubuntu14.04.1-x86_64-netboot-compute
Object name: ubuntu14.04.1-x86_64-netboot-compute
    exlist=/opt/xcat/share/xcat/netboot/ubuntu/compute.exlist
    imagetype=linux
    osarch=x86_64
    osname=Linux
    osvers=ubuntu14.04.1
    otherpkgdir=/install/post/otherpkgs/ubuntu14.04.1/x86_64
    pkgdir=/install/ubuntu14.04.1/x86_64
    pkglist=/opt/xcat/share/xcat/netboot/ubuntu/compute.ubuntu14.04.pkglist
    profile=compute
    provmethod=netboot
    rootimgdir=/install/netboot/ubuntu14.04.1/x86_64/compute

 
The rootimgdir attribute points to the location on the management node where the stateless image will reside. The first time you look at this location, it should not exist.

ls /install/netboot/ubuntu14.04.1/x86_64/compute
ls: cannot access /install/netboot/ubuntu14.04.1/x86_64/compute: No such file or directory

 
Run genimage to generate a stateless image

genimage ubuntu14.04.1-x86_64-netboot-compute

 
At this point, you have the opportunity to change any files in the image by modifying the files under the rootimgdir. It is recommended that any modifications done to your image is via postscripts. This allows the changes to be automated and repeatable. See Generate/Packing Image for more information about customizing the images.

If no changes are required, continue to the next step.

Run packimgage to pack the stateless image and create the ramdisk.

packimage ubuntu14.04.1-x86_64-netboot-compute

 

Begin Installation

Use the rsetboot command to force the compute node to boot from network ("net") on the next reboot.

rsetboot cn02 net

 
Use the nodeset command to set the next boot state for the node. This tells xCAT what do to on the next boot of the node.

nodeset cn02 osimage=ubuntu14.04.1-x86_64-netboot-compute

 
Reboot the node to start the install

rpower cn02 boot

 

Update Images

If you need to make changes and update your stateless images at any time:

• Make the necessary changes to the object definitions and postscripts
• Run genimage to re-generate the image
• Run packimage to re-pack the image and create the ramdisk
• Run nodeset to tell xCAT to reinstall on next boot
• Reboot the stateless node to pick up the new changes

 

Monitor installation

Use the rcons command to monitor the console and watch the install process.

rcons cn01
rcons cn02

 

Advanced Topics

Customizing System Packages

Refer to Add additional Software (Linux Only) for information on installing additional OS system packages using the pkglist files.

Customizing additional packages

xCAT provides an otherpkgs mechanism that allows you to install additional debian packages (*.deb) that are not provided by the distribution. To use this, you need to create and maintain a local debian package repository.

Looking at the default osimage created for netboot-compute:

# lsdef -t osimage -o ubuntu14.04.1-x86_64-netboot-compute
Object name: ubuntu14.04.1-x86_64-netboot-compute
    exlist=/opt/xcat/share/xcat/netboot/ubuntu/compute.exlist
    imagetype=linux
    osarch=x86_64
    osname=Linux
    osvers=ubuntu14.04.1
    otherpkgdir=/install/post/otherpkgs/ubuntu14.04.1/x86_64
    pkgdir=/install/ubuntu14.04.1/x86_64
    pkglist=/opt/xcat/share/xcat/netboot/ubuntu/compute.ubuntu14.04.pkglist
    profile=compute
    provmethod=netboot
    rootimgdir=/install/netboot/ubuntu14.04.1/x86_64/compute

 
The otherpkgdir specifies a directory that we can use as a base directory for the package repository.
Create a directory under here to save all the packages in your local repository

mkdir -p /install/post/otherpkgs/ubuntu14.04.1/x86_64/<my_custom_dir>

 
Install dpkg-dev on to the management node

apt-get install dpkg-dev

 
Download the debian packages and their dependency packages

cd /install/post/otherpkgs/ubuntu14.04.1/x86_64/<my_custom_dir>

#
# find the dependencies for the package <pkg_name>
#
apt-rdepends <pkg_name> | grep -v Depends

#
# download the packages
#
apt-get download -d <pkg_name>

 
Run dpkg-scanpackages to create the repository for apt

cd /install/post/otherpkgs/ubuntu14.04.1/x86_64/<my_custom_dir>

# 
# do not modify the following command
# apt requires the name 'Packages' to be used
#
dpkg-scanpackages . > Packages

 
Create an otherpkgs package list file and add the <pkg_name> to the list </pkg_name>

vi /install/custom/install/ubuntu14.04.1/compute.otherpkgs.pkglist


<my_custom_dir>/xxx.deb
...
...

 
Assign the pkglist file to the osimage object definition

chdef -t osimage  ubuntu14.04.1-x86_64-netboot-compute \
 otherpkgdir="/install/post/otherpkgs/ubuntu14.04.1/x86_64/<my_custom_dir>" \
 otherpkglist="/install/custom/install/ubuntu14.04.1/compute.otherpkgs.pkglist"

 

Installing other packages with Ubuntu official mirrors

See the following for for Installing other packages with Ubuntu official mirror

 

Known Issues

P Servers

Running Ubuntu Server 14.04 (ppc64el) VMs, rcons does not work:

To get around this, open Kimchi at URL https://<powerkvm_ip>:8001
Under "Actions", click "connect" and input the password you set before running mkvm. </powerkvm_ip>

Then you should be able to get the console.

X Servers