GPFS_statelite_nodes

Note: this is an xCAT design document, not an xCAT user document. If you are an xCAT user, you are welcome to glean information from this design, but be aware that it may not have complete or up to date procedures.

• Overview
• Other Design Considerations

Overview

Many xcat users have wanted to have nodes boot from an os image that is in gpfs, similar to nfs-root. This would be beneficial because gpfs is faster and more reliable than nfs, and this approach would take up less memory than ramdisk. Unfortunately, it would be a huge amount of work for the gpfs team to modify gpfs to run this way on the node. But something xcat could do to support a scenario close to this is the following: Provide a liteimg-like cmd that would intelligently divide up the files in the osimage (that was generated by genimage), putting the absolutely needed files in a ramdisk and putting the rest in gfps. When the node boots, it would load the small ramdisk 1st. This would start the network and gpfs, and then mount all of the other dirs of the os image that were contained in gpfs. Note that unlike nfs-based statelite, the ramdisk keeps running, we wouldn't switch root to the gpfs file system.

The liteimg-like cmd (maybe called gpfsimg) would need to:

• know where in gpfs the "rootimg" should go. This could be stored in the osimage.rootimg attribute.
• know what dirs/files needed to stay in the ramdisk. Maybe this could be in the litefile table. We should ship an example list.
• add a script to the ramdisk that would start gpfs, mount the file system, and create the bind mounts from the ramdisk to a .statelite dir, like we do for statelite
• the os image files that are in gfps will be mounted readonly (because they will be shared by many nodes), but we should also support statelite persistent files (specified in the litefile and statelite tables). In this case, statelite persistent files would be much more practical because the parallel performance of gpfs is much better than nfs running on the service nodes.

Many customers do something like this manually, loading the nodes with a ramdisk, but putting the applications and some of the big libraries in gpfs. This design is a way to automate that and optimize it (by putting more dirs/files in gpfs).

Another possible enhancement for statelite is to leverage the overlay file system, xCAT had tried the aufs in very early xCAT 2.x versions, but at that time the aufs is not in Linux kernel and it turned out that the aufs is not stable enough, so we designed the current statelite mechanism. We could investigate if the overlay filesystem is now part of Linux kernel, if yes, this could be a good enhancement for our current statelite design.

Other Design Considerations

• Required reviewers:
• Required approvers: Bruce Potter
• Database schema changes: N/A
• Affect on other components: N/A
• External interface changes, documentation, and usability issues: N/A
• Packaging, installation, dependencies: N/A
• Portability and platforms (HW/SW) supported: N/A
• Performance and scaling considerations: N/A
• Migration and coexistence: N/A
• Serviceability: N/A
• Security: N/A
• NLS and accessibility: N/A
• Invention protection: N/A