For a self-paced introduction, the Lustre 101 web-based course series developed by the Oak Ridge Leadership Computing Facility at Oak Ridge National Laboratory is a great place to start. These courses are targeted at experienced system administrators who are relatively new to Lustre.
The Lustre Manual is the most comprehensive source of information on how to set up, manage and test a Lustre file system. The Lustre Manual and other information can be found by going to the Documentation page.
Intel has created an Architecting a High Performance Storage System white-paper that describes a systematic approach to the design of a Lustre storage system in detail.
Components of a Lustre File System
Lustre’s scalable architecture has three main components:
- Metadata Server (MDS)
- Object Storage Server (OSS)
- Lustre clients
The Metadata Servers (MDS) provide metadata services for a file system and manages a Metadata Target (MDT) that stores the file metadata. The Object Storage Servers (OSS) manage the Object Storage Targets (OST) that store the file data objects.
General Steps for Installing Lustre
- Set Up Lustre Filesystem Hardware
Lustre runs on most commodity hardware with any kind of block storage device including single disks, software and hardware RAID and logical volume manager. 64-bit architectures are recommended for servers. Lustre supports Ethernet and InfiniBand networks.
Lustre allows for multiple MDSes for high availability. The size of the MDT’s backing file system should be chosen based on the total number of files planned to be stored in the Lustre file system, and the aggregate OST space should be chosen based on the total amount of data planned to be stored in the file system. Estimating space requirements early can dictate hardware requirements.
- Install Lustre Software
Lustre runs on a variety of Linux kernels from Linux distributions including RHEL, CentOS, and SLES. When using the Lustre ldiskfs OSD only, it will be necessary to patch the kernel before building Lustre. The Lustre ZFS OSD and the Lustre client code do not require patches to the kernel.
Download the required Lustre RPMs or source from the Download page and any necessary third-party packages like e2fsprogs. Metadata and Object Storage Server require the Lustre patched Linux kernel, Lustre modules, Lustre utilities and e2fsprogs installed. The clients require the Lustre client modules, client utilities and, optionally, the Lustre patched kernel.
Configuring Lustre
The Lustre Networking (LNET) kernel modules have a variety of module parameters that can be set in the /etc/modprobe.d/lustre.conf file. The type of network used and globally-available networks can be specified along with routes in a Lustre configuration.
Lustre provides a variety of configuration utilities to set up and tune the file system. These utilities include mkfs.lustre to format a disk for a Lustre service, tunefs.lustre to modify configuration information on a Lustre target disk, lctl to directly control Lustre via an ioctl interface and mount.lustre to start a Lustre client or target service.
Other configuration considerations include the degree of high availability and failover to build into the file system.
Testing Lustre
The Lustre source code contains a set of tests and testing framework that test individual Lustre components to overall benchmarking of the filesystem and network. Tests can set up and run a full battery of tests on a single node and run tests on a large cluster.
Administering and Managing Lustre
The starting point for administering and managing Lustre is to monitor all logs and console logs for system health.
Getting Help
One of the best places to start is to review the Lustre Operations Manual and the Lustre Wiki.
Review and join the Lustre community mailing lists.
Finally, if you are looking for paid, commercial support, review the OpenSFS Lustre Provider Capability Matrix for support options.
