Portfolio · UC Riverside HPCC · 2017 to 2019

UCR HPCC work, 2017-2019

Lead HPC administrator, High Performance Computing Center, UC Riverside.

The work preserved here: the HPCC AWS Cluster (HPCC Cloud Service), which let a lab launch its own private HPC cluster in AWS from the HPCC cluster with hpcc_cloud, a cfnCluster wrapper; 57 environment modules for the cluster's software stack; and the files from the Introduction to Python workshop. Every file is my own, and each 2017-2019 commit keeps its original date and message.

View on GitHub Tour the cloud service

HPCC cluster - from the operation manual, 2018

$ hpcc_cloud create new-cluster
Beginning cluster creation for cluster: new-cluster
Creating stack named: cfncluster-new-cluster
Status: cfncluster-new-cluster - CREATE_COMPLETE
Output:"MasterPublicIP"="<MasterPublicIP>"
Output:"MasterPrivateIP"="<MasterPrivateIP>"
Output:"GangliaPublicURL"="http://<MasterPublicIP>/ganglia/"
Output:"GangliaPrivateURL"="http://<MasterPrivateIP>/ganglia/"

Addresses replaced with placeholders.

143original commits, 2017-2019

57environment-module folders

66versioned modulefiles

6AWS cluster manual pages

19workshop sessions taught

01 / What's here

Five folders, one job: keep researchers computing

The repository is a portfolio of the files I wrote for UCR HPCC. Where others later edited a file, it keeps my last version before those edits.

HPCC AWS Cluster manual

The 2018 user manual for the HPCC Cloud Service: intro, AWS account setup, the data egress waiver, cluster setup and operation, and billing and budget alerts.

aws-cluster-manual/

`hpcc_cloud` on cfnCluster

Labs created, checked, listed and deleted their own clusters with one command each. The cfncluster module (1.3, then 1.4) loaded the tool from a Python virtual environment.

environment-modules/cfncluster/

Environment modules

57 module folders for the HPCC software stack, 2017-2019: CUDA and cuDNN, TensorFlow GPU, BUSCO, GATK, SPAdes, Trinity, RAxML-NG, Gaussian, Rosetta and more.

environment-modules/

AWS cluster modules

The start of a separate module repository for software on the AWS clusters HPCC supported (2018).

aws-cluster-modules/

Introduction to Python

18 example files for the HPCC Introduction to Python workshop (Oct 2018), from variable types through submitting a Python script to Slurm.

python-intro-workshop/

Slurm example

Quantum ESPRESSO submission scripts for the OpenMPI and Intel builds, with an example input file (2019).

slurm-examples/

02 / HPCC Cloud Service

The HPCC AWS Cluster

A lab-private HPC cluster in AWS, created on demand from the HPCC cluster. The manual's pitch, in its own terms:

Build a private cluster in 10 minutes.
Any number of nodes, auto scaling up and down; limit 20 to start.
Any type of compute node: high memory, high CPU single or multi node, GPUs, or the HPCC default configuration.
Keep it only as long as you need it; delete when done.
Familiar interface and job scheduler (Slurm: sbatch, squeue).
HPCC installs the software you need if it is not already there.
Build as many clusters as you need, even at the same time.
Pay only for the time you use: per-minute billing.

How it fit together

Lab AWS account

Master account + IAM users

Access key, EC2 key pair, budget alert. Covered by the UC-wide AWS agreement.

HPCC cluster

hpcc_cloud

configure, create, status, list, delete; wraps cfnCluster, loaded by the cfncluster module.

AWS, us-west-1

Stack cfncluster-<name>

Master node plus auto-scaling compute nodes, Slurm scheduler, Ganglia monitoring URL.

Researcher

ssh / scp as centos

Upload inputs, sbatch, squeue, download results, delete the cluster.

Walkthrough, from the manual

1
Create the AWS accounts

From the lab's master AWS account, open IAM and add a user for cluster work (programmatic access is all a lab member needs), assign it to a group, and download the credentials CSV with the access key ID and secret key. Then, in the EC2 console for N. California, create a key pair named for the user and save the key file. Send both files to the user privately.

Account creation page
2
Attach the account to the UC agreement

A UC-wide AWS agreement provides a data egress fee waiver of up to 15% of total monthly AWS fees. It applies to UC accounts except commercial web hosting, media streaming and MOOCs, and requires 80%+ of egress over an approved research network such as CENIC or Internet2, which normal UC usage meets. The manual walks through connecting the account to the agreement and a purchase order.

Egress waiver page
3
Configure hpcc_cloud on the HPCC cluster

Log in to the HPCC cluster and run the configure step. It prompts for the cluster template (default), the AWS access key and secret, region us-west-1, VPC name (default public), and then offers lists to choose the key name, VPC ID, master subnet and IAM user.
```
hpcc_cloud configure
```
Setup page
4
Create, check and list clusters

Creation builds a CloudFormation stack and prints the master node's public IP, used for every ssh and scp afterwards.
```
hpcc_cloud create <NameForYourCluster>
hpcc_cloud status <NameForYourCluster>
hpcc_cloud list
```

Move data in and run a job

scp -i /path/to/your/key-file.pem <local-files-to-copy> centos@<MasterPublicIP>:.
ssh -i /path/to/your/key-file.pem centos@<MasterPublicIP>
sbatch slurm-submission-script.sh
squeue

6
Download results and delete the cluster

Results are pulled back from the HPCC cluster; deleting the cluster stops the node charges.
```
scp -i /path/to/your/key-file.pem centos@<MasterPublicIP>:./<results-to-download> .
hpcc_cloud delete new-cluster
```
Cluster operation page
7
Keep costs in view

Node time and storage are the two biggest costs, so the manual advises moving only the data you need and deleting clusters when idle. Its worked example: about $0.43 per hour for the default configuration (one head node and one 8-core compute node with 16 GB RAM, no spot pricing, 25 GB of data). Billing is monthly to a purchase order, and a budget alert set at $50 a month warns when projected spend would exceed it.

Cost control and billing page

03 / Screenshots

From the 2018 training account

Original AWS console screenshots from the manual.

AWS EC2 dashboard for the N. California region showing running instances, volumes, key pairs and security groups. — Account setup The EC2 console, where the manual sends you to Key Pairs.

AWS EC2 Key Pairs page listing four key pairs with a Create Key Pair button. — Account setup Key pair management: one key per cluster user.

AWS Billing and Cost Management dashboard for March 2018 with month-to-date spend charts. — Cost control The Billing & Cost Management dashboard, bookmarked as the main billing view.

AWS Budgets landing page with a Create budget button and getting-started panels. — Cost control AWS Budgets, where the $50-a-month alert is created.

04 / Environment modules

57 packages on the HPCC software stack

Tcl modulefiles written for the HPCC cluster, 2017-2019: 57 package folders holding 66 versioned modulefiles (casm and gmt hold build notes only). Larger tags have more versions; every tag links to its folder.

Dependencies load with the package

tensorflow-gpu/1.0 runs module load cuda/8.0 and module load cuDNN/5.1, then activates its Python environment, so one command gives a working GPU stack.

Several builds side by side

RevBayes ships as 1.0.11, 1.0.11-mpi and 1.0.11-container (Singularity); Quantum ESPRESSO as 6.3 and 6.3_intel; .version files set the defaults.

Tools for admins too

hpcc_utils puts a shared bin of HPCC admin tools on the path, and cfncluster activates the virtual environment behind the AWS cluster service.

05 / Training

HPCC Introduction to Python workshop

Example scripts taught in October 2018, one file per idea, ending with a Python script that carries its own Slurm directives and is submitted with sbatch.

python-slurm.py (header)

#!/usr/bin/env python

#SBATCH --ntasks=6
#SBATCH --time=02:00:00
#SBATCH --mem=100g
#SBATCH --output=output-simple-%j.out
#SBATCH -p short,batch,intel

import os
import random
import json

slurm-examples/espresso/espresso.sh

#!/bin/bash -l
#SBATCH --nodes=1
#SBATCH --ntasks=6
#SBATCH --mem-per-cpu=1G

export OMP_NUM_THREADS=1

module load espresso/6.3

mpirun pw.x -in input_file.in >& output_file.out

Training archive

The Python workshop was one of 19 HPCC workshop sessions I taught between 2017 and 2019. The slide decks are collected in the training archive.

Open the training archive

06 / Timeline

143 commits, June 2017 to July 2019

Commits per month from the repository history (43 in 2017, 69 in 2018, 31 in 2019).

12J2017Jun 2017: 12 commits
8JJul 2017: 8 commits
7AAug 2017: 7 commits
10SSep 2017: 10 commits
4OOct 2017: 4 commits
NNov 2017: 0 commits
2DDec 2017: 2 commits
J2018Jan 2018: 0 commits
2FFeb 2018: 2 commits
19MMar 2018: 19 commits
2AApr 2018: 2 commits
3MMay 2018: 3 commits
5JJun 2018: 5 commits
4JJul 2018: 4 commits
10AAug 2018: 10 commits
1SSep 2018: 1 commits
19OOct 2018: 19 commits
1NNov 2018: 1 commits
3DDec 2018: 3 commits
7J2019Jan 2019: 7 commits
4FFeb 2019: 4 commits
4MMar 2019: 4 commits
4AApr 2019: 4 commits
11MMay 2019: 11 commits
JJun 2019: 0 commits
1JJul 2019: 1 commits

Jun 2017First module, for nQuire. CUDA 8, cuDNN 5.1 and 6, and a TensorFlow GPU module that loads both follow the same month.
Sep 2017cfncluster module added, with notes on installing cfnCluster in a virtual environment.
Dec 2017hpcc_utils module for HPCC admin tools.
Feb 2018cfnCluster 1.4 module.
Mar 2018AWS cluster module repository started; HPCC AWS Cluster manual written, with recorded setup and operation walkthroughs.
Oct 2018Introduction to Python workshop files.
May 2019RAxML-NG MPI, RevBayes in a Singularity container and as an MPI build, NCBI BLAST 2.9.0, and the Quantum ESPRESSO Slurm examples.
Jul 2019Last commit: an NCBI database module.

07 / Links

Read the source

Host names in the manual are historical and may no longer resolve. The code and docs were created for UCR HPCC; no license is granted beyond viewing.