Funded by a grant from the National Institute of Health, an IBM high performance computing system (often referred to as a supercomputer) was installed in the medical school’s data center in December 2009 and was put into operation in January 2010. During its first two years of operation the system has run over 1.3 million jobs for 200 users from both campuses. This represents over 10,000,000 CPU-hours of processing capacity or the equivalent of running your PC continuously for 1,100 years!
Thus far the system has been open to a limited number of users, but it is now time to open access to all Washington University Faculty and to establish a university wide center that will operate and sustain this capability. To accomplish these goals the Center for High Performance Computing was created by vote of the Executive Faculty of the School of Medicine during its November 2011 meeting. The Executive Faculty also approved a funding model that makes the facility cost effective for all users. The funding model consists of three components.
- Base Charge (membership fee) that is allocated uniformly to departments or programs on both on the Medical School campus and the Danforth campus.For FY12 the Base Charge is $5000For FY12, all medical school departments will contribute $5,000, unless they opt for premium membership. Danforth Campus departments can choose to contribute, however Departmental participation is required for faculty access to the system. This base annual fee cannot be allocated to grants as it is not a usage fee. The fee and mandatory contributions will be re-evaluated annually by the governing board of the center.
In November of 2012 the Executive Faculty Committee and the Dean approved a plan to drop the Usage fee for all users. Instead, the departments with the highest usage will be charged a higher Base Charge ($10000 instead of $5000).
Usage fee based on percentage of total system usage. This fee is allocated to departments, programs and large projects based on the primary appointment of each user. This fee will be billed quarterly and reflect the % of system resources utilized by that department. At the beginning of each fiscal year CHPC will provide a budget projection based on prior year actual usage. This fee can be charged to grants and is based on actual usage as measured by core-hours of processing. Currently the usage fee is based on a projected $.08 per core-hour. From historical data the average user requires approximately 20,000 core-hours which equates to $1,600 per user per year. In order to give investigators time to build these usage fees into their grant applications NO usage fee will be charged during FY12. Standard grant text will be provided on the CHPC web site for inclusion in future proposals and CHPC staff will assist with usage estimates. ICTS JIT and grant funding may be used to cover the usage fee.
Processor hours will be based on the number of processing-cores requested multiplied by the walltime that the job was active. If a user requests multiple cores, but only utilizes a single core, they will be charged for all of them since these were no longer available for others to utilize. The output file from the job will report the resources requested and utilized in the footer of the report:
[mtobias@login001 Sunjoos_Gromacs_Benchmark]$ tail gromacs_np32.o51864 Session: 16408 Limits: neednodes=4:ppn=8,nodes=4:ppn=8,walltime=24:00:00 Resources: cput=00:45:06,mem=1456184kb,vmem=8825664kb,walltime=00:01:28 Queue: dque Account: Nodes: node165 node166 node167 node168 Killing leftovers... End PBS Epilogue Mon Jun 28 08:45:45 CDT 2010 1277732745
In this case the user requested 32 cores (4 nodes x 8 processors per node) and ran for 88s. The charge would be for 32 * 88 * (1 hour/3600s) = 0.78 processor*hours. At the proposed rate of $0.08/processor*hour, this run would cost $0.063
- Premium Membership is for large projects or departments that know they will be heavy users of the system. For a fixed annual contribution (currently $100,000), all user from a premium member have priority access to the system (priority queue for launching jobs) and they have the right to request the entire system when needed. These users also have direct access to large capacity external storage systems that are directly mounted by the supercomputer. Current subscribers at this level are the Department of Radiology and the Human Connectome Project, but others are welcome.
Some members of the user community have requested access to high volume storage to support data centric applications such as genomic analysis or imaging. As an Option, CHPC can lease storage on the Radiology BlueArc storage array at the rate of $800 per Terabyte per year. This sounds like a lot of money but for that annual fee (prorated based on actual usage) you full backup (you actually have 2x the usable storage), and guaranteed availability of you data PLUS access to the 44TB high speed cache that is directly connected to the supercomputer. So you can store your research data safely and use the shared, high speed cache when you process your data. Again, this is an option only for those users who need the storage. Otherwise every user gets a storage allocation on the 9TB high speed storage pool inside the supercomputer as part of the standard fees.
CHPC offers the following services (see http://www.chpc.wustl.edu for details):
- Access to a professionally managed High Performance Computing System with
- 1856 high speed processors configured in both a distribute processing cluster and 7 high capacity, large memory symmetric multiprocessing units
- 10TB of memory
- 9 TB of internal high speed shared storage
- Usage of 35 pre-installed software packages that support molecular modeling, genomic analysis, advanced imaging, statistics and mathematical analysis (see http://chpc.wustl.edu/software.html for the complete list)
- The ability to request that new software be added to the system.
- Consultation with CHPC staff on parallel computing and software optimization
To make the Center for High Performance Computing a university wide resource it must be operated in a manner that permits full transparency and user community involvement in operations to make sure the center continues to meet the needs of all users. Two committees are in the process of formation. The Governing Board of the center will be open to 1 representative for each member entity and include the center director. The governing board will elect its own chairperson and meet at least annually to review and approve the budget and fee structure for the coming fiscal year. The governing board will also approve operating policies for the center. The Engineering Committee will be chaired by the system manager and will include faculty and staff members who are knowledgeable in high performance computing and parallel processing. This committee will guide the system manager and system administrator in establishing the best technical direction for center operations.
The goal of CHPC is to provide state of the art computing capabilities to support research at Washington University. As many of us are painfully aware the amount of data our research produces in growing at a tremendous rate. Having computing and storage capacity sufficient to keep up with this growth is a daunting challenge, but a challenge we must meet. A recent paper from researchers at the University of Arkansas pointed out, “Overall, our models indicated that investment in high-performance computing is a good predictor of research competitiveness at U.S. academic institutions.”  CHPC is one of the ways Washington University is making that investment.
 Apon A, Ahalt S, Dantuluri V, Gurdgiev C, Limayem M, Ngo L, Stealey M. High Performance Computing Instrumentation and Research Productivity in U.S. Universities, Journal of Information Technology Impact, 10( 2): 87-98, 2010. Available at SSRN: http://ssrn.com/abstract=1679248