Benchmarking

Benchmarks

The following benchmarks are selected to be used on the first day of the competition.

HPC Challenge

HPC Challenge (HPCC) will be used to score the benchmark portion of the competition. A team may execute HPCC as many times as desired during the setup and benchmarking phase, but the HPCC run submitted for scoring will define the hardware baseline for the rest of the competition. http://icl.cs.utk.edu/hpcc/

The rules described in the “Rules” section of HPCC web page on code modification does apply. For details, please visit the Rules section: http://icl.cs.utk.edu/hpcc/overview/index.html

High Performance LINPACK (HPL)

The competition will feature a “High LINPACK” award for the team submitting the highest HPL score. Additional, independent HPL runs (outside the submitted HPCC run) may be considered for the “Highest LINPACK” award if they are performed with exactly the same hardware powered on as used for HPCC run submitted for scoring. While eligible for the Highest LINPACK award, independent HPL runs will NOT count toward the team’s overall score. The HPL run must be submitted on the first day of the competition. For more information, please visit: http://www.netlib.org/benchmark/hpl/

HPCG

HPCG stands for High Performance Conjugate Gradient. It is a self-contained benchmark that generates and solves a synthetic 3D sparse linear system using a local symmetric Gauss-Seidel preconditioned conjugate gradient method. HPCG is a software package that performs a fixed number of symmetric Gauss-Seidel preconditioned conjugate gradient iterations using double precision (64 bit) floating point values. Integer arrays have global and local scope (global indices are unique across the entire distributed memory system, local indices are unique within a memory image). Reference implementation is written in C++ with MPI and OpenMP support. Please visit: http://www.hpcg-benchmark.org/ HPCG is being used on the first day of the competition. 30 minutes is the minimum time needed for the official run.

Applications

The following applications are selected to be used on the second day and third day of the competition.

FEniCS

FEniCS is a popular computing platform for partial differential equations (PDE). FEniCS enables users to quickly translate scientific models into efficient finite element code. With the high-level Python and C++ interfaces to FEniCS, it is easy to get started, but FEniCS offers also powerful capabilities for more experienced programmers. FEniCS runs on a multitude of platforms ranging from laptops to high-performance clusters. For more information about FEniCS, please visit: https://fenicsproject.org

Coding Challenge: MiniDFT

MiniDFT is a plane-wave denstity functional theory (DFT) mini-app for modeling materials. Given an set of atomic coordinates and pseudopotentials, MiniDFT computes self-consistent solutions of the Kohn-Sham equations using either the LDA or PBE exchange-correlation functionals.  The MiniDFT mini-app was excised from the general-purpose Quantum Espresso (QE) code.

For the detailed rules on the coding challenge, please README on the ISC17-SCC-miniDFT github:

https://github.com/RSE-Cambridge/ISC17-SCC-miniDFT

Teams to use GitHub issues (https://github.com/RSE-Cambridge/ISC17-SCC-miniDFT/issues) to raise questions about the code, the competition or clarify aspect of the judgement.

 

The Code Challenge competition is composed of the three parts:

  1. Run a provided input case (pe-23.LOCAL.in) on the system of the competition prior to the competition, submit the output to SCC committee by Friday, June 16th 5:00pm UTC
  2. Run during the competition the known input case (pe-23.LOCAL.in) then prove during the competition to be able to replicate a similar timing submitted previously. It is allowed to tune and modifying further the code during the competition itself but all code.
  3. Run an unknown input case provided during the competition

During the competition on Tuesday, June 20th, each team is allowed to modify the miniDFT code without any limitation apart comply with verification rules. Changes include linking optimized libraries, improving multi-threading, play with optimization flags, porting of portion (or entire) code on various accelerator flavors, rewrite parallelism, re-implement an algorithm, etc. There is no upper bound of how many lines of code can be modified.

Each team will make their entire miniDFT code publicly on GitHub before the competition starts and notify to SCC committee by Friday, June 16th 5:00pm UTC. It is allowed to tune and modifying further the code during the competition but new changes have to be publicly shared.

All teams will be able to see all GitHub public repositories. We created a GIT repository for each team, you can find all here: https://github.com/ISC17-SCC

Team members have to send their usernames and email addresses, so SCC Committee can add teams to the ISC-SCC GitHub repositories, so each team can push their own modified code there. Teams will need to produce a short report and/or presentation describing the activity done on the code. Please send by the same date these reports/presentation directly to us (via email, and box.com location), and not shared publicly.

TensorFlow

TensorFlow™ is an open source software library for numerical computation using data flow graphs. Nodes in the graph represent mathematical operations, while the graph edges represent the multidimensional data arrays (tensors) communicated between them. The flexible architecture allows you to deploy computation to one or more CPUs or GPUs in a desktop, server, or mobile device with a single API.

TensorFlow was originally developed by researchers and engineers working on the Google Brain Team within Google’s Machine Intelligence research organization for the purposes of conducting machine learning and deep neural networks research, but the system is general enough to be applicable in a wide variety of other domains as well. For more information about TensorFlow, please visit: https://www.tensorflow.org/

HPC Advisory Council has teamed up with Baidu in this year’s ISC Student Cluster Competition on the Deep Learning task. HPC Advisory Council has been working with the Baidu team to scope the type of applications and workload to be used for this year’s competition. It was decided the 12 student teams will compete on an image recognition task using TensorFlow, a popular open source library used for machine learning.

Baidu is a sponsor for this year’s ISC Student Cluster Competition. Baidu has provided expertise in the areas of Artificial Intelligence and Machine Learning, and also contributed time and resources to work closely with the HPC Advisory Council to prepare the workload and samples for teams to get familiar with image recognition tasks with TensorFlow. Baidu provided access to Baidu Cloud instances with GPU access for student teams. Teams can use the Baidu Cloud environment to experiment and prepare for the competition. Teams can use the Baidu Cloud instances for learning and training to prepare for the competition.

The last day of  the ISC Student Cluster Competition will feature the CAPTCHA image recognition challenge that will make use of the Tensorflow framework with a deep learning library called Keras, a high-level neural networks API written in Python. It is encouraged that teams to be familiar with Tensorflow and the nature of CAPTCHA. For more information about Keras, please visit: https://keras.io/

CAPTCHA ( “Completely Automated Public Turing test to tell Computers and Humans Apart”) is a type of challenge-response test used in computing to determine whether or not the user is human. The goal of the challenge is to train your defined model with TensorFlow using the Keras Deep Learning library.

Teams will be graded on the accuracy ratings of their model using the categorical_crossentropy loss, and the absolute correctness of inferencing un-trained data, which will be given on the day of the competition.  The ISC SCC organizing committee will generate and provide a number of difficulty levels of CAPTCHA verification

On the box share location (provided directly to each team), we have uploaded an example there for you to use.  It consists of image data, labels and a sample script.   The sample CAPTCHA images (captcha4_level1) have four elements, and vary in sizes of 160×60 pixels to 164×60 pixels. The sample images are to be used as training data for the Level1 CAPTCHA.   This sample data leverages 10 unique categories in the form of digits (0-9), using the to_categorical functions in Keras can be helpful.

Rules for the CAPTCHA image recognition challenge include:

The breakdown of the scores:

  • Teams will submit their code snippets and the outputs described
  • Teams must be able to achieve a high-level of accuracy when identifying the correctness of the CAPTCHA challenges
    • Not in all cases will your model achieve 100% accuracy or correctness in this type of challenge
    • Team score is based on the highest degree of accumulated accuracy,  combined with their metrics and loss results when your model is compiled, and the correctness identifying (a number of) similar untrained CAPTCHA samples
    • Teams must not exceed power budget allowed
    • A numpy random generator seed should be set as 7 in your code snippet for reproducibility of results by the judges
    • Teams are to produce and provide the Keras model, using the model.to_json and the HDF5 (.md5 model)
    • Teams are to provide their code, md5 & json files and output of the training of their best compiled model
    • Judges will verify all submitted requirements and run your model against the unseen sample data

Baidu has provided access to system resource with GPU access for all teams. Teams can use experiment and prepare for the competition, however Teams will use their own hardware for training and running the competition.

Secret Application(s)

Secret application(s) will be announced on the day of the competition.

Performance Efficiency

Please note that the performance efficiency might be considered for one of the section in the SCC. Please be familiar with the calculation of the Performance Efficiency.

To achieve the maximum performance efficiency, we can calculate using the formula below:

Max [ efficiency performance ] = max [ performance x performance/Theoretical Peak (in FLOPs)]

Max [ efficiency performance ] = max [ performance * efficiency ]

Awards – Six awards will be given

Highest LINPACK

The highest score received for the LINPACK benchmark under the power budget. Results of LINPACK must be turned in at the end of the first day.

Fan Favorite

To be given to the team which receives the most unique votes from ISC participants during the SCC.

1st, 2nd and 3rd Place Overall Winners

There will be 3 overall winner awards given to the teams that are determined by the scoring of the below. The scoring for the overall winners will be calculated using the scores from HPCC, the chosen applications, and the interview by the SCC board.

AI Award

The new award for this year for solving the CAPTCHA challenge and achieving the highest degree of model accuracy.

Scoring

The breakdown of the scores:

  • 10% for HPCC performance. (The submission for HPCC will need to be a single run with all the tests in HPCC)
  • 80% for applications runs. (The score for each successfully ran input data will be determined by the relative score to the highest scorer).
    • 10% HPCG
    • 10% FEniCS
    • 10% Secret Application
    • 25% MiniDFT
    • 25% TensorFlow
  • 10% for interview by the representatives of the SCC board.