IEEE International Conference on E-Business Engineering (ICEBE) 2023

Bio of Prof. Jie Lu

Distinguished Professor Jie Lu is a world-renowned scientist in the field of computational intelligence, primarily known for her work in fuzzy transfer learning, concept drift, recommender systems, and decision support systems. She is an IEEE Fellow, IFSA Fellow, and Australian Laureate Fellow. Currently, Prof Lu is the Director of the Australian Artificial Intelligence Institute (AAII) at University of Technology Sydney (UTS), Australia. She has published over 500 papers in leading journals and conferences; won 10 Australian Research Council (ARC) Discovery Projects as first chief investigator, and over 20 industry projects; and supervised 50 doctoral students to completion. Prof Lu serves as Editor-In-Chief for Knowledge-Based Systems and International Journal of Computational Intelligence Systems. She is a recognized keynote speaker, delivering over 40 keynote speeches at international conferences. She is the recipient of two IEEE Transactions on Fuzzy Systems Outstanding Paper Awards (2019 and 2022), NeurIPS2022 Outstanding Paper Award, Australia's Most Innovative Engineer Award (2019), Australasian Artificial Intelligence Distinguished Research Contribution Award (2022), and the Officer of the Order of Australia (AO) in the Australia Day 2023.

Talk Abstract

The talk will present how machine learning can innovatively and effectively learn from data to support data-driven decision-making in uncertain and dynamic situations, and automating the machine learning process, including source domain selection and related data stream selection. A set of new autonomous transfer learning theories, methodologies and algorithms will be presented that can transfer knowledge learnt in more source domains to a target domain by building latent space, mapping functions and self-training to overcome tremendous uncertainties in data, learning processes and decision outputs. Another set of autonomous concept drift theories, methodologies and algorithms will be discussed about how to handle ever-changing dynamic data stream environments with unpredictable pattern drifts in multiple streams by effectively and accurately detecting concept drift in an explanatory way, indicating when, where and how concept drift occurs and reacting accordingly. These new developments enable advanced machine learning and therefore enhance data-driven prediction and decision support systems in complex and dynamic real-world environments.

Bio of Prof. Jie Xu

Jie Xu is Chair of Computing at the University of Leeds, Director of the UK White Rose Grid e- Science Centre, involving the three White Rose Universities of Leeds, Sheffield and York, a co-Leader of the EPSRC-funded UK National Hub in Clouds and Distributed Computing, and Head of the Distributed Systems and Services (DSS) Theme at Leeds. Xu has worked in the field of Distributed Computing Systems for over thirty-five years, engaging closely with industrial leaders in the field. He received a PhD in Computing Science from the University of Newcastle upon Tyne, and was Professor of Distributed Systems at the University of Durham before joined Leeds in 2003.

Professor Xu is an executive member of UKCRC (UK Computing Research Committee) and a Turing Fellow in AI and Data Science. He has served as an academic expert for numerous governments and industries, such as Singapore IDA, Lenovo, UK EPSRC, and UK DTI (InnovateUK). In addition, he has extensive editorial experience, having served as an editor for IEEE Distributed Systems from 2000 to 2005, and currently acting as an associate editor of IEEE Transactions on Parallel and Distributed Systems and ACM Computing Surveys. Professor Xu is a Steering Committee member for several prestigious IEEE conferences, such as SRDS, ISORC, HASE, SOSE, JCC, and CISOSE, as well as serving on the steering board of IEEE TC on BIS. He has also been a General Chair/PC Chair for various IEEE international conferences. With over 300 academic publications, including papers in top-ranked IEEE and ACM Transactions, Professor Xu has received international research prizes, such as the BCS/AT&T Brendan Murphy Prize, and led or co-led more than 20 research projects worth over £30M. He is also the co-founder of two university spin-outs that specialize in data

Talk Abstract

In this presentation, we will share our recent experience with designing and implementing a practical system, STRONGHOLD, for training massive-scale language models with billions of parameters, with a focus on our offloading mechanism for efficiently moving data amongst GPU memory and CPU RAM/secondary storages.

Deep Learning is advancing rapidly, and with it, the size of foundation models is increasing exponentially. However, training these models requires significant GPU resources and power, which can be unaffordable for many academic and industry research teams. Even for AI teams in large companies, resources are limited, and purchasing and maintaining these devices can be prohibitively expensive. For instance, training a GPT-3 model requires over thousands of high-performance-configured A100 GPUs for continuous 3 months. Our system, STRONGHOLD, addresses this challenge by offloading model weights to CPU RAM or other secondary storages dynamically and loading them back when needed, minimizing GPU memory requirements. STRONGHOLD also allows data movement and on-GPU computation to overlap to hide the extra overhead introduced by the offloading mechanism. Compared to state-of-the-art offloading-based solutions, STRONGHOLD improves the trainable model size by 1.9x to 6.5x on a 32GB V100 GPU, with 1.2x to 3.7x improvement on the training throughput. We have successfully deployed STRONGHOLD in production to support large- scale DNN training.