Sensor Analytics Using Multi-Task Learning and Federated Learning
Date
2021
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Sensors and internet of things (IoTs) are ubiquitous in our modern day-to-day living. The past decade has been marked by the rapid emergence and proliferation of a myriad of small devices. Applications range from smart home devices that control cooking ranges to mobile phones, wearable devices that serve as fitness trackers and personalized coaches. There is a critical need for the analysis of heterogeneous multivariate temporal data obtained from individual sensors.Multi-task learning (MTL) is a machine learning framework which learns a shared representations across related tasks and naturally suited for sensor data learning. I first propose two approaches by utilizing multi-task learning framework with attention mechanism, which can jointly trains classification/regression models from multiple related tasks where data on each task is generated from one or more sensors. The temporal and non-linear relationships underlying the captured data are modeled using convolution neural network (CNN) and long-short term memory (LSTM) models. And the attention mechanism seeks to learn shared feature representations across multiple tasks for improving the overall generalizability of the machine learning model.
However, as massive data is generated from the edge devices, combined with the growth of computing power on these devices, it is attractive to learn models directly over networks with data on these distributed devices. Federated learning (FL) is a machine learning paradigm where a shared central model is learned across distributed edge devices while the training data remains on these devices. Federated Averaging (FedAvg) is the leading optimization method for training non-convex models in this setting with a synchronized protocol. But the assumptions made by FedAvg are not realistic given the heterogeneity of devices. In particular, the volume and distribution of collected data vary in the training process due to different sampling rates of edge devices and users’ preference change. The edge devices themselves also vary in their available communication bandwidth and system configurations, such as memory, processor speed, and power requirements. This leads to vastly different training times as well as model/data transfer times. Furthermore, availability issues at edge devices can lead to a lack of contribution from specific edge devices to the federated model. Therefore, I further propose an asynchronous online approach and another collaborative work with tiers in the federated learning framework to tackle these issues. Specially, to deal with the distribution change problem of the streaming data, I propose one more approach for online federated learning with concept drift. In this work, I apply techniques on both local learning and the central model aggregation to alleviate the effect on the global model performance with local data concept drift. In addition, a new method is proposed to select local clients at each round to reduce the overall communication costs.
Motivated by the growing interest and challenges in federated learning research, this thesis demonstrates the strengths of academic performance on both prediction and communication-efficient performance on multiple benchmarks. Incorporating the techniques in this thesis to develop federated learning systems with more accurate prediction, lower communication costs and resource-efficient learning. More importantly, effective online approaches are proposed to tackle the issues with streaming data in real-life setting.