Research review of federated learning algorithms

Jianzong WANG; Lingwei KONG; Zhangcheng HUANG; Linjie CHEN; Yi LIU; Anxun HE; Jing XIAO

doi:10.11959/j.issn.2096-0271.2020055

您当前的位置：

首页 >

文章列表页 >

Research review of federated learning algorithms

STUDY | 更新时间：2024-06-03

- Research review of federated learning algorithms
- Big Data Research Vol. 6, Issue 6, Pages: 2020055-1(2020)
- 作者机构：
  
  1. 平安科技（深圳）有限公司，广东深圳 518063
  2. 中国平安保险（集团）股份有限公司，广东深圳 518031
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-0271.2020055
  CLC： TP311
- Online First：2020-11，
  
  Published：15 November 2020
- 稿件说明：
移动端阅览
Jianzong WANG, Lingwei KONG, Zhangcheng HUANG, et al. Research review of federated learning algorithms[J]. Big Data Research, 2020, 6(6): 2020055-1.
DOI：

Jianzong WANG, Lingwei KONG, Zhangcheng HUANG, et al. Research review of federated learning algorithms[J]. Big Data Research, 2020, 6(6): 2020055-1. DOI： 10.11959/j.issn.2096-0271.2020055.

摘要

近年来，联邦学习作为解决数据孤岛问题的技术被广泛关注，已经开始被应用于金融、医疗健康以及智慧城市等领域。从3个层面系统阐述联邦学习算法。首先通过联邦学习的定义、架构、分类以及与传统分布式学习的对比来阐述联邦学习的概念；然后基于机器学习和深度学习对目前各类联邦学习算法进行分类比较和深入分析；最后分别从通信成本、客户端选择、聚合方式优化的角度对联邦学习优化算法进行分类，总结了联邦学习的研究现状，并提出了联邦学习面临的通信、系统异构、数据异构三大难题和解决方案，以及对未来的期望。

Abstract

In recent years

federated learning has been proposed and received widespread attention to overcome data isolated island challenge.Federated learning related researches were adopted in areas such as financial field

healthcare domain and smart city related application.Federated learning concept was introduced into three different layers.The first layer introduced the definition

architecture

classification of federated learning and compared the federated learning with traditional distributed learning.The second layer presented comparison and analysis of federated learning algorithms from machine learning and deep learning aspects.The third layer separated federated learning optimization algorithms into three aspects to optimize federated learning algorithm through reducing communication cost

selecting proper clients and different aggregation method.Finally

the current research status and three main challenges on communication

heterogeneity of system and data to be solved were concluded

and the future prospects in federated learning domain were proposed.

关键词

Keywords

references

LECUN Y , BENGIO Y , HINTON G . Deep learning [J ] . Nature , 2015 , 521 ( 7553 ): 436 - 444 .

王健宗 , 黄章成 , 肖京 . 人工智能赋能金融科技 [J ] . 大数据 , 2018 , 4 ( 3 ): 114 - 119 .

WANG J Z , HUANG Z C , XIAO J . Artificial intelligence energize Fintech [J ] . Big Data Research , 2018 , 4 ( 3 ): 114 - 119 .

KAIROUZ P , MCMAHAN H B , AVENT B , et al . Advances and open problems in federated learning [J ] . arXiv preprint,2019,arXiv:1912.04977 .

YANG Q , LIU Y , CHEN T , et al . Federated machine learning:concept and applications [J ] . ACM Transactions on Intelligent Systems and Technology , 2019 , 10 ( 2 ): 1 - 19 .

LI T , SAHU A K , TALWALKAR A , et al . Federated learning:challenges,methods,and future directions [J ] . IEEE Signal Processing Magazine , 2020 , 37 ( 3 ): 50 - 60 .

MEHMOOD A , NATGUNANATHAN I , XIANG Y , et al . Protection of big data privacy [J ] . IEEE Access , 2016 , 4 : 1821 - 1834 .

方滨兴 , 贾焰 , 李爱平 , 等 . 大数据隐私保护技术综述 [J ] . 大数据 , 2016 , 2 ( 1 ): 1 - 18 .

FANG B X , JIA Y , LI A P , et al . Privacy preservation in big data:a survey [J ] . Big Data Research , 2016 , 2 ( 1 ): 1 - 18 .

KONEČNÝ J , MCMAHAN H B , RAMAGE D , et al . Federated optimization:distributed machine learning for ondevice intelligence [J ] .. arXiv preprint,2016,arXiv:1610.02527 ,

KONEČNÝ J , MCMAHAN H B , YU F X , et al . Federated learning:strategies for improving communication efficiency [J ] . arXiv preprint,2016,arXiv:1610.05492 .

MCMAHAN H B , MOORE E , RAMAGE D , et al . Federated learning of deep networks using model averaging [J ] . arXiv preprint,2016,arXiv:1602.05629 .

MCMAHAN H B , MOORE E , RAMAGE D , et al . Communication-efficient learning of deep networks from decentralized data [C ] // Conference on Artificial Intelligence and Statistics.[S.l.:s.n] . 2017 .

LI T , SANJABI M , BEIRAMI A , et al . Fair resource allocation in federated learning [J ] . arXiv preprint,2019,arXiv:1905.10497 .

CHEN Y , SUN X Y , JIN Y C . Communication-efficient federated deep learning with layer wise asynchronous model update and temporally weighted aggregation [J ] . IEEE Transactions on Neural Networks and Learning Systems,2019:Accepted .

REHAK D R , DODDS P , LANNOM L . A model and infrastructure for federated learning content repositories [C ] // Interoperability of Web-Based Educational Systems Workshop.[S.l.:s.n] . 2005 .

LI M , ANDERSEN D G , PARK J W , et al . Scaling distributed machine learning with the parameter server [C ] // The 11th USENIX Symposium on Operating Systems Design and Implementation.[S.l.:s.n] . 2014 : 583 - 598 .

LIN Y J , HAN S , MAO H Z , et al . Deep gradient compression:reducing the communication bandwidth for distributed training [J ] . arXiv preprint,2017,arXiv:1712.01887 .

DAI W , KUMAR A , WEI J , et al . Highperformance distributed ML at scale through parameter server consistency models [C ] // AAAI Conference on Artificial Intelligence . New York:ACM Press , 2015 .

RECHT B , RE C , WRIGHT S , et al . Hogwild:a lock-free approach to parallelizing stochastic gradient descent [C ] // Advances in Neural Information Processing Systems.[S.l.:s.n] . 2011 : 693 - 701 .

HO Q , CIPAR J , CUI H G , et al . More effective distributed ml via a stale synchronous parallel parameter server [C ] // Advances in Neural Information Processing Systems.[S.l.:s.n] . 2013 : 1223 - 1231 .

FENG S W , YU H . Multi-participant multi-class vertical federated learning [J ] . arXiv preprint,2020,arXiv:2001.11154 .

KONEČNÝ J . Stochastic distributed and federated optimization for machine learning [J ] . arXiv preprint,2017,arXiv:1707.01155 .

LIU X Y , LI H W , XU G W , et al . Adaptive privacy-preserving federated learning [J ] . Peer-to-Peer Networking and Applications . 2020

HU R , GONG Y M , GUO Y X . CPFed:communication-efficient and privacypreserving federated learning [J ] . arXiv preprint,2020,arXiv:2003.13761 .

RYFFEL T , TRASK A , DAHL M , et al . A generic framework for privacy preserving deep learning [J ] . arXiv preprint,2018,arXiv:1811.04017 .

ANTONIOUS M , DEEPESH D , SUHAS D , et al . Shuffled model of federated learning:privacy,communication and accuracy trade-offs [J ] . arXiv preprint,2020,arXiv:008.07180 .

SMITH V , CHIANG C K , SANJABI M , et al . Federated multi-task learning [C ] // Advances in Neural Information Processing Systems.[S.l.:s.n] . 2017 : 4424 - 4434 .

CORINZIA L , BUHMANN J M . Variational federated multi-task learning [J ] . arXiv preprint,2019,arXiv:1906.06268 .

CALDAS S , SMITH V , TALWALKAR A . Federated kernelized multi-task learning [C ] // SysML Conference 2018.[S.l.:s.n] . 2018 .

KALLMAN R , KIMURA H , NATKINS J , et al . H-store:a high-performance,distributed main memory transaction processing system [J ] . Proceedings of the VLDB Endowment , 2008 , 1 ( 2 ): 1496 - 1499 .

YANG K , JIANG T , SHI Y M , et al . Federated learning via over-the-air computation [J ] . IEEE Transactions on Wireless Communications , 2020 , 19 ( 3 ): 2022 - 2035 .

NISHIO T , YONETANI R . Client selection for federated learning with heterogeneous resources in mobile edge [C ] // 2019 IEEE International Conference on Communications . Piscataway:IEEE Press , 2019 : 1 - 7 .

WANG J Y , SAHU A K , YANG Z Y , et al . MATCHA:speeding up decentralized SGD via matching decomposition sampling [J ] . arXiv preprint,2019,arXiv:1905.09435 .

REISIZADEH A , MOKHTARI A , HASSANI H , et al . Fedpaq:a communication-efficient federated learning method with periodic averaging and quantization [J ] . arXiv preprint,2019,arXiv:1909.13014 .

KHALED A , MISHCHENKO K , RICHTÁRIK P . Better communication complexity for local SGD [J ] . arXiv preprint,2019,arXiv:1909.04746 .

LI S Y , CHENG Y , LIU Y , et al . Abnormal client behavior detection in federated learning [J ] . arXiv preprint,2019,arXiv:1910.09933 .

SATTLER F , WIEDEMANN S,MÜLLER K R , et al . Robust and communicationefficient federated learning from nonIID data [J ] . IEEE Transactions on Neural Networks and Learning Systems . 2019

C ROTTY A , GALAKATOS A , KRASKA T . Tupleware:distributed machine learning on small clusters [J ] . IEEE Data Engineering Bulletin , 2014 , 37 ( 3 ): 63 - 76 .

JOLFAEI A , OSTOVARI P , ALAZAB M , et al . Guest editorial special issue on privacy and security in distributed edge computing and evolving IoT [J ] . IEEE Internet of Things Journal , 2020 , 7 ( 4 ): 2496 - 2500 .

SAHU A K , LI T , SANJABI M , et al . Federated optimization for heterogeneous networks [J ] . arXiv preprint,2018,arXiv:1812.06127 .

YANG K , FAN T , CHEN T J , et al . A quasi-newton method based vertical federated learning framework for logistic regression [J ] . arXiv preprint,2019,arXiv:1912.00513 .

YANG S W , REN B , ZHOU X H , et al . Parallel distributed logistic regression for vertical federated learning without thirdparty coordinator [J ] . arXiv preprint,2019,arXiv:1911.09824 .

GAO D S , JU C , WEI X G , et al . HHHFL:hierarchical heterogeneous horizontal federated learning for electroencephalography [J ] . arXiv preprint,2019,arXiv:1909.05784 .

LIU Y , KANG Y , ZHANG X W , et al . A communication efficient vertical federated learning framework [J ] . arXiv preprint,2019,arXiv:1912.11187 .

SHARMA S , XING C P , LIU Y , et al . Secure and efficient federated transfer learning [J ] . arXiv preprint,2019,arXiv:1910.13271 .

ZHAO Y , LI M , LAI L Z , et al . Federated learning with non-IID data [J ] . arXiv preprint,2018,arXiv:1806.00582 .

LIU Y , LIU Y T , LIU Z J , et al . Federated forest [J ] . IEEE Transactions on Big Data,2020:Accepted .

CHENG K W , FAN T , JIN Y L , et al . SecureBoost:a lossless federated learning framework [J ] . arXiv preprint,2019,arXiv:1901.08755 .

LI Q B , WEN Z Y , HE B S . Practical federated gradient boosting decision trees [J ] . arXiv preprint,2019,arXiv:1911.04206 .

HARTMANN V , MODI K , PUJOL J M , et al . Privacy-preserving classification with secret vector machines [J ] . arXiv preprint,2019,arXiv:1907.03373 .

ZHU X H , WANG J , HONG Z , et al . Federated learning of unsegmented Chinese text recognition model [C ] // 2019 IEEE 31st International Conference on Tools with Artificial Intelligence . Piscataway:IEEE Press , 2019 : 1341 - 1345 .

BHOWMICK A , DUCHI J , FREUDIGER J , et al . Protection against reconstruction and its applications in private federated learning [J ] . arXiv preprint,2018,arXiv:1812.00984 .

DUCHI J , JORDAN M I , MCMAHAN B . Estimation,optimization,and parallelism when data is sparse [C ] // In Advances in Neural Information Processing Systems . New York:ACM Press , 2013 .

CHILIMBI T , SUZUE Y , APACIBLE J , et al . Project adam:building an efficient and scalable deep learning training system [C ] // The 11th USENIX Symposium on Operating Systems Design and Implementation . New York:ACM Press , 2014 : 571 - 582 .

LIU Y , MUPPALA J K , VEERARAGHAVAN M , et al . Data center networks:topologies,architectures and fault-tolerance characteristics [M ] . Heidelberg : Springer Science ＆ Business MediaPress , 2013 .

BONAWITZ K , EICHNER H , GRIESKAMP W , et al . Towards federated learning at scale:system design [J ] . arXiv preprint,2019,arXiv:1902.01046 .

LI X , HUANG K , YANG W , et al . On the convergence of FedAvg on non-IID data [J ] . arXiv preprint,2019,arXiv:1907.02189 .

CALDAS S,KONEČNY J , MCMAHAN H B , et al . Expanding the reach of federated learning by reducing client resource requirements [J ] . arXiv preprint,2018,arXiv:1812.07210 .

NISHIO T , YONETANI R . Client selection for federated learning with heterogeneous resources in mobile edge [C ] // ICC 20192019 IEEE International Conference on Communications . Piscataway:IEEE Press , 2019 : 1 - 7 .

YOSHIDA N , NISHIO T , MORIKURA M , et al . Hybrid-FL for wireless networks:cooperative learning mechanism using non-IID data [J ] . arXiv preprint,2019,arXiv:1905.07210 .

SPRAGUE M R , JALALIRAD A , SCAVUZZO M , et al . Asynchronous federated learning for geospatial applications [C ] // Joint European Conference on Machine Learning and Knowledge Discovery in Databases . Heidelberg:Springer , 2018 : 21 - 28 .

XIE C , KOYEJO S , GUPTA I . Asynchronous federated optimization [J ] . arXiv preprint,2019,arXiv:1903.03934 .

YANG J L , DUAN Y X , QIAO T , et al . Prototyping federated learning on edge computing systems [J ] . Frontiers of Computer Science , 2020 , 14 : 1 - 3 .

WANG S Q , TUOR T , SALONIDIS T , et al . Adaptive federated learning in resource constrained edge computing systems [J ] . IEEE Journal on Selected Areas in Communications , 2019 , 37 ( 6 ): 1205 - 1221 .

ZHAO Y , ZHAO J , JIANG L S , et al . Mobile edge computing,blockchain and reputation-based crowd-sourcing IoT federated learning:a secure,decentralized and privacy-preserving system [J ] . arXiv preprint,2019,arXiv:1906.10893 .

LI Z Y , LIU J , HAO J L , et al . CrowdSFL:a secure crowd computing framework based on blockchain and federated learning [J ] . Electronics , 2020 , 9 ( 5 ):773.

KANG J W , XIONG Z H , NIYATO D , et al . Incentive design for efficient federated learning in mobile networks:a contract theory approach [C ] // 2019 IEEE VTS Asia Pacific Wireless Communications Symposium . Piscataway:IEEE Press , 2019 : 1 - 5 .

ISAKSSON M , NORRMAN K . Secure federated learning in 5G mobile networks [J ] . arXiv preprint,2020,arXiv:2004.06700 .

Views

3431

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Data privacy,monopoly and fairness for AI

Research on data asset circulation and trading scheme integrating blockchain and federated learning technologies

Difficulties and explorations in data privacy protection for large language models

A scalable parallel sorting algorithm by regular sampling for big data

Research on the application of government big data platform based on federated learning

Related Author

Xiaofeng MENG

Leixia WANG

Junxu LIU

ZHAO Chen

MO Zijia

SHI Min

YANG Haijun

Ying WANG

Related Institution

Postdoctoral Research Center, Industrial and Commercial Bank of China

School of Cyber Science and Engineering, Ningbo University of Technology

CyberSpace Administration of Shanghai

College of Computer Science and Technology, Zhejiang University

Big Data Development Center of Zhejiang Province

Postal code：100079
Tel：（010）53879208、53859533 Email：bdr@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：273228 Visits Today：134

⁰