1. 平安科技(深圳)有限公司,广东 深圳 518063
2. 中国科学技术大学,安徽 合肥 230026
[ "吴建汉(1998- ),男,中国科学技术大学硕士生,平安科技(深圳)有限公司算法工程师,中国计算机学会(CCF)学生会员,主要研究方向为计算机视觉和联邦学习" ]
[ "司世景(1988- ),男,博士,平安科技(深圳)有限公司资深算法研究员,CCF会员,主要研究方向为机器学习及其在人工智能领域的应用" ]
[ "王健宗(1983- ),男,博士,平安科技(深圳)有限公司副总工程师,资深人工智能总监,联邦学习技术部总经理,CCF高级会员,CCF大数据专家委员会委员,主要研究方向为联邦学习和人工智能等" ]
[ "肖京(1972- ),男,博士,平安科技(深圳)有限公司首席科学家,2019年吴文俊人工智能杰出贡献奖获得者,CCF深圳会员活动中心副主席,主要研究方向为计算机图形学学科、自动驾驶、3D显示、医疗诊断、联邦学习等" ]
网络首发:2022-09,
纸质出版:2022-09-15
移动端阅览
吴建汉, 司世景, 王健宗, 等. 联邦学习攻击与防御综述[J]. 大数据, 2022,8(5):12-32.
Jianhan WU, Shijing SI, Jianzong WANG, et al. Threats and defenses of federated learning: a survey[J]. Big data research, 2022, 8(5): 12-32.
吴建汉, 司世景, 王健宗, 等. 联邦学习攻击与防御综述[J]. 大数据, 2022,8(5):12-32. DOI: 10.11959/j.issn.2096-0271.2022038.
Jianhan WU, Shijing SI, Jianzong WANG, et al. Threats and defenses of federated learning: a survey[J]. Big data research, 2022, 8(5): 12-32. DOI: 10.11959/j.issn.2096-0271.2022038.
随着机器学习技术的广泛应用,数据安全问题时有发生,人们对数据隐私保护的需求日渐显现,这无疑降低了不同实体间共享数据的可能性,导致数据难以共享,形成“数据孤岛”。联邦学习可以有效解决“数据孤岛”问题。联邦学习本质上是一种分布式的机器学习,其最大的特点是将用户数据保存在用户本地,模型联合训练过程中不会泄露各参与方的原始数据。尽管如此,联邦学习在实际应用中仍然存在许多安全隐患,需要深入研究。对联邦学习可能受到的攻击及相应的防御措施进行系统性的梳理。首先根据联邦学习的训练环节对其可能受到的攻击和威胁进行分类,列举各个类别的攻击方法,并介绍相应攻击的攻击原理;然后针对这些攻击和威胁总结具体的防御措施,并进行原理分析,以期为初次接触这一领域的研究人员提供详实的参考;最后对该研究领域的未来工作进行展望,指出几个需要重点关注的方向,帮助提高联邦学习的安全性。
With the comprehensive application of machine learning technology
data security problems occur from time to time
and people’s demand for privacy protection is emerging
which undoubtedly reduces the possibility of data sharing between different entities
making it difficult to make full use of data and giving rise to data islands.Federated learning (FL)
as an effective method to solve the problem of data islands
is essentially distributed machine learning.Its biggest characteristic is to save user data locally so that the models’ joint training process won’t leak sensitive data of partners.Nevertheless
there are still many security risks in federated learning in reality
which need to be further studied.The possible attack means and corresponding defense measures were investigated in federal learning comprehensively and systematically.Firstly
the possible attacks and threats were classified according to the training stages of federal learning
common attack methods of each category were enumerated
and the attack principle of corresponding attacks was introduced.Then the specific defense measures against these attacks and threats were summarized along with the principle analysis
to provide a detailed reference for the researchers who first contact this field.Finally
the future work in this research area was highlighted
and several areas that need to be focused on were pointed out to help improve the security of federal learning.
ZHANG C , XIE Y , BAI H , et al . A survey on federated learning [J ] . KnowledgeBased Systems , 2021 , 216 : 106775 .
ALEDHARI M , RAZZAK R , PARIZI R M , et al . Federated learning:a survey on enabling technologies,protocols,and applications [J ] . IEEE Access:Practical Innovations,Open Solutions , 2020 , 8 : 140699 - 140725 .
BLANCO-JUSTICIA A , DOMINGOFERRER J , MARTÍNEZ S , , et al . Achieving security and privacy in federated learning systems:survey,research challenges and future directions [J ] . Engineering Applications of Artificial Intelligence , 2021 , 106 : 104468 .
YANG Q , LIU Y , CHENG Y , et al . Federated learning [J ] . Synthesis Lectures on Artificial Intelligence and Machine Learning , 2019 , 13 ( 3 ): 1 - 207 .
LI T , SAHU A K , TALWALKAR A , et al . Federated learning:challenges,methods,and future directions [J ] . IEEE Signal Processing Magazine , 2020 , 37 ( 3 ): 50 - 60 .
TRUONG N , SUN K , WANG S Y , et al . Privacy preservation in federated learning:an insightful survey from the GDPR perspective [J ] . Computers &Security , 2021 , 110 : 102402 .
KONEČNÝ J , MCMAHAN H B , YU F X , et al . Federated learning:strategies for improving communication efficiency [J ] . arXiv preprint,2016,arXiv:1610.05492 .
马嘉华 , 孙兴华 , 夏文超 , 等 . 基于标签量信息的联邦学习节点选择算法 [J ] . 物联网学报 , 2021 , 5 ( 4 ): 46 - 53 .
MA J H , SUN X H , XIA W C , et al . Node selection based on label quantity information in federated learning [J ] . Chinese Journal on Internet of Things , 2021 , 5 ( 4 ): 46 - 53 .
ABDULRAHMAN S , TOUT H , OULDSLIMANE H , et al . A survey on federated learning:the journey from centralized to distributed on-site learning and beyond [J ] . IEEE Internet of Things Journal , 2021 , 8 ( 7 ): 5476 - 5497 .
王健宗 , 孔令炜 , 黄章成 , 等 . 联邦学习算法综述 [J ] . 大数据 , 2020 , 6 ( 6 ): 64 - 82 .
WANG J Z , KONG L W , HUANG Z C , et al . Research review of federated learning algorithms [J ] . Big Data Research , 2020 , 6 ( 6 ): 64 - 82 .
LI L , FAN Y X , TSE M , et al . A review of applications in federated learning [J ] . Computers & Industrial Engineering , 2020 , 149 : 106854 .
LIU Y , KANG Y , XING C P , et al . A secure federated transfer learning framework [J ] . IEEE Intelligent Systems , 2020 , 35 ( 4 ): 70 - 82 .
KAIROUZ P , MCMAHAN H B , AVENT B , et al . Advances and open problems in federated learning [J ] . arXiv preprint,2019,arXiv:1912.04977 .
ZHAO S H , MA X J , ZHENG X , et al . Clean-label backdoor attacks on video recognition models [C ] // Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2020 : 14431 - 14440 .
BHAGOJI A N , CHAKRABORTY S , MITTAL P , et al . Analyzing federated learning through an adversarial lens [C ] // Proceedings of the 36th International Conference On Machine Learning .[S.l.:s.n. ] , 2019 : 634 - 643 .
SHAFAHI A , HUANG W R , NAJIBI M , et al . Poison frogs! targeted clean-label poisoning attacks on neural networks [C ] // Proceedings of the 32nd International Conference on Neural Information Processing Systems.Red Hook:Curran Associates Inc . , 2018 : 6106 - 6116 .
BIGGIO B , NELSON B , LASKOV P . Poisoning attacks against support vector machines [J ] . arXiv preprint,2012,arXiv:1206.6389 .
CHEN X Y , LIU C , LI B , et al . Targeted backdoor attacks on deep learning systems using data poisoning [J ] . arXiv preprint,2017,arXiv:1712.05526 .
TOLPEGIN V , TRUEX S , GURSOY M E , et al . Data poisoning attacks against federated learning systems [C ] // Proceedings of 2020 European Symposium on Research in Computer Security . Cham:Springer , 2020 : 480 - 501 .
BAGDASARYAN E , VEIT A , HUA Y Q , et al . How to backdoor federated learning [J ] . arXiv preprint,2018,arXiv:1807.00459 .
JERE M S , FARNAN T , KOUSHANFAR F . A taxonomy of attacks on federated learning [J ] . IEEE Security & Privacy , 2021 , 19 ( 2 ): 20 - 28 .
ZHOU X C , XU M , WU Y M , et al . Deep model poisoning attack on federated learning [J ] . Future Internet , 2021 , 13 ( 3 ): 73 .
FANG M H , CAO X Y , JIA J Y , et al . Local model poisoning attacks to byzantinerobust federated learning [C ] // Proceedings of the 29th USENIX Conference on Security Symposium . Berkeley:USENIX Association , 2020 : 1623 - 1640 .
BERNSTEIN J , ZHAO J W , AZIZZADENESHELI K , et al . signSGD with majority vote is communication efficient and fault tolerant [J ] . arXiv preprint,2018,arXiv:1810.05291 .
XIE C , KOYEJO S , GUPTA I . Fall of empires:breaking Byzantine-tolerant SGD by inner product manipulation [J ] . arXiv preprint,2019,arXiv:1903.03936 .
SHEJWALKAR V , HOUMANSADR A . Manipulating the Byzantine:optimizing model poisoning attacks and defenses for federated learning [C ] // Proceedings of 2021 Network and Distributed System Security Symposium . Reston:Internet Society , 2021 :18.
LIU Y F , MA X J , BAILEY J , et al . Reflection backdoor:a natural backdoor attack on deep neural networks [C ] // Proceedings of 2020 European Conference on Computer Vision . Cham:Springer , 2020 : 182 - 199 .
COSTA G , PINELLI F , SODERI S , et al . Covert channel attack to federated learning systems [J ] . arXiv preprint,2021,arXiv:2104.10561 .
LEE H , KIM J , HUSSAIN R , et al . On defensive neural networks against inference attack in federated learning [C ] // Proceedings of 2021 IEEE International Conference on Communications . Piscataway:IEEE Press , 2021 : 1 - 6 .
AONO Y , HAYASHI T , PHONG L T , et al . Scalable and secure logistic regression via homomorphic encryption [C ] // Proceedings of the 6th ACM Conference on Data and Application Security and Privacy . New York:ACM Press , 2016 : 142 - 144 .
LUO X J , WU Y C , XIAO X K , et al . Feature inference attack on model predictions in vertical federated learning [C ] // Proceedings of 2021 IEEE 37th International Conference on Data Engineering . Piscataway:IEEE Press , 2021 : 181 - 192 .
WAINAKH A , VENTOLA F , MÜßIG T , , et al . User label leakage from gradients in federated learning [J ] . arXiv preprint,2021,arXiv:2105.09369 .
NASR M , SHOKRI R , HOUMANSADR A . Comprehensive privacy analysis of deep learning:passive and active white-box inference attacks against centralized and federated learning [C ] // Proceedings of 2019 IEEE Symposium on Security and Privacy . Piscataway:IEEE Press , 2019 : 739 - 753 .
DONG Y P , SU H , WU B Y , et al . Efficient decision-based black-box adversarial attacks on face recognition [C ] // Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2019 : 7706 - 7714 .
YIN Z Y , YUAN Y , GUO P F , et al . Backdoor attacks on federated learning with lottery ticket hypothesis [J ] . arXiv preprint,2021,arXiv:2109.10512 .
CHENG M H , LE T , CHEN P Y , et al . Query-efficient hard-label black-box attack:an optimization-based approach [J ] . arXiv preprint,2018,arXiv:1807.04457 .
LI Y D , LI L J , WANG L Q , et al . NATTACK:learning the distributions of adversarial examples for an improved black-box attack on deep neural networks [C ] // Proceedings of the 36th International Conference on Machine Learning .[S.l.:s.n. ] , 2019 : 3866 - 3876 .
BAI Y , CHEN D G , CHEN T , et al . GANMIA:GAN-based black-box membership inference attack [C ] // Proceedings of 2021 IEEE International Conference on Communications . Piscataway:IEEE Press , 2021 : 1 - 6 .
ZHANG Y H , JIA R X , PEI H Z , et al . The secret revealer:generative modelinversion attacks against deep neural networks [C ] // Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2020 : 250 - 258 .
REN H C , DENG J J , XIE X H . GRNN:generative regression neural network—a data leakage attack for federated learning [J ] . ACM Transactions on Intelligent Systems and Technology , 2022 , 13 ( 4 ): 1 - 24 .
HITAJ B , ATENIESE G , PEREZCRUZ F . Deep models under the GAN:information leakage from collaborative deep learning [C ] // Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security . New York:ACM Press , 2017 : 603 - 618 .
LYU L J , YU H , YANG Q . Threats to federated learning:a survey [J ] . arXiv preprint,2020,arXiv:2003.02133 .
SONG M K , WANG Z B , ZHANG Z F , et al . Analyzing user-level privacy attack against federated learning [J ] . IEEE Journal on Selected Areas in Communications , 2020 , 38 ( 10 ): 2430 - 2444 .
BOUACIDA N , MOHAPATRA P . Vulnerabilities in federated learning [J ] . IEEE Access , 2021 , 9 : 63229 - 63249 .
WANG Z B , SONG M K , ZHANG Z F , et al . Beyond inferring class representatives:user-level privacy leakage from federated learning [C ] // Proceedings of 2019 IEEE Conference on Computer Communications . Piscataway:IEEE Press , 2019 : 2512 - 2520 .
MOTHUKURI V , PARIZI R M , POURIYEH S , et al . A survey on security and privacy of federated learning [J ] . Future Generation Computer Systems , 2021 , 115 : 619 - 640 .
LYU L J , YU H , MA X J , et al . Privacy and robustness in federated learning:attacks and defenses [J ] . arXiv preprint,2020,arXiv:2012 . 06337 .
WEI K , LI J , DING M , et al . Federated learning with differential privacy:algorithms and performance analysis [J ] . IEEE Transactions on Information Forensics and Security , 2020 , 15 : 3454 - 3469 .
GIRGIS A M , DATA D , DIGGAVI S , et al . Shuffled model of differential privacy in federated learning [C ] // Proceedings of 2021 International Conference on Artificial Intelligence and Statistics .[S.l.:s.n. ] , 2021 : 2521 - 2529 .
HU R , GUO Y X , LI H N , et al . Personalized federated learning with differential privacy [J ] . IEEE Internet of Things Journal , 2020 , 7 ( 10 ): 9530 - 9539 .
MCMAHAN H B , RAMAGE D , TALWAR K , et al . Learning differentially private recurrent language models [J ] . arXiv preprint,2017,arXiv:1710.06963 .
GEYER R C , KLEIN T , NABI M . Differentially private federated learning:a client level perspective [J ] . arXiv preprint,2017,arXiv:1712.07557 .
SUN L C , QIAN J W , CHEN X . LDPFL:practical private aggregation in federated learning with local differential privacy [C ] // Proceedings of the 30th International Joint Conference on Artificial Intelligence . California:International Joint Conferences on Artificial Intelligence Organization , 2021 : 1571 - 1578 .
DUCHI J C , JORDAN M I , WAINWRIGHT M J . Local privacy and statistical minimax rates [C ] // Proceedings of 2013 IEEE 54th Annual Symposium on Foundations of Computer Science . Piscataway:IEEE Press , 2013 : 429 - 438 .
ERLINGSSON Ú , PIHUR V , KOROLOVA A . RAPPOR:randomized aggregatable privacy-preserving ordinal response [C ] // Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security . New York:ACM Press , 2014 : 1054 - 1067 .
RASTOGI V , NATH S . Differentially private aggregation of distributed time-series with transformation and encryption [C ] // Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data . New York:ACM Press , 2010 : 735 - 746 .
AGARWAL N , SURESH A T , YU F , et al . cpSGD:communication-efficient and differentially-private distributed SGD [C ] // Proceedings of the 32nd International Conference on Neural Information Processing Systems.Red Hook:Curran Associates Inc . , 2018 : 7575 - 7586 .
ZHANG C L , LI S Y , XIA J Z , et al . BatchCrypt:efficient homomorphic encryption for cross-silo federated learning [C ] // Proceedings of the 2020 USENIX Annual Technical Conference . Berkeley:USENIX Association , 2020 : 493 - 506 .
FANG H K , QIAN Q . Privacy preserving machine learning with homomorphic encryption and federated learning [J ] . Future Internet , 2021 , 13 ( 4 ): 94 .
GENTRY C , . Fully homomorphic encryption using ideal lattices [C ] // Proceedings of the 41st Annual ACM Symposium on Theory of Computing . New York:ACM Press , 2009 : 169 - 178 .
PHONG L T , AONO Y , HAYASHI T , et al . Privacy-preserving deep learning via additively homomorphic encryption [J ] . IEEE Transactions on Information Forensics and Security , 2018 , 13 ( 5 ): 1333 - 1345 .
YANG T , ANDREW G , EICHNER H , et al . Applied federated learning:improving google keyboard query suggestions [J ] . arXiv preprint,2018,arXiv:1812.02903 .
MADI A , STAN O , MAYOUE A , et al . A secure federated learning framework using homomorphic encryption and verifiable computing [C ] // Proceedings of 2021 Reconciling Data Analytics,Automation,Privacy,and Security:A Big Data Challenge . Piscataway:IEEE Press , 2020 : 1 - 8 .
ZHU H F , MONG GOH R S , NG W K . Privacy-preserving weighted federated learning within the secret sharing framework [J ] . IEEE Access , 2020 , 8 : 198275 - 198284 .
CHA J , SINGH S K , KIM T W , et al . Blockchain-empowered cloud architecture based on secret sharing for smart city [J ] . Journal of Information Security and Applications , 2021 , 57 : 102686 .
BONAWITZ K , IVANOV V , KREUTER B , et al . Practical secure aggregation for privacy-preserving machine learning [C ] // Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security . New York:ACM Press , 2017 : 1175 - 1191 .
HAN G , ZHANG T T , ZHANG Y H , et al . Verifiable and privacy preserving federated learning without fully trusted centers [J ] . Journal of Ambient Intelligence and Humanized Computing , 2022 , 13 ( 3 ): 1431 - 1441 .
CHANDRAMOULI A , CHOUDHURY A , PATRA A . A survey on perfectlysecure verifiable secret-sharing [J ] . ACM Computing Surveys , 2022 .
FEREIDOONI H , MARCHAL S , MIETTINEN M , et al . SAFELearn:secure aggregation for private FEderated learning [C ] // Proceedings of 2021 IEEE Security and Privacy Workshops . Piscataway:IEEE Press , 2021 : 56 - 62 .
周俊 , 方国英 , 吴楠 . 联邦学习安全与隐私保护研究综述 [J ] . 西华大学学报(自然科学版) , 2020 , 39 ( 4 ): 9 - 17 .
ZHOU J , FANG G Y , WU N . Survey on security and privacy-preserving in federated learning [J ] . Journal of Xihua University (Natural Science Edition) , 2020 , 39 ( 4 ): 9 - 17 .
BARACALDO N , CHEN B , LUDWIG H , et al . Mitigating poisoning attacks on machine learning models:a data provenance based approach [C ] // Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security . New York:ACM Press , 2017 : 103 - 110 .
SATTLER F , WIEDEMANN S , MÜLLER K-R , et al . Robust and communication-efficient federated learning from non-i.i.d.data [J ] . IEEE Transactions on Neural Networks and Learning Systems , 2020 , 31 ( 9 ): 3400 - 3413 .
LIAO F Z , LIANG M , DONG Y P , et al . Defense against adversarial attacks using high-level representation guided denoiser [C ] // Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2018 : 1778 - 1787 .
XU W L , EVANS D , QI Y J . Feature squeezing:detecting adversarial examples in deep neural networks [J ] . arXiv preprint,2017,arXiv:1704.01155 .
ZHU C , CHENG Y , GAN Z , et al . FreeLB:enhanced adversarial training for language understanding [J ] . arXiv preprint,2019,arXiv:1909.11764 .
SHAH D , DUBE P , CHAKRABORTY S , et al . Adversarial training in communication constrained federated learning [J ] . arXiv preprint,2021,arXiv:2103.01319 .
FUNG C , YOON C J M , BESCHASTNIKH I . Mitigating sybils in federated learning poisoning [J ] . arXiv preprint,2018,arXiv:1808.04866 .
王健宗 , 孔令炜 , 黄章成 , 等 . 联邦学习隐私保护研究进展 [J ] . 大数据 , 2021 , 7 ( 3 ): 130 - 149 .
WANG J Z , KONG L W , HUANG Z C , et al . Research advances on privacy protection of federated learning [J ] . Big Data Research , 2021 , 7 ( 3 ): 130 - 149 .
ANDREINA S , MARSON G A , MÖLLERING H , et al . BaFFLe:backdoor detection via feedback-based federated learning [C ] // Proceedings of 2021 IEEE 41st International Conference on Distributed Computing Systems . Piscataway:IEEE Press , 2021 : 852 - 863 .
MCMAHAN H B , MOORE E , RAMAGE D , et al . Communication-efficient learning of deep networks from decentralized data [J ] . arXiv preprint,2016,arXiv:1602.05629 .
YIN D , CHEN Y D , RAMCHANDRAN K , et al . Byzantine-robust distributed learning:towards optimal statistical rates [J ] . arXiv preprint,2018,arXiv:1803.01498 .
BLANCHARD P , MHAMDI E M E , GUERRAOUI R , et al . Machine learning with adversaries:Byzantine tolerant gradient descent [C ] // Proceedings of the 31st International Conference on Neural Information Processing Systems.Red Hook:Curran Associates Inc . , 2017 : 118 - 128 .
MHAMDI E M E , GUERRAOUI R , ROUAULT S . The hidden vulnerability of distributed learning in Byzantium [J ] . arXiv preprint,2018,arXiv:1802.07927 .
SO J , GÜLER B , AVESTIMEHR A S . Turbo-aggregate:breaking the quadratic aggregation barrier in secure federated learning [J ] . IEEE Journal on Selected Areas in Information Theory , 2021 , 2 ( 1 ): 479 - 489 .
LEE H , KIM J , AHN S , et al . Digestive neural networks:a novel defense strategy against inference attacks in federated learning [J ] . Computers & Security , 2021 , 109 : 102378 .
周传鑫 , 孙奕 , 汪德刚 , 等 . 联邦学习研究综述 [J ] . 网络与信息安全学报 , 2021 , 7 ( 5 ): 77 - 92 .
ZHOU C X , SUN Y , WANG D G , et al . Survey of federated learning research [J ] . Chinese Journal of Network and Information Security , 2021 , 7 ( 5 ): 77 - 92 .
QUOC D L , FETZER C . SecFL:confidential federated learning using TEEs [J ] . arXiv preprint,2021,arXiv:2110.00981 .
LI W H , XIA Y B , LU L , et al . TEEv:virtualizing trusted execution environments on mobile platforms [C ] // Proceedings of the 15th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments . New York:ACM Press , 2019 : 2 - 16 .
CHEN Y , LUO F , LI T , et al . A trainingintegrity privacy-preserving federated learning scheme with trusted execution environment [J ] . Information Sciences , 2020 , 522 : 69 - 79 .
ZHAO Y , ZHAO J , JIANG L S , et al . Mobile edge computing,blockchain and reputation-based crowdsourcing IoT federated learning:a secure,decentralized and privacy-preserving system [J ] . arXiv preprint,2019,arXiv:1906.10893 .
DOSHI-VELEZ F , KIM B . Towards a rigorous science of interpretable machine learning [J ] . arXiv preprint,2017,arXiv:1702.08608 .
0
浏览量
2256
下载量
0
CSCD
关联资源
相关文章
相关作者
相关机构
京公网安备11010802024621