Research on iterative data cleaning of human-computer interaction

Yida LIU; Xiaoou DING; Hongzhi WANG; Donghua YANG

doi:10.11959/j.issn.2096-0271.2023048

您当前的位置：

首页 >

文章列表页 >

Research on iterative data cleaning of human-computer interaction

TOPIC: CROSS-DOMAIN DATA MANAGEMENT | 更新时间：2024-06-03

- Research on iterative data cleaning of human-computer interaction
- Big data research Vol. 9, Issue 4, Pages: 59-68(2023)
- 作者机构：
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-0271.2023048
  CLC： TP311
- Online First：2023-07，
  
  Published：15 July 2023
- 稿件说明：
移动端阅览
Yida LIU, Xiaoou DING, Hongzhi WANG, et al. Research on iterative data cleaning of human-computer interaction[J]. Big data research, 2023, 9(4): 59-68.
DOI：

Yida LIU, Xiaoou DING, Hongzhi WANG, et al. Research on iterative data cleaning of human-computer interaction[J]. Big data research, 2023, 9(4): 59-68. DOI： 10.11959/j.issn.2096-0271.2023048.

摘要

数据采集技术的进步导致了数据集规模的飞速上涨，由于数据的大规模和高复杂性引起了严重的数据质量问题，数据清洗是数据活动中必要且重要的环节。为了在保证清洗准确率的情况下有效地降低人工标注成本，提出了一种人工参与的迭代式的数据清洗方法（IDCHI）。该方法在检测模块中提出了数据选择优化方法，使分类器在初始阶段就拥有较高的准确度；并进一步提出了待人工标注数据选择方法，有效地降低人工标注的数据量。实验结果表明该方法可有效且高效地清洗错误数据。

Abstract

The advancement of data collection technology has led to a rapid increase in the size of datasets.Due to the big scale and high complexity of the data volume

serious data quality issues arise.Therefore

data cleaning is a necessary and important step in data activities.To effectively reduce human annotation costs while ensuring the accuracy of cleaning

an iterative data cleaning method (IDCHI) with human participation was proposed.This method proposed a data selection optimization method in the detection module

which enables the classifier to have high accuracy in the initial stage; and further proposed a method for selecting data to be manually annotated

effectively reducing the amount of data to be manually annotated.The experimental results show that the proposed method is effective and efficient in cleaning erroneous data.

关键词

Keywords

references

LIANG Z , WANG H Z , DING X O , et al . Industrial time series determinative anomaly detection based on constraint hypergraph [J ] . Knowledge-Based Systems , 2021 ,233:107548.

BERGMAN M , MILO T , NOVGORODOV S , et al . Query-oriented data cleaning with oracles [C ] // Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data . New York:ACM Press , 2015 : 1199 - 1214 .

SIDDIQUI T , KIM A , LEE J , et al . Effortless data exploration with zenvisage:an expressive and interactive visual analytics system [J ] . arXiv preprint , 2016 ,arXiv:1604.03583.

SHOU C L , SHUKLA A . Arachnid:generalized visual data cleaning [C ] // Proceedings of the 2019 International Conference on Management of Data . New York:ACM Press , 2019 : 1850 - 1852 .

范举 , 陈跃国 , 杜小勇 . 人在回路的数据准备技术研究进展 [J ] . 大数据 , 2019 , 5 ( 6 ): 1 - 18 .

FAN J , CHEN Y G , DU X Y . Progress on human-in-the-loop data preparation [J ] . Big Data Research , 2019 , 5 ( 6 ): 1 - 18 .

QIN X D , LUO Y Y , TANG N , et al . Making data visualization more efficient and effective:a survey [J ] . The VLDB Journal , 2020 , 29 ( 1 ): 93 - 117 .

DING X O , LIU Y D , WANG H Z , et al . SNN-AAD:active anomaly detection method for multivariate time series with sparse neural network [C ] // International Conference on Database Systems for Advanced Applications . Cham:Springer , 2023 : 253 - 269 .

SATYANARAYAN A , MORITZ D , WONGSUPHASAWAT K , et al . Vegalite:a grammar of interactive graphics [J ] . IEEE Transactions on Visualization and Computer Graphics , 2017 , 23 ( 1 ): 341 - 350 .

HEER J , AGRAWALA M , WILLETT W . Generalized selection via interactive query relaxation [C ] // Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . New York:ACM Press , 2008 : 959 - 968 .

ABEDJAN Z , CHU X , DENG D , et al . Detecting data errors [J ] . Proceedings of the VLDB Endowment , 2016 , 9 ( 12 ): 9931004 .

HANRAHAN P . VizQL:a language for query,analysis and visualization [C ] // Proceedings of the 2006 ACM SIGMOD International Conference on Management of Data . New York:ACM Press , 2006 :721.

DING X O , SONG Y C , WANG H Z , et al . Cleanits-MEDetect:multiple errors detection for time series data in cleanits [C ] // International Conference on Database Systems for Advanced Applications . Cham:Springer , 2023 : 674 - 678 .

KRISHNAN S , WANG J N , WU E , et al . ActiveClean [J ] . Proceedings of the VLDB Endowment , 2016 , 9 ( 12 ): 948 - 959 .

BEYGELZIMER A , DASGUPTA S , LANGFORD J . Importance weighted active learning [C ] // Proceedings of the 26th Annual International Conference on Machine Learning . New York:ACM Press , 2009 : 49 - 56 .

LE K H , PAPOTTI P . User-driven error detection for time series with events [C ] // Proceedings of 2020 IEEE 36th International Conference on Data Engineering (ICDE) . Piscataway:IEEE Press , 2020 : 745 - 757 .

YAKOUT M , ELMAGARMID A K , NEVILLE J , et al . Guided data repair [J ] . arXiv preprint . 2011 ,arXiv:1103.3103.

BOSTOCK M , OGIEVETSKY V , HEER J . D3 data-driven documents [J ] . IEEE Transactions on Visualization and Computer Graphics , 2011 , 17 ( 12 ): 2301 - 2309 .

XIAO H , BIGGIO B , BROWN G , et al . Is feature selection secure against training data poisoning? [C ] // Proceedings of the 32nd International Conference on International Conference on Machine Learning . New York:ACM Press , 2015 : 1689 - 1698 .

CHARFI M , GRIPAY Y , PETIT J M . Spatio-temporal functional dependencies for sensor data streams [C ] // International Symposium on Spatial and Temporal Databases . Cham:Springer , 2017 : 182 - 199 .

LUO Y Y , CHAI C L , QIN X D , et al . VisClean [J ] . Proceedings of the VLDB Endowment , 2020 , 13 ( 12 ): 2821 - 2824 .

Views

454

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Data curation technologies and applications

Data quality management of industrial temporal big data

Related Author

Minghe YU

Tiezheng NIE

Guoliang LI

Hongzhi WANG

Shengjian YU

Related Institution

Software College of Northeastern University

Guangdong Province Key Laboratory of Popular High Performance Computers

School of Computer Science and Engineering, Northeastern University

Department of Computer Science and Technology, Tsinghua University

Postal code：100079
Tel：（010）53879208、53859533 Email：bdr@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09064830号-19 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.
Total Visits：273114 Visits Today：20

⁰