- All Journals
- Al-Qadisiyah University
- Basrah University
- Kirkuk University
- Mosul University
- The Iraqi Borad for Medical Specialization
- University of Anbar
- University of Technology
- Al Mustafa University
- Al-Zahraa University for Women
- Alnoor University
- Kunooz University
- Misan University
- Ninevah University
- Tikrit University
- University of Al-Hamdaniya
- University of Kerbala
Outlier Detection on High-Dimensional Data | ||
| Journal of University of Anbar for Pure Science | ||
| Volume 18, Issue 1, June 2024, Pages 223-233 PDF (554.2 K) | ||
| Document Type: Research Paper | ||
| DOI: 10.37652/juaps.2024.146609.1191 | ||
| Authors | ||
| wasan abdalmajeed shaban* ; Murtadha M. Hamad | ||
| Department of Computer Science, College of Computer Science and Information Technology, University of Anbar, Ramadi, Iraq | ||
| Abstract | ||
| In the domain of information examination, high-layered datasets present exceptional difficulties that request specific methodologies for anomaly recognition. This paper gives a succinct outline of exception discovery in high-layered information, tending to the related difficulties and introducing a scope of procedures to really handle them." This reviles prompts expanded computational intricacy, information sparsity, and challenges in representation and translation. To battle these issues, specific preprocessing methods are fundamental, including taking care of missing information, standardization, and normalization. Machine learning algorithms assume an essential part in exception identification. This paper offers experiences into the hypothetical underpinnings of AI calculations pertinent to this errand. The paper investigates measurable based, profundity based, deviation-based, distance-based, and thickness-based strategies, revealing insight into their applications and benefits. In synopsis, this paper gives an engaged investigation of exception location in high-layered information, addressing difficulties connected with the scourge of dimensionality. With particular preprocessing procedures and a dependence on AI and profound learning calculations, the paper explores through different strategies, featuring their applications. Python arises as the favored language for its strong environment. This consolidated aide highlights the need of custom fitted ways to deal with infer significant bits of knowledge, offering an exhaustive outline with an educational substance rate going from 85% to a roughly 96%. | ||
| Keywords | ||
| Outlier detection; High-dimensional data; Curse of dimensionality; Data preprocessing; Machine learning algorithms | ||
| References | ||
|
[1] S. Aliesawi, C. Tsimenidis, B. S. Sharif, and M. Johnston, Performance comparison of IDMA receivers for underwater acoustic channels. 2010. doi: 10.1109/ISWCS.2010.5624402.
[2] K. Maharana, S. Mondal, and B. Nemade, “A review: Data pre-processing and data augmentation techniques,” Glob. Transitions Proc., vol. 3, no. 1, pp. 91–99, 2022, doi: https://doi.org/10.1016/j.gltp.2022.04.020.
[3] K. Maharana, S. Mondal, and B. Nemade, “A review: Data pre-processing and data augmentation techniques,” Glob. Transitions Proc., vol. 3, no. 1, pp. 91–99, 2022, doi: https://doi.org/10.1016/j.gltp.2022.04.020.
[4] H. Palo, S. Sahoo, and A. Subudhi, “Dimensionality Reduction Techniques: Principles, Benefits, and Limitations,” 2021, pp. 77–107. doi: 10.1002/9781119785620.ch4.
[5] U. Sivarajah, M. M. Kamal, Z. Irani, and V. Weerakkody, “Critical analysis of Big Data challenges and analytical methods,” J. Bus. Res., vol. 70, pp. 263–286, 2017, doi: https://doi.org/10.1016/j.jbusres.2016.08.001.
[6] T. Dasu and T. Johnson, Exploratory Data Mining and Data Cleaning, vol. 101. 2003. doi: 10.1002/0471448354.
[7] C. C. Aggarwal and P. S. Yu, “Outlier detection for high dimensional data,” SIGMOD Rec. (ACM Spec. Interes. Gr. Manag. Data), vol. 30, no. 2, pp. 37–46, 2001, doi: 10.1145/376284.375668.
[8] Y. K. Dwivedi et al., “Opinion Paper: ‘So what if ChatGPT wrote it?’ Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy,” Int. J. Inf. Manage., vol. 71, p. 102642, 2023, doi: https://doi.org/10.1016/j.ijinfomgt.2023.102642.
[9] D. Kaur and D. Singh, “Critical Data Consolidation in MDM to Develop the Unified Version of Truth,” Int. J. Adv. Comput. Sci. Appl., vol. 12, Jan. 2021, doi: 10.14569/IJACSA.2021.0121242.
[10] S. Ali et al., “Explainable Artificial Intelligence (XAI): What we know and what is left to attain Trustworthy Artificial Intelligence,” Inf. Fusion, vol. 99, p. 101805, 2023, doi: https://doi.org/10.1016/j.inffus.2023.101805.
[11] D. Cousineau and S. Chartier, “Outliers detection and treatment: A review,” Int. J. Psychol. Res., vol. 3, Jun. 2010, doi: 10.21500/20112084.844.
[12] C. H. Park, “A Comparative Study for Outlier Detection Methods in High Dimensional Text Data,” J. Artif. Intell. Soft Comput. Res., vol. 13, no. 1, pp. 5–17, 2023, doi: 10.2478/jaiscr-2023-0001.
[13] B. Angelin and A. Geetha, Outlier Detection using Clustering Techniques – K-means and K-median. 2020. doi: 10.1109/ICICCS48265.2020.9120990.
[14] G. Orair, C. Teixeira, Y. Wang, W. Meira Jr, and S. Parthasarathy, “Distance-Based Outlier Detection: Consolidation and Renewed Bearing,” PVLDB, vol. 3, pp. 1469–1480, Oct. 2010.
[15] M. Re and G. Valentini, “Ensemble methods: A review,” in Advances in Machine Learning and Data Mining for Astronomy, 2012, pp. 563–594.
[16] X. Zhang, J. Mu, X. Zhang, H. Liu, L. Zong, and Y. Li, “Deep anomaly detection with self-supervised learning and adversarial training,” Pattern Recognit., vol. 121, p. 108234, 2022, doi: https://doi.org/10.1016/j.patcog.2021.108234.
[17] T. Doctoral, D. Series, and C. Science, OUTLIER DETECTION ANALYSIS Data Mining Approaches for Outlier Detection Analysis Shahrooz Abghari. 2020.
[18] T. Falahi, G. Nasserddine, and J. Younis, “Detecting Data Outliers with Machine Learning,” Al-Salam J. Eng. Technol., vol. 2, May 2023, doi: 10.55145/ajest.2023.02.02.018.
[19] E. Calikus, S. Nowaczyk, M. R. Bouguelia, and O. Dikmen, Wisdom of the contexts: active ensemble learning for contextual anomaly detection, vol. 36, no. 6. Springer US, 2022. doi: 10.1007/s10618-022-00868-7.
[20] E. Calikus, Together We Learn More : Algorithms and Applications for User-Centric Anomaly Detection Together We Learn More : Algorithms and Applications for User-Centric Anomaly Detection, no. 89. 2022.
[21] A. Boukerche, L. Zheng, and O. Alfandi, “Outlier Detection: Methods, Models, and Classification,” ACM Comput. Surv., vol. 53, pp. 1–37, Jun. 2020, doi: 10.1145/3381028.
[22] Z. Zhou, G. Si, H. Sun, K. Qu, and W. Hou, “A robust clustering algorithm based on the identification of core points and KNN kernel density estimation,” Expert Syst. Appl., vol. 195, p. 116573, 2022, doi: https://doi.org/10.1016/j.eswa.2022.116573.
[23] S. Naeem, A. Ali, S. Anam, and M. Ahmed, “An Unsupervised Machine Learning Algorithms: Comprehensive Review,” IJCDS J., vol. 13, pp. 911–921, Apr. 2023, doi: 10.12785/ijcds/130172.
[24] Z. Zhao, “Ensemble Methods for Anomaly Detection,” no. December, 2017, [Online]. Available: https://surface.syr.edu/etd/817
[25] L. H. Chiang, R. J. Pell, and M. B. Seasholtz, “Exploring process data with the use of robust outlier detection algorithms,” J. Process Control, vol. 13, no. 5, pp. 437–449, 2003, doi: https://doi.org/10.1016/S0959-1524(02)00068-9.
[26] R. Purohit, J. P. Verma, R. Jain, and M. Bhavsar, “WePaMaDM-Outlier Detection: Weighted Outlier Detection using Pattern Approaches for Mass Data Mining,” 2023 Int. Conf. Adv. Comput. Comput. Technol. InCACCT 2023, pp. 243–248, 2023, doi: 10.1109/InCACCT57535.2023.10141778.
[27] A. Boukerche, L. Zheng, and O. Alfandi, “Outlier Detection: Methods, Models, and Classification,” ACM Comput. Surv., vol. 53, pp. 1–37, Jun. 2020, doi: 10.1145/3381028.
[28] O. Alghushairy, R. Alsini, T. Soule, and X. Ma, “A review of local outlier factor algorithms for outlier detection in big data streams,” Big Data Cogn. Comput., vol. 5, no. 1, pp. 1–24, 2021, doi: 10.3390/bdcc5010001.
[29] H. Ali Mohammadi and S. Chen, “Performance Evaluation of Outlier Detection Techniques in Production Timeseries: A Systematic Review and Meta-Analysis,” Expert Syst. Appl., vol. 191, p. 116371, Dec. 2021, doi: 10.1016/j.eswa.2021.116371.
[30] M. B. Al- Zoubi, “An effective clustering-based approach for outlier detection,” Eur. J. Sci. Res., vol. 28, pp. 310–316, Jan. 2009.
[31] M. Hahsler, M. Piekenbrock, and D. Doran, “Dbscan: Fast density-based clustering with R,” J. Stat. Softw., vol. 91, no. 1990, 2019, doi: 10.18637/jss.v091.i01.
[32] Z. He, X. Xu, and S. Deng, “Discovering cluster-based local outliers,” Pattern Recognit. Lett., vol. 24, no. 9, pp. 1641–1650, 2003, doi: https://doi.org/10.1016/S0167-8655(03)00003-5.
[33] M. Landauer, S. Onder, F. Skopik, and M. Wurzenberger, “Deep learning for anomaly detection in log data: A survey,” Mach. Learn. with Appl., vol. 12, p. 100470, 2023, doi: https://doi.org/10.1016/j.mlwa.2023.100470.
[34] J. Saeedi and A. Giusti, “Semi-supervised visual anomaly detection based on convolutional autoencoder and transfer learning,” Mach. Learn. with Appl., vol. 11, p. 100451, 2023, doi: https://doi.org/10.1016/j.mlwa.2023.100451.
[35] Ahmed Subhi Abdalkafor, Salah A. Aliesawi., Data aggregation techniques in wireless sensors networks (WSNs): Taxonomy and an accurate literature survey. AIP Conference Proceedings 2400, 020012 (2022); 2022.
[36] M. Landauer, S. Onder, F. Skopik, and M. Wurzenberger, “Deep learning for anomaly detection in log data: A survey,” Mach. Learn. with Appl., vol. 12, p. 100470, 2023, doi: https://doi.org/10.1016/j.mlwa.2023.100470.
[37] B. Dastjerdy, A. Saeidi, and S. Heidarzadeh, “Review of Applicable Outlier Detection Methods to Treat Geomechanical Data,” Geotechnics, vol. 3, no. 2, pp. 375–396, 2023, doi: 10.3390/geotechnics3020022.
[38] C. H. Sim, F. F. Gan, and T. C. Chang, “Outlier Labeling With Boxplot Procedures,” J. Am. Stat. Assoc., vol. 100, pp. 642–652, Feb. 2005, doi: 10.1198/016214504000001466.
[39] X. Dang and R. Serfling, “Nonparametric depth-based multivariate outlier identifiers, and masking robustness properties,” J. Stat. Plan. Inference, vol. 140, no. 1, pp. 198–213, 2010, doi: https://doi.org/10.1016/j.jspi.2009.07.004.
[40] J. Huang, D. Cheng, and S. Zhang, “A novel outlier detecting algorithm based on the outlier turning points,” Expert Syst. Appl., vol. 231, p. 120799, 2023, doi: https://doi.org/10.1016/j.eswa.2023.120799.
[41] J. Huang, D. Cheng, and S. Zhang, “A novel outlier detecting algorithm based on the outlier turning points,” Expert Syst. Appl., vol. 231, p. 120799, 2023, doi: https://doi.org/10.1016/j.eswa.2023.120799.
[42] F. Yoseph, M. Heikkila, and D. Howard, “Outliers identification model in point-of-sales data using enhanced normal distribution method,” Proc. - Int. Conf. Mach. Learn. Data Eng. iCMLDE 2019, pp. 72–78, 2019, doi: 10.1109/iCMLDE49015.2019.00024.
[43] W. P. Bensken, F. M. Pieracci, and V. P. Ho, “Basic Introduction to Statistics in Medicine, Part 1: Describing Data,” Surg. Infect. (Larchmt)., vol. 22, no. 6, pp. 590–596, 2021, doi: 10.1089/sur.2020.429.
[44] R. Dyckerhoff and P. Mozharovskyi, “Exact computation of the halfspace depth,” Comput. Stat. Data Anal., vol. 98, pp. 19–30, 2016, doi: 10.1016/j.csda.2015.12.011.
[45] E. Blanco-Mallo, L. Morán-Fernández, B. Remeseiro, and V. Bolón-Canedo, “Do all roads lead to Rome? Studying distance measures in the context of machine learning,” Pattern Recognit., vol. 141, p. 109646, 2023, doi: https://doi.org/10.1016/j.patcog.2023.109646.
[46] M. Pachgade and S. Dhande, “Outlier Detection over Data Set Using Cluster-Based and Distance-Based Approach,” IJARCSSE, vol. 2, pp. 1–5, Jun. 2012.
[47] A. Lotfi, P. Moradi, and H. Beigy, “Density peaks clustering based on density backbone and fuzzy neighborhood,” Pattern Recognit., vol. 107, p. 107449, 2020, doi: https://doi.org/10.1016/j.patcog.2020.107449.
[48] R. Bhuyan and S. Borah, “A survey of some density based clustering techniques,” arXiv Prepr. arXiv2306.09256, vol. 1, 2023, doi: 10.13140/2.1.4554.6887.
[49] Y. Zhu, K. M. Ting, Y. Jin, and M. Angelova, “Hierarchical clustering that takes advantage of both density-peak and density-connectivity,” Inf. Syst., vol. 103, p. 101871, 2022, doi: https://doi.org/10.1016/j.is.2021.101871.
[50] A. D. Outlier and I. Using, “Datasets for Validation of Stroke Clinical Outcomes,” pp. 1–23, 2020, doi: 10.1016/j.ijmedinf.2019.103988.Applying.
[51] A. Guzmán-Ponce, J. S. Sánchez, R. M. Valdovinos, and J. R. Marcial-Romero, “DBIG-US: A two-stage under-sampling algorithm to face the class imbalance problem,” Expert Syst. Appl., vol. 168, p. 114301, 2021, doi: https://doi.org/10.1016/j.eswa.2020.114301.
[52] A. Guzmán-Ponce, J. S. Sánchez, R. M. Valdovinos, and J. R. Marcial-Romero, “DBIG-US: A two-stage under-sampling algorithm to face the class imbalance problem,” Expert Syst. Appl., vol. 168, p. 114301, 2021, doi: https://doi.org/10.1016/j.eswa.2020.114301.
[53] E. Debie and K. Shafi, “Implications of the curse of dimensionality for supervised learning classifier systems: theoretical and empirical analyses,” Pattern Anal. Appl., vol. 22, no. 2, pp. 519–536, 2019, doi: 10.1007/s10044-017-0649-0.
[54] P. Hajibabaee, F. Pourkamali-Anaraki, and M. A. Hariri-Ardebili, “Dimensionality reduction techniques in structural and earthquake engineering,” Eng. Struct., vol. 278, p. 115485, 2023, doi: https://doi.org/10.1016/j.engstruct.2022.115485.
[55] Salah A. Aliesawi, Dena S. Alani, Abdullah M. Awad, " Secure image transmission over wireless network", International Journal of Engineering and Technology (UAE), Science Publishing Corporation Inc., Qatar, 7 (4) 2758-2764, 2018. | ||
|
Statistics Article View: 715 PDF Download: 317 |
||