Effective Human–AI Collaborative Intelligence

doi:10.1017/9781009587877.009

8 - Effective Human–AI Collaborative Intelligence

Published online by Cambridge University Press: 19 September 2025

Zhuoren Jiang and

Xiaozhong Liu

Edited by

Dan Wu and

Shaobo Liang

Show author details

Dan Wu: Affiliation:
Wuhan University, China
Shaobo Liang: Affiliation:
Wuhan University, China

Book contents

Get access

Summary

In today’s data-driven world, the demand for advanced intelligent systems to automate and enhance complex tasks is growing. However, developing effective artificial intelligence (AI) often depends on extensive, high-quality training data, which can be costly and time-consuming to obtain. This chapter highlights the potential of human–AI collaboration by integrating human expertise into machine learning workflows to address data limitations and enhance model performance. We explore foundational concepts such as Human-in-the-Loop systems, Active Learning, Crowdsourcing, and Interactive Machine Learning, outlining their interconnections as key paradigms. Through practical applications in diverse domains such as healthcare, finance, and agriculture, along with real-world case studies in education and law, we demonstrate how strategically incorporating human expertise into machine learning workflows can significantly enhance AI performance. From an information science perspective, this chapter emphasizes the powerful human–AI partnership that can drive the next generation of AI systems, enabling continuous learning from human experts and advancing capability and performance.

Keywords

Human-in-the-Loop Active Learning Crowdsourcing Interactive Machine Learning Human–AI Collaborative Intelligence

Information

Type: Chapter
Information: Human-AI Interaction and Collaboration , pp. 175 - 212

DOI: https://doi.org/10.1017/9781009587877.009 [Opens in a new window]

Publisher: Cambridge University Press

Print publication year: 2025

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Book purchase

Temporarily unavailable

References

Aldoseri, A., Al-Khalifa, K. N., & Hamouda, A. M. (2023). Re-thinking Data Strategy and Integration for Artificial Intelligence: Concepts, Opportunities, and Challenges. Applied Sciences, 13(12), 7082.10.3390/app13127082CrossRef Google Scholar

Arous, I., Dolamic, L., Yang, J., Bhardwaj, A., Cuccu, G., & Cudré-Mauroux, P. (2021). Marta: Leveraging Human Rationales for Explainable Text Classification. Proceedings of the AAAI Conference on Artificial Intelligence, 35(7), 5868–5876.10.1609/aaai.v35i7.16734CrossRef Google Scholar

Baldridge, J., & Palmer, A. (2009). How Well Does Active Learning Actually Work? Time-based Evaluation of Cost-reduction Strategies for Language Documentation. Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, 296–305.Google Scholar

Bartolo, M., Roberts, A., Welbl, J., Riedel, S., & Stenetorp, P. (2020). Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension. Transactions of the Association for Computational Linguistics, 8, 662–678.10.1162/tacl_a_00338CrossRef Google Scholar

Boukhelifa, N., Bezerianos, A., & Lutton, E. (2018). Evaluation of Interactive Machine Learning Systems. In Zhou, J. & Chen, F. (eds.), Human and Machine Learning (pp. 341–360). Springer International Publishing.10.1007/978-3-319-90403-0_17CrossRef Google Scholar

Brabham, D. C. (2008). Crowdsourcing as a Model for Problem Solving: An Introduction and Cases. Convergence: The International Journal of Research into New Media Technologies, 14(1), 75–90.10.1177/1354856507084420CrossRef Google Scholar

Budd, S., Robinson, E. C., & Kainz, B. (2021). A Survey on Active Learning and Human-in-the-Loop Deep Learning for Medical Image Analysis. Medical Image Analysis, 71, 102062.10.1016/j.media.2021.102062CrossRef Google Scholar PubMed

Burr, S. (2009). Active Learning Literature Survey. University of Wisconsin-Madison Department of Computer Sciences.Google Scholar

Casalini, F., González, J. L., & Nemoto, T. (2021). Mapping Commonalities in Regulatory Approaches to Cross-border Data Transfers. OECD Trade Policy Papers.Google Scholar

Chai, C., & Li, G. (2020). Human-in-the-Loop Techniques in Machine Learning. IEEE Data Engineering Bulletin, 43(3), 37–52.Google Scholar

Chandra, A. L., Desai, S. V., Balasubramanian, V. N., Ninomiya, S., & Guo, W. (2020). Active Learning with Point Supervision for Cost-effective Panicle Detection in Cereal Crops. Plant Methods, 16(1), 34. https://doi.org/10.1186/s13007-020-00575-8 CrossRef Google Scholar PubMed

Chang, J., Gerrish, S., Wang, C., Boyd-Graber, J., & Blei, D. (2009). Reading Tea Leaves: How Humans Interpret Topic Models. Advances in Neural Information Processing Systems, 22, 288–296.Google Scholar

Chaudhuri, K., Kakade, S. M., Netrapalli, P., & Sanghavi, S. (2015). Convergence Rates of Active Learning for Maximum Likelihood Estimation. Advances in Neural Information Processing Systems, 28, 1090–1098.Google Scholar

Daniel, F., Kucherbaev, P., Cappiello, C., Benatallah, B., & Allahbakhsh, M. (2019). Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques, and Assurance Actions. ACM Computing Surveys, 51(1), 1–40.10.1145/3148148CrossRef Google Scholar

De Angeli, K., Gao, S., Alawad, M., Yoon, H.-J., Schaefferkoetter, N., Wu, X.-C., Durbin, E. B., Doherty, J., Stroup, A., Coyle, L., Penberthy, L., & Tourassi, G. (2021). Deep Active Learning for Classifying Cancer Pathology Reports. BMC Bioinformatics, 22(1), 113. https://doi.org/10.1186/s12859-021-04047-1 CrossRef Google Scholar PubMed

De Melo, F. R., Flôres, E. L., De Carvalho, S. D., De Teixeira, R. A. G., Loja, L. F. B., & de Sousa Gomide, R. (2014). Computational Organization of Didactic Contents for Personalized Virtual Learning Environments. Computers & Education, 79, 126–137.10.1016/j.compedu.2014.07.012CrossRef Google Scholar

Demartini, G., Mizzaro, S., & Spina, D. (2020). Human-in-the-Loop Artificial Intelligence for Fighting Online Misinformation: Challenges and Opportunities. IEEE Data Engineering Bulletin, 43(3), 65–74.Google Scholar

Deng, D., Shahabi, C., & Demiryurek, U. (2013). Maximizing the Number of Worker’s Self-selected Tasks in Spatial Crowdsourcing. Proceedings of the 21st ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, 324–333.10.1145/2525314.2525370CrossRef Google Scholar

Diligenti, M., Roychowdhury, S., & Gori, M. (2017). Integrating Prior Knowledge into Deep Learning. 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), 920–923.10.1109/ICMLA.2017.00-37CrossRef Google Scholar

Dimensional Research. (2019). What Data Scientists Tell Us About AI Model Training Today. Alegion. https://cdn2.hubspot.net/hubfs/3971219/Content%20Offers/Drafts/Alegion_SurveyNarrative.pdf Google Scholar

Dong, J., Kang, Y., Liu, J., Sun, C., Fan, S., Jin, H., Wu, D., Jiang, Z., Niu, X., & Liu, X. (2023). Human-centred Design on Crowdsourcing Annotation towards Improving Active Learning Model Performance. Journal of Information Science, doi: 10.1177/01655515231204802CrossRef Google Scholar

Donmez, P., & Carbonell, J. G. (2008). Proactive Learning: Cost-sensitive Active Learning with Multiple Imperfect Oracles. Proceedings of the 17th ACM Conference on Information and Knowledge Management, 619–628.10.1145/1458082.1458165CrossRef Google Scholar

Dor, L. E., Halfon, A., Gera, A., Shnarch, E., Dankin, L., Choshen, L., Danilevsky, M., Aharonov, R., Katz, Y., & Slonim, N. (2020). Active Learning for BERT: An Empirical Study. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 7949–7962.Google Scholar

Druck, G., Settles, B., & McCallum, A. (2009). Active Learning by Labeling Features. Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, 81–90.10.3115/1699510.1699522CrossRef Google Scholar

Dudley, J. J., & Kristensson, P. O. (2018). A Review of User Interface Design for Interactive Machine Learning. ACM Transactions on Interactive Intelligent Systems, 8(2), 1–37.10.1145/3185517CrossRef Google Scholar

Eapen, T., Finkenstadt, D. J., Folk, J., & Venkataswamy, L. (2023). How Generative AI Can Augment Human Creativity. Harvard Business Review, 101(4), 56–64.Google Scholar

El-Hasnony, I. M., Elzeki, O. M., Alshehri, A., & Salem, H. (2022). Multi-Label Active Learning-Based Machine Learning Model for Heart Disease Prediction. SENSORS, 22(3), 1184. https://doi.org/10.3390/s22031184 CrossRef Google Scholar PubMed

Fails, J. A., & Olsen, D. R. (2003). Interactive Machine Learning. Proceedings of the 8th International Conference on Intelligent User Interfaces, 39–45.10.1145/604045.604056CrossRef Google Scholar

Fan, X., Li, C., Yuan, X., Dong, X., & Liang, J. (2019). An Interactive Visual Analytics Approach for Network Anomaly Detection through Smart Labeling. Journal of Visualization, 22(5), 955–971.10.1007/s12650-019-00580-7CrossRef Google Scholar

Fiebrink, R., Cook, P. R., & Trueman, D. (2011). Human Model Evaluation in Interactive Supervised Learning. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 147–156.10.1145/1978942.1978965CrossRef Google Scholar

Folmsbee, J., Liu, X., Brandwein-Weber, M., & Doyle, S. (2018). Active Deep Learning: Improved Training Efficiency of Convolutional Neural Networks for Tissue Classification in Oral Cavity Cancer. 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), 770–773. https://webofscience.clarivate.cn/wos/alldb/summary/8b3794b9-2bc2-4ba9-bb93–84b37012273a-0108175925/relevance/110.1109/ISBI.2018.8363686CrossRef Google Scholar

Forrester Consulting. (2020). Overcome Obstacles to Get to AI at Scale. IBM. www.ibm.com/downloads/cas/VBMPEQLN Google Scholar

Fredrikson, M., Jha, S., & Ristenpart, T. (2015). Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures. Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, 1322–1333.10.1145/2810103.2813677CrossRef Google Scholar

Gomes, R., Welinder, P., Krause, A., & Perona, P. (2011). Crowdclustering. Advances in Neural Information Processing Systems 24 (NIPS 2011).Google Scholar

Gronsund, T., & Aanestad, M. (2020). Augmenting the Algorithm: Emerging Human-in-the-Loop Work Configurations. Journal of Strategic Information Systems, 29(2), 101614. https://doi.org/10.1016/j.jsis.2020.101614 CrossRef Google Scholar

Gualo, F., Rodríguez, M., Verdugo, J., Caballero, I., & Piattini, M. (2021). Data Quality Certification using ISO/IEC 25012: Industrial Experiences. Journal of Systems and Software, 176, 110938.10.1016/j.jss.2021.110938CrossRef Google Scholar

Guo, J., Kang, Y., Duan, Y., Liu, X., Tang, S., Zhang, W., Kuang, K., Sun, C., & Wu, F. (2022). Collaborative Intelligence Orchestration: Inconsistency-Based Fusion of Semi-Supervised Learning and Active Learning. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2935–2945.10.1145/3534678.3539022CrossRef Google Scholar

Guo, J., Shi, H., Kang, Y., Kuang, K., Tang, S., Jiang, Z., Sun, C., Wu, F., & Zhuang, Y. (2021). Semi-supervised Active Learning for Semi-supervised Models: Exploit Adversarial Examples with Graph-based Virtual Labels. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2896–2905.10.1109/ICCV48922.2021.00289CrossRef Google Scholar

He, L., Michael, J., Lewis, M., & Zettlemoyer, L. (2016). Human-in-the-Loop Parsing. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2337–2342.10.18653/v1/D16-1258CrossRef Google Scholar

Hoi, S. C. H., Jin, R., Zhu, J., & Lyu, M. R. (2006). Batch Mode Active Learning and Its Application to Medical Image Classification. Proceedings of the 23rd International Conference on Machine Learning: ICML’06, 417–424.10.1145/1143844.1143897CrossRef Google Scholar

Holzinger, A. (2016). Interactive Machine Learning for Health Informatics: When Do We Need the Human-in-the-Loop? Brain Informatics, 3(2), 119–131.10.1007/s40708-016-0042-6CrossRef Google Scholar PubMed

Howe, J. (2006). The Rise of Crowdsourcing. Wired Magazine, 14(6), 176–183.Google Scholar

Hu, H., Salcic, Z., Sun, L., Dobbie, G., Yu, P. S., & Zhang, X. (2022). Membership Inference Attacks on Machine Learning: A Survey. ACM Computing Surveys, 54(11s), 1–37.Google Scholar

Ipeirotis, P. G., Provost, F., & Wang, J. (2010). Quality Management on Amazon Mechanical Turk. Proceedings of the ACM SIGKDD Workshop on Human Computation, 64–67.10.1145/1837885.1837906CrossRef Google Scholar

Jiang, Z., Gao, L., Yuan, K., Gao, Z., Tang, Z., & Liu, X. (2018). Mathematics Content Understanding for Cyberlearning via Formula Evolution Map. Proceedings of the 27th ACM International Conference on Information and Knowledge Management, 37–46. https://doi.org/10.1145/3269206.3271694 CrossRef Google Scholar

Jiang, Z., Gao, Z., Duan, Y., Kang, Y., Sun, C., Zhang, Q., & Liu, X. (2020). Camouflaged Chinese Spam Content Detection with Semi-supervised Generative Active Learning. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 3080–3085.10.18653/v1/2020.acl-main.279CrossRef Google Scholar

Jiang, Z., Liu, X., Gao, L., & Tang, Z. (2016). Community-based Cyberreading for Information Understanding. Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, 789–792. https://dl.acm.org/doi/abs/10.1145/2911451.2914744 CrossRef Google Scholar

Joshi, A. J., Porikli, F., & Papanikolopoulos, N. (2009). Multi-class Active Learning for Image Classification. 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2372–2379.10.1109/CVPR.2009.5206627CrossRef Google Scholar

Jwo, J.-S., Lin, C.-S., & Lee, C.-H. (2021). Smart Technology-driven Aspects for Human-in-the-Loop Smart Manufacturing. The International Journal of Advanced Manufacturing Technology, 114(5–6), 1741–1752.10.1007/s00170-021-06977-9CrossRef Google Scholar

Kameda, T., Toyokawa, W., & Tindale, R. S. (2022). Information Aggregation and Collective Intelligence beyond the Wisdom of Crowds. Nature Reviews Psychology, 1(6), 345–357.10.1038/s44159-022-00054-yCrossRef Google Scholar

Kapoor, A., Horvitz, E., & Basu, S. (2007). Selective Supervision: Guiding Supervised Learning with Decision-theoretic Active Learning. IJCAI’07 Proceedings of the 20th International Joint Conference on Artificial Intelligence, 877–882.Google Scholar

Kee, S., Del Castillo, E., & Runger, G. (2018). Query-by-Committee Improvement with Diversity and Density in Batch Active Learning. Information Sciences, 454, 401–418.10.1016/j.ins.2018.05.014CrossRef Google Scholar

Kovashka, A., Parikh, D., & Grauman, K. (2015). WhittleSearch: Interactive Image Search with Relative Attribute Feedback. International Journal of Computer Vision, 115(2), 185–210. https://doi.org/10.1007/s11263-015-0814-0 CrossRef Google Scholar

Kreutzer, J., Riezler, S., & Lawrence, C. (2021). Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Tasks. Proceedings of the 5th Workshop on Structured Prediction for NLP (SPNLP 2021), 37–43.10.18653/v1/2021.spnlp-1.4CrossRef Google Scholar

Le, T.-N., Sugimoto, A., Ono, S., & Kawasaki, H. (2020). Toward Interactive Self-annotation for Video Object Bounding Box: Recurrent Self-learning and Hierarchical Annotation based Framework. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 3231–3240.10.1109/WACV45572.2020.9093398CrossRef Google Scholar

Lease, M. (2011). On Quality Control and Machine Learning in Crowdsourcing. Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence.Google Scholar

Lee, T. Y., Smith, A., Seppi, K., Elmqvist, N., Boyd-Graber, J., & Findlater, L. (2017). The Human Touch: How Non-expert Users Perceive, Interpret, and Fix Topic Models. International Journal of Human–Computer Studies, 105, 28–42.Google Scholar

Liang, W., Tadesse, G. A., Ho, D., Fei-Fei, L., Zaharia, M., Zhang, C., & Zou, J. (2022). Advances, Challenges and Opportunities in Creating Data for Trustworthy AI. Nature Machine Intelligence, 4(8), 669–677.10.1038/s42256-022-00516-1CrossRef Google Scholar

Liu, B., & Ferrari, V. (2017). Active Learning for Human Pose Estimation. Proceedings of the IEEE International Conference on Computer Vision, 4363–4372. http://openaccess.thecvf.com/content_iccv_2017/html/Liu_Active_Learning_for_ICCV_2017_paper.html10.1109/ICCV.2017.468CrossRef Google Scholar

Liu, C., Zhao, F., Kuang, K., Kang, Y., Jiang, Z., Sun, C., & Wu, F. (2024). Evolving Knowledge Distillation with Large Language Models and Active Learning. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 6717–6731. https://aclanthology.org/2024.lrec-main.593/Google Scholar

Liu, P., Wang, L., Ranjan, R., He, G., & Zhao, L. (2022). A Survey on Active Deep Learning: From Model Driven to Data Driven. ACM Computing Surveys, 54(10s), 1–34.Google Scholar

Liu, X., Jiang, Z., & Gao, L. (2015). Scientific Information Understanding via Open Educational Resources (OER). Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, 645–654. https://doi.org/10.1145/2766462.2767750 CrossRef Google Scholar

Liu, Z., Guo, Y., & Mahmud, J. (2021). When and Why a Model Fails? A Human-in-the-Loop Error Detection Framework for Sentiment Analysis. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers, 170–177.10.18653/v1/2021.naacl-industry.22CrossRef Google Scholar

Liu, Z., Wang, J., Gong, S., Lu, H., & Tao, D. (2019). Deep Reinforcement Active Learning for Human-in-the-Loop Person Re-identification. Proceedings of the IEEE/CVF International Conference on Computer Vision, 6122–6131.10.1109/ICCV.2019.00622CrossRef Google Scholar

Luo, T., Kramer, K., Goldgof, D. B., Hall, L. O., Samson, S., Remsen, A., Hopkins, T., & Cohn, D. (2005). Active Learning to Recognize Multiple Types of Plankton. Journal of Machine Learning Research, 6(4), 589–613.Google Scholar

Ma, Y., Liu, J., Yi, F., Cheng, Q., Huang, Y., Lu, W., & Liu, X. (2023). AI vs. Human: Differentiation Analysis of Scientific Content Generation [arXiv preprint]. arXiv:2301.10416. http://arxiv.org/abs/2301.10416 Google Scholar

Madnick, S. E., Wang, R. Y., Lee, Y. W., & Zhu, H. (2009). Overview and Framework for Data and Information Quality Research. Journal of Data and Information Quality, 1(1), 1–22.Google Scholar

Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022). A Survey on Bias and Fairness in Machine Learning. ACM Computing Surveys, 54(6), 1–35.10.1145/3457607CrossRef Google Scholar

Monarch, R. M. (2021). Human-in-the-Loop Machine Learning: Active Learning and Annotation for Human-centered AI. Simon and Schuster.Google Scholar

Mosqueira-Rey, E., Hernández-Pereira, E., Alonso-Ríos, D., Bobes-Bascarán, J., & Fernández-Leal, Á. (2023). Human-in-the-Loop Machine Learning: A State of the Art. Artificial Intelligence Review, 56(4), 3005–3054.10.1007/s10462-022-10246-wCrossRef Google Scholar

Nguyen, D. H., & Patrick, J. D. (2014). Supervised Machine Learning and Active Learning in Classification of Radiology Reports. Journal of the American Medical Informatics Association, 21(5), 893–901.10.1136/amiajnl-2013-002516CrossRef Google Scholar PubMed

Odekerken, D., & Bex, F. (2020). Towards Transparent Human-in-the-Loop Classification of Fraudulent Web Shops. In Villata, S., Harašta, J., & Křemen, P. (eds.), Frontiers in Artificial Intelligence and Applications (pp. 239–242). IOS Press. https://doi.org/10.3233/FAIA200873 Google Scholar

Panian, Z. (2010). Some Practical Experiences in Data Governance. World Academy of Science, Engineering and Technology, 62(1), 939–946.Google Scholar

Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running Experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 411–419.10.1017/S1930297500002205CrossRef Google Scholar

Patil, Y., Amer-Yahia, S., & Subramanian, S. (2022). Designing the Evaluation of Operator-enabled Interactive Data Exploration in VALIDE. Proceedings of the Workshop on Human-In-the-Loop Data Analytics, 1–7. https://doi.org/10.1145/3546930.3547509 CrossRef Google Scholar

Pipino, L. L., Lee, Y. W., & Wang, R. Y. (2002). Data Quality Assessment. Communications of the ACM, 45(4), 211–218.10.1145/505248.506010CrossRef Google Scholar

Porter, R., Theiler, J., & Hush, D. (2013). Interactive Machine Learning in Data Exploitation. Computing in Science & Engineering, 15(5), 12–20.10.1109/MCSE.2013.74CrossRef Google Scholar

Press, G. (2022, April 14). Cleaning Big Data: Most Time-Consuming, Least Enjoyable Data Science Task, Survey Says. Forbes. www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says/Google Scholar

Rahm, E., & Do, H. H. (2000). Data Cleaning: Problems and Current Approaches. IEEE Data Engineering Bulletin, 23(4), 3–13.Google Scholar

Ramos, G., Meek, C., Simard, P., Suh, J., & Ghorashi, S. (2020). Interactive Machine Teaching: A Human-centered Approach to Building Machine-learned Models. Human–Computer Interaction, 35(5–6), 413–451.10.1080/07370024.2020.1734931CrossRef Google Scholar

Räth, T., Onah, N., & Sattler, K.-U. (2023). Interactive Data Cleaning for Real-Time Streaming Applications. Proceedings of the Workshop on Human-In-the-Loop Data Analytics, 1–3. https://doi.org/10.1145/3597465.3605229 CrossRef Google Scholar

Reddy, D. (2023). Data Engineering Challenges in AI Automation. 2023 International Conference on Computing, Electronics & Communications Engineering (iCCECE), 107–112.Google Scholar

Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1135–1144.10.1145/2939672.2939778CrossRef Google Scholar

Rouzegar, H., & Makrehchi, M. (2024). Enhancing Text Classification through LLM-Driven Active Learning and Human Annotation. Proceedings of the 18th Linguistic Annotation Workshop (LAW-XVIII), 98–111.Google Scholar

Rožanec, J. M., Montini, E., Cutrona, V., Papamartzivanos, D., Klemenčič, T., Fortuna, B., Mladenić, D., Veliou, E., Giannetsos, T., & Emmanouilidis, C. (2024). Human in the AI Loop via xAI and Active Learning for Visual Inspection. In Soldatos, J. (ed.), Artificial Intelligence in Manufacturing (pp. 381–406). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-46452-2_22 CrossRef Google Scholar

Sachan, S., Almaghrabi, F., Yang, J.-B., & Xu, D.-L. (2024). Human–AI Collaboration to Mitigate Decision Noise in Financial Underwriting: A Study on FinTech Innovation in a Lending Firm. International Review of Financial Analysis, 93, 103149. https://doi.org/10.1016/j.irfa.2024.103149 Google Scholar

Schröder, C., & Niekler, A. (2020). A Survey of Active Learning for Text Classification using Deep Neural Networks [arXiv preprint]. arXiv:2008.07267. http://arxiv.org/abs/2008.07267 Google Scholar

Settles, B. (2011). From Theories to Queries: Active Learning in Practice. Active Learning and Experimental Design Workshop in Conjunction with AISTATS 2010, 1–18.Google Scholar

Sheikh, R., Milioto, A., Lottes, P., Stachniss, C., Bennewitz, M., & Schultz, T. (2020). Gradient and Log-based Active Learning for Semantic Segmentation of Crop and Weed for Agricultural Robots. 2020 IEEE International Conference on Robotics and Automation (ICRA), 1350–1356. https://ieeexplore.ieee.org/abstract/document/9196722/Google Scholar

Sheng, V. S., & Zhang, J. (2019). Machine Learning with Crowdsourcing: A Brief Summary of the Past Research and Future Directions. Proceedings of the AAAI Conference on Artificial Intelligence, 33(1), 9837–9843.10.1609/aaai.v33i01.33019837CrossRef Google Scholar

Shoshitaishvili, Y., Weissbacher, M., Dresel, L., Salls, C., Wang, R., Kruegel, C., & Vigna, G. (2017). Rise of the HaCRS: Augmenting Autonomous Cyber Reasoning Systems with Human Assistance. Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, 347–362. https://doi.org/10.1145/3133956.3134105 CrossRef Google Scholar

Shraga, R. (2022). HumanAL: Calibrating Human Matching beyond a Single Task. Proceedings of the Workshop on Human-In-the-Loop Data Analytics, 1–8. https://doi.org/10.1145/3546930.3547496 CrossRef Google Scholar

Smailovic, J., Grcar, M., Lavrac, N., & Znidarsic, M. (2014). Stream-based Active Learning for Sentiment Analysis in the Financial Domain. Information Sciences, 285, 181–203. https://doi.org/10.1016/j.ins.2014.04.034 CrossRef Google Scholar

Smith, A., Kumar, V., Boyd-Graber, J., Seppi, K., & Findlater, L. (2018). Closing the Loop: User-Centered Design and Evaluation of a Human-in-the-Loop Topic Modeling System. 23rd International Conference on Intelligent User Interfaces, 293–304. https://doi.org/10.1145/3172944.3172965 CrossRef Google Scholar

Sourati, J., Akcakaya, M., Leen, T. K., Erdogmus, D., & Dy, J. G. (2017). Asymptotic Analysis of Objectives based on Fisher Information in Active Learning. Journal of Machine Learning Research, 18(34), 1–41.Google Scholar

Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. F. (2020). Learning to Summarize with Human Feedback. Advances in Neural Information Processing Systems, 33, 3008–3021.Google Scholar

Tajbakhsh, N., Jeyaseelan, L., Li, Q., Chiang, J., Wu, Z., & Ding, X. (2020). Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation. Medical Image Analysis, 63, 101693. https://doi.org/10.1016/j.media.2020.101693 CrossRef Google Scholar PubMed

Vaughan, J. W. (2018). Making Better Use of the Crowd: How Crowdsourcing Can Advance Machine Learning Research. Journal of Machine Learning Research, 18(193), 1–46.Google Scholar

Wallace, E., Rodriguez, P., Feng, S., Yamada, I., & Boyd-Graber, J. (2019). Trick Me If You Can: Human-in-the-Loop Generation of Adversarial Examples for Question Answering. Transactions of the Association for Computational Linguistics, 7, 387–401.10.1162/tacl_a_00279CrossRef Google Scholar

Wang, G., Zuluaga, M. A., Li, W., Pratt, R., Patel, P. A., Aertsen, M., Doel, T., David, A. L., Deprest, J., Ourselin, S., & Vercauteren, T. (2019). DeeplGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(7), 1559–1572. https://doi.org/10.1109/TPAMI.2018.2840695 CrossRef Google Scholar PubMed

Wang, Z. J., Choi, D., Xu, S., & Yang, D. (2021). Putting Humans in the Natural Language Processing Loop: A Survey. Proceedings of the First Workshop on Bridging Human–Computer Interaction and Natural Language Processing, 47–52. https://aclanthology.org/2021.hcinlp-1.8/Google Scholar

Ware, M., Frank, E., Holmes, G., Hall, M., & Witten, I. H. (2001). Interactive Machine Learning: Letting Users Build Classifiers. International Journal of Human–Computer Studies, 55(3), 281–292.10.1006/ijhc.2001.0499CrossRef Google Scholar

Wondimu, N. A., Buche, C., & Visser, U. (2022). Interactive Machine Learning: A State of the Art Review [arXiv preprint]. arXiv:2207.06196. http://arxiv.org/abs/2207.06196 Google Scholar

Wrede, F., & Hellander, A. (2019). Smart Computational Exploration of Stochastic Gene Regulatory Network Models Using Human-in-the-Loop Semi-supervised Learning. Bioinformatics, 35(24), 5199–5206.10.1093/bioinformatics/btz420CrossRef Google Scholar PubMed

Wu, X., Xiao, L., Sun, Y., Zhang, J., Ma, T., & He, L. (2022). A Survey of Human-in-the-Loop for Machine Learning. Future Generation Computer Systems, 135, 364–381.10.1016/j.future.2022.05.014CrossRef Google Scholar

Wu, Y., Wang, C., Gumusel, E., & Liu, X. (2024). Knowledge-Infused Legal Wisdom: Navigating LLM Consultation through the Lens of Diagnostics and Positive-Unlabeled Reinforcement Learning. Findings of the Association for Computational Linguistics 2024.10.18653/v1/2024.findings-acl.918CrossRef Google Scholar

Xiao, R., Dong, Y., Zhao, J., Wu, R., Lin, M., Chen, G., & Wang, H. (2023). FreeAL: Towards Human-Free Active Learning in the Era of Large Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 14520–14535. https://aclanthology.org/2023.emnlp-main.896/10.18653/v1/2023.emnlp-main.896CrossRef Google Scholar

Xu, W., Dainoff, M. J., Ge, L., & Gao, Z. (2023). Transitioning to Human Interaction with AI Systems: New Challenges and Opportunities for HCI Professionals to Enable Human-Centered AI. International Journal of Human–Computer Interaction, 39(3), 494–518.10.1080/10447318.2022.2041900CrossRef Google Scholar

Yang, J., Lan, G., Li, Y., Gong, Y., Zhang, Z., & Ercisli, S. (2022). Data Quality Assessment and Analysis for Pest Identification in Smart Agriculture. Computers and Electrical Engineering, 103, 108322.10.1016/j.compeleceng.2022.108322CrossRef Google Scholar

Yang, Q., Suh, J., Chen, N.-C., & Ramos, G. (2018). Grounding Interactive Machine Learning Tool Design in How Non-Experts Actually Build Models. Proceedings of the 2018 Designing Interactive Systems Conference, 573–584.10.1145/3196709.3196729CrossRef Google Scholar

Zhang, J., Wu, X., & Sheng, V. S. (2016). Learning from Crowdsourced Labeled Data: A Survey. Artificial Intelligence Review, 46(4), 543–576.10.1007/s10462-016-9491-9CrossRef Google Scholar

Zhou, X., Wang, Q., Wang, X., Tang, H., & Liu, X. (2023). Large Language Model Soft Ideologization via AI-Self-Consciousness [arXiv preprint]. arXiv:2309.16167. http://arxiv.org/abs/2309.16167 Google Scholar

Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., & Irving, G. (2020). Fine-Tuning Language Models from Human Preferences [arXiv preprint]. arXiv:1909.08593. http://arxiv.org/abs/1909.08593 Google Scholar

Accessibility standard: WCAG 2.2 AAA

The PDF of this book complies with version 2.2 of the Web Content Accessibility Guidelines (WCAG), offering more comprehensive accessibility measures for a broad range of users and attains the highest (AAA) level of WCAG compliance, optimising the user experience by meeting the most extensive accessibility guidelines.

Content Navigation

Table of contents navigation
Allows you to navigate directly to chapters, sections, or non‐text items through a linked table of contents, reducing the need for extensive scrolling.

Index navigation
Provides an interactive index, letting you go straight to where a term or subject appears in the text without manual searching.

Reading Order & Textual Equivalents

Single logical reading order
You will encounter all content (including footnotes, captions, etc.) in a clear, sequential flow, making it easier to follow with assistive tools like screen readers.

Short alternative textual descriptions
You get concise descriptions (for images, charts, or media clips), ensuring you do not miss crucial information when visual or audio elements are not accessible.

Full alternative textual descriptions
You get more than just short alt text: you have comprehensive text equivalents, transcripts, captions, or audio descriptions for substantial non‐text content, which is especially helpful for complex visuals or multimedia.

Visual Accessibility

Use of colour is not sole means of conveying information
You will still understand key ideas or prompts without relying solely on colour, which is especially helpful if you have colour vision deficiencies.

Use of high contrast between text and background colour
You benefit from high‐contrast text, which improves legibility if you have low vision or if you are reading in less‐than‐ideal lighting conditions.