Privacy Identification of Human–Generative AI Interaction

doi:10.1017/9781009587877.004

3 - Privacy Identification of Human–Generative AI Interaction

Published online by Cambridge University Press: 19 September 2025

Dan Wu and

Guoye Sun

Edited by

Dan Wu and

Shaobo Liang

Show author details

Dan Wu: Affiliation:
Wuhan University, China
Shaobo Liang: Affiliation:
Wuhan University, China

Book contents

Get access

Summary

Generative AI based on large language models (LLM) currently faces serious privacy leakage issues due to the wide range of parameters and diverse data sources. When using generative AI, users inevitably share data with the system. Personal data collected by generative AI may be used for model training and leaked in future outputs. The risk of private information leakage is closely related to the inherent operating mechanism of generative AI. This indirect leakage is difficult to detect by users due to the high complexity of the internal operating mechanism of generative AI. By focusing on the private information exchanged during interactions between users and generative AI, we identify the privacy dimensions involved and develop a model for privacy types in human–generative AI interactions. This can provide a reference for generative AI to avoid training private data and help it provide clear explanations of relevant content for the types of privacy users are concerned about.

Keywords

Generative AI Privacy Leakage Privacy Type Human–AI Interaction

Information

Type: Chapter
Information: Human-AI Interaction and Collaboration , pp. 43 - 81

DOI: https://doi.org/10.1017/9781009587877.004 [Opens in a new window]

Publisher: Cambridge University Press

Print publication year: 2025

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Book purchase

Temporarily unavailable

References

Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., & Zhang, L. (2016). Deep Learning with Differential Privacy. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, 308–318. https://doi.org/10.1145/2976749.2978318 CrossRef Google Scholar

Apthorpe, N., Varghese, S., & Feamster, N. (2019). Evaluating the Contextual Integrity of Privacy Regulation: Parents’ {IoT} Toy Privacy Norms Versus {COPPA}, 123–140. www.usenix.org/conference/usenixsecurity19/presentation/apthorpe Google Scholar

Baidoo-anu, D., & Ansah, L. O. (2023). Education in the Era of Generative Artificial Intelligence (AI): Understanding the Potential Benefits of ChatGPT in Promoting Teaching and Learning. Journal of AI, 7(1), 52–62. https://doi.org/10.61969/jai.1337500 CrossRef Google Scholar

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., … Amodei, D. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems, 33, 1877–1901. https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html Google Scholar

Carlini, N., Tramèr, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, Ú., Oprea, A., & Raffel, C. (2021). Extracting Training Data from Large Language Models. 30th USENIX Security Symposium, 2633–2650. www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting Google Scholar

Cascella, M., Montomoli, J., Bellini, V., & Bignami, E. (2023). Evaluating the Feasibility of ChatGPT in Healthcare: An Analysis of Multiple Clinical and Research Scenarios. Journal of Medical Systems, 47(1), 33. https://doi.org/10.1007/s10916–023-01925-4 CrossRef Google Scholar PubMed

Chang, D., Krupka, E. L., Adar, E., & Acquisti, A. (2016). Engineering Information Disclosure: Norm Shaping Designs. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 587–597. https://doi.org/10.1145/2858036.2858346 CrossRef Google Scholar

Chen, L., Sun, L., & Han, J. (2023). A Comparison Study of Human and Machine-Generated Creativity. Journal of Computing and Information Science in Engineering, 23(051012). https://doi.org/10.1115/1.4062232 CrossRef Google Scholar

Choi, J. H., Hickman, K. E., Monahan, A. B., & Schwarcz, D. (2021). ChatGPT Goes to Law School. Journal of Legal Education, 71, 387.Google Scholar

Chua, H. N., Ooi, J. S., & Herbland, A. (2021). The Effects of Different Personal Data Categories on Information Privacy Concern and Disclosure. Computers & Security, 110, 102453. https://doi.org/10.1016/j.cose.2021.102453 CrossRef Google Scholar

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). ImageNet: A Large-scale Hierarchical Image Database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, 248–255. https://doi.org/10.1109/CVPR.2009.5206848 CrossRef Google Scholar

Dockhorn, T., Cao, T., Vahdat, A., & Kreis, K. (2023). Differentially Private Diffusion Models [arXiv preprint]. arXiv:2210.09929. https://doi.org/10.48550/arXiv.2210.09929 CrossRef Google Scholar

Dwivedi, Y. K., Kshetri, N., Hughes, L., Slade, E. L., Jeyaraj, A., Kar, A. K., Baabdullah, A. M., Koohang, A., Raghavan, V., Ahuja, M., Albanna, H., Albashrawi, M. A., Al-Busaidi, A. S., Balakrishnan, J., Barlette, Y., Basu, S., Bose, I., Brooks, L., Buhalis, D., … Wright, R. (2023). Opinion Paper: “So What if ChatGPT Wrote It?” Multidisciplinary Perspectives on Opportunities, Challenges and Implications of Generative Conversational AI for Research, Practice and Policy. International Journal of Information Management, 71, 102642. https://doi.org/10.1016/j.ijinfomgt.2023.102642 CrossRef Google Scholar

Dwork, C., & Roth, A. (2014). The Algorithmic Foundations of Differential Privacy. Foundations and Trends® in Theoretical Computer Science, 9(3–4), 211–407. https://doi.org/10.1561/0400000042 CrossRef Google Scholar

Estrada, S. (2023, March 1). A Startup CFO used ChatGPT to Build an FP&A tool: Here’s How It Went. Fortune. https://fortune.com/2023/03/01/startup-cfo-chatgpt-finance-tool/Google Scholar

Finn, R. L., Wright, D., & Friedewald, M. (2013). Seven Types of Privacy. In Gutwirth, S., Leenes, R., de Hert, P., & Poullet, Y. (eds.), European Data Protection: Coming of Age (pp. 3–32). Springer Netherlands. https://doi.org/10.1007/978-94-007-5170-5_1 CrossRef Google Scholar

Fire, M., Goldschmidt, R., & Elovici, Y. (2014). Online Social Networks: Threats and Solutions. IEEE Communications Surveys & Tutorials, 16(4), 2019–2036. https://doi.org/10.1109/COMST.2014.2321628 CrossRef Google Scholar

Golda, A., Mekonen, K., Pandey, A., Singh, A., Hassija, V., Chamola, V., & Sikdar, B. (2024). Privacy and Security Concerns in Generative AI: A Comprehensive Survey. IEEE Access, 12, 48126–48144. https://doi.org/10.1109/ACCESS.2024.3381611 CrossRef Google Scholar

Jancke, G., Venolia, G. D., Grudin, J., Cadiz, J. J., & Gupta, A. (2001). Linking Public Spaces: Technical and Social Issues. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 530–537. https://doi.org/10.1145/365024.365352 CrossRef Google Scholar

Kelley, P. G., Cesca, L., Bresee, J., & Cranor, L. F. (2010). Standardizing Privacy Notices: An Online Study of the Nutrition Label Approach. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1573–1582. https://doi.org/10.1145/1753326.1753561 CrossRef Google Scholar

Kimmel, D. (2023, May 16). ChatGPT Therapy Is Good, But It Misses What Makes Us Human. Columbia University Department of Psychiatry. www.columbiapsychiatry.org/news/chatgpt-therapy-is-good-but-it-misses-what-makes-us-human Google Scholar

Kshetri, N. (2023). Cybercrime and Privacy Threats of Large Language Models. IT Professional, 25(3), 9–13. https://doi.org/10.1109/MITP.2023.3275489 CrossRef Google Scholar

Leon, P. G., Ur, B., Wang, Y., Sleeper, M., Balebako, R., Shay, R., Bauer, L., Christodorescu, M., & Cranor, L. F. (2013). What Matters to Users? Factors That Affect Users’ Willingness to Share Information with Online Advertisers. Proceedings of the Ninth Symposium on Usable Privacy and Security, 1–12. https://doi.org/10.1145/2501604.2501611 CrossRef Google Scholar

Leonard, A. (2023, September 16). “Dr. Google” Meets Its Match in Dr. ChatGPT. NPR. www.npr.org/sections/health-shots/2023/09/16/1199924303/chatgpt-ai-medical-advice Google Scholar

Li, J., Cao, H., Lin, L., Hou, Y., Zhu, R., & El Ali, A. (2024). User Experience Design Professionals’ Perceptions of Generative Artificial Intelligence. Proceedings of the CHI Conference on Human Factors in Computing Systems, 1–18. https://doi.org/10.1145/3613904.3642114 CrossRef Google Scholar

Liu, J., Lau, C. P., & Chellappa, R. (2023). DiffProtect: Generate Adversarial Examples with Diffusion Models for Facial Privacy Protection [arXiv preprint]. arXiv:2305.13625. https://doi.org/10.48550/arXiv.2305.13625 CrossRef Google Scholar

Lu, L., McDonald, C., Kelleher, T., Lee, S., Chung, Y. J., Mueller, S., Vielledent, M., & Yue, C. A. (2022). Measuring Consumer-perceived Humanness of Online Organizational Agents. Computers in Human Behavior, 128, 107092. https://doi.org/10.1016/j.chb.2021.107092 CrossRef Google Scholar

Maddison, L. (2023, April 4). Samsung Workers Made a Major Error by Using ChatGPT. TechRadar. www.techradar.com/news/samsung-workers-leaked-company-secrets-by-using-chatgpt Google Scholar

Milne, G. R., Pettinico, G., Hajjat, F. M., & Markos, E. (2017). Information Sensitivity Typology: Mapping the Degree and Type of Risk Consumers Perceive in Personal Data Sharing. Journal of Consumer Affairs, 51(1), 133–161. https://doi.org/10.1111/joca.12111 CrossRef Google Scholar

Mu, Y., Zhang, Q., Hu, M., Wang, W., Ding, M., Jin, J., Wang, B., Dai, J., Qiao, Y., & Luo, P. (2023). EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought. Advances in Neural Information Processing Systems, 36, 25081–25094. https://proceedings.neurips.cc/paper_files/paper/2023/hash/4ec43957eda1126ad4887995d05fae3b-Abstract-Conference.html Google Scholar

Narayanan, A., & Shmatikov, V. (2008). Robust De-anonymization of Large Sparse Datasets. 2008 IEEE Symposium on Security and Privacy (Sp 2008), 111–125. https://doi.org/10.1109/SP.2008.33 Google Scholar

Nissenbaum, H. (2004). Privacy as Contextual Integrity Symposium: Technology, Values, and the Justice System. Washington Law Review, 79(1), 119–158.Google Scholar

Ouyang, S., Wang, S., Liu, Y., Zhong, M., Jiao, Y., Iter, D., Pryzant, R., Zhu, C., Ji, H., & Han, J. (2023). The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactions [arXiv preprint]. arXiv:2310.12418. https://doi.org/10.48550/arXiv.2310.12418 CrossRef Google Scholar

Peris, C., Dupuy, C., Majmudar, J., Parikh, R., Smaili, S., Zemel, R., & Gupta, R. (2023). Privacy in the Time of Language Models. Proceedings of the 16th ACM International Conference on Web Search and Data Mining, 1291–1292. https://doi.org/10.1145/3539597.3575792 CrossRef Google Scholar

Phelps, J., Nowak, G., & Ferrell, E. (2000). Privacy Concerns and Consumer Willingness to Provide Personal Information. Journal of Public Policy & Marketing, 19(1), 27–41. https://doi.org/10.1509/jppm.19.1.27.16941 CrossRef Google Scholar

Porter, J. (2023, November 6). ChatGPT Continues to Be One of the Fastest-Growing Services Ever. The Verge. www.theverge.com/2023/11/6/23948386/chatgpt-active-user-count-openai-developer-conference Google Scholar

Robinson, C. (2017). Disclosure of Personal Data in Ecommerce: A Cross-national Comparison of Estonia and the United States. Telematics and Informatics, 34(2), 569–582. https://doi.org/10.1016/j.tele.2016.09.006 Google Scholar

Rumbold, J. M. M., & Pierscionek, B. K. (2018). What Are Data? A Categorization of the Data Sensitivity Spectrum. Big Data Research, 12, 49–59. https://doi.org/10.1016/j.bdr.2017.11.001 CrossRef Google Scholar

Shalaby, W., Arantes, A., GonzalezDiaz, T., & Gupta, C. (2020). Building Chatbots from Large Scale Domain-specific Knowledge Bases: Challenges and Opportunities. 2020 IEEE International Conference on Prognostics and Health Management (ICPHM), 1–8. https://doi.org/10.1109/ICPHM49022.2020.9187036 CrossRef Google Scholar

Shvartzshnaider, Y., Tong, S., Wies, T., Kift, P., Nissenbaum, H., Subramanian, L., & Mittal, P. (2016). Learning Privacy Expectations by Crowdsourcing Contextual Informational Norms. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 4, 209–218. https://doi.org/10.1609/hcomp.v4i1.13271 CrossRef Google Scholar

Smith, H. J., Dinev, T., & Xu, H. (2011). Information Privacy Research: An Interdisciplinary Review. MIS Quarterly, 35(4), 989–1015. https://doi.org/10.2307/41409970 CrossRef Google Scholar

Song, C., Ristenpart, T., & Shmatikov, V. (2017). Machine Learning Models that Remember Too Much. Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, 587–601. https://doi.org/10.1145/3133956.3134077 CrossRef Google Scholar

Stokel-Walker, C. (2023). AI Chatbots Are Coming to Search Engines: Can You Trust the Results? Nature. https://doi.org/10.1038/d41586–023-00423-4 CrossRef Google Scholar

Van Dis, E. A. M., Bollen, J., Zuidema, W., van Rooij, R., & Bockting, C. L. (2023). ChatGPT: Five Priorities for Research. Nature, 614(7947), 224–226. https://doi.org/10.1038/d41586-023-00288-7 CrossRef Google Scholar PubMed

The Vicuna Team. (2023, March 30). Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. LMSYS Org. https://lmsys.org/blog/2023-03-30-vicuna Google Scholar

Wolf, V., & Maier, C. (2024). ChatGPT Usage in Everyday Life: A Motivation-Theoretic Mixed-methods Study. International Journal of Information Management, 79, 102821. https://doi.org/10.1016/j.ijinfomgt.2024.102821 CrossRef Google Scholar

Wong, R. Y., & Mulligan, D. K. (2019). Bringing Design to the Privacy Table: Broadening “Design” in “Privacy by Design” Through the Lens of HCI. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–17. https://doi.org/10.1145/3290605.3300492 CrossRef Google Scholar

Wu, D., & Sun, G. (2023). The Credibility of the Results of Generative Intelligent Search. Journal of Library Science in China, 49(6), 51–67. https://doi.org/10.13530/j.cnki.jlis.2023048 Google Scholar

Zhang, C., Ippolito, D., Lee, K., Jagielski, M., Tramer, F., & Carlini, N. (2023). Counterfactual Memorization in Neural Language Models. Advances in Neural Information Processing Systems, 36, 39321–39362.Google Scholar

Zhang, H., Chen, J., Jiang, F., Yu, F., Chen, Z., Li, J., Chen, G., Wu, X., Zhang, Z., Xiao, Q., Wan, X., Wang, B., & Li, H. (2023). HuatuoGPT, towards Taming Language Model to Be a Doctor [arXiv preprint]. arXiv:2305.15075. https://doi.org/10.48550/arXiv.2305.15075 CrossRef Google Scholar

Zhang, Z., Jia, M., Lee, H.-P. (Hank), Yao, B., Das, S., Lerner, A., Wang, D., & Li, T. (2024). “It’s a Fair Game,” or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents. Proceedings of the CHI Conference on Human Factors in Computing Systems, 1–26. https://doi.org/10.1145/3613904.3642385 CrossRef Google Scholar

Accessibility standard: Unknown

Accessibility compliance for the PDF of this book is currently unknown and may be updated in the future.