From Queries to Prompts: Comparing User Experience in Generative AI Tools and Search Engines

Abstract

Recent advances in large language models (LLMs) and the rise of Generative Artificial Intelligence (GenAI) tools, such as ChatGPT and Copilot, are ushering in a significant shift in the way people interact with information seeking systems. This study presents a mixed-methods investigation aimed at comparing user experiences of GenAI tools and Conventional Search Engines (CSEs). Twenty-four participants completed fact-finding and browsing tasks using both types of tools. Quantitative data was gathered using Tobii Fusion eye tracking device and a paper-based NASA-TLX survey, while qualitative data was gathered through semi-structured interviews after task completion. Results revealed that GenAI prompts were significantly longer and more conversational, and GenAI tools imposed higher cognitive load during fact-finding, but less cognitive load during browsing tasks. Qualitative findings indicated that users value GenAI for abstract, creative and personalised tasks, but expressed concerns over accuracy, trust, and data privacy. This study expands the limited body of research on comparing user behaviour and experiences when seeking information using CSEs and GenAI tools. It offers a novel contribution by identifying differences in cognitive load associated with completing different task types across the different tool types, highlighting patterns in GenAI interaction behaviours, while also identifying the factors that influence user preferences, perceptions, and overall experience of GenAI tools. The paper concludes with a discussion of the implications of these findings and provides recommendations for designing GenAI tools to enhance user experience.