Mixed Methods Evaluative Research for a New Generative AI Product

Metrics

Analysis and Findings

Although the sample size was small, I calculated the average click count and mode, as these metrics still provided valuable insights into user behavior and highlighted key interaction patterns.

Usability Tasks Findings

Tools

• Pendo (intercept recruiting of current Medidata users)

• Maze (unmoderated usability test platform and data capture)

• Zoom (remote moderated usability tests and interviews)

• Condens (video hosting, editing, and text analysis)

• Tremendous (participant incentive payout)

• Google sheets, Docs (analysis and reporting)

• Notion (research repository)

In clinical trials, Data Managers are essential to ensure the accurate, secure, and efficient collection, management, and analysis of data. They commonly do this through audit trails which are detailed records of every change and data entry made during a study.

Following discovery interviews with these professionals to learn about their daily workflows, the Data Management team learned the current process to run ATRs in Medidata’s system was tedious and time consuming. Development of Medidata’s first Generative AI feature began to reduce timelines for Data Managers completing the common role responsibility of Audit Trail Reviews.

**Maybe enter the common questions and workflow image

Current ATR Process

1. Run a report in Rave Audit Trail Report to pull data

2. Download report to Excel and check if it is the correct and needed data

3. Continue this process until the correct information is found

Current average time to completion: 2 hours

In an effort to reduce task burden, Audit Trail Review would allow users to interact with a genAI chat interface to request a preview display of audit trail information related to clinical trial data entered by sites before having to download to Excel.

Study Method

A mixed methods approach was utilized. Unmoderated usability testing was completed first in an effort to collect feedback faster and more cost effectively, and make necessary changes before proceeding with moderated usability testing.

Phase 1: Unmoderated Usability Studies to gather broad, initial insights and identify pain points

Study Goals

1. Test design concepts and intuitiveness of the tool

2. Understand the tool’s usefulness

3. Evaluate the need for users to see backend data and model’s chain of thought

Recruiting and Execution

Through intercept recruiting, 11 Data Managers who were Medidata users participated. The tests were conducted through Maze.

Task Mode of Difficulty

Task 2, which required using a template, had the highest average misclick count and was rated as the most difficult to complete.

Assumption: Some difficulty with Task 2 may be attributed to a lack of capability to include progressive user assistances (PUAs) in Maze.

Even though half of participants answered that seeing the SQL behind the prompt would help them complete their task, only 1 participant mentioned the benefits of this in their open-response feedback.