Mini Research on Zero Shot Prompting

ReimaginingUX Annotation
withMLLMs

Transforming UI screenshots into structured UX annotations using language and vision models — shifting humans from tedious annotators to strategic reviewers.

Explore Demo Results
UI Element Detection & UX Annotation

Before & After Showcase

See the results with a side-by-side comparison.

Nike App

Nike App - BeforeBEFORE
Nike App - AfterAFTER

Slide the bar to compare before and after versions.

18

Elements Detected

6

Components Identified

90s

Analysis Duration

Invisible Effort in UI Design

Small Touches, Big Difference

Thoughtful details shape seamless, trustworthy experiences, often in ways we don't even notice. Here are some real-world examples of small, but impactful touches.

Duolingo speech bubble UI

Building Trust with Friendly Prompts

A candid speech bubble reduces friction and communicates trust, making users feel comfortable and understood during onboarding.

Background Statement

Problem & Solution

What is the problem we are trying to bridge?

The Pain Point: Manual UX Annotation

Painfully Slow

Manual bounding boxes & metadata tagging devour design hours, delaying projects.

Highly Inconsistent

Varied annotator styles lead to inconsistencies.

Error-Prone Process

Repetitive manual tasks increase human mistakes in labeling and classification.

Impossible to Scale

Manual workflows bottleneck innovation and can't match rapid design iterations.

The Solution: AI-Powered Automation

Automated UI Detection

AI detects and identifies UI elements in your screenshots.

Predrawn Bounding Box

Vision Language Model pre-draws UI element boundaries based on AI-extracted descriptions, accelerating the annotation process.

Component Annotation

LLM models grasp overall component function and context without explicit training.

Impact on Workflow

"Empower designers to create, not just catalogue. Let AI handle the heavy lifting."

Before: The Annotator

Bogged down by tedious, repetitive clicking. Drained by manual data entry.

After: The Strategist

Elevated to reviewer. Focused on UX quality & insights. Driving innovation at speed.

Technology Stack

Core Technologies

An overview of the primary technologies and services used in this application.

•Next.js
•Supabase
•Claude 3.7
•Vercel
•OpenAI GPT-4.1
•Moondream
•Next.js
•Supabase
•Claude 3.7
•Vercel
•OpenAI GPT-4.1
•Moondream
•Next.js
•Supabase
•Claude 3.7
•Vercel
•OpenAI GPT-4.1
•Moondream
•Next.js
•Supabase
•Claude 3.7
•Vercel
•OpenAI GPT-4.1
•Moondream
Project Highlights

How The Pipeline Works

Leveraging cutting-edge ML technology to transform UI screenshots into detailed annotations

Label & Description Extraction

AI generates clear labels and functional descriptions for each UI element.

Image Processing
UI Screenshot
LLM Extraction Results:
1
Search Field
Input element for user search queries
2
Navigation Menu
Primary site navigation controls
Auto-generated labels & descriptions from UI images

Context-Aware Prompt Engineering

Designs spatially grounded prompts that steer the model to focus on relevant UI elements based on visual hierarchy and layout context.

Optimise Zero Shot Prompts for VLM
❌ Ineffective Prompt:
"Text displaying the price of the choco croissant.
✘ Ambiguous, no anchor elements, unclear target or hierarchy.
✅ Optimized Prompt:
"Text displaying the price of the 'choco croissant' as'$5.90', with an'Add to cart' button below."
✔ Anchors price using color, nearby text, and button placement.
❌ Another Ineffective Prompt:
"Weight label of first item"
✘ Too generic, lacks specific identifiers or contextual clues.
✅ Another Optimized Prompt:
Gray text '230g' displayed to the right of 'Gnocchi with mushroom gravy' title."
✔ Specific label, clear positioning relative to other UI landmarks and content.
Clear anchoring & specificity lead to better annotation results

Automated UI Element Detection

VLM detects UI elements based on descriptions & draws bounding box coordinates.

Element Localization
Input Description:
"Gray text '230g' displayed to the right of 'Gnocchi with mushroom gravy' title."
Gnocchi with mushroom gravy230g
A delightful dish of potato gnocchi served with a rich mushroom gravy.
Weight Label
Input: Text
VLM Detection
Output: Boxes
VLMs locate elements from descriptions & output bounding box coordinates.

Tree-Based Structural Grouping

Transforms flat element lists into hierarchical trees to accurately distinguish components from their nested sub-elements.

Structural Hierarchy
Before: Flat Element List
Express Option Text
Express Option Label
Express Option Qunatity
Card
Card Title
After: Component Extraction
Card
Card Title
Express Option
Text
Label
Quantity
Organizes elements into a meaningful components

Rich Metadata Extraction

Augments each component with detailed UX metadata, including user flow impact, behavior & interaction specifications, element types, and state definitions.

Rich Metadata
Detected UI Component
Element Type
Search Input Field
Interaction Model
Accepts text input, updates on keypress
States
Default, Focus, Filled, Error
User Flow Impact
Critical path - Primary search function
Comprehensive metadata enhances usability insights

Parallelized Processing Pipeline

Processes run concurrently across images. 6 images, 160+ elements in under 6 minutes, significantly outperforming manual efforts.

Parallelized Processing
Parallel Processing Pipeline
Thread 1
Detect
Extract
Optimize
Thread 2
Detect
Extract
Optimize
Thread 3
Detect
Extract
Optimize
160+ elements
processed
~6 minutes
total time
10x faster
than manual
Concurrent processing drastically improves throughput