Softjourn R&D: Our Designers Tested AI on Real UX Work. Here's the Honest Breakdown.

The task was a standard competitive analysis: reviewing a set of competitor products against Nielsen Group UX standards (a widely used industry framework for evaluating the quality of a user experience) and producing a comparison ready to go into a client presentation.

Normally, this kind of research means spending several hours moving between sources, pulling data manually, cross-referencing standards documentation, and then formatting everything into something presentable. Most of that time is not design thinking. It is information gathering and formatting, the kind of work that is necessary but not where a senior designer adds the most value.

Using Claude and Gemini, we approached the task differently. Rather than searching manually, we built detailed prompts that directed the AI to gather relevant standards data, review competitor products against those criteria, and format the findings into comparison tables ready for a presentation deck.

The result: research that would typically take several hours was completed in under 60 minutes, roughly a 2x improvement in time. The output required review and refinement, as all AI-generated work does, but it arrived in a format that was already structured and presentation-ready rather than raw notes requiring a separate formatting pass.

What shifted was not just speed but the nature of the work itself. Traditionally, competitor research is roughly 80% data gathering, screenshots, feature lists, mood boards, and only 20% actual analysis. AI flipped that ratio. With near-instant ingestion of reference material, our designer could spend the majority of her time on the part that actually requires design expertise.

One of the more unexpected findings: the AI did not just surface feature gaps. It identified patterns in how competitors used financial terminology in their navigation, a layer of insight that informed a broader rethinking of the information architecture. That kind of synthesis would have been easy to miss in a manual research pass focused on capturing rather than interpreting.

Experiment 2: Large-Scale Asset Rebranding

This experiment came directly from a real project requirement. A client needed approximately 150 themed assets, logos, banners, and UI elements, replaced across a complex codebase, all resized and reformatted to fit updated brand specifications across various UI environments.

Done manually, this is a significant undertaking. A mid-level designer or developer would need to audit, export, rename, and replace each asset individually, verifying CSS integrity throughout. The estimated manual effort would be 12 to 16 hours.

We deployed Devin, an autonomous AI development agent (a tool that can be given a task and carry it out across a system with minimal step-by-step instruction), to handle the replacement process. Devin autonomously wrote the replacement scripts, executed the swaps, and performed self-correction when it encountered pathing errors along the way. Human involvement shifted to approximately 15 minutes of strategic prompting and final review.

That works out to a 95% reduction in total execution time, and a 98% reduction in human labor time. Human involvement shifted from owning every step of the process to 15 minutes of strategic prompting and final review. The bulk of the mechanical work was handled by the agent.

One important note worth being direct about: Devin can introduce unintended changes in other parts of a codebase as it works. A thorough final review pass is not optional. This is part of why the human-validated principle applies here as much as anywhere else. The agent's speed is only an advantage if the output is verified before it goes anywhere near production.

Experiment 3: Onboarding Flow Redesign (Where AI Fell Short)

Not every experiment delivered a clean win. This one is worth documenting with the same care.

The task was to take a 10-step onboarding flow for a fintech client product and restructure it into segmented tabs organized by user type, specifically separating the experience for Prepaid card users versus Credit card users, with different mandatory and optional steps for each group.

We used Claude, and we put genuine effort into the prompting. The business logic was described in detail. The constraints around which steps were required for which card types were spelled out explicitly.

Claude produced a restructured flow, but it could not correctly apply the product-specific rules. It grouped steps logically based on general structure without understanding which ones carried actual business consequences for a specific card type. The distinction between mandatory and optional steps for a Prepaid versus Credit user is not something that can be inferred from a general description. It requires genuine knowledge of how the product works, knowledge that lives in the team, not in a prompt.

The result: the team manually rewrote and divided the flow.

This is the category of AI failure worth naming clearly. The output looked plausible. It was structured, it addressed the prompt, and it would pass a quick visual scan. The problems were in the details, which is exactly where a flawed onboarding flow causes real user friction. 

For example, the AI grouped a step that carried mandatory compliance weight for Prepaid users into the optional category, an error that looked completely reasonable on the surface but would have created a real regulatory gap if it had gone unreviewed.

The takeaway is not that AI cannot contribute to flow design. Used as a structural scaffold or a sounding board at an early stage, it can still save time. But complex logic restructuring, where the sequencing of steps carries direct business consequences, needs to stay in human hands. The AI can help frame the problem, but it should not own the answer.

The deeper issue here was not capability but context. The AI had no access to the institutional knowledge that makes this fintech product's onboarding logic work the way it does. Which steps may carry regulatory weight for a Prepaid user, which fields are optional for Credit but mandatory for compliance reasons, and why certain steps exist at all. That knowledge lives in our team, accumulated through months of working with the product, and no prompt can fully substitute for it. 

Claude as a Day-to-Day Design Tool

Beyond the three structured experiments, our design team has built Claude into regular project work in ways that are harder to measure but equally noticeable in practice.

The most consistent use is as a starting point. Rather than opening Figma to a blank canvas, designers use Claude to generate initial UX drafts, propose UI improvements, and sketch out early ideas that can be brought directly into the design tool for refinement. The blank canvas problem, that first hour of a project spent building base grids and standard flows from scratch, has largely been eliminated.

Claude is also useful for quick client proposals. When a project needs a fast, credible first pass on a flow or layout direction, generating an AI-assisted draft gives the team something concrete to react to and refine rather than starting from zero.

One specific application: fed with detailed user personas and brand guidelines, Claude can generate UX logic and feature sets significantly faster than manual sketching of user flows and logic trees. It can also produce functional React or HTML components to test a flow interaction before a single pixel is committed in Figma, which means the team can pressure-test a user experience much earlier in the process.

Fed with detailed user personas, Claude accelerates the transition from persona to feature set by approximately 70%, cutting out much of the manual logic-sketching work that typically sits between research and design.

The pattern across all of these uses is the same one that showed up in the research experiment: Claude handles the setup work, and the designer handles the judgment.

Summary of Results