About the Client
Softjourn is a software development and consulting company with deep expertise in fintech, payments, and event ticketing, serving clients across the US and Europe. Our AI R&D program explores how AI can improve both our own work and the services we offer clients.
This case study covers one of those projects: building our AI Readiness Assessment and testing it on Redmine Candidate Search, an AI assistant our HR team uses to find available people for new projects.
We Ran Our AI Readiness Assessment on Ourselves First
Almost every company we talk to has an AI idea on the table, a pilot underway, or an AI feature already running in production. Far fewer can answer the question that comes next: is it worth investing the time and money to fully build it out?
Most readiness assessments on the market don't answer it either. They tend to deliver maturity scores and charts with little evidence behind them. We built the AI Readiness Assessment to do better: a service that ends with a clear decision and an action plan, based on what is actually happening inside the system.
Before taking the AI Readiness Assessment to clients, we put it to work on a tool from our own R&D team.
Like many companies, we run on an internal system of record. Ours is Redmine, an open-source project management platform we have customized over the years to track our people, their skills, project allocation, and time off. When a new project needs a team, our HR team searches Redmine for available specialists, which often means filtering a skills report and then checking each person's availability in a separate view.
We built Redmine Candidate Search, an AI-powered search tool, to shorten that routine. It is a Slack assistant that lets HR ask in plain language, for example, "find a senior QA with accessibility testing experience who is available for full-time work," and get a ready shortlist with each person's role, seniority, skills, and current allocation.
It started as an R&D experiment with a few goals at once: test whether Redmine could support an AI layer, build hands-on AI engineering experience, and give HR a quicker view into potential candidates.
The assistant was already in production, but it had no written business case, no usage numbers, and its original developer had moved on to other work. HR also double-checked its answers in Redmine afterward, so for now the assistant added a step rather than replacing one. In other words, it looked a lot like the AI systems clients sometimes bring to us to verify before they invest more in them.
Our AI Readiness Assessment came with real constraints on this project:
- One solution architect, working on it at about half capacity
- A framework that was still being finished while it was in use, so parts were reworked two or three times
- A system that kept changing during the review
Before Investing More in Our AI Tool, We Got Honest Feedback
We designed the AI Readiness Assessment as a short, focused review of one AI solution and one key user flow. For Redmine, that flow was "an HR manager finds available candidates for a project."
The process runs in four steps:
- Kickoff: agree on the AI flow, the problem to focus on, the scope, and the materials and access needed.
- Evidence review: documents, interviews with the people who build and use the tool, and, depending on the stage, code, logs, and live behavior. AI workflows work through the evidence alongside the architect, drafting findings, and the architect keeps only what the evidence supports.
- Report: a Decision Brief with a recommended direction (Build now, Fix first, or Defer), scores for each area assessed, findings and their impact, and a prioritized action plan. Anything that couldn't be verified is clearly marked.
- Readout: a session to walk through the findings, next steps, and any questions.
Every assessment looks at the same six areas, each answering one question:
How long an assessment takes depends on the scope we agree on at kickoff and on the solution’s stage. As a starting reference, it takes about one week of full-time work for an idea, two for a solution in development, and three for one in production. Anything that needs deeper investigation becomes a recommended follow-up. Our own pilot stretched to about a month on the calendar, because our architect worked on it part-time while finishing the framework alongside it.
"Anyone can ask AI to generate an ‘assessment framework,’ and in a few hours you’ll have 20 pages that look convincing. What took real time was applying the experience from our previous audits and making it work with agentic workflows." – Taras Romaniv, Solutions Architect, Softjourn
What Running an AI Assessment on Our Own Project Taught Us
AI drafts, people decide: AI workflows did the heavy lifting, but about 70% of the first AI-drafted findings were rejected after expert review. The AI could see how something worked but not why it was built that way, so many of the “problems” it flagged turned out to be deliberate choices made for project-specific reasons. It worked the other way too: some real issues surfaced only in the architect’s own review, missed even by the most advanced models. AI speeds the assessment up, but it doesn’t replace an architect’s judgment.
The code has the final word: Documents, interviews, and even test reports sometimes disagreed with what the code actually did. More than once, something described as a problem turned out to be a deliberate choice the team had made, and the reverse happened too. Whenever a solution is beyond the idea stage and has code, reviewing it is a standard part of the method.
People still matter: About 30% of the valid findings came from talking to people. Interviews surfaced the process, ownership, and adoption issues that a codebase simply can't show.
Advice has to fit the situation: The textbook fix would’ve been a month spent building automated tests for the AI. For a tool that hadn't yet proven its value, that would be like noticing a crooked wall and offering to rebuild the whole house. So the action plan starts with quick, inexpensive steps that answer the value question first.
Clear scope keeps it short: The assistant pulls its data from Redmine through a separate connector, which other tools could use too. We reviewed how the assistant uses that connector but saved the connector itself for its own review. Drawing that line kept the assessment focused and on schedule, and it's now a standard rule for every AI Readiness Assessment.
“I didn't treat this as an assessment we were doing for ourselves. I treated it as if it were for a real client.”— Taras Romaniv, Solutions Architect at Softjourn
The Benefits of Assessing Our Idea
The framework worked end to end on a real production AI system and produced a clear, defensible decision. For Redmine Candidate Search, the decision was to defer.
Defer doesn't mean shut it down. The recommendation is to keep the assistant running at its current low cost but fund no new features until four facts exist: how long a manual search takes, how many candidate searches the company runs each month, how many people actually use the assistant, and a written answer on whether employee data may be sent to a third-party AI model. If those facts arrive within 60 days and support the tool, the team decides again. If HR stops using it in the meantime, the tool is retired. As the Decision Brief put it, this is a question about value, not about engineering.
One of the most useful findings was also the simplest: a standard Redmine report could answer most of the questions the assistant was built for. What AI added was convenience, an answer without leaving the Slack chat. That may well be worth paying for, but nobody had measured it, and an honest assessment has to say so.
Alongside the verdict came scores across all six areas (two Low, four Medium) and a 10-step action plan. Most steps take anywhere from a few minutes to a day. The most important one, timing a manual search against an assistant search on the same request, takes about an hour. The plan also flagged two urgent technical fixes to make right away, regardless of whether we invest further in the tool.
Here are a few of the findings:
- Strength: the assistant uses each user's own login, has read-only access, and logs every request, so it works within user permissions, and every request can be traced.
- Gap: permission to send employee data to an external AI model hadn't been confirmed in writing.
- Gap: the assistant's database had no backup, so a failure could permanently lose user links and chat history.
- Gap: searching for "Swift" missed profiles that said "iOS," so HR still had to double-check results by hand.
- Measurement gap: nobody had measured manual search time or how much the assistant was actually used.
The verdict hasn't dampened enthusiasm for the tool. Our HR team still sees real value in it and has already compared its answers with manual Redmine searches on their own requests, exactly the kind of evidence the action plan asks for.
For Softjourn, the biggest result is the service itself: a validated framework, five AI workflows that support the architect at each stage, and a client pilot package ready to go. The workflows are interactive by design. They advise and assist at every step, and the architect makes every call.
"The value of the AI Readiness Assessment isn't just in finding something nobody knew. It's also gaining one clear view of the solution, accurately interpreted, with advice for moving forward." – Taras Romaniv, Solutions Architect, Softjourn
Conclusion: A Win-Win
Testing the AI Readiness Assessment on our own tool gave us a real case to learn from and an honest result we're happy to share. It also confirmed what we want the AI Readiness Assessment to be for clients: an outside view that is willing to say "not yet" when the evidence calls for it.
"It was a win-win: we got a real case to show clients, and proof that the AI Readiness Assessment works." – Oleh Sorokopud, Head of Digital Marketing, Softjourn
Next, we're running the AI Readiness Assessment with a small group of existing clients, starting with ticketing companies, and building domain lenses: extra sets of questions for areas like ticketing and finance. The same AI workflows are also being adapted for code audits and reviews of AI-generated applications.
Whatever the verdict, the next step is yours: act on the findings internally, work with another partner, or continue with Softjourn. If you're deciding whether to build, fix, or grow an AI solution, contact Softjourn for an independent, evidence-based answer in as little as one week.