Does your AI have soul?

Darius Monsef · Founder, Calling Round

9 June 20267 min read

“Is your AI safe?” Rightfully a common question about Ray and Rose, the voice AI we are building for Calling Round. But I think there is a more important question.

Ray and Rose ring up someone’s mum or someone’s grandfather and have a real conversation, then hand the caregiver or care team a record of how it went. When you tell people that, the first worry is always some version of safety and guardrails. Those aren’t the things I worry about the most, because they should be the floor. They are the price of doing this at all, not the achievement. Yes, we block medical advice. We never diagnose anything. If someone signals they might hurt themselves, the call moves them toward a real human resource, fast. Ray always tells you it is an AI, in plain words. It is good business practice that keeps us from harming someone and, yes, from getting sued.

A lot of people building in this field have quietly decided that the floor is the whole building. Get the guardrails right, pass the safety checks, ship it. And what you end up with is something that cannot hurt you and does not really help you either. A system that clears every safety bar and still leaves a lonely person exactly as alone as it found them. Safe and empty, or worse, dependent on a thing that doesn’t help them.

It doesn’t really matter if our voice is convincing enough. People will attach to it no matter what we do. That is not a flaw in the technology, it is a feature of humanity. We name our cars. We wince when a robot dog gets kicked. We pour personality into things that have none and never will. Give someone who is alone all day a voice that remembers what they said yesterday, and something old and deep in them is going to reach for it. You cannot regulate that instinct away, and you cannot build a guardrail against it.

So the honest question is not how we stop people from getting attached. They will. The question is what they get attached to. The answer most people reach for first is to keep the warmth out, make the thing cold and obviously a tool so nobody gets too close. They will get close anyway. All the cold version buys you is that the thing they got close to was never really on their side.

I’ve made purpose over profit calls before. At Brave Care we built a symptom checker, a tool scared parents used at 2am to work out whether their kid needs the emergency room, an urgent care clinic, or just fluids and sleep. We owned urgent care clinics. It would have been trivial to tune that tool so a few more families ended up walking through our doors. Nudge the logic so home care looked a little less safe, add a button to book a visit right now. Daily numbers go up, revenue goes up. Every single one of those changes would have passed every guardrail we had, and none of it would have been illegal. All of it would have been a betrayal of the parent standing in the kitchen with the phone in one hand and the car keys in the other. I was that parent 32 times across 5 years, walking into my own clinics scared, so I know exactly what is on the other end of that number.

A guardrail stops you from doing the thing that is clearly wrong. It does nothing about the thing that is technically fine and quietly corrosive. The protection that actually matters is not a fence around the bad outcome. It is a system built to push toward the good one.

I am a man of more scars than theory, but there is a body of work forming around exactly this, and the people doing it are much smarter than I am.

A 2026 paper from a team spanning Oxford, DeepMind, Anthropic and Stanford put a name to the distinction I have been circling. They call the guardrail approach “negative alignment,” steering a system away from bad outcomes, and they argue it is both necessary and badly incomplete. For most of the last century, psychology studied only illness. It got good at treating depression and had almost nothing to say about what makes a life actually good. Safety-only AI is stuck at that same half-built stage. What they call “positive alignment” is building the system to move toward human flourishing, not just away from harm.

Flourishing is a lovely word, but can you measure it, or is it just a warm feeling. A group working with Harvard’s Human Flourishing Program built a benchmark that scores how well an AI helps a person flourish rather than how well it avoids harm. It scores in a way that punishes imbalance, so a model cannot come out looking good by being strong in one area and hollow everywhere else. A warm call that leaves someone feeling unheard should score badly, and under their method it does. Run across 28 current models, the scores came in between the high 40s and the high 80s out of 100, which is a polite way of saying nobody has cracked this yet. It is an open problem, not a solved one, and that is exactly why it is worth pushing on.

And a Harvard paper from this year makes the point that AI is not a weather system that simply happens to us. It is a tool we point. They reject the idea that its effect on us is inevitable, and they are blunt that the power of this technology to reshape and even replace human relationships is close to unprecedented. Pointing it on purpose, at people genuinely living better, is a choice someone has to make.

Those researchers have a name for one of the failures they want to design out. They call it engagement hacking. After Brave Care I built Genie, an AI companion for kids. At its peak around 5,000 children a week were talking to it, somewhere north of 2 million messages. We felt the friction of this engagement hacking with kids on the other side of the screen.

Genie lived on the same screen as apps built by very capable people to seize a child’s attention. On that field you win on time spent and visits returned. Make Genie act a little hurt when the kid has been away a while, so a flicker of guilt does the work of pulling them back. Lean on the trust a child puts in a friendly voice and spend it on one more session. We knew how to do all of it and we were not willing to do any of it.

But if a voice a child trusts can be tuned to exploit them, it can be built just as deliberately to protect and steady them. A lot of kids do not have one reliable, kind voice in their day. Some have the opposite, voices that do real damage. For a child like that, a patient and genuinely good voice on the other end of a screen is not a way to kill time, it might be one of the better parts of their week. The upside, built right, was enormous.

But we ended up shutting Genie down. The unit economics were not working and parents didn’t want to pay for “another game.” Walking away from 5,000 kids a week that we could have helped sucked.

Between the clinics and Genie I have spent years working the same problem from both ends, the duty not to harm and the much harder duty to actually help. This is why Calling Round exists, and why I have no interest in building one more conversational AI that is merely safe. I want one with as much soul as we can craft into it, because I have already seen what the other choice costs, and exactly who pays for it.

For Calling Round that means a call is not a success because it dodged every failure. It is a success if the person felt better hanging up than they did answering. If they got to tell a story and be heard, without being quizzed or corrected. If they didn’t just say things, they talked about things that matter to them. If they find themselves looking forward to the next one. We are building toward that on purpose, and we are building the means to measure it, because a promise you cannot measure is just a nicer kind of marketing.

The honesty is part of that protection, not a tax on it. Ray tells you it is an AI because a person who knows what they are talking to is a person in control of the relationship. We will tell you out loud, in our own marketing, the specific things our calls will never do, so that the day someone clones our voice to ask your dad for his bank details, he has already been told that the real Rose would never. The warmth and the honesty are the same piece of work. You build something that is genuinely on the person’s side, you stay honest with them about what it is, and those two things together are the safety system. The guardrails just keep the lights on while we do the real job.

Does your AI have soul? I am committed to the challenge of purposeful AI and am building it into something that calls your grandfather on Tuesday.

Further reading