The question parents ask us more than anything else is, “Is this safe?”
And honestly, of course they do. We’re asking them to put an AI-powered toy in their kid’s room.
AI has a pretty bad track record of saying weird things. We’ve all seen the screenshots. Chatbots hallucinate, confidently make things up, get pulled into conversations they probably shouldn’t be having, or respond in ways nobody building them anticipated.
A lot of the AI world also moves incredibly fast. There’s always a newer model, another capability, another company trying to do something before everyone else does. You can get away with a certain amount of “we’ll see what happens” when you’re building software for adults.
We can’t do that. Phoenyx talks to three-year-olds. So a lot of making Phoenyx has actually been figuring out what we don’t want AI to do.
There are topics Phoenyx isn’t allowed to talk about. There are safeguards around conversations and additional rules around how it responds because the person on the other side might be a very young child. Phoenyx has no camera, so it never takes photos or videos. We don’t store children’s audio recordings.
And we wanted parents to be able to see exactly what was happening rather than asking them to trust us that everything was fine.
Parents can read their child’s conversations with Phoenyx word for word. Not an AI-generated summary of what they talked about— the actual conversation. They can also delete those conversations and their child’s data, buttons transparent through their parent app.
Originally, we thought about the transcripts pretty simply: if an AI is talking to your kid, you should be able to see what it said. But parents started using them in a way we hadn’t really thought about.
One parent was reading through her child’s conversations and found her talking to Phoenyx about something that had happened with a friend that day. Her daughter had been upset by it. She hadn’t told her mom. Her mom only realized something was wrong because it came up while she was talking to Phoenyx.
We’ve now heard versions of this more than once, and I keep thinking about it because it wasn’t really a benefit we set out to build. Every parent has probably had some version of the “How was school?” conversation.
“Good.”
“What did you do?”
“Nothing.”
And then three hours later, while their kid is playing with Legos or getting ready for bed, they’ll casually mention the thing that was actually important.
Kids don’t always tell you about their days when you ask them to. Sometimes they talk when they’re doing something else. And apparently, sometimes they talk to a stuffed bird.
That creates an interesting question for us because we obviously don’t want Phoenyx to become a replacement for a parent. We don’t want a child confiding in an AI instead of the people in their life. And we definitely don’t want to turn a toy into some creepy surveillance device that analyzes everything a child says.
But there might be something useful in the middle.
If a child happens to tell Phoenyx that someone was mean to them at school, or that something scared them, their parent has a chance to notice and ask about it later.
That feels meaningfully different from AI trying to interpret a child’s emotions or diagnose what is going on. Phoenyx doesn’t need to decide what the child is feeling. Sometimes a parent just needs the context they didn’t have before.
It has made us think about AI safety a little differently.
Most conversations about making AI safe for children are understandably about what the AI might do wrong. What if it says something inappropriate? What information does it collect? What happens to that information? Can a parent see what’s going on?
Those are still the questions we spend the most time on.
But we’re starting to see another possibility too: sometimes the same transparency we built to help parents keep an eye on the AI can help them understand their kid a little better.
I don’t think we’ve figured out exactly what that should look like yet. There are real boundaries here. Children deserve privacy too, and we don’t want every silly or emotional thing a child says turned into a warning for their parents. Being sad one afternoon is part of being a kid.
But I do think there’s something worth exploring in the fact that children talk differently when they’re playing. They say things they forget to mention when someone asks about their day. And sometimes those little comments give a parent an opening for a conversation they otherwise wouldn’t have known to have.
We started showing parents conversation history because we wanted them to be able to make sure Phoenyx was safe. We didn’t expect that sometimes, it might help them keep their child safe too.
