When Robin Singhvi shipped Gramms AI to the App Store, he hit a question that every developer building for kids will eventually face: what “age-appropriate” actually means in practice, and how to build systems that enforce it reliably. Gramms is a bedtime story app that generates personalized tales narrated in a grandparent’s cloned voice. The safety architecture behind it, Singhvi writes in a practitioner’s account, was anything but simple.
The guide, written for anyone building AI-powered products for children ages 3–10, covers age bands, parental consent, content moderation, and Apple’s review process. It’s not legal advice, but it is hard-won experience from a solo developer who had to answer these questions alone.
Age bands beat a one-size-fits-all approach
The most important design decision, he argues, is abandoning the idea of a single “kid-safe” content level. A story that works for a 9-year-old — with mild conflict and multi-step moral reasoning — may genuinely frighten a 3-year-old. And a 9-year-old will check out in about 30 seconds if the content feels like it was made for a 4-year-old.
Gramms uses three developmental age bands, each with its own prompt engineering layer. Ages 3–5 get simple vocabulary, 200–400 word stories, single-protagonist narratives, and repetitive structure with no conflict that requires resolution. Themes focus on friendship, animals, and magical helpers. Ages 6–8 get richer vocabulary, light challenge and resolution, two- to three-character relationships, and mild suspense that resolves happily, with stories running 400–600 words. Ages 9–10 get near-chapter-book structure, multi-step plots, humor that requires inference, and protagonist agency, with stories reaching 600–800 words.
Related: Rethinking Employability Skills in K–12 Education
These bands are encoded directly into the system prompt sent to the language model. The child’s age isn’t just metadata — it’s a hard constraint that shapes word choice, sentence structure, thematic scope, and narrative stakes.
COPPA compliance means blocking consent gates
COPPA, the Children’s Online Privacy Protection Act, requires verifiable parental consent before collecting personal information from children under 13. In an AI app, “personal information” is broadly interpreted. A child’s name, age, and interests all qualify. Apple reinforces this with its own App Store Review Guidelines, which require explicit disclosure of which third-party AI services process user data.
Gramms surfaces this at the point of parental consent. Before any profile is saved, the parent sees a screen naming OpenAI and Cartesia as AI vendors, explaining what data each receives, and requiring active acknowledgment. It’s not a terms-of-service checkbox — it’s a blocking action gate. Singhvi recommends designing consent as a parent-first flow, with child profile creation gated behind an account an adult controls, not a frictionless onboarding flow a seven-year-old could complete alone.
He also advises storing consent timestamps and vendor lists server-side. Gramms migrated its text-to-speech provider mid-development, which required an audit trail and a mechanism to re-surface consent disclosures to existing users.
Related: Schools nurture tomorrow’s innovators
Two-pass moderation catches what prompts miss
Prompt engineering reduces risk but doesn’t eliminate it. Language models are probabilistic systems, and a carefully constructed prompt can still produce output that slips through. For a children’s product, “mostly safe” is not an acceptable quality bar. Gramms implements a two-pass system: the first pass is prompt-level, explicitly prohibiting violence, fear-inducing antagonists, romantic content, and real-world geographic or political references. The second pass is post-generation moderation, where the full text is evaluated by a content moderation API against a children’s content policy before delivery.
When a story fails moderation, the user sees a simple “generating a new story” message. No error state, no explanation — the regeneration is silent. Singhvi also notes that moderating the narration script separately from the story text matters, since dramatic pacing cues and emotional directions for voice synthesis can introduce tonal elements not present in the raw story.
This level of caution reflects a broader shift in how developers approach children’s AI products. The early days of generative AI were marked by a kind of launch-first mentality, with safety considerations often bolted on after complaints surfaced. The Gramms approach — treating age bands, consent, and moderation as core architecture rather than add-ons — suggests the pendulum is swinging toward a more deliberate model, one where the cost of a mistake is simply too high to justify speed.
Apple review demands vendor-level transparency
Apple’s review process for apps in the Kids category is meaningfully stricter than for general apps, and AI features invite extra scrutiny. Singhvi recommends naming every AI vendor explicitly — not just “we use AI,” but “we use OpenAI’s GPT-4o-mini for story generation and Cartesia Sonic for voice synthesis.” Apple will ask, and the privacy policy, App Store description, and in-app disclosures should all reference the specific companies whose APIs process user data.
Related: Middle School Winners of Project POTUS 2025 Revealed
Child data should be kept out of training. He advises confirming in writing to reviewers that AI provider contracts include data processing agreements prohibiting use of child-associated data for model training. Both major providers offer this, but developers may need to select the appropriate API tier.
The developer also recommends separating child profiles from adult accounts at the data layer. The parent’s account owns the child profiles, and child data should never be independently addressable — only accessible in the context of an authenticated parent session. This architecture is both a COPPA best practice and a common App Review inquiry.
Testing with real children before submission isn’t an App Review requirement, but Singhvi says it’s essential. A 4-year-old’s reaction to a story you thought was gentle is a better signal than any content audit rubric. Building AI products for children isn’t harder than building for adults — it’s differently hard. The technical constraints are manageable, but the design empathy required is substantial. Every age-band decision, every consent flow, and every moderation threshold represents a judgment about what’s appropriate for a specific child at a specific developmental stage.
Families are ready for well-built AI children’s products, he argues. The anxiety isn’t about AI itself — it’s about carelessness. Demonstrating that you’ve thought carefully about safety architecture, named your vendors, built real moderation, and designed consent for parents rather than for frictionless sign-ups is how you earn that trust.
Leave a Reply