Tag: moderation

  • CAISE Notes – Issue #18

    CAISE Notes – Issue #18

    This week: how do you unpick and maybe even stop young people using technologies in ways that they find supportive?

    I’m trying to think less about social media bans at the moment. I’m bored of it, and if I’m bored of it, everyone else must be too! So, hopefully, for now at least, the stories here will be more AI-based and less social media ban focused.

    Except for the first one today 🙂 This story, about a Youth Advisory Council in Colorado, is a hopeful one because of the example of young people actually engaging with legislators. It’s not unique, and the Children’s Commissioner for England did recently advertise for 16-18 year olds wishing to take part in a Youth Advisory Council for AI. Given the questions over whether young people were listened to at all in the social media consultation process in the UK, let’s hope this group does have a voice; we will see whether the council in Colorado is heard as any legislation progresses.

    The remaining two stories this week, taken as a pair, highlight problems when you don’t listen to what young people are directly (and indirectly) saying. A piece of research coming out of the US shows that young people are using AI chatbots to help process situations where they have suffered online harms. This isn’t new, but shows that the ability to engage with something that talks back (however problematic that is in practice) can be vital in difficult or upsetting situations, where young people can’t, or don’t feel like they can, talk to adults about what has happened. This is not a new situation, of course, but it is a new permutation. If we don’t want chatbots involved (which is reasonable to say the least), how do we support young people without other places to turn?

    And this is where the last article comes in: it is a call to strip out the anthropomorphism in chatbots for young people, removal of small talk, emojis, the human touch. Allow young people to use them for more educational purposes but nothing more. There is a clear case for this, and indeed, this is the case for a number of educational tools already. There are many questions I have about the practicalities of this, but for now, I wonder about the extent to which a young person might turn to a depersonalised tool anyway, if they knew they would get a response. And then I wonder, what might that response be? How do you ensure a young person trying to use a tool to talk about something it is not allowed to is kept safe, handled in the right way to avoid making the situation worse? That’s not to say that it’s working now (by all accounts, it’s not). But it’s worth pondering these edge cases.

    📰 Three things worth your attention

    1. Colorado students want stronger state regulations of AI, social media — Colorado Newsline

    Teenagers on the Colorado Youth Advisory Council spent Friday pitching specific AI and social media policy directly to state legislators at the Capitol. These included requiring platforms to verify parental permission before anyone under 13 can open a social media account. The minimum age is already a requirement of US law, but, as we all know, it’s just not enforced. Enforcement, not the rule itself, was Pettinger’s point: without real penalties for platforms that ignore it, the age limit is decorative.

    Another recommendation was to allow under-18s to opt out of algorithmic recommendations entirely. Colorado’s Youth Advisory Council has no bill-drafting power anymore, after legislators nearly cut the program in 2025 budget talks. Whether any of this reaches a statute is unknown, but the ability of young people to interact directly with policy-makers is significant.

    Their suggestions also reiterate (for the millionth time globally) how young people want a future where the limitations sit not with them, but with the tech companies themselves.

    2. As AI becomes part of kids’ response to online harm, new Thorn research finds children turning to chatbots for guidance after online sexual interactions — PR Newswire

    New research from Thorn, a child-safety nonprofit, has found that AI chatbots are now part of how children process online harm. In a 2025 survey of over 1,000 US minors aged 9 to 17, 67% had used an AI chatbot or companion. Among the 27% who reported an online sexual interaction, 15% turned to a chatbot for guidance afterward, more than double the 7% who did the same after being bullied.

    Thorn notes that kids turning to chatbots is a sign they “feel comfortable” seeking help there, not evidence the tool is doing that job well. The report doesn’t claim chatbots are handling these disclosures safely. It claims children are handing them the job by default. This matches other reports (such as that released by Common Sense Media earlier in the year) that those young people who don’t have, or don’t feel like they have, a trusted adult to talk to will turn to AI instead.

    Fear and shame remain the biggest reasons children don’t disclose harm at all, a pattern Thorn says has been growing since 2020, and it’s now showing up in preteens as much as teens. As with so much of the concern around young people’s digital technology use, this is a far more complex issue: the more vulnerable and isolated a child is or feels, the more likely they will use this technology. This is exactly the point that Candice Odgers made in her TED talk: building, not banning, by (amongst other things) allowing easier access to human counsellors and support, could help reduce these figures.

    3. AI Chatbots for Kids Should Act Like Tools, Not Friends — The Fulcrum

    An alternative suggestion: this piece in The Fulcrum makes the case for making chatbots less warm, not more responsive. We all know the arguments against anthropomorphism. For adults this is a UX preference. For children, whose ability to distinguish simulated care from a real relationship is still developing, the piece argues it’s closer to a hazard.

    The piece cites a study in which neither general-purpose nor companion chatbots pushed back when a simulated teenager described wanting a large tattoo, a solo trip into the woods with no plan, or getting drunk quickly.

    The proposed fix is narrower than an outright ban on AI companions for minors. It’s a non-personified default setting for under-18 accounts, one designed to fail the Turing test on purpose, while leaving general chatbot use for homework and creative work intact.

    It’s a reasonable answer to the anthropomorphism problem specifically. Read next to the piece above, though, it sits awkwardly: the warmth this piece wants to strip out may or may not stop a child from talking to a chatbot if they simply need to say something to something that responds at all. Making chatbots colder might reduce one common risk while removing a stopgap measure for a more vulnerable child.

    🔁 ICYMI

    New EU rules on AI deepfakes and chatbots — RTE

    New transparency rules under the EU AI Act came into force on 2 August. AI-generated deepfakes now have to be labelled, companies must disclose when someone is talking to a chatbot rather than a human, and even AI-generated text on general-interest topics needs a label if there was no human editorial oversight. Breaches can carry heavy fines. AI law specialist Barry Scannell put the shift in Turing-test terms: for decades the goal was making AI convincing enough to pass as human, and these rules exist because that goal was reached.

    🔬 What’s new with CAISE

    We are using the summer to do a number of things: carry out our review of media and policy in the English-speaking world, analyse the school data we have already, prepare for next term, hopefully tidy up the website…and take leave before we lose it! Youyue is also juggling this with a summer internship.

    Who said that the summer was quiet?!

    → What are you seeing in your school, your research, or your own use of AI this week?

    Let me know, or share this with someone who is trying to figure it out.

  • CAISE Notes – Issue #2

    CAISE Notes – Issue #2

    This week: corporate accountability, AI in classrooms, and the questions that follow a tragedy.

    🔍 This week I’ve been thinking about…

    The story that’s stayed with me this week comes from Tumbler Ridge, British Columbia, where eight people, including students, were killed in a school shooting on February 10th. The suspected shooter had, months earlier, described violent scenarios to ChatGPT. OpenAI’s systems flagged those conversations automatically. Several employees raised concerns. And then the company decided the messages didn’t meet their threshold for alerting authorities, banned the account, and moved on.

    As someone who has done a lot of research into the importance of user privacy, I am alarmed at the thought of a world where AI companies routinely pass user conversations to law enforcement (although we know this is happening ever more in other datafied settings). These are genuinely hard calls, and false positives can have significant negative consequences.

    To bring it back to children, though: we are deploying generative AI that young people will use — in schools, in their pockets, in expensive private institutions — with institutions determining their own answers to difficult safeguarding questions. Who decides when a young person’s AI interaction warrants intervention? Who sets that threshold, and on what basis? And who is accountable when it goes wrong?

    In child safeguarding in schools, these questions have established (if complex) answers. Anyone involved in a school setting has a legal duty of care to raise safeguarding concerns. There are designated safeguarding leads. There are multi-agency frameworks. None of that infrastructure has followed AI into children’s lives. We are, in effect, treating children as early adopters of a system that hasn’t yet worked out its responsibilities to them.

    Issue #1 made the case that excluding children from technology isn’t the answer. But inclusion without protection isn’t the answer either. The work needs to be in building the frameworks, and genuinely understanding how children navigate these systems, that make both possible at once.

    📰 Three things worth your attention

    1. Meta knew parental supervision wasn’t working — and didn’t tell anyone — TechCrunch

    An internal Meta study called “Project MYST,” produced with the University of Chicago, surveyed 1,000 teens and their parents and found that household rules and parental controls had little measurable association with whether teens used social media compulsively. It also found that teens facing the most difficult life circumstances — bullying, difficult home situations — were least able to moderate their use. This didn’t come to light through publication (although it tracks with a lot of academic research): it emerged through testimony in an ongoing social media addiction lawsuit in Los Angeles. Instagram head Adam Mosseri said he couldn’t recall the study, despite a document suggesting he’d approved it. The finding matters because parental controls are consistently what legislators reach for when they want to look like they’re acting. This study suggests they’re not the lever they’re presented as, which should surprise no-one who has talked to children or parents about them.

    2. “Students are being treated like guinea pigs” — 404 Media

    A leaked investigation into Alpha School, a US private school that charges up to $65,000 a year and promises students can complete their core curriculum in two hours a day, thanks to AI, reveals a significant gap between the marketing and the reality. Internal documents show the school’s AI-generated reading comprehension questions were assessed by employees as doing “more harm than good”: questions with illogical answer choices, questions answerable without reading the article, and a hallucination rate high enough that the AI was deemed unsuitable for one key task. The company relies on AI to test the quality of its AI-generated lessons, which, predictably, doesn’t catch the problems. Meanwhile, student surveillance is pervasive: an app called StudyReel monitors screen activity, webcam footage, mouse movements and app usage, and hours of video recordings of students’ faces are stored in a Google Drive accessible to anyone with the link. One employee noted internally that they needed to get better at “selling” the surveillance to parents. Former employees credit students’ strong test scores not to the AI, but to the human tutors who cared about them.

    3. ChatGPT flagged the Tumbler Ridge shooter — and didn’t alert police — The Verge

    Months before the February 10th attack, the suspected perpetrator’s conversations with ChatGPT, describing gun violence, triggered OpenAI’s automated review systems. Employees debated escalation. Leadership concluded the posts didn’t meet the bar for an imminent, credible threat, banned the account, and took no further action. OpenAI has since said it proactively contacted the RCMP (the Canadian police force) after the shooting, though the provincial government noted that OpenAI met with a provincial representative the day after the attack without disclosing it held potentially relevant evidence. Canada’s federal AI minister has raised concerns about OpenAI’s safety protocols. There are no easy answers here, and I think that’s the point: this is hard, it requires policy, and no one has built that policy yet.

    🔁 ICYMI

    UNICEF’s Tinkering with Tech — UNICEF Digital Education

    Against the week’s heavier stories, this is worth sitting with. UNICEF’s Tinkering with Tech initiative uses micro:bit devices (or similar) and design thinking to bring hands-on AI and STEM learning to children, with a deliberate focus on girls, and on building skills rather than consuming technology. From mid-2025, AI literacy was added to the programme, which is now expanding from its initial four-country pilot to Lao PDR, Ukraine, and Uzbekistan. The approach is meaningfully different from “here is an AI tool, use it”: students identify a real-world problem their community faces, build something, reflect. Worth bookmarking for anyone thinking about what AI education that centres children, rather than the technology, can look like.

    🔬 What’s new with CAISE

    Last Tuesday I was back at the University of Kent’s Institute of Cyber Security for Society (iCSS), where I did my PhD, talking about children’s use of generative AI. My talk, “Canaries in the AI Coal Mine”, drew on Project CAISE to argue that we need to shift the conversation from what we’re keeping children away from, to what we’re equipping them to deal with.

    The numbers keep moving: over 75% of UK 13-18 year olds are already using generative AI, and in one recent survey, over half had confided something serious to an AI companion. They aren’t waiting for permission. The adults are still arguing about whether to let them in while they’re already through the door.

    I got some excellent tricky questions from the audience, including what happens if your research participants suddenly get banned from a large number of the platforms you’re studying…I guess we will find out if and when it happens!

    → What are you seeing in your school, your research, or your own use of AI this week?

    Let me know — or share this with someone thinking about these questions.