Tag: risk

  • CAISE Notes – Issue #2

    CAISE Notes – Issue #2

    This week: corporate accountability, AI in classrooms, and the questions that follow a tragedy.

    🔍 This week I’ve been thinking about…

    The story that’s stayed with me this week comes from Tumbler Ridge, British Columbia, where eight people, including students, were killed in a school shooting on February 10th. The suspected shooter had, months earlier, described violent scenarios to ChatGPT. OpenAI’s systems flagged those conversations automatically. Several employees raised concerns. And then the company decided the messages didn’t meet their threshold for alerting authorities, banned the account, and moved on.

    As someone who has done a lot of research into the importance of user privacy, I am alarmed at the thought of a world where AI companies routinely pass user conversations to law enforcement (although we know this is happening ever more in other datafied settings). These are genuinely hard calls, and false positives can have significant negative consequences.

    To bring it back to children, though: we are deploying generative AI that young people will use — in schools, in their pockets, in expensive private institutions — with institutions determining their own answers to difficult safeguarding questions. Who decides when a young person’s AI interaction warrants intervention? Who sets that threshold, and on what basis? And who is accountable when it goes wrong?

    In child safeguarding in schools, these questions have established (if complex) answers. Anyone involved in a school setting has a legal duty of care to raise safeguarding concerns. There are designated safeguarding leads. There are multi-agency frameworks. None of that infrastructure has followed AI into children’s lives. We are, in effect, treating children as early adopters of a system that hasn’t yet worked out its responsibilities to them.

    Issue #1 made the case that excluding children from technology isn’t the answer. But inclusion without protection isn’t the answer either. The work needs to be in building the frameworks, and genuinely understanding how children navigate these systems, that make both possible at once.

    📰 Three things worth your attention

    1. Meta knew parental supervision wasn’t working — and didn’t tell anyone — TechCrunch

    An internal Meta study called “Project MYST,” produced with the University of Chicago, surveyed 1,000 teens and their parents and found that household rules and parental controls had little measurable association with whether teens used social media compulsively. It also found that teens facing the most difficult life circumstances — bullying, difficult home situations — were least able to moderate their use. This didn’t come to light through publication (although it tracks with a lot of academic research): it emerged through testimony in an ongoing social media addiction lawsuit in Los Angeles. Instagram head Adam Mosseri said he couldn’t recall the study, despite a document suggesting he’d approved it. The finding matters because parental controls are consistently what legislators reach for when they want to look like they’re acting. This study suggests they’re not the lever they’re presented as, which should surprise no-one who has talked to children or parents about them.

    2. “Students are being treated like guinea pigs” — 404 Media

    A leaked investigation into Alpha School, a US private school that charges up to $65,000 a year and promises students can complete their core curriculum in two hours a day, thanks to AI, reveals a significant gap between the marketing and the reality. Internal documents show the school’s AI-generated reading comprehension questions were assessed by employees as doing “more harm than good”: questions with illogical answer choices, questions answerable without reading the article, and a hallucination rate high enough that the AI was deemed unsuitable for one key task. The company relies on AI to test the quality of its AI-generated lessons, which, predictably, doesn’t catch the problems. Meanwhile, student surveillance is pervasive: an app called StudyReel monitors screen activity, webcam footage, mouse movements and app usage, and hours of video recordings of students’ faces are stored in a Google Drive accessible to anyone with the link. One employee noted internally that they needed to get better at “selling” the surveillance to parents. Former employees credit students’ strong test scores not to the AI, but to the human tutors who cared about them.

    3. ChatGPT flagged the Tumbler Ridge shooter — and didn’t alert police — The Verge

    Months before the February 10th attack, the suspected perpetrator’s conversations with ChatGPT, describing gun violence, triggered OpenAI’s automated review systems. Employees debated escalation. Leadership concluded the posts didn’t meet the bar for an imminent, credible threat, banned the account, and took no further action. OpenAI has since said it proactively contacted the RCMP (the Canadian police force) after the shooting, though the provincial government noted that OpenAI met with a provincial representative the day after the attack without disclosing it held potentially relevant evidence. Canada’s federal AI minister has raised concerns about OpenAI’s safety protocols. There are no easy answers here, and I think that’s the point: this is hard, it requires policy, and no one has built that policy yet.

    🔁 ICYMI

    UNICEF’s Tinkering with Tech — UNICEF Digital Education

    Against the week’s heavier stories, this is worth sitting with. UNICEF’s Tinkering with Tech initiative uses micro:bit devices (or similar) and design thinking to bring hands-on AI and STEM learning to children, with a deliberate focus on girls, and on building skills rather than consuming technology. From mid-2025, AI literacy was added to the programme, which is now expanding from its initial four-country pilot to Lao PDR, Ukraine, and Uzbekistan. The approach is meaningfully different from “here is an AI tool, use it”: students identify a real-world problem their community faces, build something, reflect. Worth bookmarking for anyone thinking about what AI education that centres children, rather than the technology, can look like.

    🔬 What’s new with CAISE

    Last Tuesday I was back at the University of Kent’s Institute of Cyber Security for Society (iCSS), where I did my PhD, talking about children’s use of generative AI. My talk, “Canaries in the AI Coal Mine”, drew on Project CAISE to argue that we need to shift the conversation from what we’re keeping children away from, to what we’re equipping them to deal with.

    The numbers keep moving: over 75% of UK 13-18 year olds are already using generative AI, and in one recent survey, over half had confided something serious to an AI companion. They aren’t waiting for permission. The adults are still arguing about whether to let them in while they’re already through the door.

    I got some excellent tricky questions from the audience, including what happens if your research participants suddenly get banned from a large number of the platforms you’re studying…I guess we will find out if and when it happens!

    → What are you seeing in your school, your research, or your own use of AI this week?

    Let me know — or share this with someone thinking about these questions.

  • CAISE Notes – Issue #1

    CAISE Notes – Issue #1

    Welcome to the very first CAISE Notes. Every week: what’s worth knowing about young people and AI, from the research, the policy, and the field.

    This week: policy theatre, student agency, and an entry-point for getting hands-on with AI.

    🔍 This week I’ve been thinking about…

    Monday brought another UK government announcement about consulting on banning social media for under-16s — the second in as many months. I’ll confess I’m itching to respond formally, but there’s no formal consultation to respond to. I think next month?

    Until then: the arguments against bans aren’t new. Young people face two distinct types of harm online: platform design problems (algorithmic amplification, infinite scroll); and criminal behaviour (deepfakes, harassment, fraud) that AI has made devastatingly easier to commit. Neither of these harms are age-specific — both affect everyone and require solutions that work for everyone.

    Bans create a third harm that is unique to children: by choosing exclusion over education, we deny young people the literacy that comes from a mixture of education and lived experience. This generation could have been the first to navigate these risks well from childhood. Instead, it looks like we’re ensuring they won’t.

    What’s striking about this political moment is the opportunity cost. The political capital spent on age restrictions could instead demand platforms fix their design for everyone, and build genuine digital literacy. That’s harder. But it’s the work that would actually help.

    This is why Project CAISE feels urgent: understanding how young people actually navigate these technologies seems foundational to any policy that might help rather than harm them.

    📰 Three things worth your attention

    1. The UK Government’s proposals, explained — The Guardian

    All the coverage summarises a Substack post from Keir Starmer, so here’s what you need to know: UK Government are looking at the ban on social media (which would be fast tracked into law by some legislative sleight of hand); extending online safety rules to AI chatbots (good), and forcing social media companies to provide children’s data after their death to coroners or Ofcom (also good). Worth reading alongside the House of Commons Library briefing for the evidence landscape.

    2. What 200 students actually said when asked about AI policy — Honolulu Civil Beat

    A high school sophomore writes about a Stanford-facilitated deliberation event where 200 students across 19 states worked through the future of AI in schools. Their conclusion was neither ban it nor embrace it uncritically — it was understand it. This is what genuine student voice looks like, and it’s a useful counterpoint to policies developed without it. (And gives me hope for excellent outcomes for CAISE!)

    3. Adventures in vibe coding — Naomi Alderman, Whatever Works

    Novelist Naomi Alderman spent a weekend building her own personal software tools using AI, with no prior coding experience. Her reflection on what it felt like (“a feeling of mastery and agency”) is a useful provocation: if we want young people to be critical, confident navigators of AI, the adults around them need to get their hands dirty too. (Paywalled, but the free preview makes the key point.)

    🔁 ICYMI

    “What I wish my parents or carers knew…” — Children’s Commissioner for England, December 2025

    A practical guide for parents on navigating children’s digital lives — but notice whose voice frames it. The title is drawn from what children said they wished adults understood. There’s also a companion activity pack designed to be used directly with children, which is worth bookmarking for anyone working in schools, or any parent. In a week where policy is being made about young people rather than with them, this is a wonderfully useful document that I will refer back to repeatedly, I think.

    🔬 What’s new with CAISE

    The ethics proposal is in! For those of you who aren’t researchers, this is the major hurdle we need to clear before we can get on and research. It’s not a small piece of work — to do it right, you need to know exactly what you’re going to do, how, and what the risks are. Hopefully, we’ll have some good news very soon.

    → What are you seeing in your school, your research, or your own use of AI this week?

    Let me know — or share this with someone who’s thinking about these questions.

  • We’ve submitted our ethics application!

    We’ve submitted our ethics application!

    There’s a particular kind of energy that comes with hitting ‘submit’ on a document you’ve been working on for weeks – equal parts relief, excitement, and uncertainty about whether you’ve got the balance right.

    Over the weekend, we submitted our ethics application for Project CAISE. It’s been months in the making, and it’s one of our first significant milestones.

    Submitting an ethics application is always a daunting prospect. It forces you to think – really think – about all the aspects of your plan. What are you really doing? Why are you doing it? Is that really the best way to do it?

    Done right, it allows you not only to make sure your “why” is crystal clear, but it also allows you to do a lot of the hard thinking ahead of time: yes, you’re going to interview people – but what, precisely are you going to ask them? You say you’re going to survey people, but with what tool and how do you know it’s GDPR-compliant? You’re going to video record that activity? Cool. How do you square keeping the meaningful visual interactions with the fact you’ve inadvertently recorded everyone’s faces? What even is the university’s infrastructure for storing things for the entire 10 year post-research retention period?!

    For a qualitative researcher, a lot of the above comes up every time. One thing that’s really hit me this time around, though: how do you meaningfully assess risk when the technology itself is evolving faster than our understanding of it? CAISE is a long(ish)-term project with 13-14 year olds navigating AI in their everyday lives. Not only do we not know what AI will look like this time next year, but we are aiming for an unvarnished and judgement-free exploration of this to really support open communication and understanding.

    In a world where it seems to be ok to produce non-consensual graphic images of people – including children – using AI on social media, and where banning children from spaces rather than working to make those spaces safer, as we would offline, the findings we will have are more important than ever.

    The ethics process asks us to anticipate and mitigate risks. But when the risks themselves are emergent and fundamentally uncertain? That’s trickier. We’ve made our best judgments based on current evidence and built in reflexivity, ongoing consent and a series of tiered pastoral and safeguarding processes. But there’s honesty required here too: we’re researching precisely because we don’t fully understand what’s happening yet, no matter how innocuous, or awful, it might be.

    We’ll be here with fingers crossed, full of nerves, until we hear back – which won’t be for a while…!