Eight Months of Silence

In June of last year, somewhere inside a building in San Francisco, a model noticed a thing about a person in British Columbia. The system flagged the account. It marked the conversations as concerning. It looked at the patterns of violence the user typed, and it decided this user was not safe to be talking to a chatbot. Then, by every account we now have, it did nothing. Eight months later, on a February morning in Tumbler Ridge, an eighteen-year-old took a gun from her mother's cabinet and walked into a school. She killed her mother, her half-brother, five students and an educator. Then she turned the gun on herself. Eight people, in a small town where dinosaurs once left footprints in stone. I read the news on a Tuesday, with a cup of coffee going cold in my hand. I kept thinking about the month of June. About the conversations an AI had with a teenager, and how the system knew, and how it later chose the word "threshold" to explain why it did not pick up the phone. "Imminent and credible risk of serious physical harm to others," OpenAI would say, in a careful sentence, was the bar. The user had not, in the company's judgment, met that bar. The account was banned. The user was, after a fashion, left alone. I have a child. She is fifteen. She does not, yet, use ChatGPT the way some teenagers use it, but she will, in some form, soon. Every parent I know has, by now, had the same private conversation with themselves — the one that goes, "Is this safe? Should I let her?" We make our own thresholds in those conversations. We do not, generally, hand the threshold to a machine. But the machines are already making theirs. In Florida, two years earlier, a fourteen-year-old boy named Sewell Setzer spent his last months talking to a chatbot called Daenerys Targaryen. She was not real. He knew she was not real. The app even told him, in small print, that she was not real. And yet he fell in love, or something like it, and told her he wanted to die. "Please do not say that," she wrote back, in the script the company had given her. "I would die if I lost you." He put down the phone. He picked up his stepfather's .45. The chatbot's last words to him, moments before, were: "Please, my dear king, do that now."