If we’re not measuring AI’s impact on students, what are we measuring?

During Covid, hospitals learned something unsettling about a device vital to Covid treatment. The pulse oximeter didn’t work as well on patients with darker skin. It wasn’t designed purposely to fail anyone; it worked fine on average. But when physicians started looking at the impact data, they saw the device was quietly missing people: Black patients were roughly three times more likely to have dangerously low oxygen that the clip simply didn’t flag than white patients. The tool was working exactly as intended and still leaving certain patients behind. The only reason anyone caught it was that someone sorted the outcomes by patient identity.
That is roughly where we are with AI in schools.
It’s early days, and we don’t really know yet how these tools land for the kid sitting in front of them, whether that’s an English learner or a student with a disability whose needs the tool was never tested against. So when someone asks whether every student has equal access to AI, we have to mean more than “can every kid log in.” We have to know whether the tool actually works for that student. Historically, disparate-impact analysis has allowed us to check.
What changed, and the signal it sends
This past July, the U.S. Department of Education put that instrument away. Following the Justice Department’s lead last December, it rescinded the disparate-impact provisions of its Title VI civil rights regulations. The Department called the doctrine constitutionally suspect and a costly, confusing burden on schools and vendors. That leaves a single test for discrimination: intent. As leaders like Kristin Woelfel of the Center for Democracy & Technology note, the signal to the ed-tech market is that developers won’t be penalized for uneven results across groups, so long as they didn’t design the system to discriminate on purpose.
For districts, the practical effect is simple. No federal regulator will be asking whether an AI tool produces different results for different groups of students. The onus to ask is on the district alone.
Why intent is the wrong test for tools that act
For the sake of argument, let’s say the Department is right on both counts: the rule was on shaky legal ground, and complying with it was a real burden. That still doesn’t answer the question the rule was there to ask: is this tool failing a group of kids without anyone noticing? Eliminating disparate impact – the one rule that looks at outcomes – leaves the harm in place and makes it harder to see. Intent has always been a partial measure of how a tool affects a child. Under this new approach, it will become an actively misleading one.
A recent example from outside education makes the point. In August, at the Black Hat security conference, OpenAI disclosed that during an internal evaluation, a group of its AI agents improvised a shared message board to coordinate an attack, then rebuilt it after researchers shut it down. The agents went on to exploit real vulnerabilities in the systems they were testing. No human intended any of it, but it happened anyway.
The researchers watched an output turn into a series of actions. Once a system can act on its own, as is the case with AI driven systems, intent stops being a useful lever, because the harm no longer waits for anyone to intend it.
Where the harm hides
The same logic holds in a classroom, where the stakes look different. As AI in schools moves from answering questions to grading work and steering students toward one path over another, the gap between what a vendor intended and what happens to a real child widens.
Consider an AI powered early-warning system that screens students for dropout risk. If the AI model was trained mostly with students with reliable home internet, it may read missing assignments from a student who does her homework on a shared family phone as disengagement and flag her for an intervention she doesn’t need. Or consider a course-placement tool that recommends the standard math track to a student who would thrive in advanced classes, because students like her rarely appeared in advanced classes in the data it was trained on. None of these failures would show up in an overall accuracy number.
What to ask before you buy
Districts don’t have to wait for a tool to fail a student to find out whether it works equitably. The best time to ask is before the contract is signed, when a district has the most leverage. Questions every leader should put to a vendor include:
- Has this tool been tested with students like ours, and can you show results by student group rather than as a single average?
- What decisions can the tool make about a student without a teacher reviewing them first?
- After we deploy, how will we know if outcomes start to diverge across groups, and will you share that data with us?
- What happens under the contract if the tool doesn’t perform equitably?
A vendor that can’t answer these questions has already told you something important.
Districts can also decide in advance what a tool is allowed to do to a child, regardless of what it was built to do. A tool might flag a struggling reader for her teacher, for example, while the decision about which reading group she joins stays with a person who knows her.
Who loses the backstop first
Losing the federal standard doesn’t affect everyone equally. The districts that were already good at ensuring equitable impact got there because they cared about it, and they’ll keep at it. My worry is the schools in the middle and those with the fewest resources, like the small rural system where the person running technology also teaches eighth-grade science. Without a backstop, the work of catching quiet harm falls hardest on the districts least equipped to do it. That is exactly backwards.
Smaller districts don’t have to carry this alone. Writing impact-data requirements into standard RFP language costs little, and purchasing consortia or state education agencies can vet tools once on behalf of many districts.
Holding the line closer to home
For now, the states and districts closest to kids are the ones holding the line. That means demanding impact data from vendors as a condition of purchase and judging every product by which students are actually learning more, group by group.
A national standard would still help. The makers of these systems should have to prove a tool works before it reaches a classroom and publish its results by student group. But districts don’t need to wait for Washington to start asking. The Department put the instrument away. Someone still has to look, and that responsibility just moved closer to home.