Do GPT-4.5 and Humans Judge Similarity the Same Way?

In February 2025, OpenAI released GPT-4.5 - the company's largest chat model at the time - as a research preview. According to OpenAI, it was built to sound more natural and conversational than previous models, saying that interacting with it "feels more natural" (OpenAI, 2025).

A month later, GPT-4.5 passed a version of the Turing test at a large scale, meaning people in the study could no longer reliably tell its conversations apart from a human's (Landymore, 2025). If it could pass as human, we wanted to know whether it also "thinks" like one.

Models like GPT-4.5 are far too complex to understand by looking at their individual components. So instead of opening up the model, we took a cognitive modelling approach and applied a test from cognitive psychology to its behaviour. In humans, similarity judgements show how the mind represents and groups things, so they're a natural place to compare the two.

Our questions were: Does GPT-4.5 judge similarity the same way humans do? And if its answers look human, is the process behind them the same?

Similar behaviour does not mean the same underlying process.

We used Michael Lee's flowerpot dataset, which is made up of 16 line drawings that vary on exactly two features, the shape of the pot and the type of leaves.

They were also used in Gati and Tversky's classic 1982 study of similarity which found that when quantitative differences are introduced alongside qualitative ones, the qualitative differences seem more extreme.

The human data already existed through Michael Lee's dataset, in which people had rated how similar each pair of pots looked, on a scale from 0 to 1. For GPT-4.5, we gave it the same image with the prompt:

"Output a similarity matrix comparing the similarity between each of the 16 flowerpots in the image. The result should be a 16 x 16 similarity matrix."

GPT-4.5 returned a full similarity matrix and said it rated the pots based on their "growth stage and pot shape," reading the leaves as a plant getting older.

People gave 75 different scores across the 120 pairs, so almost every pair got its own number. This is because people naturally bring their own interpretation to what "similar" means.

GPT-4.5 was the opposite - apart from two pairs it scored 0, every score it gave was one of five values (1/6, 2/6, 3/6, 4/6 or 5/6). It also rated the pots as more alike overall, with an average of 0.56 vs. 0.48 for humans.

Looking closely at the sixths, we noticed that GPT-4.5 laid the pots out as a 4 by 4 grid, just like the image, and scored each pair by counting how many steps apart they sit.

similarity = 1 - (pot steps + leaf steps) / 6

That rule matches every cell of GPT-4.5's matrix. It is a very mathematical way to compare, yet its scores still correlate strongly with the human scores (r = 0.88 across the 120 pairs). So while GPT-4.5's answers look similar to humans', the process it used to reach them is different.

Next, we turned each matrix into a map by using NMDS, which places the 16 pots on a flat plane so that pots judged alike sit close together. Then we checked the maps against Shepard's law, one of the best-known patterns in psychology.

Shepard's law says that as two things get further apart in someone's mental space, the chance they're seen as similar drops off along a curve, starting quickly then more slowly, which human data usually follows.

Our human data followed this curve closely, while GPT-4.5's didn't follow it as well, sitting closer to a straight line. That possible linearity points to a key difference in how the model forms its mental picture compared with humans.

Hierarchical clustering joins the most similar pots first, then joins those groups until everything is one family tree called a dendrogram.

GPT-4.5's tree is almost perfectly even. It pairs pots, then fours, then all 16, using both features from the very first step. On the other hand, the human tree is messier and leans on the leaves, as people grouped the pots mainly by leaf type.

Late in the project, we found that starting NMDS from a regular MDS solution gave much better fits. We found it after our map figures were already made, which may explain why the clusters on those maps looked so unlike each other, so we treat them with caution.

The bigger open question is the one this project points to, how AI systems decide which features matter and how they turn those features into a sense of distance.