> "I decided to test GPT-4 as follows: I’d take a personality test. Then, I’d give GPT-4 some information about me and ask it to take the same personality test—but pretend to be me. I’d tell it to attempt to predict the ways I’d answer the questions based on what it knows. Then I’d measure the difference between what GPT-4 predicted my responses would be and my actual responses to see how accurate it was. I’d also ask my girlfriend, Julia, who is unusually perceptive about people and knows me quite well, to do the same task. Then I’d compare GPT-4’s accuracy against hers."
I wonder if he gave his girlfriend the information that he "give GPT-4 some information about me". Maybe if the girlfriend saw what he was emphasizing in that information, she could have better guessed what he would have put on the test. Another possibility is that he asked his girlfriend what she thought he would actually do in those situations, whereas he asked the bot to guess what it thought he would put as answers to hypothetical questions (which wouldn't necessarily match what he would actually do in those situations).
To your latter point I did actually two tests with my girlfriend: I asked her to respond "as I would" rather than with what she thought the truth was, and then did another test where I just asked her to say what she thought. She improved when I told her to predict what she thought I would say—and that's the data I used in the article. But she was still significantly worse than the best GPT model
Re the information: I don't think so. The version of GPT that had access to only my most recent tweets performed better than she did, and she reads all of my tweets. In general, I think she has much higher fidelity data on me as an individual than GPT does.
This is unsurprising. It was claimed that the algorithms developed by Cambridge Analytica and FaceBook during the Brexit campaign and the American 2016 election were able to predict behavior better than could friends or spouse:
I wonder if he gave his girlfriend the information that he "give GPT-4 some information about me". Maybe if the girlfriend saw what he was emphasizing in that information, she could have better guessed what he would have put on the test. Another possibility is that he asked his girlfriend what she thought he would actually do in those situations, whereas he asked the bot to guess what it thought he would put as answers to hypothetical questions (which wouldn't necessarily match what he would actually do in those situations).