Testing Psychometrics


People I know, know what I’m like. I’m easy to read, in that respect, if you know me. People know I’m a bit of a talker, and I can be pretty tenacious with an idea (ie stubborn and loud, OK, I’ll admit it). They also know I write for a skeptical website, and as such I often get presented with things to look at, and explain. Sometimes it’s an ‘explain this then, Mr Skeptic!’, sometimes it’s more of a ‘What’s your thoughts on this?’ In the past I’ve been given contact details of tarot card readers, flyers from religious organisations who pray away HIV (more on that coming soon, by the way), palm readings, horoscopes, Chinese stem cell therapy and everything in between. I really am that easy to read.

Which brings me to an appeal I received from a colleague, who sent me a link to a 5-minute personality test he found on a recruitment website. “Mine was pretty accurate,” is all he said. He’s a man of few words, my colleague. So, obligingly, I gave it a whirl, skeptical as to the claims the site made:

Internet Based Psychometric Testing at Internet Prices…  Professional, personality profiling for recruitment, delivered entirely online… No consultants, no trainers, no sales people… Easy to use, fast and delivered at Internet prices… Complete the personality questionnaire and discover the power and simplicity of PeopleMaps personality profiling for yourself. Read your own report. You be the judge.

Testing at Internet Prices
Professional, personality profiling for recruitment, delivered entirely online.
No consultants, no trainers, no sales people.
Easy to use, fast and delivered at Internet prices.
Complete the personality questionnaire and discover the power and simplicity of PeopleMaps personality profiling for yourself. Read your own report. You be the judge.

“Oh”, says I, “I’ll be the judge alright”. But then again, they knew I was going to say that, they can measure my personality, apparently.

So, I begun taking the test, which essentially required me to rank in order a range of adjectives about myself:

Which of the following is MOST like you?

– Strong and determined

– Enthusiastic and friendly

– Caring and sharing

– Questioning and careful

Erm, well, I’m all of those things, sometimes. None more than any other. But I diligently clicked, and found the next question near-identical:

Which of the following is MOST like you?

– Confident and outspoken

– Exciting and motivated

– Relaxed and supportive

– Methodical and disciplined

Again, at times, I’m most of those things. OK, relaxed isn’t so much me, but supportive really is – which takes precedent in terms of the couplet? I’ve no idea. I was clicking largely at random – take the test yourself if you’ve five minutes to waste – you’ll find you’re often doing the same. Especially when the pairs of adjectives are so tenuously connected.  Objective and principled. Certain and dominant. Personable and fast?! Helpful and informal?! Direct and self-sufficient? Conscientious and industrious? How do any of those pairs fit together?

My guess? They don’t. It’s all window-dressing for what is essentially cold-reading. Which was confirmed when I received my full personality profile, which I’ll present to you in a minute. Essentially, it was nothing more than astrology, horoscopes, Barnum statements. So I thought I’d show you what I got back, but I’ve gone through with a marker pen (by which I mean coloured font, this is the internet after all) and pulled each statement into three categories – green denotes an apparent hit, a character trait that I have; orange denotes a maybe, something that at-a-push I could crowbar into my personality; red is an out-an-out ‘you don’t even know me’, huge miss of a statement.

Marsh will be friendly and open and will place more reliance on relationship than facts and figures. He is animated when talking and will use his hands a lot, even on the telephone!

Consultant’s comments

He asks personal questions on very short acquaintance and wants to build a friendship before moving onto business issues. He enjoys people who stimulate his curiosity and can be fun. He does not want to be bored in any fashion and will quickly close a conversation down if it becomes too detailed.

Marsh prefers people to be light-hearted and fun. He won’t welcome routine, as far as he’s concerned variety is the spice of life. He does not enjoy working alone as he thrives on the stimulation provided by others. He would feel very restricted if he could not converse easily with colleagues in the workplace.

Consultant’s Development Advice

Marsh prefers people to be interesting and fun. He wants them to be like him, keen to share all the gossip! What’s going on in the organisation, up date the news. He believes that people are the most interesting aspect of the job and what’s happening to them is something he likes to keep tabs on. Even after he leaves an organisation he is still interested to find out what past colleagues are doing. This has a positive side as he is always able to source talent and encourage them to join him in new ventures. He would never be at a loss to put a good team together.

That comes to 12 green, 7 orange, 5 red. So from a basic analysis, that looks like a mostly good or average accuracy then? 19 out of 24 statements were either correct, or semi-correct (that’s a neat little stats trick called ‘lumping the middle’. By the same token you could say 12 out of 24 were right, the other 12 were vague or wrong.) But check out the green ones:

  • friendly and open – everyone would say this about themselves, who would say they were closed off and specifically un-friendly?
  • animated when talking and will use his hands a lot, even on the telephone – good hit, score one to the personality test
  • enjoys people who stimulate his curiosity – everyone would say this about themselves, who doesn’t enjoy stimuli and curiosity?
  • can be fun – everyone would say this about themselves, who is out there proclaiming to not be fun?
  • does not want to be bored in any fashion – everyone would say this about themselves, who actually seeks out the feeling of boredom?
  • prefers people to be light-hearted and fun – everyone would say this about themselves, who state a preference to be around boring and depressive people?
  • won’t welcome routine – not many people would disagree, some people might claim a love of routine, but percentage-wise few people would openly welcome routine.
  • would feel very restricted if he could not converse easily with colleagues in the workplace – everyone would say this about themselves, who doesn’t feel restricted by not being able to do something? Note, it doesn’t say the person enjoys being forced to converse, just that they feel restricted if they are banned from doing so. A ban is, by definition, a restriction.
  • prefers people to be interesting and fun – everyone would say this about themselves, who prefers to be around dull and boring people?
  • wants them to be like him – everyone would say this about themselves, it’s a psychological quirk that we prefer to be around people that we have something in common with – note the reading doesn’t state they have to be like the person in any specific way, just in some way.
  • Even after he leaves an organisation he is still interested to find out what past colleagues are doing – this one could go either way, sure, but it’s maybe a 70:30 chance that people like to keep in touch or informed out of idle curiosity.

Now when you look at those 11 greens, you see 8 of them are things pretty much anyone would say was true about themselves, 2 are things that MOST people would agree with, and only 1 is an outright hit (the talking-with-the-hands thing). Even the outright hit is really quite a common trait. What’s more, almost all of the specific claims made turned out to be misses:

  • asks personal questions on very short acquaintance
  • will quickly close a conversation down if it becomes too detailed
  • does not enjoy working alone
  • always able to source talent and encourage them to join him in new ventures
  • would never be at a loss to put a good team together.

So when it’s right, it’s vague; when it’s specific it’s largely incorrect. Handy. But weave it all into the context of a full-on Barnum statement, give it a mumbo-jumbo psychological-sounding word experiment to introduce it and it seems to the unsuspecting to be a very accurate reading.

Still, it’s a bit of harmless fun, right? Well, no – because the testing has real-world consequences, as the website testifies:

Imagine being able to assess the personality of your job applicants in less than 5 minutes – before you’ve even met them!

Oh imagine! Just imagine! In fact, why bother meeting them at all? Why not just get a blindfolded five-year old to pin a tail to them, like a cartoon donkey?! What’s more, the service has indeed been used, it seems:

“We had been looking for a psychometric profiler for recruitment purposes. Most seemed to be orientated to the individual – PeopleMaps’ tests are more tailored to the corporate angle. It’s convenient and quick” – Kathy Cross, HR Manager, Karcher

“Through the use of PeopleMaps I was able to reduce my shortlist of ten candidates from two very strong individuals, one of which we selected with great confidence they would fit into our small team. The profile was amazing, even scary, in its accuracy I have no hesitation in recommending PeopleMaps” – Chris Morgan, Data Liberation

Don’t worry, I’m chasing those testimonials up, just in case they turn out to be false. But vague, meaningless, generic readings are being used to help determine who gets a job and who misses out? When a simple look at the readings with a critical eye shows them to be of the flimsiest nature, yet if you’re not paying full attention you might get taken in by it? Welcome to the amazing world of cold-reading.

,

  1. #1 by Gittins on September 22, 2009 - 11:17

    I had to take a similar personality test at my current employer before starting work here. Total waste of time.

    oh and:
    “who would say they were closed off and specifically un-friendly?”

    I would. Although maybe not in a job interview situation.

  2. #2 by Colonel Molerat on September 22, 2009 - 23:16

    “Why not just get a blindfolded five-year old to pin a tail to them, like a cartoon donkey?!”

    That sounds like the best chance of my getting a new job…

  3. #3 by Andy on September 23, 2009 - 00:05

    Hmmm. Where to start (rolls up sleeves).

    1 Isn’t it a bit unscientific to do take one sample in this way?

    2 you describe those red statements as misses, which they may be but a) if they are Barnum statements how are they also misses? b) what if you are deluded in your opinion of yourself?

    3 If I was recruiting for a data input clerk position, I don’t really want someone who is gregarious and outgoing as a primary characteristic. I want a hermit who will concentrate

    4 The questions are fairly standard psychometric questions. I could look it up but you’ll probably find that the use of mosts and leasts (or in this case “more thans”) is fairly standard fayre (see what I did there?) and does bear a passing resemblance to legitimate psychology. In other words they are no better or worse than many other tests. Is your skepticism about psychometric tests in general or this one in particular, or the marketing and characterisation of it as executed by the pricks at peoplemaps?

    5 Have you had a bad experience with a psychometric test in the past? Would you like to share? 🙂

    6 better go check my shares in Myers Briggs are doing ok

  4. #4 by Andy on September 23, 2009 - 00:15

    On their website they link to ContactCentreRecruitment.com, THis is one of their companies and I note that their website expires in 2 days time. Anyone want to buy it?

    The big problem here is the company marketing the product in such a frivolous way, and with so many bullshit appeals to authority on the site.

  5. #5 by Marsh on September 23, 2009 - 01:30

    @Andy
    Just to answer a few of your questions:

    1. I actually took a look at a few different reports – sections of my colleague’s original one, plus a friend’s, as well as mine. And then I went in, clicked the boxes completely at random, and read that through for accuracy too – it scored about roughly the same as the one where I was answering honestly. I just figured this post was long enough as it is, and I primarily wanted to focus on the detail level of a single report to show how it’s possible to take something that reads as ‘spookily accurate’ (another of the testimonials on the site) down to it’s component assertions, and then examine their accuracy individually. The reports from this particular site felt more like a detailed horoscope than a legitimate psychological evaluation.

    2. The Barnum Effect doesn’t rely on every statement hitting, but by creating an overall feel of accuracy by hitting with pretty general statements. Something for everyone means just that – not everyone will go with everything, but enough people will go with enough statements for the overall feel to come over as accurate. Also, perhaps I am deluded, but here they are again, and from what you know of me, what do you think:
    – asks personal questions on very short acquaintance
    – quickly closes a conversation down if it becomes too detailed (this one, for example!)
    – does not enjoy working alone (as it happens, I’m happy either way, and I work along a lot in my day job, or when writing for this site/researching podcast stories)
    – always able to source talent and encourage them to join him in new ventures (OK, maybe this is borderline because the MSS has a lot of talented people in it, but ALWAYS? Bit of a stretch)
    – would never be at a loss to put a good team together (again, other than the MSS – which has largely put itself together – I wouldn’t really know where to start)

    3. Best way to tell if someone is gregarious or hermit-like? Interview them. Don’t presume to know their personality before they walk through the door. Which leads me to…

    4. Although I’d urge caution in accepting blanket, un-tempered claims of efficacy in these kind of tests, my main issue was this particular website (and the similar type of sites it represents), and the way it sells its reports. It doesn’t say ‘the report may be of additional use alongside standard interviewing techniques’, it says, ‘Imagine being able to assess the personality of your job applicants in less than 5 minutes – before you’ve even met them!’ That’s massively overstating the efficacy of the tool. On top of that, the readings the tool comes up with are pretty generic – so while it may bear passing resemblance to legitimate psychology, it bears very strong resemblance to a cold-read.

    5. Never really had much personal experience of psychometric testing, but from the background reading I did in preparation for this post and the general skeptical nature of assessing extraordinary claims, I think it’s fair to say these tests are as likely to give you an intimate understanding of a stranger’s personality before you ever meet them as graphology, astrology and numerology – all of which have had their heyday with recruitment and HR.

    6. If there’s any shares going cheap, let me know! 😉

  6. #6 by Andy on September 23, 2009 - 09:25

    @Marsh
    “‘Imagine being able to assess the personality of your job applicants in less than 5 minutes – before you’ve even met them!’ That’s massively overstating the efficacy of the tool.”

    Exactement.

    I don’t think that a legitimate proponent of these tests would make the above unqualified statement

    “I think it’s fair to say these tests are as likely to give you an intimate understanding of a stranger’s personality before you ever meet them as graphology, astrology and numerology – all of which have had their heyday with recruitment and HR.”

    The difference with these type of tests between graphology, astrology and numerology is that 1 there is some real psychology behind them (I don’t think I’m being a credulous believer here but am willing to be proven wrong) 2 that the resulting description and profile will vary from profile to profile but if the same answers are given in exactly the same way, the same description should emerge. That is because it is based on an automatic objective routine drawing down on text written by human beings (psychologists) and assigned based on the response variables and rules written by a human being (psychologists).

    I wonder if it’s better to do an initial screening using a consistent measurable set of criteria or to rely on fickle human beings? Everyone knows the first task in finding employment is getting past the paper sift. One large company I know of used to (probably still do) put received CV’s through an OCR to look for the keywords that were used in the advert. If those weren’t there, you didn’t pass the paper sift.

    Finally, we have a new kind of logical fallacy. “Appeal to busy call centre recruitment department”. They are chocka. This very poor website is hoping that the pressure forces them to use this pre screening tool. A concept that, as far as I can tell, has not caught on in the slightest here in Blighty.

    Finally, how the hell did you find a smiley that does your special face???

  7. #7 by Andy on September 23, 2009 - 09:51

    Having re-read my post above, I must say that I haven’t verified that these particular tests do actually follow the science I am referring to. There is an industry based on these automated tests and that process of trained people establishing the various response statements and how they are prompted by the variables.

    I’m not sure if this is an unscientific sub set or the “genuine” article that has been modified down to 10 questions by persons unknown.

    I am with Marsh completely in criticising the presentation of these tests as some kind of silver bullet for call centre recruitment. Their effectiveness is massively, as Marsh says, overstated

  8. #8 by Colonel Molerat on September 24, 2009 - 16:02

    What are proper, scientific psychometric tests used for? I wouldn’t trust one checking on me for a job interview (and it even seems a little frivioulous).
    Do they have any practical use, or are they largely theoretical/experimental?

  9. #9 by Andy on September 25, 2009 - 10:18

    @Colonel Molerat

    Supporters would say that a simple psychometric test will give an interviewer quick insight in the face of a candidate who, potentially, was “putting their best foot forward” during the interview. For example someone who was applying for a direct sales role requiring extensive travel and being away from home, but whose profile suggests a family oriented individual who is turned off by such things. They may be desperate for a job and have changed their search criteria accordingly. The profile will give the interviewer a chance to get a headstart and have their attention up in that area. They might include a question such as “your profile suggeststhat you will be happier when not travelling around. This job requires travelling by its nature, Can you talk to me about the travelling element of your last job and how you balanced that with homelife?”

    Saville and Holdsworth have been in this market for many years and specialise in creating bespoke versions which can be project, team or role related. For example there can be good reasons not to have 2 highly independent, unafraid, opinionated people with low sociability on a small project team. For some companies and for some projects it’s worth investing to get it right.

    The 3rd major application is in the review process. The psychometric tests have become so inexpensive, as evidenced by this topic, that it can be practical to use them during 6 monthly reviews. So imagine a boss with low sociability skills, a low D in the DISC format. If that boss is responsible for people maybe there have been some incidents of gruff behaviour and rubbing up staff the wrong way. A change in the profile score can indicate an improvement, along with other evidence.

    Once again, these don’t measure a person’s life experience, morals or privately held beliefs or any number of other criteria that can lead to behaviour such as environment. But they can indicate changes and they can indicate areas of interest…according to proponents 🙂

    But not as used by the site referred to in Marsh’s article above.

  10. #10 by Andy on September 25, 2009 - 10:46

    This is the type I am most familiar with and I think the test under discussion is a sub set of this family

    http://www.ttidisc.com/products.php?product=discassessment
    There are samples and stuff on the page.

    But Myers Briggs (I’m an ENTJ) and Saville and Holdsworth are big names

(will not be published)