Have you ever noticed that you rarely (if ever) meet someone who admits to having below average driving skills? My Grandfather started driving a car before UK driving tests became compulsory in 1935. So he never had to take a driving test. This was lucky for him, because he was a terrible driver. He would get distracted and cross the line into oncoming traffic, veering back at the last second when his passenger started shouting. He claimed he had never been in accident, but I expect he would have seen quite a few if he had ever thought to look in his mirrors. Few people would accept a lift from him a second time. Even as a young boy, I realised that I was in mortal danger getting into his car. I would make an excuse and make my own way by bicycle. And yet, he considered himself a good driver. After having read the excellent book Bad science by Doctor, journalist and blogger Ben Goldacre, I think I know why.
The book contains this startling graph from the paper ‘Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments’ by Kruger and Dunning :
Graphics courtesy of explorativeapproach.com (click to enlarge)
The graph shows how good subjects were at logical reasoning as measured by a test (blue), how good they thought they were (green) and how well they thought they did at the test (red). Apart from the top quartile, there was actually an inverse relationship between how skilled people thought they were and how skilled they actually were. The study also showed that the least competent individuals were also the least capable at recognizing the skill levels of others. This is the abstract from the paper:
People tend to hold overly favorable views of their abilities in many social and intellectual domains. The authors suggest that this overestimation occurs, in part, because people who are unskilled in these domains suffer a dual burden: Not only do these people reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the metacognitive ability to realize it. Across 4 studies, the authors found that participants scoring in the bottom quartile on tests of humor, grammar, and logic grossly overestimated their test performance and ability. Although their test scores put them in the 12th percentile, they estimated themselves to be in the 62nd. Several analyses linked this miscalibration to deficits in metacognitive skill, or the capacity to distinguish accuracy from error. Paradoxically, improving the skills of participants, and thus increasing their metacognitive competence, helped them recognize the limitations of their abilities.
Or, more succinctly:
Ignorance more frequently begets confidence than does knowledge. Charles Darwin
The findings fit in with the concept of ‘unconsciously incompetent’ in Maslow’s four stages of learning:
- Unconscious Incompetence: The individual neither understands nor knows how to do something, nor recognizes the deficit, nor has a desire to address it.
- Conscious Incompetence: Though the individual does not understand or know how to do something, he or she does recognize the deficit, without yet addressing it.
- Conscious Competence: The individual understands or knows how to do something. However, demonstrating the skill or knowledge requires a great deal of consciousness or concentration.
- Unconscious Competence: The individual has had so much practice with a skill that it becomes “second nature” and can be performed easily (often without concentrating too deeply). He or she may or may not be able to teach it to others, depending upon how and when it was learned.
The Dunning-Kruger effect might explain why every crank and barstool scientist appears to think they understand global warming better than the world’s top climate scientists. I am also reminded of my own experiences learning ju-jitsu. After a year or two of training, having reached the exalted rank of green belt, I thought I was pretty good. It was only some years later, as a black belt, that I realised how much I still had to learn.
What has this got to do with software? Well, it might explain some of the very poor website design, GUI design, graphics and copywriting I see from time to time. The perpetrators may be sufficiently unskilled, that they don’t even realise how unskilled they really are. It is a thorny problem. Firstly you don’t realise you have the problem. Secondly, even if someone convinces you to delegate or outsource that type of work, you are unable to accurately assess the work of others. It is particularly worrying for one man software companies (such as myself) who have to perform or oversee a very wide range of skills including: website design, user interface design, programming, testing, copyrighting, marketing, PR, documentation, support and systems admin.
By the same token, many customers with poor IT skills might not have any insight into the extent of their deficit. Below are the results of a survey I did with some of my own customers a couple of years ago:
Note how skewed the results are and bear in mind that relatively few of my customers are IT professionals. Similarly skewed results were reported recently by Patrick McKenzie in his blog:
Surprisingly many of my customers self-evaluate as comfortable with computers. 50% were “very comfortable”, and 30% were “mostly comfortable”. These numbers are, candidly speaking, not what I would have assigned on the basis of reading support requests for three years.
I don’t have any easy solutions for this problem. All you can do is:
- accept that you might not be the best judge of your own competence in all areas
- actively solicit feedback from your customers and your peers and listen carefully
- be a bit more tolerant of newb@aol.com when they blame your software for problems arising from their lack of basic IT knowledge
- console yourself that, whatever your IT shortcomings, at least you are a good driver
Successful Software 
Another reason why As hire As, while Bs hire Cs who hire Ds… Also seems correlated to an organization’s headcount, regardless of the divisions of labor.
Whenever I hear the “A hire As, while Bs hire ….” statement, I always wonder, who does hire the Bs for the downhill spiral to start? :)
And by the way, doesn’t every developer think that they’re an A?
Most of them are wrong of course (although I’m not, obviously :-).
Bs are Cs that learned something somewhere on their way.
Very interesting, almost explains why I thought I was a *great* developer 2 years ago. But now that I am so much better than back then, I am not so sure if I am actually that good.
Maybe I am seeing the 3rd quartile ‘dip’ of perceived ability, or at least on my way there. Good news overall :-)
Another excellent and thought provoking article in the series of Successful Software — thank you Andy.
Continuing Sachin’s line of thought, it would be interesting to see how the above data/graphs depend on *age* (of subjects).
One could easily argument that people who participate in the last survey aren’t your typical customers, but more engaged ones.
@Rafael
Yes, that is possible. But studies show such a skew is common, even when the survey isn’t self selected. Google ‘superiority effect’.
Senior management brings in the Bs, middle management the Cs, and HR the rest.
Pingback: Chuck Brooks For FutureWare » Blog Archive » Getting What You Pay For With Free Software
I enjoyed reading this and recognize my own development as do a lot of others I guess.
Ten years ago I knew I was a good developer, now I know I’m an average developer at best. This insight has made me more humble and thus (I think) I learn more easily from other people and see the good in their ideas and design.
It’s a bit funny: you have to realize that you’re on an average skill level in order to be able to learn more effectively.
The funniest part is when people claim to be good developers when they have no intention of improving upon themselves. I have seen a few 9-5 workers who consider that on the job experience has made them into excellent software developers when the truth seems far from that.
Getting feedback from customers, peers is probably the best way to gauge your level. The next is to accept that you may not be as good as you think and improve upon it.
Pingback: public reflection » Blog Archive » 2 -
Pingback: Blog harvest, February 2010 « Schneide Blog
Pingback: RebelFa1con™ | Bilimsel Cevaplar
Very interesting and informative article. I like this stuff.
I agree with all the points you make–I see the phenomena all the time in my “day job.” I also recognize I’ve learned enough about programming to recognize all sorts of things I don’t know :-)
For what it’s worth, though, to me your graph doesn’t seem to really support the conclusion you presented. When I look at the graph I don’t see an inverse relationship. I see everyone more or less self-assessing as equally knowledgeable–everyone sees themselves as experts, whether they are in the top or bottom quartile.
@Al C
If you look at the green line and blue lines you will that there is an inverse realtionship betweeb actual and perceived ability for the bottom, 2nd and 3rd quartile performers.
interesting spelling on the picture. “Worts”
An interesting branch of this is the intelligence parents have in the eyes of their children.
From birth to pre-teen parents start out as knowing everything but gradually deteriorate into becoming morons.
During the teen years parents are pretty near retarded and may as well have had a full lobotomy.
At around 20 years old parents start to become brighter until…
From about the mid-twenties onwards we are credited with the level of intelligence we actually have. Or at least somewhere in that general vicinity.
I KNOW I SUCK!!
Pingback: 10 things non-technical users don’t understand about your software « Successful Software
Reminds me of an old saying: Beware the artisan that claims 20 years experience when he really has 1 year of experience, 20 times. You can learn something new everywhere – just need to be open to it.
MEN ARE FOUR (not to leave out women)
THERE ARE MEN THAT KNOW NOT AND KNOW THAT THEY KNOW NOT
THERE ARE MEN THAT KNOW NOT AND KNOW NOT THAT THEY KNOW NOT
THERE ARE MEN THAT KNOW AND KNOW NOT THAT THEY KNOW
THERE ARE MEN THAT KNOW AND KNOW THAT THEY KNOW
If a man knows not and knows that he knows not
he is a student, teach him
If a man knows not and knows not that he knows not
he is asleep, awaken him
If a man knows and knows not that he knows
he is a fool, shun him
If a man knows and knows that he knows
he is a leader, follow him
I have been using a website to sell my products for about ten years. I also have a four degree in art and design. I have wasted thousands of dollars on what I call scammers.
I have noticed that there are a lot of folks out there that learn HTML
and then suddenly think they are designers. They can convince most folks that they are designers because they know technology and the others don’t. Before I hire anyone I want to see all the websites they have designed and go from there. NEVER hire anyone with references or seeing a good volume of work they have created.
I took a driving school once and the teacher said “90% of drivers think they are above average.”
It occurred to me that those 90% could possibly be right.
Consider:
* give the test to 10 people.
* 1 person scores a 4
* 9 people score a 6
* the average (9*6+1*4)/10 is 5.8
# conclusion 90% of these _are_ above average
Sadly, this is true even if they all score abysmally.
* 1 person scores 1
* 9 people score 2
* average is (9*2 + 1*1)/10 == 1.9
# 90% are above average, (but the average is horribly low)
-Jesse
I would have given your question about computer skill an intermediate ten years ago and expert about twenty-five years ago. Now I would consider myself a novice, though I have had experience in many languages, OSes, computers, etc. It isn’t just that I’m getting smarter about my ignorance, but there was a time when I had comparatively excellent computer skills on several different makes of computers, etc.
But now I’m struggling to stay abreast of a small portion of the wave. I no longer use JCL, Fortran, IBM 370s, etc. Just trying to grab hold of a good Linux distro (and I almost forgot what to call it) to use Python (much less Zope) makes me feel like a pre-novice.
So what am I? Novice? Expert? How about ten years from now? What will you be?
…and I’m a lousy driver.
Summed up very well in this quote.
“A man’s got to know his limitations.” – Dirty Harry
To follow up on Dave’s Oct. 6 comment:
To me, the most important piece of the Kruger/Dunning article is their statement:
The incompetent can gain insight about their
shortcomings, but this comes (paradoxically) by making them
more competent, thus providing them the metacognitive skills
necessary to be able to realize that they have performed poorly.
What that means, and what they demonstrate in an experiment in their paper, is that effective teaching/training relies on two parts – the content and the metacognitive role of knowing about knowing. The metacognitive aspect describes that phenomenon we have about getting better at something while simultaneously feeling not as competent about it. Self-reflection reveals while confidence blinds. But when we work with someone, success is best when we teach/train what they are supposed to know along with how they are supposed to know it.
Another great article (in case you needed external validation).
I find it better (when asessing customer’s IT skills) to ask objective questions: Have you downloaded software before (rather than “can you”).
Maybe I should be asking “and were you able to find the file you downloaded?”.
PS
I noticed something similar while teaching: the best students were the MOST worried about any mistake they made (“I want to know what the right answer is”) while the worst were unphased by repeaded failures. This perplexed me until I realized I had cause-effect reversed:
Those that worried the most were the best. The WORRYING facilitated the success. (Or to quote Any ____ of Intel: Only the Parnoid Survive.
I read somwhere that Steve Jobs does is market research by carrying around a mockup and just asking (the right) people what they think of it.
Likewise, I have NEVER trusted my own judgement and luckily have always (for my mISV) had my wife available for a second opionion. And it does matter who you ask. Not all opinions are equally valid.
Pingback: Bilimsel Cevaplar |