Valid Measures of Intellectual Competence??Summary Existing measures of intellectual competence, such as IQ tests and academic examinations, leave much to be desired. Even among persons of high IQ and highest academic qualifications there are major differences of opinion about many subjects. It is suggested that the only proper criteria for validation of measures of judiciousness are predictions of real events, and devising and testing in reality of devices. Other measures of competence would involve questions that would be excluded from IQ tests because of the controversiality of the answers. There may be practical problems with all these techniques but there appears to be no substitute. Valid Measures of Intellectual Competence?? Our society maintains a vast machinery of assessment of individuals, for the allocation of educational and employment opportunities. The desirability of selecting competent higher civil servants and competent national leaders hardly needs arguing. Yet there has been relatively little by way of assessment of most methods of assessment, and such as there has been returns a largely negative verdict (Baird, 1985). This paper suggests means of providing a valid basis for this institution. I do not report here any new empirical investigation, which may seem surprising, but the reason for this will become apparent further on. The Theoretical Background The origins of the present ideas lie in a fundamental question that I posed myself about 1981, a question that others have been content to answer without reflection. It is as follows. There are a great many questions of importance that are matters of controversy and the opinions expressed cannot all be correct. In this circumstance, how is one to decide whether one person's opinion is more correct or more realistic than another's? By considering the details and forming an opinion for oneself? But why should one give special preference to one's own opinion just because it comes from oneself? I suggest that a passable reason for doing so that stands above pure conceit would be the possession of an explanation for why others have come to their differing conclusions. So this personal-practical problem of mine leads to a theoretical question: why do people form these divergent opinions? Is it just a matter of random ideation, or is there some systematicness in the opinions? I formed the provisional opinion that at least three tendencies could be distinguished in opinions, related to the potential source of the ideas: realism, from outside, wishful thinking, from within, and conventional wisdom, from the communications of other people. Individuals appeared to differ in their balance of these tendencies. This conception led me subsequently to the production of a theory of, among other things, IQ, genius, and autism, and also to a theory of the rises and falls of civilisations and the part therein played by authoritarianism and genius. These in turn raised questions about how competence in opinion-forming can be validly assessed. I shall hereinafter use the term judiciousness to indicate this competence in forming opinions that are realistic, or correct in some objective way. Existing Methods Such as they are, existing methods of assessing competence roughly fall into two categories, the one (e.g., examinations) deriving from tradition, the other (e.g., IQ tests) from psychological science. The essential problem with the traditional methods of assessment is a lack of grounding in direct reality. Examination answers of almost all kinds are assessed entirely by one or both of two criteria: (1) accuracy/quantity of reproduction of given information; (2) the examiners' opinions of 'judiciousness'/'realisticness'/'correctness' (of presentation, selection, interpretation, evaluation, calculation, etc). Now accuracy and quantity of recall are not without value, especially in some fields such as those involving foreign languages, but they are surely of far less importance generally than judiciousness, and there is no reason to suppose that they give any indication of this latter. But who is to say what is judicious/realistic/correct? It is just the opinion of one or more examiners. Even if the examiners' assessments concur (and in many fields the degree of concurrence is very low) that may be because they concur in their prejudices rather than through any relationship to reality. With many of these procedures there is a further equally serious problem. This applies particularly to those examination questions of the form "Discuss....." or "Evaluate.....", and is equally applicable to peer review among academics; it has little application to tests in maths, physics, or language competence. This problem is that even suppposing--rather optimistically--that the assessors really are perfectly judicious, nevertheless they will have difficulty distinguishing from real judicious talent those who are skilled in imitating/conforming to the judgements of their superiors. Given that imitation is much easier to come by than judiciousness, the inevitable consequence is that in time, real judicious talent comes to be displaced by authoritarian mimicry. Thus arises the phenomenon of institutional decadence, further considered in the aforementioned theory paper. At first thought IQ tests may seem to be equally open to the charge of lack of grounding in reality criteria; however, replicated findings of correlations with basic parameters of EEG waves, and, more importantly, with non-cultural psychological measures (Jensen, 1982; Brand & Deary, 1982), imply some sort of substantial objective validity (these findings are given an integrated explanation in the aforementioned theory). But even though IQ tests are provenly valid measures of mental speed and accuracy and the consequent accumulated skills and knowledge, there is surely more to judiciousness than these qualities. There is no logically necessary correlation between intelligence (IQ) and judiciousness; even a particularly fast and perfectly reliable computer could give incorrect performance if the software were defective. The most important limitation of IQ tests must surely be that they concentrate on non-controversial questions and steer clear of those very controversies that reveal wishful-thinking, conformity, etc.. It seems likely that IQ tests measure little or none of the qualities specified in my theory of genius: wishful-thinking, pretentiousness, conformity, and presentminded attention to sensory input rather than absentminded concentration on thoughts. Do we not all know of persons who are highly intelligent and yet--we think--very foolish? Valid Measures of Judiciousness The only proper criterion by which to distinguish real judiciousness from supposed judiciousness must surely be the making of a direct test against reality itself rather than against one or more person's opinions of reality. It appears that the means of doing this are of only two kinds: prediction of what will or will not happen followed by checking what does or does not actually happen; and (a subset of the first, strictly speaking) devising of real means to bring about a specified end, followed by a real demonstration of whether or not the end is thereby achieved. I shall call these predictions and devisings respectively. A possible model for the application of these principles is provided by tests of intelligence. Therein individuals undertake a battery of tasks, such as the numerous items of a pencil-and-paper test. Factor analyses of the responses of large samples of individuals reveal one general factor, a number of special factors, and residual variance specific to each of the individual items. We can only guess whether there may likewise be one or more general factors of judiciousness, only partially tapped by any single task of prediction or devising. Conceivably we would find the known factors of ability along with factors of wishfulthinking, conformity and superficiality. It seems reasonable to expect that as with IQ test items, judiciousness test items would vary in degree of difficulty, representativeness, dependence on special or local knowledge, and amenability to coaching. It appears, however, that there is a major difference from IQ tests in the matter of practicality; it appears that items for testing by predictions and devisings would be relatively demanding of time and expertise, and furthermore that the supply of alternative items may be somewhat limited (perhaps some persons with more imagination could dispose of that problem). If these measures do indeed prove to be time-consuming, expensive, or limited in useful supply, then it may be best to confine their use to two purposes: (1) selection for higher-level posts; (2) validation of other, more convenient means of assessment. These more convenient measures could take the form of a battery of questions such as: "Is IQ hereditary?"; "Should this country have unilateral nuclear disarmament?"; "Is Keynsianism better than Monetarism?"; "What do you think the world population will be in 2020?"; "Who would have made the best President/Prime Minister in the 19__ election?". The essential requirement of the questions is controversiality--in the psychological sense if not logically or factually. If the responses are found to correlate with the abovementioned predictions and devisings, then we are on the way towards acheiving convenient valid means of assessing intellectual competence. It seems that neither the convenient measures nor the validation predictions and devisings would be stable over many years, because of cultural and scientific change, and because the correct answers could become common knowledge. But stable invalidness can be no acceptable substitute for validity, even unstable. References Baird, L. L. (1985). Do grades and tests predict adult accomplishment? Research in Higher Education. 23. 3-85. Brand C. & Deary, I. (1982) Intelligence and inspection time. In H. J. Eysenck (Ed) A model for intelligence. New York: Springer. Clarke, R. (submitted for publication) A theory of autism. [published 1993] Clarke, R. (in preparation) A theory of IQ. Clarke, R. (in preparation) A theory of genius. Clarke, R. (in preparation). A theory of decadence and renaissance: an evolutionary basis for Toynbee's theory of civilisation. Jensen, A. R. (1982) Reaction time and psychometric g. In H. J. Eysenck (Ed) A model for intelligence. New York: Springer. |