How political science is like intelligence research

NOMINATE_2
Scores generated by W-NOMINATE, a method related to factor analysis, help explain Congressional votes on a welfare reform package (from Wikipedia)

Suppose you want to know how conservative or liberal your Congressional representative is. Is there a way to do this that doesn’t involve sorting through a small mountain of votes and making an educated guess?

Political scientists have been working on this problem for decades, and have largely solved it. They can take a legislator’s voting record and use it to produce a mathematical estimate of that legislator’s ideology compared to his peers in Congress. And it’s not just votes – they can make similar estimates with campaign contributions, surveys, speeches, and even Twitter followers.

While the particular method political scientists use to make these estimates has varied over the years, conceptually all of them are variations of a technique called factor analysis.

Researchers employ factor analysis to simplify data. In this example, legislators may vote on 500 bills in a session. But to be familiar with our congressperson, we don’t have to keep track of their votes on all 500 bills – if we know his or her political leanings, specifically if the congressperson is conservative or liberal, we can predict many of their votes from that fact alone. Just one number – one “factor” – representing political ideology can summarize the data contained in all 500 votes. Factor analysis looks for these kinds of patterns that allow us to summarize larger sets of data with just a few numbers.

Political scientists weren’t the first to do this kind of analysis. Early models were inspired by work in psychology.

Which brings us to the mini-debate that has sprung up, with Sam Harris and Charles Murray on one side, and PZ Myers and Cosma Shalizi (indirectly) on the other, about IQ.

Like Myers, I’m not a fan of Sam Harris and I refuse to listen to his monotone for 2 hours. And I agree with the Eyes On The Right post that Myers quotes. Where I, at long last, finally disagree is when he dismisses the concept of IQ altogether, citing an article by Cosma Shalizi (in the comments).

Shalizi’s article – read it here – attempts to undermine factor analysis as a whole, as well as it’s specific application to the concept of g, or general intelligence (g is what IQ is supposed to measure). The main workhorses of his argument are two mathematical examples. But of the two, the first one is incorrect and the second one doesn’t prove what he claims.

The first example accidentally assumes what it sets out to prove. Shalizi generated the numbers in his matrix, which represent correlations between test scores, by choosing numbers at random between zero and one. I scratched my head when reading this part. Typically a person running a simulation would generate test scores first, and calculate the correlations directly from the scores. Instead Shalizi first generates the correlations between test scores, then creates the test scores to match the correlations.

This backwardness allows Shalizi to make a mistake he shouldn’t have made. Test scores that aren’t influenced by a common factor should have correlations clustered around zero – that is, the score on one test shouldn’t predict the scores on other tests. Shalizi, however, chooses the correlations by randomly choosing between zero and one, in effect assuming that many of the tests are correlated with each other. The factor analysis then finds a factor in this case because the numbers really do indicate an underlying factor. If Shalizi started by generating factor-less test scores, and calculated the correlations from those test scores (as opposed to randomly generating the correlations), he would quickly discover that this example doesn’t work. Factor analysis may erroneously discover a factor when applied to small amounts of data, but for larger data sets it doesn’t return spurious false positives like this. That’s just not how it works.

The second example is technically correct, but the implications are milder than he suggests. Yes, as demonstrated, IQ may be a summation of many smaller factors, but that doesn’t imply that it’s not a real or useful concept. The NOMINATE ideology scores used by political scientists are also based on many small factors – votes on issues ranging from abortion to taxes to NATO. Yet we all know the difference between conservative and liberal, and given a placement along the left-right axis we can make decent predictions about a person’s views on any number of issues. Clearly the left/right distinction in American politics is not a figment of our imagination, or a statistical artifact, even though it combines many smaller issues into one category. Likewise, though IQ may be the result of many smaller factors (and genetically speaking, I imagine it is), that doesn’t imply it’s not real. It’s correlated – not perfectly correlated, but correlated – with a variety of things, like knowing what the word “vicarious” means, solving MENSA puzzles, and finishing college. These correlations just happen to match one definition of the word intelligence, in the same way that factor analysis of rollcall votes leads to scores that match our understanding of the liberal-conservative spectrum in politics.

Political scientists debate the biases in scores generated by political factor analysis techniques, but there’s basically no debate that the left-right political spectrum is a real thing, and these scores have served as the backbone for a huge amount of research in the last 30 years. As far as I can tell g is in the same place.

My expertise is limited to statistics and social science, so I won’t comment on the debate about to what extent IQ is determined by genes. And I’m skeptical, to put it mildly, of Charles Murray’s policy views. But factor analysis is real.

 


Disclaimer: Yes, I did read The Bell Curve, about 10 years ago. I think it’s got some rigorous arguments mixed with some really weird arguments. And no, I didn’t encounter any censorship: I rented the book from my local library with no mishap.

Advertisements

Common sense is better than free speech

dont_read_comments

Clichés about free speech are so common in discussions about college speakers that I despair even thinking about it. You can’t swing your arm without hitting at least two cringe-worthy bromides – and both spoken with total condescension and certainty. So to end the despair and to help the self-righteous off their high horses (and to give my arms more room), I’m proposing a radical alternative: let’s look at evidence.

Is unregulated free speech the best way to get the truth? What do the people most serious about pursuing truth think? I suggest starting with scholarly journals, the highly regarded publications that form the backbone of the scientific community. When faced with the decision to publish a paper, do journal editors bark out overused catchphrases about free speech and publish everything? Clearly not. Journals restrict speech radically and extensively. Only a select few authors ever get to publish in the most prestigious journals, and it can take years of effort for them to do so. Editors discriminate so zealously that simply failing to cite the right literature in the bibliography can keep a paper from being published. And far from being a drag, this is the secret to the journals’ success!

Or take another truth-seeking institution – the classroom. When creationists propose that public schools teach creationism alongside evolution, do scientists welcome these proposals as virtuous exercises in “open debate and dialogue”?

Perhaps a third truth-seeking institution will be different: the scientific conference. Do geologists, for example, allow flat-earthers to speak at their conferences because “the solution to bad speech is more speech”? Obviously not.

On the other hand, we’ve all seen the disasters of unmoderated comments sections. Why anyone thinks this kind of wild west free-for-all would be an effective way for college students to learn or debate is beyond me. And due to legal restrictions that prevent public universities from barring bad speakers, “unmoderated comments section” appears to be the current state of college speaking.

To be totally honest I find it hard to care about college speakers, even if the policy is a mess. How much does this actually matter? Very little, in my experience. I suspect this is an overwhelmingly symbolic issue for all sides of the debate. What really annoys me are the allegedly smart people making flat-out unbelievable claims. Are thinking people really supposed to believe that the college speaking tour of Milo Yiannopoulos, the world’s most famous internet troll, is an event that will educate college students? It sounds more like a scene out of the movie Idiocracy. Yet this is an argument I see again and again, without a shred of evidence and without any hint of humility. It’s as if we’ve traveled back to the Middle Ages – what kind of fool would look at evidence when we have the words of the ancient philosophers to rely on? (I will not name the guilty parties to avoid embarrassing anyone. But if they persist I might.)

Anyway, it’s easy to see the solution to this problem. Ideally colleges would be able to enforce policies that prevent self-aggrandizing vile ignoramuses from taking advantage of their lecture halls, in much the same way that a forum moderator would lay out the ground rules for participating in a discussion board. The co-opting of colleges by morons and internet trolls is an embarrassing failure of public policy, not a victory of rationalism.


 

Update, 4/21/17:
Cass Sunstein writes:

Officials at the University of California at Berkeley this week canceled a scheduled speech by Ann Coulter, the conservative author, on the ground that the school could not “find a safe and suitable venue” for her. … It should go without saying that at colleges and universities, free speech is indispensable, and interferences with it are deplorable.

My hot take: It should not “go without saying” that trolls have the right to speak at publicly-funded college facilities intended for education. Trolls are not “indispensable” college speakers, and banning them is not “deplorable”. Colleges would do just fine without them. I am very confident of this. There are many colleges Ann Coulter has not visited. Those colleges have not fallen into barbarism.