The New Yorker on the Poets & Writers MFA Rankings
More shouts and murmurs and general disgruntlement over the Poets & Writers MFA rankings: the New Yorker dismisses the “less scientific approach” of P&W and encourages its readers to “listen to P&W’s own F.A.Q. page: ‘Should I rely on these tables to choose where to apply? No.’” :
Dissatisfaction with academic rankings is nothing new. A few years back, the Annapolis Group, a conference of a hundred liberal-arts colleges, voted to fight back against US News; college presidents pledged to withhold information on which the magazine bases its lists. But the qualms with the P&W list seem to run deeper. The idea that only the opinions of prospective students should count strikes many as very strange: “It’s analogous to asking people who are standing outside a restaurant studying the menu how they liked the food,” said Leslie Epstein, the director of the writing program at Boston University.
Then there's Best American Poetry’s subtly-titled “Poets and Writers MFA Rankings: Garbage In Garbage Out” where columnist Stacey Harwood dissects the ranking's weaker points. For instance:
The methodology says the responses are "votes" and that those programs receiving the most "votes" are the ones the respondents hold in highest esteem. Yet if you read elsewhere you learn that the "votes" are the programs to which respondents are applying; there may be no connection between which program one might hold in high esteem and those to which one individual is applying (eg. I may think Houston is the best but because I live and work and have family in NYC, it isn't practical for me so it's not on my list of programs I'll be applying to- this likelihood isn't captured in the ranking).
Every time the author says, "it is reasonable to assume," know that it is also reasonable to assume something other that what follows. Such claims are not backed up with any evidence, anecdotal or otherwise.
Most of the programs ranked got fewer than half the number of votes than the top ranked program - there are many many ways to account for this gap and yet the only explanation given is that it reflects the "esteem" with which potential applicants hold the program but it could reflect many other possibilities.
A good deal of the "methodology" section is devoted to history of MFAs and to discrediting other polls - irrelevant in a discussion of methodology. I could go on. The infelicities of the writing style are too numerous to mention. Strip away all of this filler and you see how little substance there is to the ranking.