Thursday, June 05, 2008

China's SATs

Slate has an article on China's SAT tests.

This goes back to my old posts about huge discrepancy between GRE scores of foreign students (we routinely reject dozens if not hundreds foreign students - mostly from China, but also from Korea, India and soviet block) with Physics GRE scores that are 950+, but accept plenty of domestic students with scores as low as 600-700.

Now, in the comments I was labeled as somewhat naive because apparently it is common knowledge that foreign students do so well on the exams because a) they cheat b) they prepare specifically for the exam, sometimes taking months or years off. And even if they don't cheat or prepare specifically the scores don't matter because they don't correlate much with research performance anyways.

These sentiments strike me as a bit strange. Sure, some students cheat, but I refuse to accept that foreign students are intrinsically more morally and ethically corrupt than american students. Besides, this explanation requires cheaters to reside in western european countries like Romania, Czech republic or Russia, in China, India, Korea and Japan - but of course not in US. For me this would require a "willing suspension of disbelief".

As to single-mindedly training for months for GRE - aren't these "good" qualities we look for in students? The excuse along the lines of "domestic students just chose not to study for GRE, but they are really smart and we will accept them anyways" is a wrong-headed argument. We wouldn't accept a student with poor grades over someone who got straight A's, simply because this student decided not to waste their time preparing for the exams, right?

As to previous research experience - I have both foreign and domestic students in my group, and foreign student has actually more hands-on undergrad experience than a domestic student. In either case, I don't consider spending a summer in research lab as an undergrad doesn't give someone much of a headstart considering long and tedious 6-7 year PhD process.

The way I see it foreign students face several disadvantages - the letters of recommendations carry less meaning because they are often from scientists or professors that are unknown to most US professors - we tend to look more favorably at letters from people we know. The letters style and content is often not the "gold standard" of a perfect letter of recommendation. US professors don't know how to properly calibrate grades and rankings - we know that best physics student at MIT is exceptionally good, and probably much better than the best student at Podunk Liberal Arts College. But what about differences between Fudan University and University of Science and Technology of China? Or IIT Delhi vs. IIT Kanpur?

Finally, the language barrier often is the most serious hurdle to overcome. Most students get a lot better over the first two years in US, but some struggle even after 5-6 years.

The real question remains - suppose someone came up with unbiased blind selection of best undergrads, based on their undergraduate academic record and potential to succeed, what would be the foreign/domestic distribution of such blind ranking/selection process?

34 comments:

quantummoxie said...

Those tests (SATs, GREs, etc.) are all a bunch of hooey (except in a few instances). Studies have shown (for years and years) that they only test a student's ability to take that particular test and certainly do not give any indication of academic ability (I know people who did very, very well on them and bombed in college while I also know people who did not do well on them who have PhDs and are quite competent).

Incoherent Ponderer said...

quantummoxie - I heard this argument before and I find it rather dangerous.

Sure, and maybe bad grades in college or high school don't prevent someone from becoming an outstanding scientist, doctor or businessman or a politician. And someone who never even finished college or high school could be plenty smart.

But all things being equal, and occasional outlier, there IS a strong correlation between grades, test results and other academic metrics and the performance.

Otherwise, why are we grading exams and asking our own kids to study hard, if all of it is a bunch of hooey? If a student who is failing to grasp basics of statistical analysis and is on path of getting C-, is competing for a job with a student who understands complex mathematical concepts and has been acing his classes - should the company hire C-student or A-student? If both have about the same chance of success, why bother with grades, or SAT or GRE scores at all?

This is precisely the argument I hear from some parents who protest any kind of standardized testing on grounds that the tests don't measure anything meaningful, other than test-taking ability. But of course this is not true. Asking a student to name current secretary of state, or ten european countries or explain the concept of mass or temperature DOES test something - maybe not their creativity or originality of thinking, but it DOES test some general knowledge, and walking away from tests by labeling them meaningless is equivalent to further lowering the standards to which we hold our students.

okham said...

Quantummoxie, could you please give a reference to a few of the studies that you mention ? I would be very interested in reading them.

okham said...

I completely agree with IP. My experience is that GRE scores correlate quite well not only with graduate coursework performance, but also with research ability (mostly in theoretical, perhaps, but also in experimental research).
Also, lunchtime stories of "the student with straight A's who could not operate a vacuum pump" or "the student who could not solve a second-degree algebraic equation but made fundamental theoretical advances" are worth telling precisely because they are exceptional. In general, my experience is that coursework performance is actually a pretty good indicator not only of background strength, but also seriousness and motivation, which impact research performance.

Schlupp said...

Okham , I don't know the two students you mention, but I know the one who couldn't solve a quadratic equation, wanted an A, thought he could make fundamental theoretical advances and was no longer permitted near any vacuum pumps.

As to cheating... hm, I have to admit that cheating is a national sport in MyCountry and a few others nearby. It's not that we are so particularly unethical, it is just not considered a very bad thing to give you buddy the answer to question 7. (We make up for it by not jaywalking.) Since everyone is aware of the issue, teachers and professors take measures against it where needed. (And we have lots of oral exams.) Now, large scale organized cheating on important tests is something different, but it could be that some people are not aware of the difference.

Anyway, I'd guess that Americans do worse because they can get away with it: If everyone knew they'd have to have a higher score, they'd do special preparation for the tests as well. As it is, they might be right in assuming that their energy is better spent in working toward a good recommendation letter.

okham said...

Okham , I don't know the two students you mention, but I know the one who couldn't solve a quadratic equation, wanted an A, thought he could make fundamental theoretical advances and was no longer permitted near any vacuum pumps.

Schlupp, that narrows it down to just about 90% of the student population...

PS I have to say it did not use to bother me so much but now PhysioProf has got me to hate word verification too....

sylow said...

Do I have to remind anyone writing comments here that physics grad students are required to pass a qualifier exam in many US universities BEFORE they can attain PhD candidacy and qualifier exam is waay harder than GRE physics test which is essentially undergrad level... Many many full physics professors would fail qualifier exams in top schools like Berkeley if they took it cold. Can anyone point me to some data which correlate GRE physics scores and qualifier success rates? If the arguments of the previous commentators here hold, there should be pretty good correlation presumably. Even better, the graduation rate of the grad students and their GRE scores? I guess we should do more than speculate here...

okham said...

Do I have to remind anyone writing comments here that physics grad students are required to pass a qualifier exam in many US universities

No, it's not necessary. First off, it is a well known fact, secondly it is irrelevant to the subject of this discussion. We are talking excellence in research; the qualifier exam is passed by typically over 80% of graduate students (see here, for instance), and so it can hardly correlate with anything.

On the other hand, if you google "correlation GRE academic achievement" you get a conspicuous number of results, most of which point studies showing at least a moderate correlation (this is a review of a series of such studies).

Anonymous said...

The US has been getting the best graduate students from around the world in the past. Is there any surprise that the rest of the world put together can produce far more talented undergraduates than the US? Physics GRE scores tell a simple truth - the AVERAGE foreign grad student is a way better student/researcher than the AVERAGE US grad student.

Mark said...

Don't high scores typically present in applications from China correlate somehow with inability of western examiners to tell one eastern face from another, or a face in front of them from a picture on the examination form?

sylow said...

Okham, what I was trying to say is that IF high physics GRE score is an indicator, it should translate into higher success rates in qual exams since they both test textbook material knowledge. Therefore, I wanted to know whether a grad student who came in with a higher GRE score is more likely to pass qual. I did NOT ask the overall pass rate of qual which is closer to 50-60% in big state schools like UIUC and Berkeley. These schools typically admit twice as many students as they can support as RA's because they need plenty of first year TA's and they need to get rid of a large fraction of the incoming class somehow.

Second, nobody here yet presented an argument on why someone who knows Griffith's electromagnetism by heart would be a better researcher than another individual who does not. Research by definition involves things that are not written in textbooks. I, as a computational physicist, would be more interested in hiring a student/postdoc who is a prolific C programmer with little physics knowledge in my field rather than someone who memorized J.D. Jackson but never wrote a script in his entire life thus I fail to see what a GRE or qual exam tells anything about anyone's research potential. My own PhD advisor is a person who never took any of those exams and knows only Maxwell's equations from Jackson's book yet he has 180 papers in science citation index as of today.

okham said...

Okham, what I was trying to say is that IF high physics GRE score is an indicator, it should translate into higher success rates in qual exams

Sylow, let me repeat this: if pass rate for qualifying is very high (80% or above), it cannot correlate with anything. It simply means that qualifying exam is irrelevant.
Now, from personal experience and countless conversations with colleagues, it is my belief that pass rate for qualifying exam is embarrassingly close to 100% nationwide.
I gave you a reference to it being over 80% at UCSD, which is ranked 16th nationwide, and runs a very rigorous program. I cannot imagine it being much lower elsewhere, as everyone is starved for graduate students these days. Regarding the numbers at Urbana and Berkeley -- I think you are talking about one-shot pass rates, but students can take the exam twice, which brings the overall pass rate to 75-80% (more information here -- interesting factoids: 25% of departments among the top 30 do not require a comprehensive exam, and 25% of departments nationwide allow students to retake the exam an indefinite amount of time, i.e., 100% pass rate).

Second, nobody here yet presented an argument on why someone who knows Griffith's electromagnetism by heart would be a better researcher than another individual who does not.

Because we were discussing whether there is a correlation or not, not why there should be a correlation. If you were to look at the average GPA of faculty in the top 30 (eh, make it 300) physics departments, do you think it would be on average closer to 4.0 or 2.0 ?

quantummoxie said...

1. I never said I thought all testing was hooey. I was specific to tests with formats similar to the GREs and SATs.

2. Some resources on criticisms of such tests (including a very interesting study by MIT) can be found here:
http://www.insidehighered.com/news/2006/10/06/sat
and here:
http://en.wikipedia.org/wiki/SAT#Criticism

3. Many smaller colleges have gone SAT-optional including some top tier schools such as Holy Cross, Bates, etc. While I disagree with the tone of the article, this explains a bit more about that:
http://www.usatoday.com/news/education/2006-04-04-standardized-tests_x.htm

4. Many people who are exceedingly intelligent do not perform well on certain types of exams and tests for many reasons. See, for example, this article:
http://www.wired.com/medtech/health/magazine/16-03/ff_autism

5. I've been very involved in secondary and post-secondary education in various forms for most of my adult life and have literally three generations of educators in my family. The knowledge (though the evidence may be more recent) that these types of tests are a poor judge of ability has been around for decades (see above).

quantummoxie said...

Oh, and I meant to add, that, while qualifiers and comprehensives are good, they need to be administered blind. I know one person who was flat-out told he would fail his quals no matter how many times he took them because the department was trying to ruin his advisor's career (he was tenured so they couldn't just can him - instead they flunked all his grad students on the quals).

The key to doing any of this correctly - fairly and with positive results - is to really pay attention to the folks who spend their careers on effective testing (there's a whole branch of education research focused on this). They've done all the work. Now people just need to implement it. The trouble is that people are inherently averse to change.

okham said...

Quantummoxie, I have nothing but respect for your experience of educator, but I also have plenty of respect for mine. So, maybe we can compare notes one of these days, but for the time being I'd say let's stick to studies, if they are available.

Again I accept the notion that tests are fallible and that there are exceptions -- any test (including medical ones) is only valid an instrument on average, i.e., occasionally it will fail. But this does not mean that it should not be relied on and/or that it is not useful -- it simply means that it ought not be the only piece of evidence (which no reasonable person would dispute).

I have looked at the references that you have furnished, and they all make the point that SAT test is not infallible. I have no problem conceding that, but still find studies that claim an average statistically significant correlation with academic achievement pretty convincing.

quantummoxie said...

okham,

First, I meant no offense. I happen to feel very strongly about this topic and can get a little hot under the collar when I perceive something as unfair or biased (which is how I view SATs).

Second, I just spent this last semester serving on a committee that spent endless hours discussing assessment and I was pleased to find that there were some excellent alternatives to the usual format (off the top of my head, the name of the test I liked the best isn't coming to me, but I can ask a colleague with a less porous memory if you want the name).

Third, most of these colleges that have dropped the SAT as a requirement did so after conducting their own study. Even the U. of California system found the SAT I to be a poor determiner of academic performance: http://www.ucop.edu/news/archives/2001/oct25art1.htm

Fourth, studies are great, but results can be interpreted in many ways. Take the following article, for instance. It notes that high school GPA is a good predictor of college GPA in athletic training programs, but then notes in parentheses that the r-squared value for the fit is 0.38!! The same sentence says it predicts "more than one third of the professional program success." The implication is that the test is good but I'd hardly call something good if it has a 33% success rate.
http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=155524

One of the problems, of which I am acutely aware, is that the best measures of success are hard to quantify. Liberal arts colleges have the advantage of fewer numbers of applicants and thus have the ability to spend more time on each applicant. Frequently the personal interview and personal essay weigh the most heavily and, while I have not found a study to compare it, I would bet these are far better indicators of success than SATs. Of course, large universities don't have the ability to rely as heavily on these criteria. That's why I think there needs to be a major change in how we test. If I remember the name of the one I liked from my committee work I'll post it.

ali said...
This post has been removed by the author.
sylow said...

Okham, if everybody is passing qual as you claim, why should we care about anyone's GRE score?? I mean, if these folks already know J.J. Sakurai by heart, they also know Griffiths as well presumably.
Am I mistaken?

I also dispute the notion that it requires a formal education to succeed in anything in life. Famous Freeman Dyson does not even have a Ph.D. and Bill Gates does not even have a bachelor's degree.

okham said...

I also dispute the notion that it requires a formal education to succeed in anything in life. Famous Freeman Dyson does not even have a Ph.D. and Bill Gates does not even have a bachelor's degree.

Sylow -- I see what you are saying... why bother with applications, GRE scores, letters of recommendation, undergraduate transcripts... just pick 50 random people each year as your entering graduate class, and they'll be just as good as everyone else.
Well, don't worry, we are almost getting there :-)

sylow said...

Okham, if those seemingly "random" 50 students had previous research experience as undergrads and even better coauthored a scientific paper, I would accept them. That is what all grad schools do anyway. Take a look at the information provided in Princeton physics' grad admission's webpage to see what I am talking about.

Being a successful researcher or being a successful entrepreneur has absolutely nothing to do with GPA's, test scores or textbook material knowledge.

Incoherent Ponderer said...

sylow, so if we have 445 graduate school applications and 200 of them have undergraduate experience - but you only have 35 slots, who would you take? top 35 applications alphabetically, geographically, chronologically?

okham said...

IP -- it's simple: first take the ones who have the lowest GPA. Clearly, these must be the best and brightest, who understood right away what science is all about and adjusted their priorities consequently. They did poorly on purpose, even though they could easily have done well (particularly telling are grades like D+ or C- in QM, if one is to spot real talent). That's how things work, did you not know it ?
(If that does not narrow it down enough, identify all the ones who scored in the bottom quartile on their GRE, especially the quantitative part. These are true "hidden gems").
These bright your men and women are the ones who have no trouble going through graduate school, by memorizing Griffiths' and Sakurai's books and passing the qualifying exam.

Incoherent Ponderer said...

okham - even if we accept for a moment a premise that GRE Physics or grades or passing qualifier on their own don't have much relationship with research potential, the basic premise of GRE Physics, or graduate exam or qualifier is: you have X amount of time to learn Y, you will be quizzed on it later. Figure it out, this test is important for your career.

So even if it's a stupid meaningless exam, if some students can't figure out how to do well on
this stupid meaningless exam, I want to know why? What other things they won't be able or simply can't figure out? Is it because they can't glue their butts to the chair for 5-10 hours a day for a few weeks (or months) until they are well-prepared? Is it because they don't care?

As experimentalist, I would rather see an exam where one has to carefully take apart and then put together a vacuum chamber, while soldering wires for feedthroughs and measuring connections with a multimeter. Such test would be more meaningful - but other tests will do fine, as they test for things that are also necessary for doing well in grad school.

Anonymous said...

I was an American student who came from "Podunk" Undergrad University and scored in the 500s on my physics GRE, probably because I can't memorize formulas and didn't study as hard as I should have. Nevertheless, I was accepted to a top ranked physics program probably because of research experience and letters of recommendation. I did poorly on my course work during my PhD but did quite well in my experimental research and published a couple of highly visible publications. During my final year of my PhD without having done a postdoc, I was offered and accepted an Assistant Professor position at a very well known Research 1 University.
I think one has to be careful with the weight they put on physics GRE scores and even success in Q exams, since their direct correlation with success in science is questionable. Success in science requires hard work, creativity, and passion more than anything else and those exams can't measure that.

Schlupp said...

I think the American students do not study as much because they don't have to. Since they know they will be accepted with a lower score, it would really be a waste of time to study more.

okham said...

IP -- let me give you the following example: why do we periodically go to the doctor and have blood work done ? Because the medical establishment believes that some tests correlate with the presence of health conditions. Are these tests always reliable ? Of course not, nor would any sane and competent doctor take them as the final word, but they are believed to be helpful, if properly interpreted and placed in the broader context of a person's condition. As a patient, do I care to understand in depth how these tests work ? Not really, to be honest.

Pretty much the same applies to GPA and GRE. Virtually every school on this planet (yes, even Princeton) regards them as valuable predictors of a student future performance, and this claim is backed by a body of studies.
There are disagreements as to the strength of the correlation, to be sure, but that some correlation exists seems out of question. Should GPA and GRE score be the only criterion for admission ? No reasonable person says that. Is it possible that a student with low GPA and GRE score may turn out to be an excellent researcher ? Of course. Does it happen often ? Not really.

So, while I can certainly imagine that on occasion a student with a lower GPA may be preferred to another with a higher one for reasons such as demonstrated ability in research at the undergraduate level, it would puzzle me (just like it does you) if I saw a pattern of admissions in which students with lower GPA or GRE scores were given priority.

sylow said...

Okham, you read what anonymous wrote above (It is not me, by the way). From my own circle of colleagues, I can give you several examples.
1. My own PhD advisor, who never took any of those tests, yet he has close to 200 papers in citation index as of today
2. My postdoc mentor, who again did not take any of those silly tests, has a paper which has 7500 citations as of today. There are only a handful of papers which have >5000 citations in physics literature.
3. A collegue of mine, who was not in top %25 percentile of physics GRE has 65 single author papers as of today
4. Another colleague of mine who was at top %1 of physics GRE exam has only 8 papers as of today. He earned his PhD in 1996.

These are only people I know personally. I am sure there are lots of people like this in your own circle too. Rather than speculating here constantly, you should provide us some data which actually correlates omeone's research performance with his GPA, test scores etc.

okham said...

My own PhD advisor, who never took any of those tests, yet he has close to 200 papers in citation index as of today

My dad never took any of those tests and has no publication. Same goes for my mom and my brother. That's 3 against 1. Coincidence ? I don't think so...

A collegue of mine, who was not in top %25 percentile of physics GRE has 65 single author papers as of today

Interesting. I wonder if that is true of everyone who is not in the top 25%... Seems like a very solid hypothesis but, hey, let's not jump to conclusions... we need at least one more case -- once we have it, I say we just assume that it is true.

These are only people I know personally.

Yeah, but let's face it, how many more examples do you need ? Come on, once you can cite three, four, that ought to settle it...

Rather than speculating here constantly, you should provide us some data which actually correlates someone's research performance with his GPA, test scores etc.

You are right Sylow. So far I have only given speculations, like links to studies (see my 12:52 post), which clearly is not the way to go about discussing this issue. I could at least have made up a number or two...
Let me go ask a couple colleagues of mine what their GPA is. Maybe I'll ask my neighbor and his wife too. I say that will give us a pretty damn good indication.

quantummoxie said...

okham,

Yikes! I thought I was a little hot under the collar there, but you're getting pretty harsh on the sarcasm there with sylow!

Sylow has a point, though and instead of ridiculing it you ought to do precisely what you are asking of him (and me). Now, I have found studies on both sides of the issue - studies that support a correlation and studies that do not. I'm sure the physics GRE is a better test than the SAT I for its specific purpose (it doesn't fall prey to the potential biases in written and oral communication). My only complaint about it is that it penalizes people with certain learning disabilities who can't remember equations very well or transpose things in their heads. While you can get accommodations for such things, it is far from easy and, in my experience, far too many people fall through the cracks.

Now, personally, I like IP's idea of some sort of hands-on testing (even though I'm a theorist). I'll tell you that the three biggest things that are indicative of future success in anything are, in no particular order, 1. creativity, 2. perseverance, and 3. critical thinking/logic skills. Now, a student with all three should clearly be above average (i.e. above a C) I would think, but I have seen students with A's who had none of those skills.

Let me give you an example. A few years ago I had a student in my introductory class who got an A- (the only A or A- in a class of 40). Yet she complained because my labs are not a set of instructions that, if followed, easily produce high grades. I make my students think - in fact I make them devise their own procedure (I follow the methods used by Tom Moore at Pomona). She actually said to me, "I just want someone to tell me what to do." This student was one of our college's top students the year she graduated and went onto Med School. I told her I didn't want a surgeon operating on me who had that attitude. I also tell my students, when they complain that they just wasted an hour doing the whole thing wrong, that getting the wrong answer is not wasting time. Failure teaches you something, especially in science.

In short, the best tests are those that test critical thinking, thinking outside the box, and perseverance (not giving up in the face of adversity). Such tests are hard to come by, however, which is why, while I would agree that absurdly low standardized test scores are probably a decent indicator, mediocre standardized test scores are likely no better than good test scores.

The problem is that both you and IP are, like many folks at large universities, stuck on pure statistics which reduces people to numbers. While, statistically, the status quo is likely to give you halfway decent students most of the time, how many even better students have you missed out on by not analyzing the skills I mentioned?

okham said...

Yikes! I thought I was a little hot under the collar there, but you're getting pretty harsh on the sarcasm there with sylow!

Wow... "harsh" ? I thought I was just being silly... be that as it may, it's clearly time for me to abide to the famous commandment "Thou shalt not bore" and (un)gracefully bow out of this.
After all, on reading my own comments, I see that I have repeated the same things more than once, so clearly I am ineffective at making my point.

One thing though: it's not fair to keep asking me to "provide references". I have provided them. You and Sylow may obviously find them unconvincing, and I respect that. But anecdotal evidence just does not do it for me.

quantummoxie said...

One thing though: it's not fair to keep asking me to "provide references". I have provided them.

I looked back through the post and, indeed, you did provide them. My apologies. On the other hand, so did I.

arl said...

I actually think GRE tests are a good indicator if the student takes them twice (at least). Two high scores says something about the student; I think that means the student will do just fine in grad school, but maybe it only means (s)he studies too much or memorizes a lot, not necessarily a bad thing.

Two mid-ranging scores cannot really tell anything about the student, thus many other things should to be used to evaluate a student.

Two low scores, I would have serious problems with that application. Either the student doesn't care, or the student cannot learn basic physics.

GRE-type knowledge proves to be very useful for a grad student. When you go to a conference or a talk in your department, if you do not have those "little" equations in mind the whole talk will be over your head. Also, the ability to determine orders of magnitude quickly and correctly is very important when doing physics; it can tell you immediately if you are going in the right direction.

And yes, I have heard that argument of people with low scores being competitive in grad school. I have to disagree, one thing is to make it through grad school, another very different is to be competitive. I am foreign physics student, though not chinese, at a top 20 university in the US and I have to say that in every new class, the best student is always a chinese or indian. The american students, and me, are not bad at all. We can understand and get good grades in grad school, but there is a clear difference in how much we know to how much the chinese or indian know. Could we be like them? Maybe, but for one reason or another we aren't and they deserve credit for that.

Anonymous said...

I will expand on my previous anonymous anecdotal evidence. I am the Professor that scored in the 500s on the Physics GRE which if I remember correctly is in the bottom 25th percentile. I am not suggesting that we admit students with low physics GRE scores but we shouldn't rule out students who have an otherwise strong application - good rec. letters, impressive research experience, etc., if they just happened to score low on the physics GRE.

Same goes for a low GPA. My grad school grades were pretty bad and I did get cut during the final round of a prestigious postdoctoral fellowship due to that. I didn't really care, because I received a much better offer as a faculty at a top research university instead, but I was very lucky that they didn't ask to see those grades.

In my case, there was usually one defect in my application packet - for grad school that defect was the physics GRE and for my postdoc/faculty applications that defect was my grad school grades.

I agree with arl that we need to judge all aspects of the application packet. I wouldn't reject a student if they have a strong overall application but I wouldn't hold it against another student if they have impressive rec letters and research experience but lack a good GPA or GRE score. It could be evident that their passion lies with research and not with taking tests. This can be learned by having the student spend a summer in my lab. I would prefer to have a student in my lab who is passionate about research rather than somebody that loves solving exam or hw problems. One important aspect to research is defining the problem. Physics courses teach us how to solve problems but they rarely teach you how to define a new and exciting problem. Some of the discoveries and the kinds of research that gets into big journals that land you jobs in big research universities are those where the researcher simply asked the right question.

One of the greatest scientists in history and perhaps the greatest experimentalist would have probably failed the physics GRE - Michael Faraday.

quantummoxie said...

I like Anonymous' take on all this. Personally, I never took the GREs. In fact I never took a qualifying exam either. Having received a diagnosis of attention deficit disorder while I was writing my PhD thesis, I suspect I would not have done well had I taken those exams. On the other hand, I nailed my oral defense and my thesis has a growing list of citations to it.

Anyway, what I really want in a student researcher (and I'm a theorist) is someone who can think critically and has the ability to learn. I'm less interested in the exact knowledge a student has than in his or her ability to rapidly acquire new knowledge. I'd rather put students through some sort of intelligence-like test similar to a neuro-psych test (which is what they subjected me to when I was diagnosed with ADD - it's a roughly six-hour test).

My point in all this, which I did state in a previous post, is that while the GREs may possibly be some indicator of future success, we're using a format that is now approaching half-a-century old. The advances we have made in understanding how people learn and how to assess that learning just in the past two decades alone warrants developing a better option. Why a better option hasn't been developed or hasn't caught on if it has been developed is debatable but as a general cynic I suspect it is related to College Board's monopolistic promotional tactics, but that's a purely unsubstantiated opinion on my part.