quizivex wrote:The PGRE is a valuable metric for comparing students from different undergraduate institutions. It's important to have a standardized metric, just like med schools have MCAT and law schools have LSAT.
The PGRE asks questions about circuits, Doppler shift and point charges... and people try to say that the test has gender or racial bias. Sigh.
I disagree. The evidence from ETS themselves show that the GRE (granted, not the Physics GRE, but others have shown the same for all subject tests**) shows a much stronger correlation to racial and gender attributes than outcomes. There are many other studies that show standardized testing does not properly evaluate knowledge. I don't really intend to use this space to debate the merits of whether or not standardized testing has a place in our educational system.
**By others, I mean Staussan from my linked Miller & Staussan article. I don't know if his work is published yet, but he has gone on a talk tour around the country to talk to Physics and Astro departments and present his evidence and his arguments for abolishing the Physics GRE. Sorry that I can't link to something but maybe your dept was one of the ones visited on this tour. I only changed my mind about the test after seeing this presentation and through discussions with him.
quizivex wrote:Indeed, the PGRE or any standardized test could be improved. Perhaps the astronomy community would be better off having an astronomy GRE. But there should still be a standardized test. Tests play an even bigger role in grad school. The quals/prelims are arguably less fair because instead of 100 questions spread over all physics topics there's only a few hit or miss problems, and it's pass/fail.
This might be an astronomy/physics difference then. Tests play a very small role in the graduate programs I've been a part of. Very few graduate classes have exams at all (mostly pass/fail final projects) and those with exams are often oral exams that are also pass/fail. So, I don't see how a standardized test can really help pick students for graduate programs---you don't need to do well on standardized tests to do well in grad school. I think a pass/fail test is much more fair than one graded with a score. If you are really just checking whether or not a student meets a requirement, then you only care about pass or fail. If you want to compare scores like 880 vs. 760, well, then you really better be very very confident in your ability to come up with these metrics. Why do we put so much trust in ETS and the PGRE to have such precise scores? (Note: the standard deviation on these scores are also quite large, up to 200 points).
My opinion is that the burden of proof of usefulness of PGRE scores lies on ETS. And from the evidence I've seen (e.g. the astronomy fellowship outcomes paper**), there is reasonable doubt that the PGRE score as actually useful. There is some evidence that suggests it may be harmful (e.g. Staussan). Therefore, my opinion is that until we are sure that the PGRE actually does something useful, we shouldn't be using it as there is risk of harm but not much evidence for utility.
**A note regarding the Levesque et al. fellowship paper: This work was not meant to be absolute proof (nor a rigorous study) disproving the effectiveness of the PGRE. I would view it as evidence showing that there are potential problems with the PGRE score usage. They also acknowledge that winning an astronomy fellowship isn't the only indicator of success. However, in case you are not familiar with these specific fellowships, they are indeed the pinnacle of postdoc appointments and the majority of these fellows go onto tenure track faculty positions immediately after holding these fellowships. That is, you wouldn't say people who don't win one aren't successful, but it's reasonable to say that if you do win one of these, you definitely count as "successful". Or, to put it another way, if there is a metric that marks these fellows as "not suitable for grad school", then that would not be a very good metric!!
And that's simply what the paper shows: many common uses of PGRE cutoffs in grad admissions would have excluded people who would have gone on to win these fellowships. That by itself may not be a big problem, okay, not all metrics are going to be perfect. However, when combined with the evidence that performance on these exams are biased by gender and race, then you are now implementing a metric that may unfairly exclude people who are capable of success on the basis of gender or race. And to me, doing this knowingly and without evidence for the utility of the PGRE is just not ethical.