Setting Score Goals

  • If you want to know something about the GRE subject test in physics then chances are you will find it in here.
  • If something about the physics GRE it isn't already discussed in here then please put it in here.

Post Reply
User avatar
HappyQuark
Posts: 762
Joined: Thu Apr 16, 2009 2:08 am

Setting Score Goals

Post by HappyQuark » Wed Apr 24, 2013 8:01 am

Hi all, long time no see.

I did some fun plots for old times sake and thought I'd share. The info can also be found at PhysicsGrad.com

I'm not sure whether it is common knowledge among the forum, but the difficulty of the practice GREs varies quite a bit. If you compare PGRE9677 to PGRE0877, for example, a raw score of 50 would give you a 720 in the 0877 exam and a 850 in the 9677. This is obviously important because if you want to get a certain score on the subject GRE, you need to know how many questions you need to get right.

On the 9677 test, for example, you only needed to get a raw score of 67 to get a a 990. A good technique for this test, then, would have been to try to complete 75-80 questions that you are sure you know. It seems like ETS have tried to stop this from happening and now in the 0877 test you need to get a minimum of 82. These two examples are the extremes so I decided to compare all the subject tests by plotting the percentile score vs the raw score (See Below).

Image
PDF Link

So the tests are very different but we obviously want to know which raw score we should be gunning for in the upcoming 2013 exams. Looking at the Percentage vs Scaled score and comparing the 2011 and 2012 scores pulled from the users profiles on this site, it is clear that the most recent exams very closely matches the 2008 practice exams terms of overall score scaling.

Image
PDF Link

So for the upcoming tests, a reasonable breakdown of the "Raw Scores" to "Scaled Score" would be

Raw Score | Scaled Score
17 - 31 | 500-590
32 - 45 | 600-690
46 - 59 | 700-790
60 - 71 | 800-890
72 - 100 | 900-990

soil
Posts: 7
Joined: Tue Dec 11, 2012 7:15 am

Re: Setting Score Goals

Post by soil » Thu Apr 25, 2013 3:24 am

Thanks to your kind information :)

I just took my April's test and now waiting the result...

User avatar
quizivex
Posts: 1031
Joined: Tue Jan 09, 2007 6:13 am

Re: Setting Score Goals

Post by quizivex » Thu Apr 25, 2013 1:14 pm

It's important to note that the "difficulty of the test" and "percentiles" are completely unrelated! You can't assume that because the "percentile vs. scaled score" plots for recent years line up with 2008, that the raw scores needed for each scaled score in upcoming tests will be similar to the 0877 practice test.

The "difficulty of the test" is measured by what raw score you must get to earn a given scaled score. The scaled scores are intended to measure a given level of performance independent of the difficulty of the particular version of the test, and independent of the performance of the other test takers.

On the other hand, the "percentile vs. scaled score" is determined entirely by the curve set by the pool of recent students taking the test. In the 0877 practice test booklet, they state "The table on page 91 contains, for each scaled score, the percentage of examinees tested between July 2007 and June 2010 who received lower scores.." So the percentiles are averages over a 3 year period, and will not substantially change from one year to the next.

HappyQuark's second graph shows that scaled scores have been increasing over the years. This is because the students have gotten better at the test over the years. The reason the "percentile vs. scaled scores" from 2011 and 2012 line up closest with the 2008 test is that 2008 is by far the most recent of all the years the practice tests were released.

The "difficulty of the test" is a function of one individual test only. There are several versions of the test given on every test date, and students in the same exam room have disagreed over how difficult the test was for that reason. The 0877 test is a sample size of one.

Incidentally, to measure difficulty of tests over the years, the scaled scores should be compared with raw scores. A plot of "raw score vs. percentile" is virtually meaningless because percentiles change over time.

Overall, we cannot use the 2008 test to estimate "which raw score we should be gunning for in the upcoming 2013 exams". It will be dangerous if PGRE.com users go into the test expecting it should be as "easy" as the 08 test and find that it isn't. One of the issues that makes the subject GRE experience difficult is that students won't know in advance how difficult their version of the test is. So if they're having an easy/hard time, they won't know if it's because they're doing well/bad, or if the test itself is easy/hard.... it's possible to mess yourself if you count on needing to get a certain number of questions right to reach a goal.

User avatar
HappyQuark
Posts: 762
Joined: Thu Apr 16, 2009 2:08 am

Re: Setting Score Goals

Post by HappyQuark » Fri Apr 26, 2013 8:15 am

quizivex wrote:It's important to note that the "difficulty of the test" and "percentiles" are completely unrelated! You can't assume that because the "percentile vs. scaled score" plots for recent years line up with 2008, that the raw scores needed for each scaled score in upcoming tests will be similar to the 0877 practice test.

The "difficulty of the test" is measured by what raw score you must get to earn a given scaled score. The scaled scores are intended to measure a given level of performance independent of the difficulty of the particular version of the test, and independent of the performance of the other test takers.

On the other hand, the "percentile vs. scaled score" is determined entirely by the curve set by the pool of recent students taking the test. In the 0877 practice test booklet, they state "The table on page 91 contains, for each scaled score, the percentage of examinees tested between July 2007 and June 2010 who received lower scores.." So the percentiles are averages over a 3 year period, and will not substantially change from one year to the next.

HappyQuark's second graph shows that scaled scores have been increasing over the years. This is because the students have gotten better at the test over the years. The reason the "percentile vs. scaled scores" from 2011 and 2012 line up closest with the 2008 test is that 2008 is by far the most recent of all the years the practice tests were released.

The "difficulty of the test" is a function of one individual test only. There are several versions of the test given on every test date, and students in the same exam room have disagreed over how difficult the test was for that reason. The 0877 test is a sample size of one.

Incidentally, to measure difficulty of tests over the years, the scaled scores should be compared with raw scores. A plot of "raw score vs. percentile" is virtually meaningless because percentiles change over time.

Overall, we cannot use the 2008 test to estimate "which raw score we should be gunning for in the upcoming 2013 exams". It will be dangerous if PGRE.com users go into the test expecting it should be as "easy" as the 08 test and find that it isn't. One of the issues that makes the subject GRE experience difficult is that students won't know in advance how difficult their version of the test is. So if they're having an easy/hard time, they won't know if it's because they're doing well/bad, or if the test itself is easy/hard.... it's possible to mess yourself if you count on needing to get a certain number of questions right to reach a goal.
This is a fair point. I forgot to include the Scaled Score vs Raw Score plot which, as you pointed out, is relevant.

Image

From this plot, it's pretty clear that the relationship between the scaled score and raw score is nice and linear. Where the ambiguity comes in is the cutoff at the 990, as this will affect the slope. on the 0877 test the cutoff cited is a raw score of 82, in the 0177 test it was 85, 9677 is 67, 9277 is 76 and 8677 is 84.

Where things apparently get interesting, as you pointed out, is the comment by ETS where they say:

"The percent scoring below the scaled score is based on the performance of 14,395 examinees who took the Physics Test between July 1, 2007 and June 30, 2010".

If I'm understanding your post, you are saying that your interpretation of this sentence is that there are multiple versions of the test (which I agree with and am aware of) but that each of these tests have significantly different levels of difficulty which, on average, is the cited value they give. I've been interpreting this sentence as the percentile score is found as the average of the test takers over 3 years but that the relative difficulty of the different tests and the corresponding scaled score and raw score is roughly similar.

The thing that doesn't make sense to me about your interpretation, If correct, is this makes the average values they cite completely useless. In fact the table they give at the end of each practice exam is completely meaningless. When I took the practice exams, it was my understanding that when I calculated my raw score on a practice test, I could look at the score conversions table in the back to see what my raw score and percentile rank would have been and, so far as I can tell, this is how everyone else has interpreted it. So, for example, if I get a raw score of 75 on the 0877 exam, it was always my understanding that I could interpret that as a scaled score of 920 and that I was in the 89th percentile (plus or minus some small variation). If you are correct, then in reality getting a raw score of 75 on the 0877 test may have really corresponded to anything between, say, a scaled score/percentile of 750/64% to even as high as a 990/95% (like it was in the 9677 test).
quizivex wrote:There are several versions of the test given on every test date, and students in the same exam room have disagreed over how difficult the test was for that reason. The 0877 test is a sample size of one.
I don't know if forum comments are a good indicator as to the difficulty of the tests. Since, presumably, some people are smarter and/or more well prepared than other people taking the test it would seem reasonable to me that you'd get varied descriptions as to how difficult a test is, even if the tests were identical. Regardless, this is unfortunately an unknown that is relevant. Based on the graphs and their wording on the data, it seems more reasonable to me that the GRE problems are different between tests but of roughly similar difficulty with the 990 cutoff corresponding to a raw score somewhere around 75-85. If these assumptions are correct then I think this data can be of some use. If not then we may just be *** out of luck.

User avatar
quizivex
Posts: 1031
Joined: Tue Jan 09, 2007 6:13 am

Re: Setting Score Goals

Post by quizivex » Fri Apr 26, 2013 9:56 am

HappyQuark wrote:If I'm understanding your post, you are saying... that there are multiple versions of the test... but that each of these tests have significantly different levels of difficulty which, on average, is the cited value they give.
No. The percentiles corresponding to each scaled score designate what percentage of students got below the given scaled score in the 3 previous years, regardless of what version of the test they all took. The raw score to scaled score table is a completely different stat which only applies to the particular test booklet... Such tables on the practice tests are valid for gauging your performance on that one test, even the very old practice tests.
HappyQuark wrote:The thing that doesn't make sense to me about your interpretation, If correct, is this makes the average values they cite completely useless. In fact the table they give at the end of each practice exam is completely meaningless. When I took the practice exams, it was my understanding that when I calculated my raw score on a practice test, I could look at the score conversions table in the back to see what my raw score and percentile rank would have been and, so far as I can tell, this is how everyone else has interpreted it.
The scaled scores are intended to show the level of performance independent of what version of the test was taken. A student who gets a 720 with a raw score of 65 on a hard test would presumably get a higher raw score on an easier test, and end up with the same 720 scaled score in theory.

Percentiles are determined by the scaled scores of the recent pool of test takers. In fact, some forum users have pointed out (myself included) that their percentiles for their tests have changed from the time they took the test to the time they applied to schools. That's because the 3-year pool of recent test takers changes each year. The percentiles in the practice test booklets from the 80's and 90's will not be useful today because the 3 year period for determining those percentiles is way out of date... but the 2008 percentiles are probably close, though not exact, to what they will be in 2013.

I hope your questions are answered somewhere in there. :)

User avatar
Skullgrid
Posts: 59
Joined: Sat Aug 21, 2010 6:50 pm

Re: Setting Score Goals

Post by Skullgrid » Fri Apr 26, 2013 4:41 pm

quizivex wrote: In fact, some forum users have pointed out (myself included) that their percentiles for their tests have changed from the time they took the test to the time they applied to schools. That's because the 3-year pool of recent test takers changes each year.
I can confirm this. My percentile score from the first time I took the test (Oct '11) has changed since the original score report.



Post Reply