The March 10, 2018 edition of the Wall Street Journal featured a fascinating article on its Review section cover. This article, written by industrial-organizational psychology professors at the University of Minnesota, explores several test related myths the authors have identified. While we may not agree with every aspect of the article, the bottom line is that testing matters. It matters for admission. It matters for future success. And it matters for students to take the tests seriously. Read the article and see what you think.
The Truth About the SAT and ACT
Myths abound about standardized tests, but the research is clear: They provide an invaluable measure of how students are likely to perform in college and beyond
By
This Saturday, hundreds of thousands of U.S. high-school students will sit down to take the SAT, anxious about their performance and how it will affect their college prospects. And in a few weeks, their older peers, who took the test last year, will start hearing back from the colleges they applied to. Admitted, rejected, waitlisted? It often hinges, in no small measure, on those few hours spent taking the SAT or the ACT, the other widely used standardized test.
Standardized tests are only part of the mix, of course, as schools make their admissions decisions. They also rely on grades, letters of recommendation, personal statements and interviews. But we shouldn’t kid ourselves: The SAT and ACT matter. They help overwhelmed admissions officers divide enormous numbers of applicants into pools for further assessment. High scores don’t guarantee admission anywhere, and low scores don’t rule it out, but schools take the tests seriously.
And they should, because the standardized tests tell us a lot about an applicant’s likely academic performance and eventual career success. Saying as much has become controversial in recent years, as standardized tests of every sort have come under attack. But our own research and that of others in the field show conclusively that a few hours of assessment do yield useful information for admissions decisions.
Unfortunately, a lot of myths have developed around these tests—myths that stand in the way of a thoughtful discussion of their role and importance.
Myth: Tests Only Predict First-Year Grades
Longitudinal research demonstrates that standardized tests predict not just grades all the way through college but also the level of courses a student is likely to take. Our research shows that higher test scores are clearly related to choosing more difficult majors and to taking advanced coursework in all fields. At many schools, the same bachelor’s degree can be earned largely with introductory courses or with classes that approach the level of a master’s degree. Students with high test scores are more likely to take the challenging route through college.
Tests also predict outcomes beyond college. A 2007 paper published in the journal Science presented a quantitative review across thousands of studies and hundreds of thousands of students, examining the predictive power of graduate-school admissions tests for law, business, medicine and academic fields. It showed that the tests predict not only grades but also several other important outcomes, including faculty evaluations, research accomplishments, degree attainment, performance on comprehensive exams and professional licensure.
High-school and college grades are excellent measures for selecting students who are prepared for the next level. But we all know that a grade-point average of 3.5 doesn’t mean the same thing across schools or even for two students within a school. As high-school GPAs continue to go up because of grade inflation, having the common measure provided by admissions test scores is useful.
Myth: Tests Are Not Related to Success in the Real World
Clearly there are many factors, beyond what is measured by tests, that have an impact on long-term success in work and life. But fundamental skills in reading and math matter, and it has been demonstrated, across tens of thousands of studies, that they are related, ultimately, to job performance.
A 2004 meta-analysis published in the Journal of Personality and Social Psychology looked at results from a test that was designed for admissions assessment but was also marketed as a tool for making hiring decisions. Though originally intended as a measure of “book smarts,” it also correlated with successful outcomes at both school and work.
Longitudinal research has demonstrated that major life accomplishments, such as publishing a novel or patenting technology, are also associated with test scores, even after taking into account educational opportunities. There is even a sizable body of evidencethat these skills are related to effective leadership and creative achievements at work. Being able to read texts and make sense of them and having strong quantitative reasoning are crucial in the modern information economy.
Myth: Beyond a Certain Point, Higher Scores Don’t Matter
Some might concede that these skills are important—but only up to a point, beyond which higher scores don’t matter. It’s an understandable intuition, but the research clearly shows that, all else being equal, more is better.
One of us examined four large national data sets and found no evidence, in either work or academic settings, of a plateau where all relatively high scorers were roughly equal. If anything, the relationship between scores and success increased as scores went up. One theory for why this occurs is that people who score higher are more likely to seek out highly complex academic and work settings, where their cognitive skills are especially important.
A remarkable longitudinal study published in 2008 in the journal Psychological Science examined students who scored in the top 1% at the age of 13. Twenty years later, they were, on average, very highly accomplished, with high incomes, major awards and career accomplishments that would make any parent proud.
Yet, even within that group, higher scores mattered. Those in the top quarter of the top 1% were more likely than those merely at the bottom quarter of the top 1% to have high incomes, patents, doctorates and published literary works and STEM research.
Cognitive skills are not the only factor in success, of course. Our own research has demonstrated that, with certain elite cohorts, like applicants for executive positions, the abilities measured by tests are still important but less so than other characteristics. This is the same phenomenon as in professional basketball, where differences in height become less important among the extremely tall. This highlights the need to assess multiple characteristics with high-quality measures.
Myth: Common Alternatives to Tests Are More Useful
Admissions staff often rely on letters of recommendation, interviews and student essays and personal statements to create a complete picture of a student. It’s a worthy goal. Success is not just a function of high-school grades and test scores.
Unfortunately, most of these tools are not stellar indicators of future success. Letters of recommendation have some modest utility, but research shows that evaluations of student essays and personal statements have almost no relationship to how students ultimately perform. It is well known that traditional interviews are poor predictors (though structured interviews are much more effective). Problems with traditional interviews and letters of recommendation are so pervasive that many schools are looking for better options.
We know from extensive longitudinal research that many aspects of a person’s personality are associated with important life outcomes. Unlike typical personality measures, new measures that are resistant to faking in high-stakes settings are being developed. These measures can more accurately test a student’s character, getting at critical characteristics such as curiosity, empathy, resilience and determination. In addition, “situational judgment tests” that evaluate a person’s judgment in key school situations have been successfully used for medical school admissions and are being developed for admissions at all levels.
Myth: Tests Are Just Measures of Social Class
Admissions tests aren’t windows into innate talent; rather, they assess skills developed over years of education. They evaluate a student’s capacity to read and interpret complex prose, think critically and reason mathematically.
How well students develop these skills is influenced, of course, by many factors, including educational quality, high expectations, stable communities and families, and teacher behavior. It is a tragic reality that these factors are not equally distributed across social class and race in the U.S.
Studies have documented, for example, that the number of words and encouragements spoken to little children varies by socioeconomic status and that these differences are related to the development of verbal reasoning skills. Obviously, some kids from less well-off families grow up in a home environment where they encounter complex vocabulary and sentence structures, but many more do not.
Though we see exceptionally skilled students from all walks of life, the reality is that there is a correlation between test scores and social class. This doesn’t mean, however, that success on standardized tests and in college is simply dependent on class.
Our own comprehensive look at the issue, including a review of the existing literature and analysis of several large national data sets, showed that the tests were valid even when controlling for socioeconomic class. Regardless of their family background, students with good tests scores and high-school grades do better in college than students with lower scores and weaker transcripts.
Standardized tests are not just proxy tests of wealth, and many students from less affluent backgrounds do brilliantly on them. But the class differences in skill development are real, and improving the K-12 talent pipeline would be a huge benefit to the country.
Myth: Test Prep and Coaching Produce Large Score Gains
If tests were easily coached and coaching was only available to the wealthy, there would be an equity problem, even if tests are generally useful. Commercial test prep is clearly expensive, so this is a critical issue.
Researchers have conducted a mix of experimental studies and controlled field studies to test this question. They have generally concluded that the gains due to test prep are more on the order of 5 to 20 points and not the 100 to 200 points claimed by some test prep companies.
One review found a typical gain of 15 to 20 points on the math portion of the SAT and 8 to 10 points on the verbal portion. One of us conducted a more in-depth analysis of 4,248 high-school students and, after controlling for prior scores and the differing propensity of students to seek coaching, we estimated a gain of 14 points on the math test and 4 points on the verbal.
These are just averages, and among students who prep, a small percentage do realize 100 point gains. Why? The research suggests that they fall into two overlapping groups. The first consists of students who are fundamentally well prepared but are rusty on some basic concepts. The second group has not put even basic effort into understanding the questions and the flow of the tests. Gaining simple familiarity is one of the surest ways to achieve quick increases in scores.
Most experts want students to prep. Tests are generally more valid when everyone has had preparation because scores then reflect the application of fresh skills and not differences in basic familiarity with the test. The College Board, which administers the SAT, has partnered with Khan Academy to offer free test prep. Such training is valuable, and having accessible prep materials helps to improve both student scores and the validity of the test.
Myth: Tests Prevent Diversity in Admissions
Do standardized tests have a negative impact on the admission of a racially diverse student body? A good test of this would be to look at schools where admissions tests are optional for applicants and compare them to schools that use the tests. Recent research demonstrates that testing-optional schools have been enrolling increasingly diverse student bodies. But the same is true of schools that require testing.
Similarly, in a 2012 study, we examined a sample of 110 colleges with a total of 143,000 students to see whether the admitted student body consists mostly of those from wealthier families or reflects the socioeconomic profile of the applicant pool as a whole. It turned out that the social class of the enrolled students mirrored the applicant pool.
If there is a social-class filter, it affects who is prepared for college and who chooses to apply. This deserves national attention, since there are many talented and hardworking students who, as we have said, are not getting the sort of education that would prepare them for college.
Ideally, students applying to college should be evaluated on many different pieces of information, including their academic skills, curiosity, drive and teamwork. But test scores should have an important role in admissions decisions. Differences in skill investment and development over the course of many years cannot be overcome quickly.
Some schools take addressing these gaps as their mission, while others assume an advanced baseline of skills and focus on pushing their students toward higher levels of achievement. Not all schools have the same goals, and that’s fortunate, given the realities of talent development across students in the U.S.
Standardized tests are just tools—very effective tools—but they provide invaluable information to admissions offices. They identify those students who need help catching up with fundamental skills and those who are ready to tackle advanced material and rapidly accelerate in their learning.
Drs. Kuncel and Sackett are professors of industrial-organizational psychology at the University of Minnesota. This essay is adapted from their chapter in “Measuring Success: Testing, Grades and the Future of College Admissions,” a new edited volume published by Johns Hopkins University Press. In the past they have received research funding from the College Board, which administers the SAT.
https://www.wsj.com/articles/the-truth-about-the-sat-and-act-1520521861