Efficacy of Informal Logic Courses

The Efficacy of Undergraduate Critical Thinking Courses

A Survey in Progress

Tim van Gelder

Preprint 1/2000
Department of Philosophy
University of Melbourne
Parkville VIC 3052
http://www.philosophy.unimelb.edu.au

Web address of this paper: http://www.philosophy.unimelb.edu.au/reason/efficacy.html

Note: This report is a work-in-progress. It is presented here as a webpage for ease of distribution and updating. Please contact me with details of any studies not covered here, or to provide corrections, supplementations, or discussion. In particular, if your favorite study is not reported here, please let me know.

Last update: Tue, Sep 26, 2000
Note: this paper is obsolete. We are producing a new updated version, which will incorporate a number of studies not mentioned here. It will be posted in a month or two.

The truth is that we know very little about reasoning and how to teach it. The one thing we thought we knew—namely, that formal discipline is an illusion—seems clearly wrong. Just how wrong, and therefore just how much we can improve reasoning by instruction, is now a completely open question.
- Lehman et al, 1988

Introduction
Focus & Clarification
Table 1
Table 2
Other Resources
Discussion
References

Introduction

Each year, in hundreds of colleges and universities around the world, students enrol in one-semester courses variously known as "Critical Thinking," "Informal Logic," "Introduction to Reasoning," etc..

One standard rationale for these courses is that they improve students' ability to think critically or engage in informal reasoning. The courses are often—perhaps normally—sold to students on this basis. In particular, CT/IL courses are often billed as being superior for this purpose to formal logic classes.

A student taking such a course might reasonably ask: What reason do we have to think that the standard rationale holds water? That is, what is the evidence that traditional one-semester courses in CT or IL actually do improve CT or informal reasoning ability? What is the evidence that such courses are any better for this purpose than other courses that might be taken instead?

This report surveys the available empirical evidence on the efficacy of courses in critical thinking or informal reasoning. Probably the two most important conclusions to emerge from the survey are:

Currently it is difficult to make a convincing case that CT/IL courses make an appreciable difference to CT or informal reasoning skills.

There is a serious need for more and better research on this issue.

For further discussion, see below.

Focus & Clarification

There has been in some places a tendency to assimilate critical thinking and informal reasoning. In my opinion they should be distinguished, with informal reasoning being just one component of critical thinking. My own primary interest is in informal reasoning. Nevertheless, in what follows I will blur over the difference and use the term "critical thinking" (CT) to cover either or both, depending on context.

The ideal demonstration of the efficacy of courses in CT would:

concern a course in CT
perform pre, post- and follow-up testing of a substantial number of students
perform the same testing on a well-constructed control group or groups
use an independently validated test of CT
find an effect that is not only statistically signficant but also substantial over and above (a) maturation, and (b) the effect of being at college

I am yet to discover any demonstrations meeting these reasonable but strict requirements. However there have been studies which approximate, to some greater or lesser degree the above design. The closest such approximations are summarized in Table 1.

To qualify for inclusion in Table 1 a study must:

concern a one-semester, undergraduate course (or courses)
concern a course in CT
attempt to measure improvement in general CT ability, and not just some narrow component such as performance on the Wason card test
generate real data, i.e., quantify observations
be published, or at least written up in a form suitable for distribution

Some noteworthy studies which fail to meet one or more of these criteria are summarized in Table 2.

Note that there are many different kinds of evidence one might bring to bear in attempting to understand the impact and value of a CT course. The focus of this report is on studies of the kind described above, but this should not be taken to imply that such studies are the only kind of evidence worth taking seriously. Different kinds of questions are best addressed with different kinds of investigation and evidence. The question "What is the efficacy of a CT course in improving CT skills?" is best addressed with methodolgically and statistically rigorous empirical studies. However other kinds of questions—e.g., what are typical patterns of misunderstanding? how do students apply their skills outside the classroom?—need rather different approaches.

Note that the summaries presented in Tables 1 and 2 should be taken as guides only. Interested readers should consult the original sources for accurate and detailed information.

Table 1

Studies of the efficacy of one-semester courses in critical thinking.

Study (alph. order)

Design

Testing Instrument

Result

Conclusion*

Notes

Annis & Annis, 1979

Pre- and post-testing 32 students in an introductory logic course. Comparison with students in Introductory Philosophy, Ethics; and a control group.

Watson-Glaser Critical Thinking Appraisal

"The multivariate analysis revealed a significant main effect of course on...[three subtests]... Which course a student was in had no significant effect on Recognition of Assumptions, Evaluation of Arguments, nor the total score on the Watson-Glaser." (149)

"Which course a student was in had no significant effect on... the total score on the Watson-Glaser..." (149)

"In general the results of the study indicate that Logic has a consistent impact on certain aspects of critical thinking..." (150)

Results section sketchy and difficult to interpret; appears to contain crucial gaps. Also: was the Introductory Logic course informal or formal logic?

Facione, 1990

Detailed, large-scale study comprising four experiments involving pre- and post-testing students in a variety of CT courses and comparing with controls

CCTST: California Critical Thinking Skills Test: College Level

Statistically significant differences detected for students in CT courses. Control group showed (borderline) nonsignificant improvement.

"We can be confident that the CCTST succeeds in detecting the growth in CT skills which is hypothesized to occur during college level instruction specifically designed for the purpose of CT development."

Primary goal of study was to validate CCTST. If validity of CCTST is assumed, study indicates efficacy of CT instruction.

Pascarella, 1989

Pre- and post-testing of 47 students at beginning and end of first year at college; comparison with 20 matched controls not attending college.

Watson-Glaser Critical Thinking Appraisal

precise data on relation between improvement on WGCTA and logic courses not reported in this paper

"other specific measures of the college experience such as curricular emphasis (number of science and logic courses)...failed to have significant partial associations with critical thinking, independent of the study covariates."

Efficacy of CT courses only a very minor aspect of this study. Lack of "significant partial associations" leaves open possibility that science and logic courses are having some effect.

Rainbolt & Rieber, 1998

Test of a web-based approach to teaching. 2 instructors taught two sections of CT each using same text, each instructor teaching one section using web and one section using traditional approach.

CCTST: California Critical Thinking Skills Test: College Level

Gain using web approach: 2.42 and 3.77 questions correct. Gain using traditional approach: 2.0 and 2.8 questions correct.

the students in the web class did significantly better than those in the control classes. It is interesting to note, however, that the improvement which seems to have resulted from use of the web is much smaller than the improvement that seems to have resulted from using regular (tenure-track) faculty instead of part-time instructors...Much more testing needs to be done but the initial results of this method appear promising.

Study intended to compare two approaches to CT, not to assess the efficacy of CT courses. Nevertheless study appears to be picking up larger improvements than typically found in Facione's study using the same test.

van Gelder, 2000

Pre- and post-testing of students in an introductory reasoning class based on the Reason! software. Control group of students taking same course taught by traditional means.

1. Watson-Glaser Critical Thinking Appraisal
2. Locally developed test, somewhat similar to the GRE Writing Assessment (exp. group only)

Gains for experimental group; effect size approx. .45. Control group showed no improvement on WG.

1. Informal reasoning can be improved through instruction.
2. Courses based on intensive practice appear to be more effective than traditional courses.
3. Appropriately designed software can facilitate intensive practice.

no follow up testing in this study, but more studies of this kind are being conducted.

* The entry in this column is not necessarily the main conclusion of the study. Rather, of the conclusions drawn by the author(s), it is the one most relevant to the topic of this report.

Table 2

Some other studies relating to efficacy of reasoning instruction at improving critical thinking.

Study

Design

Test

Result

Conclusion

Notes

Reason for not including on Table 1

Beckman, 1956

"Non-equivalent pretest-posttest control group"; 303 students in 8 colleges and universities

Watson-Glaser Critical Thinking Appraisal

"No significant differences between the experimental and control classes. The differences in mean gain between colleges was significant."

All information here drawn from McMillan 1987.

It is not clear that this study concerns CT courses in the sense relevant here.

Glaser, 1941

high school students; experimental and control group

Test which developed into the Watson-Glaser CTA

"significant difference in performance"

All details here provided by DH

not college level

Leblanc (unpublished)

2 pre- and post-test studies; undergrads in one-semester CT class

Selection of seven questions of the kind found in LEMUR software package

mean gains of 1.8 and 1.25 questions correct out of seven

details communicated by Leblanc

not published; also test questions very closely related to taught material?

Lehman et al 1988

Subjects were first and third year graduate students in law, medicine, psychology and chemistry at University of Michigan. One study was cross-sectional, one longitudinal.

Specially developed reasoning test consisting of four sets of questions: statistical, methodological, conditional and verbal reasoning

for details see paper

"The results show that training in both psychology and medicine can affect statistical reasoning about everyday-life problems, methodological reasoning about everyday-life problems employing a number of rules related to the confounding principle, and reasoning about problems that logicians can solve using the material conditional in deductive logic. In addition, training in the law affects [only] conditional reasoning. Training in chemistry does not seem to affect any of these kinds of reasoning." (440) "The present results indicate that 20th century psychologists have been too quick to conclude that formal discipline is not possible and that rule training has little generalized educational potential..." (441)

not a study of a CT course

Stenning et al., 1995

35 Stanford undergraduates taking two different versions of a 12-week introductory logic course, one based on Hyperproof (Barwise 1994) and the other a traditional course.

Two tests developed by the authors:
1. a "blocks world" test based on Hyperproof graphics
2. A "pseudo-GRE" reasoning test

Significant improvement found on the blocks world test and on the "analytical subscale" or the GRE test.

"The results show good transfer of learning on both conventional and HP courses, but strong interactions between pre-existing individual differences and methods of teaching...We believe this study provides a useful existence proof that logic courses of a variety of kinds can transfer to other reasoning."

found that in some cases instruction can actually reduce performance

concerns effect of formal logic instruction on CT

van der Pal & Eysink, 1999

80 undergraduate students underwent various forms of instruction, 3-5 hours duration, using Tarski's World (Barwise 1993). Pre-, post- and one-week follow-up testing.

A test consisting of "four arbitrarily selected Wason selection tasks"

Significant improvement found in both post- and follow-up tests; up to 1.28SD on one way of calculating improvement ("logic index").

"The results revealed a facilitating effect of the complete Tarski's World instruction on the Wason selection tasks...This study signifies the importance of interactive graphical representations when used in combination with formal systems for logic instruction..." (p. 338)

a laboratory-type study, not a study of a one-smester course in CT or a general test of CT

Gehrett, 2000

Pre- and post-testing of 12 students (juniors and seniors) in a one-semester philosophy subject with substantial CT content

Cornell Critical Thinking Test, Level X

"While one student showed a loss, all the others gained. The average gain was 6.6 score points, and five students gained more than ten points." (p.32)

"I must stress that I did not do a formal analysis of the test scores, but I think improvement is clearly demonstrated. Obviously, this single test cannot be taken too seriously..." (p.32)

Paper contains many interesting qualitative observations about improving CT skills in a one-semester subject

Not an undergraduate course

Other Resources

Allen et al, 1999.

Abstract: This meta-analysis considers the impact of various methods of improving public communication skills on critical thinking. The results indicate that communication instruction improves the critical thinking ability of the participants (longitudinal designs, r=.18, cross-sectional designs, r=.20). Forensic participation demonstrated the largest positive impact on critical thinking improvement, but all communication skill experiences demonstrated significant improvement. The cumulative evidence indicates that communication skill instruction generates, using the Binomial Effect Size Display (BESD), a 44% increase in critical thinking ability...

Facione & Facione, 1997

A very large scale study of critical thinking in nursing students using the California Critical Thinking Skills Test and the California Critical Thinking Disposition Inventory. The study finds modest but statistically significant gains in critical thinking, suggesting that "overall something good is happening in the area of critical thinking skills." (p.26) Also, "gains appear to be concentrated in the evaluation scale... and inductive reasoning scale... These results suggest that the students' formal reasoning skills, that is, their skills at analytical thinking and drawing deductive conclusions based on abstract thought structures, are not the main focus of growth. However, the students have improved their skills at drawing probabilistic inferences and evaluting the relative strength of a body of information provided on behalf of a purported conclusion. This is an excellent result. It suggests that nursing students are learning to consider a wider range of possibilities and available data as they form judgements about what to believe or what to do." (27-8)

McMillan, 1987.

Abstract: "Twenty-seven studies are reviewed that investigate the effect of instructional methods, courses, programs, and general college experiences on changes in college students' critical thinking. Only two studies used true experimental designs; most were nonequivalent pretest-posttest control group designs. The results failed to support the use of specific instructional or course conditions to enhance critical thinking, but did support the conclusion that college attendance improves critical thinking. What is lacking in the research is a common definition of critical thinking, good instrumentation to provide specific measurement, and a clear theoretical description of the nature of an experience that should enhance critical thinking."

Pascarella & Terenzini, 1991

Chapter 4 provides a comprehensive summary of research on the development of "Cognitive Skills and Intellectual Growth" during undergraduate education. It describes overall changes in critical thinking as well as how various factors apparently do, or do not, affect growth in critical thinking. Noteworthy is their conclusion that attendance at college has a net positive effect on critical thinking, over and above maturation effects, and that a disproportionate share of this improvement takes place in the first year. Apparently a new version of this book will be coming out soon.

Lipman & Gazzard 1988

This paper contains a table, similar to table 1 above, detailing some 11 studies all showing significant improvement on reasoning skills in students who had completed Philosophy for Children programs. It is certainly an impressive advertisement for the Philosophy of Children program. However this table is contained in a section entitled "Evidence of Effectiveness" [of PforC] and one wonders whether all the evidence for non-effectiveness had simply not been mentioned. For another more recent study claiming effectiveness for PforC, see (Morehouse & Williams, 1998).

Discussion

The following points are tentative, based on the evidence listed above.

The available evidence on the efficacy of standard CT courses is the proverbial dog's breakfast. There is only a small number of studies bearing closely on the topic; these studies are heterogeneous, and in each case there are reasons why it is difficult to draw any firm conclusions about CT courses in general.

Currently it is difficult to make a convincing case—i.e., one that would survive hard scrutiny from good critical thinkers—that CT courses are of any substantial benefit. On one hand there are various studies indicating no significant benefit from CT instruction. On the other, there are some studies which do appear to find some benefit.

Currently one of the strongest improvement effects for a CT course is that described in (van Gelder 2000). Note, however, that the improvement in this case was gained in a highly non-traditional course, based on intensive practice in a software environment. It does not offer much support for the idea that traditional CT courses substantially improve CT, and indeed in that study the traditionally-taught comparison group showed no improvement on the Watson-Glaser.

The belief, common among CT teachers, that CT courses are better for improving CT than formal logic courses does not appear to be supported by the available evidence, such as it is. (I have not been actively seeking out studies on the efficacy or otherwise of formal logic courses at improving CT skills.)

It is possible to make a somewhat better case that CT can be improved through education. It appears that there are a number of educational strategies which improve CT either more substantially, or more reliably, than CT courses. Crudely, if you want your child to grow up to be a critical thinker, you should have her engage in Philosophy for Children; send her to college; have her participate in forensics and give her a touch of formal logic; and round it off with a PhD in psychology.

An important question, which is left unresolved by these studies, is whether CT courses harm their students. It appears possible that typical CT courses actually reduce CT performance. Studies discussed by Pascarella & Terenzini indicate that college students usually make substantial gains in CT at college and particularly during their first year. This improvement is partly due to maturation and partly due to being at college. A study of a first-year CT course should register this improvement. If the study finds no improvement, or no significant improvement, it is possible that the CT course is hindering the normal development of CT. Note that Stenning et al. did in fact conclude that for some students under certain circumstances logic training inhibits general reasoning.

There appears to be an urgent need for further, high quality studies. The CT community claims that taking CT courses improves CT skills. It ought to be possible to provide better evidence.

Acknowledgements

Thanks for assistance to Jon Barwise, Edwin Coleman, Peter Facione, Dale Hample, David Hitchcock, Jill LeBlanc, Craig McInnis, Michael Scriven, Keith Stenning, Neil Thomason, Jelke van der Pal, Mark Weinstein, and various contributors to the ARGTHRY list.

References

Allen, M., Berkowitz, S., Hunt, S., & Louden, A. (1999) A meta-analysis of the impact of forensics and communication education on critical thinking. Communication Education, 48, 18-30.

Annis, D., & Annis, L. (1979) Does philosophy improve critical thinking? Teaching Philosophy, 3, 145-152.

Barwise, J., & Etchemendy, J. (1993) The Language of First Order Logic. Stanford CA: Center for the Study of Language and Information.

Barwise, J., & Etchemendy, J. (1994) Hyperproof. Chicago: Chicago University Press.

Beckman, V. E. (1956) An investigation of the contributions to critical thinking made by courses in argumentation and discussion in selected colleges (Doctoral dissertation, University of Minnesota). Dissertation Abstracts International, 16, 2551A.

Facione, P. A. (1990) CCTST experimental validation and content validity: CCTST technical report #1. ERIC, ED 327-549. 1-19. Millbrae, CA: The California Academic Press.

Facione, N. C., & Facione, P. A. (1997) Critical Thinking Assessment in Nursing Education Programs: An Aggregate Data Analysis. Millbrae, CA: The California Academic Press.

Gehrett, C. (2000) Doing philosophy in high school: One teacher's account. Inquiry: Critical Thinking Across the Disciplines, 19, 27-35.

Glaser, E. M. (1941) An Experiment in the Development of Critical Thinking. New York: Columbia University Teacher's College.

Lehman, D. R., Lempert, R. O., & Nisbett, R. E. (1988) The effects of graduate training on reasoning: Formal discipline and thinking about everyday-life events. American Psychologist, 431-442.

Lipman, M., & Gazzard, A. (1988) Philosophy for Children: Where are we now? Thinking, S2-S11.

McMillan, J. (1987) Enhancing college student's critical thinking: A review of studies. Research in Higher Education, 26, 3-29.

Morehouse, R., & Williams, M. (1998) Report on student use of argument skills. Creative and Critical Thinking, 6, 14-20.

Pascarella, E. (1989) The development of critical thinking: Does college make a difference. Journal of College Student Development, 30, 19-26.

Pascarella, E. T., & Terenzini, P. T. (1991) How College Affects Students: Findings and Insights from Twenty Years of Research. San Francisco: Jossey-Bass.

Rainbolt, G. & Rieber, S. Critical Thinking on the Web Information Page. http://www.gsu.edu/~wwwphl/241web/demostart.html

Stenning, K., Cox, R., & Oberlander, J. (1995) Contrasting the cognitive effects of graphical and sentential logic teaching: reasoning, representation and individual differences. Language and Cognitive Processes, 10, 333-354.

van der Pal, J., & Eysink, T. (1999) Balancing situativity and formality: the importance of relating a formal language to interactive graphics in logic instruction. Learning and Instruction, 9, 327-341.

van Gelder, T. J. (2000) Learning to reason: A Reason!-Able approach. Submitted to Proceedings of the Fifth Australasian Cognitive Science Society Conference, Melbourne Jan 2000. Singapore: World Scientific.

Study (alph. order)	Design	Testing Instrument	Result	Conclusion*	Notes
Annis & Annis, 1979	Pre- and post-testing 32 students in an introductory logic course. Comparison with students in Introductory Philosophy, Ethics; and a control group.	Watson-Glaser Critical Thinking Appraisal	"The multivariate analysis revealed a significant main effect of course on...[three subtests]... Which course a student was in had no significant effect on Recognition of Assumptions, Evaluation of Arguments, nor the total score on the Watson-Glaser." (149)	"Which course a student was in had no significant effect on... the total score on the Watson-Glaser..." (149) "In general the results of the study indicate that Logic has a consistent impact on certain aspects of critical thinking..." (150)	Results section sketchy and difficult to interpret; appears to contain crucial gaps. Also: was the Introductory Logic course informal or formal logic?
Facione, 1990	Detailed, large-scale study comprising four experiments involving pre- and post-testing students in a variety of CT courses and comparing with controls	CCTST: California Critical Thinking Skills Test: College Level	Statistically significant differences detected for students in CT courses. Control group showed (borderline) nonsignificant improvement.	"We can be confident that the CCTST succeeds in detecting the growth in CT skills which is hypothesized to occur during college level instruction specifically designed for the purpose of CT development."	Primary goal of study was to validate CCTST. If validity of CCTST is assumed, study indicates efficacy of CT instruction.
Pascarella, 1989	Pre- and post-testing of 47 students at beginning and end of first year at college; comparison with 20 matched controls not attending college.	Watson-Glaser Critical Thinking Appraisal	precise data on relation between improvement on WGCTA and logic courses not reported in this paper	"other specific measures of the college experience such as curricular emphasis (number of science and logic courses)...failed to have significant partial associations with critical thinking, independent of the study covariates."	Efficacy of CT courses only a very minor aspect of this study. Lack of "significant partial associations" leaves open possibility that science and logic courses are having some effect.
Rainbolt & Rieber, 1998	Test of a web-based approach to teaching. 2 instructors taught two sections of CT each using same text, each instructor teaching one section using web and one section using traditional approach.	CCTST: California Critical Thinking Skills Test: College Level	Gain using web approach: 2.42 and 3.77 questions correct. Gain using traditional approach: 2.0 and 2.8 questions correct.	the students in the web class did significantly better than those in the control classes. It is interesting to note, however, that the improvement which seems to have resulted from use of the web is much smaller than the improvement that seems to have resulted from using regular (tenure-track) faculty instead of part-time instructors...Much more testing needs to be done but the initial results of this method appear promising.	Study intended to compare two approaches to CT, not to assess the efficacy of CT courses. Nevertheless study appears to be picking up larger improvements than typically found in Facione's study using the same test.
van Gelder, 2000	Pre- and post-testing of students in an introductory reasoning class based on the Reason! software. Control group of students taking same course taught by traditional means.	1. Watson-Glaser Critical Thinking Appraisal 2. Locally developed test, somewhat similar to the GRE Writing Assessment (exp. group only)	Gains for experimental group; effect size approx. .45. Control group showed no improvement on WG.	1. Informal reasoning can be improved through instruction. 2. Courses based on intensive practice appear to be more effective than traditional courses. 3. Appropriately designed software can facilitate intensive practice.	no follow up testing in this study, but more studies of this kind are being conducted.

Study	Design	Test	Result	Conclusion	Notes	Reason for not including on Table 1
Beckman, 1956	"Non-equivalent pretest-posttest control group"; 303 students in 8 colleges and universities	Watson-Glaser Critical Thinking Appraisal	"No significant differences between the experimental and control classes. The differences in mean gain between colleges was significant."		All information here drawn from McMillan 1987.	It is not clear that this study concerns CT courses in the sense relevant here.
Glaser, 1941	high school students; experimental and control group	Test which developed into the Watson-Glaser CTA	"significant difference in performance"		All details here provided by DH	not college level
Leblanc (unpublished)	2 pre- and post-test studies; undergrads in one-semester CT class	Selection of seven questions of the kind found in LEMUR software package	mean gains of 1.8 and 1.25 questions correct out of seven		details communicated by Leblanc	not published; also test questions very closely related to taught material?
Lehman et al 1988	Subjects were first and third year graduate students in law, medicine, psychology and chemistry at University of Michigan. One study was cross-sectional, one longitudinal.	Specially developed reasoning test consisting of four sets of questions: statistical, methodological, conditional and verbal reasoning	for details see paper	"The results show that training in both psychology and medicine can affect statistical reasoning about everyday-life problems, methodological reasoning about everyday-life problems employing a number of rules related to the confounding principle, and reasoning about problems that logicians can solve using the material conditional in deductive logic. In addition, training in the law affects [only] conditional reasoning. Training in chemistry does not seem to affect any of these kinds of reasoning." (440) "The present results indicate that 20th century psychologists have been too quick to conclude that formal discipline is not possible and that rule training has little generalized educational potential..." (441)		not a study of a CT course
Stenning et al., 1995	35 Stanford undergraduates taking two different versions of a 12-week introductory logic course, one based on Hyperproof (Barwise 1994) and the other a traditional course.	Two tests developed by the authors: 1. a "blocks world" test based on Hyperproof graphics 2. A "pseudo-GRE" reasoning test	Significant improvement found on the blocks world test and on the "analytical subscale" or the GRE test.	"The results show good transfer of learning on both conventional and HP courses, but strong interactions between pre-existing individual differences and methods of teaching...We believe this study provides a useful existence proof that logic courses of a variety of kinds can transfer to other reasoning."	found that in some cases instruction can actually reduce performance	concerns effect of formal logic instruction on CT
van der Pal & Eysink, 1999	80 undergraduate students underwent various forms of instruction, 3-5 hours duration, using Tarski's World (Barwise 1993). Pre-, post- and one-week follow-up testing.	A test consisting of "four arbitrarily selected Wason selection tasks"	Significant improvement found in both post- and follow-up tests; up to 1.28SD on one way of calculating improvement ("logic index").	"The results revealed a facilitating effect of the complete Tarski's World instruction on the Wason selection tasks...This study signifies the importance of interactive graphical representations when used in combination with formal systems for logic instruction..." (p. 338)		a laboratory-type study, not a study of a one-smester course in CT or a general test of CT
Gehrett, 2000	Pre- and post-testing of 12 students (juniors and seniors) in a one-semester philosophy subject with substantial CT content	Cornell Critical Thinking Test, Level X	"While one student showed a loss, all the others gained. The average gain was 6.6 score points, and five students gained more than ten points." (p.32)	"I must stress that I did not do a formal analysis of the test scores, but I think improvement is clearly demonstrated. Obviously, this single test cannot be taken too seriously..." (p.32)	Paper contains many interesting qualitative observations about improving CT skills in a one-semester subject	Not an undergraduate course