A continuing debate on the bilingual advantage: some recommendations from Kenneth Paap.

My conversation with Barbara Treccani started last february with this post in Italian: I bilingui sono più intelligenti? Ma che esagerazione!

In June, I was intrigued by the discussion between two leading groups in the research on the bilingual advantage and I summarized the dispute by matching some excerpts from their last published papers, in this post: Bilingualism and cognitive advantage: Bialystok versus de Bruin.

The journal Cortex promoted a Discussion Forum with the article by Paap and colleagues and then hosted the debate in the Bilingualism Forum.

Some days ago, I shared the post Do bilingual people have a cognitive advantage? by Neuroskeptic on Facebook and the conversation with Barbara Treccani continued.

My questions were: What do you think is the way out? Preregistration as Neuroskeptic says? What are the criteria for continuing the research on this topic?

She was very kind in writing the following answer and in sharing – with the permission of the author - an email by Paap to Klein, Craik and all contributors:

“when the issue of Cortex was published there was a lively debate between the invited contributors. In response to the email exchange between Ray Klein and Gus Craik (shared with us, the other authors), Kenneth Paap sent a long letter in which he summarizes a number of critical points that, I think, respond to your concerns.

I asked Ken for permission to share his thoughts because I believe that they are particularly relevant. Neuroskeptic talks of "depressing reading" about the lack of participation by the main supporters of the bilingual advantage (especially the group from Toronto). I do not get depressed but I am very puzzled. Judging by what I see and hear, we go beyond a simple "bias" towards results that show an advantage for the bilinguals.

Here is the letter:”

"Gus, Ray, and all intererested colleagues,

I applaud the willingness that Gus has shown to debate the issues as the Toronto group has, in the past, repeatedly declined invitations from myself and/or Ray to engage in joint symposiums or to comment on our published articles. In fact two such invitations were declined (not by Gus) by characterizing my work as “belligerent, anti-intellectual, and unprofessional.” The Toronto group has made no comments nor asked any questions when I presented evidence challenging the bilingual advantage hypothesis at Psychonomics in 2012, 2013, 2014 or 2015. Perhaps Toronto doth protest too little.

Gus complains that the “only recommendations I can find in the piece are that more null results should be published.” But Ray explicitly traces many of the recommendations articulated in Hilchey & Klein (2011), Hilchey, Saint-Aubin, & Klein (2015), and Paap, Johnson, & Sawi (2015). To name only a few:

(1) Acknowledge (rather than quibble with the observation) that there is a confirmation bias operating as researchers relegate null results to the file drawer and reviewers and editors give preference to positive results. I think there are “fervent true believers” and I admit that I lean toward “skepticism”. This language seems mild compared to accusations of being “belligerent, anti-intellectual, and unprofessional.”

(2) Stop doing severely underpowered studies with 40 or far fewer participants per language group and acknowledge that all studies with very large samples sizes have yielded null results.

(3) Stop ignoring and provide a response to the 80%+ of published tests that reported null results, especially those that are close replications of seminal studies (as endorsed by Jared, 2015; Morton; 2015; Cortex). Do bear in mind that Ray was quoting the phrase “colossal disregard for evidence”, but regardless of the phrasing there appears to be no reaction from proponents of the bilingual advantage hypothesis to the large-n studies, using many measures of EF, and including highly proficient, balanced, and early bilinguals that show no effects of bilingualism.

(4) Acknowledge (or simply stop citing) that many early reports of bilingual advantages had confounds that may have caused the group differences. Further, do a better job of matching language groups in new studies so that confounding factors need not be “controlled” by the inappropriate use of ANCOVA. I think “scholarly” lapses is actually a gentle way of characterizing some of the inappropriate interpretations of published Group x Trial Type interactions pointed out by Ray’s group or in our articles (and endorsed by Wagermakers, 2015, Cortex). These are not “abusive accusations”, they are specific critiques of analyses and interpretations, and certainly pale in comparison to “belligerent, anti-intellectual, and unprofessional”.

Gus observes that “there are many such positive results in the literature from a number of investigators, relating to children, young and older adults.” No one is ignoring those! The primary purpose of our Cortex target article was to discuss in detail the plausible alternative causes of those positive results. As we have recently enumerated, some very solid appearing results, like Prior & MacWhinney’s bilingual advantage in switching costs, have miserable replication records, viz. 3 replications out of 27 new tests. A large number (but modest proportion) of positive findings does not establish that a phenomena is real. Meta-analyses (of a biased database) consistently show a very small overall effect size and in the most recent case an effect size of zero (von Bastian et al., 2015, Psychonomics).

Gus also notes “an increasing number of largely uncontested reports of brain differences associated with bilingualism”. Both Rays’ group, my group, and the BCBL have acknowledged that there are functional and structural differences in the brains of bilinguals and monolinguals; but we have all pointed out that the reorganization can lead to better, worse, or no change in behavioral performance on tasks that require general EF. More important, studies showing differences in neuroscientific data usually show no differences in behavior. Also, the neuroscientific data is often ambiguous. Three reviews have pointed out specific instances where studies have shown opposing differences in BOLD or structure, but nonetheless interpret the difference as evidence for a bilingual advantage. These brain differences are interesting in their own right, but by themselves neither support nor disconfirm the bilingual advantage hypothesis.

Gus also points out that there are “reports of beneficial effects in some patient populations”. In our response to the comments on our Cortex target article we conclude that this evidence is also mixed: “Six studies have used a prospective cohort design following individuals without dementia at baseline… A study by Wilson et al. (2015) is the only one that resulted in a bilingual advantage. The other studies all produced nonsignificant results (Crane et al., 2009; Lawton et al., 2015; Sanders et al., 2012; Yeung et al., 2015; and Zahodne et al., 2014) with three trending in the direction of a monolingual advantage. Thus, the longitudinal studies show very little support for a bilingual advantage with the exception of the Wilson et al. study. One might note that the Wilson et al. study examined only the number of years of language instruction before age 18, did not measure current language use, only examined the onset of mild cognitive impairment, and found an advantage only on the non-amnestic tests. As we said in our target article, if the prospective studies are weighted more heavily, there is little evidence that bilingualism protects against cognitive decline.”

Finally, Gus suggests that “a more mature response to the contradictions and failures to replicate would be to investigate the conditions under which positive effects are and are not found…” This strategy has been ardently pursued by ourselves (see in particular Paap, Johnson, & Sawi (2014, JCP) and many other research groups for several years. A large section of our Cortex target article is devoted to evaluating this literature. Evaluating this effort led to our conclusion that if bilingual advantages exist they “are restricted to very specific and undetermined circumstances.” I personally think more research in this vein should be conducted, but also point out that Hartsuiker (2015, Cortex), for one, suggests that continuing to do so is a waste of time.

Cheers,

Ken

PS. I hope Ray and Gus have triggered a continuing dialog."

And I hope a continuing dialog will spread to other discussed topics in psychology and neuroscience, to find a way against bias and bad faith.