Schunn, C. D., Crowley, K., & Okada, T. (1998). The growth of multidisciplinarity in the Cognitive Science Society. Cognitive Science, 22(1), 107-130.

[Disclaimer: There may be minor difference between the online and the journal versions.]

Carnegie Mellon University

University of California, Santa Cruz

Nagoya University

Author notes: Christian Schunn, Department of Psychology; Kevin Crowley, Department of Psychology; Takeshi Okada, Department of Educational Psychology. Portions of the data have been presented in partial form in Schunn, Okada and Crowley (1995), and Schunn, Crowley and Okada (1996). Work on this project was support by a grant from the Mitsubishi Bank Foundation to all three authors. We would like to thank Johanna Moore, Craig Stark, and the UCSC Cognition and Culture group for their assistance, and John Anderson, Bobby Klatzky, David Klahr, Don Norman, Daniel Dennett, Paul Thagard, and James Greeno for valuable comments on earlier drafts. Correspondence regarding this manuscript should be addressed to: Christian Schunn, Department of Psychology, MSN 3F5, George Mason University, Fairfax, VA 22030-4444 USA; E-mail: schunn@gmu.edu.

A phenomenon of great interest to the understanding of the process of science is the emergence of a new discipline. New disciplines can develop as an outgrowth from one particular discipline (e.g., psychology from philosophy), or through the combination of multiple disciplines (Strauss, 1982). In this article we focus on a recent instance of this second process: The emergence of a new discipline called cognitive science.

Definitions of cognitive science typically emphasize the field’s multidisciplinary roots (Collins, 1977; Hardcastle, 1996; Simon, 1982; Simon & Kaplan, 1989). Among the disciplines named as contributors to the foundations of cognitive science are anthropology, artificial intelligence, education, engineering, human-computer interaction, linguistics, medicine, neuroscience, philosophy, psychology, sociology, and others. However, cognitive science is thought to be an emergent discipline in and of itself, rather than just the superset of its constituent disciplines. Cognitive science forums have stated explicitly from the beginning that they place a strong priority on material that is relevant to audiences from numerous disciplines, and that material relevant to only one discipline is inappropriate (e.g., Collins, 1977). Moreover, it has been argued that cognitive science is not just cognitive psychology with some additional bells and whistles (Hardcastle, 1996).

Multidisciplinary fields are thought to arise when members of historically distinct disciplines have something to gain from one another (cf. Clarke, 1990). This "something" could be new issues, new theories, new methodologies, or simply a new audience for discussing common problems and ideas (e.g., Dunbar, 1995; Okada, Schunn, Crowley, Oshima, Miwa, Aoki, & Ishida, 1995; Okada, Crowley, Schunn, & Miwa, 1996; Thagard, 1994). For example, in 1950s, Herbert A. Simon, an economist and political scientist, worked with Allen Newell, a systems researcher with a mathematics background, and Cliff Shaw, an expert programmer, to create a new kind of computer program called Logic Theorist. Their multidisciplinary collaboration not only addressed the immediate concerns of the research project, but also opened up new questions and new methodological possibilities that eventually contributed to the foundation of cognitive science. In many ways, Simon, Newell, and Shaw’s collaboration might be considered a canonical example of the power and possibility of multidisciplinary cognitive science research.

In this article we examine the extent to which the current state of cognitive science reflects the multidisciplinary ideals of its foundation. Is it the case that constituent disciplines have merged to form a new discipline, or is it the case that cognitive science forums have become places where psychologists present psychology, computer scientists present computer science, linguists present linguistics, etc. Have cognitive scientists simply cited work from other disciplines while continuing to conduct traditional within-discipline research, or have they integrated new theories and methodologies to form a unique mode of cognitive science research. Are cognitive scientists working only with their disciplinary colleagues, or are they seeking out multidisciplinary collaborations?

We approach these questions through analysis of the social worlds of cognitive science. The creation of a new discipline is thought to involve the development of new social worlds through a process of social negotiation (Strauss, 1979), where a social world is defined as a group with shared commitments to certain activities sharing resources of many kinds to achieve their goals (Strauss, 1982). Two particular kinds of social worlds are especially important for an academic discipline (Gearson, 1983; Kling & Gearson, 1977, 1978): a production world in which the activities produce something (e.g., books, journals articles) and a communal world in which the activities focus on the establishment and maintenance of communities of people committed to each other and their shared goals (e.g., departments, societies, etc.).

We focus our investigation on a case study of one historically influential force in the field of cognitive science: the Cognitive Science Society, consisting of an important production world in cognitive science (the journal Cognitive Science) and an important communal world in cognitive science (the Annual Meeting of the Cognitive Science Society). The issue of selecting a case for study is non-trivial because a complex field such as cognitive science has numerous outlets, all of which are slightly different in disciplinary structure. We focus on the journal Cognitive Science and the Annual Meeting of the Cognitive Science Society because they are the only outlets with the title ‘cognitive science’ that are not restricted to particular subissues or particular geographical regions; their mission statements include many of the disciplines typically thought of as part of cognitive science; and there are no other clear candidates that would better represent the field of cognitive science. The other obvious journals that might be candidates for cognitive science outlets each have important flaws as representatives of cognitive science research: Cognition includes little computer science, and Brain and Behavioral Sciences has a scope larger than cognitive science. Further, BBS is a very atypical research forum. Although we believe the activities of the Cognitive Science Society can serve as a useful first-approximation of the state of cognitive science, it is important to acknowledge the possibility that analysis of other journals or conferences (or more likely, combinations of different journals or conferences) may reveal a more complete picture.

Beyond contributing to an understanding of how new scientific disciplines are created, we hope that our findings may also be of some practical use to cognitive scientists. The ideal and actual nature of cognitive science has been a frequent topic of conversation among cognitive scientists. One reason for these conversations may be the realization that multidisciplinary science has particular reasons to pay attention to the relative influence of constituent disciplines on editorial policies, reviewing practices, conference organizational strategies, research funding, and the training of future scientists. By providing concrete data on these issues, we hope to support dialogue about whether the current status of the Cognitive Science Society (and perhaps cognitive science as a whole) meets the aspirations of cognitive scientists.

The journal Cognitive Science was first published in 1977. It has always been a quarterly journal with approximately three to five papers per issue. In 1980, Cognitive Science became the official journal of the Cognitive Science Society and was automatically sent to all members. That same year, an editorial by Allan Collins stated that the range of the journal had broadened to include linguistics, philosophy, developmental psychology, cognitive anthropology, and neuroscience (Collins, 1980). In 1985, the journal merged with Cognition and Brain Theory, which had similar agenda although a greater focus on philosophy and neuroscience (Waltz, 1985; Ringle & Arbib, 1984).

The subtitle of the journal has reflected these changes (Table 1). Although the Collins editorial acclaimed the broadening of the journal's scope in 1980, it was not until 1988 that the subtitle of the journal reified the growth of cognitive science beyond artificial intelligence, psychology, and language. At this time, the new subtitle changed "language" to "linguistics", added neuroscience and philosophy, and alphabetized the list of disciplines. The most recent change in 1997 saw the addition of education and anthropology.

Table 1. Changes over the years in the subtitle for the journal Cognitive Science.

To assess the level of multidisciplinary work in the journal, we gathered three measures: 1) the departmental affiliations of authors who published in the journal; 2) the methodology used in the reported research; and 3) the discipline of the literature cited in articles published in the journal. We gathered data on major non-solicited articles; invited/special issue papers, editorials, book reviews, comments, and responses were excluded. To examine the evolution of these measures throughout the history of the journal, we compared the 73 articles published in the first five years of the journal (1977—1981), the 75 articles published in the middle five years (1984—1988), and the 73 articles published in the most recent five years (1991—1995). Inter-rater agreement for all coding reported below exceeded 90%.

The places where cognitive scientists work are one measure of their disciplinary background. The first authors of articles published in the journal were coded as being affiliated with Psychology, Computer Science, Linguistics, Philosophy, Neuroscience, Education, Cognitive Science, Industry (e.g., Xerox PARC), and Other (e.g., Anthropology, Business, Mathematics). Ideally, one would prefer to code affiliations at the level of cognitive psychology and artificial intelligence rather than just psychology and computer science, since those subareas are the constituents of cognitive science. However, department affiliations are typically listed only at the bigger levels of psychology and computer science. Only first authors were coded because the journal typically does not list departmental affiliations for other authors.

Table 2. Percentage of first authors with given departmental affiliations for the different time ranges of the journal Cognitive Science.

In general, the departmental affiliations of first-authors have remained similar throughout the history of the journal (Table 2). Although there have been minor (and non-significant) fluctuations in the relative percentage of psychology and computer science affiliations over the years, these two disciplines have consistently accounted for the majority of first-author affiliations. Authors from linguistics, philosophy, neuroscience, industry, and other fields have been only occasional contributors to cognitive science, and, if anything, are becoming less frequent contributors in recent years.

The only affiliation to change significantly over time is cognitive science itself. From a minor role in early years, first authors affiliated with cognitive science departments have increased their participation to account recently for almost one-fifth of all articles. This increase probably reflects increases in the number of cognitive science departments in existence. In support of this interpretation, an on-line search of the affiliation fields in the PsycInfo and ERIC library databases revealed that there the number of departments and institutes of cognitive science worldwide had grown to at least 47 by 1996. In other words, the primary change at the level of communal worlds relating to cognitive science was the creation of a new set of communal worlds directly focused on cognitive science.

A second measure of the multidisciplinary status of the Cognitive Science Society can be obtained by examining the methodologies of the work reported in the journal. Since the previous analyses revealed that the journal is dominated by scientists affiliated with psychology and computer science departments, we focus particularly on the primary methodologies from those two disciplines: empirical studies of behavior and computer simulation. Each article was coded into one of five mutually-exclusive categories: simulation only; empirical study only; simulation+empirical study; simulation of data; or neither simulation or empirical study. To be coded as having a simulation, an article must have described a running computer model or a mathematical model for which detailed predictions were produced–purely verbal models or flow diagrams were not counted. To be coded as having an empirical study, articles had to provide full presentation of the study including methods and results. The simulation+empirical study category was a simple conjunction category. The simulation of data category was used for articles presenting simulations of empirical data sets that had been presented elsewhere. To be included in this category a particular data set had to be identified–references to general human or animal phenomena were not sufficient.

Table 3. Within in each time range, the percentages of journal contributions categorized according to each methodology category.

Table 3 presents the changes in use of simulation and empirical studies throughout the history of the journal. Over the years, about half of the articles published in Cognitive Science have relied on a monodisciplinary method, presenting either empirical studies only or simulations only. The proportion of articles presenting only empirical studies increased 10% in recent years, while articles presenting just simulations declined somewhat between early and middle years before stabilizing in recent years. Articles combining simulation with empirical studies or empirical data sets continue to remain less prevalent than monodisciplinary methods. However, the use of multidisciplinary methods in the journal has almost doubled from middle to recent years, with most of that growth coming from an increase in the proportion of articles presenting simulations of previously published data sets. In early and middle years, articles containing neither simulations nor empirical studies were quite common, accounting for nearly one-third of all articles; recently the publication of these articles has dropped precipitously.

These findings suggest that, although the largest share of research in the Cognitive Science Society continues to rely on monodisciplinary methodology, there has been a recent rise in the prevalence of work combining the methods of computer science and psychology within the same article. The increased use of multidisciplinary methods could reflect changes in the Society or cognitive science as a field, or it could reflect changes in psychology and computer science in general. For example, perhaps mainstream computer scientists have begun conducting a significant number of empirical studies, or mainstream psychologists have begun conducting a significant number of simulations.

To assess this possibility, we coded recent issues of two main journals from within each of the disciplines of artificial intelligence and cognitive psychology (see Table 4). To assess whether another multidisciplinary journal might reveal a different picture of the field of cognitive science than Cognitive Science, we also analyzed methodology in Cognition, a journal that contains significant amounts of philosophy, neuroscience and linguistics in addition to psychology. We coded recent issues from each of the journals until there a sufficient number of them to produce reliable estimates. Since the journals varied greatly in frequency and number of papers per issue, this produced codings of: July 1995—January 1996 of JEP:LMC, all issues of 1995 of Cognitive Psychology, November 1994—December 1995 of AI, Volumes 20—23 (mid 1995 — mid 1996) of Machine Learning, and October 1995—February 1996 of Cognition.

Table 4. From recent issues of journals within cognitive psychology and artificial intelligence, the percentages of contributions categorized according to each methodology category.

From these results, one can see that there are fairly few simulations in mainstream cognitive psychology and no empirical studies in artificial intelligence. Thus, the multidisciplinarity found in Cognitive Science is unlikely to be a simple reflection of changes within the constituent disciplines of psychology and computer science–the researchers publishing in Cognitive Science are neither prototypical cognitive psychologists nor prototypical computer scientists. This difference is especially clear in the high proportion of papers that contain neither simulation nor experiment in the mainstream artificial intelligence (typically taking the form of mathematical proofs)–such papers occur far less frequently in cognitive science. Another point of note in Table 4 is that there are few simulations and a large percentage of psychology experiments in the journal Cognition. From these results, it seems clear that Cognition does not represent a more rounded perspective of the field of cognitive science than does the journal Cognitive Science.

A third measure of the multidisciplinary nature of cognitive science is the literature cited by authors who publish their work in the journal Cognitive Science. For each article, we coded every reference by the discipline it represented: psychology, artificial intelligence, linguistics, neuroscience, philosophy, education, business/economics, medicine, engineering, cognitive science, other social science, or other. The disciplines of journal articles were assigned according to the classifications in Ulrich’s international periodical directory. The disciplines of conferences were assigned according to the name of the conference. The disciplines of technical reports were assigned according to the department publishing the report.

Books and book chapters had no clear external means for determining their disciplinary affiliation and were coded according to a best guess of their content. When a book or book chapter was too ambiguous to code, coders were instructed to exclude that item. This occurred for 39% of book references (20% of all references). Given the disciplinary ambiguity of books and book chapters in general, we were faced with the decision of excluding all such references or excluding only those references that were too ambiguous to code. We present analyses that include the codeable books and book chapters because some disciplines publish much more often in books than in journals or conferences; because coders had good agreement on those books and chapters that were coded; and because when the data were analyzed excluding all books and book chapters, the overall pattern of results did not change. It is possible that the excluded references may have included disproportionately many multidisciplinary references, thereby producing a bias in our results. However, careful investigation of a subset of these ambiguous references suggests that this was not the case. The three most common reasons that references were not coded were non-English titles, very short book titles that proved to contain work from only one discipline, and missing departmental affiliations from dissertation and technical report references.

As the reference coding involved significant labor, we coded references from three years within each of the age ranges (i.e., 9 years total). These were selected by taking the first, middle and last year within each five-year age range. This produced 1075 references from beginning years, 1976 references from the middle years, and 2298 references from recent years.

As a first approximation of disciplinary influence, we counted the number of references to previous work from each of the different disciplines. Consistent with the affiliation and methodology findings, the citation data suggest that psychology and computer science have historically remained the most influential disciplines in the journal (Table 5). Citations of psychology sources increased somewhat between the beginning and middle time periods and remain the most frequently cited source in recent times. Computer science was initially the most frequently cited discipline, although in middle times citations plunged 40% before recovering somewhat to represent slightly more than one-fifth of all citations. This drop in computer science references and number of authors with computer science affiliations in the early 1980s coincides with the formation of AAAI, which provided another important outlet for artificial intelligence research.

Table 5. For Beginning (’77-’81), Middle (’84-’88), and Recent (’91-’95) time periods, the mean percentage of citations to work in each discipline.

Note. *=p<.05, **=p<.01 for changes across time.

The only other disciplines to represent more than 10% of citations at any point in the historical eras we sampled are cognitive science and linguistics. The citations from cognitive science sources have increased steadily over the years, rivaling computer science in recent times by accounting for just under one-fifth of all citations. Although citations of linguistics sources have remained relatively low and even decreased slightly over the history of the journal, it is interesting to note the participation of linguistics in the journal is larger in the citation data than in the affiliation data. While very few linguists publish in Cognitive Science, it appears that a fair number of Cognitive Science authors read linguistics.

To determine whether these averages were representative of individual articles, we calculated the percentage of each article's reference list that was affiliated with each of the constituent disciplines of cognitive science. In Table 6 we present findings using a conservative criterion of disciplinary influence (a discipline was said to have influenced an article if at least 25% of the citations were affiliated with the discipline) and a more liberal criterion (at least 5% of the citations were affiliated with the discipline).

Table 6. For Beginning (’77-’81), Middle (’84-’88), and Recent (’91-’95) time periods, the percentage of papers with an influence of each discipline type, using conservative (>25%) and liberal (>5%) criteria.

Note. *=p<.05, **=p<.01 for changes across time.

When measured by the conservative criterion, the pattern of disciplinary influence was consistent with the overall percentage of citations from each discipline: Psychology and computer science have historically influenced the greatest number of articles in the journal. Cognitive science and linguists have also played a role, with cognitive science recently displacing linguistics as the third most influential discipline. Other disciplines were rarely major influences.

When measured by the liberal criterion, a somewhat different picture emerges. First, as one would expect, the influence of every discipline increased. The increase was most pronounced for the field of cognitive science. Although it was a major influence on less than one-fifth of recent articles when analyzed with the conservative criterion, it influenced almost every recent article (93%) with the liberal criterion. Consistent with prior findings, psychology is an increasingly strong influence, computer science remains important, and linguistics has assumed a decreasingly important role. Analyses using the liberal criterion also revealed that neuroscience has steadily increased in influence over the history of the journal.

Another way to consider these data is to analyze how many articles referred to multiple disciplines. Using the conservative criterion (>25%), slightly fewer than half of the papers referred to multiple disciplines (see Table 7). However, using the liberal criterion (>5%), almost all papers referred to prior work from multiple disciplines (see Table 7). These frequencies of multidisciplinarity did not change over the history of the journal (F(2,133)<1), nor did the mean number of influences per paper (F(2,133)<1).

Table 7. For each time range, the mean number of influences per paper (with standard deviations), and the percentage of papers categorized as multidisciplinary (# influences _ 2), using conservative (>25%) and liberal (>5%) criteria.

The finding that authors are reading work from other disciplines does not necessarily indicate that their research is multidisciplinary. Similarly, using methodologies from multiple disciplines does not necessarily imply reading a great deal of work from multiple disciplines. These possibilities raise the question: What is the relationship between the kinds of methodologies that are used and the disciplines that are read? An analysis of this relationship may reveal particular kinds of mono and multi-disciplinary research. For example, it may be that a particular discipline is cited only exclusively when that discipline’s methodology is used. As a related possibility, it could be that researchers are jumping disciplines–using methodologies from other disciplines to address questions in that discipline. For example, a cognitive psychologist could use simulations to address psychological issues, or could use simulations to address artificial intelligence issues. If either of these options were the case, then the citations should be primarily to the discipline associated with the methodology. By contrast, if researchers are bringing another discipline’s methodologies to bear on their own research issues, then there should be a high level of multidisciplinary referencing associated with each research methodology, including the monodisciplinary methods.

Table 8. For papers of each methodology type in the journal, the percentage of papers with more than one discipline of large influence (>25% of references), and the percentage of psychology and computer science references, using all years that were coded combined and with Recent years alone in parentheses.

Table 8 shows that this is generally the case, indicating that researchers are borrowing methodologies from other disciplines to address issues within their own disciplines. However, there is one clear exception–papers presenting only psychological experiments, which rarely made heavy references to multiple disciplines, instead made heavy references to psychology and little reference to computer science. Since previous analyses had indicated that computer scientists were not yet conducting experiments, this particular result is consistent with the general findings that pure cognitive psychology is considered a perfectly valid candidate for inclusion in Cognitive Science.

Summary

From the analyses of departmental affiliations, self-rated backgrounds, and discipline citations, a consistent picture of the journal Cognitive Science has emerged: it is dominated by psychology and computer science. The other definitionally-constituent disciplines of the discipline of cognitive science–linguistics, philosophy and neuroscience–appear to be unequal partners in the journal. It is then of no great surprise that all the senior editors of the journal and all the chairs of the Cognitive Science Society have been computer scientists and cognitive psychologists. Interestingly, cognitive science as a discipline of its own with its own communal and production worlds is becoming increasingly more common in the journal, but even in most recent years, accounts for at most 20% of the papers.

From all three sources of data another important trend emerged: Multidisciplinary work has increased over the years in the journal such that between 30 and 50 percent of recent work is multidisciplinary. Thus, despite the dominance of psychology and computer science, the contributors to the journal are more than just a collection of psychologists and computer scientists producing traditional psychology and computer science research–they are reading work from multiple disciplines, and they are borrowing methodologies from each other.

The Annual Meeting of the Cognitive Science Society first convened in 1979. It has grown steadily from 34 presentations, to its 1992 high of 199 presentations, before beginning to gradually drop in recent years to as low as 114 presentations in 1996. Throughout its history, the conference has been one of the primary communal world for cognitive scientists to meet, network, and exchange the latest findings from their ongoing research.

Study 2 had two goals. The first was to replicate the overall findings of Study 1. The journal Cognitive Science accepts only a small percentage of submitted manuscripts, and it is possible that this high selectivity may have underestimated the extent of multidisciplinary research in the Cognitive Science Society. Perhaps more multidisciplinary work is occurring, but the journal’s stringent peer-review process favors publication of work using established methodologies by scientists with proven track records of publishing within their constituent disciplines. The conference is fairly selective in relation to other conferences, however its acceptance rate is still much higher than that of the journal (e.g., 55% of submitted papers were accepted at the 1995 conference).

The second goal of Study 2 was to extend our analyses beyond the limits of the data that can be obtained by analyzing published work. In Study 1 we assessed where first authors worked, what methods they utilized, and what literatures they cited. But the emergence of a multidisciplinary field is more than just the activities of individual scientists. Active research collaborations between scientists from different disciplines may be one of the primary vehicles for interdisciplinary exchange. Over the history of the conference, cognitive science collaborations (as measured by the proportion of multi-authored papers) have gradually increased (Schunn, Okada, & Crowley, 1995). At recent conferences, over 70% of papers had more than one author. In Study 2, we focus particularly upon the collaborative aspects of cognitive science research using a questionnaire that probed cognitive scientists about the work leading up to their conference presentations.

All authors of multi-authored papers and posters to be presented at the 17th annual conference in Pittsburgh, PA were contacted two months prior to the conference by e-mail. Permission to conduct the interview studies was obtained from the conference organizers in advance of contacting the participants. Ninety-six multi-author papers and posters had been accepted, 94 for which we had valid e-mail addresses, consisting of 222 total authors. Seventy-five (34%) questionnaires were completed and returned, representing at least one author from 56 (60%) of the papers. This response rate is fairly high for questionnaire methodology, yet, as with all questionnaire-based research, these data may suffer from self-selection and response biases.

The multiple-choice questionnaire we developed was based upon exploratory interviews with cognitive scientists and previous questionnaire studies (Okada et al., 1995; Schunn, Okada, & Crowley, 1995). Three sections of the questionnaire are relevant to the current analyses. The first section assessed the primary backgrounds and professional status (faculty, graduate student, etc.) of the participant and the participant's collaborators. The second section asked participants to estimate how successful they thought the project had been and how likely they were to continue working with their collaborators. Finally, the third section asked participants to estimate how often communication had occurred within the collaboration, the means of communication (face-to-face meetings, e-mail, etc.), the mesh or clash of the collaborators’ background knowledge and intellectual styles, and the benefits and frustrations of the collaboration.

The unit of analysis for most of the analyses presented below is a paper rather than a participant, therefore, we used only one response per paper. In particular, we used the data from the highest ranking author (i.e., first then second then third, etc..), on the assumption that they would have the most detailed knowledge of the project. In the small number of cases in which responses were received from multiple authors of the same paper, the responses were generally quite consistent. Moreover, changing the unit of analysis from paper to respondent did not change any of the results obtained in our analyses. Inter-rater reliability for all of the coding categories exceeded 90%.

We present two measures of disciplinary background: The departmental affiliations coded from the proceedings of the 1995 conference and the self-reported disciplinary affiliations from the questionnaires. Replicating the findings of the journal analysis, most presenters at the conference were from psychology or computer science departments (Table 9). The relative dominance of psychology was more pronounced at the conference, with psychologists accounting for almost half of all affiliations, while computer scientists accounted for 25%. Scientists from departments other than psychology and computer science did present at the conference, but in far fewer numbers. The departmental frequencies found at the 1995 conference are very close to those found from our previous study of the 1994 conference (Schunn, Okada & Crowley, 1995), suggesting that the 1995 meeting was not an atypical conference.

Table 9. Disciplinary affiliations of presenters at the 1995 conference

The finding that self-reported backgrounds were consistent with departmental affiliations is evidence that departmental affiliations serve as a good approximation of disciplinary background. Because these analyses include all authors of a paper, these findings also suggest that the journal analyses did not necessarily distort disciplinary participation by focusing on the first authors of articles.

As in Study 1, each poster or paper was coded as involving: simulation only; empirical study only; simulation+empirical study; simulation of previously published data; or neither simulation or empirical study. The methodology was coded for all accepted papers and posters from four conferences, two chosen from the Middle age range (1984 and 1987), and two chosen from the Recent age range (1992 and 1995). No conferences were coded from the Beginning age range because the conference was quite small then and non-peer reviewed.

Table 10. Within in each time range, the percentages of conference contributions categorized according to each methodology category.

Methodologically, there were some differences between the conference and the journal (see Table 10). In particular, at the conference, there were more simulations, fewer multidisciplinary papers, and fewer papers using neither simulation nor empirical studies. However, the overall high proportion of empirical studies and the recent growth in papers using multidisciplinary methodologies were quite consistent with the journal findings. Thus, we have discovered that the primary journal results do generalize to the conference, suggesting that the rigorous journal review process did not result in an atypical sample. Now we turn to issues that go beyond the ones addressed by the journal analyses.

Ninety-six of the 135 presentations at the 1996 conference were multi-authored. What percentage of these collaborations involved individuals from different backgrounds? Based upon the questionnaire responses, collaborations were coded as multidisciplinary using the following very strict criterion: the backgrounds of the other co-authors had to be different from any of the backgrounds listed for the respondent (sometimes respondents gave more than one background), and cognitive science as a background was considered to be the same as all other backgrounds. Using this strict criterion, 57% of the multi-authored papers were multidisciplinary. This number is surprisingly high given that a strong determinant of collaboration is physical proximity (Kraut, Egido, & Galegher, 1990) and the constituent disciplines within cognitive science are frequently located in different buildings.

One important dimension that one might expect to interact with multidisciplinarity is whether the collaborations are peer or apprenticeship collaborations–one might expect fewer multidisciplinary collaborations among apprenticeship collaborations. To investigate this possibility, collaborations were divided into peer and apprenticeship collaborations according to following criterion: they were coded as apprenticeship if one of the co-authors was of a different professional status level. Four status levels were used: faculty/clinician/senior researcher, postdoc/junior researcher, graduate student, undergraduate/research assistant. There were 16 peer collaborations and 58 apprenticeship collaborations. Surprisingly, there was a high level of multidisciplinary collaborations for both peer (50%) and apprenticeship (59%) collaborations.

To see whether multidisciplinary collaborations were equally frequent among each of the disciplines, we examined the frequency of collaborations among each of the disciplines (see Table 11), along with the frequency of monodisciplinary collaborations within each discipline. Interestingly, cognitive psychology was the only discipline in which researchers were more likely to collaborate with someone from their own discipline than someone from another discipline. Furthermore, almost all monodisciplinary collaborations were from cognitive psychology or computer science, suggesting that those two disciplines are the only ones considered legitimate instances of cognitive science without contributions from some other discipline. Along those lines, almost all multidisciplinary collaborations involved either a cognitive psychologist or a computer scientist. These results suggest that most work at the conference involves one of these two disciplines and perhaps, although not necessarily, another discipline.

Table 11. The frequency of collaborations among each of the disciplines at the 1995 conference.

There are multiple interpretations to the increase in multidisciplinary research in the journal and at the conference. It could be that the multidisciplinary work depends upon researchers from different disciplines coming together to do the work. Alternatively, it could be that researchers from the constituent disciplines are learning the theories and methodologies from other areas and are capable of doing multidisciplinary work themselves. To investigate this issue, we calculated the proportion of papers using multiple methodologies for multi- and mono-disciplinary collaborations, as well as single-authored papers (see Table 12). Surprisingly, the proportion of papers using multidisciplinary methodologies was remarkably similar across all collaboration types. Although Psychology/Artificial Intelligence collaborations result in slightly higher use of multidisciplinary methodologies, the difference is not statistically significant (c2(1)<1). These results suggest that individuals from each background were beginning to adopt the methodologies from the other discipline: cognitive psychologists were doing their own simulations, and computer scientists were paying attention to psychological data. Note, however, that the computer scientists, as yet, were not conducting their own psychological experiments; instead they were simulating results from psychological experiments.

Table 12. Percentage of the 1995 conference papers that had multidisciplinary methodology (i.e., both simulation and experiment or simulation of a particular experiment), for single authored papers, all multidisciplinary collaborations, psychology/AI collaborations specifically, all monodisciplinary collaborations, and psychology/psychology and AI/AI collaborations specifically.

Now that we have seen how often multidisciplinary work occurs in the journal and at the conference, we can use the questionnaire data to address the question, does multidisciplinarity affect the process of doing scientific research? We divided this question into two components. First, we assessed whether basic characteristics of the research process differed between multi- and monodisciplinary collaborations. If cognitive science research requires multidisciplinarity, then one would expect that the reported similarity of backgrounds would affect the reported structure of the collaboration, most notably the successfulness of the collaboration. Second, we examined whether different features were associated with successful research for multidisciplinary collaborations than for monodisciplinary collaborations. Such an analysis might provide useful suggestions for how one might foster a greater degree of multidisciplinary work in cognitive science.

Table 13. For the questionnaire given to presenters at the 1995 conference, percent of participants responding ‘Yes’ to ‘Was the project very successful’, ‘Did your co-authors frequently present alternative hypotheses’, ‘Did your co-authors have different research styles’, and ‘Did you feel that you had an equal status relationship with your co-authors?’ for multidisciplinary and monodisciplinary collaborations.

Note. *=p<.05

Table 13 presents four dimensions along which we compared multi- and monodisciplinary collaborations. Most surprisingly, multidisciplinary collaborations were not rated more successful than monodisciplinary collaborations. As a converging piece of evidence, multidisciplinary collaborations were no more likely to be a paper vs. a poster, which is determined by reviewer ratings, than were monodisciplinary collaborations (65% vs. 67% respectively). Yet, multidisciplinary collaboration did differ from monodisciplinary collaborations on several other important dimensions that one would expect to have an impact on the self-reported successfulness of the collaboration. First, participants were more likely to say that their co-authors frequently came up with alternative hypotheses in a multidisciplinary collaboration situation. Second, the multidisciplinary collaborators were more likely to say they had a different research style than their co-authors. Finally, multidisciplinary collaborators were more likely to say that they had an equal status relationship. This result held even when the relationship was inherently one of unequal status (see Table 14). Thus, there are benefits of multidisciplinary collaborations, even if they are not immediately apparent in the reported successfulness of the project.

Table 14. For the questionnaire given to presenters at the 1995 conference, percent of participants responding ‘Yes’ to ‘Did you feel that you had an equal status relationship with your co-authors?’ for multidisciplinary and monodisciplinary, and peer and apprenticeship collaborations.

If multidisciplinary collaborations were not associated with higher success ratings, then why did researchers engage in such collaborations? An analysis of the benefits and frustrations listed by collaborators provides some preliminary clues as to why multidisciplinary collaborations were not rated as more successful. Table 15 presents the benefits that were listed by at least one third of the participants. Three related benefits were rated as occurring more often in the multidisciplinary collaborations than in monodisciplinary collaborations: having different ideas, having a stimulating relationship, and challenging each other’s ideas.

Table 15. The percentage of participants in multi- and mono-disciplinary collaborations listing the following components as beneficial to their particular collaboration, broken down by those thought beneficial in both collaboration types and those though beneficial primarily in multidisciplinary collaborations (N=55).

However, multidisciplinary collaborations might also have more frustrations. Overall, participants listed fewer frustrations than benefits–thirty-six percent of the participants responded that they had no frustrations with their collaborations. Table 16 presents the frustrations listed by at least 5% of the participants. While reports of most types of frustrations were equally frequent in both collaboration types, it is very interesting to note that the one frustration that was reported more often in multidisciplinary collaborations than monodisciplinary collaborations was having ideas that were too different. Thus, having different ideas were reported both as beneficial and a source of frustrations in multidisciplinary collaborations, which may explain why multidisciplinary collaborations were not reported as more successful overall.

Table 16. The percentage of participants in multi- and mono-disciplinary collaborations listing the following components as frustrations in their particular collaboration, broken down by those thought beneficial in both collaboration types and those though beneficial primarily in multidisciplinary collaborations (N=50).

How did these factors relate to the reported success of the project and whether the collaborators would continue to work with one another? To answer these questions, we examined which of the following questionnaire responses predicted those two outcomes (success and continue working). To reduce the possibility of spurious relationships, we focused on six factors that previous research has suggested might be related to success (e.g., Dunbar, 1995; Okada & Simon, in press; Okada et al., 1996; Thagard, 1994): i) whether they had an equal status relationship; ii) how similar their initial ideas were; iii) how frequently their collaborators proposed alternative hypotheses; iv) how similar their research styles were; v) how similar their backgrounds were; and vi) how often they communicated during the collaboration. Stepwise regressions were conducted separately for multi- and mono-disciplinary collaborations.

For monodisciplinary collaborations, none of the factors predicted the reported successfulness of the project. For multidisciplinary collaborations, by contrast, reported similarity of initial ideas (r=.40, p<.01) predicted successfulness and communication frequency (partial r=.28, p<.1) marginally predicted reported successfulness. That is, collaborations with more similar initial ideas and more frequent communications lead to greater reported success. [Footnote: There was no evidence of any such relationships in the monodisciplinary collaborations (r=.02, p>.9 and r=-.09, p>.5 for similarity of initial ideas and communication frequency respectively).] These results suggest that multidisciplinary collaborations require some initial points of agreement, and then require frequent communications to come to some synthesis beyond what the individual collaborators would produce on their own.

A different pattern emerged for predicting who was likely to continue to work together. For multidisciplinary collaborations, none of the factors were statistically significant, although the role of having an equal status relationship marginally predicted (r=.26, p<.1) likelihood of continuing to work together. By contrast, for monodisciplinary collaborations, two very different factor emerged. For those collaborations, participants were more likely to say they would continue to work with their collaborators when they felt that their collaborators frequently proposed alternative hypotheses (r=.44, p<.02) and when their collaborator had a similar background (partial r=.40, p<.05).

While some of these results were somewhat weak and may be biased by the belief systems of the respondents, taken together, these findings suggest the following model of multidisciplinary collaborations: they require some similarity of perspective to begin with and benefit from having multiple perspectives if the collaborators meet frequently to resolve differences. If the collaborators cannot develop an equal status relationship (a hallmark of multidisciplinary collaborations), then they are not likely to continue to work together. By contrast, a different model emerges for monodisciplinary collaborations: participants prefer their collaborators to present alternative views, yet they also prefer some commonalties in perspectives. However, for such collaborations, features of the collaboration as a whole do not predict success. Instead, it must be features of the individuals themselves that are predictors of success.

We have presented four main findings based on a case study of the evolution of the Cognitive Science Society, a primary production and communal world of cognitive science research. First, we have found that participants in the Society are primarily cognitive psychologists and computer scientists, and that scientists from other disciplines typically thought of as full partners in cognitive science (e.g., philosophy, linguistics, and neuroscience) are but a small minority. Second, despite this focus on psychology and artificial intelligence, a large proportion of work being done in the Society is multidisciplinary, as measured by collaboration makeup, methodology use, and citation of previous theories and results. That is, the Society is undergoing a growth of a new communal world, and there were changes in the activities of the corresponding production worlds. However, there is still a significant proportion of monodisciplinary work being done in the Society, most of which is done by cognitive psychologists. Third, researchers from each of the disciplines are adopting ideas and methodologies from other disciplines. In particular, we found that psychologists are using simulations more frequently, and that computer scientists are paying more attention to data. Fourth, we found that multidisciplinary collaborations are different in their structure than monodisciplinary collaborations. In particular, they involve combining different research styles, more frequently proposing alternative ideas, and having a more equal status working relationship. Moreover, we found that different factors predict successfulness in multidisciplinary collaborations than in monodisciplinary collaborations.

How general are these findings? They were fairly consistent over the life of the Society, over numerous measures, and between the journal and conference. However, the extent to which our findings generalize beyond the Cognitive Science Society to the discipline of cognitive science as a whole depends on the extent to which the activities of the Cognitive Science Society reflect a representative cross-section of the discipline. This, in turn, depends in large part on how one chooses to define "cognitive science". As we stated in the introduction, although one might argue that the activities of the Cognitive Science Society can serve as a useful first-approximation of the discipline, it is important to recognize that analysis of other journals and conferences may reveal a different picture about the current state of the field of cognitive science.

Another factor that could limit the generality of our findings is an American bias. While neither the journal nor the conference are restricted to Americans, a majority of research published in those two sources is conducted by researchers at American institutions. Furthermore, the conference almost always occurs in the US (on one occasion it occurred in Canada). However, analyses of the German and Japanese Cognitive Science Societies have found a similar dearth of philosophy, linguistics, and neuroscience (Jameson, 1997; Okada, Crowley, Schunn, & Miwa, in prep). Moreover, a recent survey of cognitive science research across the world has revealed that at least one of our findings is quite general: Psychology and computer science tend to be central components of cognitive science (Denis, 1996).

Why have linguists, philosophers, and neuroscientists not taken a greater role in the Society’s activities? It is likely that several factors contribute. For discussion purposes, we will assume that the causes are not just simple factors like an editorial/reviewer bias, large asymmetries in the size of each discipline, or the existence of adequate within-discipline production and social worlds. Moreover, there was some evidence against these simple factors: 1) the explicit editorial policies of both the journal and the conference clearly invite submissions from the other constituent disciplines of cognitive science; 2) simple on-line searches of the WorldCat, Journal, Proceedings, and Conference databases available in FirstSearch of work produced from 1990—1995 revealed that there were approximately as many psychology conference papers as philosophy papers, linguistics produced over 25% more conference papers than did psychology, and while linguistics produced fewer journal articles and books than psychology, philosophy produced as many; and 3) the existence of specialized forums does not remove the need for an overarching journal or conference which spans the discipline, which are typically quite prestigious and reach a wider audience. Since the causal factors are likely to vary by discipline, we will consider two particularly interesting cases: linguistics, an older, stable discipline; and cognitive neuroscience, a new, rapidly-growing discipline.

In our analyses, we found that linguistics has historically been the third most cited of the constituent disciplines behind psychology and computer science. Yet, for both the journal and the conference analyses, we found that participation by linguists has never exceeded 4% of all articles or conference papers. To investigate possible causes, we conducted several citation analyses. First, we counted the number of citations to the journal Cognitive Science in the journal Linguistic Inquiry, the linguistics journal with the highest impact factor, for all main articles (ignoring squibbs, remarks, and reply articles) for the years 1980, 1987, and 1994. We found no references to Cognitive Science articles in any of the years. For comparison, the corresponding numbers for the mainstream psychology journal, Psychological Review were: 5% in 1980, 19% in 1987, and 14% in 1994. Thus, mainstream linguistics articles appear not to cite Cognitive Science papers. Second, we examine how often Cognitive Science articles by linguistics are cited by anyone in contrast to Cognitive Science articles by psychologists and computer scientists. The citation analysis was done for papers from 1980, 1986, and 1991, years selected randomly from the three age ranges used for the other journal analyses. Citations were counted 2, 4, and 8 years after publication. Psychology papers (N=14) received 3.4 citations per year; computer science papers (N=10) received 2.1 citations per year; and linguistics papers (N=2) received only 1.3 citations per year. Thus, it appears that linguists have less to gain from publishing in Cognitive Science.

The case of the rapidly-growing field of neuroscience is likely to be different. We did find a small growth of cognitive neuroscience within the Cognitive Science Society’s conference and journal. Moreover, there are reasons to suspect that our measures may have underestimated the presence of cognitive neuroscience within the Society: many cognitive neuroscientists may have been trained in traditional departments (e.g., psychology, medicine), use methodologies common to psychology and computer science (i.e., psychological experimentation and computational modeling), and may still reside in traditional departments. Assuming, however, that there is little neuroscience in the Cognitive Science Society, there are several possible causes. First, since cognitive neuroscience is a very young and rapidly growing field, it may be that researchers within cognitive neuroscience feel sufficiently challenged by growth within the discipline. Second, neuroscience is already a very multidisciplinary field, and there are many disciplines that compete with cognitive science for the attention of neuroscientists.

In addition to data on the current status of the Cognitive Science Society, we have also presented data on changes in its makeup over time. From this historical data, one might extrapolate to make predictions for the future of the Society. One prediction seems quite clear: we found no evidence that linguistics or philosophy will become more prevalent in the Society in the near future. If anything, the reverse was true. However, there was a small trend for neuroscience becoming an increasingly greater part of the Society. The most important growth in the Society was the emergence of a full, separate discipline called cognitive science. This was seen in a growth of researchers listing departments or institutes of cognitive science as their primary affiliation, researchers listing cognitive science as their primary background, citations of work in cognitive science journals, and single-authored papers using methodologies from multiple constituent disciplines. Moreover, analyses of methodology and departmental affiliations in the conference and journal for 1996 and 1997 showed a general continuation of these findings. Thus, while the Cognitive Science Society is still primarily a conjunction of researchers from different disciplines, it is on its way to becoming a separate discipline with distinct communal and production worlds.

Our analyses uncovered a fairly high level of multidisciplinary collaboration within the Cognitive Science Society. What are the consequences of this multidisciplinary research? We have found that scientists in multidisciplinary collaborations valued the difference of ideas and stimulating intellectual relationship with their collaborators. However, several scientists noted that such differences were also a source of frustration. We also found that the more frequent communication on the project became, the more successful the scientists felt the project was. These data converge to tell us that an important aspect of multidisciplinary collaboration is a balance of similarities and differences of ideas among collaborators. Differences of ideas could cause problems such as communication difficulty, slower pace of progress, and frustrations among collaborators. In order to make a multidisciplinary collaboration productive, such problems have to be overcome with frequent and effective communication measures among collaborators.

The conclusions that we have made regarding what leads to a successful collaboration should, of course, come with a cautionary note. We measured the success of the projects using the scientists’ self-evaluations. Such a measure might not reflect the success of their projects objectively. Some discoveries have to wait for decades to get appropriate evaluation until communities of science and society understand their importance. Especially novel research which might result from multidisciplinary collaborations may require a long time to develop and become accepted in the general research community.

In this paper we have explored a topic that is frequently and fervently discussed by cognitive scientists with anecdotal data at best: Is cognitive science truly multidisciplinary? We hope that our data may be used to support productive dialog regarding the development of the Cognitive Science Society and the discipline of cognitive science itself, both in the production worlds (e.g., editorial and reviewing practices) and communal worlds (e.g., professional society, conference, and departmental levels). Of potential relevance to these practical issues, we have conducted analyses of what factors lead to successful multidisciplinary research. We found that frequent meetings are important for success, suggesting that the occurrence of multidisciplinary research may be increased by providing further support for more frequent meetings. For example, one might call for the establishment of greater support infrastructure such as interdisciplinary institutes, as physical proximity is known to be a strong factor in the occurrence of collaborative research (Kraut, Egido, & Galegher, 1990). We hope that a better understanding of the state of the Cognitive Science Society can be used to make better informed policy decisions in these days of transition and continued growth in cognitive science.

Number of visits to this site since 11/1/00: