A Randomized, Controlled Trial of Computer-Assisted Cognitive Remediation for Schizophrenia

Cognitive impairment in schizophrenia accounts for significant variation in community outcomes, such as work performance (1–3). The class of behavioral treatments known as cognitive remediation specifically targets memory, attention, reasoning, and similar capacities, with the ultimate aim of enhancing everyday functioning. The authors of a 2007 meta-analysis concluded that remediation yields moderate improvements in cognitive performance, symptoms, and psychosocial functioning (4). Fisher et al. (5, 6) have recently reported even larger gains in the first trials of a novel remediation program. Moreover, participants generally find remediation programs enjoyable and cognitively stimulating (7–9). Still, the body of research faces important challenges. First, the aspects of cognition that are targeted and the conceptual models that underlie different remediation programs vary substantially, making it difficult to identify essential intervention elements. Some interventions have used broadly targeted cognitive training (10, 11), while others have focused on perceptual processes (5, 6, 12), attention (13, 14), executive functioning (8), or social cognition (15, 16). Certain programs emphasize repetitive drill and practice to make cognitive processing more automatic (so-called "bottom up" approaches; 5, 17, 18), while others employ problem-solving training and other metacognitive strategies to guide the use of cognitive resources ("top down" approaches; 8, 19). Programs have also used different therapeutic modalities, e.g., paper and pencil (8, 20) or computer (10, 18), in different settings, e.g., inpatient treatment (7, 21) or vocational rehabilitation (11, 22).

Second, relatively few studies reviewed in the meta-analysis used rigorous clinical trial methods. Weaknesses included small study groups (14, 23), treatment-as-usual control groups (10, 11, 17), unblinded assessments (20, 24), and unspecified plans for statistical analysis (10, 14, 25). Studies that avoided most of these weaknesses were more apt to yield weak or negative results (9, 18, 26, 27). Further, while the primary rationale offered for cognitive remediation has been to generate durable improvements in psychosocial functioning (4), most reports have provided only results on neuropsychological measures and only immediately after treatment. This is problematic because it is difficult to entirely avoid overlap between the remediation exercises and the neuropsychological tasks used to measure their effect. Indeed, the key neuropsychological measures in some studies were quite similar to the remediation training exercises (17, 18), and apparent cognitive enhancement could reflect task practice. Only a minority of studies have examined the durability of treatment effects over time or the generalization of these effects to functional outcomes (4, 11, 28, 29). Moreover, in these studies, the effect of cognitive enhancement per se may be confounded with the effects of other supportive elements, e.g., process groups (28) or on-the-job cognitive supports (11). In short, the evidence supporting cognitive remediation for schizophrenia remains inconsistent and incomplete.

Our goal in the current study was to develop a broadly targeted computer-assisted cognitive remediation program and submit it to a rigorous clinical trial in a large group of schizophrenia patients. We hypothesized that—relative to a condition designed to control for nonspecific treatment effects—cognitive remediation would improve performance on the remediation exercises, neuropsychological composites (representing episodic memory, working memory, attention, executive functioning, and processing speed), and proxy measures of everyday functioning and that these improvements would be sustained over a 3-month follow-up period.

Method

All procedures were approved by the institutional review board of the University of Maryland, Baltimore, and all participants provided written informed consent. There were two parallel studies, one funded by the National Institute of Mental Health (NIMH study) and the other through the Department of Veterans Affairs Rehabilitation Research and Development Service (VA study). The main differences between the studies related to study aims beyond this clinical trial. For the present analyses, data from these studies were combined.

Development of Computer-Assisted Cognitive Remediation Program

At the time this research program began (about 2002), there was no leading program of cognitive remediation for schizophrenia. Wykes and co-workers (8) taught organized problem-solving strategies in intensive, one-on-one sessions but used plain paper-and-pencil exercises. Bell et al. (17) took advantage of computer technology but employed repetitive and unadorned exercises borrowed from treatment for brain injury. Medalia and Revheim (30) addressed motivational issues with engaging, educational software and supportive coaching, but many details of this program were flexible and difficult to implement in a standardized clinical trial. Computer-assisted cognitive remediation was designed to integrate the strengths of these programs. Its aim was to enhance the selection, execution, and monitoring of cognitive operations to allow more efficient performance of cognitive and everyday tasks. Its key elements included 1) training in organized problem solving, 2) guided practice of engaging computer-based training exercises, and 3) supportive, one-on-one training sessions. Other adaptations geared to the special learning and motivational needs of individuals with schizophrenia were drawn from Bellack's approach to skills training (31). These included an emphasis on concrete strategies for problem solving (e.g., verbalization of details to promote encoding), visual prompts summarizing recommended strategies, practice in a variety of problem contexts, an explicit focus on generalization beyond the treatment setting, and frequent, immediate social reinforcement. The program was refined and its feasibility and tolerability were tested in schizophrenia patients in a series of small, preliminary trials conducted over 9 months. Although the program was evolving during this period, we carefully monitored participant improvement on the selected exercises during the small trials. The participants' consistent improvement provided encouraging evidence of cognitive change in response to remediation training. In this period, we also developed training exercise metrics and we standardized and manualized the remediation and control interventions in preparation for a clinical trial.

Clinical Trial Design

Individuals who agreed to participate in the clinical trial and met the study criteria were assigned to experimental groups by using an adaptive urn randomization procedure (32) that adjusted for sex, race, psychiatric diagnosis, site, and lifetime history of substance dependence. The participants were not informed that they were assigned to "treatment" or "control" conditions; rather, individuals in both groups were told that the aim of the study was to determine whether participation in a "computer activities program" improved thinking skills. Both conditions consisted of 36 sessions lasting 1 hour each. The participants received financial compensation for their time in assessments and training sessions.

Participants

Participation was open to individuals diagnosed with schizophrenia or schizoaffective disorder. Diagnoses used the Structured Clinical Interview for DSM-IV (SCID), information from the participants' mental health care providers, and medical records. Eligible individuals were 21 to 60 years old, clinically stable on regimens of second-generation or low-dose first-generation antipsychotics, without a history of significant brain trauma, neurological disorder, or substance dependence within the previous 3 months, and without physical limitations precluding effective use of computer-based exercises. Recruitment for the trial began in 2004 and concluded in 2006, with treatment sessions and follow-up assessments continuing into 2007. Individuals were recruited from the Veterans Integrated Services Network 5 (the VA Medical Centers at Baltimore and Perry Point, Md., and Washington, D.C.) (70.1%), community psychiatry clinics of the University of Maryland Medical System, Baltimore (23.4%), and Mosaic, Inc., Catonsville, Md. (6.5%).

Remediation Training

Training was organized in three phases (Figure 1). During the first phase, trainers introduced a simple and general problem-solving approach, which was reinforced consistently through all phases of training. The participants were prompted to identify the challenges in each exercise, articulate a plan to address them, implement the plan, monitor its effectiveness, and adjust their strategy as needed. Computer-assisted cognitive remediation shaped these problem-solving techniques through extensive practice. Master's-level trainers guided participants at an individualized pace through a varied curriculum of engaging, educational computer exercises, selected to gradually enhance processing speed, attention, working memory, episodic memory, and executive functioning, i.e., reasoning and problem solving (30) (Figure 1). Time in individual sessions was split; practice of cognitive exercises (roughly two-thirds of each session) alternated with trainer prompts, queries and feedback, and strategy review. The training sessions were videotaped and reviewed in a weekly supervision meeting to promote consistency across different participants and trainers and allow adjustments to individual participant needs. We sought to complete three remediation sessions per week, with a maximum of 15 weeks allowed for completion of the 36-session training program.

Active Control Condition

This condition was designed to control for nonspecific treatment effects. It specified an equal number of one-on-one computer sessions with the same trainers who conducted the remediation sessions. It offered supportive trainer interactions and matched experience with computers and varied computer activities. Control activities were selected for game-like properties and low cognitive demand. Participants in this condition did not receive problem-solving training or guided practice on the exercises used in the remediation condition. The control sessions were also videotaped and reviewed in supervision meetings.

Assessment

The participants were assessed before treatment, immediately after treatment, and 3 months after treatment. The assessments included symptom measures, a comprehensive battery of cognitive measures, a measure of self-perceived cognitive functioning, and proxy measures of everyday functioning (Table 1).

Enlarge table

The cognitive assessors (who also administered the functional measures) were research assistants, trained and supervised by a neuropsychologist (D.D.). The assessors were blind to the participants' assigned condition and had no other role in the project that would undermine blinding. Two aspects of the assessment protocol were not fully blinded. First, while initial symptom ratings were completed before randomization, the posttreatment and follow-up ratings were completed by investigators (W.T., S.M.) or by postdoctoral fellows under their supervision. We did not hypothesize any treatment-related changes in symptoms, and these were considered secondary outcome measures. Second, during the course of remediation training, participants in the treatment group completed metrics developed directly from the remediation exercises. A given metric was administered by the remediation trainer before work on a given exercise and then several sessions later, after completion of work on that exercise, to document any improvement in performance. The participants in the control condition also completed the remediation exercise metrics with their trainers, on the same schedule as that followed by the participants receiving remediation training but without intervening training and practice.

Planned Analyses

Analyses followed a modified intent-to-treat approach, including all participants who "engaged" in treatment (defined as attending at least three sessions) and completed postparticipation assessments. The first set of primary analyses tested the treatment effect on the 10 metrics that were derived from the remediation exercises. The second set addressed whether treatment generalized to produce an effect on five composite variables representing processing speed, working memory, controlled attention, episodic memory, and executive functioning (grouped as in Table 1). The third set of primary analyses tested treatment effects on the University of California at San Diego (UCSD) Performance-Based Skills Assessment (42) and the Maryland Assessment of Social Competence (43). All analyses used linear regression to estimate the mean treatment effect at posttreatment after controlling for baseline performance and study site. To strike a balance between controlling for type I errors and avoiding type II errors, the t values associated with the regression coefficients for each variable were tested for statistical significance by using an alpha level of p<0.01. Effect size for a given variable was calculated as the difference after treatment between the treatment and control conditions divided by the pooled standard deviation. The second and third tiers of analysis were repeated for individuals who completed the 3-month follow-up assessment. The exercise metrics were not administered at the follow-up. Secondary analyses using the same approach examined treatment effects on the Brief Psychiatric Rating Scale (BPRS) (45), Scale for the Assessment of Negative Symptoms (SANS) (46), and Schizophrenia Cognition Rating Scale (44) ratings.

Results

Group Characteristics

Figure 2 is a diagram of study participation for the 107 individuals who signed consent forms. Of the 35 participants who engaged in the remediation condition, 30 (85.7%) completed at least 30 remediation sessions (mean=32.2). Thirty-four participants, including four who did not finish treatment, completed posttreatment assessments. Of the 28 who engaged in the control condition, 20 (71.4%) completed at least 30 sessions (mean=34.3). Twenty-seven participants, including seven who did not complete the control condition, provided posttreatment assessments. There were differences in baseline characteristics depending on the recruitment sites (38 subjects from the VA sites and 25 from non-VA sites). The mean age of the participants recruited at the VA (mean=50.3 years, SD=6.4) was significantly higher than that for those recruited from non-VA sites (mean=43.2 years, SD=7.5) (t=3.99, df=60, p<0.001); the subjects from the VA also had more education (mean=13.1 years, SD=1.4, versus mean=11.5 years, SD=1.4) (t=4.39, df=61, p<0.001) and were more likely to be male (84.2% versus 48.0%) (χ²=9.39, df=1, p<0.01). The VA and non-VA participants did not differ in regard to race, diagnosis, or substance use history. The participant groups also did not differ in baseline performance on the objective cognitive measures or proxy measures of everyday functioning, on self-reported cognitive status, or on symptom ratings. Data from all of the 61 participants who completed posttreatment assessments (see Figure 2) were included in the main intent-to-treat analyses. The subjects in the intent-to-treat analyses were divided as follows between the VA recruitment sites (N=37) and non-VA sites (N=24). Within the VA and non-VA samples, the remediation and control groups showed the same patterns of response to treatment. Nevertheless, study site was included as a covariate in all relevant analyses.

Table 2 shows the baseline characteristics of the treatment and control groups. We found no significant differences between the groups on demographic or clinical variables or on performance on baseline cognitive or community functioning measures. Most participants were on stable regimens of second-generation antipsychotics. One control participant took no antipsychotic medication.

Enlarge table

Treatment Effects

Table 3 and Figure 3 summarize the posttreatment effects on the remediation exercise metrics and the cognitive, functional, and symptom measures. Regression modeling indicated that the results on all but two of the 10 exercise metrics improved significantly more in the remediation condition than in the control condition (p≤0.01). The mean effect size, favoring the remediation condition, was 0.53 across all 10 metrics. There were no significant effects favoring the treatment group on any cognitive, functional, or symptom measure, even at a "trend" level (i.e., p≤0.1) (the mean effect size for the cognitive composites was 0.07, and for the functional proxies, it was –0.12). Among the cognitive measures, the executive performance composite showed nonsignificant movement in the hypothesized direction (effect size=0.28), but this likely reflected regression to the mean in both groups (see baseline differences in Table 2). The difference closest to significance was for the UCSD Performance-Based Skills Assessment (effect size=-0.33, p=0.11); however, this result favored the control condition (in all other cases, p>0.59).

Enlarge table

Because the average age of individuals in this study was higher than in other remediation studies, for the treatment group we explored the association of age with treatment-related changes on the exercise metrics and the cognitive, functional, and symptom measures. There were no significant associations for the exercise metrics or symptom measures. Among the primary outcome measures, improvement on the Maryland Assessment of Social Competence was positively correlated with age (r=0.46, N=27, p<0.05). No other cognitive or functional changes were significantly associated with age (r=–0.10 to r=0.16, p>0.37 in all cases).

Table 4 presents the results of the 3-month follow-up assessments measured against baseline performance. Four of the five neuropsychological composites showed small changes in the hypothesized direction, and scores on the UCSD Performance-Based Skills Assessment, Schizophrenia Cognition Rating Scale, and SANS also showed numerical improvement. However, none of these changes approached statistical significance.

Enlarge table

Discussion

In the current study we set out to meet a high standard of rigor in the development and testing of a computer-assisted cognitive remediation program for people with schizophrenia. Drawing on work by leaders in the fields of cognitive remediation and behavioral treatment for schizophrenia, computer-assisted remediation provided supportive, graduated training in selecting, executing, and monitoring cognitive operations to promote more efficient performance of cognitive and everyday tasks. Demanding clinical trials methods were used to test the program. We recruited and randomly assigned a substantial study group. The participants received up to 36 sessions of training. The computer-assisted remediation was contrasted with an active experimental condition that controlled for nonspecific elements of the remediation training, including supportive therapist interactions and exposure to interesting computer activities. Both conditions were manualized, and weekly supervision using session videotapes assured fidelity to the training model and consistency across sites and trainers. Outcomes were assessed at three levels: proximally, on the remediation training exercises; intermediately, on neuropsychological measures not involved in the training; and more distally, on proxy measures of everyday functioning. The neuropsychological and functional outcomes were designated as the primary outcomes and assessed by raters who were blind to treatment condition. A 3-month follow-up assessment was included to test the durability of training effects. Data analysis followed a prespecified plan, used an intent-to-treat model, and included steps to control for multiple comparisons.

Remediation training was effective in improving performance across the full range of training exercises (the effect size was approximately 0.5) (4, 47). However, the effect of training did not generalize to outcomes beyond the training exercise metrics. There was no evidence of a benefit to the remediation group in performance on neuropsychological composites representing episodic memory, working memory, attention, executive functioning, or processing speed or on performance-based measures of instrumental activities of daily living (the UCSD Performance-Based Skills Assessment) and interpersonal effectiveness (the Maryland Assessment of Social Competence). Analyses of data from the 3-month follow-up yielded similar results. Secondary analyses suggested that there was no training effect on self-described cognitive performance (the Schizophrenia Cognition Rating Scale), general symptoms (the BPRS), or negative symptoms (the SANS), either immediately after treatment or at the 3-month follow-up.

Our main hypotheses were not supported. Several explanations are possible. Statistical power is a common weakness of remediation studies. Our study group was as large as or larger than those in 22 of the 26 studies included in the 2007 meta-analysis (4) and, even with a stringent alpha level, had power to detect medium to large effect sizes of the sort reported in some studies (5, 19, 20, 48). However, the study was not powered to detect the small to medium treatment effects reported in other studies (11, 15, 17, 22). Our rationale was simply that medium to large neuropsychological effects would be more likely than smaller effects to yield meaningful functional benefits. Two considerations help to mitigate power concerns. First, the results of this study did not pivot on the stringent alpha level and would have been essentially unchanged at a more relaxed, and hence more powerful, alpha level (e.g., p≤0.1). Second, the original power calculations were based on single, independent statistical comparisons. However, the final analyses showed near-perfect multivariate consistency across all of the significant results (i.e., outcomes on the training exercise metrics) and nonsignificant results (i.e., the primary outcome measures not involved in training). This clear contrast in the full set of results adds weight to the individual significant and nonsignificant results considered in isolation.

The participants in the study were primarily chronically ill, urban, and middle-aged. Future work may show that appropriate remediation is more effective for individuals earlier in the course of illness, although the first studies addressing this hypothesis failed to demonstrate a strong response to remediation in younger individuals (26, 49). In the current data, furthermore, age was unrelated to treatment-related improvement on the exercise metrics or on cognitive or symptom variables. The only significant age correlation, for the Maryland Assessment of Social Competence, showed enhanced treatment response for older participants. Poor motivation is another challenge in these studies. Completion rates in the current study were likely aided by financial compensation for participation. However, the supportive, one-on-one training and engaging multimedia exercises were incorporated in the remediation program with motivational challenges in mind and appear to have been effective. Of the participants who completed the first three remediation sessions, 85.7% completed the full program (compared with 71.4% in the control condition; see Figure 2), and motivation was sufficient to garner significant gains in the remediation group across the training tasks.

Experienced researchers hold different views as to whether cognitive remediation programs—targeting memory, attention, reasoning, and similar capacities with the ultimate aim of enhancing everyday functioning—can improve the lives of people with schizophrenia. Proponents cite the recent meta-analysis by McGurk et al. (4). That review of 26 remediation studies suggested that cognitive remediation yields average effects of 0.41 on cognitive measures not involved in training, of 0.35 on measures of psychosocial functioning, and of 0.28 on symptom measures. However, the contrast of these conclusions with those of another recent review illustrates the difficulties faced in trying to synthesize the literature. Great Britain's National Institute for Health and Clinical Excellence (NICE) reviewed the cognitive remediation literature in the course of updating its 2002 schizophrenia treatment guideline (50). NICE agreed that cognition is an appropriate target for treatment and noted a number of supportive findings. However, applying criteria more restrictive than used in the 2007 meta-analysis (e.g., no mixed cognitive/vocational remediation trials, at least 10 subjects per arm, follow-up data) to select 20 studies for quantitative review, NICE found "little consistent advantage of cognitive remediation over standard care and attentional controls" and "no consistent evidence that cognitive remediation alone is effective in improving the critical outcomes, including relapse rates, rehospitalization, mental state, and quality of life" (50, pp. 223–224).

While other recent studies are more encouraging (5), the current study offers incremental support for the cautious conclusions reached by NICE. The central aim of computer-assisted cognitive remediation was to enhance strategic, "top-down" control of the selection, integration, and monitoring of cognitive operations, in turn boosting efficiency in performance of cognitive and everyday tasks. A metacognitive approach has been suggested by commentators (51), and in the McGurk et al. meta-analysis, interventions that included strategy training improved functional outcomes relative to straight "drill and practice" interventions (4). However, the approach did not yield generalized benefits in the current trial. More generally, the remediation literature is mixed and open to diverging interpretations. As reflected in the number of fundamentally different approaches to cognitive remediation, the field has yet to settle on a conceptual model (or models) of how remediation might produce desired changes or to determine which changes would yield the greatest functional benefit. Linked to this issue is whether, as has been suggested (4), remediation needs to be embedded in a broader psychosocial rehabilitation program (e.g., supported employment) or combined with a cognition-enhancing pharmacological intervention in order to be optimally effective. A focus on such threshold issues may help to move the research forward.