Types of studies

To be included trials had to be double‐blind and randomised.

Types of participants

Patients had to suffer from major depression (meeting DSM‐IV or ICD‐10 criteria). Trials in children (< 16 years) were not eligible. In previous versions of this review (Linde 1998; Linde 2005a) trials not restricted to patients with major depression had been included.

Types of interventions

Experimental intervention
Preparations of hypericum (St. John's wort). Trials investigating combinations of hypericum with other herbs were excluded.

Control intervention
Placebo or synthetic antidepressants (tricyclic and related antidepressants, selective serotonine reuptake inhibitors, serotonine‐noradrenaline reuptake inhibitors). Trials using clearly inadequate synthetic antidepressants (e.g., benzodiatepines) or a dosage clearly below the lower thresholds recommended in current guidelines (Härter 2003, ICSI 2007) were excluded.

Experimental and control treatments had to be given for at least four weeks.

Main comparisons
The following comparisons were performed:
1. hypericum extracts vs. placebo
2. hypericum extracts vs. standard antidepressants

Types of outcome measures

Primary outcome
To be included, trials had to measure clinical outcomes such as depression scales or symptoms. Trials that measured physiological parameters only were excluded. The primary outcomes of interest were

1. Effectiveness: treatment response

2. Safety: the proportion of patients who dropped out due to adverse effects

Secondary outcomes
1. Effectiveness: remission, depression scales such as the Hamilton Depression Scale (HAMD), the Clinical Global Impression Index (CGI), the Montgomery‐Asberg Depression Rating Scale (MADRS), patient‐rated depression scales

2. Safety: total proportion of drop‐outs, proportion of patients reporting adverse effects

Selection of studies

Two reviewers independently decided on eligibility for the revised inclusion criteria. Disagreements were resolved through discussion. Due to reading errors, disagreements occurred for two trials in which not all patients had major depression and which were excluded after discussion (Vorbach 1994, Winkel 2000). In two trials both reviewers had problems with assessing eligibility: For one small, older trial (Lehrl 1993) the publication did not state that inclusion was limited to patients with major depression, but we had a statement of the sponsor obtained for our 1998 update that all patients met the criteria. As this information could not be verified for this update, we decided to exclude the trial. A Chinese trial (Gu 2001) referred to a Chinese classification. As this classification is not completely comparable to ICD‐10 and DSM‐IV, we decided to also exclude this trial.

Data extraction and management

Primary study characteristics and results were extracted by at least two independent reviewers using a pretested form. In particular, we extracted diagnoses and main inclusion criteria, age, gender, duration of episodes, baseline depression scores, country of origin, number and type of study centers, numbers of patients who were randomised and analysed and who completed protocols, the number and reasons for drop‐outs and withdrawals, numbers of patients reporting adverse effects, and the number and type of adverse effects that were reported.

We assessed numbers of patients who were classified as responders based on score improvements on the Hamilton Rating Scale for Depression (HAMD), the Clinical Global Impression Index (CGI; subscale global improvement rating as at least "much improved"), or any other clinical response measurement. Missing or additional information was sought from authors/sponsors.

Most trials measured clinical outcomes with the Hamilton Depression Scale (HAMD) and the Clinical Global Impression Index (CGI). The HAMD is an observer‐rated scale that focuses mainly on somatic symptoms of depression (Hamilton 1960). The original version includes 21 items, but a version with 17 items is more commonly used in clinical trials. Most studies using the HAMD report the number of 'treatment responders' (patients achieving a score less than 10 and/or less than 50% of the baseline score). When available, we extracted means and standard deviations before, during and after treatment as well as the number of 'responders'. The CGI (CGI 1970) is an observer rated instrument with three items (severity of illness, global improvement, and an efficacy index). We extracted the number of patients rated as 'much improved' or 'very much improved' for global improvement. As recently the Montgomery‐Asberg Depression Rating Scale (MADRS (Montgomery 1979)) and the remission criterion for the HAMD (usually a score of less than 8 at the end of treatment) have been gaining importance as outcome criteria, we also checked all trials for the reporting of these measures. As the DS (Depression Scale von Zerssen (von Zerssen 1996)) was the most often used patient‐rated instrument in the included trials, we extracted post‐treatment data for this scale, if available. For additional post‐hoc analyses, one reviewer (KL) also extracted data for other self‐rating instruments.

Assessment of risk of bias in included studies

The main part of the update process of this review was completed before the new risk of bias tool of the Cochrane Collaboration (Higgins 2008) was available. The methodological quality of each trial was assessed by at least two independent reviewers using scales developed by A. Jadad et al. (Jadad 1996) and by one of the reviewers (KL). The results of the quality scoring are displayed in the table of included studies.

The Jadad scale has three items adding up to a maximum score of five points. 0, 1 or 2 points can be given for randomisation (explicit statement that allocation was randomised and description of an adequate generation of the random sequence), 0, 1 or 2 points for double‐blinding (explicit statement that patients and evaluators were blinded and that treatments were indistinguishable), 0 or 1 point for description of drop‐outs and withdrawals (numbers and reasons for all compared groups separately). The display in the table of included studies is as follows (examples): 2‐2‐1 (full score in every item), 1‐0‐0 (only statement on randomisation).
The second quality scale, the "Internal Validity Scale" (IV), which has been used in other reviews on complementary medicine (Linde 1996b, Linde 1997) has six items with possible scores of 0, 0.5 or 1 point for each. Items 1 through 6 refer to statement of random allocation, adequacy of randomization concealment, baseline comparability, blinding of patients, blinding of evaluators, and likelihood of selection bias after allocation, respectively. Results are displayed by item in the table of included studies (e.g., 1‐1‐1‐0.5‐1‐1 represents a full score, with the exception of blinding of patients which was stated but treatment and placebo might have been distinguishable).

The assessments in the Jadad and IV scores are solely based on the information provided in the publication (as additional information could not be gathered for all studies). In the table 'Characteristics of included studies', however, additional information provided from authors or sponsors was included. This table also contains information on allocation concealment and attrition.

Measures of treatment effect

Our primary outcome measure, to assess the effectiveness of St John's wort versus placebo and versus other antidepressants, was the proportion of responders (according to the Hamilton Depression Scale (first preference) or other responder measures (second preference)) at the end of treatment, or in case of treatment phases longer than 6 weeks, at the time point defined for primary outcome measurement by the study investigators.

Secondary outcome measures were: proportion of responders according to HAMD, proportion of responders according to CGI, mean HAMD after treatment (or, if this was not available, difference after treatment ‐ baseline), at 2, 4, 6 to 8 weeks, and mean DS score after treatment (or, if this was not available, difference after treatment ‐ baseline).

The main outcome measure for the safety analysis was the proportion of patients who dropped out due to adverse effects. Secondary measures were the total proportion of drop‐outs and the proportion of patients reporting adverse effects.

Dichotomous outcomes
We used responder rate ratios (= relative risks = proportion of responders in the treatment group/proportion of responders in the control group) and their 95% confidence intervals for the analysis of treatment response. Responder rate ratios greater than 1 indicate better response in the hypericum group.

Due to highly variable frequency of side or adverse effects reported, odds ratios instead of rate ratios were calculated in the safety analyses. Odds ratios less than 1 indicate that fewer events occurred in the hypericum group.

Continuous outcomes
For HAMD and DS scores we calculated mean differences (also termed weighted mean differences). Negative mean differences indicate better response in the hypericum group.

Unit of analysis issues
Two trials with more than one hypericum group were included in the analyses as follows: Laakmann 1998 included an extract available on the market and an additional experimental extract with low hyperforin content which was never on the market and only used for control reasons. We did not include the data from the group receiving the experimental low‐hyperforin extract in the analyses. Kasper 2006 et al tested two dosages (600 and 1200 mg) of an available product. We pooled the data from these two groups to prevent that the control group of this trial would have been included in the analyses twice.

Dealing with missing data

Dichtomous outcomes
Responder proportions were calculated according to the intention to treat principle, counting drop‐outs as non‐responders. For the comparison hypericum extracts vs. standard antidepressants responder proportions were also calculated on a per protocol basis (as this is considered more appropriate to assess the equivalence of two treatments).

Continuous outcomes
If means and standard deviations from intent to treat analysis with missing values replaced were available, we preferably used these data. In other cases we used analysis based on available data.

Obtaining missing data
If the number of patients responding to treatment and means and/or standard deviations of HAMD scores after completion were not reported, we always tried to contact first or corresponding authors and/or sponsors to obtain these data. In general, we also tried to obtain other missing details on methods and secondary outcomes from authors or sponsors, but the extent to which we were doing this depended on the cooperation of authors/sponsors and the amount of missing information in the publications. We did not impute or recalculate missing standard deviations as these were unavailable only for a few secondary outcomes in a minority of trials.

We tried to contact authors and/or sponsors of 27 of the 29 included trials; for two trials (Kalb 2001; Laakmann 1998) this was considered unnecessary. We did not receive responses for five trials (Behnke 2002; Brenner 2000; Fava 2005; Harrer 1999; Moreno 2005). Very limited additional information was available or needed for three studies (Bjerkenstedt 2005; Volz 2000; Woelk 2000). We obtained relevant additional information to a variable extent from authors, sponsors, or both for the remaining 19 trials.

Data synthesis

The following comparisons were performed:
1. hypericum extracts vs. placebo: a) for dichotomous outcomes (response rate ratios); b) for continuous outcomes; c) for drop‐outs and adverse effects
2. hypericum extracts vs. standard antidepressants: a) for dichotomous outcomes; b) for continuous outcomes; c) for drop‐outs and adverse effects

All main analyses were performed using RevMan 5.

Due to the clinical diversity of the studied populations, the hypericum extracts and the comparison drugs used, we considered that the included studies did not estimate a common underlying effect, but rather that each individual study estimated its single and unique underlying effect. Thus, the application of random effects model in all analyses seemed to be appropriate.

The primary analysis for the comparison of response rate ratios (= relative risks) under treatment with hypericum extracts or placebo was a random‐effects intention to treat meta‐analysis stratified by study precision (above or below median of variance of treatment effect).

Assessment of heterogeneity
Heterogeneity of trials' results was tested with the Chi‐squared test, and the I‐squared statistic was calculated to give an estimate of the degree of heterogeneity. I‐squared values over 50% indicate considerable heterogeneity (Higgins 2003).

Investigation of heterogeneity and subgroup analyses
Predefined subgroup analyses were performed (a) including only trials with response operationalised with the HAMD score; (b) including only trials with response operationalized with the CGI; (c) for the type of extract investigated; and (d) comparing trials originating from German‐speaking countries and from other countries. Weighted mean differences for HAMD scores were calculated after therapy, at 2 to 3, 4, 6 to 8 weeks, and for differences compared to baseline values. For DS scores we calculated after therapy values, for MADRS score after therapy values and differences compared to baseline. As only relatively few studies used the DS, we performed an additional post‐hoc random effects analysis calculating standardised mean differences for any available patient‐rating scale (preferably end of treatment values, but if these were not available also differences from baseline) to investigate whether findings from physician‐rated instruments could be broadly reproduced.

The primary analysis for the comparison of responder rate ratios under treatment with hypericum extracts or standard antidepressants was a random effects intent to treat meta‐analysis stratified for type of synthetic antidepressant (selective serotonine inhibitors or older antidepressants). Predefined subgroup analyses were performed (a) using per protocol data; (b) stratified for country (German‐speaking Europe versus other countries); (c) including only trials with response operationalised with the HAMD score; and (d) including only trials with response operationalised with the CGI.

Additional meta‐regression analyses were performed to investigate the influence of country of origin (German‐speaking versus not German speaking), precision and HAMD baseline values on study findings (responder ratio and mean difference of HAMD scores after treatment) both in placebo and standard antidepressants comparisons. According to current recommendations of experts (Thompson 1999, Lipsey 2000), random effects meta‐regression analyses were carried out using the restricted information maximum likelihood (REML) method. A main advantage of this approach is that it accounts for residual between‐trial heterogeneity. Both univariable and multiple regression models were fitted. We calculated the proportion of explained heterogeneity variance by dividing the heterogeneity explained by the independent variable(s) through the total heterogeneity variance present in random‐effects meta‐analysis. When referring to a whole model, this coefficient was termed R². When referring to the contribution of single covariates the coefficient was termed β². In univariable meta‐regression analyses these coefficients are mathematically equal. In multiple meta‐regression analyses, sum of β² values for all covariates may be slightly different from R². For all meta‐regression analyses the Statistical Package for the Social Sciences (SPSS; Chicago, Illinois) v13.0 software using additional macros by Wilson (Wilson 2002; Lipsey 2000) was used.

Assessment of reporting biases
Visual analysis of funnel plots was performed to identify possible publication bias (Sterne 2001). Furthermore, the asymmetry coefficient was calculated for formal examination of publication bias (Egger 1997).