Rasch measurement

Rasch measurement [18] represents a philosophically different approach to measurement and instrument validation in comparison to the Classical Test Theory (CTT) approaches that typically dominate the validation literature [19]. To contrast, the CTT approach to validation often aims to formulate a model that best reflects as much variance in collected data as possible. Confirmatory factor analysis and other forms of structural equation modelling are common statistical analysis approaches to this end. Consequently, fitting a model to explain a set of data is the aim of this approach, with the best-case outcome being a model with limited unexplained variance and being reflective of a theoretically-meaningful understanding of the latent variable of interest. Rasch measurement places importance on whether the data collected from an instrument’s items provides an invariant representation of an underlying ability or trait of interest. The focus is therefore not on crafting a model or framework that reduces unexplained data variance (and is hopefully theoretically defensible), but for data from measurement items to correctly infer an underlying latent factor [19].

A good understanding of the latent variable of interest to the scale developer and their crafting of the measure items is therefore important, and may undergo several iterations of calibration and refinement, as the items constructed to reflect a latent variable must meet several fundamental measurement requirements per the Rasch model [19]. The underlying factor measured by the items must be singular in nature (i.e., unidimensional), otherwise measurement contamination is inevitable. Bond and Fox [19] provided a good example of early thermometers that varied in their readings on account of temperature and atmospheric pressure, demonstrating a deficiency in accurate instrumentation on the basis of multidimensional measurement. Another key assumption of the Rasch model is that item responses must reflect a linear relationship between the ability or level of a trait a respondent possesses, and their score on the measurement instrument. For example, individuals taking a measure of general mental ability should become progressively less-likely to correctly answer questions of increasing difficulty with respect to how difficult they are in relation to the abilities of the individual. The likelihood (as expressed via a logistic function in the Rasch model) of higher scores on a measure should therefore reflect a linear relationship between individual ability and item difficulty [20]. Items in a Rasch-validated measure must also be invariant in nature. The lack of systematic variation in item responses attributable to facets beyond the underlying factor that the scale is attempting to measure (e.g., the gender of the person providing the response) provides the basis for context-free measurement. These assumptions of the Rasch model represent an exciting approach to instrument validation and refinement in the human sciences, bringing our approaches to measurement closer to that of the fundamental measurement options available to the physical sciences [19].

Rasch modelling of the ProQOL

Rasch modelling provides several considerations for measure validation, and subsequent refinement, with the stricter measurement principles as outlined prior in comparison to a CTT approach to measure validation. An instrument congruent with Rasch measurement principles has not only theoretical validity-based advantages (e.g., evidence for construct validity), but has practical advantages in affording clear comparisons between scores on a measure (e.g., change scores) due to the interval-level data it can provide [20]. The latter is not possible with traditional summed ordinal-level scoring provided by Likert-style measures, which the ProQOL currently recommends [1], and used in typical CTT approaches to measure validation, although in the latter case this issue is rarely heeded [19]. Additionally, Rasch modelling provides indications of measure reliability that extend beyond the sample-based coefficients typically appearing in the nursing literature, such as the reporting of a Cronbach’s alpha value [20], substantiating validity on the stability of a measure’s item difficulty and person ability ordering. The ProQOL therefore appears to be a valid candidate to this end, due to the ambiguity regarding its measurement properties as outlined prior (e.g., [14]).

There are several questions regarding the response format of the ProQOL that have practical implications for the recommended summed scores approach to interpreting the measure. The Likert-style scale (ranging from ‘Never’ to ‘Very Often’) used by participants completing the ProQOL inherently assumes that participants view each item as being equally difficult to respond to. Realistically, the ProQOL items (e.g., “I am (not) happy” versus “I feel overwhelmed because my case (work) load seems endless”) appear to represent different levels of the underlying variable being addressed, and would therefore be unlikely to encourage participants to consider their responses uniformly. The degree to which participants view items as being more or less likely to be rated highly is an integral part of the Rasch model’s accounting for item difficulty, therefore this aspect of the ProQOL can be scrutinised as part of the planned analysis. It is similarly unclear as to whether all five of the response categories provided to participants are being effectively utilised, or whether certain neighbouring response categories can be collapsed together for reasons of parsimony, which can be established as part of the Rasch analysis procedure. If nurses can convey the same information with fewer response options, then this modification can assist with streamlining the administration of the ProQOL. Furthermore, the recommended 25th and 75th percentile cut-points indicative of low, average, and high levels of CS, STS, and BO outlined by Stamm [1] assume that responses on the ProQOL reflect a well-targeted (i.e., capturing a range of person abilities based on each scales’ items), interval-level measure. This is yet to be demonstrated in the ProQOL literature, and warrants substantiation via Rasch analysis.

Therefore, we seek to examine the measurement properties of the three scales of ProQOL via a Rasch analysis approach. In doing so, we aim to substantiate the construct validity and reliability of the measure, and provide estimates of interval-level normed scores for use in a practitioner setting as a screening tool.

Participants

Registered nurses or nurse officers (age x̅ = 46.48 years, SD = 11.78) from Australian hospitals participated in the study (N = 1615), with the sampled nurses being approximately the same average age as those registered nationally (age μ = 44.40 years; [21]). Sampled participants were mostly sourced from public sector (N = 664) and private sector (N = 651) workplaces, with fewer participants working in the aged care sector (N = 273). The sample was predominantly female (N = 1360), although the proportion of male nurses (N = 116, 7.2%) within the sample was similar to the proportion of male nurses within the occupation broadly (i.e., approximately 10%; [21]). Most of the sampled nurses held less senior roles, with nurses of grade one through three (N = 1058) forming the majority of the sample in comparison to more senior nurses of grades four and greater (N = 557).

Linacre [22] suggested that, for item calibrations / person measures to be stable within 0.5 logits, at 99% confidence, then a minimum sample size between 108–243 participants (reflective of limited to best targeting, respectively) is required. Our total sample of nurses exceeded these minimum recommended values. Smith et al. [23] have suggested that mean square estimates of fit adequacy are relatively stable across sample sizes for polytomous Rasch models such as the one suggested for this study.

Measure

Data analysis

The software Winsteps Version 3.92.1 [24] was used to conduct the Rasch analysis. We examined the adequacy of the items as polytomous partial-credit Rasch models [25] in each instance, to account for the prospect of participants heterogeneously perceiving and responding to the Likert-style scale for each item. Misfitting items were identified on the basis of Infit and Outfit mean-square coefficients outside of the range of 0.60 < X < 1.40, as this range reflects reasonable item fit for a Likert-style scale [19]. The reliability of person ordering and item ordering estimates were inferred on the basis of the (more-conservative) real reliability and model reliability and separation coefficients, and Cronbach (KR-20) α, calculated as part of Winstep’s model summary output. Larger estimates of reliability were considered reflective of better measurement properties [19]. The univariate structure of the Rasch model was estimated on the basis of a principle components analysis of the Rasch model residuals [19], with off-factor clusters with eigenvalue coefficients greater than 2.0 investigated further for potential issues regarding a multivariate structure [24]. Disattenuated correlations between the residual clusters were similarly examined to help inform decisions made regarding the prospect of a multivariate structure, with disattenuated r² values greater than 0.50 (i.e., 50% shared variance between the clusters) indicating that a multivariate solution is less likely [24]. Issues with item multicollinearity were detected with standardised residual correlations greater than r = .80. Item response category calibrations were made in accordance to the guidelines suggested by Linacre [26] where monotonicity of the response categories, sufficient observations per response category, category Outfit mean-squares coefficients being less than 2.0 were observed. Examination of Differential Item Functioning (DIF) as an indicator of item invariance was conducted on the basis of nurse gender, seniority (nurse rank ≥ 4; nurse rank < 4), and nurse age (50 ≥ years; < 50 years, in addition to X_age ≥ years; < X_age years). DIF contrast values greater than 0.43, indicative of small-to-moderate effects of differential responding [27], which were statistically significant (p < .05) per their corresponding Mantel χ² coefficient, were considered as evidence of DIF.

Procedure

The data used in this study is derived from a large-scale survey of nurses conducted in 2013 (see [28] for details). Human research ethics approval was provided by the University of Queensland’s Behavioural and Social Sciences Ethical Review Committee (Approval 2013000887), and by Curtin University’s Human Research Ethics Committee (Approval SONM25-2013). Following human research ethics committee approval, an electronic survey was emailed to members of the Queensland Nursing and Midwifery Union (QNMU) who had a current email address. The survey included a number of structured measures including the ProQOL as well as several demographic questions. An even number of nurses from the public, private and aged care sector were invited to complete the anonymous survey. In total, 13,739 nurses across the three sectors were invited to complete the survey. A total of 2679 (19.50%) nurses completed either some or all of the survey. For the purpose of this study only the nurses who completed the all of ProQOL items were included.

Compassion satisfaction

Invariance testing.

We examined measurement invariance across factors of age, gender, and nurse grade seniority. Consistent with the Australian Institute of Health and Welfare’s [21] contrasting of nurses on either side of the 50 years of age threshold, we examined measurement invariance on the CS scale items for nurses older or younger than 50 years of age. Additionally, we examined age-related invariance via splitting nurses on the basis of being younger or older than the mean age of the sample (X_age = 46.48 years). Neither method of examining invariance across age groups demonstrated variant response characteristics, as all contrasts demonstrated statistically non-significant Mantel χ² coefficient probabilities and/or differential item functioning (DIF) contrasts < 0.43 logits, the latter indicating a non-meaningful proportion of item variance across demographic categories ([24], [27]). No evidence of DIF was similarly presented based on the gender of nurses. We conducted our final test of invariant responding on the CS scale based on the grade seniority of the nurses, contrasting registered nurses (or nurse officers) at grades 1 through 3, to those at grade 4 or greater. We found no evidence of invariant responding when contrasting nurse grades along those categories. In summary, the CS scale of the ProQOL, with its modified response scoring approach, appeared to demonstrate good item fit (see Table 1), and invariant responding across several demographic characteristics. We present in Table 2 interval-level scores for the CS scale based on the modified scoring format, and Table 3 presents the 25^th, 50^th, and 75^th percentile cut-point equivalents to Stamm [1] using our modified scoring format. Table 4 presents bivariate correlations and coefficients of central tendency between the modified scoring measure items and the summed measure scores representing CS.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Item measure and fit coefficients, point-correlation with measure coefficients, and coding reference for reduced ProQOL-21 item set.

https://doi.org/10.1371/journal.pone.0193478.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Raw scores and corresponding normed scores, standard errors, and percentile rankings for the modified ProQOL-21 scales.

https://doi.org/10.1371/journal.pone.0193478.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Percentile cut-points for modified ProQOL-21 scoring approach.

https://doi.org/10.1371/journal.pone.0193478.t003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Bivariate correlations and coefficients of central tendency for modified ProQOL-21 scoring approach items.

https://doi.org/10.1371/journal.pone.0193478.t004

Burnout and secondary traumatic stress

Our initial examination of the burnout and secondary traumatic stress measures indicate notable limitations in measurement adequacy (see S1 Results). We found that the burnout measure indicated the best-fitting model when considered as a two factor measure; a consequence of the limited item pool in each minor scale was mediocre reliability and measure targeting. Similarly, a reduction in items for the secondary traumatic stress scale after consulting evidence of item-misfit, alongside evidence of mediocre targeting and 60% of the scale indicating concerns with differential item functioning, suggested that the item-factor relationships proposed by Stamm [1] were difficult to support with our analyses. Returning to the pool of twenty items from the burnout and secondary traumatic stress scales, we tested the prospect of the ProQOL demonstrating adequate measurement properties for a latent compassion fatigue factor, reflected by items from both scales. This approach is consistent with Stamm’s [1] description of Burnout (BO) and Secondary Traumatic Stress (STS) reflecting a general construct of Compassion Fatigue (CF), or the negative consequences of caring for clients, therefore we considered this revised approach theoretically justifiable.

Dimensionality.

After inclusion of the 10 Burnout and 10 Secondary Traumatic Stress items, we conducted an initial inspection of dimensionality to support the proposed singular CF underlying construct, before considering item fit, invariance, and other considerations of measurement adequacy. While the raw variance explained by the measures was less than 50% (Eigenvalue = 16.24, 44.8% variance), this appeared to be muddied by a substantial off-factor cluster of items as suggested by a notable first contrast (Eigenvalue = 3.04, 8.4% variance). This cluster of items consisted of BO scale items, specifically items 4, 17, 29, 15, and 1 (per the item ordering in [1]). We consulted the disattenuated correlations between this cluster (see Fig 1) and the remaining two clusters, and noted that these items appeared to have very little shared variance with the remaining items from the BO and STS scales; r = .429 (r² = .184, or 18.4%) with the second cluster, and r = .288 (r² = .083, or 8.3%) with the third cluster. Due to the limited shared variance of these items, insinuating they were unlikely to represent the same underlying factor as the remaining 15 items, we removed these items and re-examined dimensionality.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Principal components analysis standardised residual plot for compassion fatigue measure.

Standardised residual contrast plot indicating a five item cluster (circled) with notable separation along the Y axis from the remaining Compassion Fatigue items.

https://doi.org/10.1371/journal.pone.0193478.g001

A second test of dimensionality among the remaining 15 items from the secondary traumatic stress and burnout scales indicated a potentially-significant off-factor cluster (Eigenvalue = 2.19, 6.7% of variance). This cluster consisted of items 21, 26, and 19 from Stamm’s [1] burnout subscale, although in this instance the disattenuated correlations between this cluster and the remaining clusters did not seem to indicate multidimensionality as prominently. The relationship between clusters 1 and 2 (r = .682, r² = .465, or 46.5% shared variance) and clusters 1 and 3 (r = .793, r² = .629, or 62.9% shared variance) suggested that near-half or greater than half of the variance between the clusters was shared, respectively. Consequently these items did not seem to be clear candidates for removal from the model at this stage of analysis, and were retained as indicators of CF. The overall quantity of raw variance in the underlying construct explained by the measure at this stage of analysis was over 50% (Eigenvalue = 17.49, 53.8% variance explained).

Invariance.

As item fit, calibration, and unidimensionality were reasonably justified for the CF measure, tests of item invariance across gender, seniority, and age categories were conducted. While nurse seniority and age categories did not indicate potential concerns with variant responding, examining the differential item functioning (DIF) of the CF items on the basis of nurse gender suggested invariant responding (see Fig 2) for item 26, “I feel ‘bogged down’ by the system” [1]. Male nurses (item difficulty = -2.05 logits) found item 26 significantly easier to endorse in comparison to their female counterparts (item difficulty = -1.47; Mantel χ² = 4.54, p = .033, DIF contrast = 0.57 logits). As this degree of difference represented a slight-to-moderate level of invariant responding ([24], [27]), we created a male-response and female-response version of item 26, before adding these items to the model after removing the general-response version of item 26, as a means of accounting for the DIF of this item ([24], [32]). Following the introduction of these items, item 25 (“As a result of my (helping), I have intrusive, frightening thoughts”; [1]) demonstrated notable DIF between male (item difficulty = 1.62) and female (item difficulty = 2.08; Mantel χ² = 5.70, p = .017, DIF contrast = 0.46 logits) nurses. Male nurses did not perceive this item as difficult to endorse as female nurses, and this item was reflective of a small-moderate degree of differential item functioning ([24], [27]). Splitting item 25 into gender-variant versions using the same approach outlined prior did not reveal any further evidence of DIF. Table 1 presents the item difficulties, recoding modifications, and fit coefficients, and we present in Table 2 the interval level scoring of the final modified CF scale. Table 3 provides cut-points per Stamm’s [1] 25^th, 50^th, and 75^th percentile delineations.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Person differential item functioning (DIF) plot for compassion fatigue items.

Significant (Mantel χ² p < .05) and notable (DIF contrast coefficient > 0.43; [27]) gender-variant responding was identified. Items 25 and 26 (dotted-circled) had significant, notable differences in item-difficulties between male and female nurse respondents.

https://doi.org/10.1371/journal.pone.0193478.g002

Theoretical implications

Our findings suggest several important theoretical implications for the use of Stamm’s [1] ProQOL, particularly within the profession of nursing. Primarily, the difficulty in substantiating the validity of the ProQOL’s burnout and secondary traumatic stress items as distinct, well-functioning scales is a point of concern given the measure’s use in healthcare practitioner research (e.g., [4], [5], [6]). Particularly in instances where research has been conducted using the burnout or secondary traumatic stress scales as indicators of their inferred latent factors (e.g., [2]), the adequacy of citing an alpha coefficient and the validity claims presented in the measure’s manual are potentially questionable forms of evidence. Regarding the former, alpha is frequently and incorrectly regarded as an indicator of unidimensionality/homogeneity, and widely-cited “cut-off” values such as a > .70 are not guarantees of measure reliability [33], therefore alpha coefficients demonstrated by previous research using the ProQOL do not dismiss this validity query. Instead, this finding reinforces questions raised by authors such as Bride et al. [14] and Ray et al. [7] about the limited representation in the literature regarding the measure’s construct validity, substantiated by our mixed evidence to this end. Researchers working with data collected from the ProQOL may wish to use our revised ProQOL-21 item and scoring approach (see S1 Syntax for SPSS Syntax) if they wish to make inferences about compassion satisfaction and compassion fatigue. If researchers intend to examine relationships with burnout and secondary traumatic stress as inferred by the ProQOL, we caution that this approach may require further clarification from future literature to confirm its tenability, specifically regarding evidence of construct validity for these measures.

Echoing the advice provided by authors such as Linacre [24] and Bond and Fox [19], our findings do not necessarily indicate that the items provided by Stamm [1] are unsalvageable. Despite the limitations in accounting for noteworthy off-factor variance, and/or poorly reflecting participant ability at the estimated level of item difficulty, Bond and Fox [19] recommend that findings such as these serve as a signal to researchers to “…‘find out why’ not ‘throw it out’”. As we noted in the Results, many of the predictors we set aside in this iteration of analysis appeared to account for variance outside of compassion fatigue. While we have attempted to provide contextual explanations as to why scoring on these items did not conform to what was expected (i.e., an assumed underlying compassion fatigue factor driving participant responses), the concepts behind the original item generation by Stamm [1] may still be valid, albeit requiring some rewording or further clarity enhancement to better reflect facets of the intended underlying factor. Take, for example, item 5 (“I jump or am startled by unexpected sounds”; [1]). While this item presumably sought to reflect the increased arousal experienced by healthcare professionals with secondary traumatic stress [34], we argued that it was likely to also capture variance associated with the acoustic startle reflex [30]. Therefore an alternative item, intended to address the facet of increased physiological arousal argued to be reflective of individuals experiencing secondary traumatic stress, is prospectively a valid future focus for an addition to the original ProQOL items. It is with this in mind that we note that the conceptualisation of burnout and secondary traumatic stress are not inherently flawed, a stance that is reinforced by the sound reasoning of previous authors in this field. Instead, the ProQOL requires further iterations to address the paucity of well-fitting items [19] relevant to the burnout and secondary traumatic stress subscales. Therefore, for the purposes of theory generation and model testing, we would recommend that authors intending to measure the latent factors of burnout and secondary traumatic stress consider alternative available measures (e.g., the short burnout measure of [35]) until a future item-generation iteration of the ProQOL has been calibrated and validated.

In terms of theory-generation relevant to substantiating a general Compassion Fatigue construct, our investigation of the construct validity of the ProQOL indicates that a mix of items originally from the burnout and secondary traumatic stress (STS) scales, when combined, produces a valid, short-length measure of compassion fatigue. Rather than separating burnout and STS as separate constructs, the present analysis indicates that the construct is better captured as a combination of STS items that capture aspects of the trauma associated with caring, along with other items related to emotional and physical exhaustion associated with caring. Stamm [9] suggested that the overlap between the burnout and secondary traumatic stress scales was due to distress common to both conditions. Our results suggest that rather than two independent but related conditions as proposed by Stamm [9], compassion fatigue can be meaningfully construed as one construct, which may assist researchers examining broader negative facets of healthcare provision, and their anticipated outcomes.

Practical implications

The reduced number of items for the revised ProQOL-21 lends itself to greater ease of administration in a practitioner setting as a screening tool, per Stamm’s [1] recommended use of the scale. Both the compassion satisfaction and compassion fatigue measures presented in our Results demonstrated excellent measurement characteristics, therefore practitioners can confidently apply our revised approach to scoring and inference as part of their intended use as a screening tool. In both clinical and research settings this brief version of the ProQOL means that it is possible to measure nurse professional quality of life in a more efficient and less time-consuming manner. This is important, as nurses are often asked to complete many workplace surveys, therefore a shortened version of this measure can reduce the prospect of being over-burdened in this regard.

We are aware that the ProQOL has been and continues to be used extensively around the world [1]. In order to use this existing pool of data, we have developed an SPSS syntax file available to other researchers and practitioners (see S1 Syntax) to re-score their ProQOL data according to the two scale structure reported here. This will enable researchers and practitioners to determine a total score for CS and CF that can then be compared with data collected by different research groups across different settings.

Limitations

Our male sample (N = 116) was considerably fewer than our female sample, which may have had implications for the estimation of error terms for this subsample. Future studies examining the measurement properties of the ProQOL should try to sample additional male participants, as it would assist in bolstering the confidence of accuracy for the measure’s estimations within this minority group of nurses. While the low response rate in this study is common in nursing research [36], the potential issue of participant bias in participating nurses may be a prospective limitation of this research. To temper this prospect, we note that the demographic characteristics of the nurses participating in this study were highly similar to those reported by national data on employed nurses [21], therefore biased participant selection was not a clear threat to these findings.

Directions for future research

The results of this study open up many avenues for future research. As noted previously, for the ProQOL to be regarded as a valid measure of burnout and secondary traumatic stress in particular, further research must be directed to iterating additional items that address the notable gaps and measurement shortfalls for these subscales. However, should future researchers wish to examine compassion satisfaction and compassion fatigue, our revised scoring procedures (which is further facilitated with the available SPSS syntax in S1 Syntax) provide a rigorous footing with which to explore these constructs as indicated by the existing ProQOL item set. Studies of this nature are essential steps in the creation of theory-driven interventions aimed at building resilience and reducing compassion fatigue [2]. To this end, two important directions for future research appear necessary regarding the modified version of the ProQOL. Due to the use of Australian participants in this study, the potential for differential item functioning on the basis of cultural differences could not be explored. As previous studies (e.g., [16]) have noted intercultural variations when examining the factor properties of the ProQOL, it would be valuable for research conducted in other countries to confirm the adequacy of the measure properties as part of the iterative measure refinement process we propose. Furthermore, while the construct validity of the measure was the focus of inquiry in this article, other facets of validity (e.g., predictive validity) merit exploration. While we have presented fundamental bivariate correlation coefficient information to demonstrate the revised-scoring procedure measures’ negative relationship with each other as anticipated in the CS/CF literature [1], future research may wish to examine predictive validity via other variables (e.g., resilience; [2]) following the measure modifications.

The use of the partial-credit Rasch model in our analyses, serving to maximise the model’s person-targeting characteristics, has implications for the presentation of the ProQOL test items and their response options. While a variable series of response anchors and response choice lengths may have advantages in reducing acquiescent-responding biases [37], we are concerned about the potential for this to be a clarity-inhibiting element in scale presentation. Therefore we recommend practitioners and researchers employ post-data-collection recoding to adjust for the category response limitations highlighted in our analyses. Additional items may allow future iterations of this measure to apply a consistent scoring approach across all items without the targeting limitations outlined in our Results (i.e., demonstrate good targeting via a rating scale model). In addition to addressing item-difficulty gaps and item misfit issues by generating additional items for the ProQOL, and in-turn validating these items are indicators of a valid measurement model, a future iteration of the measure may be able to uniformly present a well-targeted series of response options without needing to have item-level variations in response category behaviour.