The following is a summary of the psychometric properties of the ECI based on a sample of children assessed in two Midwestern states. A more complete description of the sample and the ECI’s technical soundness properties are available in the following publications:
Greenwood, C., Walker, D. Buzhardt, J. (2010). The Early Communication Indicator (ECI) for Infants and Toddlers: Growth Norms from Two States. Journal of Early Intervention, 32, 5, 310-334.
Greenwood , C. R., Carta, J. J., Walker , D., Hughes, K., & Weathers, M. (2006). Preliminary investigations of the application of the Early Communication Indicator (ECI) for infants and toddlers. Journal of Early Intervention, 28, 178 -196.
This summary addresses the following questions:
What is the reliability of ECI scores when administered and scored in the field by trained early childhood home visitors and center-based educators?
What is the alternate forms reliability between the two toy sets used for the ECI?
What are the expected patterns of growth for the ECI, and is this growth influenced by gender, IFSP status, or home language?
What are the growth patterns of the ECI’s four key skill elements, and are they influenced by IFSP status?
How does performance on the ECI correlate with performance on other standardized language assessments (criterion validity)?
Unless otherwise indicated, this information is based on findings from Greenwood, Walker, & Buzhardt (2010).
Child Sample and Data Descriptives
Table 1 describes the sample of children that received ECI assessments. ECI assessors in the two states were EHS program staff who conducted ECI assessments as a regular part of the services provided to families on their home-visiting caseload or children in their center. The number of staff per program that used the ECI varied. In total, the number of staff involved was 262 in State 1 and n = 241 in State 2. The EHS service model in the majority of programs was a home-visiting model in which program staff visited homes and worked with parents on intervention goals in the home setting. For example 62% of all ECI’s collected were in homes compared to 36% in Centers, and 2% other (e.g., grandparent home, etc.). Using the online data system to enter their ECI assessment data, it was possible for program staff to view individual children’s ECI growth charts. Local program directors/coordinators could view and print children’s growth charts, as well as produce a program-wide summary report.
Table 1. Children’s Descriptives
The total number of ECI assessments collected was 16,688. Sixty-three percent of assessments (10,496) were collected in State 1 and 37% (6,162) in State 2. Ninety-six percent of all ECIs were administered in English, 4% in Spanish. By state, the proportion of ECIs administered in English was 93.9% (State 1) versus 99.4% (State 2). A handful of ECIs in each state were administered in other languages (e.g., Creole, Vietnamese, Cantonese, etc.).
What is the reliability of ECI scores when administered and scored in the field by trained early childhood home visitors and center-based educators?
In all, 246 paired interobserver assessments, 206 from State 1 and 40 from State 2, were available for analysis (see Table 2). Assessor training required that assessors achieved both administration fidelity and coding reliability, The analysis of inter-rater reliability for this report focused on ECI score reliability rather than percentage agreement (e.g., Hartmann, 1977; Walker & Buzhardt, 2010). Pearson r was used to compare the covaration in primary and reliability assessors’ scores with alpha set at .01 because of the large sample size. The overall Pearson r was strong at .96 (.96 in State 1 versus .86 in State 2) indicating a pattern of similarity between assessors scores. The equivalence of the groups’ total communication scores was estimated by calculating confidence intervals around the mean difference between the groups, overall and within each state (Seaman & Serlin, 1998). Perfect equivalence is indicated by a 0 at the low end of the interval. For overall and State 1, the low end of the confidence interval was near 0 at .03 and .01 respectively, indicating near equivalence. The confidence interval for State 2 however was not as equivalent at .67 (see Table 2).
Table 2
Interobserver Agreement Summary.
What is the alternate forms reliability between the two toy sets used for the ECI? (Greenwood et al, 2006)
This test of reliability compares Total Communication scores formed when observations were made using alternate toys, in this case the Barn versus the House. Alternate forms reliability was r = .72 for ECI Total Communication mean level. This reflects a moderate to high level of reliability – highly similar scores result regardless of the toy used.
Were Children’s Patterns of Growth in ECI Total Communication Influenced by Gender, IFSP Status and Home Language Environment? The growth pattern in total communication for all children was increasing monthly with a mean slope of 0.97866 (SE = .02228) responses per minute per month and positively accelerating by .01022 (SE = .00060) response per month. Level-2 analyses of the influence of gender, home language, and IFSP status, indicated no differences in the trajectories of boys versus girls, nor between children who heard English versus Spanish at home. Differences were noted, however, in the growth of children with versus without an IFSP in terms of mean intercept and slope but not acceleration. The difference between IFSP groups in mean intercept at 36 months was -6.74025 responses per minute, t[5189], p = .0001, d = .71 and the mean difference in slope was -0.219, t[5189, p = .002, d = .31). Similar values for slope were 0.79 responses per minute per month for children with IFPS versus 1.01 for children without IFSPs.
What was the pattern of growth for each of the four ECI key skill elements (i.e., gestures, vocalizations, and single- and multiple- word utterances), and how were they influenced by differences in IFSP status?
Based on the raw data, we anticipated finding a relatively stable trend in gestures and an increasing trend in infants’ and toddlers’ vocalizations followed by their decline after about 15 months of age as the use of single and multiple words began increasing. Level-1 growth modeling essentially confirmed these patterns. With the exception of gestures, patterns of growth for children with IFSPs versus those without indicated slower development of early communication.
How does performance on the ECI correlate with performance on other standardized language assessments (criterion validity)? (Greenwood et al, 2006)
Criterion validity is designed to test whether or not the ECI Total Communication score correlates to other measures of early communication. Moderate to high levels of correlation indicate that the ECI does measure early communication. Two criterion validity measures were selected to assess expressive communication and to provide a multi-method, multi-informant dimension to the validity study.
The Preschool Language Scale – 3 (PLS-3; Zimmerman, Steiner, & Pond, 1992) is a standardized norm-referenced measure of receptive and expressive language skills. It was selected because of its technical adequacy, wide use, and coverage of the entire age range of interest. Only the expressive subtest of the PLS-3 was used.
The Caregiver Communication Measure (CCM; Walker, Hart, Linebarger, & Parsley, 1998) provided a parent’s report of the child’s use of five categories of communication: gestures, sounds, understanding of words, use of specific words, and grammar. This second criterion measure was created to tap parents’ perceptions of their child’s language and communication skills. The goal was to gather additional information that might not be apparent from ECI assessments taken in childcare setting.
The following correlations demonstrate relatively strong criterion validity for the ECI:
Pearson r = .62 (ECI Total Communication mean level vs. Preschool Language Scale [PLS-3])
Pearson r = .51 (ECI Total Communication mean level vs. Caregiver Communication Measure [CCM])
The total number of ECI assessments collected was 16,688. Sixty-three percent of assessments (10,496) were collected in State 1 and 37% (6,162) in State 2. Ninety-six percent of all ECIs were administered in English, 4% in Spanish. By state, the proportion of ECIs administered in English was 93.9% (State 1) versus 99.4% (State 2). A handful of ECIs in each state were administered in other languages (e.g., Creole, Vietnamese, Cantonese, etc.).
What is the reliability of ECI scores when administered and scored in the field by trained early childhood home visitors and center-based educators?
In all, 246 paired interobserver assessments, 206 from State 1 and 40 from State 2, were available for analysis (see Table 2). Assessor training required that assessors achieved both administration fidelity and coding reliability, The analysis of inter-rater reliability for this report focused on ECI score reliability rather than percentage agreement (e.g., Hartmann, 1977; Walker & Buzhardt, 2010). Pearson r was used to compare the covaration in primary and reliability assessors’ scores with alpha set at .01 because of the large sample size. The overall Pearson r was strong at .96 (.96 in State 1 versus .86 in State 2) indicating a pattern of similarity between assessors scores. The equivalence of the groups’ total communication scores was estimated by calculating confidence intervals around the mean difference between the groups, overall and within each state (Seaman & Serlin, 1998). Perfect equivalence is indicated by a 0 at the low end of the interval. For overall and State 1, the low end of the confidence interval was near 0 at .03 and .01 respectively, indicating near equivalence. The confidence interval for State 2 however was not as equivalent at .67 (see Table 2).
Table 2
Interobserver Agreement Summary.
What is the alternate forms reliability between the two toy sets used for the ECI? (Greenwood et al, 2006)
This test of reliability compares Total Communication scores formed when observations were made using alternate toys, in this case the Barn versus the House. Alternate forms reliability was r = .72 for ECI Total Communication mean level. This reflects a moderate to high level of reliability – highly similar scores result regardless of the toy used.
Were Children’s Patterns of Growth in ECI Total Communication Influenced by Gender, IFSP Status and Home Language Environment?
The growth pattern in total communication for all children was increasing monthly with a mean slope of 0.97866 (SE = .02228) responses per minute per month and positively accelerating by .01022 (SE = .00060) response per month. Level-2 analyses of the influence of gender, home language, and IFSP status, indicated no differences in the trajectories of boys versus girls, nor between children who heard English versus Spanish at home. Differences were noted, however, in the growth of children with versus without an IFSP in terms of mean intercept and slope but not acceleration. The difference between IFSP groups in mean intercept at 36 months was -6.74025 responses per minute, t[5189], p = .0001, d = .71 and the mean difference in slope was -0.219, t[5189, p = .002, d = .31). Similar values for slope were 0.79 responses per minute per month for children with IFPS versus 1.01 for children without IFSPs.
What was the pattern of growth for each of the four ECI key skill elements (i.e., gestures, vocalizations, and single- and multiple- word utterances), and how were they influenced by differences in IFSP status?
Based on the raw data, we anticipated finding a relatively stable trend in gestures and an increasing trend in infants’ and toddlers’ vocalizations followed by their decline after about 15 months of age as the use of single and multiple words began increasing. Level-1 growth modeling essentially confirmed these patterns. With the exception of gestures, patterns of growth for children with IFSPs versus those without indicated slower development of early communication.
How does performance on the ECI correlate with performance on other standardized language assessments (criterion validity)? (Greenwood et al, 2006)
Criterion validity is designed to test whether or not the ECI Total Communication score correlates to other measures of early communication. Moderate to high levels of correlation indicate that the ECI does measure early communication. Two criterion validity measures were selected to assess expressive communication and to provide a multi-method, multi-informant dimension to the validity study.