Connecting high school dropouts to employment and education: an impact study of the National Guard Youth ChalleNGe Program
IZA Journal of Labor Policy volume 5, Article number: 10 (2016)
This paper presents findings from the multi-year, random assignment study of the National Guard Youth ChalleNGe Program (NGYCP), an intensive residential program for high school dropouts. The “second chance” program gives youth an opportunity to earn a high school credential and prepare to enter the workforce or continue on to post-secondary education. The NGCYP study sample includes 1173 youth, ages 16–18, from ten program sites across the USA. Positive impacts on employment and educational outcomes were sustained 3 years after entering the study, with some evidence of older participants benefiting more than younger participants. Implications for research and practice are discussed.
JEL codes: J1, I26
Young people who drop of out of high school face long odds of success in a labor market that increasingly values education and skills. In the USA, between 3.5 and 6 million young adults, ages 16 to 24, can be classified as high school dropouts, those without a high school credential and not currently enrolled in school (Northeastern University Center for Labor Market Studies 2009, U.S. Bureau of Labor Statistics 2014). While many of these young adults eventually earn a high school credential—a diploma or more often a General Educational Development (GED)Footnote 1 certificate—a long delay may place them at a serious disadvantage in competing for jobs and going on to college.Footnote 2 Early work experience is closely linked to future earnings and labor market success.Footnote 3 In addition, there is growing evidence of the limited value in the labor market of the GED without further educational attainment (Heckman et al, 2010). More dramatically, for those that do not reconnect to the educational system and attain at least a high school credential, the individual and societal costs are high. High school dropouts face higher rates of unemployment, poverty, criminal involvement, and health difficulties. As a result, this population contributes significantly less in taxes from earnings while requiring more public spending in other areas such as incarceration costs and public assistance benefits (Northeastern University Center for Labor Market Studies 2009).
In response to the range of issues associated with dropping out of school, prevention and “second chance” programs have been developed to assist or reconnect this higher-risk youth population. These programs often offer a mix of education, training, employment, and supportive services to youth ranging in age from roughly 16 to 24 years of age. The programs themselves vary widely in terms of the specific populations they target (such as youth with disabilities or those involved in the justice system), the scale (such as large federally funded national programs vs. smaller community-grown models), the residential or non-residential setting, and the particular emphasis or combination of services offered. In recent years, there has also been an education shift in focus from GED obtainment as a goal towards high school diploma-granting opportunities and post-secondary education trajectories (Bloom, 2010). Ultimately, the goal is to help these youth successfully transition to adulthood with improved skills and/or credential and become productive, contributing members of society.
This paper presents results of the multi-year evaluation of one of these programs targeted at high school dropouts, the large-scale, residential, National Guard Youth ChalleNGe Program (NGYCP). The evaluation was led by MDRC, a nonprofit, nonpartisan research organization. This paper presents longer-term impacts on a range of educational and vocational outcomes for participants approximately 3 years after they joined the study. Shorter-term results published by MDRC showed promising results on high school credential receipt and employment, among other outcomes.
2 Background on success of youth interventions
Interventions targeting youth who have dropped out of high school have had mixed results, among those that have been rigorously evaluated. Indeed, a review of training programs operated under the Job Training Partnership Act in the 1980s found that these second chance programs produced few (and sometimes even negative) impacts on earnings (Bloom 2010; Orr et al. 1997). For example, JOBSTART, New Chance, and Job Corps have all struggled to produce sustained improvements in employment or earnings, although they have shown success in increasing GEDs and vocational certificates (Cave et al. 1993; Quint et al. 1997; Schochet et al. 2003, 2006). In response to the challenge of achieving enduring effects in this population, researchers note the importance of tracking youth over time to see how the impacts unfold in the short-term, medium-term, and longer-term (Rodríguez-Planas 2010). Relatedly, the question has also been raised about whether these relatively brief programs for high school dropouts should theoretically result in long-lasting effects or just a shorter burst of changes or outcomes for a young person.
Long-lasting effects are especially challenging for residential programs, in which participants often struggle to maintain positive changes as they reenter their peer groups and communities. For example, adolescents may arrive home to difficult living situations or friends that may be a negative influence. As a result, gains made during residential programs often diminish or disappear when participants return to their communities (Godley et al. 2001).
Although some programs may have been insufficiently intensive, or were evaluated prematurely, researchers and other experts on youth programming have suggested that many of the prior programs erred by not utilizing a positive youth development (PYD) model (Bloom 2010). While other programs might focus on “fixing” a young person’s problems, the hallmark of the PYD approach is a view of youth as positive resources to be developed (Roth and Brooks-Gunn 2003; Larson 2000; Lerner 2005). Similar to ChalleNGe, the federally funded YouthBuild program utilizes a PYD approach to individually support the youth it serves. YouthBuild gives youth leadership opportunities throughout the program, which provides vocational training and instruction towards a GED or high school diploma.
3 The ChalleNGe model
In the late 1980s and early 1990s, the Center for Strategic and International Studies (CSIS) led an initiative to develop new approaches to working with young high school dropouts. CSIS concluded that features of the military structure could be beneficial for at-risk youth and that the National Guard, with its strong community service mission, was well-suited to operate such a program (Cullinan et al. 1992). Congress funded a ten-site pilot of ChalleNGe in 1993, and funding was made permanent in 1998. The program currently operates in more than 30 locations across the USA and approximately 121,000 young people have graduated from the program since it began.
Utilizing a PYD model and run through the National Guard Bureau, ChalleNGe is an intensive program designed to “reclaim the lives of at-risk youth” who have dropped out of high school and give them the skills and values to succeed as adults.Footnote 4 As one program director said, “It’s not about tearing these kids down. You have to build them up” (Bloom et al. 2009). The 17-month program is divided into three phases: the pre-ChalleNGe phase (2 weeks), the residential phase (20 weeks), and the post-residential phase (1 year). During the first two phases (totaling 22 weeks), the participants live at the program site, often on a military base. Throughout the program, ChalleNGe uses its specific setting, mix of activities, and the relationships it tries to create with youth—along with the education and training it offers—to promote positive change in the youth it serves. This change can potentially be seen through a youth’s attitudinal shift, improved education and employment outcomes, or other markers as steps towards independence and adulthood.
To start, the 2-week orientation period can be very challenging, both physically and mentally, much like military training. Candidates who complete pre-ChalleNGe are formally enrolled in the program as “cadets” and move to the residential phase. The curriculum for this phase is structured around eight core components that reflect current thinking about how to promote PYD: leadership/followership, responsible citizenship, service to community, life-coping skills, physical fitness, health and hygiene, job skills, and academic excellence. Cadets spend the largest share of each day in the education component. At the time of the study, most programs helped participants prepare for the GED exam, and a small number offered a high school diploma.
The program environment is described as “quasi-military”: the daily schedule is highly structured with almost no downtime, and staff closely monitor cadets at all times.Footnote 5 A typical 100-bed program employs a multidisciplinary staff of approximately 50 people. The largest contingent of the staff is the cadre (or team leaders), who directly supervise the cadets day and night. The cadre come from a variety of backgrounds; however, a large majority have some military experience, ranging from one term of service to retirement from a full military career. In addition, all programs have a minimum of six full-time instructors. The instructors teach the GED subject courses, including math, science, writing and language arts, and computer skills, as well as other parts of the core components, including responsible citizenship and job skills. The ChalleNGe programs vary among states in the sources of their instructional staff, with some hiring instructors directly and others using teachers from local school districts or community colleges. Finally, a team of counselors work with cadets both in the context of one-on-one career and psychological counseling, as well as group classes, including life-coping skills, job skills, and health and hygiene. Most of the counseling staff hold either bachelor’s or master’s degrees in psychology, social work, mental health, alcohol and drug counseling, or school counseling. Note that while ChalleNGe utilizes a military framework (such as discipline, facilities, and staff) to accomplish its objectives, there are no military service requirements for participants.
Showing competency on the core components and having an acceptable placement lined up are requirements of successfully completing the residential phase or “graduating” from ChalleNGe. Throughout the residential phase, the cadets work with program staff to arrange a post-residential placement. Acceptable placements include employment, education, and military service. After graduation, participants move into the 1-year post-residential phase, which includes the placement and a structured mentoring program. The post-residential phase was developed to support youth’s transition back into the community and to directly address the common challenge of erosion of effects upon reentry into the community. The ChalleNGe mentoring program differs from traditional mentoring in that young people nominate their own mentors. Mentors tend to be drawn from family friends, extended family members and godparents, school and extracurricular staff, and religious leaders. ChalleNGe initiates the mentoring relationship during the residential phase, after the staff has screened and trained the mentors.Footnote 6
Although the National Guard Bureau collects extensive data on program participation and participant outcomes that indicate the positive influence the program can play in participants’ lives, ChalleNGe had not been rigorously evaluated prior to this study (National Guard Bureau 2010).
4.1 Design and data
Youth were recruited for the study through participating ChalleNGe sites.Footnote 7 The 12 participating sites were not chosen randomly; rather, there was an effort to identify programs that had stable staffing and that tended to receive more applicants than they could serve, a prerequisite for conducting a random assignment evaluation. In addition, the Department of Defense authorized a modest amount of funding to support enhanced recruitment efforts by the programs that participated in the evaluation.
Although 12 programs across the country agreed to participate in the evaluation, there were some incoming classes in which the number of applicants was too small to allow random assignment to take place. As a result, random assignment was conducted in only 10 of the 12 participating programs in late 2005 through early 2007. Random assignment was conducted for 18 incoming class cycles across the ten sites, for a total of 3074 participants. Due to different program sizes and numbers of participating class cycles, the number of participants varied greatly by site. At the low end, one site included 92 research participants, and, at the high end, one site included 941 research participants.
To be eligible for the ChalleNGe program, youth must have dropped out or been expelled from school, be unemployed, drug-free at the time of entry into the program, not currently on probation or parole for anything beyond juvenile status offenses, not serving time or awaiting sentencing, not under indictment or charged, and not convicted of a felony or capital offense. The program is open to both males and females, although about 80 % of the participants nationwide are male.
As part of their application to a participating ChalleNGe program, applicants signed consent forms to participate in the study and completed a baseline survey prior to randomization. Footnote 8 Of the 3074 participants, 2320 youth were assigned to the program group and 754 youth were assigned to the control group.Footnote 9 Follow-up surveys were conducted, on average, at approximately 9 months, 21 months, and 3 years after participants entered the study.
This study focuses solely on outcomes from the 3-year survey in order to assess longer-term impacts.Footnote 10 The survey was fielded to 1507 youth, yielding a total of 1173 sample members (722 program group and 451 control group) who completed the survey, for an overall response rate of 78 % (79 % of program group members and 76 % of control group members).
It is important to note that the survey fielding sample was not drawn to represent the full research sample. Given the varying samples from the individual sites, the impact results discussed below pool across all sites and provide impact estimates that weight each of the ten participating sites equally. Similarly, the random assignment strategy that maximized the number of youth served by the programs throughout this period created varying random assignment ratios. Therefore, the survey sampling aimed to adjust for these issues as much as possible. For example, the stratified sampling plan drew a higher percentage of sample members from smaller sites and from the control group. Within these parameters, the subsample was drawn randomly from the full research sample. However, as discussed below, there are limitations on the study’s generalizability beyond the survey sample. For the remainder of this paper, the survey sample will be the primary discussion of this study.
Demographic information about study participants indicate that ChalleNGe is serving a diverse group of high school dropouts (see Table 1). Approximately 88 % were male and the average age at the time of study enrollment was just under 17 years of age.Footnote 11 Just over 40 % of the sample described themselves as white, while over 30 % identified as black and another 18 % as Hispanic. Twenty-six percent of youth reported someone in their household received public assistance and fewer than half of the participants lived in two-parent households. More than two thirds of the sample (83 %) had completed tenth grade or lower. Eighty-three percent reported having been suspended from school, and 18 % had been convicted of a crime.
This study focuses on key program outcomes encompassing the major objectives of the intervention, including self-reported measures of youth educational, vocational, and employment outcomes.
GED/high school diploma was measured using two single-item questions asking whether youth had received a high school diploma and whether they had received a GED.
College credit was measured using a single-item question asking if youth had received any credit towards a college degree.
Vocational training and certification was measured using two independent single-item questions that asked if youth had participated in vocational training and if youth had received a trade license or training certificate.
Military enlistment was measured using a single-item question asking if youth had enlisted in the military since random assignment.
Employment and earnings measures were derived from a series of questions including the start and end date of each job they had since random assignment, hours worked per week, and wages. These measures compile the self-reported information on employment and earnings in the 12 months prior to the survey interview and at the time of the interview.
Involved in a productive activity combined responses to a series of questions and includes participation at the time of the survey interview in any of the following: employment, school, GED programs, vocational activities, the military, or a residential program. A separate measure utilized the earlier GED/high school diploma measure and ongoing participation in any of the following: employment, post-secondary education, vocational activities, the military, or a residential program.
Living independently was based on a single-item question that asked where a youth lived at the time of the survey interview.
Married or living with a partner was based on a single-item question that asked youth about their current marital status.
4.4 Statistical procedures
First, we conducted tests to verify that random assignment was successful in creating statistically equivalent groups at baseline. Then, program impacts tested for statistical significance in comparisons of the regression-adjusted means of the program and control groups for all outcome variables.Footnote 12 An intent-to-treat analysis is utilized, and thus, all members of each group were included regardless of their participation in the ChalleNGe program. As mentioned above, given the large variation in site sample sizes, the comparisons are drawn across all sites for a pooled estimate. The regression model controlled for sample member baseline characteristics and was weighted to account for differences by sites in each of the following aspects: site sample size (so each of the ten sites contributes equally to the results), survey response rates, and program versus control ratios. Given the aim of the impact analysis to understand the effects of the program across different sites, the decision was made to weight each site equally. This means, however, that the impacts presented do not represent the distribution of youth for the entire study. For the subgroup analyses, impacts for each subgroup were calculated, and an H-statistic was generated to assess whether the difference in impacts between subgroups was statistically different.
5.1 Initial equivalence and attrition
Table 1 presents a comparison of baseline characteristics between participants assigned to the treatment group and those assigned to the control group. The table shows a handful of statistically significant differences between the two groups, but one group does not appear more disadvantaged or “at risk” than the other. For example, on average, those in the program group were more likely than the control group to have completed eleventh grade. However, this group also reported higher rates of conviction and drug or alcohol use. Overall, a logistic regression predicting treatment status among respondents confirmed that baseline characteristics did not significantly predict treatment status among those who responded to the 38-month survey (p = 0.25).
As discussed earlier, although the 3-year follow-up survey did not attempt to represent the full sample of baseline participants, analyses were conducted to assess the degree to which survey respondents (the sample included in the current study) were similar to the remainder of the larger sample at baseline. An analysis of baseline characteristics between these two groups revealed some small, but statistically significant, differences that limit the generalizability of the results (see Appendix Table 4). As was to be expected given the sampling plan’s aims, the survey respondents included significantly fewer members of the program group than the remaining sample in an effort to represent the program and control groups more equally. In addition, compared to those who were not included in the 38-month survey, survey respondents were more likely to be male, white, or Hispanic and less likely to be black, less likely to receive public assistance, more likely to have been convicted, and differed in their family composition and who suggested that they should apply to ChalleNGe. A logistic regression confirmed that there are systematic differences of baseline characteristics between the two groups that limit the generalizability of the results beyond the survey sample. Again, these differences were to be expected given the survey sampling plan, the variation in site sample sizes, and the demographic differences between the programs’ applicants.
5.2 Intervention received
Based on data from ChalleNGe’s Management Information System (MIS), among all youth assigned to the program group, 82 % started the pre-ChalleNGe phase. Sixty-eight percent of the program group completed the 2-week Pre-ChalleNGe residential phase and enrolled in the residential phase. Fifty-three percent of the program group graduated from ChalleNGe by finishing the residential phase and its requirements. Among those who completed pre-Challenge and formally enrolled in ChalleNGe, approximately 78 % graduated from the program. These percentages are similar to national averages for ChalleNGe program completion among enrollees. Unfortunately, participation rates among the survey sample were not available.
At the time of the 3-year follow-up survey, 56 % of participants in the treatment group reported that they were still in contact with their mentors. In addition, a total of eight youth assigned to the control group attended ChalleNGe. Survey data also indicated that at the 3-year mark, within the control group, 13 % had attended a residential program other than ChalleNGe. They had received an average of 9 months of high school classes and an average of 4.4 months of GED classes. Within the treatment group, they had received an average of 7.6 months of high school classes and an average of 4.8 months of GED classes (this may include both classes taken within the context of ChalleNGe as well as in other contexts). Notably, as previously mentioned, this study employs an intent-to-treat analysis, including all youth who were assigned to a given group during randomization regardless of actual participation in the program.
Results of the impact analyses indicated that participation in the ChalleNGe program led to significant improvements on educational and employment outcomes (see Table 2) 3 years later. Youth in the program group were significantly more likely than the control group to have obtained a high school diploma or GED (p < .01, effect size = 0.33), with 72 % of those in the program group having earned a GED or high school diploma, compared with 55 % of the control group.Footnote 13 Notably, this impact was driven largely by differences in GED receipt. The program group was also nearly twice as likely as the control group to have earned at least some college credit (p < .01, effect size = 0.34). Additionally, youth in the program group were more likely than the control group to have received vocational training since entering the study (p < .01, effect size = 0.14). However, there were no differences reported between the two groups with respect to receipt of a trade license or learning certificate.
Exposure to ChalleNGe led to better employment outcomes for those in the program group, compared to the control group, 3 years later. Specifically, the program group was employed for a greater number of months with higher earnings (p < .01, effect size = 0.20 and 0.16) than the control group in the past year. The program group reported earnings of approximately $13,500 in the previous 12 months, about $2,300 higher than the control group average of $11,250. At the time of survey, the program group was also more likely than the control group to be employed and earning an hourly wage of $10 or more (p < .05, effect size = 0.12). It is interesting to note, given the National Guard involvement in the program, that there were no significant differences in military enlistment between the two groups.
Looking across outcomes, at the time of the survey, the two groups also differed on whether they had a high school credential and were involved in another productive activity, such as employment, educational or residential programs, vocational training, or military enlistment (p < .01, effect size = 0.23). The program group was more likely than the control group to be living on their own, but rates of marriage or living with a partner did not differ.
5.4 Impacts for subgroups
Given that effects of other youth interventions have varied by certain subgroups, such as age in the JobCorps evaluation, we investigated whether ChalleNGe is more or less effective for specific groups of participants. Specifically, analyses examined impacts based on age, high school academic performance, whether a family member suggested ChalleNGe participation (a possible sign of family involvement), and previous involvement with the justice system. Among these, differences mainly emerged in looking at effects by participant age. The results presented in Table 3 suggest that the program may have slightly more positive effects, for both educational and employment outcomes, on older participants (those who enter at age 17 or 18) than on younger participants (those who enter at age 16). Significant differences in impacts by age group were observed for receiving a high school diploma or GED and earning college credit (p < .05), with older youth faring better on both variables. ChalleNGe appears to have reduced high school diploma receipt (p < .05) among the subgroup of younger sample members; some of them probably obtained a GED through the program when they otherwise would have obtained a regular diploma. Among older sample members, those in the program group were also more likely than the control group to have been employed in more months and earned more money (p < .05).
Given the relatively small number of young women included in the evaluation, this study was unable to test directly for differences in impacts between men and women. However, a sensitivity analysis removing the women from the sample resulted in a similar pattern of impacts to that of the full sample. In addition, small sample sizes within some sites limited investigation of differences in impacts across the ten sites. Moreover, the evaluation did not include a systematic cataloging of site characteristics as part of the implementation research, compromising our capacity to make site-level attributions.
The results indicate that the National Guard Youth ChalleNGe Program has significant positive effects on youth participants 3 years after entering the program. Specifically, compared to the control group, youth who participated in the ChalleNGe program earned more money and were more likely to obtain a high school diploma or GED, earn college credit, and be employed. These labor market outcomes differentiate ChalleNGe from some of the youth interventions mentioned earlier, such as JOBSTART and New Chance.
The results of this evaluation suggest that the skills, competencies, and credentials of vulnerable youth can be improved and sustained in the context of sufficient support and opportunities. By employing a PYD approach, the ChalleNGe program provides youth with an environment that allows them to gain such assets during these crucial transitional years. Once off-track, youth face serious challenges as they attempt to re-engage and become a participating, productive member of society. As such, those who, for a substantial period of time, are not in school, not working at a job, not in the military, or not married to someone connected to any of these institutions are far more likely during later adulthood to struggle with poverty, be on welfare, work inconsistently, and be unmarried (Danziger and Ratner 2010).
Moreover, in a twenty-first century labor market that increasingly prioritizes education and skills, young people who drop out of high school face greater challenges (Berlin et al. 2010). It appears that ChalleNGe can provide educational, occupational, and economic benefits that endure for 3 years. Particularly since post-secondary education is becoming increasingly necessary in the twenty-first century economy, the fact that program participants were significantly more likely than those in the control group to go on to receive college credit is promising. Given the high percentage of youth who received a GED certificate, continuing on to post-secondary education seems crucial. Studies have shown that GED holders earn significantly less than high school graduates (Tyler 2005).
Beyond the educational outcomes, it is encouraging that ChalleNGe was able to produce improvements in employment and income 3 years later. The employment rates—about half of the sample employed—mirror that of earlier follow-up survey responses. Even though respondents are getting older, this lack of an increase in employment over time is not surprising given the sharp rise in unemployment nationally that accompanied the most recent recession. In 2009, when the survey was administered, the unemployment rate for youth reached 15 %, the second-highest rate since the government began tracking these data.Footnote 14
Overall, ChalleNGe appeared to help youth move along the path in the transition to adulthood. The young people had received a high school credential and were involved in another productive activity to further themselves or contribute to society, whether it be further school, training, or employment. In addition, the young people were showing their independence by moving out of their parents’ homes and living on their own.
It should also be noted that not all of those assigned to the program group actually made it past registration and the pre-ChalleNGe phases to enroll in (68 %) and graduate from (53 %) the ChalleNGe program. The impacts of the program on the group of ChalleNGe graduates are thus likely to be larger than the impacts reported in this study. It is a notable accomplishment that, even with the use of intent-to-treat analyses, the program was able to achieve sustained effects on educational and employment outcomes within an at-risk population. Due to the variation in dosage of intervention received within the treatment group, it will be important for future research to explore the influence of dosage on program impacts.
However, as noted earlier, MDRC has presented earlier impacts for this study sample. Comparing the 3-year results with those from a survey administered at 21 months, there is a narrowing gap between the two groups on earning a high school diploma or GED (from 24 to 16 percentage points). While impacts on other educational outcomes did not measurably shift, the impact on employment at the time of the survey increased slightly (from 5 to 7 percentage points). Overall, larger percentages of the research sample had earned college credit and participated in vocational training at the 3-year mark, indicating the sample’s continued progress—at an average age of 20 years old—to continue improving towards adulthood. Given these changing ages and the relatively young age of participants, longer-term follow-up of the research sample could further tell the story of the transition to adulthood and the possible longer-term effects of the ChalleNGe program.
Subgroup analyses also provided valuable insight into variation of impacts among participants. Specifically, stronger educational impacts were observed for those entering the program when they were older (17 or 18 years old) than for those entering when they were younger (16 years old) for obtaining a GED or high school diploma and college credit. One reason for this difference in impact is that youth who are 16 years old may be more likely to return to high school after dropping out than youth who are 17 or 18 years old. In fact, within the younger group, significantly fewer youth in the treatment group than in the control group received a high school diploma, indicating that participation in ChalleNGe may have inadvertently discouraged high school attendance among younger youth by making it possible for them to earn a GED. It has been well-established that a high school diploma is of more value in the labor market than a GED. This impact could be viewed as a potentially negative unintended consequence of the program and suggests the need for caution in making opportunities for earning a GED available to youth who are still of an age when they could remain in or return to high school.Footnote 15
Differences in impacts between older and younger participants may also be influenced by the fact that older participants may be more mature and better able to take advantage of the opportunities offered by the ChalleNGe programs. The JobCorps evaluation similarly found evidence of greater effects for older participants. In addition, qualitative differences between students who leave school at 16 years old versus those who leave school at 17 or 18 years old may contribute to greater impacts among older participants. Generally, these results suggest the benefit of focusing second chance programs such as ChalleNGe on older youth who may have fewer options for obtaining high school credentials and who may approach the experience with greater seriousness and focus.
Although this study has a number of strengths, including the use of random assignment, data collected 3 years following entry into the ChalleNGe program, and a large, national sample, there are also a number of limitations. As noted earlier, differences between those who responded to surveys and those who did not respond may have biased the results and limit the generalizability of results beyond the survey sample. It is hard to know the direction of the bias, however, based on the differences in baseline characteristics between the survey respondent sample and the remaining random assignment sample. The survey sampling plan did draw sample members randomly, within the confines of the plan discussed earlier, so there are no selection bias issues. This sampling plan may limit generalizability in one sense, but the plan and the pooled, weighted impact estimates attempt to more equally represent the different programs in the study.
Similarly, the study could not include all youth who applied to these programs during the random assignment period. As discussed, some cohorts did not include enough applicants to allow random assignment to occur at a site. The weighting scheme utilized in the pooled analysis—which gave equal weight to each site—helps to account for these potential sampling variations. Specifically, programs that fared better with recruitment, and included more youth in the sample, do not overwhelm the smaller program samples in the analysis of outcomes. In addition, the study relied solely on self-reported measures of outcomes, which are influenced by individual bias and subjectivity.
Program effects also may have been inflated by the fact that participants in the study were drawn from ChalleNGe sites with stable staffing that received more applicants than they could serve, possibly indicating more successful programs. In addition, the ability to analyze and understand site-level differences in this evaluation was limited due to sample sizes and the implementation research involved. The analysis pooled results across sites to create the best overall estimate possible of ChalleNGe, using weights and regression adjustment. However, while technically the pooled results do not represent a national impact of ChalleNGe, the study sites do look like other ChalleNGe sites based on the available national performance data. Further implementation research to catalog the site differences and interpret site-level variation could be helpful for program administrators, funders, and policymakers.
Moreover, although it is significant that this study was able to collect data on participants more than 3 years after entry into the study, the field would benefit from studies that follow participants for a longer period of time. Such studies would help to determine whether improved educational and occupational outcomes are sustained throughout the complicated transition to adulthood and beyond. Finally, although the ChalleNGe program serves a vulnerable group of youth by virtue of their status as high school dropouts, due to the voluntary and intensive nature of the intervention, it tends to attract a youth who are particularly motivated to make change, limiting the external validity of the study. Specifically, the results cannot be generalized beyond the youth who would be interested and eligible in the ChalleNGe program.
Despite these limitations, this study significantly adds to the body of knowledge about youth interventions that can have enduring effects across key years in the transition to adulthood among high school dropouts. The young people in this study were off-track, with dwindling opportunities to re-engage and become participating, productive members of society. Rather than target their deficits and problematic behaviors, ChalleNGe utilized a PYD approach to “reclaim the lives of at-risk youth” and 3 years later the evidence remains in the employment and education outcomes discussed here. More broadly, the findings of this evaluation suggest that an emphasis on PYD may be particularly useful for youth who have dropped out of high school. The results of this study provide hope that ChalleNGe can offer some youth a second chance to become productive members of our society.
Note that recently, some states have begun using tests other than the GED as a high school credential, such as HiSET and TSAC. The GED is used as shorthand throughout this paper and is the test that was available to sample members during the time of this study.
One oft-cited longitudinal study of an 1988 eighth-grade cohort by the National Center for Education Statistics found that 63 % of high school dropouts had gone on to earn a high school credential (received their diploma or earned a GED) 8 years after their scheduled high school graduation date (http://nces.ed.gov/pubs2005/2005026.pdf).
Adapted from the National Guard Youth ChalleNGe Program’s mission statement.
As part of the quasi-military approach, cadets are divided into platoons and squads, live in barracks, have their hair cut short, wear uniforms, and are subject to military-style discipline.
For a more detailed description of the ChalleNGe program, refer to Bloom et al. (2009), Millenky et al. (2011), or the Challenge website, www.ngycp.org. Overall, fidelity to the intervention model during the Residential Phase was relatively high across participating sites. However, delivery of the intervention during the Post-Residential Phase was more uneven.
Most ChalleNGe programs have recruiters who travel around the state to discuss the program with interested young people, parents, school principals, and other youth professionals.
For the duration of the study, those interested in applying to ChalleNGe needed to agree to also take part in the study in order to continue with the enrollment process. However, the study procedures allowed for a small number of hardship cases to bypass study involvement in special circumstances. For applicants under 18 years of age, their parents or guardians signed consent forms during the application process as well.
A greater number of youth were assigned to the program group than the control group because the primary goal was to fill the number of available program slots. During the study period, program managers told MDRC how many applicants they needed to accept in order to meet the graduation target, assuming normal patterns of attrition. Random assignment was conducted if the number of qualified applicants was at least 25 greater than the number needed to meet the graduation goal.
The study has a slightly greater percentage of males than the overall percentage for ChalleNGe programs due to the fact that some programs needed to accept all female applicants into the program for logistical reasons and therefore could not include them in the random assignment pool. While ChalleNGe serves youth aged 16–18, youth who were younger than 16.5 years of age were excluded from the study. The Department of Defense made the decision to exclude the youngest applicants from random assignment (though not from the programs) in order to reduce the number of young people who, if they were assigned to the control group, would be barred from reapplying for ChalleNGe for several class cycles. Owing to this rule, the characteristics of the participants in the study do not necessarily match those of all the young people who participated in the programs during the cycles when random assignment occurred.
Baseline covariates in the model included age, gender, race, highest grade completed, if the participant lived in a two-parent household, and whether the participant applied to ChalleNGe because of interest in the military.
Interestingly, 15 % of the program group reported having both a high school diploma and a GED certificate. This is unusual, since a GED is usually seen as an alternative to a high school diploma. These program group members were concentrated in a few ChalleNGe programs with special arrangements. For example, one is an alternative high school and another has an arrangement with the state legislature that allows young people who complete ChalleNGe and pass the GED exam to receive a state high school diploma.
The 15 % unemployment rate is representative of young people ages 20 to 24 in the USA in 2009. On average, survey respondents were 20 years old. In the same year, the unemployment rate for 18- and 19-year-olds was higher at 23 % (U.S. Bureau of Labor Statistics, 2010).
Heckman cites research that argues more broadly that the availability of the GED causes some youth to choose this option over completing high school and receiving a more traditional diploma (Heckman et al. 2010).
Center for Strategic and International Studies
General Educational Development
Management Information System
National Guard Youth ChalleNGe Program
positive youth development
Berlin G, Furstenberg F, Waters MC (2010) The transition to adulthood: introducing the issue. Futur Child 20:3–18
Bloom D (2010) Programs and policies to assist high school dropouts in the transition to adulthood. Futur Child 20:89–108
Bloom D, Gardenhire-Crooks A, Mandsager C (2009) Reengaging high school dropouts: early results of the National Guard Youth ChalleNGe Program evaluation. MDRC, New York
Cave G, Bos H, Doolittle F, Toussaint C (1993) JOBSTART: final report on a program for school dropouts. MDRC, New York
Cullinan M, Eaves JK, McCurdy D, McCain J (1992) Forging a military youth corps: a military-youth service partnership for high school dropouts. The final report of the CSIS National Community Service for Out-of-School Youth Project. Center for Strategic and International Studies, Washington, DC
Danziger S, Ratner D (2010) Labor market outcomes and the transition to adulthood. Futur Child 20:133–158
Godley SH, Godley MD, Dennis ML (2001) The assertive aftercare protocol for adolescent substance abusers. In: Wagner E, Waldron H (eds) Innovations in adolescent substance abuse interventions. Elsevier Science Ltd., New York, pp 311–329
Heckman, James J., Humphries, John Eric, & Mader, Nicholas S. (2010). The GED. NBER Working Paper No. 16064.
Larson RW (2000) Toward a psychology of positive youth development. Am Psychol 55:170–183
Lerner RM (2005). Promoting positive youth development: theoretical and empirical bases. White paper prepared for Workshop on the Science of Adolescent Health and Development. Retrieved June 1, 2011 from ase.tufts.edu/iaryd/documents/pubPromotingPositive.pdf.
Millenky M, Bloom D, Dillon C (2010) Making the transition: interim results of the National Guard Youth ChalleNGe evaluation. MDRC, New York
Millenky M, Bloom D, Muller-Ravett S, Broadus J (2011) Three-year results of the National Guard Youth ChalleNGe evaluation. MDRC, New York
National Guard Bureau (2010) An assessment of the Youth ChalleNGe Program: 2010 performance and accountability highlights. AOC Solutions, Inc., Chantilly
Northeastern University Center for Labor Market Studies (2009) Executive summary—consequences of dropping out of high school: joblessness and jailing for high school dropouts & the high cost for taxpayers. Northeastern University, Boston
Orr LL, Bloom HS, Bell SH, Doolittle F, Lin W, Cave G (1997) Does training for the disadvantaged work? Evidence from the National JTPA study. Abt Associates, Cambridge
Quint JC, Bos JM, Polit DF (1997) New chance: final report on a comprehensive program for young mothers in poverty and their children. MDRC, New York
Rodríguez-Planas, Núria (2010). Mentoring, Educational Services, and Economic Incentives: Longer-Term. Evidence on Risky Behaviors from a Randomized Trial. IZA Discussion Paper No. 4968.
Roth JL, Brooks-Gunn J (2003) What exactly is a youth development program? Answers from research and practice. Appl Dev Sci 7:94–111
Schochet PZ, McConnell S, Burghardt J (2003) National Job Corps study: findings using administrative earnings records data. Final report. Mathematica Policy Research, Princeton
Schochet PZ, Burghardt J, McConnell S (2006) National Job Corps study and longer-term follow-up study: impact and benefit-cost findings using survey and summary earnings records data. Mathematica Policy Research, Princeton
Tyler J (2005) The general education development credential: history, current research, and directions for policy and practice. In: Comings J, Garner B, Smith C (eds) Review of adult learning and literacy, volume 5. Lawrence Erlbaum Associates, Boston, pp 45–84
U.S. Bureau of Labor Statistics. 2009 Annual Averages — Household Data — Tables from Employment and Earning: Employment status of the Civilian Noninstitutional Population by Age, Sex, and Race. 2010. http://www.bls.gov/cps/aa2009/cpsaat5.pdf.
U.S. Bureau of Labor Statistics. Labor force status of 2013 high school graduates and 2012-13 high school dropouts 16 to 24 years old by school enrollment, educational attainment, sex, race, and Hispanic or Latino ethnicity, October 2014. http://www.bls.gov/news.release/hsgec.t01.htm.
The author gratefully acknowledges the study’s Project Director, Dan Bloom, and other members of MDRC’s ChalleNGe evaluation team, the participating ChalleNGe programs and their staff, the youth who participated in the study, and the support of the Bill & Melinda Gates Foundation, Charles Stewart Mott Foundation, Edna McConnell Clark Foundation, John D. and Catherine T. MacArthur Foundation, MCJ Foundation, Robert Wood Johnson Foundation, William and Flora Hewlett Foundation, and the US Department of Defense.
Responsible editor: David Neumark
The IZA Journal of Labor Policy is committed to the IZA Guiding Principles of Research Integrity. The author declares that she has observed these principles.
Megan Millenky is a Research Associate in MDRC’s Health and Barriers to Employment policy area. Millenky’s current work focuses on evaluations of programs for at-risk young people. Prior to coming to MDRC, Millenky received a master’s degree in public policy from Georgetown University. While at Georgetown, she worked at the US Government Accountability Office and the Center on Budget and Policy Priorities.
About this article
Cite this article
Millenky, M. Connecting high school dropouts to employment and education: an impact study of the National Guard Youth ChalleNGe Program. IZA J Labor Policy 5, 10 (2016). https://doi.org/10.1186/s40173-016-0065-x