1. Introduction

Primary schools are considered a key intervention context through which Physical Activity (PA) levels can be increased in order to prevent childhood obesity and sedentary behavior related diseases [1]. Through sports, Physical Education (PE) and play, children of all ages are encouraged to participate in PA. Yet the risk of related injury is often neglected in these efforts to promote daily PA.

In children, PA promotion is usually focused on leisure time PA. Although the number of injuries expressed per 1,000 hours of physical activity associated with these activities is lower than for organized PA (i.e. PE and sports), absolute numbers as well as costs associated with these injuries are significantly higher [2, 3]. Also, leisure time PA is related to a relatively large number of forearm fractures [3]. Overall forearm fractures account for up to one third of childhood fractures [4], and about 40% to 50% of children sustain at least one fracture during childhood5. Also given a steady increase in the incidence of forearm fractures over the last 30 years [4], it is evident that preventive measures are warranted.

Currently, little is known about the mechanisms underlying the observed increase in forearm fractures. Regarding the injury mechanism, it has been described that 80% of forearm fractures occur after a fall [6]. Since falling is an inescapable part of a child’s daily play and PA, it is unlikely that falls can be completely avoided. The injury resulting from a fall could, however, be reduced when children are trained to properly react when falling. Martial arts falling techniques have, for example, shown to decrease forearm ground reaction forces [7, 8] thereby potentially reducing the risk of sustaining a forearm fracture [7]. In the Netherlands, an educational program, consisting of exercises based on martial arts techniques, was developed to improve fall skills in children. In this study, the process of implementation was evaluated.

This process evaluation, alongside a cluster-RCT, was performed to measure the extent to which the intervention was delivered and received as intended. Such information is not only essential to understand how the intervention worked within it’s intended context, but also to prevent a Type III error. This type of error occurs when researchers conclude that an intervention was not effective when, in fact, the lack of significant study findings was due to inferior implementation and not due to the design or efficacy of the intervention itself [9].

In the current process evaluation, the RE-AIM framework [10] was used. In this framework five dimensions of intervention quality are described; 1) Reach of the target population, 2) Effectiveness of the intervention, 3) Adoption of the program by target settings, 4) Implementation and 5) Maintenance of behaviour change. Exploring these different dimensions of intervention quality will give a better understanding of the strengths and limitations of the program and measure the extent to which the intervention was delivered as intended.

2. Methods

2.1. Design and Participants

The study design and participants have been described in detail elsewhere [11]. In short, the effectiveness of the intervention program to improve fall skills was evaluated in a cluster-RCT during the school year 2009-2010. Participating Dutch primary schools (n = 36) were randomly allocated to an intervention or control condition. In the beginning of October 2009, children in the intervention group (n=1,397) started receiving an eight-week intervention program as part of their PE classes. Children in the control group (n=1,920) received their regular PE classes. Participating children were recruited from classes 5 – 8 (age: 7-12 years). Parents of the participating children received a passive, informed consent form which explained the nature and procedures of the study. If either parent or child did not want to participate, they could withdraw from the study at any moment. This study was registered in the Netherlands Trial Register (trial code=2031). The intervention was implemented by the Dutch Consumer Safety Institute. The study was performed alongside this implementation and therefore no ethical approval was obtained. To secure privacy, this study was registered at the Dutch Data Protection Authority.

2.2. The Intervention Program

The fall-related injury prevention program revolved around martial arts (judo) techniques that taught children to distribute the impact energy associated with a fall over a larger contact area, and to convert the fall into a rolling motion [12]. The most important aspect of the lessons was to avoid blocking a fall with extended elbows. Furthermore, children were taught to keep their head from touching the ground and to slap the ground hard with their hand to stop the rolling of the body.

Exercises were taught in an 8-week program, consisting of a weekly 1-hour lesson, increasing in difficulty during the program. Intervention lessons replaced regular PE classes. Teachers were instructed through a written manual, an instruction DVD and instruction cards designed to support the children during the exercises. Furthermore, a booklet with background information was part of the instruction material. The practice ball that was used in some of the exercises was also provided.

2.3. The RE-AIM Framework

We used the RE-AIM framework to evaluate the process of implementation of the educational program. The definitions used are displayed in table 1.

Table 1. Description of the Used Definitions of the Five Components of the RE-AIM Framework [10]

2.3.1. The Questionnaires

An overview of the different components of this process evaluation is provided in table 2. For the process evaluation, three types of questionnaires were used:

a questionnaire on demographic details, PA behavior and self-perceived fall skills;

a process evaluation questionnaire;

continuous injury registration during the school-year (when a child sustained an injury).

Table 2. Description of the Outcome Measures Registered in the Process Evaluation during the Study

The questionnaire on demographics, PA behavior as well as the continuous injury registration have been described in detail elsewhere [11].

Self-perceived fall skills were assessed with 2 questionnaire items using a 5 point likert scale: 1) Do you think you can keep your arms close to your body when you fall? And; 2) Do you think you can keep your head from the ground when you fall?

The process evaluation questionnaire included 1) the extent to which children were exposed to the intervention, reported by both children and teachers; 2) the extent to which the exercises were implemented as intended; 3) the extent to which exercises were repeated after the eight week program was finished; and 4) the attitude of both teachers and children towards the educational program.

2.3.2. The Fall Skills Test

To gain insight in the actual performance of taught fall skills, a fall skills test was performed in a convenience subsample of children from the intervention and the control group at baseline, follow-up 1 and follow-up 2. Eight schools located in the proximity of Amsterdam were approached for this test (4 control, 4 intervention). One class from each school was selected such that all grades were accounted for (grades 5, 6, 7 and 8). The intervention grade 8 only comprised 12 children, therefore an intervention 8th grade from another school was included to obtain a comparable number of pupils in each age category.

The items of the fall skills test were derived from the instruction material and comprised of four basic falls. Children were asked to 1) fall sideways while seated; 2) fall sideways while kneeled; 3) fall backwards while seated; and 4) fall backwards while crouched. Falls were executed on a crash mat. Each fall was scored with ‘yes’ (=1 point) or ‘no’ (= 0 point) for three subcomponents. These subcomponents were ‘breaking the fall with arms stretched’, ‘touch the crash mat with head’ and ‘slapping the crash mat to stop rolling motion’. Since the score for the slapping motion was inverse to the other scores, this score was inverted before summing all scores. A high sum score indicated an observed bad falling technique.

2.4. Statistical Analysis

Before starting further analysis, baseline variables of the control and the intervention group were analysed for differences by an independent samples t-test for continuous variables, and χ2 statistics for categorical variables. These analysis were performed using SPSS. Differences (i.e. odds ratios) between the intervention and control group in the probability of injury and the severity of injuries (yes/no) were calculated using a mixed logistic regression model with random intercept. Intra cluster correlations (ICC) will be given. A generalized linear mixed model (the Gllamm procedure in Stata 11.2 (StataCorp, Chicago, Illinois)) [13-15] was used for the computations.

Both responses to self-reported fall skills and the fall skills test were analysed using MLWin multiple three level linear regression models. For analysis of the self-perceived knowledge of fall skills questions, change scores were calculated between baseline and follow-up 1 and follow-up 2. Baseline differences between both study groups were considered as potential confounders and therefore included in all fully adjusted models.

3. Results

3.1. Reach

PE education is obligatory on Dutch primary schools. Since all children who partook in PE classes were exposed to the educational program, reach was approximately 100%. Over 95% of children remembered executing the exercises. To determine if the participants in the study were representative for all children in the Netherlands, comparisons were made between characteristics of the study population and population data of 9 to 15 year olds in the Netherlands (ref CBS). No differences were observed regarding gender (51% boys) and ethnicity (85% western).

3.2. Effectiveness

A cluster RCT was conducted to evaluate the effectiveness of the educational program. The results regarding injury incidence have been described elsewhere [11]. In summary, unadjusted logistic regression analyses showed a significant decrease in the number of fall-related injuries in the intervention group (OR 0.50, 95% CI (0.34 to 0.74)). However, after adjustment for clustering and correction for interaction by baseline level of PA results failed to reach significance (OR 0.66, 95% CI 0.17–2.55). A trend (p=0.087) was found suggesting that the educational program was more effective for children with low levels of habitual PA. No significant differences were reported for injury severity.

Within the same cluster RCT, changes in self-reported fall skills were assessed (Table 3). At follow-up 1 children in the intervention group showed improved self-reported ability on both questions. This effect remained statistically significant at follow-up 2 for self-reported ability to keep the hands close to the body.

Table 3. Mean Change in Self-perceived Fall Skills in 7-12 Year Old Children After Participation in an Intervention Program to Improve Fall Skills

In regards to the fall skills test no differences were found between the control and intervention groups at baseline. Compared to control group, test scores had significantly improved in the intervention group at both follow-up 1 and follow-up 2(Table 4).

Table 4. Mean Fall Skills Test Scores of 7-12 Year Old Children

3.3. Adoption

Schools were recruited from a waiting list of a pilot project. This pilot was undertaken to ascertain whether schools were interested to participate in improving fall skills. The registration for this pilot project exceeded expectations; 250 (out of little under 7000) Dutch primary schools could not participate and were approached for this study. Thirty-six (14%) of these 250 schools were willing to participate.

After randomization, three schools dropped out from the intervention group. One school dropped out before the start of the study because the PE teacher changed job, one school failed to return the baseline questionnaires, and one school was lost to follow-up because of long-term absence of the contact person. Thirty-three schools (16 control, 17 intervention) completed the study.

3.4. Implementation

A total of 28 (72%) questionnaires was returned by the teachers. Each intervention school returned at least one questionnaire (Table 5). Overall, teachers appreciated the educational program with an average of 7.3 on a scale of 1 to 10. The appreciation of the teachers of the separate instruction materials is displayed in table 5. All users reported to have used the instruction cards, while most of them (86%) had used the written manual. Additional materials were used by about half (range: 21% - 68%) of the teachers; those materials were informative and not necessary to execute the lessons. Teachers reported to require an average of 17 minutes (SD = 12 minutes) to prepare for the lessons.

Table 5. Results of Teacher Reports (n=28) on Appreciation and Use of the Material Developed for the Intervention Program to Improve Fall Skills

The difficulty of the fall skills exercises was rated ‘good’ for grades 5 and 6 (7-9 years of age) and ‘easy-good’ for grades 7 and 8 (10-12 years of age). A number of specific exercises was not executed by the teachers (16 teachers from 13 schools). These exercises included the practice ball (6 teachers from 5 schools) and falling from rings and from wall bars (7 teachers from6 schools). Reasons to exclude these exercises were that there were no facilities (2 teachers from 2 schools), that there were too many pupils in a group to execute the exercises (2 teachers from 1 school) and too little time to prepare children for the exercises (5 teachers from 5 schools). Seventeen teachers (from 10 schools) reported that they sometimes had changed the exercises.

Eight teachers (from 7 schools) reported to have executed all 8 lessons. On average 5.3 out of the total 8 lessons was executed. Nine teachers reported that they did not have enough time to perform all lessons, 11 teachers indicated that 8 lessons were too much and 5 teachers reported that some of the PE classes were cancelled. Two out of the 8 teachers that executed all 8 lessons reported to have executed only a part of each lessons.

Experiences of children with the educational program were rather good; an average score of 7.8 points on a 1 to 10 scale. Fifteen percent of the children reported that the program was ‘boring’. The difficulty of the program was rated as easy by 96% of pupils. When asked if they thought they had learned how to fall without being hurt, 90% of pupils agreed.

3.5. Maintenance

Twenty six teachers (67%) from 14 schools returned the follow-up 2 questionnaire, 10 teachers (from 8 schools) had reported repetition of the exercises after completion of the 8 week program. When asked about future use, 54% of the teachers (from 10 schools) had reported that they were planning to use the program in the future. Reported potential problems for national implementation of the educational program were; difficulties with fitting the educational program into the existing annual PE program (10 teachers from 10 schools), too many lessons (n=15 teachers from 10 schools), and the low intensity of the exercises (n=4 teachers from 4 schools).

4. Discussion

The first goal of the current process evaluation was to identify strengths and limitations of the educational program. A major strength of the program is that all children that partake in PE-classes are reached. This includes those children that have reported to be at increased injury risk, namely those with low levels of habitual PA. Another strength is that both teachers and children were generally very positive in their rating about the educational program. The outcome that the majority (96%) of the children rated the exercises as ‘easy’ is a possible strength, meaning that most children could execute the exercises. This finding is, on the other hand, a limitation since this could also mean that the exercises were not challenging enough. The latter might indeed be true for children aged 10-12 years; for this age groups teachers rated the exercises as ‚easy-good’ as well. The ratings of the children are, of course, dependent on the extent to which the intervention was received as intended.

The second goal of this process evaluation was to measure to which extend the intervention was delivered as intended. Adoption of the educational program was low: On average, 5.3 out of 8 lessons was executed and 89% of the teachers failed to implement the program as intended. The responses of the teachers suggested that the program was less suitable for current Dutch PE classes, where different exercises are usually offered in a circuit form. Other teachers suggested that the exercises should be spread over an entire school year to keep children interested in the program, instead of over 8 consecutive weeks.

One of the reasons to use the RE-AIM framework to assess the educational program, was to rule out a type III error. It is a fact that no decreased injury risk could be reported even though fall skills did improve. This would suggest that the type III error could be rejected. The reported limitations of the program in combination with the small, yet significant, changes in fall skills makes us to conclude that a type III error cannot be completely ruled out.

4.1. Limitations

The current process evaluation is of course subject to bias. One possible source of bias in the current study lies in the recruitment procedure. School were recruited from a waiting list for fall-clinics that had received national press coverage. As such, we had recruited more directly from a pool of schools that was already interested in the topic. This has likely facilitated recruitment, but may also have biased the outcomes since the intervention was implemented in schools with an interest in fall-injury prevention. It could be argued, however, that this is a common phenomenon in implementation research where mainly motivated participants agree to participate [16]. As such, we believe that selection bias may have been minimal and that results of the intervention program can be implemented broadly.

Another limitation of this process evaluation is that most results were based on self-reports. This can have resulted in over-reporting of compliance and in giving socially desirable answers. However, if this has been the case, the results should be regarded as promising since even without teachers fully complying, changes were reported in fall skills. Focus group interviews with teachers and children might be helpful to improve compliance and to guide further development and implementation of the educational program.

5. Conclusions

The intervention evaluated in the current study is, to our knowledge, the first educational program that aimed to increase fall skills in 7-12 year old children. Results of the process evaluation using the RE-AIM framework are promising: Even though adoption of the program was lower than expected, the fall skills of children were significantly improved. For future use, changes to the program are advised to improve adherence of teachers. Shorter lessons and a spread of lessons over a school-year instead of eight consecutive weeks could make it easier for teachers to adopt the program in their teaching routine.

ACKNOWLEDGEMENTS

We would like to thank Myrthe Bartelink, Mariëtte Geuke and Tessa Könemann for their administrative support, and Yos Lotens and 2Basics for developing the educational program ‘Vallen is ook een sport’.