Evaluation Gets Results!
Number 21 - December 2000
L. Debuhr & B. Adelson
In 1993, the education division at the Missouri Botanical Garden initiated the Natural Science Institute for Elementary Teachers. This teacher training programme was developed to help the education division test a training protocol which we hoped would provide biological content enhancement and upgrade teachers’ pedagogical skills and leadership abilities. The purpose of this case study is to describe the process used to evaluate the Natural Science Institute for Elementary Teachers and present some of the results from the process. It is not the goal of this report to go into a detailed description of the results of the evaluation, but to feature the highlights. The consequences of the evaluation for education programmes at the Missouri Botanical Garden will also be discussed.
The Natural Science Institute for Elementary Teachers was started as a pilot project in 1992, when three teachers were hired during the summer to organise and teach a special summer programme for children. These teachers were paid for six weeks of work - two weeks for planning and preparation, three weeks for the delivery of classes to children, and one week to conclude the programme by preparing a brief report. Several months later, one of the teachers reported that the summer experience had been very helpful to him, in that he had incorporated many of the same lessons from the summer classes in his own teaching for the first time. He also mentioned that the opportunity to work with other teachers and interact with them as they planned and carried out the programme was a valuable experience.
Essentially, the pilot project started in 1992 and the results and comments from the teachers hired to conduct the programme provided the first evaluative data for this case study, and represented a type of formative evaluation that was critical to the success of the project. The formative information from the pilot study significantly influenced the nature of the project as the Natural Science Institute was designed. A proposal was submitted to a major science foundation in the United States in 1993 to develop the Natural Science Institute, and the project was funded for 4 years as a result of that proposal.
Description of the Natural Science Institute for Elementary Teachers
The target audience for the Natural Science Institute for Elementary Teachers was classroom teachers in grades K-5 who worked predominantly in schools with high percentages of students from economically disadvantaged backgrounds or ethnic minorities. National research indicates that these schools are in more critical need for improved science teaching and better classroom practices. The content focus of the Natural Science Institute was on organismal and ecological biology concepts, and conformed to the U.S. National Science Standards (National Research Council 1996).
The project had three components. The first component was a series of five workshops, initially for nine days and later ten days in length. The purpose of the workshops was to introduce organismal and ecological biology concepts to the participants. Garden staff used investigative and hands-on lessons to teach these concepts. The second component was a summer science programme for children taught by the participating teachers. The purpose of this summer programme was to allow the teachers the opportunity to work in teams to organise and teach many of these science concepts using the same lessons and activities they learned in their own training, and to receive support and guidance from each other as well as from the garden education staff. This reflective practice component was thought to be critical for helping the teachers incorporate new ideas and new lessons into their own classrooms. The final component of the programme was that the teachers were required to return to their classrooms and begin incorporating more organismal and ecological biology into their own teaching. In addition they were to serve as a resource to other teachers in their schools.
Were We Hitting the Target Audience?
The first objective of the programme was to provide training for teachers from schools with high percentages of students from economically disadvantaged backgrounds or ethnic minorities. The question we were most interested in answering was: were we hitting the target audience? The method used to evaluate this outcome was to collect demographic data on the students enrolled in the classes for which the teachers were responsible. Over the four years of the project, 59 teachers completed the Natural Science Institute. A total of 40 of these teachers taught in schools with more that 50% ethnic minority students, and 28 teachers were in schools with 80% or more of the students from ethnic minorities. Altogether, 31 teachers taught in schools with more than 50% of the students from economically disadvantaged families, with 23 of those in schools with 80% or more of the students economically disadvantaged. From these results, we were generally confident that we were hitting our target audiences without excluding participation by teachers from other less disadvantaged schools.
Did the Training Session Help Teachers Learn?
The second objective of the project was to ensure that participating teachers were more knowledgeable about science concepts and science teaching practices after undergoing training. Two questions were asked: how effective were the individual training sessions? and did teachers’ levels of science knowledge increase? A series of five workshops were scheduled with the participants between January and May. These workshops focused on organismal and ecological science concepts, and used hands-on, investigative activities to teach these concepts to the participants.
The first question above was answered with short evaluation surveys given to the participants at the end of each session, and a longer summative evaluation at the end of the entire training programme. The basic questions on the survey were:
- Was the science content information adequate and organised in a useful manner?
- Were the activities well prepared and presented?
- Which activities do you expect to be most useful to you in the classroom?
- Which were of least use to you and why?
- Are we generally meeting your expectations? If not, why not?
- How can we improve the delivery of the programme?
The number of individual comments made by teachers over the four years of the project are too numerous to mention. The importance of the survey to us was that the feedback from the participants allowed us to identify weaknesses and strengths in the training and to make changes in future sessions and future years. As the project progressed each year, we were able to identify science concepts that teachers had difficulty understanding (such as adaptation), and we could schedule more instructional time with these concepts. We were also able to make some changes in the format of the training as a result of the evaluation. For example, the number of days of the workshops was increased from nine to ten each year and the organisation of the concepts, and the sequence in which they were introduced during the sessions, was changed in response to teachers' comments.
The second question (whether the teachers’ levels of science knowledge increased) was answered with content pre-tests given to participants on the very first day of the training sessions, and a post-test given on the last day of the training programme. The results of the pre-test and post-test were analysed using paired t-test protocols. We were disappointed in the first year of the project when the mean difference between the pre-test scores and the post-test scores was only 7 percentage points. Although this was a statistically significant difference (p < 0.05) our sense was that the participants learned a great deal more than reflected in the post-test scores. Therefore, we re-examined the pre-test and post-tests and discovered that the exam questions did not represent the content of the course very well. The pre- and post tests were rewritten for subsequent years to be more closely aligned with the curriculum of the training sessions. During the second, third, and fourth years of the project, the mean difference between pre-test and post-test scores were 15.5, 17.0, and 18.4 percentage points, respectively. We felt more confident that the participants’ science knowledge increased as a result of the project.
One final evaluation method used to assess this outcome was a summative survey at the very end of the training session. The purpose of the summative evaluation was to collect additional data about the training programme as a whole, rather than about each individual session. The questions asked of the participants were:
- Do you feel your content knowledge in ecology and organismal biology increased as a result of this programme? - The answer was an overwhelming yes, confirmed by pre-test and post-test scores.
- Which content areas were of greatest benefit? - There was no clear area that seemed to be more beneficial than others.
- How can we improve the delivery of this aspect of the programme? - The one important comment that came from this question was the desire for us to build in more time for the participants to process information and to think about the content in groups. The number of days for the training sessions was increased.
- What activities were of greatest use? - Many different activities were mentioned by the participants, and no specific activity was mentioned in a significant way. It was clear that those activities which were most active and hands-on were preferred over activities that were more passive.
Does Reflective Practice Work?
The third objective of the project was to test the effectiveness of the reflective practice model used in the teacher training. The basic questions we asked were: did the reflective practice help? and were participants teaching different concepts and using new lessons and activities?
To answer the first question, a summative evaluation was administered at the end of the summer reflective practice component of the programme. Teachers were asked to assess how valuable the summer science programme was to them as a way to try out new science activities. The overwhelming response to this question was that the opportunity to teach in the summer science class was the most important part of the programme. We also asked them how we could improve their preparation for the summer programme? As a result of this question, we increased the planning time given to the teachers prior to the summer programme from one week to two weeks. Essentially, the results of the evaluation were responsible for fundamental changes in the structure of the programme.
The method used to answer the second question was to use written surveys and personal interviews with participants. The teachers were asked to list concepts and specific lessons and activities from the programme that they now include in their own classroom that they had not taught before. Although the specific concepts and lessons used by the participants varied from one teacher to the next, each participant reported that they taught an average of 5-6 new science concepts in their classrooms, and use an average of five new activities and lessons from the programme.
Were Teachers Helping Other Teachers?
The final outcome of the programme was to create a cadre of teachers who could serve as resources for other teachers. The fundamental question we asked was: were teachers being used as informal and formal resources for the classroom? The method we used to answer this question was to ask the participants to maintain activity logs. For one year after the teachers participated in the training sessions, they were asked to keep a log of the formal and informal activities that they had conducted to help other teachers. They were then asked to give these logs to the project staff at regular intervals so that the project staff could calculate the number of times that the participants had served as a resource and determine the nature of the assistance that they gave. Examinations of these logs indicated that the participants were conducting a minimum of 2-3 formal workshops each year for other teachers in their schools and were used an average of ten times a year as an informal resource. In the latter capacity, the teachers were being asked to serve on curriculum committees, to recommend lessons and activities that other teachers could use, to provide assistance in obtaining materials, to organise building science classrooms, and to help select future participants in the programme.
Some Generalisations from the Evaluation
Because of the success of the reflective practice approach, the Missouri Botanical Garden will continue to use the basic training model in future efforts. This represents a significant change in the way we view long-term teacher development. Our optimism about the structure of the training as a model for reflective practice has been reinforced. Feedback from participants during the entire project indicates that the combination of training and supervised practice was very helpful for teachers prior to their incorporating some of the concepts and lessons into their own classrooms.
Teachers participating in the programme have gained new knowledge and skills in organismal and ecological biology and are incorporating more science lessons into their own classrooms. They are also more knowledgeable about ways to teach these topics in an interactive and investigative manner. By extension, the children in the classes taught by the participating teachers will benefit from the teachers’ new knowledge and skills. The reflective practice approach does appear to be working in changing what teachers do in the classroom.
The confidence level of the teachers in science changed. At the beginning of the teacher training sessions, the participants expressed a high level of anxiety and concern about the job they would be asked to do. By the end of the year, those same teachers were able to successfully conduct in-service programmes for other teachers and to teach science to children without hesitation. All participating teachers showed a high level of commitment to the programme and involvement in the project.
Consequences of the Evaluation
In addition to the impact the evaluation of the Natural Science Institute had on how the education division operates teacher training programmes, the evaluation has also resulted in a number of consequences for the Missouri Botanical Garden. First, the success of the project as documented in the evaluation, led to continued funding of the programme after the initial four years. When the initial four year grant was completed, the garden was asked to continue the training programme for the St. Louis Public School District. The school district agreed to pay for the programme from their own operating budget. It is rare that a project funded from an external grant is later built into a school district funding, and the success in this case is a result of changes made in the programme based upon evaluative data and the documentation revealing that the programme was effective in achieving its goals.
A second major consequence of the project evaluation, is that it led to a second grant from the same agency that funded the Natural Science Institute initially. This second programme funded by this foundation had many of those characteristics of the first programme that were shown to be effective. It is clear that a successful project, as determined by an evaluation component, will lead more easily to additional support.
Finally, confirmation that the reflective practice component of the programme is effective in helping to achieve changes in the classroom, has resulted in a change in philosophy about teacher training at the Missouri Botanical Garden. We have reduced the number of one and two day short-term workshops in favor of longer-term training programmes that include some kind of reflective practice component.
National Research Council. (1996) National Science Education Standards. National Academy Press, Washington, D.C. USA.
L’Evaluation Donne Ses Résultats
Dans cet article sont décrits les conséquences et résultats d’une large évaluation menée par l’Institut des Sciences pour les enseignants du primaire du Jardin Botanique de Missouri. Ce projet de formation des enseignants a utilisé un mélange d’évaluations formatives et récapitulatives, de pré et post tests de connaissances, d’enquêtes, d’entretiens personnels, et de compte-rendus d’activités a permis d’arriver à quatre conclusions pour ce projet et de répondre à une série de questions spécifiques. L’évaluation démontré que le projet était attractif pour le public visé, que les connaissances scientifiques des participants étaient en augmentation, que le protocole choisi permettait une meilleure incorporation en classe des compétences et connaissances acquises et que les participants au programme deviennent eux-même des personnes ressources pour les autres enseignants. L’évaluation a aussi permis de recevoir des fonds supplémentaires pour des programmes de formation des enseignants et à un changement fondamental de la nature des programmes pédagogiques destinés aux enseignants au Jardin Botanique de Missouri.
La Evaluacion Consigue Resultados
En este trabajo se presentan los resultados y las consecuencias de una evaluacion comprensiva del Natural Science Institute for Elementary Teachers (el Instituto Cientifico Nacional para Profersores Elementales) en el Jardin Botanico de Missouri. Este proyecto de formacion de profesores utilizo una combinacion de evaluaciones formativas y anadidas, examenes de conocimiento cientifico antes y despues, encuestas, entrevistas personales, y notas de actividades para llegar a cuatro evaluaciones del proyecto y contestar algunas preguntas especificas. Las conclusiones generales de la evaluacion fueron que el proyectop atrahia la audiencia deseada, el conocimiento de conceptos cientificos de los participantes aumentaba, que un protocolo de ‘practica reflectiva’ resultaba en una mejor incorporacion de nuevas abilidades y lecciones en las aulas, y que los participantes en el programa se convierten en recursos formales e informales para otros profesores. La evaluacion tambien resulto en mas fondos para los programas de fomacion de profesores, y en cambios fundamentales en los programas de formacoion profesorial en el Jardin Botanico de Missouri.