Main points 2.1 The purpose of this chapter is to inform the Legislative Assembly about the work we did on provincial testing of students in the Anglophone sector, by the Department of Education. This section highlights significant observations and summarizes our work in two charts (Exhibits 2.1 & 2.2). Highlights 2.2 Some of our observations and comments include the following. • Individual students’ results on provincial tests are reported to the school; however, they are not included in the student’s mark, and they are not reported to the parent/guardian. • Some districts and some schools do testing that is similar to provincial testing, which is not incorporated into the student’s mark. The Department is not monitoring to ensure all the testing is needed, and there is no integration of the testing for efficiency - to ensure the appropriate party is doing appropriate testing. Students could be writing too many tests. • We examined the provincial testing schedule and while we found most of the schedule to be consistent with the education plan, we observed some inconsistencies and risks. • The Department has a good process for marking provincial tests. • While the Department administers many provincial tests, there are no documented policies and limited procedures to protect the integrity of the data and the quality of the information generated. Without documented policies and procedures with adequate monitoring, the quality and reliability of the information gathered through provincial testing is at risk. The Department could be gathering and using bad information. While the Department has guidelines, we saw evidence that schools were not always following them when the provincial test was written. • The results from provincial testing show that the percentage of students meeting the proficiency standards for literacy has increased since 2004; during the same time, the percentage of students meeting the proficiency standards for math has decreased. The Department should take care to ensure that improvements in performance in one area do not come at the expense of declines in performance in any other subject area. (The results from national and international testing show New Brunswick students achieving at levels below their peers from across Canada.) • We believe that different purposes for provincial testing would result in different testing and reporting strategies. For example, if the purpose for provincial testing is to serve as a system check (“to keep the public informed about the educational system’s general health”), then a sample of students could be tested, which would likely have less cost and be less disruptive to learning time. However, if the purpose for provincial testing is to measure individual students’ achievement, then the results should be included in the student’s mark and reported to the parent or guardian. • Although the Department has been doing provincial testing for over thirty years, it has not yet completed a strategic plan for provincial testing. The Department provided us with a draft document labeled Assessment Framework and evidence that the Department has been working on it over the past four years. The Department informed us that it was in the process of finalizing the Assessment Framework. • The Minister’s advisory committee, which is required by legislation, was inactive from 2004 until March 2009. • One of the provincial tests is a graduation requirement – the ELPA (English Language Proficiency Assessment). It tests for appropriate achievement of grade 8 literacy standards. The test is given to students in grade 9. Students not successful on the test can rewrite it in grades 10, 11 and 12. Policy 316 “Graduation Requirements” provides a comprehensive listing of requirements necessary for a student’s graduation, which includes the ELPA. We are surprised that the Department expects only a grade 8 literacy level as one of its requirements for high-school graduation. • In 2008, 43% of grade 9 students (and 62% of the repeat writers in grades 10-12) failed the reading component of the ELPA. Thirty-three percent (33%) of grade 9 students and 42% of the repeat writers failed the writing component. • In October 2008, eighty-seven schools had students who wrote the grade 7 provincial test in English literacy, which tested for appropriate achievement of grade 6 standards. The provincial target is that 85% of students reach or exceed appropriate achievement. Only two of the eighty-seven schools met the target. • With its Accountability Report 2008, we are pleased to see that the Department has made progress in performance reporting. The Department has performance indicators with targets, and is measuring and publicly reporting actual performance in comparison to the targets. Despite actual performance figures being low when compared to targeted performance on provincial tests, the Department is demonstrating accountability by reporting performance. 2.3 Despite the above observations, we commend the Assessment and Evaluation branch within the Department of Education for the work they do. We observed dedicated staff members doing a large volume of work and noted the following. • With limited resources, the Assessment and Evaluation branch administers a large number of provincial tests, which has been increasing since 2005-06. With the Province’s French Immersion program, a literacy test may require the preparation of two different tests, one for the English program and a second for the French Immersion program. • Recently, the branch has started doing additional analysis of the results of the provincial tests. The information provided to the Department and the districts appears very useful. • In December 2008, after a two-year recess from publicly reporting school results of provincial testing, the branch generated school report cards that each student took home to parents or guardians and that are available to the public on the Department’s web site. What we found 2.4 Our observations, conclusion and recommendations for each of our two objectives are summarized in Exhibits 2.1 & 2.2. Exhibit 2.1 Strategic direction for provincial testing Exhibit 2.2 Administration of provincial testing Introduction and background information 2.5 The purpose of this section is to give an overview of the chapter’s layout; present the objectives of our work; and provide background information on the public school system in New Brunswick. Overview of the chapter’s layout and content 2.6 The purpose of this chapter is to inform the Legislative Assembly about the work we did on provincial testing of students in the Anglophone sector. We began with a section which summarized main points. This section provides background information on the public school system in New Brunswick and the Assessment and Evaluation branch of the Department of Education, which is responsible for administering provincial testing for Anglophone educational services in New Brunswick. Next, we present our observations and recommendations relating to the Department’s strategic direction for provincial testing of students. The following section presents our observations and recommendations relating to the Department’s process of administering its provincial testing of students. The appendix shows the recommendations that we made to the Department, along with the Department’s response to the recommendations. 2.7 This chapter identifies issues that require the Department’s attention and areas where improvement is needed to protect the quality of information obtained by provincial testing in the Anglophone sector. We include some of the results of students’ performance on provincial tests. Our report also includes references to the province of Alberta. (Alberta’s vision for education is stated as, “The best Kindergarten to Grade 12 education system in the world.” And, the results of national and international testing of students show that Alberta is a leader in education, both within Canada and internationally.) What we examined and the objectives of our work 2.8 Assessment and Evaluation is a branch within the Department. The branch is responsible for developing and conducting the provincial testing program. 2.9 Provincial tests are prepared by the Department for several grade levels and various subjects. The results from provincial tests are reported to schools, districts, parents/guardians and the public in varying levels of detail. 2.10 We began our work during 2008 and reported our findings to the Department in July 2009. The objectives of our work were: • to assess the Department’s strategic direction for its provincial testing of students in the Anglophone sector; and • to assess the Department’s process of administering its provincial testing of students in the Anglophone sector. The public school system in New Brunswick Organization of the school system 2.11 The Department describes the organization of the school system as follows. Since 1967, the provincial government has had sole responsibility for financing public schools and is committed to equal opportunity for all students. The Minister of Education prescribes curriculum and establishes educational goals and standards. Serving Canada's only officially bilingual province, New Brunswick's education system offers students the opportunity to learn in both French and English through two parallel but separate education systems. Each linguistic sector of the Department of Education is responsible for its own curriculum and assessment. The public education system has 14 school districts -- five French and nine English. District Education Councils (DECs), consisting of publicly and locally elected members, are responsible for establishing the direction and priorities for the school district and for making decisions as to how the districts and schools are operated. Mission statement, budget and statistics 2.12 The Department’s mission statement for New Brunswick schools is, “To have each student develop the attributes needed to be a lifelong learner, to achieve personal fulfillment and to contribute to a productive, just and democratic society.” 2.13 The Department of Education has an annual budget of $963.1 million for 2009-2010 ($941.6 million for 2008-2009). 2.14 Statistics for each sector are shown in box 1. When kids come first (WKCF) – the Department’s education plan 2.15 In June 2007, the Department released When kids come first, its plan for public education in New Brunswick. The Department describes the government’s education strategy as follows, “When kids come first, our five-year framework for increasing student learning and academic performance is an ambitious plan. It is based on the vision that we can build the best education system in Canada, an integral component of our self-sufficiency agenda.” 2.16 The five-year plan has three goals (box 2), eight commitments, 142 actions and 23 targets to be achieved by 2013. 2.17 For more information on the school system in New Brunswick, see the Department’s web site: www.gnb.ca/0000/. Assessment and Evaluation branch 2.18 This section presents information on the Assessment and Evaluation branch for the Anglophone sector of the Department. It states the mandate for the branch and provides a brief description, discusses the provincial testing schedule and gives some general information on provincial testing. Who is responsible for provincial testing? (mandate and description) 2.19 Responsibilities for provincial testing are assigned to the Minister of the Department, the principal of each school and the “provincial testing and evaluation advisory committees” in sections 37 & 38 of the School Administration Regulation - Education Act. 2.20 Assessment and Evaluation is a branch within the Anglophone Educational Services division of the Department. The branch is responsible for developing and conducting provincial testing programs at both elementary and secondary school levels. The mandate of the branch is stated as follows, “The Evaluation Branch is responsible for monitoring student achievement for the purposes of public accountability and improving teaching and learning.” 2.21 The branch began in 1975. A task force in 1975 did “a study of Provincial Testing and Evaluation Needs in the Province of New Brunswick.” (“Until 1972 the Department of Education administered High School leaving examinations. At that time, the Province entered into a four-year contract with Educational Testing Service, Princeton, New Jersey to provide tests based on the NB curriculum in nine subject areas. Writing the tests was on a voluntary basis.”) 2.22 In addition to the provincial testing done by the Department, the branch is also responsible for administering the national and international tests in which New Brunswick participates (see boxes 4 and 5). National and international tests provide measures of how New Brunswick students perform in relation to the rest of Canada, as well as other industrialized nations. Provincial testing schedule – grade level and topic 2.23 Provincial testing in the Anglophone sector is done at each of the three school levels: elementary, middle and high school. Provincial tests are conducted on language arts (reading and writing) and mathematics. A provincial test on science is being piloted. In 2008-09 students wrote provincial tests at several grade levels for various subjects as listed here. • Kindergarten - Teacher's School Readiness Inventory • Grade 2 literacy • Grade 4 literacy • Grade 5 mathematics • Grade 6 science (pilot) • Grade 7 English literacy • Grade 8 mathematics • Grade 9 English Language Proficiency Assessment and Reassessments (ELPA). (Reassessments are written at the same time for students in grades 10, 11 and 12 who have yet to be successful. Success on the ELPA is a requirement for graduation with a New Brunswick high school diploma.) • Grade 10 French oral proficiency • Grade 10 French literacy • Grade 12 French oral proficiency • “off year” for national and international assessments (NB students participated in the national assessment in 2007 and will participate in the international test in 2009.) General information on provincial testing 2.24 For more information on the Assessment and Evaluation branch, see the Department’s web site: www.gnb.ca/0000/. Objective 1 - Strategic direction for provincial testing of students 2.25 The purpose of this section is to provide additional information on the observations shown in the chart presented earlier for objective 1. 2.26 The objective was: to assess the Department’s strategic direction for its provincial testing of students in the Anglophone sector. 2.27 We comment on the following: • the Minister’s advisory committee; • a strategic plan for provincial testing in the Anglophone sector; and • provincial testing’s role in the Department’s education plan. The Minister’s advisory committee The Minister’s advisory committee is required by legislation 2.28 Section 38 of the School Administration Regulation - Education Act states the requirement of a provincial testing and evaluation advisory committee to the Minister, for “the purposes of consultation, advice, liaison and communication with respect to the provincial testing and evaluation programs”. It also states that the committee is to have eleven members who meet at least annually. Committee members are to serve a two-year appointment and may be reappointed. The committee was inactive for more than four years 2.29 At the beginning of our work, the Department informed us that the Minister’s Advisory Committee on Testing and Evaluation (MACTE) had existed in the past, but did not exist at the time we started our work. The Department planned to reconvene the committee. 2.30 The Department provided documentation (minutes of meetings and correspondence with members) demonstrating the following. • The Minister’s Advisory Committee on Testing and Evaluation was inactive from 2004 until March 2009. The Department is not complying with legislation if it operates without an active provincial testing and evaluation advisory committee to the Minister. • The committee was active from November 1999 to June 2004. • The committee had twelve members in 1999, only nine members in 2001-2002, and ten members appointed for 2003-2005. • The committee expressed concern regarding their influence, i.e. their role and their recommendations to the Minister. The committee has been reconvened 2.31 The Department told us the committee has been reconvened and provided documentation from its first meeting on March 30th, 2009. Conclusion 2.32 The Department is not complying with legislation unless the Minister’s advisory committee is active. Recommendations 2.33 The Department should ensure the Minister’s advisory committee actively operates in compliance with legislation. The representation on the Minister’s advisory committee and the frequency of its meetings should allow the advisory committee to serve its purpose. 2.34 The Department should document clearly the committee’s terms of reference, which should include its role and responsibilities. 2.35 The Department should develop an orientation program for new committee members. Strategic plan for provincial testing in the Anglophone sector The importance of a strategic plan 2.36 Strategic planning results in a clear purpose with goals and priorities for a program. It also identifies the type of information that must be gathered in order to assess the achievement of the program’s purpose. A strategic plan also identifies risks and the actions needed to minimize or prevent key risks. A good plan also includes a process for reviewing and amending the plan. 2.37 Once a strategic plan is prepared and implemented, it is important to monitor operations to ensure the plan is followed. Strategic plan for provincial testing has not been completed 2.38 The branch responsible for provincial testing has been operating since 1975. The description of the Evaluation Program Component in the 2009-2010 Main Estimates says that the purpose is to “develop and administer a provincial comprehensive policy on educational evaluation.” This statement has been repeated in the Main Estimates each year since 1992-93, and perhaps earlier. 2.39 The Department has recently committed to completing an Assessment Policy Framework. The document called When kids come first (June 2007) included the following action step, “Work with education partners to develop an Assessment Policy Framework that clearly articulates what is tested, why and how, and that ensures provincial testing reflects critical thinking and problem-solving skills.” This action step was to be implemented in phase 1 of When kids come first, which covered the first two years of the plan. 2.40 We also saw evidence suggesting that the Department has been working on developing a provincial testing plan over the past four years. • A report dated April 2005 called Believing in Achieving 2005 included the following statement, “The provincial assessment program in the anglophone sector is being revised… These revisions will be finalized within the next year.” • We reviewed a document dated May 2006 titled, “A Framework for Improving Student Learning – Anglophone Sector (Revised Draft) – A Discussion Paper – Prepared for the N.B. Department of Education by Stevenson & Cooper”. This report included a proposed policy, guidelines and recommendations for the provincial assessment program and for improvement planning. • The Department provided us with a draft document labeled Assessment Framework and informed us that it was in the process of finalizing the document. 2.41 The history of provincial testing indicates to us that the Department has placed more importance on testing in recent years. While the Department has been working towards putting in place a framework, the Department has not yet completed it. Recommendation 2.42 With the assistance of the Minister’s advisory committee, the Department should complete its strategic framework for provincial testing of students. Purpose of provincial testing 2.43 The Department’s Annual Report states the following for the Assessment and Evaluation branch. The responsibilities within Assessment and Evaluation include monitoring student achievement through provincial examinations and monitoring school effectiveness through a school review process. The objective of both is to improve teaching and learning and to keep the public informed about the educational system’s general health. Provincial assessments function as a reasonable and cost-effective gauge of an individual student’s or school’s overall achievement. 2.44 This information was out of date when published (the school review process was no longer done by the branch), and the objective statement does not clearly articulate the purpose of provincial testing of students. In comparison, in Alberta the purpose of provincial testing is clearly stated as follows. The purpose of the Achievement Testing Program is to • determine if students are learning what they are expected to learn • report to Albertans how well students have achieved provincial standards at given points in their schooling; • assist schools, authorities, and the province in monitoring and improving student learning. 2.45 The draft Assessment Framework the Department showed us includes a purpose statement that is similar to Alberta’s statement. Part of the process to finalize the framework would include getting input from the Minister’s advisory committee on the proposed purpose statement. We believe that the Department needs to finalize the purpose of provincial testing, because it will help to ensure that testing is carried out in the most effective way. 2.46 We believe that different purposes for provincial testing would result in different testing and reporting strategies. For example, if the purpose for provincial testing is to serve as a system check (“to keep the public informed about the educational system’s general health”), then a sample of students could be tested, which would likely have less cost and be less disruptive to learning time. However, if the purpose for provincial testing is to measure individual students’ achievement, then the results should be included in the student’s mark and reported to the parent or guardian. Conclusion 2.47 The Department has not finalized its strategic plan for provincial testing. While the Department has been working on an Assessment Framework, it needs to receive input from the Minister’s advisory committee on the draft framework, and then finish and adopt the resulting framework. It is important that the final framework include a clear statement of purpose for provincial testing. Recommendation 2.48 With the assistance of the Minister’s advisory committee, the Department should determine and document its purpose for provincial testing. Provincial testing’s role in the Department’s education plan 2.49 Strategic direction for the Department’s work is documented in When kids come first, which states the Department’s vision, goals, commitments, targets and action steps. 2.50 “To help measure the progress of When Kids Come First, …23 targets, linked to the eight commitments, are used to determine whether the education and school systems are improving relative to their current status . . .” The Department uses provincial testing to measure progress on eight of its targets, as shown in box 6. 2.51 We noted an inconsistency between one of the Department’s goals and two of its targets. Goal #2 states, “Every child will leave Grade 5 having mastered the tools to learn – reading, writing and numeracy.” However, targets #4 and #5, state “90% of kids”. Objective 2 – Process of administering provincial testing of students 2.52 The purpose of this section is to provide additional information on the observations shown in the chart presented earlier for objective 2. 2.53 The objective was: to assess the Department’s process of administering its provincial testing of students in the Anglophone sector. 2.54 We comment on the following: • the schedule for provincial testing; • similar testing done by districts and schools; • addressing the special needs of students; • the marking process; • reporting results of provincial tests; • documented policies and procedures; and • the English Language Proficiency Assessment (ELPA). Observations on the schedule for provincial testing in the Anglophone sector 2.55 The schedule for provincial testing in the Anglophone sector is presented in box 7. It shows that literacy is tested four different times, at grades 2, 4, 7 and 9. It shows that math is tested two different times, at grades 5 and 8. It shows that provincial testing in science at grade 6 is in its third year as a pilot. Inconsistencies between the provincial testing schedule and the Department’s education plan 2.56 Provincial tests provide information for measuring progress towards attaining specific targets stated in the Department’s education plan (WKCF). While we found most of the schedule to be consistent with the education plan, we observed the following inconsistencies. • WKCF states eight commitments. The second commitment states, “To Work Urgently on Literacy, Numeracy and Science - Reading, writing, math and science concepts are the tools for learning. We will set high standards for students to master these skills by Grade 5 and improve them throughout higher grades. …” While the education plan indicates mastering science by grade 5, the schedule for provincial testing (box 7) shows that there is no provincial testing in science before grade 6. (Prior to 2002-03, science was tested at both grades 3 and 5.) • The fifth target in WKCF states, “90% of kids reach the numeracy standard at Grade 3 and again at Grade 5; 20% exceed it.” The schedule for provincial testing shows that there is no provincial testing in numeracy at grade 3. However, the education plan states an action of adding provincial testing in numeracy at grade 3 within two years, which would be June 2009. There is a risk of lost learning due to the timing of two tests. 2.57 Most of the provincial tests are written by students at the end of the school year, often in early June. This seems to be an appropriate time given that the test measures the achievement standards for the end of a grade. For example, the grade 2 provincial literacy test measures the achievement of literacy standards for the end of grade 2. 2.58 We note two tests are not written at the end of the school year. The grade 7 English literacy provincial test is usually written in October of grade 7 and measures the achievement of literacy standards for the end of grade 6. The grade 9 English language proficiency test is usually written in January of grade 9 and measures the achievement of literacy standards for the end of grade 8. It is possible, and we believe it is likely, that teachers review information to help students prepare for tests. When provincial tests are not written in the grade consistent to the achievement standards being tested, teachers may review the previous year’s information rather than teaching the current year’s curriculum. There is a risk of lost learning due to provincial testing of only one subject within a grade. 2.59 When provincial testing addresses only one subject at a specific grade level, as opposed to several subjects in the grade’s curriculum, there is a risk of teachers, “reassigning instructional time from subjects that are not being tested provincially.” 2.60 The schedule for provincial testing shows that only one subject is tested at a grade level. Prior to 2002-03, this was not the case; provincial tests at grades 3 and 5 were done in English reading, math and science. In 2002-03, the schedule was similar except for the elimination of testing science at grade 3. In 2003-04, the schedule for provincial testing changed to testing only one subject at a specific grade level; testing literacy at grade 2 and testing math at grade 5. In 2005-06, testing literacy began at grade 4 and at grade 7. 2.61 Box 8 shows that the percentage of students meeting the proficiency standards for literacy has increased since 2004. It also shows that during the same time, the percentage of students meeting the proficiency standards for math has decreased. We note that there is provincial testing in literacy two times (at grades 2 and 4), before the first provincial test in math (at grade 5). We are unable to comment on the change in performance since 2004 for other subjects, such as science or social studies, because there is no provincial testing of them in New Brunswick. While it is acceptable to focus improvement efforts, care must be taken to ensure that improvements in performance in one area do not come at the expense of declines in performance in any other area. 2.62 We also observed that the provincial testing schedule for the Anglophone sector in New Brunswick is different from most of the other provinces. (The Department provided us with information on the provincial testing programs in the other Canadian provinces.) Most provinces, including Alberta, do provincial testing of more than one subject within a grade. 2.63 Box 9 shows the provincial testing schedules for New Brunswick and Alberta. It portrays single-subject testing in New Brunswick and provincial testing of multiple subjects in Alberta. We question why the Department does provincial testing of only one subject at a specific grade level, which has the risk of teachers focusing on the subject having a provincial test rather than the entire curriculum for the grade. There is a risk of lost learning due to the increasing number of tests. 2.64 Within the past four years the number of provincial tests written by students in the Anglophone sector has doubled; it has gone from five to eleven. And, the Department plans to add at least one more provincial test. • The number of provincial tests conducted in the 2007-08 academic year was eleven; see box 7. • The number of provincial tests conducted in the 2003-04 academic year was five. They were: grade 2 literacy, grade 5 math, grade 8 math, the middle level English Language Proficiency Assessment (ELPA), and grade 12 French second language oral proficiency. • The Department plans to add a test for numeracy in Grade 3, as stated in the Department’s education plan, When kids come first. • According to the Charter for Change, the government will “reinstate Provincial Achievement Examinations to ensure all students are learning the essential skills.” 2.65 Each provincial test has a cost, not only a financial cost to the Department for administering the test, but also a cost to the schools due to the consumption of valuable learning time and due to stress on students. The provincial testing schedule has been increasing, which has disadvantages 2.66 A stable provincial testing schedule allows for trend analysis over time, which can provide valuable information. Some of the provincial tests have been in place for many years, such as the grade 8 math test and the French oral proficiency test. However in general, the provincial testing schedule has been increasing, as demonstrated by box 7. Box 7 also indicates that the ELPA changed from grade 8 to grade 9 in 2005-06. It is typical that once the timing of a provincial test is set, it is written during the same month each year. However, the timing of the ELPA is unstable; it has changed each year since being moved to grade 9 in 2005-06. 2.67 Valuable information from trend analysis is forfeited whenever the provincial testing schedule changes. There are other disadvantages that can occur. For example, a changing schedule can hurt operational planning, and it can create an impression of disorder. Conclusion 2.68 The provincial testing schedule has the following weaknesses: it is not completely consistent with the education plan; there is a risk of lost learning due to: the timing of two tests, the testing of only one subject within a grade, and the increasing number of tests; and, the schedule is subject to change. Recommendation 2.69 In consultation with its stakeholders, the Department should challenge its current provincial testing schedule, explore alternatives for obtaining the information required to serve its purpose with the least disruption to the students’ learning time, and revise its provincial testing schedule as needed. Similar testing done by districts and schools 2.70 In addition to the tests prepared by the teacher that are used to generate a student’s mark and the provincial tests that are done by the Department, students write other tests. We observed the following: • Some districts and some schools do testing that is similar to provincial testing, which is not incorporated into the student’s mark. • The Department does not monitor which schools or districts are conducting other tests or how many are done. • “Testing does not make kids smarter; teaching does.” Both teacher and student time and effort is consumed preparing for and writing tests that are not reflected in the students’ course marks. The Department does not know how much learning time is used for conducting other testing done by the districts and schools. It is not monitored. • There is no coordinated integration of provincial testing, district testing and testing by the schools. • Parents or guardians may, or may not, be aware of the other tests; and they may, or may not, be informed of the results. 2.71 Monitoring the testing done by different parties would reduce the risk of duplication and excessive testing time. Conclusion 2.72 The Department is not monitoring testing done by some districts and some schools that is similar to provincial testing and not incorporated into the student’s mark. The Department does not know which schools or districts are conducting other tests or how many are done. Recommendations 2.73 The Department should monitor all testing done by schools and districts that is similar to provincial testing. 2.74 The Department should monitor the amount of students’ learning time that is used for preparing for and writing provincial testing and all other similar testing done by the districts and schools to ensure that it is not excessive. 2.75 The Department should develop practices to ensure that different testing of students complements one another and there is no duplication of efforts. Provincial testing and the special needs of students Why is this important? 2.76 Students with special needs have a special education plan to help them at school. Their plan indicates if they should be exempted from participating in provincial testing or if they require an accommodation when writing the provincial test. Box 10 describes exemptions and accommodations. 2.77 It is important that exemptions and accommodations are appropriately granted in order to protect the quality of the information obtained by provincial testing. Accommodations should be given to students that require them; and, only an eligible student should be provided with an accommodation. Appropriate use of accommodations enhances the validity of the results of the test. Documented guideline 2.78 The Department’s document, Provincial Assessment Program - Guidelines for Exemptions and Accommodations allows provincial testing to be as inclusive as possible by addressing the special needs of students. This document provides guidance on exemptions and accommodations for each of the provincial tests. It also describes the process to be followed and provides the required forms to be used to obtain exemptions and accommodations for students. In addition to being provided to the schools, the document is available on the Department’s web site for parents or guardians. 2.79 The process for obtaining an exemption is different than the process for obtaining an accommodation. Currently, the process for administering exemptions and accommodations is manual with paper forms. Exemption from writing the test 2.80 Our observations on the Department’s process for administering exemptions from provincial testing included the following. • The exemption process appears reasonable. It is clearly documented with standard forms. Forms requesting exemption are monitored by the branch for compliance with the documented process. • Approximately 2% of students are granted an exemption from provincial testing. • Students with exemptions are included when reporting provincial results on provincial tests. They are counted as not having the appropriate standard of achievement on the provincial test. Accommodations for writing the test 2.81 Our observations on the Department’s process for administering accommodations for provincial testing included the following. • The process for providing accommodations is clearly documented with a standard form. Forms requesting accommodations are processed by the branch and the requested accommodation is provided. • The Department does not monitor the requests for accommodations to ensure only eligible students are given accommodations. We examined a sample of 57 students (from four different districts) that had been provided with an accommodation. We determined that each of the 57 students had a special education plan. • The Department does not monitor the number of students needing accommodations, the number of accommodations provided or the type of accommodations required and provided. We examined the requests for one type of accommodation for the 2008 grade 7 English literacy test. We observed that one school with 33 grade 7 students requested accommodations for 9 students (27% of the students writing the test were provided with the accommodation) and a different school with 272 grade 7 students had no requests for the accommodation (0% of the students writing the test were provided with the accommodation). Conclusion 2.82 The Department has documented guidance for granting exemptions from writing provincial tests and providing accommodations for writing provincial tests, to address the special needs of students. The Department is not adequately monitoring to ensure that exemptions and accommodations are appropriately provided. While we were unable to determine that accommodations are provided to all students that are eligible, it appears that only eligible students are given an accommodation. Recommendation 2.83 The Department should strengthen the process for providing accommodations for students with special needs to ensure that: • all eligible students receive accommodations and in the form to which they are entitled; and • only eligible students receive accommodations and only in the form to which they are entitled. The marking process 2.84 Our observations on the marking process for provincial tests included the following. • The staff member responsible for preparing the provincial test is also responsible for its marking. The individual is present at all times during the marking session to oversee the process and address any concern that may arise. • The company contracted for packing and shipping the tests is also present at the marking session. They distribute the tests to ensure all markers are kept busy at all times. • The marking session begins with a briefing and training period so markers are comfortable with their role before the actual marking. • Markers report that they enjoy serving on the marking committee; they comment that it is a great learning experience. • The markers document common errors. The Department uses this information to produce a “list of common errors”, which is circulated to the teachers so they are able to help students with the common weaknesses identified on the provincial test. Conclusion 2.85 The Department has a good process for marking provincial tests. Reporting results of provincial testing in the Anglophone sector 2.86 This section provides our observations on how the Department reports the results of students’ performance on provincial tests. After the provincial test has been written by the students and marked by the marking committee, the Department collects the data and reports the results by student, by class, by school, by district and for the Province. We make comments on reporting individual students’ results, reporting schools’ results and reporting results to the public. Reporting individual students’ results 2.87 Our observations on reporting individual students’ results included the following. • The Department states the following on its website, “Provincial assessments function as a reasonable and cost-effective gauge of an individual student’s or school’s overall achievement.” • Individual students’ results on provincial tests are reported to the school; however, they are not included in the student’s mark. (In Alberta, “individual teachers may use provincial achievement test results in determining final grades for their students.”) • Individual students’ results on provincial tests are not reported to the student or their parent or guardian for most of the tests. Individual students’ results are provided to the student for two of the eleven provincial tests; the English Language Proficiency Assessment (ELPA) and the French oral proficiency test. • Before each provincial test, an Information Bulletin for Parents and Guardians is sent home with the students. While the Information Bulletin is different for each provincial test, in general it provides the same information regarding when the test will be written, what it will be like and how it will be marked. It also tells how the results will be reported. “Districts and schools will be provided provincial, district and school level information, as well as individual student results.” We noted that for the grade 7 provincial English literacy test, the Information Bulletin also stated, “…schools will receive individual student results to share with parents/guardians.” However, parents or guardians were not provided with their student’s results or given instructions for obtaining their student’s results. • The Department is aware that one district provides parents or guardians their student’s results for at least some of the provincial tests. • While we did not review literature on all Canadian provinces, we did note that Alberta and Nova Scotia report the individual student’s results to the parents or guardians and Saskatchewan does not. Reporting schools’ results 2.88 Our observations on reporting schools’ results included the following. • After eleven years of producing an annual Report Card that provided provincial, district and school level information on provincial testing, there was no public reporting of these results for two academic years (2005-06 and 2006-07), • In December 2008, the Department released the first annual school achievement reports. Each school received a Report on Achievement. The report has the following traits, which make it an excellent report. The report shows achievement levels on provincial tests over a three-year period, which allows the reader to see progress over time. The report shows the provincial targets for each test along with the overall performance results for the school, district and the Province, which allows the reader to make comparisons. Results for schools having five or fewer students who wrote the test are not reported, which respects individual students of small classes. The report is easy to read and understand. • The school Report on Achievement was given to each student in every school to take home to their parents or guardians. • The Department’s information system is not well-suited for generating the Report on Achievement. For example, producing the 228 school reports involved manually entering three years’ of provincial testing data and manually verifying its accuracy. Reporting results to the public 2.89 Our observations on reporting results to the public included the following. • The Report on Achievement for each school and each district is available on the Department’s web site for the public. • The Department’s web site also provides the Provincial Report which provides performance results for each provincial test, with graphs providing a three-year comparison. It also provides additional information for each test, such as the number of students writing the test. • It is not explicitly clear how to find the results of provincial testing on the Department’s web site; the results are not shown with the other information on the Assessment and Evaluation branch. • In December 2007, the Department released its Benchmark Report for the 2006-07 school year. “This Benchmark Report is designed to give parents and the public a complete picture of where we currently stand on the various indicators of success. Next year, and in the years that follow, we will report on the progress we have made.” In March 2009, the Department released An Accountability Report on the targets of When kids come first 2008 (Accountability Report) for the 2007-08 school year. • The Benchmark Report and the Accountability Report include provincial results for most of the provincial tests. They include results from national and international tests. We did note that the ELPA results were not reported in the 2007 and 2008 reports. • We commend the Department for its report titled, A Benchmark Report on the targets of When kids come first – 2007 and the successor, An Accountability Report 2008. With these reports, the Department has made progress in performance reporting. The Department of Education has performance indicators with targets, and the Department is measuring and publicly reporting actual performance in comparison to the targets. Despite actual performance figures being low when compared to targeted performance on provincial tests, the Department is demonstrating accountability by reporting performance. Conclusion 2.90 Since the purpose of provincial testing has not been clearly documented, we were unable to determine the appropriateness of the reporting of the results. Recommendations 2.91 The Department should document a policy on reporting individual students’ results on provincial tests. 2.92 The Department should assess its information system needs with regards to reporting the results of provincial testing and ensure that its needs are met. 2.93 The Department should ensure that the results of provincial testing are easily identified on its web site. Documented policies and procedures The importance of policies and procedures 2.94 Policies and procedures document the rules of a program and describe the proper steps in performing tasks. Not only do policies and procedures inform staff members how to fulfill their day-to-day responsibilities successfully, but they also provide a basis for monitoring activities to ensure consistency in operations. No policies and limited procedures 2.95 The Department has no documented policies and limited procedures for provincial testing in the Anglophone sector. The following observations indicate a need for documented procedures. 2.96 We observed that the marking team is composed differently for different provincial tests. Sometimes the districts are asked to select representatives to serve on the marking team and other times the Department selects individuals. Without documented guidance, there is little assurance that the selected individuals have the experience and objectivity that a marking team requires. 2.97 The Department does have Administrative Guidelines for Principals and Teachers (the rules that schools must follow when conducting provincial testing) and Guidelines for Exemptions and Accommodations. These appear to be useful documents; however, they are not always followed. For many readers, the term “guideline” means a suggestion, rather than a requirement. This may explain why the guidelines are not always followed. 2.98 We saw cases where the Administrative Guidelines for Principals and Teachers had not been followed. (For example, writing with pencil although the rules state, “students must use a pen with dark blue or black ink,” and inserting additional loose papers, while the rules instruct that responses are to be written in the booklet and state, “there should not be additional pages of student work stapled into the booklet”.) We also saw where a teacher changed the test. (The test instructed students to write on the topic of being a student in middle school. One teacher instructed her students to write on the topic of storm days instead.) The Department is aware of these and other cases where the rules had not been followed. 2.99 The Department does not have policies and procedures for monitoring the schools or enforcing the guidelines. In order to protect the integrity of the information obtained from provincial testing, documented procedures are needed for identifying the misuse of provincial testing materials, documenting the misuse, and reporting it for corrective action. 2.100 The Department should have documented procedures for each significant component of provincial testing. These include: developing and preparing provincial tests; packaging and transporting tests to and from the schools; selecting committees and marking tests; accumulating data and reporting the tests’ results; and storing and retaining provincial testing documentation. Also, there should be documented procedures for how districts, schools and teachers are to use the results of provincial testing. And, to ensure that the policies and procedures are followed and updated as needed, quality control practices need to be documented and implemented. Conclusion 2.101 The Department has no documented policies and limited procedures for provincial testing in the Anglophone sector. Recommendations 2.102 The Department should develop documented policies and procedures for the provincial testing program. 2.103 The Department should provide training, as necessary, to help ensure the policies and procedures are understood and followed. This should include training teachers on the administrative guidelines and the use of provincial testing results. 2.104 The Department should develop and implement quality control practices to ensure the policies and procedures are followed and updated as needed. The English Language Proficiency Assessment (ELPA) What is the ELPA? 2.105 The ELPA (English Language Proficiency Assessment) is a provincial test on English literacy. Similar to the other provincial literacy tests, the ELPA has both reading and writing components. 2.106 Different from the provincial literacy tests at grades 2 and 4, the ELPA is written only in English. (In grades 2 and 4, French Immersion students take a French provincial literacy test.) 2.107 Also different from the provincial literacy tests at grades 2 and 4, success on the ELPA is a requirement. “In order to graduate with a New Brunswick diploma, a student (with the exception of those officially exempted for valid reasons) must demonstrate the ability to read and write at an ‘Appropriate’ standard of achievement as determined by the English Language Proficiency Assessment. Unsuccessful students will have opportunities for reassessment in Grades 10, 11, and 12.” Because the ELPA is a graduation requirement, it appears similar to Alberta’s diploma examinations, which are described in box 11. History of the ELPA 2.108 The following information provides a brief history of the ELPA. • In the 1990s, New Brunswick had provincial exams for various subjects in high school. Student marks were comprised of 70% class mark and 30% provincial exam. To receive a NB high school diploma, there was a required achievement of at least 50% on the English mark. During the period 1997-2004, the provincial exam in English was replaced by the ELPA. • The ELPA was introduced in 1994 in grade 8 as a pilot and implemented in 1995. • Success on the ELPA became a requirement for graduation in June 2001. • The timing of the ELPA shifted from grade 8 to grade 9; it was written for the last time in grade 8 in 2003-04, and it was written for the first time in grade 9 in 2005-06. Comments 2.109 Our observations on the ELPA included the following. • While the ELPA is written during grade 9, it tests for competency of the literacy achievement standards for end of grade 8. • Of the eleven provincial tests, it is the only one that has a success requirement. • Students have up to five opportunities to succeed on the test. • Forty-three percent (43%) of grade 9 students failed the reading component of the ELPA in 2007-08. Sixty-two percent (62%) of the repeat writers (in grades 10-12) failed the reading component of the ELPA in 2007-08. Thirty-three percent (33%) of grade 9 students and 42% of the repeat writers (in grades 10-12) failed the writing component of the ELPA in 2007-08. • The success rate on the ELPA has decreased significantly since 2004. See box 12. • The number of students repeating the test may exceed the number of first-time writers. In January 2009, there were 6,265 grade 9 students who wrote the ELPA, and there were 4,247 repeat writers (from grades 10 through 12) for the reading component and 3,007 repeat writers for the writing component. (Since students only repeat the component that they were not successful on, it is possible that a student may only have to repeat one component. Some students may have to repeat both components.) In 2008, there were 6,447 grade 9 students who wrote the ELPA, and there were 4,042 repeat writers for the reading component and 4,938 repeat writers for the writing component. The Department could not provide information on the success rate by grade level of the repeat writers. • The cost of the ELPA and reassessment is approximately $250,000. This is double the cost of the other literacy tests, which are done at grades 2, 4 and 7 and cost approximately $120,000 - $140,000. (These figures do not include salaries of branch staff members.) • The ELPA is the cause for the recently added (2006) grade 7 English literacy provincial test. From a request brought forward by the teacher population, the provincial test on English literacy at grade 7 was established to prepare students for the ELPA, which is a graduation requirement. The grade 7 English literacy test is written in October, and it tests for competency of the literacy achievement standards for end of grade 6. In October 2008, eighty-seven schools had students who wrote the grade 7 English literacy test. The provincial target is that 85% of students reach or exceed appropriate achievement. Only two of the eighty- seven schools met the target. Seventy-nine schools had fewer than 80% of the students reach or exceed appropriate achievement, of which eight schools had fewer than 50% of the students reach or exceed appropriate achievement. • While the opportunity to rewrite the test four times has been in practice since 2004, the Department has not done a cost/benefit analysis to determine if five opportunities for writing the test is appropriate. Conclusion 2.110 The information on the ELPA does not reflect well on the education system in New Brunswick. It is difficult to understand why success on a grade 8 literacy test is one of the requirements for a high-school diploma; this appears to be a very low expectation for a high-school graduate. The results of the ELPA show that the Department is promoting a large number of students into high school that do not have the literacy skills required to cope with the high school curriculum. 1. Province of New Brunswick, Report Card 2001 Anglophone School Districts - Department of Education - Evaluation Branch. 2. Province of New Brunswick, Report Card 2005 Anglophone School Districts - Department of Education - Evaluation Branch. Conclusion 2.111 In regards to our objectives, we concluded that the Department has not yet finalized a strategic plan for its provincial testing of students in the Anglophone sector; and because the purpose for provincial testing has not been clearly documented, we were unable to conclude on the appropriateness of the Department’s process of administering its provincial testing of students in the Anglophone sector. 2.112 Some fundamental elements of an effective program are missing, such as: a clear purpose, a strategy or master plan, documented policies and procedures, and program monitoring. The Department is doing a lot of provincial testing of students in the Anglophone sector. However, the relevance and the reliability of the information obtained through provincial testing is at risk because the program does not have documented policies and adequate procedures with proper monitoring to protect the quality of the information. We made recommendations to address the weaknesses we identified. We believe our recommendations are consistent with the plan stated in The Charter for Change to “Examine the best practices for student evaluation.” 2.113 The information on the ELPA does not reflect well on the education system in New Brunswick. 2.114 Last year we reported our work on Adult Literacy Services in the Department of Post-Secondary Education, Training and Labour. We noted the Canadian results of the 2003 International Adult Literacy and Skills Survey (IALSS); the average proficiency scores of the adult population aged 16-65 in New Brunswick were below the Canadian averages across all domains measured (literacy, numeracy and problem solving). The international survey results indicated that in Canada, approximately 50% of high-school graduates do not have the skills needed in today’s world. We commented that, “In New Brunswick, we have a free education system. We have legislation that requires children to attend school until the age of eighteen. We have a curriculum set by the Province. We have teachers that must have specific qualifications.” And, we questioned, “Why do we have people without the appropriate literacy skills?” Finally, we quoted the following, “…learners in adult literacy programs represent, at the simplest level, the failure of schools to educate all children to graduation.” With our work on provincial testing in the Department of Education this year, we wonder if perhaps New Brunswick’s high- school graduation requirement for a grade 8 literacy level plays a part. 2.115 Finally, we remind our readers, that although provincial testing can provide “high-quality information that is reliable and valid, it cannot provide all the information that a school needs to know about students’ achievement. School and [district] results from provincial tests are best interpreted within the context of local quantitative and qualitative information.” ... “Achievement tests can assess only part of what is to be learned.” Appendix - Recommendations 2.116 The purpose of this section is to present our recommendations in the areas where we think change is appropriate. Our recommendations to the Department are presented along with the Department’s response to each recommendation. Recommendations relate to each of our two objectives, which address the following topics: • strategic direction for provincial testing (box 13); and • process of administering provincial testing (box 14).