Background 6.1 This chapter examines the development of performance measurement and effectiveness reporting in the Province of New Brunswick over the last ten years. This year marks the tenth anniversary of our Office’s effectiveness reporting mandate and also the tenth anniversary of the Province of New Brunswick's first annual report policy. 6.2 Our Office carried out and reported on its first value-for-money audit in 1989. This expansion of the Office’s role into value-for-money auditing was accomplished by adding to the list of reportable items under the Auditor General Act cases where: • money has been expended without due regard to economy or efficiency; • procedures have not been established to measure and report on the effectiveness of programs, where, in the opinion of the Auditor General, the procedures could appropriately and reasonably be used; and • procedures established to measure and report on the effectiveness of programs were not, in the opinion of the Auditor General, satisfactory. 6.3 These amendments to our legislation in December 1988 established the so-called “Three E’s” (economy, efficiency, and effectiveness) in legislation. From the outset, our Office realized that reporting under that third “E” of effectiveness would be a challenge, especially given the fact that there was no effectiveness reporting framework in place. Paragraph 5.15 of our 1989 Report raised the issue with the following statement: At present there is no consensus on the concept of providing information on the effectiveness of programs in some form of public document. It is in this context that we have carried out the audit [of the Facilities Management Branch of the Department of Supply and Services] and provide our recommendations relating to effectiveness information. 6.4 In this chapter we would like to look at some of the highlights of that challenge over the last ten years. And as we provide that retrospective, we also discuss the somewhat simultaneous development of the Province's annual reporting and performance measurement policies. Scope 6.5 This chapter gives an overview of comments in previous Reports of the Auditor General relating to annual reports of departments and agencies and effectiveness reporting. As well, the chapter addresses various policy and reporting developments by the Province, beginning with the original policies of 1989, the annual report policy AD-1605 from 1991, and the implementation of "Performance Measurement: Supplement to the Main Estimates" in 1996-97. 6.6 Similar to some of our earlier Reports, we have also carried out a review of annual reports to determine if government departments are complying with the annual report policy. In this respect, the scope includes the 1997-98 reports of government departments, the latest ones available at the time of our review. 6.7 We had three objectives in carrying out this special study: • to provide an overview of our audit efforts and the progress by the Province in the areas of performance measurement and effectiveness reporting over the last ten years; • to review annual reports of government departments for 1997-98 using our scorecard methodology developed in prior audits; and • to promote the use of the annual report as an accountability document. Results in Brief 6.8 The performance measurement system in New Brunswick continues to expand and improve each year. 6.9 We have seen improvement in the level of compliance with the annual report policy. We recommend the policy be revised to include direction that each annual report should indicate how it complies with the policy. 6.10 The Legislative Assembly, through its Public Accounts Committee (and possibly the Crown Corporations Committee), could have a role to play in improving annual reports, “the major accountability document for the Legislative Assembly and the general public.” First Value-for-Money Audits 6.11 Our first value-for-money audit, in 1989, covered the Facilities Management Branch of the Department of Supply and Services. That audit included an effectiveness reporting component. We actually suggested a number of indicators the Department might use in reporting on the Branch’s effectiveness. 6.12 We faced a difficulty, however, in that the government of the day had no generally accepted manner or document for addressing this need for performance information. Recognizing this, we made a recommendation that: The information to be included in the Department of Supply and Services Annual Report, the New Brunswick Main Estimates, and the New Brunswick Public Accounts should be reviewed. The purpose of this review would be to establish the types and amounts of information to be reported to enable the Legislative Assembly and the public to form an opinion on the effectiveness of the facilities management function. The review should establish which documents should include this information or if another document should be created to report this information. 6.13 The Department responded that it had some difficulty with our recommendations. It noted that both the Main Estimates and the Public Accounts “are prepared in accordance with an established format that is set by government and we have no input into same.” The Department did agree to undertake a review of the annual report, however, “to determine the information that should be included.” 6.14 Our second value-for-money audit was in the Nursing Home Services Branch of the Department of Health and Community Services. And again, we faced similar issues when it came to our comments and recommendations on effectiveness reporting. The Department responded as follows: While there is no objection to participating in efforts to develop refined accountability reporting, additional resources and central agency leadership are seen as prerequisites. Since central government agencies have a direct interest in such matters, they should take the lead role in this area and establish, in consultation with operational departments, a suitable general framework. First Government- Wide Study of Annual Reports 6.15 It was in this context that in 1989-90 our Office carried out its first government-wide study of annual reports. It was seen as an important background study to help the Office fulfil its new-found responsibilities in the area of effectiveness reporting. 6.16 The Office developed audit criteria in five key areas: timeliness; physical layout; goals, objectives and related performance data; financial disclosure; and understandability. In developing criteria we were aided by the fact that the government had issued two major annual report policy statements shortly before the audit began. The first, dated 1 March 1989, was developed by the Premier’s Office. It was superseded by a policy dated 8 September 1989 from the Cabinet Committee on Communications. We took great interest in the fact that one of the policies spoke of reporting “the results obtained by expenditures.” We saw this statement as a reference towards reporting on the effectiveness of the expenditure of funds. 6.17 Our general conclusion was that the annual reports did not present the information needed by the Legislative Assembly and the public to judge program effectiveness. There was insufficient performance information to tell the accountability story. As well, reporting was not timely. The Auditor General’s major recommendation was as follows: I recommend the Cabinet Committee on Communications revise the requirements for annual reports. Annual reports should reflect how effective departments and agencies have been in the use of government funds. New Annual Report Policy 6.18 We were pleased, and somewhat surprised, by how quickly and thoroughly the government responded to our recommendation. Our public Report was issued in late 1990. On 26 June 1991 the Cabinet Committee on Communications issued a new annual report policy which addressed our recommendation. It answered the questions of where and how performance information should be reported, the very questions that departments had raised in our first two value-for-money audits. 6.19 This new policy was given Executive Council approval on 6 December 1991. It was incorporated into the government’s administrative policy manual system as Policy AD-1605 and remains virtually unchanged to this day. And it clearly established that reporting was everybody’s responsibility. That is, the government stated that the policy applied to all four parts of the civil service. This included government departments and agencies, Crown corporations, hospital corporations, and school boards. 6.20 We would like to quote from various sections of the policy to illustrate the manner in which it has established a framework for performance reporting. 6.21 The objective of the policy is stated as follows: The prime function of an annual report is to be the major accountability document by departments and agencies for the Legislative Assembly and the general public. It serves as the key public link between the objectives and plans of a government entity and the results obtained. 6.22 We believe this was a key statement. In setting out the “prime function” of the annual report, it clearly established where government departments and agencies are to report accountability information. 6.23 The policy goes on to describe various elements or standards of content: To the degree possible, departments and agencies should give a clear account of goals, objectives and performance indicators. The report should show the extent to which a program continues to be relevant, how well the organization performed in achieving its plans and how well a program was accepted by its client groups. Actual and budget financial information in summary form and a narrative explaining major variances as well as other aspects of financial performance are to be included in all annual reports. 6.24 By covering these areas, the policy addresses key aspects of performance reporting. 6.25 Recognizing the importance of understandability, the policy also calls for annual reports to be written in clear and simple language. In order to emphasize the importance of timeliness, the policy calls for all reports to be submitted to the Clerk of the Legislative Assembly by 1 November following each 31 March fiscal year. 6.26 We were also pleased to see that the new policy covered several areas of interest identified by MLAs in a survey regarding annual reports. We had carried out this survey in conjunction with our first review of annual reports. Value-for-Money Audits After Policy Implemented 6.27 After the policy was implemented, our work regarding the third “E” of effectiveness became a little easier. At least we knew where to look for performance information. 6.28 From our 1992 Report on, we have tended to have at least one major audit each year that focuses on a department or agency annual report. Policy AD-1605 provides ready-made criteria for such an audit. In almost every case we have made recommendations for improvement. We have never been able to conclude that any annual report that we examined was in full compliance with the policy. 6.29 We have covered a wide variety of organizations from the Department of Finance’s Tax Administration Division to the Department of Transportation’s road construction activity to the Environmental Trust Fund. We have extended our reach to cover Crown corporations such as the New Brunswick Geographic Information Corporation and the Lotteries Commission of New Brunswick. In 1998 we communicated with the regional hospital corporations on this issue as part of our governance study. At the conclusion of a discussion on public reporting we made the following recommendation: We recommend that regional hospital corporations comply fully with the provincial annual report policy in future corporate annual reports. Across-the–board surveys 6.30 In addition to detailed examinations of annual reports and departmental effectiveness measures, we have also carried out two general studies of annual reports across government. The first of these was more of a "report card" style of study where we reviewed each annual report and "checked off" if there was something there that seemed to fit the policy requirement. There was no formal assessment (or assurance provided) of the quality or reliability of the information. 6.31 For example, departments are required to present summary financial information with variance analysis. We did not assess the quality of the variance analysis. If the department had an analysis, it received the checkmark. 6.32 Our 1995 Report contained our survey of deputy ministers' self-assessment of their own annual reports. We had provided each deputy minister with a checklist of the policy requirements and asked them to carry out their own analysis. We were pleased that all the deputy ministers responded to our survey. The deputy ministers’ responses included several frank admissions that the corresponding departments did not meet aspects of the policy. For example only four departments felt they had information on client acceptance as called for in the policy. A number of deputy ministers admitted the annual reports did not discuss program relevance. 6.33 We were encouraged to see that in a number of cases the deputy ministers spoke of planned improvements in their report. Departments and agencies saw themselves moving towards compliance with the policy. Performance Indicators 6.34 One requirement in the policy is to "give a clear account of ...... performance indicators." In a number of our audits of departmental annual reports, we had negative comments with respect to this aspect of compliance. For example, our 1995 audit of the Department of Advanced Education and Labour contained the following recommendation: We recommend the Department ensure that benchmarks and performance indicators are established for all key activities of the college system. The results should be reported publicly as part of the Department’s annual report. 6.35 In the Department of Economic Development and Tourism in 1998 we recommended that The Department should develop performance indicators to report against all strategic objectives. This would allow for an evaluation of the degree of success of the Department in achieving its strategic objectives and therefore its mission. 6.36 To some extent, this one area of non-compliance was contemplated by the policy. One paragraph of the policy reads as follows: It is recognized that management information systems in many departments and agencies do not produce sufficient relevant data to meet this goal. However, over time, departments and agencies are expected to develop performance indicators and to include this information in their annual reports. 6.37 For the first several years that the policy was in existence, we had to consider this qualifying statement. Commencing, however, with the budget process for 1996-97, the government has embarked on a performance indicator process that should see that qualifier becoming less and less relevant. 6.38 In 1995 the government announced a performance measurement initiative which was implemented by the Department of Finance for the 1996-97 budget year. All departments and agencies were asked to establish public benchmarks and other measurements to assess the performance and impact of the programs they administer. 6.39 Now in its fifth year of development, the plan has been based on the concept of incremental progressive improvement. In the latest report Performance Measurement, Supplement to the Main Estimates 1999-2000, fifteen departments have identified core businesses and goals, with related performance indicators and targets for most key result areas. One of the major factors in its success to date has been acceptance and buy-in to the process by departments, demonstrated by their co-operation and dedication to ongoing improvement of the system. 6.40 The Province currently measures performance only at the departmental program level. Departments are required to prepare performance information as part of the Main Estimates process. Departments clarify their mandates through the identification of their core business areas and for each area identify the key goals. For each key goal, indicators are outlined together with targets and sources of information. This information is collected from all departments and consolidated into a supplement to the Main Estimates. 6.41 The responsibility for the performance measurement initiative has been recently transferred from the Department of Finance to the Executive Council Office. This office acts as developer and co-ordinator of the initiative, as well as advisor to departments. All performance indicators and targets are subject to approval each year, currently by the Board of Management. Departments are then required to publish the results in their annual reports. The initiative in perspective 6.42 The performance measurement system in New Brunswick continues to expand and improve each year. We have been informed that this approach has been viewed as very successful by government, allowing officials to deal with growing pains as encountered and to plan ongoing strategies. This has also allowed a gradual evolution and growth of the process, while ensuring departments have never been overwhelmed by it as yet another central agency request. One key executive put it succinctly - “While recognizing the limitations of our current process, we are encouraged by its demonstrated improvement each year.” 6.43 Our Office is encouraged to see the development of this process as well. We recognize that the publication Performance Measurement, Supplement to the Main Estimates was not developed directly to support the reporting requirements of the annual report policy. Departments were never directed they had to do this in order to comply with that policy. Performance Measurement was instead a product of central direction stemming from a commitment in a 1995 government platform document. But nonetheless, Performance Measurement does provide an important link to the annual report discussion on performance. And we believe this to be a good thing. It should lead to overall improvements in the annual reports of departments as the years progress. 6.44 A key improvement to the performance measurement system is training sessions currently being provided to key departmental personnel to encourage better use of performance measurement as an internal management tool. With these more recent developments it is hoped to change the focus from that of just reporting to one of using the information received to improve both planning and decision-making at all levels throughout government. Review of Annual Reports for 1997-98 6.45 As noted, we believe it is important to continue emphasizing the annual report policy. It is a key part of a focus on accountability. As the policy states “the annual report is to be the major accountability document by departments and agencies for the Legislative Assembly and the public.” 6.46 It is from this perspective that we approached our second objective, to review annual reports of government departments for 1997-98 using our scorecard methodology developed in prior audits. This year we reviewed the annual reports of the fifteen government departments which presented indicators in Performance Measurement, Supplement to the Main Estimates for the year ended 31 March 1998. This was the most recent fiscal year for which all annual reports were available. We did not include the reports of Crown agencies this year. We wanted to focus on the departments only, because the Crown agencies have been excluded to date from the performance measurement initiative of government. 6.47 Our review was similar to that of previous years. It looked at the policy requirements and compared these requirements to the reports. In other words, the review asked, what does the policy call for and is it easily identifiable in the report? The review was not designed to evaluate the quality of the information; just whether or not the information required was presented. 6.48 Our questions arising from a review of the requirements of the policy are as follows: • was a report prepared? • are the goals/objectives of the organization stated? • are performance indicators presented? • does the report discuss achievement of plans? • is there discussion of program relevance? • are details available on level of client acceptance? • is actual and budget financial information presented? • does the report explain variances from budget? • was the report tabled with the Clerk of the Legislative Assembly by 1 November? Was a report prepared? 6.49 We are pleased to report that all departments included in our review prepared annual reports. Are the goals/objectives of the organization stated? 6.50 All departmental reports have stated goals and objectives in some recognizable form. This is a major improvement over our previous surveys. Perhaps part of the reason for this change can be attributed to the performance measurement system. The performance measurement system requires departments to identify the key goals for each core business area. Are performance indicators presented? 6.51 All of the reports reviewed contained a section that outlined key performance indicators for the year. Does the report discuss achievement of plans? 6.52 As a general rule, reports follow the department’s organization structure. That is, each branch or section of the department has a discussion of its various activities for the year. They are often activity oriented. To quote for illustrative purposes from the Introduction of one report – “the annual report ....... for 1997-98 outlines the activities and major events undertaken.” These activities are not necessarily linked to any strategic objectives and operating plans although in at least four cases they appear to be linked to the program component objectives present in the Main Estimates. Some departments have presented information from their strategic planning processes, but this is not directly referenced to the various activities reported on elsewhere in the report. 6.53 In our 1998 Report we commented at some length on reporting with respect to achievement of plans in a more detailed examination of the 1996-97 annual report of the Department of Economic Development and Tourism. We made the following recommendation as part of that work: Once departmental strategic objectives have been defined, it would be useful to have performance indicators and narrative descriptions of achievements presented by strategic objective in the annual report. This would make it much easier for a reader to evaluate the degree of success of the Department in achieving its strategic objectives. 6.54 This is a recommendation that probably still has general applicability. Some form of improved reporting is necessary to enable the reader to determine how well departments are doing in achieving their plans. Is there a discussion of program relevance? 6.55 This topic was not given separate attention in any report we reviewed. But of course a number of departments referred to their process of strategic planning. And, as noted, the performance indicator process includes the identification of core business areas. Therefore, one could not say relevance has been ignored. 6.56 When we carried out our 1995 survey of deputy ministers, we received several interesting comments on program relevance. One respondent reported: It is my sincere opinion that program relevance should not be discussed as an issue in the annual report. Discussing the merits or faults of each program or service offered would be like second-guessing the decisions taken by government. Once decisions are made by government to eliminate programs no longer relevant, or to implement new programs this information could be part of the annual report, along with information on the process used to determine why such a change was necessary. 6.57 Another in reference to certain regulatory programs stated: We have traditionally assumed that the relevance of such requirements was subject to extensive public and legislative debate when the statutes were initially passed, and as a result, presenting rationales for their application in an annual report is no longer required. 6.58 These comments, and our observations on the general lack of discussion of the issue, point out that program relevance may be one area where additional guidance on compliance might be needed. Are details available on level of client acceptance? 6.59 As with program relevance, client acceptance was not a topic that the reports systematically addressed. It should be noted, however, there are often statistics on the volume of users of programs. This does provide some information on the extent to which intended beneficiaries are taking advantage of programs. But we believe that in order to fully answer the question on client acceptance, there should be some discussion of the extent to which various users judge a program to be satisfactory. In this regard, at least one performance indicator did speak clearly to the issue. “The percentage of client satisfaction with archives services as reported through annual customer surveys” is an indicator in the Department of Supply and Services. Is actual and budget financial information presented? 6.60 All departments presented budget versus actual financial information. Does the report explain variances from budget? 6.61 Eleven departments had narratives explaining variances from budget. Four of those surveyed did not. Was the report tabled with the Clerk by 1 November? 6.62 Three departments tabled their reports prior to 1 November. Another department came close by tabling on 2 November, the first working day in November. Two other departments tabled in the first week of November. 6.63 This is a fair improvement over our last two surveys. In both 1993 and 1994 only one department met the deadline, tabling its report in late October. Nonetheless, we believe there is room for continued improvement. Less than half of the departments had a report on time or within the first week of the deadline. One department did not table until 29 March 1999 according to the Clerk’s records. Another tabled in February. Three tabled in January. 6.64 Perhaps one of the reasons for the failure to table reports on time is the fact that little attention is given to them until the public meetings of the Public Accounts Committee. In this regard we would encourage the Public Accounts Committee to meet as early as possible after the 1 November reporting deadline. Not only would this result in a more timely review of the year just past but it would also allow for input into the budget process for the next fiscal year. Where to from here? 6.65 In general, we have seen improvement in the level of compliance with the annual report policy. We expected to find some level of improvement simply through the implementation of the performance measurement system. We are pleased by this progress. 6.66 Departments are to be commended as well for the attention paid to improving reports. From time to time we are asked by departmental representatives to provide some advice with respect to their upcoming annual reports. We are certainly pleased to be able to share our experience with departments when we receive such requests. While we are able to discuss our past reviews of annual reports with departments, we do not have a definitive format or template that we can give to guide them through the process. 6.67 We noted how most departments are organizing their reports around the branches or sections of the department. One department appears to have developed a more strategic approach in that the information is organized under three core business areas. Each core business section describes various services and programs provided along with certain statistical information. Performance measures are presented in a separate section that is also organized by the same core business areas. 6.68 At least four other departments have organized their reports by program objective. Objectives are stated for the various program components followed by a section on various activities carried out during the year. These objectives are essentially the same as those presented in the Main Estimates. Another department began each section of the report with a subtitle “Mandate” followed by “Major Achievements for 1997-98.” After these achievements are discussed, the reader is informed on “Priority Projects for 1998/99”, the upcoming fiscal year. 6.69 Perhaps what we need is some sort of model around which the annual reports can be structured and further improved. For example, earlier in this chapter we stressed the importance of linking strategic goals and achievement of plans. We repeated a recommendation made in our 1998 Report. Organizing annual reports to tie into the strategic objectives may be one acceptable approach. Organizing it under core business areas may be another. 6.70 We have talked about the issues of program relevance and client acceptance as not being particularly clear in the annual reports. We also talked about the issue of timeliness. These could become part of the focus of improvements in the process. 6.71 In the interim, we would like to repeat a recommendation we made in our 1995 Report. In that Report, we noted we were pleased to see the interest deputies demonstrated regarding compliance with the policy. We went on to state the following: We believe that it would be useful for departments and agencies to discuss their compliance with the policy in a section of their annual reports. A department or agency could place a chart near the front of the report listing each of the aspects of the policy. In a second column the chart could show where in the report each element of the policy was complied with. For instance, in response to the requirement to state goals and objectives, the chart could refer the reader to such things as excerpts from the strategic plan or an overview message from the deputy minister. Regarding budget and actual financial information, the chart could refer the reader to the financial statements contained elsewhere in the report. Recommendation 6.72 We recommend that the annual report policy be revised to include direction that each annual report should indicate how it complies with the policy. 6.73 Our Office has actually adopted such an approach. We have included a chart of this type in our chapter “Office of the Auditor General” for the last several years. The Role of the Annual Report 6.74 Over the years our Office has continued to be a steady supporter of the annual report as an accountability document. This is consistent with our Office’s strategic goal to “Promote accountability and performance reporting by government, by individual departments and by Crown agencies.” With the development of the Province’s performance measurement program, we look forward to continued improvements in annual reports. And we intend to continue to focus on the importance of performance indicators in keeping with our “effectiveness” legislation. 6.75 We are pleased that the message seems to be getting through. In the 5 June 1998 meeting of the Public Accounts Committee a member began his questioning of a deputy minister with the following remarks: We have had sessions here with the Auditor General, and he impressed upon us the importance of performance indicators. Does your department function with performance indicators? Is there some sort of measuring stick you utilize to assess whether you are efficient or whether you are achieving certain goals or objectives? 6.76 Another member questioned a deputy minister as follows: Your performance measurements begin on page 33. I see that you are making good progress on the first goal, which is operating costs for procurement services per total dollar value of goods and services procured. Are you thinking you can continue to improve upon that, or are you getting to the point of more or less maximum efficiency? 6.77 The deputy minister responded: We are trying to, but as you know, savings are harder and harder to find. We are hoping to keep proceeding on that one. 6.78 In a third example, a member began directly with the performance indicators of a government department as presented in the annual report. His questioning started as follows: Another area I want to review with you is performance measurement. I find it quite fascinating to look at the government’s move into performance measurement..... In this case you have all the challenges that are being faced in this year, and you have indicators that you do not appear to be hitting. Have we set the goals too high? Are they realistic? Should they be reassessed? 6.79 This type of questioning, focused on the performance indicators of the departments and agencies, could lead to a whole new level of debate on the effectiveness of government organizations. It could transform the functioning of the Public Accounts Committee and the Crown Corporations Committee as they work towards ensuring accountability is achieved when public money is spent. 6.80 The Legislative Assembly, through its Public Accounts Committee (and possibly the Crown Corporations Committee), could have a role to play in improving annual reports. By examining specific reports and comparing them to the policy, the members could create an impetus for improved compliance with the policy. A model report might be a possible outcome from committee deliberations aimed at improving the annual report, “the major accountability document for the Legislative Assembly and the general public.”