Summary 2.1 The purpose of this chapter is to inform the Legislative Assembly of New Brunswick about the review we did on the public-private partnership (P3) for the Eleanor W. Graham Middle School and the Moncton North School. 2.2 In May 2008, the government announced that two schools, one in Rexton and one in Moncton, would be delivered using a P3 model. In October 2008, the government announced that it would consolidate the construction, maintenance, and rehabilitation of these schools into one project. The net present value (NPV) of the project, including payments to the private sector as well as quantified risks, was estimated at $93.9 million. 2.3 Our key findings and conclusions are listed in Table 2.1. The recommendations and the responses from various Departments are shown in Table 2.2. Table 2.1 Results in Brief Criterion Findings Conclusion Objective 1 - To determine the process for identifying the two school project as a potential P3. No associated criteria Unsupported P3 decision: We found no evidence that a formal preliminary analysis was performed to support the decision of adopting a P3 approach before it was publicly announced. We were unable to determine the rationale for the decision. Subsequent to the announcements, the Department of Supply and Services (the Department) decided to prepare a value for money (VFM) assessment to see whether the P3 approach would deliver value for money. Approval by Legislature long after the agreement signed: The project was not included in the capital estimates (i.e. for approval by the Legislative Assembly) until fiscal year 2010- 11, although the Province had already signed multi-year agreements with a successful bidder in September 2009. The commitment involves making annual payments of approximately $5.1 million for a 30 year period. The Legislative Assembly had no opportunity to debate this commitment in advance of the decision being made. In our view, government should obtain approval of the Legislature through the budget process before a multi-year P3 contract is signed. We concluded there was no formal process to support the identification of this project as a P3 candidate. Criterion Findings Conclusion Objective 2 - To assess the business case on which the Department’s decision to adopt the P3 approach for the two school project was based. All significant assumptions made by the Department should be supported and documented. We found that assumptions associated with discount and inflation rates, as employed in the VFM analysis, were supported. However we did not agree with the Department’s assumption to include an additional net present value (NPV) of $14.2 million for the traditional model to reflect the maintenance and life cycle deficit. The assumption was based on the expectation that the Province would not adequately fund these costs under the traditional model. Although we understand why this assumption was made, in our view, if the Province is willing to commit funds to maintain the school at documented standards under the P3 model, it should be willing to do the same under the traditional model. Treating one model differently in the VFM analysis distorts the comparison. Based on the VFM analysis, the Department concluded the P3 approach provided $12.5 million VFM to taxpayers for this project over the traditional approach. However, after adjusting for the effect of the maintenance and lifecycle cost assumption (NPV $14.2 million), the traditional model would deliver $1.7 million VFM over the P3 approach. We concluded that not all significant assumptions were supported and documented. We had concerns about the appropriateness of certain assumptions made, and the accuracy of the resulting comparison between the P3 and traditional models. Criterion Findings Conclusion Objective 2 - To assess the business case on which the Department’s decision to adopt the P3 approach for the two school project was based. All assumptions made by the VFM consultants should be reviewed and challenged (by the Department). There was little documentation available showing that the Department had reviewed the assumptions upon which the VFM analysis was based. We were unable to conclude on whether the assumptions were adequately reviewed and challenged by the Department, as sufficient documentation was not available to enable us to make a judgment. VFM analysis should be in-line with common industry practice. Consistent with common industry practice, the VFM report prepared for the project included: * a comparison between the P3 model and the traditional approach; * all relevant cost components (i.e. design, construction, financing maintenance, and operation); * assessments of risks; and * quantification of risks. However, we noted three areas where the VFM analysis was deficient in comparison with common industry practice. These included: * sensitivity analysis (i.e. only risk quantification was subject to this analysis); * timing of preliminary VFM analysis (i.e. the analysis was not completed prior to announcing the P3 project); and * reporting of VFM results (i.e. the Department did not comply with the government’s P3 protocols that require fair and transparent reporting). We concluded that the VFM analysis completed was partially in line with common industry practice. However, there were some significant deficiencies. Due diligence should be performed to review the value-for-money report. The only review was an informal one completed by departmental employees who had been involved in developing the VFM report, and therefore were not independent. We concluded that a formal independent due diligence review of the VFM report was not completed by the Department. Criterion Findings Conclusion Other observations No associated criteria We encountered two additional reportable matters during the completion of our work: 1) Were capital budgeting practices appropriate? Provincial budget restraint has led to the deferral of needed maintenance and rehabilitation work at schools. The Department, in its VFM analysis, recognized that deferral of maintenance leads to significantly higher future costs, and is therefore not a cost-effective long-term solution for the Province’s budget shortfalls. If government approves the construction of a new school, regardless of the construction method (P3 or traditional), the long term cost of operating and maintaining the facility should be factored into the decision at that time and protected in future budgets. There are mechanisms by which this can be done, such as statutory appropriations. A statutory appropriation should exist for ongoing maintenance and repair of provincially owned schools. Otherwise, unanticipated school closures like the 2010 mid-year school closure of Moncton High School and Polyvalente Roland-Pépin in Campbellton will continue. 2) Was due process followed in selecting project advisors? A process advisor was paid approximately $107,000 and a financial advisor was paid approximately $565,000 for their services on the project. Both advisors were engaged by the Department without calling for public tenders or any other form of competition. In our opinion, due process was not followed in engaging these project advisors. Table 2.2 Summary of Recommendations Recommendation Department’s Response 2.26 The Department of Supply and Services should conduct a preliminary assessment to identify the best procurement approach prior to a Cabinet decision on how to proceed (P3 or traditional approach). Department of Supply and Services response: Agreed. Government has established Partnership New Brunswick in the Department of Transportation. Since its inception Partnership NB has developed standard procedures for assessing potential P3 projects and determining if there is value for money in implementing a candidate project as a P3. The Department of Supply and Services is committed to working closely with Partnership NB to ensure capital projects are screened to determine the feasibility of P3 procurement approach. 2.31 The Department of Finance should have the government obtain approval of the Legislative Assembly, during the budget process, for future year P3 funding commitments in advance of entering into such contracts. Department of Finance response: We agree that Members of the Legislative Assembly should have the opportunity to debate P3 projects. To that end, the Minister of Finance has tabled the 2012-13 Capital Estimates, including a multi-year capital expenditure plan that clearly identifies the future year funding for all P3 projects. 2.71 The Department of Supply and Services should document the development of significant assumptions for VFM analysis, especially the assessment of their reasonableness. Department of Supply and Services response: Agreed. The Department of Supply and Services will work with Partnership NB to ensure key processes are documented appropriately. 2.72 The Department of Supply and Services should review assumptions made by its VFM consultant. Reviews and important discussions should be properly documented. Department of Supply and Services response: Agreed. The Department of Supply and Services reviewed all assumptions made by the consultants through iterative reviews of the risk matrix and drafts of the VFM report. We will work with Partnership NB to formalize the review and documentation of these assumptions on future projects. Table 2.2 Summary of Recommendations (continued) Recommendation Department’s Response 2.73 The Department of Supply and Services should obtain the discounted cash flow model from its consultant as part of the arrangement for future P3 projects. Department of Supply and Services response: Agreed. The Department of Supply and Services will work with Partnership NB to ensure this is incorporated into the Standard P3 process. 2.96 The Department of Supply and Services should perform a sensitivity analysis which includes all key variables in the project cost estimate process. Department of Supply and Services response: Agreed. The Department of Supply and Services will work with Partnership NB to ensure this is incorporated into the Standard P3 process. 2.97 The Department of Supply and Services should inform the public of key information in the P3 process. Department of Supply and Services response: Agreed. The Department of Supply and Services is committed to working with Partnership NB on the mechanism to ensure appropriate public disclosure. 2.103 The Department of Supply and Services should perform an independent due diligence review of the value for money assessment for each proposed P3 project. Department of Supply and Services response: Agreed. The Department of Supply and Services is committed to working with Partnership NB on the development of policies and procedures to be followed for the due diligence review of each P3 project. The Department is also committed to ensuring all policies and procedures are adhered to. 2.111 To ensure provincially owned schools are properly maintained over their useful lives, the Department of Supply and Services in cooperation with the Departments of Finance and Education should: Department of Supply and Services response: Agreed. The Department of Supply and Services has acquired and is in the process of implementing an Asset management System when completed it will be available to other Departments including Education. Department of Finance response: Each year the capital budget process carefully evaluates the funding required to maintain existing assets prior to consideration of funding for new projects. 1. develop and implement an asset management system that provides for and prioritizes multi-year maintenance and capital repair needs of the schools; and 2. implement budgeting measures to protect the long-term funding stream required for sufficient ongoing maintenance of the schools. Table 2.2 Summary of Recommendations (continued) Recommendation Department’s Response 2.117 The Department of Supply and Services should tender or solicit multiple fee estimates when engaging advisors for P3 projects, given the significant cost of these services. Department of Supply and Services response: Agreed. While the Department of Supply and Services adhered to all requirements of the Public Purchasing Act, the Department recognizes the value of soliciting multiple proposals and will incorporate this as good practice. Background 2.4 In recent years, the government of New Brunswick has increasingly used the public-private partnership (P3) approach to deliver public infrastructure. Examples include the twinning of Route 1, the Moncton Law Courts, and more recently the new psychiatric hospital in Restigouche County. In February 2011, the government confirmed the creation of a new division within the Department of Transportation called Partnerships New Brunswick. See Appendix 1 for more information on this organization. 2.5 A public-private partnership is a partnership arrangement in the form of a long-term performance-based contract between the public sector and the private sector (usually a team of private sector companies working together) to deliver public infrastructure for citizens . P3 projects typically include both a capital component and an ongoing service delivery component of non-core services for a specific period. The private sector partner in a P3 project often owns the infrastructure for the term of the contract and provides contracted services, as is the case for the P3 project under review. It receives periodic payments from the public sector partner once operation of the infrastructure has commenced, contingent on the private sector partner’s performance in supplying the services. Typically, the ownership of the asset is transferred to the public sector at the end of the agreement. 2.6 The pricing of risk, and the subsequent inclusion of that costing of risk into the financial analysis supporting a decision to proceed with a P3, is a huge factor in determining whether a P3 "gets the green light." One of the key rationales offered by P3 proponents is that the private sector can manage certain risks much more effectively than the public sector. This makes it more economical for the government to seek out P3 arrangements. 2.7 In May 2008, the government announced during a presentation of the 2008-09 Main Estimates of the Department of Education that two schools - Eleanor W. Graham Middle School in Rexton and Moncton North School in Moncton - would be built using a public-private partnership. Subsequently, in October 2008, the government announced the two schools would be packaged into one project. The net present value (NPV) of the project, including payments to the private sector as well as quantified risks, was estimated at $93.9 million. The intention was to have a private sector partner design, construct, finance, operate and maintain both schools. 2.8 The P3 for the two schools followed a two-stage selection process. A request for expressions of interest was issued in December 2008, from which a short list of proponents was selected. Then the proponents in the short list were invited to participate in the second-stage request for proposals. The development firm Scotia Learning Centres Inc. was selected as the preferred proponent in August 2009. Shortly after, a special interest entity called Brunswick Learning Centres Inc. was created by the firm for the purpose of this P3 project. The final Project Agreement was signed between the Province and Brunswick Learning Centres Inc. on 11 September 2009. 2.9 Eleanor W. Graham Middle School covers 5,574 sq. metres (60,000 sq. feet) and accommodates 350 students from Grades 6 to 8. The Moncton North School covers 10,219 sq. metres (110,000 sq. feet) and accommodates 650 students from kindergarten to Grade 8. The key dates and milestones are listed in Table 2.3. 2.10 Construction for both schools was completed ahead of schedule with less than 1% of construction costs in change orders. This percentage is much lower than typical projects delivered through the traditional approach. Table 2.3 Summary of key dates and milestones Date Event May 2008 Government announced the two schools would be built using a public-private partnership October 2008 Government announced the two schools would be packaged into one project December 2008 Request for expressions of interest was issued August 2009 The development firm Scotia Learning Centres Inc. was selected as the preferred proponent September 2009 Final Project Agreement signed November 2009 Construction started August 2010 Eleanor W. Graham Middle School was substantially completed September 2010 Eleanor W. Graham Middle School opened October 2010 Moncton North School was substantially completed November 2010 Moncton North School opened 2.11 The Province has made a significant financial commitment to the private sector partner. According to the winning proposal, the Province will make annual payments of approximately $5.1 million over the 30 year operating period which covers construction, operation, maintenance and rehabilitation. As per the project agreement, the payments are subject to an annual increase based on the consumer price index. As well, the payments will be adjusted in case the private sector partner fails to perform the contracted services up to the service standards set in the agreement. This will require diligent monitoring on behalf of the Department of Education. Objectives and Scope 2.12 Our objectives were: * to determine the process for identifying the two school project as a potential P3; and * to assess the business case on which the Department’s decision to adopt the P3 approach for the two school project was based. 2.13 Our work included: * interviews with staff of the Departments of Supply and Services (DSS), Education, and Finance; * review and analysis of project related documents which were produced by DSS, the Department of Finance and the VFM consultant engaged by DSS; * review of the Province’s guidelines and legislation with respect to public-private partnerships; and * research into the reports and practices of public- private partnerships in other jurisdictions, including other Canadian provinces, the United Kingdom and Australia. Why we chose this project 2.14 We decided to focus on this P3 school project for two reasons. First, this P3 contract is long term in nature and imposes a significant financial obligation to the Province for 30 years. 2.15 Secondly, during our scoping stage we became aware the Province was considering other P3 school projects. We believe our findings and recommendations could provide information critical to the decision process for those projects. Following the change in government in the fall of 2010, our Office was asked to present an interim report on our findings. We presented a short preliminary report to the Department which in turn was shared with the Board of Management (the Board). The Department recommended to the Board, in part due to our interim findings, to proceed with the construction of the new Riverview school and another new Moncton North school using its traditional procurement method rather than a P3. Prior to the Board's direction, the Department considered constructing Moncton North and Riverview schools using a P3 model. Detailed Observations Objective One - Determine the Decision Making Process 2.16 Our first objective was to determine the process for identifying the two school projects as potential P3’s. 2.17 The Eleanor W. Graham Middle School/Moncton North School P3 project represents a significant investment in the public education system of New Brunswick. Due to its long term nature and high dollar value, it is critical to assess the costs and benefits of all reasonable and relevant alternative procurement models. It is also important for legislators and taxpayers to know that all plausible alternatives have been considered. This type of assessment allows the decision makers to choose the option that offers the best value for money (VFM). VFM is defined by Her Majesty’s Treasury (HM Treasury) as: the optimum combination of whole-of-life costs and quality (or fitness for purpose) of the good or service to meet the user’s requirement …. In assessing and delivering VfM it is also important to note that VfM is a relative concept which requires comparison of the potential or actual outcomes of alternative procurement options. 2.18 The most common approach to evaluate different procurement alternatives is a VFM assessment. The main purpose of a VFM assessment is to compare the estimated full costs to deliver the required infrastructure and services using a traditional procurement method with that of the same project using a P3 model. 2.19 This assessment should be carried out early in the process, as is recommended by other jurisdictions both nationally and internationally. The Value for Money Assessment Guidance published by the HM Treasury in November 2006 recommends “procuring authorities should begin detailed assessments of the VfM of…. projects at the earliest stage possible.” 2.20 In its National PPP Guidelines, Infrastructure Australia states: It is important that procurement alternatives are analyzed thoroughly so that a robust recommendation can be made of a preferred procurement method. A rigorous analysis simplifies the decision- making process and ensures that the best procurement method, whether a PPP or an alternative, is pursued. 2.21 For infrastructure projects managed by Infrastructure Ontario (IO), the IO Board does not approve release of Request for Proposals (RFP) unless, among other factors, positive VFM is demonstrated by procuring a project using alternative financing and procurement, according to a document called “Assessing Value for Money – A Guide to Infrastructure Ontario’s Methodology”. Partnerships BC also recommended in 2009 the best practice for the quantitative analysis of infrastructure project procurement options is to present the objectives, scope, program delivery options analysis and recommendation for the preferred service delivery option before the final decision can be made. Unsupported P3 Decision 2.22 In May 2008, the government announced it would deliver the Eleanor W. Graham Middle School/Moncton North School project using a P3 approach. We found no evidence, however, that any kind of formal preliminary analysis was performed to support the P3 decision made by the Cabinet. We were advised there was some informal discussion among senior officials in the Department of Finance and the Department of Supply and Services during the budget process, but we were unable to determine the rationale for this decision. 2.23 Additionally, the government announced in October 2008 that the two schools would be packaged in one project and the model of P3 chosen was design-build- finance-maintain-operate (DBFMO). We found no evidence that this decision was supported by any type of formal assessment showing DBFMO as the most cost- effective form of P3 for this project. 2.24 We found that neither the Department of Supply and Services, which executed the decision, nor the Department of Education, which manages the agreement after the schools open, were officially involved in the decision making process with respect to the initial determination of the procurement method. 2.25 Subsequent to the P3 decision, the Department decided to conduct a value-for-money (VFM) assessment. The assessment was prepared after the decision to follow the P3 approach was made. Typically, value-for-money of alternatives is evaluated and compared prior to deciding between the alternatives (not after the fact as the government did in this case). Since the P3 decision was already made when the VFM analysis began, there is a potential risk that the VFM analysis could be biased to support the Cabinet’s P3 decision. Recommendation 2.26 The Department of Supply and Services should conduct a preliminary assessment to identify the best procurement approach prior to a Cabinet decision on how to proceed (P3 or traditional approach). Approval by Legislature Long After the Agreement Signed 2.27 We also noted that the project was not included in the capital estimates until fiscal year 2010-11, although the government announced the decision to build the two schools through a P3 in May 2008, and the final project agreement was signed in September 2009. In other words, the government was already committed to this multi- million dollar project long before the Legislative Assembly had the opportunity to debate it or approve the associated multi-year financial commitment. 2.28 The Financial Administration Act states: 32(1)No contract is to be made by which money is to be paid during the fiscal year in which the contract is made unless there is a sufficient unencumbered balance in the applicable appropriation. 32(2)Every contract made by the Province after the commencement of this section provides for payment of public money is deemed to contain the following term: No payment is to be made by the Province under this contract in any fiscal year unless an appropriation against which the payment is to be charged is made in that fiscal year. 2.29 Our interpretation of the Act is that, in subsection 32(1), contracts should not be entered into which will require a payment in the fiscal year unless there is an appropriation by the Legislature sufficient enough to pay for it. Subsection 32(2) states no payment is to be made for a contract unless there is an appropriation. The Act is not clear with respect to situations where contracts are entered into in one year but contract terms and conditions are such that payment will not occur until a future year. As a result contracts can be entered into, resulting in a commitment of future year funding without having an appropriation until future years. It is only when an appropriation is required that such amounts are included in the Main Estimates and the annual budget process, hence subject to public debate in the Legislature. 2.30 In our view, government should obtain approval of the Legislature through the budget process before multi-year P3 contracts are signed. Recommendation 2.31 The Department of Finance should have the government obtain approval of the Legislative Assembly, during the budget process, for future year P3 funding commitments in advance of entering into such contracts. Conclusion on Objective One 2.32 We concluded there was no formal process to support the identification of this project as a P3 candidate. Objective Two - Assessing the VFM Analysis 2.33 Our second objective is to assess the business case supporting the Department's decision to adopt the P3 approach for the two school project. 2.34 We found the Department did not prepare a business case for this project. It conducted a VFM analysis. Based on the results of the VFM analysis, the Department recommended that the government enter into a P3 contract with the successful proponent. 2.35 We developed four criteria to guide our work under this objective: 1. all significant assumptions made by the Department should be supported and documented; 2. all assumptions made by the VFM consultants should be reviewed and challenged (by the Department); 3. VFM analysis should be in-line with common industry practice; and 4. due diligence should be performed to review the value-for-money report. 2.36 Essentially, we wanted to look at the assumptions integral to the VFM analysis that supported the government's P3 approach for this project to determine if: * the major assumptions were supported and documented; and * the Department reviewed and challenged any of the assumptions developed by the consultant. 2.37 We deal with these four criteria in the sections that follow. Prior to entering into this discussion, however, we believe it would be useful to identify the key terms underlying the notion of value-for-money as calculated in the VFM analysis. VFM Analysis Process 2.38 The VFM analysis is the key component of the decision making process. A brief explanation of key terms is provided in Appendix 2. In order to conduct this VFM analysis, the Department has to create a public sector comparator (PSC). The PSC is an estimate of the "whole- of-life" cost of a hypothetical situation. That is, what the cost of the Eleanor W. Graham Middle School/Moncton North School project would be if it had been delivered by government through the traditional procurement approach. Whole-of-life cost includes designing, constructing, financing, maintaining and operating both schools during the 30 years and 11 months contract period in accordance with the required output specification. 2.39 In addition to looking at the costs of this PSC, the Department also had to estimate the total whole-of-life costs to the Province of delivering the same project to the identical specifications using a P3 approach. The difference between the estimated total project costs under each model generates a remainder the Department and the advisor call VFM. The government should only have adopted the P3 approach if the VFM analysis demonstrated it was the option that provided the best value for money. 2.40 The cash flow streams differ, both in terms of dollar amounts and timing, between the PSC and the P3 approach. For example, in the PSC, the Department would make a series of progress payments throughout the construction of the two schools. Once the two schools open, the Department of Education would begin making payments to various parties to operate and maintain the schools. In the P3 approach, the Department of Education will make a series of periodic payments covering the whole cost of two schools, and the vendor's profit component, throughout the 30 years and 11 months contract period. 2.41 The most common method used to analyze these differences in cash flow streams is the discounted cash flow analysis. This method follows the concept of time value of money (i.e. a dollar today is worth more than a dollar in the future). In using this method, the respective cash flows must be expressed in dollars as at a single date in time, known as the base date, so that cash flows that occur in different periods can be added together into one total amount (i.e. net present value or NPV). The NPV of the PSC and the P3 approaches can then be compared to determine which one provides better value for money. 2.42 This NPV approach is built upon a number of key assumptions. These include the interest rate to be used, the costs of various components, the inflation rate, and the costs of various risks. A change in one of these major assumptions can change the result significantly. 2.43 In the case of the Eleanor W. Graham/Moncton North schools P3 project, both the Department and the Department’s VFM consultant contributed to the key assumptions used in the analysis. This means it is essential that any assumptions the Department contributed are both well documented and well supported. For those instances where the Department’s VFM consultants developed the assumptions on their own, it is crucial that the Department review and challenge them. This notion lies behind our first two criteria for this objective: 1. all significant assumptions made by the Department should be supported and documented and 2. all assumptions made by the VFM analysis consultants should be reviewed and challenged (by the Department). Criteria 1 & 2 - Quality of the Assumptions 2.44 We have chosen to look at criteria one and two under the same heading. The quality of the assumptions will affect the result of the NPV calculation of the cash flow streams. If the assumptions made by either the Department or its consultant are faulty, then the result of the value-for- money analysis will be misleading. One invalid assumption could change the conclusion. Decision makers relying on the analysis could then make a wrong decision, committing the Province to a 30 year contract that ties it to a more expensive option. Given this, we believe it is important to examine all the key assumptions together to determine their validity in order to appropriately address the notion of value-for-money. What Were the Key Assumptions? 2.45 We prepared a listing of key assumptions and presented it to officials from the Department. They agreed with our assessment. 2.46 We categorized these key assumptions under the headings: * total design and construction costs; * maintenance and life cycle cost deficit; * discount rate; and * inflation rate. Re-evaluate the Results 2.47 After reviewing the key assumptions we re-evaluated the results of the VFM analysis. 2.48 The report prepared by the VFM consultants shows that the P3 approach would provide a net benefit of $12.5 million to the Province when compared to the traditional approach. However, after considering our adjustment as shown in Table 2.4 below, the traditional approach would have resulted in net benefit to the Province of $1.7 million. The major items in the table are discussed in detail in the following sections. Table 2.4 VFM Assessment Comparison Between the Department and Office of Auditor General (in millions, NPV as of 2009) VFM Assessment Comparison Between the Department and Office of Auditor General (in millions, NPV as of 2009) Cost component Total costs per the Department’s assessment Total costs after OAG adjustment PSC P3 PSC P3 Misc. project costs and land cost $3.0 $3.0 Base design and construction cost estimates 41.1 41.1 Transferred risks 9.9 9.9 Retained and 50% shared risks 26.4 24.0 26.4 24.0 Maintenance and life cycle related costs 23.71 9.51 Others 2.1 2.1 Cost of winning proposal 74.6 74.6 Provincial income taxes (0.5) (0.5) Adjustment for quality (4.2) (4.2) Total costs $106.4 $93.9 $92.2 $93.9 VFM P3 offers $12.5 million in VFM over the PSC PSC offers $1.7 million in VFM over the P3 1. See details in Table 2.6 and section “Maintenance and life cycle costs” Total Design and Construction Costs for the PSC 2.49 The Department estimated it would cost $41.1 million (base cost estimate) including typical contingencies of $7.0 million or 20%, if the project were to be delivered using a traditional procurement approach or PSC. The details are listed in Table 2.5 below: Table 2.5 The Department’s Base Cost Estimate for the Traditional Procurement Model The Department’s Base Cost Estimate for the Traditional Procurement Model Education Program 111,902 ft2 x 1.47% Systems & Services (Dept of Education Target) 164,500 ft2 164,500 ft2 x $180/ft2 =$29.6 million $29.6 million x 1.12% Design Contingency =$33.2 million $33.2 million x 1.05% Leadership in Energy and Environmental Design (LEED) Allowance =$34.8 million + $34.8 million x 8% Design Fees + Expense =$2.8 million + $34.8 million x 5% Tender Contingency =$1.7 million + $34.8 million x 5% Construction Contingency =$1.7 million + Total Design, Tendering, Misc. and Construction *: =$41.1 million *Plus insurance costs during construction ($53,000), not including land costs. 2.50 The Department also pointed out that undertaking this project using the traditional model would involve other risks such as cost overrun, in addition to the contingencies taken into consideration in the base cost estimate in Table 2.5. 2.51 The Department and its VFM consultant undertook a risk analysis of the project. The Department provided the dollar impact and the probability of occurrence of each risk. Using this information, the VFM consultant quantified the estimated risk outcome. 2.52 The risks associated with design and construction for the PSC were quantified at a net present value (NPV) of $33.4 million, which is 81% of the $41.1 million base design and construction estimate. We did not find evidence that the Department compared the total amount of quantified risk with actual experience from prior school construction projects to assess the reasonableness. In our view, historical cost information is an important tool to validate project costs including estimated risk costs. Maintenance and life cycle costs 2.53 There are three components involved in our re-evaluation of maintenance and life cycle costs for the PSC model. These are: * the relative NPV’s of maintenance cost ($5.0 million) and lifecycle cost ($4.5 million) as required by the RFP; * the NPV of the maintenance and lifecycle deficit ($21.4 million) projected to arise because the Department expected the Province would not adequately fund these costs for the PSC model; and * the NPV’s of the funds the Department expects the Province would typically invest to maintain ($1.7 million) and rehabilitate ($0.6 million) the facilities under the PSC model. 2.54 In its value-for-money analysis, the Department assumed that the traditional approach to operating the schools would result in underfunding of both maintenance and life cycle costs. This was based upon the historical provincial practice of deferring necessary maintenance and rehabilitation on public infrastructure. Therefore, the Department estimated the NPV’s of maintenance and lifecycle costs under the traditional model to be only $1.7 million and $0.6 million respectively. Both figures are much lower than the required standard in the RFP totaling $9.5 million as shown in Table 2.6. 2.55 The Department then calculated a maintenance and life cycle deficit associated with this expected underfunding which totaled $21.4 million. The Department based the calculation of this figure on two further assumptions. First, each dollar not spent on necessary maintenance today will result in $4 of required spending in 10 years. Second, each dollar not spent on necessary life cycle costs today will result in $4 of required spending in 15 years. We did not attempt to verify the accuracy of these multipliers. 2.56 Based upon these figures, the Department estimated that the NPV of maintenance and life cycle costs under the traditional approach would total $23.7 million, as shown in Table 2.6, which is $14.2 million more than the $9.5 million total costs of maintenance and lifecycle as required by the RFP. 2.57 We understand why this assumption was made, however, we do not believe that the possibility of a future government decision (i.e. to not approve adequate funding in the budget to cover necessary maintenance and lifecycle costs) provides sufficient rationale for adding a net $14.2 million to the cost of the PSC model for purposes of the VFM analysis. In our view, if the Province is willing to commit funds to maintain the schools at a documented standard under the P3 approach, then it should be willing to do the same under the PSC model. Treating one model differently than the other in the VFM analysis distorts the comparison. 2.58 Therefore, as shown in Table 2.6, we have concluded that maintenance and life cycle costs associated with the PSC model in the VFM analysis have been overstated by $14.2 million. Table 2.6 Comparison of Costs Regarding Maintenance and Life Cycle for PSC ($ millions, NPV as of 2009) Comparison of Costs Regarding Maintenance and Life Cycle for PSC ($ millions, NPV as of 2009) Department estimate Total costs related to maintenance and life cycle $23.7 OAG estimate Maintenance cost as required by the RFP and project agreement 5.0 Life cycle cost as required by the RFP and project agreement 4.5 Total costs related to maintenance and life cycle $9.5 Difference – overstatement of maintenance and life cycle costs for PSC $14.2 Discount rate 2.59 As we mentioned previously, the net present value (NPV) calculation depends primarily on two main inputs: the estimated cash flows of a project, and the rate at which these cash flows are discounted (the discount rate), from future periods to a common base date. 2.60 In carrying out NPV analysis, the choice of discount rate is important as it can have a significant impact on the outcome. If an inappropriate discount rate is selected, there is a significant risk that it could result in the wrong choice of procurement method. 2.61 In accordance with governmental policy , the discount rate used by the Department was benchmarked to the government of New Brunswick’s long term borrowing rate with a similar time frame. The Department informed the VFM consultant on 10 March 2009 that the discount rate to be used was 5.3%. The Department did not have documents supporting the rate decision. We were told the Department obtained the rate quote verbally from the Department of Finance. The Department of Finance subsequently confirmed the estimated 30-year government borrowing rate on 10 March 2009 was 5.27% as quoted by one investment dealer. The Department of Finance also stated that the 5.30% rate used was likely an average from several investment dealers. 2.62 Based on our research, different infrastructure organizations are using different methodologies in determining the discount rate. For example, Partnerships BC is basing the discount rate on the cost of capital for a particular project. Infrastructure Ontario uses the most current weighted average cost of capital, which, in its view, is the simple average of the long-term provincial debt (bonds with terms of one to 30 years). To neutralize the effects of daily fluctuations on the discount rate, a ten-day rolling average of this simple bond yield average is used as the standard discount rate. Internationally, Infrastructure Australia recommends the use of different discount rates under the PSC and the P3 depending on which party will bear the systematic risk. 2.63 Given the discount rate chosen was in accordance with policy and the fact that the VFM consultant agreed this was the correct rate, we have no concern with the rate chosen. However, given practices in other jurisdictions and the fact the related Administration Manual Policy (AD-6701) was last updated in 1977, the Department should consider alternatives in future P3 initiatives. 2.64 Upon our request, the consultant calculated the impact on the NPV of changing the discount rate by +/- 1% and the NPV varied by less than $2 million. This indicated that the discount rate variations result in immaterial changes in results. The VFM consultant did not provide the discounted cash flow model to the Department so we were unable to assess its appropriateness. Inflation Rate 2.65 The annual inflation rates assumed in this case are: * 2.7% annually for design and construction costs, based on Nova Scotia construction historical inflation data; and * 2.0% for other elements, based on the Consumer Price Index (CPI) forecast from the Conference Board of Canada and the long-term targeted CPI rate by the Bank of Canada. 2.66 The rates used were supported appropriately; however, there is no evidence this assumption was reviewed and challenged by the Department. In fact, when we requested the backup for the rates, the Department had to ask the VFM consultant to provide supporting documents. The Department had no written policies and procedures to guide it in reviewing the assumptions. Documentation 2.67 The Department did not document its work in preparing and reviewing the VFM report adequately. Departmental management indicated that its staff met several times with its VFM consultant to discuss the VFM analysis including the risk assessment. However, the Department was unable to provide sufficient documentation to support its statement. 2.68 The Department showed us two draft versions and the final version of the risk assessment. Allocation and quantification of risks were revised based on the comments from the Department. This demonstrates that the Department staff participated in the process of assessing risks. 2.69 However, the Department did not document any discussions. We were unable to determine what was discussed and why some changes were made. Conclusion on Criteria 1 and 2 2.70 Not all significant assumptions were supported and documented. We had concerns about the appropriateness of certain assumptions made, and the accuracy of the resulting comparison between the P3 and traditional model. We were unable to conclude on whether the assumptions and risk assessment were adequately reviewed and challenged by the Department as sufficient documentation was not available to enable us to make a judgment. Recommendations 2.71 The Department of Supply and Services should document the development of significant assumptions for VFM analysis, especially the assessment of their reasonableness. 2.72 The Department of Supply and Services should review assumptions made by its VFM consultant. Reviews and important discussions should be properly documented. 2.73 The Department of Supply and Services should obtain the discounted cash flow model from its consultant as part of the arrangement for future P3 projects. Criterion 3: VFM Analysis was in-line with Common Industry Practice 2.74 P3s are relatively new initiatives in New Brunswick, while some other jurisdictions nationally and internationally have a more mature P3 market. These other jurisdictions have published guidance regarding the P3 general process and methodology to assess VFM. These documents were developed over time and based on a large number of real infrastructure projects. It is important for the Department to follow commonly used industry practice. 2.75 We reviewed the final version of the VFM report prepared by the Department’s VFM consultant. In general, the VFM analysis included in the report was consistent with many, but not all, of common industry practices. 2.76 Based on our research, we believe a VFM analysis should include as a minimum: * For comparison purposes, a Public Sector Comparator (PSC) to establish the total cost of the project under a traditional procurement approach. The most likely and achievable procurement approaches should be assumed in the PSC, so that a realistic cost comparison between the PSC and the P3 can be achieved. * Risk assessment including risk allocation. This allows total costs of PSC and P3 models to be adjusted to reflect the impact of risks. 2.77 Additionally, all adjustments made to the total costs of the PSC and P3 should be supported. 2.78 We found that the PSC prepared in the VFM analysis included all relevant cost components throughout the life cycle of the project, including design, construction, financing cost, operating cost, regular maintenance, major maintenance and cyclical renewal required to maintain the service potential of the facilities. 2.79 The Department assumed that under the traditional approach, the project will be realized through multiple contracts (i.e. the Province awards one or more design contracts to external professionals). The construction is divided into different lots that will be performed by different private construction companies. The operation and maintenance would be carried out through short-term contracts. 2.80 Given that the project includes two schools, the approach adopted appears to be reasonable. We were advised by the Department that two new schools delivered using the traditional construction method in Boiestown and Restigouche East adopted a similar approach. 2.81 Risk assessment was performed. Risks were categorized and allocated between the Province and the private partner. Each risk was quantified based on its likelihood, significance and dollar impact. The total value of the risks was included in the total cost of the PSC, as the Province would retain all the risks under the traditional procurement approach. 2.82 One significant adjustment was made to the total cost of the P3. The adjustment for quality was deducted from the total cost of the P3. It was supported. The private partner’s proposal exceeded the minimum requirements of the design standards as set out in the RFP. The Department decided it was willing to pay for the additional features and quality. To allow for a valid comparison between the PSC and P3, the cost of the proposal was adjusted for a net present value (NPV) of $4.2 million. Areas Where VFM Analysis Not Conducted Using Common Industry Practice 2.83 We did notice three areas where the VFM analysis was not consistent with common industry practice. These are in the areas of sensitivity analysis, timing of the preliminary VFM analysis, and reporting of VFM results. Sensitivity Analysis 2.84 It is important to test the impact of changes in key assumptions used in the VFM assessment, as the project contract is long-term in nature. It is generally difficult to make assumptions with real precision in such a case. Sensitivity analysis would help the decision makers to understand the significance of changes to key variables in terms of the project costs of both approaches. 2.85 One variable (e.g. inflation, discount rate) would be changed while others are being held constant in a typical sensitivity test. In this way, the decision makers can determine how sensitive the estimate of the cost of the project is to changes in that particular variable. 2.86 It is common industry practice in Canada and internationally to perform sensitivity analysis on key variables such as: discount rate, estimated design and construction costs, inflation rate, etc. 2.87 The only sensitivity analysis documented for this project was on the risk quantification in the VFM assessment prepared by the Department’s VFM consultant. It increased and decreased the value of all risks by 25% for both the PSC and P3 to see how sensitive the total cost was to changes in the value of risks. The other key variables were not tested in terms of sensitivity, therefore it is impossible for the decision makers to identify how changes in the key variables could affect the overall outcome. Such analysis would have helped decision makers make a more informed decision. Timing of Preliminary VFM Analysis 2.88 Under objective one, we indicated that the government did not carry out a formal preliminary VFM analysis prior to announcing that the Province would proceed with the Eleanor W. Graham/Moncton North School project using a P3 approach. 2.89 In paragraph 2.19, we stated that the VFM assessment should be carried out early in the process. We noted how both domestic and international jurisdictions support this viewpoint. However, the Cabinet did not follow this common industry practice. Informing the Public of Key Information in the P3 process 2.90 The Province has protocols for P3’s which establish the ground rules that all P3’s should follow. The protocol states that the procurement process must be fair and transparent. We believe it is essential for the government to ensure the transparency in P3’s through timely disclosure of procurement related information to the public while protecting commercially sensitive information of a private partner. 2.91 Some information was disclosed through the process for this project. The Request for Expressions of Interest was advertised on the New Brunswick Opportunities Network, and the preferred proponent was announced through a news release. Additionally, the Department issued the VFM report prepared by the VFM consultant in response to a Right to Information request rather than under the direction of a general policy, but significant key information was concealed through shading of text. To the best of our knowledge, this is the only information the Department provided in this regard. 2.92 Other key information was not made available to the public, especially the VFM report which includes detailed analysis of how the P3 method would deliver value for money. If the government does believe a P3 approach is the better alternative for this project, it should fully inform the public by disclosing the VFM report. In our view, the current practice of the Department is not consistent with the transparency requirement in the Province’s P3 Protocols. 2.93 Partnerships BC published an updated version of “Procurement Related Disclosure for Public Private Partnerships” which outlines what should be disclosed at various stages of procurement. It recommends some key information should be disclosed, such as the request for quotations, short-listed parties, the request for proposals, preferred proponent, the VFM report, and the final project agreement. Infrastructure Ontario has similar disclosure practices. With the disclosure of the key documents, including the rationale for various important decisions made, the public will be well informed during the entire process. 2.94 Conversely, in New Brunswick there is no policy regarding procurement related disclosure for P3 projects. Given the Department has entered into other P3 agreements, it is becoming increasingly important to establish a standard disclosure policy. A consistent approach will not only help ensure transparency but also inform private partners in advance as to what type of information will be disclosed. Conclusion on Criterion 3 2.95 The VFM analysis completed was partially consistent with common industry practice. However, there were some significant deficiencies. Recommendations 2.96 The Department of Supply and Services should perform a sensitivity analysis which includes all key variables in the project cost estimate process. 2.97 The Department of Supply and Services should inform the public of key information in the P3 process. Criterion 4: Due Diligence Should be Performed to Review the Value for Money Report 2.98 The VFM report is one of the most important documents in the decision making process regarding which procurement option the Department should follow. An independent review of the VFM report is critical, as that report provides the key quantitative evidence on which approach would deliver the best value for money for the Province. 2.99 We believe the Department should be responsible for the accuracy of the VFM report. The report should be reviewed independently either by staff members of the Department who have not been involved in the VFM analysis process or by an independent third party. The Department should also document the results and analyses. 2.100 We found there was no independent due diligence review performed to ensure the reasonableness and completeness of the assumptions and conclusions. According to the Department, the only staff members who were involved in the VFM analysis informally reviewed the paperwork of the VFM consultant. However, no documented evidence of such reviews could be provided. 2.101 Furthermore, the Department does not have written policies and procedures on how due diligence should be performed. Conclusion on Criterion 4 2.102 A formal independent due diligence review of the VFM report was not completed by the Department. The informal review conducted by the Department staff was not clearly documented. Recommendation 2.103 The Department of Supply and Services should perform an independent due diligence review of the value for money assessment for each proposed P3 project. Other Detailed Observations 2.104 During this review, we encountered two additional reportable matters of significance and of a nature that should be brought to the attention of the Legislative Assembly (Section 15(2) Auditor General Act). These matters are being reported under the headings of: 1. Were capital budgeting practices appropriate? 2. Was due process followed in selecting project advisors? Were Capital Budgeting Practices Appropriate? 2.105 If government approves the construction of a new school, regardless of the construction method (P3 or PSC), the long term cost of operating and maintaining the facility should be factored into the decision at that time and protected in future budgets. There are mechanisms by which this can be done such as statutory appropriations. For example, the 2011-12 Main Estimates show that the New Brunswick Highway Corporation has a statutory appropriation of over $36.9 million for maintenance and other related costs for designated sections of the highway network. 2.106 A similar statutory appropriation should exist for ongoing maintenance and repair of provincially owned schools. Otherwise, unanticipated school closures like the 2010 mid-year school closure of Moncton High School and Polyvalente Roland-Pépin in Campbellton will continue. 2.107 We also noted the Department of Transportation’s Asset Management System, initiated in 2009, which is used for long term planning and budgeting in maintaining highway infrastructure, with the objective of identifying the “right treatment at the right time for road improvements at the lowest cost to the taxpayer ”. A similar approach is needed for provincially owned schools. 2.108 The deferred maintenance problem of provincial schools came to light in connection with the 2010 mid-year school closures of Moncton High School and Polyvalente Roland- Pépin in Campbellton. 2.109 The 2005 Report of the Auditor General regarding Education – Facilities Management contained the following recommendation: * We recommend the Department of Education annually advise the government of: i. The estimated level of expenditures necessary to appropriately maintain school facilities; and ii. The major repairs that have been deferred because of limited funding and the projected risks associated with deferring the major repairs. 2.110 Our 2009 follow up work indicated the Department of Education was investigating asset management systems in a multi-year planning process in order to address these issues. The status of this recommendation was that although the Department agreed with the recommendation, they had not yet implemented it. Recommendation 2.111 To ensure provincially owned schools are properly maintained over their useful lives, the Department of Supply and Services in cooperation with Departments of Finance and Education should: 1. develop and implement an asset management system that provides for and prioritizes multi-year maintenance and capital repair needs of the schools; and 2. implement budgeting measures to protect the long term funding stream required for sufficient ongoing maintenance of the schools. Was due process followed in selecting project advisors? 2.112 During fiscal years 2008-09 to 2010-11, the Department engaged a process advisor and a financial advisor to aid in the development of the project agreement, the procurement process, and the VFM analysis. The breakdown of the expenditures is listed in Table 2.7 below: Table 2.7 Breakdown of Fees Paid to Process Advisor and Financial Advisor Breakdown of Fees Paid to Process Advisor and Financial Advisor Moncton North/Eleanor Graham Description of main duties 2008-09 2009-10 2010-11 Total Process Advisor Ensure the procurement process is fair $17,301 $90,287 $0 $107,588 Financial Advisor Conduct the VFM assessment and provide financial expertise throughout the procurement process $0 $526,530 $38,540 $565,070 2.113 The process advisor was doing another project for the Department at the same time. The Department stated it was trying to realize some cost savings by getting the same advisor to do both projects. 2.114 The financial advisor provided financial consulting services particularly in the area of Value for Money analysis. As per the Department, the financial advisor had P3 expertise in other jurisdictions in Canada and globally. In the financial advisor’s proposal dated 17 July 2009, the fees anticipated for the VFM assessment up to 31 August 2009 were $105,000. As per the Department and the financial advisor, the budget was respected. The rest of the $565,070 fee paid was for additional financial advisory services throughout the procurement process. 2.115 In our opinion, due process was not followed to engage project advisory services, although it is unclear whether the Department is compelled to solicit multiple bids in hiring P3 advisors. The Department believes it complied with the Public Purchasing Act. 2.116 One could interpret the clause 27 (c) of the New Brunswick Regulation 94-157 under the Public Purchasing Act to say that both of these contracts were exempt from public tender, as both were with chartered accountants. Of course, one could interpret section 27 (c) more narrowly to apply it only to audit services offered by a chartered accountant, rather than consulting services. In any case, it is quite conceivable that these consulting services could have been provided by a broader range of practitioners and consultants other than chartered accountants. Further, if the Department needed to rely on a certain narrow range of firms, there would have been nothing in the Public Purchasing Act preventing it from obtaining competitive fee estimates from various service providers on this shorter list. Indeed, our understanding is that there are other consulting firms which are able to provide similar services. Recommendation 2.117 The Department of Supply and Services should tender or solicit multiple fee estimates when engaging advisors for P3 projects, given the significant cost of these services. ? Appendix 1: Partnerships New Brunswick 2.118 The government confirmed in February 2011 the establishment of Partnerships New Brunswick, a division within the Department of Transportation, to serve all departments and agencies of the Province as well as other entities such as municipalities, institutions and other provinces by providing support to clients in the VFM assessment and delivery of public-private partnership projects. 2.119 The services Partnerships New Brunswick can provide are: * Screening potential P3 projects * Business Case (VFM) assessments * Market sounding * Advising on retention of technical, legal, and financial advisors * Developing and sharing standard procurement processes and documentation * Providing advice on procurement processes and documentation: ? Risk transfer, project definition and scope ? RFQ and RFP ? Evaluation of proponent qualifications and proposals ? Development of Reference Cases ? Development of project agreements and requirements 2.120 During an interview with the staff of Partnerships New Brunswick, we were informed that all the services it is offering are optional at this point in time. Departments and provincial organizations are not required to go to Partnerships New Brunswick to develop an infrastructure project. Appendix 2: Glossary of Terms Term Definition Net Present Value (NPV) The equivalent value at a given time of a stream of future cash flows, calculated by discounting the actual values at the appropriate discount rate. Discount Rate The rate used to calculate the present value of future cash flows. Public Sector Comparator (PSC) The hypothetical, risk-adjusted whole-of-life cost of a public sector project if delivered by government. P3 Cost Estimate The net present value of a series of cash flows required by the proponent in its proposal. Value-for-Money A quantitative and qualitative assessment of the costs and benefits of the traditional procurement approach versus the P3 approach. Retained Risk The value of those risks or parts of a risk that government bears under a P3 project. Transferred Risk The value of those risks (from government's perspective) that are likely to be allocated to the private party under a P3 project. Shared Risk The value of those risks that are likely to be shared between the government and the private partner. Maintenance cost Regular non-capital expenditures required during the operation period to maintain the assets. Life cycle cost Capital expenditures required during the operation period to replace or perform major maintenance on assets that have a lifespan which is shorter than the contract term. Definition by Partnerships BC At time of printing, Department of Education was renamed Department of Education and Early Childhood Development AD- 6701 Administrative Policy on Present Value Analysis of Expenditure Decisions Department of Transportation’s 2009-10 Annual Report, page 10 Public-Private Partnership: Eleanor W. Graham Middle School and Moncton North School Chapter 2 Chapter 2 Public-Private Partnership: Eleanor W. Graham Middle School and Moncton North School 12 Report of the Auditor General - 2011 3 Report of the Auditor General Department of Supply and Services - Public-Private Partnership: Chapter 2 Eleanor W. Graham Middle School and Moncton North School Report of the Auditor General - 2011 9 Department of Supply and Services - Public-Private Partnership: Eleanor W. Graham Middle School and Moncton North School Chapter 2 Department of Supply and Services - Public-Private Partnership: Chapter 2 Eleanor W. Graham Middle School and Moncton North School 13 Report of the Auditor General - 2011 Department of Supply and Services - Public-Private Partnership: Eleanor W. Graham Middle School and Moncton North School Chapter 2 10 Report of the Auditor General - 2011 Department of Supply and Services - Public-Private Partnership: Eleanor W. Graham Middle School and Moncton North School Chapter 2 Department of Supply and Services - Public-Private Partnership: Chapter 2 Eleanor W. Graham Middle School and Moncton North School 40 Report of the Auditor General - 2011 Report of the Auditor General - 2011 41 Department of Supply and Services - Public-Private Partnership: Eleanor W. Graham Middle School and Moncton North School Chapter 2 Report of the Auditor General - 2011 14