Background 6.1 In our 1993 Report, we noted the Department of Finance completed a major re-engineering study, the first of its kind in this government. One of the key areas identified in the study was the need for a change in the management of revenue. In recognition of this need, the Department issued a Request for Proposals (RFP) for a Revenue Management System (RMS) on 14 October 1994. Four companies submitted proposals for evaluation by the Department. On 13 December 1994, the Department selected the IBM Consortium (referred to as IBM) as the successful bidder, and the project officially commenced on 16 January 1995. 6.2 The RMS project consisted of a number of projects designed to improve revenue management within the Department. Over a period commencing in the fiscal year ended 31 March 1995 and ending 31 March 1999, the RMS was supposed to move through 4 phases or “plateaus”, beginning with so-called early start initiatives and ending at the fourth plateau with universal electronic commerce. The anticipated result was an integrated RMS with six major functional components consisting of customer registration, revenue processing, compliance management, information provision, legislation/policy development and various support tools. The benefits of the new system were supposed to allow the system to pay for itself. As the project progressed, the pending implementation of the Harmonized Sales Tax (HST) caused the Department to rethink the RMS project. This was because, according to the Department’s Cost/ Benefit Profile, up to $37 million of the projected benefits of $51.2 million were to come from the improved collection and management of Provincial Sales Tax (PST). It was felt many of these benefits would disappear under harmonization, as tax collection would become a federal responsibility. 6.3 The Department informed us it started negotiating the termination of RMS with IBM as soon as the HST memorandum of understanding was signed with the government of Canada. The government of New Brunswick terminated the RMS project on 26 June 1996. 6.4 Our Office has maintained an ongoing interest in systems implemented through a public/private partnership arrangement. In previous Reports, we have looked at the NBCase Human Resources Development project and the New Brunswick Integrated Justice project. In early 1996, we wrote the Department to obtain some initial background information on the RMS. The RMS next came to our attention as a result of a transaction selected in our 1996-97 expenditure sample. While examining this sample item and related documentation, we identified a number of issues with potential value-for-money implications. Scope 6.5 Our work covers both of the Department’s contracts with IBM relating to the RMS project. Total expenditures were $9,874,518. Exhibit 6.1 lists all the projects and their related costs. Exhibit 6.1 Cost of the RMS Source – Spreadsheets provided by the Department 6.6 Our work included: • a review of the Department’s planning, budgeting, monitoring and controlling of the RMS; • a review of the travel expenses paid by the Department to IBM; • an assessment of the reasonableness of IBM’s charge-out rates for RMS employees; • a review of project completion notices for each project; • a review of the benefits calculated by IBM for the RMS; • a review of IBM’s profit calculation for the RMS; and • discussions with the Department of its plans for future revenue collection improvements that were identified but not carried out. Results in brief 6.7 The RMS was terminated due to the anticipated agreement with the government of Canada for the Harmonized Sales Tax. 6.8 The Department of Finance did not have a signed contract until after the project was terminated. 6.9 The Department started working on several RMS projects before the Business Case was completed. In our opinion, work on the other projects should not have proceeded until the Business Case was finished and the net benefits were quantified. 6.10 We had a number of concerns with the Department’s monitoring and controlling of the RMS project. 6.11 Hourly charge-out rates paid to the consortium were well in excess of standard government of New Brunswick rates. 6.12 The method of calculating profit was not clearly defined in the contract. 6.13 We recommend the Province analyze its experience to date with public/private partnerships and CPP arrangements. Introduction to findings 6.14 At the outset, we would like to note that our discussion with the Department indicated that despite the early termination of the RMS, eleven of thirteen projects started were completed. However, we found official completion notices for only three of the projects. We did not perform any testing to determine if the projects had been successfully implemented. 6.15 We have organized our findings according to areas of concern. Where appropriate we have made recommendations, some of which apply to the government as a whole, not just the Department of Finance. These recommendations should be useful to the government for future system development projects and public/private partnerships. Contracts not signed until after project terminated 6.16 The contracts between IBM and the Department were not signed until after the RMS project was terminated. The project was awarded on 16 January 1995, but the contracts were not signed until 29 June 1996 (contract one) and 23 July 1996 (contract two). The government officially terminated the project on 26 June 1996; neither contract was signed by this date. 6.17 A signed contract would have clearly established how the Province was to compensate IBM, protecting both parties. This was especially important in this case as the RFP response from IBM provided the broadest cost estimate – “between $2 and $40 million but definitely more than two.” In addition IBM identified many benefits, however none were quantified. In our opinion, given the unknown magnitude of the project, a signed contract should have been essential before work on the project started. 6.18 In our 1995 Report, we wrote the following with respect to the Andersen Consulting NBCase project: We believe that all of the undertakings should have been set out in a contract prior to the firm commencing this work. 6.19 We hold the same position with respect to the RMS. In our opinion, the consortium should not have been allowed to proceed in the absence of a signed contract. Recommendation 6.20 We recommend that the government of New Brunswick develop a policy to proceed with public/private partnerships and other major system developments only after a contract has been signed between the Province and the related parties. Departmental response 6.21 It was the department’s desire to have a signed contract shortly after the completion of the business case, as evidenced in the letter to IBM dated July 11, 1995; however, negotiations took longer than anticipated. In the interim, the Project did produce a Project Charter that established the framework and guiding principles for working together. Also, near completion was a draft Business Alliance Model that was to represent an umbrella agreement under which detailed contracts would be executed. We concur with your recommendation that future work of this nature should not commence prior to a signed contract or series of contracts. Work started before net benefits established 6.22 The Business Case project report indicated that there were eight “early start initiatives”, of which the Business Case was one. The Business Case was supposed to be completed by June 1995, however, documentation showed that significant resources were allocated to the Business Case up to October 1995 and some work continued even under the second contract. With one exception, we noted that work on all the other projects began prior to October 1995, before the net benefits had been documented in the Business Case. It should be noted that three of the early start initiatives (Electronic Commerce Pilot, Activity Based Costing, Decision Support) had no benefits specifically documented in the RMS Business Case Summary Report dated 7 March 1996. 6.23 Discussions with the Department indicated it adopted this approach because it was necessary to get some systems running as soon as possible. In our opinion, this early start approach goes against the very principle of the project; that is, that the benefits pay for the costs. Without the benefits being identified and quantified there was no justification to start the project. Recommendation 6.24 We recommend that the Province develop a policy that major system development projects and public/private partnerships should proceed only after net benefits are quantified. Departmental response 6.25 Charges continued against the business case initiative to update the report as new information came to light. In addition, charges were incurred in 1996 to review the impacts of harmonization on the business case. 6.26 In addition to the Cost/Benefit Profiles document completed in conjunction with the Business Case Summary Report, each early start initiative was to identify any tangible benefits. For some initiatives such as Project Management, Change Management and Communications the benefits were intangible, but these initiatives play an important role in the overall success of the Project. For this reason, the Project Charter identified that each initiative must be feasible in terms of cost/benefit; however, this did not mean that each initiative must pay for itself. The benefits from the overall project could be used to pay for individual initiatives. Concerns with project monitoring and control 6.27 The Department of Finance appears to have been quite thorough in certain aspects of project monitoring and control. In particular, a great deal of attention was placed on ensuring that all time charges billed by the consortium were supported by timesheets. Equipment purchases were monitored to ensure that the Department obtained the equipment at the “best value.” In addition, the Department carried out some analytical work on travel expenses as a means of ensuring that these charges were reasonable. 6.28 However, we have a number of concerns with the Department’s overall project management. These are detailed below. Project statements varied in quality 6.29 The terms of the RMS contracts required a project statement to be prepared for each project. A project statement outlines such important items as scope, objectives, delivery schedule, required roles and resources, costs and anticipated benefits for a project. This information would have assisted the Department in better monitoring and controlling each project. 6.30 With this in mind, we looked at the project statements for the RMS. Our objectives were to assess the quality of the planning and budgeting information available, to ensure benefits were identified, and to ensure the statements complied with the terms of the contracts. We expected to find the following based on paragraph 3.3 of the contracts: • a schedule providing evidence that a time frame had been established for commencement and completion of the project; • an estimate of the proposed human resources and their roles explaining who would have been working on the project and at what job classification; • an estimate of proposed human resource utilization showing the expected amount of time to have been spent on the project by each person; • an estimate of costs integrating human resources and other budget items; and • an estimate of benefits to determine if, in fact, a project should proceed and to provide a projection of the probability of cost recovery. 6.31 Only two project statements met the first three criteria. These were the Business Case, and the Activity Based Costing project. None of the project statements met all five of the criteria. 6.32 The project statements which did have some form of proposed resources and planned resource utilization, tended to be very nonspecific in nature, not giving enough detailed information to do budget comparisons against actual figures. Recommendation 6.33 We recommend that the Province develop a policy of requiring major systems projects to have reasonable estimates of costs and benefits prior to work commencing. Departmental response 6.34 Although the information identified existed for each initiative, it was not all contained within the Project Statements. The department concurs that in future such information should be incorporated in the signed statements. Several budgets existed 6.35 We had difficulty in finding one approved budget figure to use in conducting our work. Prior to the delivery of the Business Case in October 1995, most of the projects had some form of budget in the planning documentation or project statements. After delivery of the Business Case, the Department told us it submitted a project budget extracted from the Business Case to the Policy and Priorities Committee of Cabinet (P&P). Our understanding is that the P&P decision did not specifically approve a budget figure. Departmental comments 6.36 Throughout the development of the business case the projected cost of the Project continued to be modified. For this reason various projected budgets existed until the final Business Case Summary Report. When asked if the budget was approved at P & P, the department advised your officials that it submitted to the Policy and Priorities Committee its Business Case Summary Report as an attachment to the memorandum, which included a cost/benefit summary (copy of the page from the Business Case Summary Report was provided). In summary, Record of Decision No. 95-31-06 of Policy and Priorities Committee recommended to Executive Council approval for the development of a Revenue Management System and noted three requirements: 1) the creation of a strategic business alliance with IBM; 2) agreement in principle to proceed with the development of a framework and a business plan for a Special Operating Agency; and, 3) agreement in principle for legislative and policy changes. 6.37 Had the Project not been terminated, the department planned to seek approval from Board of Management of its Business Case and Business Alliance Model. Instead, Board of Management Minute Number 96.0384 reviewed the proposed contract to IBM and made recommendations. Board of Management Minute Number 96.0575 approved the transfer of funds. Variances and/or change orders 6.38 Our review of documentation also showed that the project team prepared twenty separate change orders totaling just over $1.5 million. In a document provided to us on 21 October 1997 the Department included these change orders as increases to its budget in a comparison of budgeted expenditures to actual. We are uncertain as to whether these items are true budget amendments or if they may have been more correctly classified as variance explanations. In our view, a budget amendment should not normally be made, as one major purpose of a budget is to provide a means of comparing planned performance to actual results. The more a budget is amended, the less value this comparison will have. Departmental comments 6.39 All costs related to Project Statements were based on a limitation of expenditures and not fixed price. This meant any increase in expenditures required a change order. A few change orders resulted in fixed price due to the nature of the work being performed. 6.40 The change orders during 1995 are more representative of requirements that had not been identified or envisioned at the time a Project Statement was produced. However, once the decision was made to terminate the Project, all initiatives were reviewed establishing what the projected cost would be to bring the initiatives to completion by October 31, 1996, (closing date for the Project). The remaining effort and projected cost were reviewed by the department. A number of Project Statements covered the period ending March 31, 1996, and new Statements for the next fiscal were to be prepared. Given that the Project was terminating, a decision was made to prepare change orders extending the existing Project Statements to the time frame required to close each initiative. The change orders identified the effort and cost to bring each initiative to completion by October 31, 1996. Department did not monitor effort 6.41 Most of the project statements had some estimated budget for the level of human resources required in either hours or days. However, the Project Director informed us that the Department did not compare actual hours or days worked to the project statements. 6.42 In our opinion, this was a significant oversight. We believe it is necessary to monitor the level of effort on a project in order to determine the percentage of completion. In addition, in comparing budget to actual results, it is necessary to be able to split out this human resource component from such things as equipment, materials, expenses and rentals. This allows the managers to take corrective action on any component of the project that exceeds budget at a particular point in time. Without tracking effort against budget, we believe the control function is impaired. Recommendation 6.43 We recommend the Province monitor budgeted human resource effort to actual effort in carrying out major systems projects. Departmental response 6.44 Early in the Project, IBM and the department established a Project Control Office. The role of the Project Control Office was to: establish and enforce project management standards and methodologies; manage the project logs; track and report project status; manage project facilities and inventories; initiate purchase orders; manage and report the project financials; manage the agreements between IBM and its sub-contractors; maintain a project library; process time sheets; and, provide administrative support. The Project Control Officer was accountable to both GNB and IBM’s Project Director. 6.45 All classifications established for outside resources were reviewed by the department’s Project Director and challenged where necessary. Performance issues were addressed as they arose with negotiated solutions. 6.46 Although the Project Control Officer was to monitor projected effort to actual, the department’s Project Director also maintained records on projected cost to actual as the bulk of the budget (84% of the cost of the Project before interest and profit) related to effort. Had the actual development work commenced for an integrated revenue management system, increased emphasis would have been placed on monitoring effort to actual as the range of resources and classification would grow, as well as costs for hardware, software, etc. Travel claims should be reviewed 6.47 Documentation provided by the Department stated that IBM would follow the government of New Brunswick’s guidelines for its travel claims. The Department provided the guidelines to IBM. Indeed, there should have been no confusion as to what expenses would have been allowed. 6.48 We tested 28% of the dollar value of the claims submitted by IBM. We found some possible overpayments, ineligible expenses, or items that we felt the Department might want to investigate. Clause 6.7 of the contracts allows the Province to audit and verify any invoice presented by IBM up to four years after the acceptance of all services and deliverables. Based on our testing, it may be useful for the Department to further examine expense claims in order to identify any misunderstandings or errors in travel claim preparation that may have occurred on the part of the consortium. Recommendation 6.49 We recommend that the Department of Finance examine expense claims for the RMS project. Departmental response 6.50 It has always been the plan of the department to do a further review of receipts; however, it is the department’s view that there has not been a significant overcharge and it will ensure this is the case. No final cost benefit analysis 6.51 One of the thirteen projects included in the RMS was Project Management. One objective of this project was to track the benefits associated with the implementation of the RMS. Project Management attempted to track these benefits until May 1996. By that time, the Department knew the RMS was to be terminated and the project ceased to track RMS benefits. In effect, the project moved from the “funded out of savings” approach to a regular “pay for service” approach. 6.52 The Project Director informed us that benefits up to May 1996 were estimated at $1.9 million. We believe it would have been useful for the Department to do a final comparison of benefits received to the total $9.9 million in payments to IBM. 6.53 In addition, we believe the Department may have benefited by linking IBM’s payment to the benefits received. IBM’s response to the RFP said “costs of the project will be fully funded by the Company until benefits resulting from the project can be realized.” Most projects identified in the Business Case were projected to have positive net benefits. And the Department has stated eleven of these projects were completed. Since the partners had been working for a year and a half without a signed contract and under the assumption IBM would be paid out of benefits, we wonder why this approach could not have been continued until the work was completed. Under this scenario the payments to the consortium may have been significantly less. Since the benefits calculations were stopped in May 1996, we are not able to report on the amount of benefits received. Departmental comments 6.54 After [May 1996], it was determined the cost of maintaining a resource to monitor benefits was not adding value. The Project was to be terminated and the initiatives were in their winding-down phase preparing for closure on October 31. Only three projects had completion notices 6.55 Both contracts required each project to have a completion notice. Clause 14.3 said: Upon completion of each Project, IBM shall submit to the Province a written confirmation (the “Project Completion Notice”) that all Services and Deliverables for the Project have been completed and are ready for acceptance testing. Upon receipt of a Project Completion Notice, the Province shall during the Project Acceptance Period carry out such tests on the Services and Deliverables as the Province considers necessary (the “Acceptance Tests”) for the purposes of determining whether the Services and Deliverables meets the Project Acceptance Criteria. The Province shall, within thirty (30) days from the expiration of the Project Acceptance Period, provide IBM with (i) written notice that the Services and Deliverables meet the Project Acceptance Criteria (the “Project Acceptance Certificate”); or (ii) written notice advising IBM that the Services and Deliverables, or a part thereof, do not meet the Project Acceptance Criteria (the “Project Deficiency Notice”). 6.56 Finance informed us that although eleven projects were completed, it had not obtained the required completion notices for them all. Only three projects had completion notices on file. Finance did, however, do a checklist to ensure it had received the key deliverables for each project. Departmental comments 6.57 The department recognized the need for completion notices as a way to identify that each initiative had delivered what it was supposed to. However, once the Project started to scale back on effort and resources in 1996, in lieu of completion notices an agreement was reached whereby a listing was prepared of all deliverables from each initiative. Prior to any approval of payment to IBM, this listing was reviewed and receipt of each deliverable was confirmed. Documentation to this effect was shown to your auditors. We believe the confirmation of all deliverables by listing serves the same purpose as completion notices and this should not be an issue. Charge-out rates appear high 6.58 During the oral presentation IBM committed to pricing “personnel at NB market rates.” There is no further definition in the meeting notes or the contract as to what NB market rates actually meant. We did note, however, some concern in the Department of Finance about the reasonableness of the rates charged. 6.59 An internal memo dated 7 March 1996 discussed a comparison of IBM rates to those under the government of New Brunswick’s (GNB) Informatic Professional Service Contract of Supply. (The Department of Supply and Services prepared this Contract of Supply for use by all government departments. It contains rates for various information technology professionals from twenty consulting companies.) The memo indicated that if one compares the IBM rates including their 15% profit margin to the average GNB Contract of Supply rates, IBM’s rates are over 100% higher. The memo added, the Department’s comparison was very conservative since the highest rates from the Contract of Supply were being compared to the lowest IBM rates for each category. 6.60 Another memo questioned how IBM’s daily rates were affected by the number of productive days per year. It said IBM’s calculations were based on a productive year being 175 days. This meant they were asking 20% more than if we assume a productive year of 210 days, 26% more if we assume 220 days. 6.61 Since neither party had the protection of a signed contract when the project terminated, we would have expected the Department to attempt to negotiate the rates to a lower amount. One reason we were given for the high rate structure is that IBM was using rates quoted to the government of Canada for so-called common purpose procurement (CPP) projects. Supposedly these rates include a risk premium to cover the various contingencies implicit in these CPP projects. However, in this case, since the project was terminated and the consortium was essentially compensated for all hours billed, it appears the main elements of risk to IBM were removed. We would note as well that nothing in the contracts states that government of Canada rates are equivalent to NB market rates. Recommendations 6.62 We recommend the Province develop a RFP policy requiring the responses to specify charge-out rates in detail. This would assist the RFP evaluation committee in recommending a successful bidder. Terms such as “NB Market Rates” should be clearly defined. 6.63 We recommend that a contract should not be signed until clear and reasonable rates are established. Departmental response 6.64 As there were no GNB CPP rates established, the department was able to review, on a confidential basis, CPP rates IBM charged to their other clients. The department found the rates being discussed to be comparable. 6.65 The department does concur with your findings that rates should be clearly established prior to work commencing and contracts signed. Charging commercial profit 6.66 The Pricing Strategy in the RMS contracts indicated the amount paid would include “IBM commercial profit of 15%.” We were uncertain as to the definition of “commercial profit” as it was not defined in the contracts. 6.67 As discussed in the previous section, it is our opinion that the charge-out rates appeared high. One might argue they were sufficiently high enough to have included a profit component. This was particularly so once the project moved from the “funded out of savings” approach to a regular “pay for service” approach. This change removed an element of risk. 6.68 If this was the case, the Department should have questioned being charged an additional 15% commercial profit on services. The total profit that was charged on services in contract one was $1,051,433. In contract two this amount was $185,652, bringing the total to $1,237,085. 6.69 IBM also charged 15% commercial profit on “integrated materials.” This amounted to $40,846 over the two contracts. Integrated materials include things such as computer hardware and software. IBM purchased these items on behalf of the Province at standard government rates or, as the Department termed it, at the “best value.” Then the Province was to reimburse IBM for the cost of the purchases. IBM charged the Province an additional 15% on the cost of all these purchases. In our opinion, this is not appropriate. It certainly does not meet the term “best value.” If the Province knew IBM was going to charge profit on the purchase of integrated materials, the Province would have been better off purchasing the materials itself. 6.70 We asked the Department what steps it had taken to ensure “commercial profit” had been calculated correctly. We were informed at one point that someone from the Office of the Comptroller assessed the reasonableness of the calculation. However, we could not find any written report from the Office of the Comptroller on this matter. Recommendation 6.71 We recommend all key terms be defined in contracts prior to their signing. In particular, the method of calculating profit components should be clearly defined. 6.72 We recommend the Department exercise its right to audit under clause 6.7 of the contracts to determine whether it can receive a refund for all or a portion of the commercial profit. Departmental response 6.73 Although the contract allowed for 15% commercial profit, which is the percentage used in the formula, we concur with your recommendation that the method of calculating profit components, in future, should be more clearly defined. 6.74 It was the decision of government to harmonize its sales tax with the GST. IBM entered into the business alliance fully prepared to accept the risk of being paid from benefits. Up until the decision was made to commence harmonization negotiations, IBM was carrying that risk. It was the decision of the department that IBM be paid in accordance with the business alliance discussions to that point. These commitments included established rates, interest and profit. Future considerations 6.75 Throughout this chapter, we have made specific recommendations prompted by our findings related to the RMS project. In our opinion, however, a measure of the success of this chapter is how the Province will use these findings to better manage major systems development projects, common purpose procurement (CPP) arrangements and public/private partnerships in the future. 6.76 One area of consideration for the Province should be an analysis of the method of procurement. It should consider the pros and cons of doing work itself versus contracting out for services. It should consider the value of developing an independent business case and breaking a large project down into many smaller areas as opposed to the integrated methodology used for common purpose procurement. 6.77 When proceeding with CPP, the Province must recognize two important risk factors. These are a lack of independence in preparation of the business case and the fact that there is no separate determination of the best way to achieve benefits. These risks must be carefully weighed against the potential benefits of common purpose procurement prior to requesting proposals. Lack of independence in the preparation of the business case 6.78 Our understanding is that in the first project, the Business Case, IBM was to identify the benefits, costs and investments required for the RMS. If there were no net benefits from the RMS, IBM would not proceed on the other projects. This approach is typical of CPP projects where the government is seeking a partner to provide a solution to a range of business problems and issues. 6.79 This approach appears to put IBM, or any CPP consultant, into a position where they have control over the amount of future work they will receive. This is particularly true since they are involved in identifying the benefits. By making the consultant’s work on other projects contingent on a successful business case, the Province puts the consultant in a potential conflict of interest position. No separate determination of best way to achieve benefits 6.80 The arrangement entered into with IBM combined into one analysis both the benefits and the costs. If the costs could be paid from benefits, the project proceeded. In other words there appears to be a strategy that no matter how high the costs, the project should proceed so long as the benefits are greater. Our understanding again is that this is a typical approach for a CPP arrangement. This approach does not follow a traditional business case analysis, and as a result potential savings and efficiencies may be lost. 6.81 What we would have expected to see was the identification of benefits, financial or otherwise, if a particular decision or course of action is followed. Then we would expect to see a separate and distinct determination of the best way to achieve those anticipated benefits. Carrying out the project with the same firm that prepared the business case may not be the best solution, but only a solution. There may be more effective ways to achieve the results, such as contracting out some or all the work, doing the work in small projects with existing staff and buying software packages from another company. There does not appear to have been this type of analysis in the IBM business case, but rather an implicit acceptance that the IBM solution was the best solution for the Province. The Department’s Plans 6.82 The Department appears to be heading in a good direction with its new Preliminary Analysis/System Architecture (PA/SA). This new focus covers approximately twenty new projects, which will be developed over three or four years. These projects will be tendered separately, and no public/private partnership or CPP arrangement will exist. Where possible, the Department is going to use existing software rather than try to develop and market its own software. 6.83 This proposed method of many small projects appears to have several benefits. Smaller projects result in less risk. If they fail, the cost is lower. Also, with smaller projects, the Department can more easily manage costs and resources. This helps with the timeliness of implementation, resulting in less risk of the projects becoming outdated before they can be implemented. Recommendation 6.84 We recommend the Province analyze its experience to date with public/private partnerships and CPP arrangements. It should determine whether these procurement methods are delivering solutions in the most economical and efficient manner possible.