Interim Evaluation No: R/SK/EDIS/03.POHL		

	Interim Evaluation of the European Union
Pre-Accession Instrument Phare

Country: Slovak Republic

Sector: EDIS

Programme covered:

2002/000.585 Preparation of the National Fund and Implementing Agencies for EDIS implementation within the Stage II ( Gap Plugging ).

Author: POHL Consulting

Date: August 4th 2004
	
	This report has been prepared as a result of an independent evaluation by the POHL Consulting and Associates being contracted under the Phare programme.

The views expressed are those of the POHL Consulting and Associates and do not necessarily reflect those of the European Commission.

ABSTRACT

Scope of the evaluation

This Interim Evaluation covers the project “Preparation of the National Fund and Implementing Agencies for EDIS Implementation within the Stage II (Gap Plugging), under the project number 2002/000.585. - The programme used technical assistance as a delivery mechanism and funding for this programme amounted to 0.7 MEUR from the 2002 Phare budget.

While it had originally been intended to complete the IE on this programme on 15th January 2004, the submission date for this report was changed at the request of the Ministry of Finance to February 21st 2004 due to the heavy workload in preparing for the submission of the accreditation documents.  It was also apparent, however, that there had not been adequate consultation with the stakeholders on the purpose or scope of the evaluation prior to the Kick Off Meeting. The first mission for this IE eventually took place in early February 2004 and the second in mid-April 2004
The scope of the evaluation was elaborated and agreed during the Consultant’s meetings with the stakeholders in Bratislava during January 2004 and further refined during their second mission in April 2004. It was agreed with the Commission Services that due to the limitation of the time available for the EDIS component of the Interim Evaluation, the scope of the exercise would be primarily on the evaluation of the Stage II “Gap Plugging“ component and its constituent elements. However, it was agreed that this would not be treated in isolation from Stages I and III since the 3 components were mutually interdependent.
Due to time limitations, this is not a performance audit but is primarily a "snapshot" of the EDIS process with the focus on Stage II “Gap Plugging“. This report is not, therefore, a standard Interim Evaluation product. Nevertheless, this IE is important in that it assesses the degree to which measures designed to prepare Slovakia for accession in the context of EDIS accreditation have been successful.
There was no Interim Evaluation Report produced by the EMS Consortium in the past and this is the first IE to be undertaken for this programme.

Key achievements, findings and recommendations

Relevance. The relevance of the EDIS Stage II support is not in doubt. Slovakia is, in conjunction with other acceding countries obliged under the Accession Treaty to be EDIS-compliant by accession. Phare support to assist the Slovak authorities was, therefore, fully relevant. Unlike some other acceding countries, however, the approach in Slovakia has been a dedicated Phare product rather than available ad-hoc financing from project preparation facilities. The final delivery of assistance via the FWC, however, has been the same. The only downside with the programming approach in Slovakia was that it made timeliness of contracting and intervention more demanding, with the result that this downside risk in fact materialised and Phare funds for the Compliance Assessment component (Stage III) were not contracted and lost due to problems with the tendering process (lack of proposals, ineligibility of candidates, disagreements on the TORs). As a result this component was funded by the Slovak authorities themselves.

In terms of the material components of Stage II, 3 of them (Development of IT, Development of Human Resources and the Development of Management Procedures), were clearly identified in the Stage I Gap Assessment Report as areas of major concern and ones that required specific intervention. The final component, Compliance Assessment, was also provided for within the budget lines although this was eventually funded from Slovak funds rather than through the Phare budget due to the problems outlined above.

The relevance of the programme in general and its constituent components is not in doubt and was a crucial part of the overall EDIS accreditation process, linking Stage I: Gap Assessment with Stage III: Compliance Assessment and eventually with Stage IV: Verification Audit.

Efficiency. The initial deadline for the submission for Stage IV Commission Accreditation was September 2003. In fact, delays occurred due to a variety of factors and the submission eventually took place in December 2003, some 3 months later than envisaged. Although the timetable for the Stage I was maintained, time was lost in particular in Stage II due to problems with tendering and contracting and in Stage III where protracted problems with the tendering process eventually led to the loss of all of the funding for this component (€132,000) and this element being funded by the Slovak authorities themselves. 

The Stage IV Verification Audit is due to start in third week of April 2004, which will leave little time for remedial action to be undertaken by the time of accession. As a consequence, there is an urgent need for a contingency plan to allow for the continuation of contracting pending verification. 

It should be added that the Slovak authorities have put in place a wide series of structures to monitor the implementation of EDIS including Steering Committees and Working Groups for specific components. While the Steering Committees and WGs have met frequently and thus have had a positive impact, some concerns have been expressed about their efficiency in translating the results of the meetings into actions.

Effectiveness. The effectiveness of Phare intervention for the EDIS process has two aspects. It concerns the Phare financing of Stage II but also more widely, a judgement call on the overall effectiveness in assisting Slovakia to meet the EDIS accreditation by accession to the EU on the 1st May 2004. 

During the contractor’s discussions with the agencies concerned, there was general appreciation of the work undertaken in the Development of Human Resources and of it providing the basis for HRD in the key agencies. 

By contrast, the expertise provided under the component concerned with the Development of Management Procedures was considered poor, badly focused and disjointed by all agencies contacted. As a result, its effectiveness was unsatisfactory.

For the development of IT, the work undertaken was reasonably sound and provided the basis for the development of strong back up systems, disaster recovery systems and support services in the NF and the IAs. It also provided an in depth analysis of the systems and hardware in place as well as identifying further gaps not included in the initial Gap Assessment Report.

In terms of effectiveness, concerns have been raised over whether the use of a horizontal approach to providing assistance was the most appropriate mechanism rather than one that was tailored to the needs of the specific Ministries and their specific requirements. In hindsight, it would appear that the approach was not well suited to the needs of the specific agencies and in some cases led to a dilution of resources and very ineffective impact.

 Impact and Sustainability. The key issue here, in the narrow sense, is whether the initiatives undertaken in the context of the Stage II exercise were effective, had the desired impact and provided the basis for long term sustainability in the post-accession period when EDIS accreditation has been secured. In terms of the Development of Human Resources, the work has had a positive impact, while that for the Development of IT was sound although limited in scope. For the Development of Management Procedures, the impact has been poor and much additional work has been required by the Slovak authorities themselves to ensure that procedures, manuals etc are in place.

Recommendations include the following:

•	Given the imminence of the May 1 accession date, a plan of action needs be agreed for the period from May 1 until EDIS accreditation for Slovakia within the time bounds set under the Accession Treaty. A joint approach will be needed between Slovakia as a new Member State and the Commission’s Representation Office. The Slovak authorities under the NAO must continue to build through their operational Working Groups a sufficient critical mass that tackles the basis of the longer-term sustainability and impact concerns.

•	A delay in giving a decision for accreditation means that is within the bounds of the Accession Treaty for the continuation of the current DIS arrangements by the Representation Office from May 1.While the contractors were advised that there will be a notification to the effect that contracting can continue under the DIS with the new PRAG Guidelines, there has as yet been no formal notification to the Slovak authorities. It is important that this is done quickly so that the Slovak authorities can take the necessary actions on the implementation of the outstanding contracts.

•	The existing co-ordination structures need to continue beyond May 1, since their role will be critical in the post accession period especially if further Gaps need to be plugged and an interim solution needs to be found using the DIS system. The JMC will continue. The Steering Committee will need to stay in operation as will a number of the Working Groups especially the one for Financial Management. The number of WGs continuing to operate will depend very much on the conclusions of the Verification Audit.

•	Each of the three core areas of Human Resources, Management Systems and IT are a dynamic that will need ongoing fine-tuning and revision. This is best undertaken under a combined approach of CS monitoring and appraisal through various peer and audit reviews and spot checks on the one hand and thorough continued Institution Building actions on the other. Twinning should be continuously programmed for the foreseeable future under the Transition Facility. 

•	Given the problems that have occurred using a horizontal approach to the provision of assistance, intervention in the future would be best targeted at the individual Ministries themselves and assistance focused on their specific needs rather than to apply a homogenous approach.

EDIS	
Number	Type of Assistance	Programme/ project title	Implementing Authority	
2002/000.585	TA	Preparation of the National Fund and Implementing Agencies for EDIS implementation within the Stage II (Gap Plugging)	Ministry of Finance- National Fund	
GS – Grant Scheme, INV – Investment, JSPF – Joint Small Project Fund, TA – Technical Assistance, TWIN –
Twinning,

TABLE OF CONTENTS  

ABSTRACT	i
TABLE OF CONTENTS	1
PREFACE	2
GLOSSARY OF ACRONYMS	3
EXECUTIVE SUMMARY	5
Main Report	11
1.	SCOPE OF EVALUATION	11
2.	EVALUATION RESULTS	14
2.1 	Relevance	14
2.2 	Efficiency	14
2.3 	Effectiveness	15
2.4 	Impact and Sustainability	16
ANNEXES	19
ANNEX 1. Indicators of Achievement OF SECTOR OBJECTIVES AND EVALUATION CRITERIA	20
ANNEX 2. LIST OF INTERVIEWS	21
ANNEX 3. LIST OF Documents Referred to in the INTERIM EVALUATION	22
ANNEX 4 - DISSENTING VIEWS	23

PREFACE
This Interim Evaluation covers Phare assistance to “EDIS” under the following Programme Component:

2002/000.585 	“Preparation of the National Fund and Implementing Agencies for EDIS implementation within the Stage II (Gap Plugging)”

This Interim Evaluation has been prepared by Pohl Consulting and Associates of Germany during the period from December 2003 to April 2004. The factual basis for this evaluation has been based on an analysis of the formal Programme documentation, interviews with relevant Ministries and agencies and published material. The Interim Evaluation examines the progress of the programme towards the objectives stated in the formal programming documents, i.e. Terms of Reference. The report is intended to provide management information for the benefit of the Commission Services and to the Slovak authorities. It draws conclusions and puts forward recommendations. It provides a general assessment of programme under consideration. The present IE is unusual in several respects in terms of focus and the time available.

Comments requested on the Draft Version were received from the following parties.

Parties Invited	Comments Received	
Ministry of Construction and Regional Development/ Regional Development Support Agency 	                           No	
National Agency for Small and Medium Enterprises 	                           No	
Office of the Slovak Government/ Aid Co-ordination Unit 	                           Yes	
Ministry of Finance/ Central Finance and Contracting Unit 		Yes	
Ministry of Finance/National Fund 		No	
Commission Services at the Delegation 	                           Yes	

GLOSSARY OF ACRONYMS

AC			Accession Country
ACU 			Aid Co-ordination Unit
CC			Candidate Country
CFCU 			Central Finance and Contracting Unit
CS 			Commission Services
ECD 			European Commission Delegation
EU 			European Union
ESC 			Economic and Social Cohesion
ESF 			European Social Fund
FM 			Financing Memorandum
FWC			Framework Contract
GAR			Gap Assessment Report
HLWG			High Level Working Group
IA 			Implementing Agency
IB			Institution Building
IE 			Interim Evaluation
IT 			Information Technology
JMC 			Joint Monitoring Committee
MEUR			Million(s) of Euro
MS			Member State
NAC 			National Aid Co-ordinator
NADSME 		National Agency for Small and Medium Enterprises
NF 			National Fund
NPAA 			National Programme for Adoption of the Acquis Communautaire
PAO 			Programme Authorising Officer
PCM			Project Cycle Management
PF 			Project Fiche
SF 			Structural Funds
SK/SR 		Slovak Republic
SKK 			Slovak Crowns
SME 			Small and Medium Enterprise
SMSC 			Sectoral Monitoring Sub-Committee
SR/SK 		Slovak Republic
TA 			Technical Assistance
ToR 			Terms of Reference

FINANCIAL DATA
Financial and Contractual Data of the Sector ‘EDIS’

Programme 
Number	Programme
Title	Expiry
Date for contracting	Expiry
Date for
Disbursement	Allocation (in MEUR)	Phare Funds	Co-financing	
				Phare	Co-financing	Commitment	Disbursement	Commitment	Disbursement	
Preparation of the National Fund and Implementing Agencies for EDIS implementation within the Stage II (GapPlugging) 	
2002/000.585	Development of IT	30/09/2003	30/09/2004	170,000	0	57,720.27

	57,720.27

	0	0	
2002/000.585	Development of Human Resources	30/09/2003	30/09/2004	199,000	0	159,850	159,850	0	0	
2002/000.585	Development of Management Procedures	30/09/2003	30/09/2004	199,000	0	179,384.58	179,384.58	0	0	
2002/000.585	Compliance Assessment	30/09/2003	30/09/2004	132,000 	0 	0
	0
	0	0	

	Total			700.000	0	396,954.85	396,954.85	0	0	
Source: National Fund information dated 4/2004

EXECUTIVE SUMMARY 
Slovak Republic-EDIS

This Interim Evaluation has been prepared by Pohl Consulting and Associates of Germany and covers Phare assistance to the ‘EDIS’ Sector under the Phare project "Preparation of the National Fund and Implementing Agencies for EDIS implementation within the Stage II (Gap Plugging). 

Scope and Objectives

This project comprises four Programmes under the Phare Horizontal Programme preparing candidate countries for EDIS, totalling 0.7 MEUR in funding. These are shown in the table below. Overall co-ordination of the programme rests with the Ministry of Finance. Other central institutions to be accredited under EDIS are also involved in the programme. This Programme relates to the preparation of Slovakia for the management of Phare pre-accession funds. The primary focus of this IE will be on the Stage II "Gap Plugging" exercise although due reference will be made to the work undertaken in the other 2 stages, Stage 1: Gap Assessment and Stage III: Compliance Assessment.

Evaluation Results

The table below shows the consultants' ranking of each of the programme components and the overall ranking. Only the Development of Human Resources and the Development of IT show a satisfactory evaluation with the overall assessment being Unsatisfactory.  The loss of funding for the Compliance Assessment ranks this component Highly Unsatisfactory despite its critical importance.

Criterion

Project	Relevance	Efficiency	Effective-ness	Impact	Sustain-ability	Verbal
Overall
Rating	
Preparation of the National Fund and Implementing Agencies for EDIS implementation within the Stage II (GapPlugging)	
Development of IT	1	1 	1	1	1	S	
Development of Human Resources	1 	1 	1	1	0 	S	
Development of Management Procedures	2	-1	-1	-1	-1	U	
Compliance Assessment	1	-2	-2	-2	-2	HU	
Taking into account the contextual constraints on the evaluation, the sector overall is rated to be “Unsatisfactory”.	
Total for the Sector 	1	-1	-1	-1	-1	U	

This IE is non-standard in several respects. The primary focus is strictly on the performance of the Phare expenditure under the above-stated 2002 EDIS Stage II "Gap Plugging" exercise which although intended for financing of Stages II and III was in fact deployed only for Stage II.

However, the consultants were requested to adopt a broader approach in the IE and to place the Stage II evaluation in the broader context of the whole EDIS accreditation and to comment on the Stages I: Gap Assessment and Stage III: Compliance Assessment. 

This means in essence that the IE is in essence a mix of interim and ex-post assessment, with critical appraisal, to the degree possible, on the larger picture of the switch from DIS to EDIS in the run-up to EU accession on May 1, 2004 and the transfer of the ex-ante visa from the Commission Services to the Slovak NAO. 

It should be stressed at the outset, however, that a severe time constraint for this exercise has limited the extent of the analysis and the IE. It is hoped, however, that this will complement the extensive in depth verification audit that will take place over the coming weeks.

The IE is based on available but limited quantitative data on the programme and qualitative information based on a mix of questionnaire, interviews and available documentation. All the key parties in the EDIS process were interviewed and in general the Consultant received substantial information and active co-operation. 

It is a "snap-shot" IE review of the Phare intervention. However, the wider and long-term focus on the durability of the EDIS intervention, i.e. impact and sustainability, are also given significant attention as per the Commission Services’ focus on the EDIS process. 

However, this is neither a thorough IE, nor a performance or systems audit that alone can genuinely answer the process-specific issues in depth. These issues will be covered in the detail in the Verification Audit that is to take place in April/May 2004 and will provide a detailed insight into the degree to which there is compliance.

Relevance

The relevance of the EDIS Stage II support is not in doubt. Slovakia is, in conjunction with other acceding countries obliged under the Accession Treaty to be EDIS-compliant by accession. Phare support to assist the Slovak authorities was, therefore, fully relevant. Unlike some other acceding countries, the approach in Slovakia has been a dedicated Phare product rather than available ad-hoc financing from project preparation facilities but the final delivery of assistance via the FWC has been the same.

The only downside with the programming approach in Slovakia was that it made timeliness of contracting and intervention more demanding, with the result that this downside risk in fact materialised and Phare funds under the FM were lost for a variety of reasons. In summary, Phare financing was fully relevant. 

The framework for the output in Stage II or Gap Plugging was based on the rigour of the approach and comprehensiveness of the assessment in Stage I or Gap Assessment. 

The 3 key areas addressed in the Stage II Gap Plugging exercise (Development of IT, Development of Human Resources and the Development of Management Procedures) were clearly identified in the Stage I exercise as critical areas of concern and ones that needed to be addressed before Compliance Assessment could be undertaken. There is, therefore, concrete evidence of the relevance of the components that needed to be addressed in the Gap Plugging exercise.

It should be stressed, however, that the activities undertaken in Stage II identified a number of gaps that had not emerged from the Stage I Gap Assessment Report indicating that the work undertaken had not been as comprehensive as had been envisaged or that the authorities were having to work with a “ moving target “ and changing scenarios.

Efficiency

At the HLWG meeting of 6th March 2003, the Action Plan presented by the Slovak Authorities indicated that the completion dates for Stage II and Stage III respectively would be March-May 2003 and June 2003. However, the minutes of the JMC of June 2003 stated that the completion of Stage II would be by August 2003, which was 2 months later than indicated earlier.

Stage III was due to start in August 2003 and be completed by the beginning of September 2003. The submission for Stage IV Commission Accreditation was due in September 2003.

In fact delays occurred in this timeframe, although the timetable for the Stage I was maintained and completed by December 2002. Time was lost in particular for Stage II due to problems in contracting; and in Stage III through problems with the tendering process, lack of tenders, disagreements over the content an scope of the TOR etc with the result that these funds were not contracted and lost with the result that the Slovak authorities funded it themselves from national sources.

The submission for Stage IV Commission Accreditation was due in September 2003 but eventually only occurred in December 2003. 

In terms of efficiency, it is apparent that there were protracted delays in launching each of the components, which had an adverse knock on effect in terms of meeting the time schedule for compliance assessment and submission of the verification documents. Changes in structures and new personnel were also contributory factors in the delays that occurred.

Effectiveness

The effectiveness of Phare intervention for the EDIS process has two aspects. It concerns the Phare financing of Stage II but also more widely, a judgement call on the overall effectiveness in assisting Slovakia to meet the EDIS accreditation by accession to the EU on 1 May 2004. 
In terms of overall effectiveness of the individual components, discussions with the beneficiaries concluded that the Development of Human Resources was satisfactory with a high quality TA team, good collaboration, sound recommendations and positive intervention.

By contrast, the Development of Management Procedures component was considered to be unsatisfactory especially with regard to public procurement and financial management issues. The quality of TA was poor and the delivery was questionable and in many cases unrelated to the specific needs of the individual institutions. Training was also poor and limited to a one-day training seminar for all agencies.

For the IT component, the project was effective in identifying systems and hardware in each of the relevant agencies and in setting a framework for the development of back up facilities, disaster recovery systems and security systems in each of the key agencies. It was also important in establishing that systems in the Ministries were not homogeneous and that the use of a generic approach and a “one size fits all” was not appropriate.

Finally, the loss of funding for Stage III Compliance Assessment was due to a mixture of poor programming, problems with contracting, disagreements between the Commission Services and the MoF on the nature and scope of the TOR with the result that the Slovak authorities funded this element themselves from national sources.

Impact and Sustainability

Impact and sustainability are generally only assessed in an ex-post context after the termination of Phare intervention. More fundamentally, sustainability in the end-case will be judged on the smoothness of transition from the pre-accession world of the ex-ante visa held by the ECD to one where it is waived and responsibility is assumed by the NAO. 

The key issue here, in the narrow sense, is whether the initiatives undertaken in the context of the Stage II exercise were effective, had the desired impact and provide the basis for long-term sustainability in the post-accession period when EDIS accreditation has been secured.

During the contractor’s discussions with the agencies concerned, there was almost a unanimous commendation of the work undertaken in the Development of Human Resources and of its impact in providing the basis for HRD in the key agencies. The quality of the TA, the expertise provided and the focus on individual Ministries requirements was considered very satisfactory and providing a basis for long-term sustainability. 

By contrast, the expertise provided under the component concerned with the Development of Management Procedures was considered poor, badly focused and disjointed by all agencies contacted. As a result, any long-term sustainability could not be attributed to the actions taken in this element.

For the development of IT, the work undertaken was sound and provided the basis for the development of strong back up systems, disaster recovery systems and support services in the NF and the IAs. It also provided an in depth analysis of the systems and hardware in place as well as identifying further gaps not included in the initial Gap Assessment Report.

As for the agencies under the EDIS accreditation, the main issues concern contracting, tendering and procurement. The main weight of projects rests with the CFCU and despite the turnover of staff it appears to have key core members in place with sufficient focus on how to compensate for the administrative-deficit by the targetted use of TA. There is also a positive sense that the NADSME is showing medium-term improvement. The NF has also reinforced its staffing.

Conclusions

General Observations

•	Phare support was critical for financing Stage II or Gap Plugging, whilst Stages I and III were financed by Slovak authorities. Since the stages are sequential, the output of any stage rests in part on the input from the prior stage. By corollary, the implied impact and sustainability of Stage II cannot be judged in isolation and without recourse to the progress and results of Stage III. 

•	Phare support under FM2002/000.585 was not fully contracted leading to a loss of funding to Slovakia. The loss of funding was doubly costly for Slovak authorities, as the implied loss of Phare money did not mean delayed financing but instead required the use of scarce domestic resources. The causes of the loss of the funding  include problems with the tendering process (lack of submissions, ineligibility of candidates), disagreements over the TOR etc.)

Specific Observations

•	Evaluated in the narrow spectrum of the IE logic, the overall assessment of the Stage II Gap Plugging exercise is satisfactory.  This is based on the assumption, however, that the Compliance Assessment Component does not form part of this IE since it was contracted from national sources.

•	However, there was substantial variation in the 3 components of the Stage II TA. The overall score for the HR component is judged as satisfactory for the quality of delivery and the implied impact ascertained from the interviews with beneficiaries. However, the assessment of the management procedure component is unsatisfactory due to a number of factors including poor TA, lack of expertise, deficiencies in the use of the horizontal approach etc. The third component under Stage II for the IT systems is assessed as reasonably satisfactory in terms of defining system and hardware deficiencies and identifying additional gaps to be plugged. 

•	In terms of the EDIS process, the design of Stage II was determined on the basis of the output of Stage I. The Gap Assessment Stage I was not exhaustive, confirmed by “virtual gaps” that were non-existent during Stage I emerging during Stage II. Notwithstanding the above comment, the EDIS process has an in-built corrective mechanism in the form of a Stage III exercise to gauge the preparedness of the national authority and IAs to meet the accreditation and absorb any shortcomings form the earlier stages. 

Recommendations

•	Given the imminence of the May 1 accession date, a plan of action needs be agreed for the period from May 1 until EDIS accreditation for Slovakia within the time bounds set under the Accession Treaty. A joint approach will be needed between Slovakia as a new Member State and the Commission’s Representation Office. The Slovak authorities under the NAO must continue to build through their operational WGs a sufficient critical mass that tackles the basis of the longer-term sustainability and impact concerns. 
•	A delay in giving a decision for accreditation means that is within the bounds of the Accession Treaty for the continuation of the current DIS arrangements by the Representation Office from May 1.While the contractors were advised that there will be a notification to the effect that contracting can continue under the DIS with the new PRAG Guidelines, there has as yet been no formal notification to the Slovak authorities. It is important that this is done quickly so that the Slovak authorities can take the necessary actions on the implementation of the outstanding contracts.
•	The existing co-ordination structures need to continue beyond May 1, since their role will be critical in the post accession period especially if further Gaps need to be plugged and an interim solution needs to be found using the DIS system. The JMC will continue.. The Steering Committee will need to stay in operation as will a number of the Working Groups especially the one for Financial Management. The number of WGs continuing to operate will depend very much on the conclusions of the Verification Audit.

•	Each of the three core areas of Human Resources, Management Systems and IT are a dynamic that will need ongoing fine-tuning and revision. This is best undertaken under a combined approach of CS monitoring and appraisal through various peer and audit reviews and spot checks on the one hand and thorough continued Institution Building actions on the other. Twinning should be continuously programmed for the foreseeable future under the Transition Facility. 

•	Given the problems that have occurred using a horizontal approach to the provision of assistance, intervention in the future would be best targeted at the individual Ministries themselves and assistance focused on their specific needs rather than to apply a homogenous approach.	

Main Report

1.	SCOPE OF EVALUATION

1.	This Interim Evaluation has been prepared by Pohl Consulting and Associates of Germany and covers Phare assistance to the ‘EDIS’ Sector under the Phare Preparation of the National Fund and Implementing Agencies for EDIS implementation within the Stage II (Gap Plugging) for the year 2002. 

2.	This programme comprises four projects under the Phare Horizontal Programme preparing candidate countries for EDIS, totalling 0.7 MEUR from the Phare 2002 budget. Overall co-ordination of the programme rests with the Ministry of Finance NAO/National Fund ) This Programme relates to the preparation of Slovakia for the management of pre-accession funds.

Financial and Contractual Information for the Components under Evaluation

Programme/Component Title	Allocated
(€)
	Committed
(€)*
	Disbursed
(€)*	
Development of IT	170,000	57,720.27	57,720.27	
Development of Human Resources	199,000	159,850	159,850	
Development of Management Procedures	199,000	179,384.58	179,384.58	
Compliance Assessment	132,000 	0	0	
Total 	700,000	396,954.85	396,594.85	

Source: Perseus 4/2004

3.	By April 2004, all of the projects were closed.. All of the funding for Compliance Assessment amounting to some 19% of the overall project budget was not contracted and lost and was covered by funding from the Slovak Authorities themselves.

1.2	Performance of Activities

1.2.1	Component 1: Development of IT Activities and Outputs.

4.	The work on this component started in June 2003 and was completed in August 2003 when the Final Report was submitted by the contractor, Astec Global Consultancy. Although the work undertaken by Deloitte Touche in the preparation of the Gap Assessment Report (Stage I) identified the key gaps in operations continuity, backup and recovery, physical and logical security and applications and data, the contractor undertook a thorough review of all systems and hardware in the various agencies and identified IT gaps that had been overlooked in the Stage I process. 

5.	The key finding of this initial survey was that the approach adopted in Stage I had been too generic with the assumption that there were homogenous systems in place in all the Ministries. This was clearly not the case and there were wide variations in systems, resources, hardware etc. 

6.	The contractor presented the initial analyses based on the initial questionnaire distributed to all involved institutions and experts, and started technical discussions with the representatives of the IT Departments in the relevant Ministries and Agencies. 

The experts recommended that in the scope of Stage II work, the development of detailed templates and best practice policies and procedures for operational continuity and backup/recovery was necessary. In addition, it was stressed that formal training should be provided on best practice principles and guidelines in these areas. It was envisaged that following this, the individual agencies and other bodies would be able to use the detailed templates for their own plans and needs. 

7.	The Final Report was presented to the NF, NAC and IAs in August 2003 and finalised after remarks were discussed and accepted by the relevant institutions.

Effects:

8. 	From the beginning, the IT component was a collaborative process with TA experts working alongside and cooperating with the national experts from the various IT Departments of the relevant ministries and agencies. There was, therefore, a definitive transfer of know how.

9.	All gaps identified in the Auditors´ report (Gap Assessment) were tackled and plugged.  However, there are still some outstanding issues that need to be developed with additional investment required for the IT technology in all institutions involved.  

10.	There was a positive assessment of the inputs of the experts and of the quality of the work provided. All of the IT departments have developed back up and disaster recovery systems 

1.2.2	Component 2: Development of Human Resources

Activities and Outputs

11. 	The Development of Human Resources within the core institutions was identified as a key pre-requisite for EDIS accreditation and developing the human resource capacity of the key institutions. The Gaps identified in Stage 1 were translated by the expert team into Development and Training Plans. The Development Needs Analysis was carried out through the preparation of individual Personal Development Plans for all relevant staff during June 2003. Account was taken of both current and future development needs arising from the Staff Plan. The finding from the needs analysis created the basis for the Development and Training Plan for the remainder of 2003. The Personal Development Plans were reviewed in December 2003 and further developed in 2004.

Effects

12.	 All related institutions evaluated the impact from this part of TA highly. The work undertaken has had a positive effect in the provision of well-focused training but also in familiarising staff with the planning of training and personal improvement. Other positive inputs from the experts included excellent communication with the national bodies, the logical co-ordination of activities, the justification of training needs with the top management of institutions and very good management of training components themselves. However, impact and sustainability can be seen as questionable since some managers do not seem to be really committed to the results of this component. In some agencies, however, such as NADSME, the project had a very positive impact on HRD and has provided the basis for the on-going monitoring of human resource needs and staff development geared to the EDIS requirements.

1.2.3	Development of Management Procedures

Activities and Ouputs
 
13. 	This project was tendered 2 times and the contract was eventually awarded in June 2003 to ECODES. Whilst having a key role to play in developing management procedures in key institutions, the quality of the TA support and the delivery of services were poor and universally criticised by all agencies concerned. The team was weak, not well qualified in the specific areas required and incapable of targeting assistance to the individual Ministries. Particular problems were encountered in the areas of public procurement and financial management where the TA provided was very poor. The attempt to use a horizontal approach rather than one tailored to the needs of the individual Ministries proved ineffective with little in the way of specific guidance for individual agencies. Training was very brief and misdirected with only a one-day course provided for all agencies involved.

Effects
 
•	14.	The general conclusion was that the exercise had very few positive effects. 
2.	EVALUATION RESULTS

2.1 	Relevance

In relation to the IE of the Stage II Gap Plugging exercise, the following comments can be made:

15.	In relation to Development of Human Resources, this component was clearly relevant and specified in the Gap Assessment Report produced in December 2002 by Deloitte Touche. The recognition of the need of a strong core of administrative and managerial personnel to the success of the EDIS was clearly stated and the development of a strong HRD policy identified for the NF itself and for the key agencies.

16.	However, HR issues are a dynamic process and ultimately the process involves more than the adoption of training plans and job descriptions to the creation of a core cadre of well-motivated, trained and remunerated staff. There needs to be a constant reappraisal of capacities and if it is found to be a binding constraint during the Commission’s Stage IV Accreditation Stage, one solution to raise total relevance to need is to deploy Phare funds to the hiring of experts at these organs and/or to top-up salaries to reflect increased responsibility. There is also the need for a constant reappraisal of staffing needs and re-training as workloads intensify and change. 

17.	With regard to the design arrangements for the Development of Management Procedures, overall, it was envisaged that this component would play a pivotal role in ensuring that operational procedure, manuals etc would be prepared within the key Ministries. There is, however, clearly a disappointment with this exercise. 

18.	With regard to the Development of IT, once again the Gap Assessment Report identified this as an area of concern and proposed remedial action in the areas of security, disaster recovery and back up systems.
2.2 	Efficiency

In the narrower sense related to the Stage II Gap Plugging exercise, the following comments are relevant on the efficiency axiom:

19. 	As far as co-ordination, management and monitoring issues of Development of IT the overall impression gained for the interviews is that it was useful but potential efficiency was constrained by the limitations imposed by the lack of an effective needs analysis or ex-ante evaluation. 

20. 	Development of Management Procedures received a negative feedback from stakeholders. The IE ranking for the Phare intervention in Stage II is unsatisfactory. Problems were highlighted in terms of the level of preparation of the EU experts, the poor mix between the expert team and overall sense that despite positive elements that the intervention was poorly focussed on the true needs of the agencies. 

21.	It is understood that complaints were made internally to the NF, the lead of the contract, by other agencies and that Slovak authorities did also endeavour to act with the FWC, but the restrictions of the EDIS Action Plan provided a restraint. This is unfortunate though understandable – unfortunate since the address of any remaining deep structural flaws may still not fully covered beyond the narrow tunnel of the checklist for Stage III, and may therefore be picked up by Commission auditors during Stage IV, in which case they would still need to be addressed.

22.	The Phare intervention to assist the Development of Human Resources component was overall assessed to be positive, and therefore assessed to be efficient. The Gap Plugging team in Stage II provided guidance on best-practice HR strategy and tools and procedures. 

In the wider sense, the following comments apply:

23.	The submission for Stage IV Commission Accreditation was due in September 2003 but in effect only occurred in December 2003 some 3 months later.

24.	 In fact delays occurred in the overall timeframe, although the timetable for the Stage I was maintained. Time was lost in particular for Stage II with protracted delays in contracting; and in Stage III where for a variety of reasons concerned with tendering (poor submissions, lack of tenders, the ineligibility of candidates, disagreements over TORs etc), the funds were not contracted and lost and the work was eventually funded from the Slovak authorities themselves.

2.3 	Effectiveness

25.	The input of the Phare support in Stage II for the Gap Plugging in the Development of IT was effective broadly in meeting the Stage I Gaps. However, the lack of a comprehensive IT strategy and approach based on a needs analysis in general limits the long-term effectiveness of the intervention unless this is simultaneously addressed. 

26.	Development of Human Resources part was more effective. Substantial efforts have been made to ensure best-practice methods have been integrated and there is a general agreement that the quality of the TA provided was good and well directed. In some cases, for example at NADSME, the project acted as a catalyst to the development of a coherent HRD strategy geared to the needs of EDIS. Most of the institutions have made determined efforts to reinforce and expand their administrative capabilities to meet with the increased workloads that have been imposed upon them. 

However, effectiveness in a dynamic rather than static one-point in time requires that the relative gaps do not worsen. The latter cannot be guaranteed without a major injection of resources into the key administrative and operational nodes of activity, particularly in the context of the transfer of the ex-ante visa. 

27.	Development of Management Procedures is considered unsatisfactory for the Phare supported Stage II. When taking the possible additional use of Phare funds lost that might have been used for Stage III, the ranking diminishes further since it would have been possible to devote greater resources to tackling the inadequacies of Stage II deliverables. 

2.4 	Impact and Sustainability

28.	The key issue here, in the narrow sense, is whether the initiatives undertaken in the context of the Stage II exercise were effective, had the desired impact and provide the basis for long term sustainability in the post-accession period when EDIS accreditation has been secured.

29.	During the contractor’s discussions with the agencies concerned, there was a consensus that the work undertaken in the Development of Human Resources was sound and that it had had a positive impact in providing the basis for HRD in the key agencies. The quality of the TA, the expertise provided and the focus on individual Ministries requirements was considered very satisfactory and providing a basis for long term sustainability. 

30.	By contrast, the expertise provided under the component concerned with the Development of Management Procedures was considered poor, badly focused and disjointed by all agencies contacted. As a result, any long-term sustainability could not be attributed to the actions taken in this element.

31.	For the development of IT, the work undertaken was sound and provided the basis for the development of strong back up systems, disaster recovery systems and support services in the NF and the IAs. It also provided an in depth analysis of the systems and hardware in place as well as identifying further gaps not included in the initial Gap Assessment Report.

2.5	Recommendations

2.5.1	General 

32.	Given the imminence of the May 1 accession date, a plan of action needs be agreed for the period from May 1 until EDIS accreditation for Slovakia within the time bounds set under the Accession Treaty. A joint approach will be needed between Slovakia as a new Member State and the Commission’s Representation Office. The Slovak authorities under the NAO must continue to build through their operational WGs a sufficient critical mass that tackles the basis of the longer-term sustainability and impact concerns. 

33.	The existing co-ordination structures need to continue beyond May 1. The JMC will need to perform its role.. With regard to the status of the Steering Committee it is likely that this will need to stay to act as a focal point for the coordination of on-going activities. As for the WGs, the need for them will be determined by the outcome of the verification audit although it is likely that the WG for financial management will have to remain.

35.	Although it appears that the Slovak Authorities will be allowed to operate under the DIS with the new PRAG Guidelines after May 1st pending a decision on accreditation, this decision needs to be communicated to the Slovak Authorities quickly so that a contingency plan can be put in place.

2.5.4	Summary of Conclusions and recommendations
Conclusion	Recommendation	Output	Reference/
Paragraph	Responsibility	Deadline	Debriefing Results	
Possible delay in receiving EDIS accreditation.	Given the imminence of the May 1 accession date, a plan of action needs be agreed for the period from May 1 until EDIS accreditation for Slovakia.	Action Plan on EDIS accreditation.	32	MoF/ECD	Urgent		
Need to accelerate contracting.	The NAO and the ECD should jointly aim to front-load all contracting by April 30th so that PRAG rules apply for implementation.

	Front Loading of Contracts	37	MoF/IA	Urgent		
Coordination Structures need to be maintained	Decision to be made in light of verification audit on structures to be retained, frequency of meetings etc.	Coordinating Structures	33	MoF, IA	Urgent		

2.5.5	Summary of Evaluation
Criterion

Project	Relevance	Efficiency	Effective-
ness	Impact	Sustain-
ability	Verbal
overall
rating	
Preparation of the National Fund and Implementing Agencies for EDIS implementation within the Stage II (GapPlugging)	
Development of IT	1	1 	1 	1 	1	S	
Development of Human Resources	1 	1 	1	1	0 	S	
Development of Management Procedures	2	-1	-1	-1	-1	U	
Compliance Assessment	1	-2	-2	-2	-2	Hu	
Taking into account the contextual constraints on the evaluation, the sector overall is rated to be “Unsatisfactory”.	
Total for the Sector 	1	-1	-1	-1	-1	U	

ANNEXES

ANNEX 1. Indicators of Achievement OF SECTOR OBJECTIVES AND EVALUATION CRITERIA

Indicators of Achievement 	Valid as Indicator	Remarks 	Reference to evaluation
criteria	
2002/000.585		Indicators were not specified 		

ANNEX 2. LIST OF INTERVIEWS

INSTITUTION 	INTERVIEWEE 	DATE	
EMS Consortium	Mr. Dietmar Aigner
EMS evaluator
	21/01/2004	
Central Finance and Contracting unit
Ministry of Finance of SR
Stefanovicova 5
817 82Bratislava	Ms. Silvia Czuczorová,
Director
Ms. Daniela Migrová,
Ms. Anna Oszlányiová	11/02/2004	
Delegation of the European Commission
Panska 3
SK-811 01 Bratislava	Mr. Michel Doumont
Deputy
Phare/ISPA Section
	16/01/2004	
Delegation of the European Commission
Panska 3
SK-811 01 Bratislava	Mr. Dieter Thiel,
Head of Phare Unit
Mr. Jochen Schult,
Financial Manager
Mr. Marek Madry
Task Manager	11/02/2004,
15/02/2004
14/04/2004	
Ministry of Construction and Regional
Development, Agency for Regional
Development Support Agency
Spitalska 8	Ms. Annamaria Macurova
Lawyer
Ms Zuzana Andrássy
Director	11/02/2004
15/04/04	
National Agency for Small and Medium
Enterprises
Prievozska 30
SR-821 05 Bratislava	Ms. Zuzana Cigánková,
Financial Manager, 
Mr. Daniel Pitoňák,
Ms. Barbora Vargová
	12/01/2004	
Office of the Government of the SR
Aid Co-ordination Unit
Namestie slobody 1
SK-813 70 Bratislava	Ivan Fecenko
Director
	12/2003	
Ministry of Finance of the Slovak Republic
Stefanovicova 5
817 82 Bratislava 15	Mr. Tvaroška,
State Secretary,
Mrs.Maria Kompisova
Mr. Mário Virčík,
Ms. Ľubica Slimáková,
Ms. Petra Pazerini
Ms. Mária Hrachovcová,
Mr. Peter Bagin	11/02/2004,
19/02/2004*,
20/02/2004
23/02/2004
24/02/2004
14/04/04
16/04/04

Office of the Government of the SR
Aid Co-ordination Unit
Namestie slobody 1
SK-813 70 Bratislava	Mr. Martin Orth
Task Manager
	13/02/2004,
20/02/2004**	
Telephone interviews are indicated with * and e-mail/fax correspondence with **	

ANNEX 3. LIST OF Documents Referred to in the INTERIM EVALUATION

Name of Originator	Date	Title of Document	
Office of the Government 	2001 	National Programme for Adoption of the Acquis	
EMS Consortium	December 12th 2003	Country Phare Evaluation Review: R/SK/CPER/03.112	
Government of the Slovak
Republic/ European Commission	December 2003 	Relevant programming documentation:
 such as Work programme, ToRs, Contracts, Contractors’ reports for the Development of Human resources, Development of Management Procedures, Development of IT Component	
Ministry of Finance/ACU	January 2004	Presentation of Phare EDIS application,
Submission report,
Minutes of the HLWG 1-3,
JMC minutes,
Institutional arrangements of Ex-ante approval of Phare and Transition Facility projects in the SR within EDIS system,
Governmental Decree nr. 1103/2003 on Shifting the competence from the European Commission to the national institutions related to ex-ante approval within the Extended Decentralised Implementation System for the Phare, ISPA and Transition Facility programmes in the Slovak Republic,
Checklist for EDIS request for the National Aid co-ordinator/Aid co-ordination unit
Minutes of Steering Committee Meetings
Action Plan for Compliance Assessment, EDIS Phare
Work Programme Action Plan 6, EDIS Phare
Gap Assessment Report, Phare EDIS Stage I
Annex 1 to Gap Assessment Report
Compliance Assessment Report, Phare EDIS Stage 3

Documents requested but not made available (with reasons): not applicable.
ANNEX 4 - DISSENTING VIEWS

There were a number of dissenting views on the part of the Aid Coordination Unit (ACU) and these are summarised below.

i)	The Evaluation Unit at EC Headquarters in Brussels was not approached for comments on the Interim Evaluation and this should be undertaken to provide adequate checks and balances.

ii)	The evaluators were requested to use their expertise to  suggest some example of measurable indicators but this was declined since this was not in their TOR.

iii)	The evaluators were requested to elaborate more on the action plan but this was declined since it was considered to be outside the scope of the TOR.

iv)	The component relating to Compliance Assessment should not have been included in the IE since it was funded from national sources and not from this programme. Scoring it Highly Unsatisfactory due to the loss of funds was not therefore correct. That is why the report, according to the Aid Coordination Unit, gives a somewhat distorted impression of what actually happened (achievement of objectives) under Compliance Assessment stage of EDIS. The evaluators included this component despite its funding from national sources since it was based purely on the degree to which the objectives of this component were realised. The scoring was not a reflection of the performance of the project.