Evaluating European Commission NMP and Horizon 2020 SME and FTI Proposals (2008 - present)



Richard Ball undertakes a role as an independent expert for the evaluation of SME Instrument, Fast Track to Innovation (FTI), and previously FP7 NMP proposals on behalf of the European Commission (EC). 


Working individually, the task for each expert is to give a score for each criterion, with explanatory comments, and prepares an ‘individual evaluation report (IER)’. 



After carrying out an individual evaluation, an expert will join other experts who have evaluated the same proposal in a consensus group, to agree on a common position, including comments and scores. 



After the consensus phase a panel of experts:

 

• reach an agreement on the scores and comments for all proposals within a call, checking consistency across the evaluations
• if necessary, propose a new set of marks or revise comments, and resolve cases where evaluators were unable to agree
• rank the proposals having a qualifying score and give a priority order for proposals with the same score

 

Before notifying coordinators of the final evaluation results, the EC reviews the results of the experts’ evaluation and puts together the final ranking list.The EC then:

 

• produces a ranked list of proposals, including all proposals with scores above the qualifying score
• draws up a list of proposals for possible funding from that list. The number of proposals in the list depends on the available budget.

 

After the finalisation of the evaluation, applicants receive the Evaluation Result Letter (ESR).

Observer - Reviewing Past Evaluation Processes and Providing Recommendations on Improvement for Future European Commission FTI Programmes (2018 - present)



To undertake a role as observer for the H2020 European Innovation Council (EIC) Fast Track to Innovation (FTI) programme, the instructions for the evaluation process were as follows:

 

1. Observe the process of the evaluations (Individual Expert Reports (IER) and Evaluation Summary Reports (ESR).
2. Select proposals from each call deadline and from across the 15 keyword panels.
3. Review and make suggestions on the evaluation grid and how it could be improved.
4. Review and comment on the approach in general with a view to the future.

 

Keston observations are carried out in two stages. First, a review of the briefing material to ensure familiarity with the call process and coverage and to assess the quality of the briefing material. Secondly, selected evaluations were accessed to look at the comments and markings that assessors had provided and assess the ways that individual evaluators had interpreted the guidance notes. Comments and scores were also reviewed for consistency, with particular attention paid to large scoring variations, and especially where one evaluator’s scoring or comments were vastly different from the other three. A number of conclusions and recommendations were drawn across the following subcategories:

 

• Evaluator Markings
• Evaluator Comments
• Scale and Complexity
• Transparency
• Throughput Time
• Efficiency, Reliability and Usability
• Impartiality, Fairness and Confidentiality
• Conformity with Rules
• Quality of Process
• Overall Quality of Evaluation
• Quality of Briefing Materials
• Quality of Scoring Sheet
• Quality of Monitor’s Report Template

 

Independent Review of the Technology Developer Accelerator Programme (2015-2019)



The Advanced Propulsion Centre (APC) has developed and implemented the Technology Developer Accelerator Programme (TDAP), a programme aimed at small and medium sized technology developers (TDs) with low carbon propulsion technologies that can be applied to the automotive sector. Working with Beta Technology, Keston has undertaken several independent reviews of the TDAP, in successive waves, including an assessment of economic impact and an opinion on its design and implementation.



The reviews have been undertaken through collecting and collating evidence of the effectiveness of the interventions.

Project Technical Advisor (PTA) for the European Commission on collaborative R&D projects (2012 – 2017)



On behalf of the European Commission, and through Beta Technology, Keston provided project monitoring services for six European-funded projects. These were large, multi-partner, multi-national projects lasting between three and five years and with budgets ranging from 5m to 15m Euros each. Most of the activity was undertaken remotely but did entail meetings with the project consortia on a six monthly basis.

Monitoring and Evaluation of the UK’s Micro and Nano Technologies Facilities Network (2007-2012)



The UK Government established a range of facilities offering world class MNT technology. A key emphasis of these facilities was to understand and stimulate their markets to generate growth as well as developing technology. Richard Ball was assigned as a monitor to a number of these projects for a six year period, with the monitoring having both a strategic and business focus. The monitoring focused on the following aspects of the projects:

 

• Impact and additionality, particularly that the facilities demonstrate that investment has created new innovative projects through the exploring of new technologies and markets.
• Business Planning.
• Management and Governance.
• Business Development.

 

The contract was successful in supporting UK Government, via the Technology Strategy Board, in monitoring and supporting the facilities projects. 

Development of the PTA monitoring framework for the European Commission (2011)



Working with Beta Technology, Keston assisted in the development of the framework through which a large number of European funded projects were monitored and evaluated. The requirement was to provide a framework to assist the EC directorate to monitor the progress on projects funded in Theme 4 – Nanosciences, Nanotechnologies, Materials and new Production Technologies within the 7th Framework Programme. The framework ensured that the participants in the projects performed correctly their activities to the best possible program requirements.



The monitoring framework that was developed operated to:

 

• Provide advice to the EC Project Officers;
• Monitor the progress of the projects;
• Assess periodic and final reports of the projects;
• Propose workable solutions to any contractual and/or managerial matters;
• Develop links between projects of a similar nature.

 

Development of a pilot environmental technology evaluation scheme as part of the European Environmental Technologies Action Plan (2006-2009)



Through Beta Technology, Keston worked on the development of an evaluation scheme for environmental technologies on behalf of the European Commission. 



A total of 15 technologies were put through the pilot scheme in three areas – waste water, soil remediation, and energy technologies (renewables and energy efficiency). The project results have fed into the process of developing a proposed mechanism and recommendations for a European-wide evaluation scheme. 



The project involved collaboration at international level with other evaluation schemes in the USA and Canada. 



Recommendations from the pilot were used to draft a policy proposal. 

Other Monitoring and Evaluator Roles



Keston has also been responsible for evaluating proposals for various other funding schemes and monitoring projects throughout Europe and Australia. Most notably, these include:

 

• International expert for the evaluation of the project “Centre for Research and Utilization of Renewable Energy - CVVOZE” on behalf of the Czech Ministry of Education, Czech Republic (2013).
• Expert Evaluator of projects supported under the Priority Axes 1 and 2 OP RDI.
• Expert evaluator for Czech-Norwegian Research Programme (CZ09).
• Eurostars (Eureka Programme) evaluations (2011 to present).
• The Flanders ‘Strategic Initiative Materials’, Belgium.
• The Czech Ministry of Education, Czech Republic, (evaluator for a call for proposals for regional R&D centres in the Czech Republic – and member of Panel of International Experts, allocating €250m of funding to technology R&D centres in a variety of disciplines, including biotechnology, ICT, engineering and materials science (2009 to present)).
• The Great Southern Development Commission, Australia, Independent Assessor for the evaluation of grant submissions under the Regional Grant Scheme (2005 to 2016).