Duration: M2 - M24
Project leader: OBU
Other participants: INSEAD, AUP, UTA, OBU, CELN, STC

Deliverables

• D4.1 Evaluation approach, methodology, document (including a first draft of the key indicators)– M06 - PP
• D4.2 Result of the formative evaluation – M09
• D4.3 Evaluation plan (including a consolidated version of the indicators) – M12
• D4.4 Final evaluation report (includes both the assessment of the result and the strategic evaluation) – M23

Objectives

The objective of this workpackage is to develop a robust and effective evaluation program and document the results of evaluation, for all concepts and models developed within the project, starting with early prototyping of the concepts developed in WP1 and WP2 through to final demonstrators.

Evaluation within the project occurs at three different levels.

• formative evaluation: the objective is to identify the users’ needs and to define or validate usage scenarios
• summative evaluation : The objective is to evaluate the work that has been produced
• substantive evaluation : The objective consists in assessing the substantive value of the knowledge (technical components or approaches) that has been generated in this project.

It will rely on a set on the specification of well defined set of key indicators (qualitative or quantitative) that will be used to access the achievements of the project (including the value of the ideas and the knowledge assets generated in this project).

Description of work

• T4.1: Formative evaluation: Will consist in the elaborating high-level scenarios with users, and collecting user’s preliminary feedbacks. This information will be used as an input for the elaboration of the conceptual framework (relevance), and will be used for adjustment purpose in the design.
• T4.2 : Elaboration evaluation frameworks for attention (oriented toward effectiveness rather than efficiency) and for advanced user interface (with cognitive rather than ergonomic orientation). A particular focus will be the evaluation of the substantive value contributed by each of the mechanisms (for instance protocols of tests with or without artificial characters, with out without support for attention and direction, could be considered). Besides It will include the specification of a set of key indicators helping to assess the achievements project in all its dimensions.
• T4.3 : Definition of the evaluation plan (summative evaluation) Consists in the concrete operationalization of the methodology (definition of the characteristic of the test groups, specification of the evaluation action and of the schedule of their execution)
• T4.4 : Analysis of the result (summative evaluation)
• T4.5 : Strategic Evaluation (Lesson learned, prospective, etc.).

Milestones and expected result

• M4.1 Evaluation approach. – M06

  • The project has now a good idea of the methodological part of the evaluation (methods that will be used for the evaluation, balance between qualitative versus quantitative evaluation, first draft of the key indicators, etc.).

• M4.2 Formative evaluation has been completed – M09

  • The result of this evaluation (which first result will be available before M09) will be able to validate, guide, and help to adjust the design.

• M4.3 Evaluation Plan is ready – M12

  • This report will provide the information necessary for organising the work of the validation pilot.
• M4.4 Evaluation Report – M23