D2.1 Requirements and Specification - CORBYS

CORBYS 

Cognitive Control Framework for Robotic Systems 

(FP7 – 270219) 

Deliverable D2.1 

Requirements and Specification 

Contractual delivery date: Month 6 

Actual submission date: 31st July 2011 

Start date of project: 01.02.2011 Duration: 48 months 

Lead beneficiary: The University of Reading UR Responsible person: Prof. Atta Badii 

Revision: 1.0 

Project co-funded by the European Commission within the seventh Framework Program 

Dissemination Level 

PU Public X 

PP Restricted to other program participants (including the Commission Services) 

RE Restricted to a group specified by the consortium (including the Commission Services) 

CO Confidential, only for members of the consortium (including the Commission Services)

D2.1 Requirements and Specification 

Document Contributions History 

Author(s) Date Contributions 

Prof. Atta Badii (UR) 24-02-2011 

14-03-2011 

21-03-2011 

25-03-2011 

Requirements Eng. SoA Section Document Structure 

Requirements Eng. Domain Knowledge Framework 

Requirements Eng. Deliverable Integrated Structure 

Requirements Eng, Progress Verification-Structuring 

Rationale 

Zlatko.Matjacic (IRSZR) 18-02-2011 Background Information 

Zlatko.Matjacic (IRSZR) 25-03-2011 Gait Rehabilitation Domain Knowledge 

Matthias Spranger (NRZ) 14-03-2011 Gait Rehabilitation Requirements Document II 

Mathias Spranger (NRZ) 30-03-2011 Comments on UR’s document 

Matthias Spranger (NRZ) 14-04-2011 Gait Rehabilitation Requirements and User Types 

Michael.Vandamme (VUB) 13-04-2011 State-of-the Art (Gait Rehabilitation Systems) 

Marco Creatura (BBT) 15-04-2011 State-of-the-Art in Non-Invasive Brain Computer Interface 

Detection of Motor Activities (BBT) 

Daniel Polani (UH) 19-04-2011 State-of-the-Art in Behaviour Generation, Anticipation and 

Initiation 

Hanne Opsahl Austad 28-04-2011 State-of-the-Art in Sensors and Perception 

(SINTEF) 

Sinisa Slavnic (UB) 09-05-2011 State-of-the-Art in Architectures for Cognitive Robot 

Control 

Ali Khan 

Prof. Atta Badii 

Daniel Thiemert (UR) 

11-05-2011 State-of-the-Art in Situation Assessment 

Requirements Eng. Domain Knowledge (Gait rehabilitation 

meeting with experts 28/04/2011) 

Roko Tschakarow 20-05-2011 State-of-the-Art in Smart Integrated Actuators 

Rajkumar Raval 

Ali Khan (UR) 

26-05-2011 State-of-the-Market – First Demonstrator 

Frode Strisland (SINTEF) 27-05-2011 Review of Human Sensing requirements 

Contribution to CORBYS glossary 

Marco Creatura (BBT) 30-05-2011 Reviewing BCI requirements (chapter 6.4 and 7.3) 

Danijela Ristic-Durrant 30-05-2011 State-of-the-Art in Robotic Autonomous Systems for 

(UB) 

examining Hazardous Environments 

Sinisa Slavnic (UB) 30-05-2011 Revision of Control sub-system requirements 

Ali Khan 

02-06-2011 Situation Assessment Architecture specific requirements 

Daniel Thiemert 

Prof. Atta Badii (UR) 

UI-REF methodology 

Rajkumar Raval 

Ali Khan (UR) 

02-06-2011 Task 4.3 specific requirements 

Christoph Salge (UH) 03-06-2011 Revision of SOIAA specific requirements 

Svetlana Grosu (VUB) 06-06-2011 Requirements specific to: 

Development of Low level control units (Task 7.4) 

Experimenting and Evaluating simulations (Task 6.4) 

Christoph Salge (UH) 13-06-2011 Provided SOIAA output requirements 

Christoph Salge (UH) 14-06-2011 Tasks 6.1 and 6.5 specific requirements 

Frode Strisland (SINTEF) 15-06-2011 Requirements specific to system integration (WP8) 

Sinisa Slavnic (UB) 17-06-2011 Revision of requirements specific to Tasks 5.6, 6.2, 6.3, 6.6 

II


Matthias Spranger (NRZ) 04-07-2011 

and 9.4). 

Addition of requirements specific to Task 3.3 

Requirements specific to Task 9.5 

NRZ, IRSZR, UR, UB, 30-06-2011 Chapter 4 revision – domain knowledge, background, 

VUB, OB, OBMS. 

requirements, demonstrator I 

OB 07-07-2011 Task specific requirements 

OBMS 07-07-2011 Task specific requirements 

SCHUNK 07-07-2011 Task specific requirements 

Markus Tuttemann (OB) 08-07-2011 State-of-the-Market (Demonstrator I) reduced / revised 

NRZ, IRSZR 10-07-2011 Chapter 4 revision – agreed, finalised requirements for 

demonstrator I 

Marco Creatura (BBT) 12-07-2011 Updating BCI requirements: BCISW11, BCI1 and 7.3.2.4 

Cornelius Glackin (UH) 13-07-2011 Revision SOIAA section 7.7 

UR 18-07-2011 Review, corrections in State-of-the-Art sections (Chapter 10 

onwards) 

Danijela Ristic-Durrant 

Adrian Leu (UB) 

16-07-2011 Requirements and SoM Demonstrator II added (Section 4.7, 

4.8) 

Section 9.2 requirements added, revised 

Matthias Spranger (NRZ) 23-07-2011 Chapter 4 revision 

Frode Strisland (SINTEF) Revision section 5.2, chapters 8 and 10 

BBT 26-07-2011 Deliverable 2.1 review 

UB 27-07-2011 Deliverable 2.1 review Chapters 1-9 

UB 28-07-2011 Review of SOA sections 

UR 28-07-2011 Final integration of Reviewers comments, extensive editing 

and formatting 

III


Glossary 

Term Definition 

CORBYS Roles 

CORBYS User ANY user interacting with the CORBYS system, for example, 

in case of gait rehabilitation system, users with the following roles: a 

patient, a practitioner or an engineer; 

In case of a mobile robotic system, users with the following role: a 

hasardous area examination officer. 

CORBYS End-user The companies/entities that use/exploit (aspects of) CORBYS technology 

in their commercial products or services. 

CORBYS Expert / Professional user A professional dealing with the CORBYS system based on a need to do 

technical maintenance, repairs or system configurations. For example: a 

Practitioner, a Developer, an Engineer. 

Operators/Tester/Assessor/Configurer/ 

Expert/ Practitioner user/end-user/ 

CORBYS Patient The person receiving gait rehabilitation therapy aided by the CORBYS 

system 

CORBYS Practitioner The medical professional configuring and assessing rehabilitation therapy 

aided by the CORBYS system. 

CORBYS Domain Knowledge 

Sensor Fusion Method used to combine multiple independent sensors to extract and 

refine information not available through single sensors alone. 

Situation Assessment Estimation and prediction of relation among objects in the context of their 

environment. 

Cognitive Control Capability to process variety of stimuli in parallel, to “filter” those that are 

the most important for a given task to be executed, to create an adequate 

response in time and to learn new motor actions with minimum assistance. 

Human-Robot Interaction Ability of a robotic system to mutually communicate with humans. 

Neural plasticity Ability of neural circuits, both in the brain and the spinal cord, to 

reorganise or change function. 

Security Robots Mobile platforms equipped with different sensors (cameras, laser, 

scanners, etc.) which allow autonomous or teleoperated navigation. 

Cognitive Processes Processes responsible for knowledge and awareness, they include the 

processing of experience, perception and memory. 

BCI Brain Computer Interface 

CORBYS Technology Components 

CORBYS User Interface User interface which suits the needs of the 

patient/therapists/developers/cognitive modules 

Environmental Sensors Sensors measuring the (physical) environment of the human-robot system, 

e.g. collision avoidance sensors. 

SAWBB Situation Awareness Blackboard 

SOIAA Self-Organising Informational Anticipatory Architecture 

Smart Actuators Highly integrated Mechatronic units incorporating a motor and the 

complete motion control electronics in one single unit. 

CORBYS Patient Sensors The sensor systems taking measurements on the patient 

CORBYS Physiological Sensors Sensors measuring physiological parameters on the patient, for example 

heart rate, EEG, muscle tension 

CORBYS Patient mechanical sensors Sensors measuring movements, forces, torques, joint angles and similar at 

any given position on the patient. 

Patient Inertial Measurement Units Sensors providing inertia information for specific parts of the human 

body. 

IV


Table of Contents 

1 ABSTRACT ..................................................................................................................................................... 1 

2 EXECUTIVE SUMMARY AND REPORT SCOPE ................................................................................................... 1 

3 INTRODUCTION ............................................................................................................................................. 2 

4 REQUIREMENTS ENGINEERING ANALYSIS BASE .............................................................................................. 5 

4.1 KNOWLEDGE ELICITATION FROM CLINICAL EXPERTS FOR DEMONSTRATOR I ...................................................................... 5 

4.2 ENVIRONMENT OF THE DEMONSTRATOR I: GAIT REHABILITATION USER‐ROBOT INTERACTION ............................................. 11 

4.3 REQUIREMENTS FOR DEMONSTRATOR I .................................................................................................................. 13 

4.4 REQUIREMENTS FOR DEMONSTRATOR II ................................................................................................................. 16 

4.5 REQUIREMENTS ENGINEERING METHODOLOGY (UI‐REF) SALIENT FEATURES ................................................................ 17 

4.6 STATE‐OF‐THE‐MARKET ...................................................................................................................................... 21 

5 MECHATRONIC CONTROL SYSTEMS ............................................................................................................. 28 

5.1 CANONICAL SUB‐SYSTEMS .................................................................................................................................... 28 

5.2 HUMAN SENSING SYSTEMS (TASKS 3.1, 3.2; SINTEF) .............................................................................................. 28 

5.3 ROBOTIC SYSTEM MODELLING AND INTEGRATION OF MOTOR CONTROL UNITS (TASK 6.2, UB) ......................................... 35 

5.4 DESIGN AND DEVELOPMENT OF THE MOBILE PLATFORM (TASK 7.1, OBMS).................................................................. 38 

5.5 DESIGN AND DEVELOPMENT OF THE POWERED ORTHOSIS (TASK 7.2, OB) ..................................................................... 39 

5.6 DESIGN AND INTEGRATION OF ACTUATION SYSTEM (TASK 7.3, SCHUNK) .................................................................... 42 

5.7 DEVELOPMENT OF LOW‐LEVEL CONTROL UNITS (TASK 7.4, VUB) ................................................................................ 43 

5.8 REALIZATION OF THE ROBOTIC SYSTEM (TASK 7.5, SCHUNK) .................................................................................... 45 

6 HUMAN CONTROL SYSTEM .......................................................................................................................... 46 


6.2 BCI DETECTION OF COGNITIVE PROCESSES THAT PLAY KEY ROLES IN MOTOR CONTROL AND LEARNING (TASK 3.3, UB) ........... 47 

6.3 BCI SOFTWARE ARCHITECTURE (TASK 3.4, BBT) ..................................................................................................... 48 

6.4 ARCHITECTURE DECOMPOSITION AND DEFINITION (TASK 6.1, UH) .............................................................................. 50 

6.5 INTEGRATION OF COGNITIVE CONTROL MODULES (TASK 6.3, UB) ................................................................................ 51 

6.6 EXPERIMENTING AND EVALUATING SIMULATIONS (TASK 6.4, VUB) ............................................................................. 54 

6.7 ARCHITECTURE REVISION AND IMPROVEMENT (TASK 6.5, UH) ................................................................................... 55 

6.8 FINAL ARCHITECTURE INTEGRATION AND FUNCTIONAL TESTING (TASK 6.6, UB) ............................................................. 55 

7 ROBOHUMATIC SYSTEMS (GRACEFUL ROBOT‐HUMAN INTERACTIVE‐COOPERATIVE SYSTEMS) .................... 59 


7.2 BCI COGNITIVE INFORMATION (TASK 3.5, BBT) ...................................................................................................... 59 

7.3 DEVICE ONTOLOGY MODELLING (TASK 4.1, UR) ..................................................................................................... 63 

7.4 SELF‐AWARENESS REALISATION (TASK 4.2, UR) ...................................................................................................... 64 

7.5 ROBOT RESPONSE TO A SITUATION (TASK 4.3, UR) ................................................................................................... 68 

7.6 SOIAA: SELF‐ORGANIZING INFORMATIONAL ANTICIPATORY ARCHITECTURE (TASKS 4.4, 5.1, 5.2, 5.3, 5.4; UH)............... 74 

7.7 USER RESPONSIVE LEARNING AND ADAPTATION FRAMEWORK (TASK 5.5, UR) .............................................................. 81 

7.8 COGNITIVE ADAPTATION OF LOW LEVEL CONTROLLERS (TASK 5.6, UB) ......................................................................... 82 

8 SYSTEM INTEGRATION AND FUNCTIONAL TESTING (WP8, SINTEF) ............................................................... 85 

8.1 CONFORMANCE TESTING ON SUB‐SYSTEM AND SYSTEM LEVEL ..................................................................................... 85 

8.2 INTEGRATION OF SUB‐SYSTEMS ............................................................................................................................ 90 

9 EVALUATION (WP9, IRSZR) .......................................................................................................................... 94 

V


9.1 EVALUATION METHODOLOGY, BENCHMARKING, METRICS, PROCEDURES AND ETHICAL ASSURANCE (TASK 9.1, UR) ............... 94 

9.2 TRAINING ON CORBYS SYSTEM (TASK 9.2, IRSZR) ................................................................................................. 96 

9.3 CONTINUOUS ASSESSMENT OF THE TECHNOLOGY UNDER THE DEVELOPMENT (TASK 9.3, IRSZR) ....................................... 97 

9.4 EVALUATION OF THE RESEARCHED METHODS ON THE SECOND DEMONSTRATOR (TASK 9.4, UB) ........................................ 98 

9.5 EVALUATION AND FEEDBACK TO DEVELOPMENT (TASK 9.5, NRZ) ................................................................................ 99 

10 STATE‐OF‐THE‐ART IN SENSORS AND PERCEPTION (SINTEF) ....................................................................... 101 

10.1 INTRODUCTION TO SENSORS AND PERCEPTION ....................................................................................................... 101 

10.2 SENSOR PRINCIPLES FOR PERCEPTION IN HUMAN‐ROBOT INTERACTION ........................................................................ 101 

10.3 INTERPRETATION OF MULTIPLE SENSOR SIGNALS ..................................................................................................... 109 

10.4 SUMMARY ON TECHNOLOGY GAPS AND PRIORITIES FOR DEVELOPMENT IN CORBYS ...................................................... 111 

11 STATE‐OF‐THE‐ART IN SITUATION ASSESSMENT (UR) ................................................................................. 112 

11.1 INTRODUCTION ................................................................................................................................................ 112 

11.2 SITUATION ASSESSMENT .................................................................................................................................... 113 

11.3 RELEVANT APPROACHES .................................................................................................................................... 116 

11.4 SUMMARY ON TECHNOLOGY GAPS AND PRIORITIES FOR DEVELOPMENT IN CORBYS ...................................................... 120 

12 STATE‐OF‐THE‐ART IN BEHAVIOUR GENERATION, ANTICIPATION AND INITIATION (UH) ............................. 120 

12.1 INTRODUCTORY COMMENTS .............................................................................................................................. 120 

12.2 INFORMATION‐THEORETIC PRINCIPLES ................................................................................................................. 122 

12.3 SELF‐ORGANISED BEHAVIOUR AND GOAL GENERATION ........................................................................................... 125 

12.4 BEHAVIOUR ANTICIPATION, GENERATION AND INITIATION ....................................................................................... 128 

12.5 TECHNOLOGICAL GAPS ...................................................................................................................................... 130 

13 STATE‐OF‐THE‐ART IN ARCHITECTURES FOR COGNITIVE ROBOT CONTROL (UB) .......................................... 131 

13.1 ARCHITECTURES FOR COGNITIVE CONTROL OF ROBOTIC SYSTEMS ............................................................................... 132 

13.2 COGNITIVE ARCHITECTURES USED FOR CONTROLLING DIFFERENT ROBOTIC SYSTEMS ....................................................... 135 

13.3 CORBYS ENABLING POTENTIAL AND CONSTRAINTS (CURRENT GAPS/SHORTCOMINGS) ................................................... 143 

14 STATE‐OF‐THE‐ART IN SMART INTEGRATED ACTUATORS (SCHUNK) ........................................................... 143 

14.1 INTRODUCTION TO SMART INTEGRATED ACTUATORS ............................................................................................... 143 

14.2 BASIC ACTUATOR TECHNOLOGIES ........................................................................................................................ 143 

14.3 CONTROL TECHNIQUES, INTERFACING, STANDARDISED DRIVE MODULES ...................................................................... 145 

14.4 CORBYS ENABLING POTENTIAL AND CONSTRAINTS (CURRENT GAPS/SHORTCOMINGS) ................................................... 146 

14.5 TECHNOLOGY INNOVATION REQUIREMENTS GAPS FILTER ELEMENTS ........................................................................... 146 

15 STATE‐OF‐THE‐ART IN NON‐INVASIVE BRAIN COMPUTER INTERFACE (BBT) ................................................ 147 

15.1 INVASIVE VS. NON‐INVASIVE BCI TECHNOLOGY AND ROBOTICS ................................................................................ 147 

15.2 HARDWARE FOR NON‐INVASIVE BRAIN‐COMPUTER INTERFACES ............................................................................... 149 

15.3 SOFTWARE FOR NON‐INVASIVE BRAIN‐COMPUTER INTERFACES ................................................................................ 152 

15.4 THE ROLE OF EEG ARTEFACTS IN NON‐INVASIVE BCIS ............................................................................................ 155 

15.5 DECODING THE COGNITIVE PROCESS REQUIRED IN CORBYS .................................................................................... 157 

16 STATE‐OF‐THE‐ART IN GAIT REHABILITATION SYSTEMS (VUB) .................................................................... 163 

16.1 GAIT REHABILITATION ....................................................................................................................................... 163 

16.2 GAIT REHABILITATION ROBOTS ............................................................................................................................ 165 

16.3 ROBOT CONTROL STRATEGIES FOR GAIT ASSISTANCE ................................................................................................ 170 

17 STATE‐OF‐THE‐ART IN ROBOTIC SYSTEMS FOR EXAMINING HAZARDOUS ENVIRONMENTS ......................... 174 

VI


17.1 ROBOTIC SYSTEM FOR AUTOMATED SAMPLING ....................................................................................................... 175 

17.2 INTELLIGENT AUTOMATED INVESTIGATION OF A HAZARDOUS ENVIRONMENT ................................................................ 176 

18 CONCLUSION ............................................................................................................................................. 177 

19 REFERENCES .............................................................................................................................................. 178 

VII


Table of Figures 

FIGURE 1: NORMAL WALKING IN SAGITTAL PLANE (TOP) AND IN FRONTAL PLANE (BOTTOM) 7 

FIGURE 2: TOE WALKING IN SAGITTAL PLANE 8 

FIGURE 3: CROUCH GAIT IN SAGITTAL PLANE 9 

FIGURE 4: STIFF KNEE GAIT IN SAGITTAL PLANE (TOP) AND IN FRONTAL PLANE (BOTTOM) 10 

FIGURE 5: STIFF KNEE GAIT WITH CIRCUMDUCTION IN SAGITTAL PLANE (TOP) AND IN FRONTAL PLANE (BOTTOM) 11 

FIGURE 6: USUAL WORKFLOW IN A REHABILITATION UNIT 12 

FIGURE 7: PRIORITISATION PROCESS IN UI‐REF 19 

FIGURE 8: DISTRIBUTION OF TOTAL COST OF MS IN EUROPE (YEAR 2005) BY RESOURCE USE COMPONENTS 22 

FIGURE 9: NON‐FATAL INJURIES PER 1000 BY SEX AND AGE GROUP 24 

FIGURE 10: ASENDRO EOD ROBOT 25 

FIGURE 11: DRAGON RUNNER 26 

FIGURE 12: TECHNOROBOT RIOTBOT WITH REMOTE CONTROL 27 

FIGURE 13: IROBOT 510 PACKBOT 27 

FIGURE 14: EOD‐ROBOTER MADE BY TELEROB 28 

FIGURE 15: WILLHELM EINTHOVEN’S SETUP FOR MEASURING THE ECG SIGNALS (LEFT) 102 

FIGURE 16: ILLUSTRATION OF THE ECG SIGNAL AND THE EFFECT OF AN ABRUPT MECHANICAL DISTURBANCE. 103 

FIGURE 17: 12 LEAD ECG 103 

FIGURE 18: ECG ELECTRODES. TO THE LEFT IS AN EXAMPLE OF DISPOSABLE ELECTRODES 104 

FIGURE 19: THE LEFT IMAGES SHOW THE BIOMETRICS LTD EMG MONITORING SYSTEM. 104 

FIGURE 20: NONIN ONYX II 9560 AND DISPOSABLE 7000A ADULT SENSOR 106 

FIGURE 21: THE XSENS MTX INERTIAL MONITORING UNIT OFFERING SIMULTANEOUS 107 

FIGURE 22. TEKSCAN F‐SCAN® SYSTEM , 108 

FIGURE 23: GONIOMETERS FROM PLUX (LEFT) AND BIOMETRICS LTD 108 

FIGURE 24: HIDALGO EQUIVITAL MEASURING UNIT (LEFT) AND BELT (RIGHT) 109 

FIGURE 25: SINTEF ESUMS CHEST UNIT BELT 109 

FIGURE 26: LAMBERT'S APPROACH TO SEMANTIC FUSION 113 

FIGURE 27: CASE‐BASED REASONING PROCESS 117 

FIGURE 28: THE ILLUSTRATION OF TYPICAL THREE LAYERS ARCHITECTURE 134 

FIGURE 29: ARMAR ARCHITECTURE (BURGHART ET AL.2005) 136 

FIGURE 30: A HUMANOID ROBOT ARMAR (BURGHART ET AL.2005) 137 

FIGURE 31: IMA MULTIAGENT‐BASED COGNITIVE ROBOT ARCHITECTURE (KAWAMURA ET.AL. 2004 138 

FIGURE 32: THE LAYERS OF ICUB ARCHITECTURE (SANDINI ET AL. 2007) 140 

FIGURE 33: ICUB COGNITIVE ARCHITECTURE (SANDINI ET AL. 2007) 140 

FIGURE 34: ARCHITECTURE OF CARE‐O‐BOT (HANS ET. AL. 2001) 141 

FIGURE 35: CARE‐O‐BOT 142 

FIGURE 36: ROTARY ELASTIC CHAMBERS – ACTUATOR (IAT BREMEN) 144 

FIGURE 37: FLEXIBLE FLUIDIC ACTUATOR (AIA KIT, KARLSRUHE) 144 

FIGURE 38: DYNAMIXEL SMART ACTUATORS INCLUDING REDUCTION GEAR, CONTROLLER, MOTOR AND DRIVER 145 

FIGURE 39: SCHUNK MODULAR SMART ACTUATOR SYSTEM WITH ROTARY AND LINEAR DRIVES 146 

FIGURE 40: EEG DRY DEVICES 151 

FIGURE 41: DRY AND PORTABLE RECORDING EEG SYSTEM PROVIDED FROM G.TECH: DRY ELECTRODES (LEFT) 152 

FIGURE 42: EXAMPLE EEG ARTEFACTS 155 

FIGURE 43: VOLUNTARY HAND MOVEMENT PHASES: MOTION INTENTION, 158 

FIGURE 44: GAIT REHABILITATION ROBOTS: END‐EFFECTOR BASED (E.G. HAPTICWALKER) 165 

FIGURE 45: END‐EFFECTOR TYPE DEVICES: (FROM LEFT TO RIGHT) HAPTICWALKER®, G‐EO®, ARTHUR 166 

FIGURE 46: COMMERCIALLY AVAILABLE EXOSKELETON TYPE DEVICES: 167 

FIGURE 47: BILATERAL PROTOTYPES: (FROM LEFT TO RIGHT) LOPES, PAM/POGO, WALKTRAINER 168 

VIII


FIGURE 48: UNILATERAL AND SINGLE JOINT PROTOTYPES: 168 

FIGURE 49: ASSISTIVE EXOSKELETONS: (FROM LEFT TO RIGHT) REWALK., BODY WEIGHT SUPPORT ASSIST, SUBAR 169 

FIGURE 50: POWER AUGMENTING EXOSKELETONS: (FROM LEFT TO RIGHT) BLEEX, SARCOS EXOSKELETON, 169 

FIGURE 51: HIGH‐LEVEL CONTROL STRATEGIES IN ROBOT‐ASSISTED GAIT REHABILITATION: 170 

FIGURE 52: BASELINE SAMPLE COLLECTION FOR LABORATORY ANALYSIS (BRUEMMER ET AL. 2002) 174 

FIGURE 53: MOBILE SECURITY ROBOTS 175 

FIGURE 54: AUTONOMOUS VS. MANUAL INVESTIGATION OF CONTAMINATION AREA INCLUDING SAMPLING 176 

FIGURE 55: (A) HARDWARE SETUP OF THE RECOROB RECONNAISSANCE ROBOTIC SYSTEM, 176 

IX


Table of Tables 

TABLE 1: WHO‐MONICA PROJECT 6 EU POPULATION. 21 

TABLE 2: PREVALENCE (PER 100 000) OF MS IN EUROPE BY AGE (BEST ESTIMATES) 23 

TABLE 3: INCIDENCE (PER 100 000/ YEAR) OF MS IN EUROPE 23 

TABLE 4: MAIN PLATFORM COMPARISON 153 

X


1 Abstract 

Deliverable D2.1 Requirements and Specification formalises the prioritised requirements for the CORBYS 

Cognitive Control Architectecture and demonstrator domains as exemplars of use cases to be evaluated within 

the CORBYS project. The focus of CORBYS is on robotic systems that have a symbiotic relationship with 

humans. Such robotic systems have to cope with highly dynamic environments as humans are demanding, 

curious and often act unpredictably. CORBYS will design and implement a cognitive robot control 

architecture that allows the integration of 1) high-level cognitive control modules, 2) a semantically-driven 

self-awareness module, and 3) a cognitive framework for anticipation of, and synergy with, human behaviour 

based on biologically-inspired information-theoretic principles. These modules, supported with an advanced 

multi-sensor system to facilitate dynamic environment perception, will endow the robotic systems with highlevel 

cognitive capabilities such as situation-awareness, and attention control. This will enable the adaptation 

of robot behaviour, to the user’s variable requirements, to be directed by cognitively adapted control 

parameters. CORBYS will provide a flexible and extensible architecture to benefit a wide range of 

applications; ranging from robotised vehicles and autonomous systems such as robots performing object 

manipulation tasks in an unstructured environment to systems where robots work in synergy with humans. 

The latter class of systems will be a special focus of CORBYS innovation as there exist important classes of 

critical applications where support for humans and robots sharing their cognitive capabilities is a particularly 

crucial requirement to be met. CORBYS control architecture will be validated within two challenging 

demonstrators: i) a novel mobile robot-assisted gait rehabilitation system CORBYS; ii) an existing 

autonomous robotic system. The CORBYS demonstrator to be developed during the project will be a selfaware 

system capable of learning and reasoning that enables it to optimally match the requirements of the user 

at different stages of rehabilitation in a wide range of gait disorders. 

2 Executive Summary and Report Scope 

This deliverable document (D2.1) reports the activities, effort and work performed under Work Package 2 

(WP2 Requirements and Specifications) of the CORBYS project. This document defines the end-user groups 

and end-user requirements, elicits requirements with defined priorities, and specifies in detail the list of 

requirements for all the sub-systems as well as the inter-dependencies. The document also reviews state-ofthe-art 

achievements in the Science and Technology areas relevant to the project. 

An introduction is given in Chapter 3, followed by the Requirements Engineering Analysis Base presented in 

Chapter 4. This includes sections on requirements engineering methodology UREIRF salient features, and 

knowledge elicitation from clinical partners regarding the first demonstrator, consisting of end-user 

demographics and gait biomechanics in normal and pathological walking. Requirements for the first 

demonstrator and the second demonstrator are also given in Chapter 4. Requirements elicitation 

methodologies employed, together with the involvement of stakeholders, establish a prioritised hierarchy of 

requirements to be fulfilled by the project during its lifetime; these are also reported in this chapter. This 

chapter concludes with a state-of-the-market relevant to CORBYS solutions. 

Chapter 5 focuses on the mechatronic control systems of CORBYS such as human sensing systems, robotic 

system motor control units, the mobile platform of the gait rehabilitation system, powered orthosis, actuation 

systems etc. Chapter 6 reports the human control system of CORBYS including non-invasive BCI detection 

of cognitive process for motor control and learning, cognitive control modules. Chapter 7 deals with the 

robohumatic systems i.e. graceful robot-human interactive cooperation systems. This includes self-aware 

realisation, situational response, user responsive learning and adaptation, anticipation etc. 

1


Chapter 8 and 9 detail system integration and functional testing for the CORBYS solutions including 

conformance testing at sub-systems and system level, formulation of appropriate evaluation methodologies, 

identification of relevant benchmarks, metrics and procedures. 

Chapters 10 – 17 report state-of-the-art in relevant topics such as sensors and perception, situation assessment, 

anticipation and initiation, cognitive robot control architectures, smart integrated actuators, non-invasive BCI, 

gait rehabilitation systems and hasardous area examining robots. Chapter 18 concludes the report. 

3 Introduction 

In this introductory chapter we motivate a framework for focusing on the relevant domain knowledge prerequisites 

of CORBYS requirements engineering process. Accordingly we set out some domain observations; 

thus identifying a set of important perspectives, referred to as facets, of the relevant domain knowledge that 

this section has to explicate and formalise to prepare the way for the subsequent requirements engineering 

phase. 

We conclude this chapter by setting out indicative content markers for each of the 9 facets thus identified. 

This will serve to characterise the expected structure of the relevant knowledge to be contributed by respective 

Partners for each facet as deemed relevant to the design and evaluation of the CORBYS Cognitive System. 

This analysis is aimed at concluding a rational structuring of each section of the deliverable document so as to 

avoid gaps that could otherwise emerge in the relevant knowledge map later when we hope to integrate the 

constituent parts to be contributed by each Partner per their specialist area of expertise. 

Deriving the analysis base for relevant knowledge acquisition 

For clarity, it should be stated that in this document we distinguish between the following four subsystems of 

the CORBYS architecture: 

i) The CORBYS Mechatronics i.e. the CORBYS Control Hardware Embodiment: by which we 

refer to that assembly of sensors, actuators, and control hardware such as servos, differential 

motors, and other parts of the embodiment of any integrated framework of mechanical, electronic 

and control components. 

ii) The CORBYS Cognitive System i.e. the CORBYS Intelligent Initiative-Taking Intelligent 

Controller: by which we refer to the intelligent sub-systems that enable the smart (i.e. the robotic) 

capability of an adaptively-assistive architecture such as CORBYS. 

In particular in usage-contexts whereby such an architecture is capable of seamless integration 

with the human body-area systems to deliver close real-time responsive man-machine 

cooperativity in support of a person’s well-being, life/work-style goals in a highly personalised 

manner, we classify such a system as displaying what we refer to as intimate-assistive or “robohumatic” 

capabilities. 

This means that although the core design architecture of CORBYS cognitive system should be 

generally applicable to various usage-contexts, the design of its situation-awareness framework 

that integrates directly with the target host platform and user environments will have to be 

mindful of the degree to which the application domain requires intimate man-machine assistivity. 

The more closely the CORBYS system has to integrate with human body-area systems in 

achieving its man-machine co-working , the more critical it would be to ensure that the design of 

its situation-awareness is optimally adaptive (including consideration of additional ethical 

2


compliance criteria including informed consent, ability to withdraw at any time, ethical approval 

etc.) whereas CORBYS cognitive co-working applications that typically do not involve direct 

machine-human-body-area systems integration should pose less exacting requirements re human– 

factors and related technology acceptance criteria. For example CORBYS Demonstrator I, the 

robot assisted human gait rehabilitation application domain will involve more “robo-humatic” 

user (therapist/patient) design requirements than the CORBYS Demonstrator II the hazardous 

area examining robotic system. 

A CORBYS User Interface: for the practitioner/end-user, who would be re-configuring CORBYS, 

(re) setting its performance goals and monitoring the outcomes of such performance and his/her 

own recommended scenario e.g. therapy. The CORBYS Assist interface is to provide an 

appropriate interface for Operators/Tester/Assessor/Configurer/ Expert/ Practitioner user/enduser/ 

any man-in-the-loop, who may or may not be the person engaged with the CORBYS system 

in routine co-working but who at any rate may need to have a simple interface through which to 

(re)configure and/or (re)set the CORBYS system for various operational modes to satisfy specific 

performance targets and to assess the system performance e.g. a Gait Therapist or Emergency 

Hazardous Area Examining Officer 

iii) CORBYS data logging system: to serve off-line (outer-loop) learning, pattern discovery and 

Case Based Reasoning (CBR) so that insights can be harvested by the system for off-line learning 

and evolutionary performance refinement; as well as this becoming a source of experiential 

knowledge that can be shared amongst the user community of practice for most beneficial sociotechnical 

deployment of the CORBYS architecture in a given domain e.g. in gait therapy 

management. 

iv) CORBYS Evaluation Environment: which must conform to appropriate Testability Framework 

Criteria thus providing a proving ground for safe, sense-ful, non-trivial, realistic, repeatable and 

scale-able validation and use-ability evaluation of all the above sub-systems. 

Accordingly for the purpose of our requirement engineering analysis we distinguish between what we refer to 

as the CORBYS Mechatronics or the embodiment of the target CORBYS system the specification and 

configuration of which will be expected to be largely application dependent so that it could integrate with 

legacy application-specific host environments 

and, 

the cognitive system which is that part of the CORBYS architecture which is to provide the intelligent adaptive 

capabilities e.g. as needed in graceful man-machine teamwork in a given application arena. This technical 

objective of man-machine cooperativity support which is essentially the goal of the CORBYS architecture 

requires the semantic integration of several interacting dynamic environments as listed below. 

By graceful cooperativity support we mean that the intervention steps taken by the CORBYS cognitive system 

should fluidly blend in with the human/co-worker activity flow to create a seamless confluence of mutual 

effort i.e. efficient and effective orchestration of teamwork towards the shared goal whilst ensuring that the 

limits of human’s/co-worker’s comfortable performance are not exceeded nor under-actualised as deemed 

appropriate. 

This deliverable has to focus on the parametrics of the environments involved i.e. the states and process flows 

of the entities in the domain environments with which CORBYS has to gracefully interact. As such this 

3


deliverable will have to formally set out all the relevant epistemology (the structure of knowledge) and 

teleology (the theories of sense-ful action and purpose) essentially at two levels of abstraction: i) Generic 

CORBYS Framework Architecture Requirements, ii) Domain-Specific Requirements for each Proof-of- 

Concept Demonstrator Domain; i.e. for the Gait Therapist Assistant, and, the Hazardous Area Examining 

Assistant applications. 

Thus the aim of this deliverable is to focus on a formal explication of the knowledge needed to help specify 

the requirements to be fulfilled by the CORBYS Architecture during the various modes of its operational life 

i.e. during: 

i. configuration 

ii. usage 

iii. goals (re)setting 

iv. maintenance, 

v. validation and 

vi. refinement 

Phases of the CORBYS system operation 

It is also important to clarify that we recognise at least two CORBYS user groups: 

i) the CORBYS Professional User or CORBYS Expert User who would prescribe the manner in 

which the CORBYS system should be (re)configured, and have its goals (re)set so as to provide a 

sense-ful and useful service with maximum benefit to the end-user, and, 

ii) the CORBYS End-User who would be the human engaged in some activity that is intended to be 

directly supported through timely and gracefully cooperative initiatives taken by the CORBYS 

system to help the human in achieving his goals. 

Thus the CORBYS Consortium Partners should make contributions for this deliverable, each according to 

their expertise-related responsibilities and the level of planned effort; to respond to the questions re: 

What the design, development and validation of a CORBYS system needs to know about: 

process-flows, states, spaces, observability, controllability, degrees of freedom, resources constraints, 

optimal timing, interfaces etc relevant to the intervention steps by a cognitive system architecture to support 

man-machine interactive-cooperativity for the integrated man-machine system as a whole to achieve a given 

target state (goal state) as may be set by the domain experts e.g. i) by a gait therapist, or, ii) by a hazardous 

area examination officer – who are the typical practitioner users in the two CORBYS Project Demonstrator 

domains. 

Accordingly the design of the CORBYS framework architecture has to be informed by the epistemology i.e. 

the structure of knowledge relevant to both the general principles of man-machine mixed initiative-taking 

system design, and, the ontology of the two application arenas chosen as the Demonstrator proving grounds. 

Such application domain ontology includes the semantic parametrics of the entities, states and processes, 

goals as well as the spatio-temporal and resource constraints appertaining to each of the two demonstrator 

domains 

From the above observations follows our action plan which is to formalise the relevant knowledge structured 

within 9 facets of the domain with a focus of our analysis base as outlined in the indicative contents to be 

agreed under each of the relevant 10 sections set out below. For each of the facets, by reference to a uniform 

4


analysis set (of entities, states, processes, constraints and requirements) we seek to examine and identify all 

the requirements relevant to the design and validation of CORBYS as a framework Cognitive Systems 

architecture and for its demonstrator domains. 

The 9 facets of our requirements analysis and elicitation are as follows: 

i) The Human System 

ii) The CORBYS Mechatronic System 

iii) The CORBYS Cognitive System 

iv) The CORBYS Learning (on/offline) Component 

v) The CORBYS Testability, Validation and Usability Evaluation Framework 

vi) The Operational Environment Parametrics 

vii) The Demonstrator Application I: Robot-assisted Gait Rehabilitation 

viii) The Demonstrator Application II: Robotic examination of hazardous/contaminated areas 

ix) Socio-technically acceptable CORBYS Solution System and its Business Model Sustainability 

4 Requirements Engineering Analysis Base 

4.1 Knowledge elicitation from clinical experts for Demonstrator I 

This section sets out the requirements of the first CORBYS demonstrator resulting from discussions among 

relevant partners of the Consortium including NRZ, IRSZR, OB, OBMS, VUB, UB, and UR. 

4.1.1 EndUser Demographics 

End-users of the CORBYS demonstrator I, gait rehabilitation system, are patients, who could have the 

following diseases and handicaps: 

� Lesions or diseases of the central nervous system such as traumatic brain injury (incidence of severe 

TBI in Germany 30.000), stroke (of 250.000 patients suffering from stroke yearly in Germany approx. 

30.000 could benefit from a walking aid system), encephalitis, hypoxic brain injury before or during 

birth Preterm birth (= infantile cerebral palsy, incidence in Germany 3000), or during later life (= due 

to cardiac disease or suffocation), degenerative brain diseases such as Parkinson´s disease (36.000), 

inflammatory brain diseases such as multiple sclerosis (56.000), or brain tumors. Lesions of the 

central nervous system lead to spastic paresis (=increased muscle tone) and the need for re-modelling 

by exerting repeated exercises and active-corrective. 

� Diseases and trauma of the spinal cord (5000). Lesions of the spinal cord also lead to a spastic paresis, 

but there is little chance for re-modelling 

� Diseases of the peripheral nervous system such as Guillain Barré syndrome, polyneuropathies, or 

neuromuscular diseases (disabling approx. 10.000). Lesions of the peripheral nervous system lead to a 

flaccid paresis (= reduced muscle tone) and the need for support and gaining strength 

� Orthopedic problems such as endoprothesis of knee (in Germany 160.000) or hip (in Germany 

210.000). Patients would benefit from a gait training system during the first 1-2 weeks after operation 

Depending on the site and size of the lesion, patients suffer from pareses of one to four limbs: hemi- (arm and 

leg of one side), para- (both legs), tetra- (all four limbs) or monoparesis (one limb). Additionally, other 

movement disorders can occur alone or in combination with paresis: ataxia (uncoordinated movements) or 

5


extrapyramidal gait disorders with involuntary movements (e.g. tremor, ballistic or other not controllable 

movements. 

However, it is not wise to define groups of patients, neither by etiological diagnosis nor by gait patterns, since 

gait patterns are highly individual. The reasons for this are: 

� the aetiology of a disease (e.g. stroke, traumatic brain injury, etc) may lead to very different symptoms 

(e.g. hemiparesis, spasticity, etc.) 

� the symptoms can be more or less severe, leading to very different requirements. 

� Additionally, different symptoms can appear simultaneously and influence each other, e.g. a certain 

degree of spasticity in case of severe weakness is necessary to stand and walk. 

� And on top, cognitive functions and capacities, pain, anxiety etc. are often also involved and may 

change the gait pattern again. 

4.1.2 Gait biomechanics in normal and pathological walking 

4.1.2.1 Normal walking 

Figure 1 displays positions of human trunk, pelvis and lower extremities in seven instances of gait cycle 

(between two consecutive contacts of the same leg). In general human walking may be decomposed into 

sagittal, frontal and transversal movement, with sagittal and frontal movement being the main contributor to 

dynamic stability as well as to forward propulsion and forward progression. When observing human walking 

we notice: 

� alternating phases of flexion and extension in ankle, knee and hip joint in sagittal plane (red arrows) 

� alternating phases of pelvic internal and external rotation (red box) 

� alternating phases of hip abduction and adduction combined with ankle, knee and hip flexion/extension 

during WEIGHT TRANSFER and FOOT CLEARANCE in frontal plane (red arrows) - this also 

implies the necessity for pelvic sideways movement (red arrows) 

� alternating phases of pelvic up and down movement/tilt in frontal plane (red box) 

� ankle varus/valgus in frontal plane (blue arrows) 

6


Figure 1: Normal walking in sagittal plane (top) and in frontal plane (bottom) 

in seven instances of gait cycle (GC): initial contact (IC), weight acceptance (WA), mid stance (MSt), terminal stance (TSt), 

initial swing (ISw), mid swing (MSw) and terminal swing (TSw). Red boxes indicate normal pelvic rotation and tilt whereas 

arrows indicate normal movements 

Only a well coordinated and synchronised movement in all joints can assure dynamic stability, reliable foot 

contact/positioning, proper weight transfer, propulsion and sufficient foot clearance. Therefore, normal 

bipedal locomotion as imposed by CORBYS gait rehabilitation system must satisfy the following four 

requirements: 

� antigravity support of the body (exoskeleton) 

� stepping movements 

o reliable foot contact/positioning – flexion/extension in ankle, knee and hip (exoskeleton) 

o WEIGHT TRANSFER and FOOT CLEARANCE – combined and synchronised pelvic rotation/till 

and ankle, knee and hip flexion/extension as well as hip abduction/adduction and internal/external 

rotation (exoskeleton) 

o forward progression – ankle, knee and hip flexion/extension (exoskeleton) 

� adequate degree of dynamic equilibrium (moving platform) 

� propulsion (exoskeleton) 

All four requirements need to be coordinated simultaneously and continuously. A viable approach would be 

to first have the CORBYS system in the »follow me« mode where the powered orthosis and moving platform 

would minimise the interface forces between the walking subject and the CORBYS system. Alternatively, 

CORBYS system could impose a normal walking pattern and the needed moments in the joints could be 

estimated from the current/torque supplied by exoskeleton motors. 

4.1.2.2 Pathological walking 

Pathological walking may be caused by a long list of diseases, however, the abnormalities they impose on the 

7


biomechanics of walking fall into four functional categories: 

� deformity (muscle and joint contractures on allowing for a sufficient passive mobility) 

� muscle weakness (insufficient muscle strength) 

� impaired control (sensory loss, spasticity: selective control is impaired, primitive locomotor patterns 

emerge: mass flexion and mass extension in all three joints, muscles change phasing, proprioception 

is impaired) 

� pain 

The CORBYS gait rehabilitation system should be concerned only with the impaired control and muscle 

weakness. The other impairment categories should be treated in other ways (physical therapy, casting and 

medical treatment). The general approach in walking rehabilitation should be either to train a “normal” 

walking pattern, or, if muscle weakness or muscle contracture will not allow a normal pattern, a 

compensatory, “optimised” movement patterns is permitted (or adequate orthotic aids are given) in order to 

achieve functional walking. 

The following section addresses the four most frequently occurring gait deviations due to impaired control, 

describes which and how principal walking mechanisms are affected and demonstrates how CORBYS system 

should behave to restore normal walking pattern. 

4.1.2.2.1 Toe walking 

Figure 2: Toe walking in sagittal plane 

Principal characteristics of toe walking are: 

� initial contact with forefoot (metatarsals joints) 

� pronounced plantar flexion / dorsiflexion deficit in ankle throughout the gait cycle 

� the heel touches the ground in mid stance or may even remain above the ground throughout the stance 

phase; likewise, in terminal stance a premature heel rise is common 

� increased knee and hip flexion throughout the gait cycle 

They impose: 

� insecure initial contact 

� propulsion is achieved predominantly by rapid hip extension instead of forceful extension in all joint 

CORBYS system should restore: 

� adequate ankle dorsiflexion prior to contact for proper heel contact and weight transfer (this may 

normalise the movement in other joints as well) 

� forceful extension in ankle, knee and hip during push off to assure normal propulsion 

8


� adequate ankle dorsiflexion and knee flexion in swing phase for normal foot clearance 

4.1.2.2.2 Crouch gait 

Figure 3: Crouch gait in sagittal plane 

Principal characteristics of crouch walking are: 

� plantigrade locomotion (podials and metatarsals flat on the ground) 

� pronounced knee flexion / knee extension deficit throughout the gait cycle 

� pronounced ankle dorsiflexion throughout the gait cycle 

� increased hip flexion 

They impose: 

� propulsion is achieved predominantly by rapid hip extension instead of forceful extension in all joint 

� low C of G due to pronounced knee flexion in stance phase 


� normal ankle plantar flexion and knee extension prior to contact for proper heel contact and weight 

transfer 

� normal knee extension to raise the C of G (combined with pelvic vertical translational DOF) 


4.1.2.2.3 Stiff knee 

9


Figure 4: Stiff knee gait in sagittal plane (top) and in frontal plane (bottom) 

Principal characteristics of stiff knee walking are: 

� inadequate knee flexion throughout the gait cycle 

� excessive dorsiflexion 

� inadequate knee flexion stimulates excessive contralateral trunk tilt and pelvic vertical movement to 

assure sufficient foot clearance 

They impose: 

� insecure and unstable weight transfer due to constant knee extension 

� propulsion is achieved predominantly by rapid hip extension instead of forceful extension in all joints 

� excessive pelvic vertical movement in swing phase compensates the lacking knee flexion to assure 

sufficient foot clearance 


� normal knee flexion to allow secure and stable weight transfer in initial contact and midstance 


� normal knee flexion to assure sufficient foot clearance (excessive pelvic vertical movement will no 

longer be necessary) 

4.1.2.2.4 Stiff knee with circumduction 

10


Figure 5: Stiff knee gait with circumduction in sagittal plane (top) and in frontal plane (bottom) 

Principal characteristics of stiff knee walking with circumduction are: 

� inadequate knee flexion throughout the gait cycle 

� excessive dorsiflexion 

� inadequate knee flexion stimulates contralateral trunk tilt and pelvic vertical movement to assure 

sufficient foot clearance 

They impose: 

� insecure and unstable weight transfer due to constant knee extension 

� propulsion is achieved predominantly by rapid hip extension instead of forceful extension in all joints 

� combined excessive pelvic vertical movement and hip adduction in swing phase compensates the 

lacking knee flexion to assure sufficient foot clearance 


� normal knee flexion to allow secure and stable weight transfer in initial contact and midstance 


� normal knee flexion to assure sufficient foot clearance (excessive pelvic vertical movement and hip 

adduction will no longer be necessary) 

4.2 Environment of the demonstrator I: gait rehabilitation userrobot 

interaction 

CORBYS demonstrator should be an institution-based gait locomotion rehabilitation training instrument. It 

should be an intelligent gait rehabilitation device; a mobile gait trainer. Patients could be treated both as 

inpatients and outpatients. The system could be integrated in the treatment workflow of a rehabilitation 

department which is illustrated in Figure 6. 

The gait rehabilitation robotic system could be integrated in this workflow in the following way: 

- The patient´s gait is analysed on admission on a treadmill with sensors measuring kinematics, forces 

and resistance in hip, knee and ankle joint as well as force on ground via foot plates parts. The CORBYS 

11


professional user (therapist) is optimizing the patient´s gait with his/her manual techniques, the system is 

“learning” the optimal gait applicable for this individual patient (=”teaching mode”). The data relevant 

for the support given by the robotic system are transferred to the system. The sensors used as a diagnostic 

tool on the treadmill will not be the same as in the robotic system. 

Clinical, informal, formal and apparative diagnostic procedures and 

Specific and goal orientated 

treatment 

Reassessment of achieved goals 

setting new goals 

Figure 6: usual workflow in a rehabilitation unit 

- The patient is performing overground walking with the system on his own, but under supervision of a 

therapist. The system is optimising his gait and measuring the efforts, energy and result. This increases 

the treatment time without therapist and increases the time for reciprocal repetitive training 

- The therapist is evaluating the parameters measured by the system and reevaluates the gait on the 

treadmill, again optimising the gait and by this changing the walking parameters of the system. 

Alternatively the system itself corrects the gait with its cognitive structure by approximating the patients 

gait to normal gait pattern or to the therapist assisted, learned gait pattern“. 

As a result, the gait rehabilitation robot system could allow more independence from the need to have 

therapist supervision at all time. The therapist may supervise more than one patient at the same time more 

than one robot system. The patients will have more repeatative training time, which leads to a more affective 

rehabilitation outcome. However, the role of therapist cannot completely be replaced by the technology. 

The gait rehabilitation robot system should interact with the patient at following different levels: 

Level 1: co-operation of the powered orthosis and mobile platform with the user on the level of basic 

overground walking: walking along the straight line at a constant speed. This level would cover the “gentle” 

alteration of the existing walking patterns: symmetry, weight-bearing capabilities, “normalisation” of 

kinematic and kinetic patterns in all three joints of lower extremity in all three planes of motion, balance 

control (COP vs. COM control). This level is not concerned with “cognitive” issues as this should be 

12 

Clinical, 

(Re)Integration into patient´s 

environment


automated movement. This level, however, should allow incorporation of “targeted walking” input that can 

be determined by a CORBYS professional user. For example, first the existing patterns like toe walking, 

crouch, stiff-knee, knee hyperextension, Trendelenburg etc. would be identified and appropriate changes in 

the desired kinematic and kinetic patterns would be made. The iterative nature of walking should be exploited 

in the design of a suitable control of powered orthosis and mobile platform. 

An important aspect of Level 1 operating mode would be training of isolated components of skills required in 

walking such as proper weight transfer training (starting from appropriate placement of the leg in swing onto 

the ground, followed by weight transfer during double stance and concluded by adequate control of trunk in 

the frontal plane during the single stance). 

Level 2: starting/stopping the walking, changing the speed of walking, turning during walking – change of 

walking direction. This level would be heavily concerned with cognitive interaction between the man and 

machine. A very motivating treatment modality would be avoiding the obstacles on the route that would 

represent a purposeful journey for a patient (going from the therapy session to their room for example). 

Level 3: the pathologic gait pattern of the patient is corrected by the gait rehabilitation robot system itself in 

an autonomous mode by comparing the measured joint angles, torques, velocities etc with a normal gait 

pattern or the CORBYS professional user assisted/corrected gait pattern and correcting false movements of 

the patient. 

4.3 Requirements for demonstrator I 

REQUIREMENTS FOR THE FIRST DEMONSTRATOR 

MANDATORY DESIRABLE, OPTIONAL 

1. Inclusion criteria to use the CORBYS gait rehabilitation system 

All symptoms leading to disturbed gait: 

- weakness of legs 

- increased tonus (spasticity) of legs 

- lack of coordination of legs 

- involuntary movement of legs 

- sensoric disorders of legs: disturbed afferences 

- Contractures < 30° in ankle, knee and hip 

- weakness of legs and trunk 

- lack of coordination of legs and trunk 

- involuntary movement of and trunk 

- in this case, the system must also support the trunk, 

which is unstable 

2. Exclusion criteria for use of the CORBYS gait rehabilitation system 

- blindness 

- cognitive impairments to use the system 

- no arm or hand function 

- massive, suddenly outbreaking dynamic spasticity, ballistic 

movement disorder 

- grand mal epilepsy with frequent seizures 

3. Informal diagnostic measurements by the robot system 

- degree of passive movement in hip, knee and ankle joint - analysis of balance problems 

(analysis for fixed contractures) 

- measurement of fatigue / tiredness: Measure cycle 

- resistance to a passive movement (analysis for dynamic time 

contractures) 

Co-ordination will find out mistakes and 

abnormalities and thus allow to identify tiredness as 

Within a gait cycle the following parameters should betired 

patient will increase mistakes and this can be 

measured 

detected by co-ordination feedback 

- torque (= voluntary muscle strength) in hip, knees and ankle 

joints both sides 

13


- velocity of movements in hip (sagittal) and knee both sides 

- changes of angles of joints over time within the gait cycle 

(=curves of gait cycle: hip sagittal, knee and ankle joints) 

- orchestration of coordinated movements in all joints as 

measured by the changes of joint angles over time 

- “foot landing”: first contact of which part of the foot to the 

ground (should be the heal) 

These measurements must be compared with normal gait 

pattern 

Evaluation measurements: 

- how much torque / energy was supplied by system both to 

support the weak muscle and to hamper spasticity 

- patient’s perspective, expectations, experience and opinion 

regarding the demonstrator should be documented. 

4. Standardised diagnostic measurements 

- step length 

- timed walk tests: steps per time, 6 min test. 10 m test 

5. General requirements to system design 

- adaptation to patient´s height: 160-200 cm 

- adaptation to patient´s weight: 50 kg-100 kg 

- infinitely variable body weight release to decrease ground 

reaction force provided through the frame of the machine 

rather than overhead support (harness). In addition, the 

powered orthosis can provide body weight support as well. 

The ability of mobile platform to hold the patient could 

contribute to the design of orthosis. 

- max. width 100 cm 

- Vmax: 3 km/h 

- walking curves 

- degrees of freedom (max angles): 

- Pelvis: (up, down) translational actuated; (forward, 

backward) translational actuated; (left, right) translational 

actuated; frontal, rotational passive (actuated indirectly through 

hip motion).. As the exoskeleton will be attached to mobile 

platform, active DoF on the mobile platform induce pelvis 

movements. Design considerations in pelvis are considerably 

challenging. Since the orthosis is attached to the moving 

platform the system is redundant, which implies that more 

than one design solution is possible that would meet the 

necessary movement requirements. The above describes one 

design possibility. 

- Hip: 3: sagittal (40 °) and frontal (10°) actuated, (desired) 

horizontal (10°) passive 

- Knee: 1: sagittal (70°) actuated 

- Ankle: 2: sagittal (90°)actuated, horizontal passive 

- sensor: 

- measurement of angles in all actuated joints 

- measurement of torques applied by the system in all actuated 

joints 

- measurement of velocity of movement in all actuated joints 

- measurement of the patient´s resistance against a movement 

generated by the system 

active or passive compliance to deal with spasticity: 

- system must slowdown movement in case of useful, but 

increased movement and in case of spasticity 

14 

- adaptation to patients weight: 40 kg-125 kg 

- actuator in both ankle joints 1 

- safety: obstacle identification including stairs up 

and down, collision avoidance 

- system should identify patient´s intention via EEG, 

speech, eye tracking or weight shift, etc for 

initiation of gait, steering, turning, stopping etc. 

- system should evaluate, if the identified intention is 

meaningful 

- system utilisation without continuous supervision 

of a therapist 

- turning on the spot 

- autonomous mode: gait rehabilitation robot 

systemoptimizeses the pathologic gait pattern by 

itself, withoud the impact of the therapist, by 

comparing the patients gait pattern with a 

normalized gait parametes including angle, torque, 

and velocity in all actuated DoF in hip, knee and 

ankle. 

- modular configuration of the system: frame and 

exoskeleton can be separated, exoskeleton can be 

split: the less severely affected a patient is, the 

fewer modules will be needed: frame, actuators, 

sensors 

- handling: One therapist must setup system within 

10 minutes 

- handling: patient must set up system by himself 

- utilization in everyday life, at home


- system must not give way in case of little spasticity 

(isometric reaction), but give way in case of massive 

spasticity (isotonic reaction) 

- degree of giving way in case of massive spasticity (in % of 

normal strength) must be adjustable individually and joint by 

joint by the therapist 

- safety: patient must not fall with the system or sink down 

within the system 

- handling: System must be optimised easily by the therapist in 

case of pressure ulcera 

6. Treatment goals 

- indoor walking on plane and even surfaces 

- outdoor walking 

- economic walking 

7. Treatment methods 

- system learns optimal gait automatically by comparing the 

analysed gait pattern of the patient with the normal gait 

pattern via feedback loops 

- system learns optimal gait for the individual patient during 

training session with therapist on the treadmill (=”teach-in”) 

- for both legs independently 

- system repeats the therapist´s treatment manouvres 

15 

- motivation of the patient is important and hence the 

use of reward-based positive psychology is 

beneficial. In the CORBYS demonstrator, 

envisaged to be a moving platform, visual 

motivational feedback is foreseen to be distracting 

and hence not advisable. However, audio 

motivational feedback may be used. CORBYS 

professional (physiotherapists) may encourage the 

patients by using pep talk and raising the volume of 

their voice (verbal encouragement). 

4.3.1 Cognitive capability 

CORBYS demonstrator should not just be a rehabilitation system but also a system with cognition. It should 

learn the areas in the gait cycle which need improvement and present this to the CORBYS professional user 

(therapist). Two main aspects in which CORBYS cognitive capability can contribute/innovate: 

• Cyclical nature of gait (Propelling force - push off, pull off - and efficient weight transfer) 

• movement planing (inituation, stopping, turning) 

In the former cognitive aspect, CORBYS should support all three planes i.e. sagittal, frontal and transversal. It 

should learn over many cycles of gait what the problem is, and subsequently introduce corrective measures 

not intra-cycle but inter- cycle, or inter-session. It can also support maintenance of balance (dynamic 

walking). In contrast to state-of-the-art systems where therapists adjust gait parameters such as gait velocity, 

CORBYS can control this adjustment. For example, if the patient is tired, adjust the speed to suit the 

performance. Another relevant example can be of spasticity – if the patient is experiencing spasticity, unlike 

state-of-the-art systems such as Lokomat which can be programmed to switch off the motors upon reaching a 

threshold, the CORBYS demonstrator should be able due to its cognitive capabilities to adjust parameters 

automatically to control or dampen spasticity using mechanical actuation. In general, the CORBYS gait 

rehabilitation system should be able to identify the support needed by the patient, provide it at that level and 

then gradually lessen the support. Dampening of the spasticity should also be supported using motorised 

stimulation (not Function Electrical Stimulation). 

In the second cognitive aspect, CORBYS should assist the patient in movement planning during over ground 

walking with the CORBYS demonstrator I. CORBYS can assist the patient and provide them with a 

purposeful goal to move around. It can provide help the patient such as planning the movement, execution of


the movement, making a turn, planning the route etc. 

Regarding the "cognitive" aspects of the project one can actually think of having cognitive control, situation 

awareness etc. at both levels of control: lower-level that is concerned with automated movement of lower 

extremities (here "cognitive" control of weight shifting manoeuvre could be very relevant) and higher-level 

that should be concerned with purpose of movement - going to designated place while making all necessary 

turns and obstacle avoidance. Also, in this way both categories of patients, more impared and less impared, 

may be considered. With the more impaired, the emphasis will be on "lower-level" aspects: proper weight 

bearing and weight shifting manoeuvres as well as appropriate automated movement of legs while with less 

impaired patients the movement planning and trajectory execution would be a challenge. 

4.3.2 Ethical compliance 

� Informed consent will be obtained from all end-users (patients) 

� The possibility for the patient to quit experiments at any time without having any disadvantage for 

their further treatment will be ensured 

� Votes of the local ethical committees (in Bremen, Germany and Ljubljana, Slovenia) will be obtained 

4.4 Requirements for demonstrator II 

The robotic system which will be used as the second demonstrator has been developed by UB within the 

German national project RecoRob (Hericks et al., 2011). It consists of a 7DoF (degrees of freedom) robot arm 

which is mounted on the mobile platform. The robot is equipped with sensors for environment perception as 

well as with sensors for platform navigation and robot arm control. In this project two experimental setups 

will be designed in which the robot works in a team with a human to investigate contaminated/hasardous 

environments. Both experimental scenarios of the second demonstrator will be designed to test different 

functionalities of the CORBYS cognitive control architecture, with the emphasis on alternating the humanrobot 

lead taking in exploratory scenarios. In both scenarious different testings will be carried out, partial 

testing of some aspects of cognitive architecture as well as full testing of complete functionalities. An example 

of partial testing is demonstration of goal selfgeneration in autonomous object manipulation. In full testing, 

the robot is considered as a partner of an external agent (e.g. firefighter) in a cooperative activity. 

In the first scenario named augmented teleoperation, the robot is teleoperated by the human so that it is sent 

into the contaminated area to collect samples used to determine contamination levels. The robot’s primary 

task is to follow the operator's instructions for navigation and manipulation. However, if communication fails, 

the robot has to be able to take the initiative in completing the task, in spite of the loss of human command. 

Also, if an unexpected obstacle is sensed, the robot has to be capable of reasoning to “veto” dangerous human 

commands to avoid running into obstacles. In this case, in order to complete the task, the robot has to takeover 

the goal-setting initiative and to self-generate a goal. In the second experimental scenario, the robot is a 

co-worker in investigation of hasardous environment. The robot has the role of a transportation robot as it 

helps the human in carrying the containers with the collected samples. At the beginning of the investigation 

mission, the robot follows the human partner keeping the constant distance between them. Based on sensory 

information (e.g. vision sensors) the robot analyses the human’s behaviour and deduces the human's goal. If 

there is an unexpected change in human behaviour such as, for example, a change in direction of movement so 

that human is approaching the robot, the robot has to change its behaviour so to stop and to allow the human 

to place the containers with the samples collected onto the robot. 

In order to enable the evaluation of CORBYS cognitive control architecture on the second demonstrator the 

16


following requirements have to be fullfiled. 

1. Integration (wrapping) of modules 

of existing control architecture (e.g 

robot arm control module) of the 

second demonstrator into CORBYS 

control architecture. 

2. Analysis of usability of existing 

sensors for CORBYS demonstration 

REQUIREMENTS FOR THE SECOND DEMONSTRATOR 

MANDATORY 

High level CORBYS cognitive control modules to be evaluated on existing 

robotic system for examining hasardous environments. A detailed strategy 

will be defined later. It depends on the requirements of CORBYS control 

(software) architecture. 

Existing sensors (e.g. vision sensors) are to be analysed in the CORBYS 

context. It is to be tested whether existing sensors can provide necessary 

information for cognitive modules. Should additional sensors be integrated? 

Should the functionalities of the vision module be extended? 

3. Situation Awareness Situation Awareness input to semi-autonomous (autonomous) robot control 

4. Identification and anticipation of 

human co-worker purposeful 

behaviour 

Cognitive input to semi-autonomous (autonomous) robot control for 

overtaking initiative when needed 

4.5 Requirements Engineering Methodology (UIREF) Salient Features 

Requirements Elicitation and Prioritisation 

UI-REF (Badii 2008) incorporates an integrated set of models, techniques and tools to allow the system 

designers to negotiate articulate and prioritise the set of use-contexts and their related requirements and 

evaluation context criteria and priorities in a way that overcomes the problems of needs articulation and 

memory bias on the part of the users. To arrive at a prioritised set of requirements, it will be necessary at least 

to: 

i) agree and set out the list of domain prototypical entities or actors as stakeholders; 

ii) define the characteristics of the prototypical entities involved in the typical usage arena envisaged for 

the target system (e.g. actors, devices, system of systems); 

iii) establish the generalisation ontology of usage-contexts each related to their distinct features as needed 

by the relevant user (sub)groups in their target prototypical scenarios; 

iv) define the key differentiators of usage contexts (context switches) and the prototypical actors’ needs 

hierarchies in each of the identified prototypical target context-scenarios; 

v) define the prototypical workflows and state diagrams, thus establishing the domain user/practitioner’s 

(sub)-goal and (sub)-task hierarchies; 

vi) deduce the user’s needs priorities in terms of ICT-enabled features to facilitate user’s task fulfilment in 

each situated context-scenario of the application domain as identified and demarcated (situated-usageclass) 

under respectively iii) and iv) above. 

User-intimate approaches often yield a vast amount of raw data and need appropriate abstraction and context 

layering, to reflect the natural partitions within the domain. This is so as to arrive at actionable insight as to 

the most deeply-valued needs for most users belonging to each of the target usage-context types within the 

17


spectrum of usage-context classes to be addressed by the target system. 

Specifically in identifying all domain objects and actors and delineating the boundaries of roles and 

responsibility spaces, rights/privileges for each actor and/or device in the domain, we are essentially 

negotiating a phenomenological analysis of the domain with the users. By further specifying domain 

knowledge, taxonomies and a tentative ontology for the domain and negotiating this with various user 

communities, it is possible to conclude an appropriately partitioned ontology of the world of the users for 

whom the system is intended. Such generalisation ontology serves as a values expression language of the 

most-deeply-valued-needs for various usage-contexts. 

Building on this increasingly deeper understanding paves the way for formalising the domain knowledge 

structure including tacit knowledge, causal, process-related and structural knowledge thus adequately 

specifying the domain knowledge structure. This comprises a most important element of the experientiallyderived 

tactical/ strategic problem solving knowledge. The domain knowledge is clearly the provenance of 

the various user-classes (as distinguished by their usage-contexts) who are to use the system in pursuit of their 

everyday practice. It is these communities of practice who are expected to make available to the requirements 

engineers and other stakeholders their domain knowledge so as to promote deeper understanding of their 

domain requirements including end-to-end interoperability and meta-operability across all implicated entities 

including legacy norms, processes, policies, etc. In eliciting the domain knowledge we can establish the goal 

structures knowledge for the application domain. 

UI-REF advocates that the requirements are classified into the following descending-order priority categories 

for implementation; these range from mandatory, as the highest priority class, through to desirable, as the 

medium priority class, to optional, the lowest priority class. Prioritisation of requirements is deduced from 

careful analysis of user-stated priorities which can be aided also by a consideration of Purpose-Hurry- 

Frequency Criteria set re degrees of intimacy and immediacy of the required services and patterns of 

interactive online support required to facilitate the user’s life/work-style patterns. 

The above priority levels are formally elaborated as a spectrum of target platform core affordances plus 

additional ones as follows: 

Mandatory – These are those core design features which are perceived by the majority of a user group as 

offering the most needed added-value(s) and that can be accommodated by the target system. This category is 

expected to include the selected functional and non-functional requirements (look-and-feel interfaces) that are 

the common core to all the usage-contexts within the target usage spectrum; including features supporting 

scalability, modularity and open design so as to enable the incremental evolution of the system to offer further 

features to satisfy future requirements and customisation as appropriate. 

Desirable – These are those features that are desirable, but not highest priority design features as candidates to 

be accommodated as far as possible, within the resources and technological constraints appertaining to the 

lifecycle of the project. 

Optional – These are those features said by some users to be of the lowest priority and/or are anyway highly 

contextualised to particular (sub)-sectors of the user group and as such falling into the less common, and/or 

possibly more controversial and conflictual category. 

Once the raw user-stated requirements are aggregated through all elicitation instruments, channels and 

modalities, they have to be transcribed, tabulated and cross-checked to prune duplications and delete clearly 

out-of-scope requirements. UI-REF promotes a negotiation-based resolution of requirements into the three 

18


categories to reflect the priorities of the majority of users. 

Next additional checks have to be done to flag up, for negotiation with the stakeholders, the possible 

deletions, demotions, promotions and new additions of specific requirements to be consensually resolved into 

the set of mandatory, desirable and optional requirements for a first prototype. The need for the following 

refinement steps arises as a natural consequence of the fact that the users in stating their requirements can not 

be expected to be either exhaustive or factor-in the technology, market and practice constraints (SoA, SoM, 

SoP); the trends of which they are not necessarily expected to be fully aware. Further, users are expected to 

articulate their own perceived requirements which may or may not be complete and may be incompatible with 

other users’ requirements or project resources or in conflict with the technological and/or market imperatives 

and trends. 

Figure 7: Prioritisation process in UI-REF 

Accordingly it will next be necessary to “factor-in” the influence of the push-pull forces and their dynamics 

over the near to medium term to ensure that the target system to be delivered will represent the highest rate-ofreturn 

on investment for all stakeholders in order to have the highest chance of take-up and widest diffusion, 

usability and technology-policy-process interoperability and convergence potential given the current and 

emergent technological and practice environment that it will have to integrate with i.e. it will be as scalable 

and sustainable as possible. Such pull and push factors representing constraints and affordances invoked as 

requirements filters and augmenters can be best understood by performing respectively a SoA, SoP and SoM 

analysis, where the State-of-X is represented by the latest update on the state of current modus operandi, gaps, 

and, available enabling and emergent innovations from the viewpoint of X. 

Prioritisation in UI-REF is performed at several levels including stakeholder, intimate or non-intimate usagecontext-type, 

specific usage-contexts within each type, usage-context implementation sequencing, usability 

sensitive evaluation of system functionalities whose usability assessment is ranked and weighted so as to 

reflect their priorities in the order of the most-deeply-valued needs of the user as prioritised per the UI-REF 

requirements engineering process. Such prioritisation processes will use a variety of situated techniques as 

appropriate to best suit particular users and their usage-contexts for example Nirvana, Ablation and Noah’s 

Arc, virtual user, nested videos, Frequency-Purpose-Hurry (FPH)-based analysis of user specified needs and 

thus the resulting UI-REF Effects-Side-Effects-Affects multi-dimensional impact prediction and evaluation 

matrices. 

19


Total requirements: 345 (Mandatory: 309, Desirable: 22, Optional: 14) 

In total, 345 requirements have been gathered, out of which 309 are classed as mandatory, 22 as desirable, and 

14 as optional requirements. All the requirements are detailed under chapters 5 – 9 i.e. mechatronic control 

systems, human control system, Robohumatic systems, system integration and functional testing for the 

CORBYS solutions, and evaluation. 

Using technology filters, specific usage-context filters for demonstrator I and II, 44 main requirements for the 

CORBYS project have been selected falling under three main categories: Cognitive systems, Demonstrator I 

specific, Demonstrator II specific. 

4.5.1 Cognitive systems 

1. HSS8: Sensor output related to identification of psycho-physiological states 

2. HSS9: Sensor output related to identification of intentional states 

3. SAWR16: Current context of SAWBB 

4. RRS6: Identification of reflexive capability to be enabled by the FPGA sub-system 

5. RRS7: Identification of ‘Navigation’ related reflexive behaviour enabled by the FPGA sub-system 

6. RRS8: Identification of ‘Obstacle avoidance’ related reflexive behaviour enabled by the FPGA subsystem 

7. RRS9: Identification of ‘Safety’ related reflexive behaviour enabled by the FPGA sub-system 

8. RSM8: BCI Architecture 

9. FAI12: User interface 

10. FAI13: User interface for patients 

11. FAI14: User interface for therapist 

12. FAI15: User interface for developers 

13. DMI2: Motor intention study 

14. DMI3: Algorithm for detection of motor intention 

15. ADD1: SOIAA Module 

16. FAI8: Self-motivated gait and goal Generation sub-system 

17. SOIAA1: Sensorimotor data from the robot platforms 

18. SOIAA6: Models and algorithms for the identification and anticipation of human purposeful 

behaviour 

19. FAI7: Robot cognitive control architecture 

4.5.2 First Demonstrator: Mobile Robotassisted Gait Rehabilitation System 

20. HSS6: Sensor output related to physical effort assessments 

21. HSS7: Sensor output related to gait parameter assessments 

22. HSS15: Online access to past rehabilitation sessions 

23. RSM11: Modelling of the walking in the rehabilitation system 

24. MPD4: Body weight support. 

25. ACT4: Actuator models for robotic system modelling and integration of motor control units 

26. ACT5: The actuator system of the orthosis 

27. ACT6: Actuation of the mobile platform 

28. LLC1: Mechanical design of the powered orthosis and mobile platform 

29. LLC2: Electro-mechanical components 

30. LLC5: Low level control loop 

31. REAL2: Integration of the actuators to the orthosis 

32. REAL3: Integration of the actuators for the mobile manipulator 

33. REAL4: Sensor integration into the orthosis 

34. HSS21: Intended duration of continuous usage of physiological sensors 

20


35. HSS34: Physiological sensor biocompatibility issues 

36. HSS36: Physiological sensor hygienic issues 

37. MPD10: Patient stability 

4.5.3 Second Demonstrator: Robotic Systems for examining Hazardous Environments 

38. EASD1: Detailed specification of the evaluation scenarios. 

39. EASD2: Integration (wrapping) of modules of existing control architecture (e.g. robot arm control 

module) of the second demonstrator into CORBYS control architecture. 

40. EASD3: Analysis of usability of existing sensors for CORBYS demonstration 

41. EASD4: Situation Awareness 

42. EASD5: Identification and anticipation of human co-worker purposeful behaviour 

43. EASD6: Extension of existing sensor modules (e.g. vision) 

44. EASD7: Evaluation of CORBYS cognitive control architecture on the existing robotic system 

4.6 StateoftheMarket 

This section presents a state-of-the-market on systems relevant to CORBYS. In the first subsection, focus is 

given to the market for gait rehabilitation systems, and in the second subsection, focus is shifted to the market 

for autonomous robotic systems for assistance of humans in examining hazardous areas. 

4.6.1 First Demonstrator: Mobile Robotassisted Gait Rehabilitation System 

The following subsections reports statistics on the neurological conditions identified by the clinical partners in 

the CORBYS Consortium as most relevant in terms of the application of CORBYS technologies to be 

developed, to gait rehabilitation for patients suffering from these conditions. 

4.6.1.1 Stroke 

Table 1 below shows data from the WHO-MONICA Project (WHO MONICA Project cited in Major and 

Chronic Diseases Report 2007) reported for the age range 35-64 years as to mean stroke event rates derived 

from the last 3 years of surveillance. Annual change in stroke events and 28-days case fatality are also 

reported (Major and Chronic Diseases Report 2007). 

Table 1: WHO-MONICA Project 6 EU population. 

Age-standardised average attack rate per stroke events (fatal and non fatal) per 100,000: mean of the last 3 years of the 10year 

surveillance in men and women ages 35-64 years; 28-day case fatality; average annual trend in 10 years of stroke events. 

21


4.6.1.2 Multiple Sclerosis 

The cost per MS case in Europe ranges from €10 000 to €54 000, with a mean of €31 000 (Sobocki P 2007 

cited in Major and Chronic Diseases Report 2007). The distribution of the estimated total cost of MS in 

Europe in 2005 by resource use components is reported in Figure 8. 

Figure 8: Distribution of total cost of MS in Europe (year 2005) by resource use components 

22


MS prevalence by gender, age and European Country (where data were available) and MS total annual 

incidence rates by European Country are summarised in Table 2 and Table 3 (Major and Chronic Diseases 

Report 2007). 

Table 2: Prevalence (per 100 000) of MS in Europe by age (best estimates) 

Table 3: Incidence (per 100 000/ year) of MS in Europe 

23


4.6.1.3 Injuries 

4.6.1.3.1 Fatal injuries 

The injury pyramid for EU shows on the top a total of 250000 fatal injuries. These injuries mostly comprise 

of children, adolescents and young adults. The Netherlands has the lowest rate of fatal injuries in EU and 

estimated more than 100,000 lives could be saved each year if the countries lower their injury mortality rate to 

the current Netherlands rate (Injuries in the European Union Statistics Summary 2005-2007). 

4.6.1.3.2 Nonfatal injuries 

An estimated 60 million people every year are medically treated for an injury. 42 million people are treated in 

hospital for injuries every year and out which 7 million are admitted for severe injuries which is about 19000 

people every day. The total estimated cost of injury related hospital treatment is €15 billion, this is because of 

the patients receive more than 50 million hospital days of treatment every year accounting to 9% of all 

hospital days (DG Sanco 2004 cited in Injuries in the European Union Statistics Summary 2005-2007). There 

are more than 3 million people permanently disabled in the European Union due these non-fatal injuries 

(Labour Force Survey 2002 cited in Injuries in the European Union Statistics Summary 2005-2007). Figure 

below shows non-fatal injuries per 1000 by sex and age group. 

4.6.1.3.3 Trends 

Figure 9: Non-fatal injuries per 1000 by sex and age group 

Fatal home and leisure injuries are growing because of falls among elderly people. Non-Fatal injuries are also 

on the rise in the home and leisure areas where as in traffic and work-place it is level. There is going to be 

rise in the number of disabled people because of the decline in fatal injuries and stable or increasing non-fatal 

injuries. The injury hotspots that are identified are children, adolescents, older people, vulnerable road users, 

sports, product and services related accidents, interpersonal violence, and self harm. 75% of all injury deaths 

are due to accidental injuries and 25% are due to intentional injuries. The most fatalities occur due to suicide 

24


and road injuries both in absolute and relative terms (in relation to the number of hospital treated injuries). 

70% of all injuries in EU are treated in hospitals. Home, leisure and sports are the biggest domain of all 

hospital treated injuries accounting for 74% (Assessing disability 2004 cited in Injuries in the European Union 

Statistics Summary 2005-2007). Road injuries account for 10% of all hospital treated injuries or a total of 4.3 

million victims every year as estimated by EU IDB (Injuries in the European Union Statistics Summary 2005- 

2007). 

4.6.1.4 Wheel Chair Market Europe (Business Wire 2009) 

The top five countries in wheelchair market are Germany, United Kingdom, Italy, Spain, and France. The 

market increased by 5.8% in total units sold and 5.2% in value in 2008 in the top 5 countries. The wheelchair 

market is expected to grow steadily after 2008 and increase to approximately 300,000 units sold in 2011 

which will amount to €737.3 million in sales. More people are surviving accidents and people live longer 

because of better medical treatment and equipment. This has resulted in an increase in the growth of 

wheelchair market. 

4.6.2 Second Demonstrator: Robotic Systems for examining Hazardous Environments 

Nowadays intelligent remotely operated vehicles are often employed in scenarios including investigation of 

environments like toxic or irradiated environments or areas hardly reachable by a person or dangerous for a 

person to walk through. Many intelligent vehicles have been developed in the past years and some of them 

are already available on the market. 

Remotely operated vehicles have a large variety of sensors for environment perception as well as a robust 

platform with wheels or chains for propulsion, but they do not act independently, being remotely operated by 

a person. Some remotely operated vehicles include some intelligence on board in order to monitor its 

surroundings and sense dangerous situations (such as detecting and avoiding obstacles), which makes the 

remote control significantly easier for the operator. The wireless communication channel is critical for remote 

controlled vehicles, because it should enable a real-time transfer of all data required by the user. 

Remotely operated vehicles exist in different size and weight categories, depending on the applications which 

they should serve. They are employed in military as well as civilian applications. Many models exist on the 

market, modified to specific applications. The scenarios in which such vehicles are useful vary from 

reconnaissance and sample collection to Explosive Ordnance Disposal (EOD). Some of these vehicles are 

presented in the following paragraphs. 

Figure 10: Asendro EOD robot 

25


ASENDRO EOD developed by Diehl BGT Defence (Modular Robot ASENDRO, 2007) is a robot which 

allows rapid and reliable detection of suspicious objects as well as dangerous zones and is the first small robot 

worldwide which can be used for both reconnaissance and EOD (Explosive Ordnance Disposal) tasks. The 

robot is equipped with a manipulator arm including a two-jaw gripper which is used for moving and handling 

objects such as door and window handles. The control of the arm is done through the new type of 

telepresence technology, which means that the arm is moved synchronously with the operator's hand or head 

movement. This feature enables the operator to act as if he himself were on site. Stereo cameras are attached 

at the tip of the arm, which transmit their data to a head-up display. Thus, the operator can see threedimensionally 

"with the robot's eyes" and estimate distances from objects. 

The modularity of the ASENDRO EOD allows it to become becomes the reconnaissance robot ASENDRO 

SCOUT through replacement of the payload module. Controlled from a safe distance, the robot explores the 

operational site. A thermal and a video camera is then mounted on the extendable reconnaissance arm and 

transfers images as well as sound to the command centre. 

Figure 11: Dragon runner 

The Dragon Runner (Dragon Runner Reconnaissance Robot) originally developed by the National Robotics 

Engineering Centre (NREC) in 2002-2003, is a rugged, ultra-compact, lightweight and portable 

reconnaissance robot developed for urban operations (UO). The prototype model of the Dragon Runner 

measures around 23cm in length, 20cm wide and 7.5cm tall with a weight of about 7kg. The basic robot 

operates as a tough, low-observable ground sensor providing corner views to users moving at a speed of about 

8km/h. 

In May 2007, Automatika was acquired by Foster-Miller, a QinetiQ North America company, which made the 

Dragon Runner commercial. In November 2009, QinetiQ received contracts for around 100 Dragon Runner 

robots from the UK Ministry of Defence (MoD) to support its military operations in Afghanistan. The value 

of the contract was £12m ($19m) and includes the provision of technical services and spares. 

26


Figure 12: TechnoRobot RiotBot with remote control 

TechnoRobot specialises in the design and manufacture of robots intended for the security and defence 

sectors. After two years of development, TechnoRobot has launched RiotBot (TechnoRobot – Rapidly 

Deployable Remotely Operated Less-Lethal Support Robos), a robotic platform requested for years by 

different international agencies involved in high-risk operations for its security and defence. To design the 

RiotBot, a strong R&D team was formed within the enterprise by more than 20 technicians working closely 

alongside various special operations tactical experts. 

RiotBot was designed to intervene in high-risk missions to remove the threat to human team members. The 

principle objective is to minimise injury within both the intervention group and the opposing force. To that 

end, the RiotBot is equipped with a nonlethal weapon. It is also equipped with a laser site for greater weapon 

acuracy and a powerful flashlight for nighttime operations. RiotBot is equipped with six-wheel-drive 

capabilities, and can reach speeds exceeding 20km/h. A light-weight portable control unit allows the RiotBot 

operator to view images captured by the robot's camera in real time from over a kilometer away. 

Figure 13: iRobot 510 Packbot 

510 PackBot (GROUND ROBOTS – 510 PACKBOT®) produced by iRobot is a tactical mobile robot that 

performs multiple missions while keeping warfighters and first responders out of harms way. Through its 

modularity, adaptability and expandability it can be used in various scenarios like bomb disposal, surveillance 

/ reconnaissance and hazardous material detection. 510 PackBot easily climbs stairs, rolls over rubble and 

navigates narrow passages with sure-footed efficiency, traveling at speeds of up to 9 km/h. 

510 PackBot relays real-time video, audio and other sensor readings while the operator stays at a safe standoff 

distance. The operator can view a 2-D or 3-D image of the robot on the control unit, allowing for precise 

positioning. PackBot also features a game-style hand controller for faster training and easier operation in the 

27


field. More than 3,000 PackBot robots have been delivered to military and civil defence forces worldwide. 

Figure 14: EOD-Roboter made by telerob 

tEODor (telerob) stands for "Explosive Ordnance Disposal and observation robot" and is a remote controlled 

EOD robot produced by telerob. It has unique features such as a programmable six-axis manipulator, 

additional linear axis, automatic tool exchange, integrated diagnostic system and parallel operation of up to 

five disruptors. It sold more than 350 units in 39 countries to military and police units worldwide. 

5 Mechatronic Control Systems 

5.1 Canonical subsystems 

� Sensoring, data acquisition, fusion and interpretation (TASK 3.1, SINTEF) 

� Sensor network design (TASK 3.2, SINTEF) 

� Robotic system modelling and integration of motor control units (TASK 6.2, UB) 

� Design and development of the mobile platform (TASK 7.1, OMS) 

� Design and development of the powered orthosis (TASK 7.2, OB) 

� Design and integration of actuation system (TASK 7.3, SCHUNK) 

� Development of low-level control units (TASK 7.4, VUB) 

� Realization of the robotic system (TASK 7.5, SCHUNK) 

5.2 Human sensing systems (Tasks 3.1, 3.2; SINTEF) 

Relevant tasks: 

� Sensoring, data acquisition, fusion and interpretation (TASK 3.1, SINTEF) 

� Sensor network design (TASK 3.2, SINTEF) 

CORBYS Overarching Requirements on the human sensing systems (physiological sensors system excluding 

BCI components) and interfaces to other WPs and partners. 

5.2.1 Functional Requirements 

5.2.1.1 Processes 

On selecting physiological sensor components for CORBYS and properties of individual sensor 

components 

28


Requirement No. HSS1 

Name: Sensors implemented in the CORBYS system 

Description: The actual sensors selected will be defined in the detailed specification 

Reason / Comments: Comment 

Anticipated sensors: 

� Heart rate 

� EMG (muscular activity) 

� EDR and/or humidity/sweat sensors 

� Inertial measurement units (3 axis accelerometer, gyroscope and magnetometer) 

� The requirements for mechanical sensing (force, torque, angular joint movements, 

force/pressure distribution) need to be discussed with the consortium 

Indicative priority Mandatory 

Requirement No.: HSS2 

Name: Sensor locations 

Description: The actual sensors selected will be defined in the detailed specification. 

Reason / Comments: Comment: 

A complete discussion on the physical/mechanical patient-robot interface configuration is 

required here. Some aspects related to sensing: 

� Homing positions of inertial sensors 

� Positions of robot mechanical support to the patient (such as limb fixation) and 

patient movement actuators 

� Actuator position movement vs. limb position movement: Both the robot and the 

body sensors can be used to find e.g. the position and angle of a knee. Which is 

best, or is redundancy required? 

� Can certain sensing components be integrated in the robot itself, rather than being 

mounted on the patient? 



Name: Patient user size 

Description: Adult users. Height, weight circumferences will be discussed with clinical partners 


� A complete discussion on the physical/mechanical patient-robot interface 

configuration is required here. Some aspects related to sensing: 

� Homing positions of inertial sensors 

� Positions of robot mechanical support to the patient (such as fixing on of limb) and 

patient movement actuators 

� Actuator position movement vs. limb position movement: Both the robot and the 

body sensors can be used to find e.g. the position and angle of a knee. Which is 

best, or is redundancy required? 

� Can certain sensing components be integrated in the robot itself, rather than being 

mounted on the patient? 


Sensor Output 


Name: Sensor output of primary parameter values 

Description: Definition on how human sensor parameters are shared with the rest of the CORBYS 

system, as well as in export to log files: Details are to be defined. 

Reason / Comments: (Example: 

Heart rate will be in [beats/minute], based on integration of values during the past 5 seconds. 

The value will be updated every second) 

29




Name: Safety-related sensor output – information derived from sensor fusion of multiple sensors 

Description: Status information or flags should be raised if sensor readings indicate a potentially 

hazardous situation. To be discussed with partners with stakes in the design of CORBYS 

control system 

Reason / Comments: Detailed definition will be developed later, and will depend upon the entire CORBYS 

system configuration 



Name: Sensor output related to physical effort assessments 

Description: Details are to be defined. 

Reason / Comments: Comments: 

� Detailed definition will be developed later, and will depend upon the entire 

CORBYS system configuration 

� Synergies with BCI measurements and development of artificial muscle actuators 

must be explored 



Name: Sensor output related to gait parameter assessments 





� Synergies with BCI measurements and development of artificial muscle actuators 

must be explored 



Name: Sensor output related to identification of psycophysiological states 





� Synergies with BCI measurements must be explored 



Name: Sensor output related to identification of intentional states 





� Synergies with BCI measurements must be explored 


Some overarching design constraints on the physiological sensors and the rest of the CORBYS system 


Name: Data processing and signal analysis requirements 

Description: Details not yet decided. 

Reason / Comments: Generally speaking, the project needs to sum up all the data processing requirements, both 

capacity and platform wise. 

30




Name: Integrating physiological sensor measurements (SINTEF) with BCI (BBT) 

Description: Details not yet decided. 

Reason / Comments: Discussions between BBT and SINTEF regarding finding a shared platform for integrating 

sensor signals. 



Name: Interfacing physiological sensor measurement system with the main CORBYS cognitive 

robot control system 

Description: 

Reason / Comments: The main alternatives for feeding the sensor data stream into a computer: 

� Developing a dedicated data acquisition printed circuit board allowing integration 

with the computer using standard cables (parallel, serial RS232 or USB) 

� Purchase a dedicated data acquisition board from for example National Instruments 

� Integrate sensors using standards based wireless communication, e.g. Bluetooth or 

WiFi. 



Name: Mains requirements 

Description: It must be anticipated that some of the measurement equipment will require 220V/50Hz 

Reason / Comments: The project needs to sum up all the power requirements 



Name: Sensor network architecture requirements 

Description: 

Reason / Comments: The project needs to compile a summary of all sensors and actuators (with detailed operation 

characteristics and worst-case values) in order to specify the total sensor network 

architecture. 



Name: Online access to past rehabilitation sessions 

Description: Online access to past (and possibly ongoing) therapy sessions implies a software 

architecture solution, as well as probably a WiFi node on the CORBYS system 

Reason / Comments: A software architecture incorporating the requirements will have to be implemented. 



Name: CORBYS system intermittence, delay and synchronisation requirements for sensors and 

actuators 

Description: 

Reason / Comments: A shared understanding of signal propagation will have to be reached between the partners. 


5.2.1.2 Interfaces 

User interfaces, hardware/software interfaces i.e. what the system must connect to. 


Name: Number of physiological sensor probes on the patients 

Description: The detailed number is TBD. No limitations are stated at this stage. 

31


Reason / Comments: Comment: One important design challenge in order to make the system usable is to select a 

minimum sensor configuration that gives all essential information while allowing easy 

mounting and dismounting of a low number of sensor devices. 



Name: Number of physiological sensor probes on the patients 

Description: For ease-of-use purposes, it will be desirable to combine several sensors into single devices, 

thereby reducing the experienced system complexity 

Reason / Comments: 

Indicative priority Desirable 


Name: Signal connection of physiological sensors to the CORBYS system 

Description: Sensor data signals will be sent through electrical wires/cables 




Name: Signal connection of physiological sensors to the CORBYS system 

Description: For ease of use purposes, possibilities to make some sensor units transmit data using 

wireless communication protocols will be considered 


Indicative priority Optional 

5.2.2 Non Functional Requirements 

5.2.2.1 Performance 


Name: Intended duration of continuous usage of physiological sensors 

Description: Anticipation: A therapy session will last up to 2 hours 

Reason / Comments: Usage scenario perspectives must be provided by clinical partners. The intended duration of 

usage affect the sensor technologies to chose from 




Description: Anticipation: 8 hour sessions 





Description: If CORBYS becomes a “community walker” gait assistance system, usage sessions could 

last from morning to evening. 



5.2.2.2 Safety and reliability 

Physiological sensors safety and comfort to use: 


Name: Electrical measurement system safety 

32


Description: Within Consortium electronic systems are acceptable as long as they are tested and deemed 

safe for the CORBYS users (e.g. complete user shielding from 220V/50Hz). 


It is important to discuss local human subjects research requirements with respect to 

carrying out human subjects testing using “investigational” sensors and concepts like 

CORBYS 

Keep in mind that the system should be safe for both patient and professionals 




Description: CE Medical device standard electrical safety 





Description: Medical device CE approvals on all sensor components (For a commercial product after 

CORBYS) 




Name: Sensor systems should not be invasive or excessively obtrusive 

Description: In vivo (implanted) sensor systems are not a part of the CORBYS physiological 

measurement system 

Sensor concepts probing human fluidic samples (blood, urine, saliva etc) are excluded 

Sensor concepts probing human body openings (such as rectal core temperature 

measurements and breath air gas analysis) are excluded 




Name: Limitations in the range of acceptable users 

Description: Based on user safety concerns, the physiological measurements system might not be used on 

patient groups such as: 

� Patients with electronic implants 

� Patients with certain dermatologic conditions 

� Patients with limitations in cognitive capabilities 

� Others - to be decided 




Name: Mounting and removal sensors on the patient 

Description: The physiological sensors will be mounted and removed by trained clinical rehabilitation 

professionals 

Reason / Comments: Usage scenario perspectives must be provided by clinical partners 



Name: Mounting of individual sensor components directly on the user’s skin 

Description: Certain physiological sensors (for example electrode based) can be placed at optimum 

measurement locations, directly on the skin of the patient. 

Reason / Comments: Usage scenario perspectives must be provided by clinical partners : 

This requirement implies that some sensors will have to be placed under the patient’s layers 

of clothing, and possibly expose some body parts during mounting 

33




Name: Mounting of individual sensor components directly on the user’s skin 

Description: Less optimal, but more user friendly locations can be used. 




Name: Placement of sensor components on the patient 

Description: All sensor components should be clearly marked in order to reduce the risk of placing 

sensors in incorrect measurement positions (e.g. mix left and right) 




Name: Placement of sensor components on the patient 

Description: Preferably automated detection mechanisms to avoid the risk of incorrect placement 



5.2.2.3 Other 


Name: Physiological sensor biocompatibility issues 

Description: Sensors should not cause irritations/inflamatic responses or pain during the designated 

duration of CORBYS rehabilitation sessions. 

Sensors can be temporarily attached to the patient using e.g. medical grade adhesive tape 




Name: Physiological sensor biocompatibility issues 

Description: EC Medical device standard biocompatibility testing of all materials interfacing the patient 




Name: Physiological sensor hygienic issues 

Description: Physiological sensor interfacing the patient’s skin directly should be possible to clean or 

replace from patient to patient: 

Single use probes 

Multiple use probes that have smooth surfaces and that can be cleaned in appropriate 

detergents 




Name: Time required to mount or dismount all physiological sensors 

Description: For a trained user, it should be possible to mount all sensors within the maximum time 

required for the entire start-up and shut-down times targeted for the entire CORBYS system. 

Time allocated for the physiological sensor system alone is TBD. 

Reason / Comments: The consortium need to discuss this issue considering all the different procedures 

(adjustments, configurations, fixing of body and limbs, mounting of sensors, initiation of 

protocols) required prior to or after a therapy session. 


34


5.3 Robotic system modelling and integration of motor control units (TASK 6.2, 

UB) 


� Robotic system modelling and integration of motor control units 

Modelling of the robotics systems should be used for evaluating the concepts, algorithms and speeding-up 

development process of hardware and software. The following systems will be modelled: 

� actuation units, 

� sensor units, 

� environment (human will also be modelled as an environment subpart), 

� robot’s motor control units and cognitive modules. 

The developed models will be the starting point on research on robot control and cognitive control 

architectures. MSC Adams, MATLAB will be used for modelling and simulation of the robotic system, its 

components and environment. 



Inputs: 

Requirement No. RSM1 

Name: Model of mechanical construction of the demonstrator 

Description: CAD file: STEP or similar 

Reason / Comments: The form of the input will be determined in discussion with the partner responsible 



Name: Actuator models 

Description: CAD file: STEP or similar 




Name: Low level control algorithms 

Description: MATLAB, Program Code, Documentation 




Name: Signal processing algorithms 





Name: Self-awareness modules 

35






Name: Reflection modules 





Name: SOIAA architecture 





Name: BCI architecture 





Name: Patient motion data 

Description: Motion data files 



Outputs: 

The output of this task is not a functional sub-system, rather appropriate simulation environment where 

functionality of all other sub-systems will be tested using realistic models of robotic systems and their 

environment. 


Name: Training data 

Description: Output of the modelling and simulation process will be training data in the form of 

kinematics and dynamics data of the modelled robotic system and interaction with its 

environment. 



Processing: 


Name: Modelling of the walking in the rehabilitation system 

Description: The model of human body will be coupled with the model of the rehabilitation system and 

walking in the system will be modelled. Simple position controller will be used to “guide” 

the patient walk. 




Name: Modelling of different behaviours of the system 

36


Description: More complex control algorithms will be simulated and behaviour of the controllers will be 

observed as well as interaction forest and torques between human body model and the 

rehabilitation system. 




Name: Cognitive control 

Description: The higher control structures will be modelled and the behaviour of the system will be 

observed when system is controlled by cognitive modules. 



Data flow: 


Name: Software inter-module communication 

Description: Since several software modules will be used in this task (cognitive modules, sensor modules 

and other), it is necessary to agree with the responsible partners about the communication 

protocols. 





Name: Hardware in the loop simulation 

Description: It is possible to include mechatronical sub-systems of the robotic system directly in the 

simulation in order to prove concepts and test algorithms. 

Reason / Comments: Will be discussed with partners. 


5.3.1.3 Goals and expectations 


Name: Sub-system evaluation 

Description: Help in evaluating the concepts, algorithms and performance of the researched robotic 

system. Results will be used for improvements of simulated sub-systems. 



5.3.1.4 Operating environment 


Name: MATLAB and MSC Adams 

Description: The robotic system and all other software modules will be modelled using MATLAB and 

MSC Adams software on Windows platform. 



37


5.4 Design and development of the mobile platform (Task 7.1, OBMS) 

In this task the design and development of the mobile robotised platform will be carried out. 



Inputs 

Requirement No. MPD1 

Name: Requirements and mobile platform specifications 

Description: The actual mobile platform will be defined in the detailed specification. 


A complete discussion on the number and type of DOFs, their position and specifications 



Name: Modelling, simulation, optimization 

Description: Modelling, simulation and optimization of the robotic system, its components and 

environment, as the starting point on research on robot control and cognitive control 

architectures will performed by UB in Task 6.2. 



Processing 


Name: Mechanical design of the mobile robotic platform. 

Description: Design of the mobile robotic platform based on requirements and specifications defined in 

deliverables 2.1 and 2.2 




Name: Body weight support. 

Description: Infinitely variable body weight release to decrease ground reaction force provided through 

the frame of the machine rather than overhead support (harness). 

Reason / Comments: The ability of mobile platform to hold the patient could contribute to the design of powered 

orthosis 


Requirement No.: MPD5 

Name: Mechanical development (construction) 

Description: 




Name: Actuation of the mobile platform 

Description: The actual actuators selected will be defined in the detailed specification. 




38


Name: Sensor integration 




Output 


Name: CAD model of the mobile platform for robotic system modelling 

Description: CAD model for robotic system modelling and integration of motor control units in Task 6.2 

Reason / Comments: The developed models for simulation of the robotic system, its components and environment 

will be the starting point on research on robot control and cognitive control architectures. 



Name: Functional testing 

Description: 






Name: Patient stability 

Description: 


Patient must not fall with the system or collapse within the system 



Name: Obstacle avoidance 

Description: Obstacle identification including stairs up and down, collision avoidance 



5.5 Design and development of the powered orthosis (Task 7.2, OB) 

In this task the design and development of the exoskeleton/orthotic device, which will be later integrated into 

a mobile platform, will be carried out. 



Inputs 

Requirement No. POD1 

Name: Requirements and orthosis specifications 

Description: The actual orthosis will be defined in the detailed specification. 

39




(RangeOfMotion, torques and forces), cycle numbers 


Requirement No. POD2 

Name: Modelling, simulation, optimisation 

Description: Modelling, simulation and optimisation of the powered orthosis system, its components and 

environment, as the starting point on research on robot control and cognitive control 

architectures will performed by UB in Task 6.2. 



Processing 

Requirement No.: POD3 

Name: Mechanical design of the powered orthosis 

Description: Design of the powered orthosis based on requirements and orthosis specifications defined in 

deliverables 2.1 and 2.2 

Reason / Comments: An integrated design of appropriate type of joints and joint mechanisms or adaption 

interfaces allowing integration of the actuator system (to be developed by SCHUNK in 

WP7), and sensorics (to be developed by SINTEF in WP3), is preferred. 

Modular design will make the sensors-actuators integration to the joints more convenient as 

well as later the complete (powered) HKAFO integration to the mobile platform (to be 

developed by OBMS in WP7). 



Name: Design according to patient user size 

Description: Adaptation to patient’s height and weight 

Reason / Comments: The orthosis design should be adaptable so that the resulting orthosis can be easily fitted to 

various body schemas that are anatomical structures of end-users (patients with different 

age, size and weight). The specification of these data shall be provided in cooperation with 

the clinical partners. 



Name: Mechanical development (construction) 

Description: Definition of adaptation interface to actuators in cooperation with SCHUNK and to drive 

frame with OBMS (especially concerning pelvic motion). 

Mandatory requirements for clinical application (DOF) 

Reason / Comments: To calculate the needed strength of joint and bars the load values have to be known. 



Name: Sensor integration 


Reason / Comments: The sensory needed for the mechanical values shall be integrated in the orthosis (size and 

position have to be delivered by the partner providing the sensor), sensory for e.g. vital 

parameters have to be applied separately and not being implemented in the orthosis. 



40


Name: Actuation of the orthosis 

Description: Appropriate actuators will be defined in the detailed specification. 



Output 


Name: CAD model of the orthosis for robotic system modelling 

Description: CAD mode for robotic system modelling and integration of motor control units in Task 6.2 

Reason / Comments: The developed models for simulation of the robotic system, its components and environment 

will be the starting point on research on robot control and cognitive control architectures. 



Name: Functional testing 

Description: 


The system shall be tested in conjunction with motors (SCHUNK) and driving frame 

(OBMS) to ensure that no safety hazards occur and that all mechanical components work 

together 



Name: 

Functional system to be attached to mobile platform. 

Description: 


Functional system to be attached to mobile platform in Task 7.5 (SCHUNK) 




Name: 

Mobile platform interface module 

Description: An interface module on orthosis to be able to attach it to the mobile platform 




Name: 

Patient interface 

Description: A patient interface that fits the orthosis to the patient’s body (orthopaedic requirements) 






Name: Mounting time required to mount the system 

Description: A therapist must setup system within 10 minutes 

41





Name: Handling 

Description: Patient must set up system by himself 



5.6 Design and integration of actuation system (Task 7.3, SCHUNK) 

In this task, depending on established requirements SCHUNK will provide adequate actuators for actuating 

the mobile platform demonstrator and the powered orthosis demonstrator’s joints. 



Inputs 

Requirement No. ACT1 

Name: Requirements and actuators specifications 

Description: The actual actuators will be defined in the detailed specification. 




Requirement No. ACT2 

Name: Modelling, simulation, optimization 

Description: Modelling, simulation and optimization of the robotic system, its components and 

environment. 



Processing 

Specification is needed on operation range, load situations, cycle times and load cycles, 

voltage range, power supply, cabling, electrical interfacing, fusing, temperature range, IP 

protection, brakes and emergency situations. 

Requirement No.: ACT3 

Name: 

Smart and safe actuators 

Description: Smarter and safer actuator control by integrating force and torque sensors into the control. 

Reason / Comments: Research and develop new functions for smarter and safer actuator control by integrating 

force and torque sensors into the control. The research on the actuators is important, because 

of high miniaturisation and safe operation demands. Conformance for safe stop operation 

(STO) must be performed. 


42


Output 


Name: Actuator models for robotic system modelling and integration of motor control units (Task 

6.2) 

Description: The developed models will be the starting point of research on robot control and cognitive 

control architectures, modelling and simulation of the robotic system, its components and 

environment. 




Name: 

The actuator system of the orthosis 

Description: The actual actuators selected will be defined in the detailed specification. The actuator 

system of the orthosis will be developed by SCHUNK: output power, power source, motor 

characteristics (Voltage, current consumption, torque, velocity) with nominal and absolute 

ratings. and will be integrated to the orthosis through the Task 7.5 

Risk analysis in conformance with the European machine guidelines is expected. 




Name: Actuation of the mobile platform 

Description: The actual actuators selected will be defined in the detailed specification. The actuator 

system of the mobile manipulator will be developed by SCHUNK: output power, power 

source, motor characteristics (Voltage, current consumption, torque, velocity) with nominal 

and absolute ratings. and will be integrated to the mobile platform through the Task 7.5 

Risk analysis in conformance with the European machine guidelines is expected. 






Name: Safety 

Description: The safety solutions integrate intrinsic safety design, but also necessary dual channel 

(redundant) sensor integration and observation according to man machine cooperation 

standard DIN/ISO 10218 

Reason/Comments 


5.7 Development of lowlevel control units (Task 7.4, VUB) 

From the control point of view, the computational and sensing abilities of the robotic system impose 

additional limitations on the robot’s performance. The key contribution to a control system will consists of 

distributed set of interacting microcontroller units which will provide robust full monitoring and easy 

expandability. 

43




Inputs 

Requirement No.: LLC1 

Name: Mechanical design of the powered orthosis and mobile platform 

Description: Number and type of DOFs, their position and specifications 

Reason/comments To define main characteristics of the LLC is necessary to clarify specifications of all 

actuated DOF 



Name: Electro-mechanical components 

Description: Actuators for every DOF of powered orthosis and mobile platform; output power, power 

source, motor characteristics (Voltage, current consumption, torque, velocity) with nominal 

and absolute ratings. 

Reason/Comments The performance of CORBYS control architectures and the constraints on actuators assume 

a special importance in functionality of whole system. Controller manipulates the system 

inputs to obtain the desired effect on the output of the system. In order to have more 

accuracy in control some characteristics of electro-mechanical components to be discussed 

with partners. 



Name: Controller Inputs 

Description: Determine the sensors that will be connected to the controller unit to define number and type 

of the inputs required. Data sheet characteristics for all sensors. 

Reason/Comments After the sensors will be chosen it will be necessary to classify which ones will be connected 

to an acquisition data system and ones that can be read and used by LLC as feedback in low 

level control loop. 


Outputs 


Name: Controller Outputs 

Description: Specific output ports required to drive actuators or provide information to other control 

subsystems. 

Reason/Comments Besides the driving of the actuator it is necessary to provide power, references or data to 

other modules of the system, so the peripherals and special hardware components should be 

considered. 


Processing 


Name: Low level control loop 

Description: Developing a control scheme to accomplish low level control for each DOF of the system. 

Reason/Comments After all input and output parameters will be defined it will be necessary to develop a 

control scheme and implement the low level source code. LLC will operate in accordance 

with the main processing unit real time cycle, all to be discussed with partners. 


44


Data flow 


Name: Communication protocol 

Description: Every LLC will be a node in the system network which will receive and provide data, 

communication protocol should be fast, safe and reliable. 

Reason/Comments Considering real time operation it is crucial to have a real time communication protocol. 

LLC should integrate modules necessary to accomplish this task. 




Name: Test interface 

Description: Test Graphic User Interface 

Reason/Comments A graphical user interface should be designed to test the functionality of the LLC before 

being integrated in the system. 





Name: Dimensions and placement in system 

Description: 

Reason/Comments The placement of LLC will influence hardware design complexity and wiring architecture. 

To be discussed with the Consortium. 




Name: Safety functional parameters 

Description: Will be chosen to conform with the general requirements for safety in human-robot 

interactions (ex: admissible voltage for motor drivers) 



5.8 Realization of the robotic system (Task 7.5, SCHUNK) 

In this task SCHUNK will be responsible for assembly of complete robotic system that should be fully 

functional on completing this task. 



Inputs 

Requirement No. REAL1 

Name: Completed components of the robotic system. 

45


Description: The finalised robotic system will include mobile platform and powered orthosis both with 

integrated actuators, sensors, batteries and on-board electronics. All based on CAN bus 

communication and battery power supply. 



Processing 

Requirement No.: REAL2 

Name: 

Integration of the actuators to the orthosis 

Description: Manufacturing and assembling tasks for small scale miniaturized actuators based on 

electronic motors and motor control circuitry. Create drawings, bill of materials, 3D models 

and discuss them with other involved partners (OB, OBMS). 




Name: Integration of the actuators for the mobile manipulator 

Description: Manufacturing and assembling tasks for small scale miniaturised actuators based electronic 

motors and motor control circuitry. Create drawings, bill of materials, 3D models and 

discuss them with other involved partners (UB). 




Name: Sensor integration into the orthosis. 

Description: Assembly and electrical interfacing tasks. Software integration of data interface. Develop 

diagnostic software together with partner SINTEF and UB. 



Output 


Name: Functional test of mobile manipulator and orthosis. 

Description: Diagnostics software to control the actuators and fetch sensor data 



6 Human Control System 


� BCI detection of cognitive processes that play key roles in motor control and learning (TASK 3.3, 

UB) 

� Design of a brain computer software architecture (TASK 3.4, BBT) 

� Cognitive control Architecture decomposition and definition (TASK 6.1, UH) 

� Integration of cognitive control modules (TASK 6.3, UB) 

� Experimenting and evaluation (TASK 6.4, VUB) 

� Architecture revision and improvement (TASK 6.5, UH) 

� Final architecture integration and functional testing (TASK 6.6, UB) 

46


6.2 BCI detection of cognitive processes that play key roles in motor control and 

learning (Task 3.3, UB) 

Relevant task: 

� BCI detection of cognitive processes that play key roles in motor control and learning 

� The goal of this task is to detect cognitive information related to motor intention (e.g. intention of legs 

motion) via Brain-Computer Interface. Electroencephalography (EEG) data will be acquired during 

motor execution experiments on healthy subjects. The aim is to investigate the feasibility of 

distinguishing between intended movement and unintended movement. 



Inputs: 

Requirement No. DMI1 

Name: Experimental protocol design 

Description: Documentation that describes the Motor intention BCI experiments through three main 

elements: synopsis, data collection and analysis. 

Reason / Comments: This protocol is necessary to conduct the motor intention studies 



Name: Motor intention study 

Description: EEG data in .dat format recorded from healthy subjects 

Reason / Comments: This data is necessary for offline signal processing 


Outputs: 


Name: Algorithm for detection of motor intention 

Description: MATLAB Program Code 

Reason / Comments: This algorithm detects the intention of movement from stored EEG data 


Processing: 


Name: Cue-based motor intention detection algorithm 

Description: MATLAB Program Code (bci_Calibration.m) 

Reason / Comments: This algorithm detects motor intention based on a calibration session that contains EEG data 

locked to an event signal (offline). 



Name: Self-paced motor intention detection algorithm 

Description: MATLAB Program Code (bci_Process.m) 

Reason / Comments: This algorithm presents the second approach to detect motor intention based on data 

recorded online. 

47




This task will use hardware and software that is available at UB to conduct the studies. Therefore, no 

interfaces to other sub-systems are necessary to conduct the BCI experiments at UB. 

6.2.1.3 Roles and responsibilities 

To conduct the experiments that will help to investigate the detection of human cognitive information related 

to the motor tasks, the design of a detailed experimental protocol is necessary. This protocol will be designed 

and discussed in collaboration with BBT partner. The experiments will be conducted by UB with healthy 

users. 

UB will provide results of EEG offline and online analysis for the detection of motor intention to BBT in 

order to integrate the program code into the Brain Computer Interface software architecture that will be 

developed in task 3.4. 


The goal of this task is the detection of motor intention (e.g., before actual leg or feet movement is executed). 

It is expected that the signal processing can not differentiate between intention of moving right from left leg or 

foot. 


Laboratory: University of Bremen 

6.2.1.6 Resources 

Hardware: 

Data will be acquired through a Porti32 amplifier (Twente Medical Systems International, Netherlands). 

Software: 

BCI2000 framework software for signal acquisition and stimulus presentation will be used. 

Materials: 

33 Electrodes and 3 EEG caps are required. 

6.3 BCI Software Architecture (Task 3.4, BBT) 


� Design of a brain computer software architecture (TASK 3.4, BBT) 

This task will provide software architecture for the integration of the BCI-related devices and of the neural 

decoding mechanisms required in CORBYS (Task 3.5). 

48




Requirement No. BCISW1 

Name: BCI communication interface 

Description: External interface that communicates with other subsystems using a TCP/IP messages 

protocol. 

Reason / Comments: Easy communication and integration between subsystems. 



Name: Therapist GUI 

Description: The graphical user interface (GUI) allows the therapist to interact with the BCI software. 

Reason / Comments: Facilitate the use of the BCI software to non-computer experts. 



Name: Subject GUI / User GUI 

Description: In the training and decoding process subjects are asked to perform some tasks. 

Reason / Comments: The graphical user interface (GUI) displays commands to the subject (e.g. a visual cue 

indicating that the subject intends to start walking). 




Name: BCI software portability 

Description: The BCI software can run over different operating systems (Windows, Linux, etc.) 

Reason / Comments: The BCI software can be easily moved from an environment to another. 




Name: EEG sensor cap size 

Description: The cap is available in 3 sizes (small, medium and large); the most appropriate one needs to 

be chosen depending on the subject head circumference. Anyway the medium-sized cap is 

suitable for over 95% of all adult subjects. 

Reason / Comments: An appropriate cup allows achieving a better signal quality. 



Name: EEG Electrodes location and number 

Description: The EEG electrodes are inserted via small holes in the cap. Their position on the scalp, 

indicated on the cap according to the extended international 10/20 system, and number 

depends on what brain areas are activated during a specific cognitive task. Ongoing 

CORBYS research will identify those (Tasks 3.3 and 3.5). 

Reason / Comments: Specific electrodes placements and number related to the cognitive processes required in 

CORBYS are needed. 



49


Name: Subject screen / User screen 

Description: The graphical user interface (GUI) displays commands to the subject (e.g. a visual cue 

indicating that the subject has to start walking). 

Reason / Comments: Hardware needed for Requirement No.BCISW3 



Name: Therapist screen 

Description: The graphical user interface (GUI) allows the therapist to interact with the BCI software. 

Reason / Comments: Hardware needed for Requirement No. BCISW2 



Name: BCI processing unit 

Description: The minimum computing power needed depend on the results of the ongoing CORBYS 

research (e.g. laptop, netbook, personal digital assistant, etc.). 

Reason / Comments: Hardware required for the BCI software. 



6.3.2.1 Other 


Name: EEG system montage 

Description: A fast and easy EEG system montage, cap and electrodes placement, is required. Associated 

with these requirements the most appropriate system will be used. 

Reason / Comments: A rehabilitation scenario where the subject feels comfortable is desired. 



Name: EEG system portability 

Description: A reduced size and weight EEG system is required. Associated with these requirements the 

most appropriate system will be used. 

Reason / Comments: A rehabilitation scenario where the subject feels comfortable is desired. 


6.4 Architecture decomposition and definition (Task 6.1, UH) 

The possible approaches of including cognitive structures developed in WP4-5 into robot control structures 

will be investigated here. In particular, it will define the coupling of the SOIAA architecture developed in 

WP5 to the CORBYS framework. The core SOIAA architecture will obtain information about the current 

status of the human-robot system primarily in the form of low-semantics, uninterpreted sensoric 

measurements, such as (depending on scenario) location, positions of human and robot, angles of human 

limbs, forces and accelerations, and transform this into movement patterns proposed by SOIAA. SOIAA will 

then send a request to the CORBYS architecture to carry out these movements. The interface to the CORBYS 

architecture will filter the behaviours (i.e. movement commands) to be constrained with respect to known 

constraints in dynamics with respect to fundamental feasibility, stability, directionality and other requirements 

determined externally or explicitly imposed and stored in the CORBYS knowledge base (see Task 5.4). This 

ensures compliance with fundamental feasibility and safety constraints imposed on the system. 

In the second phase, the CORBYS-SOIAA interaction will be extended to support a higher-semantics 

50


(interpreted) sensoric data stream from CORBYS to SOIAA. It will combine the semantically richer, 

interpreted, sensoric and metasensoric data provided by the CORBYS architecture (e.g. from the BCI interface 

or from the higher levels of the knowledge base) with the low-semantics SOIAA core via fusion techniques. 

These higher-level data can be beneficial to the human intention estimate of SOIAA, by providing more 

refined and informed estimates. However, since they may be “tainted” by misinterpretation or wrong 

assumptions in estimating the human-robot take-over/hand-over goal-setting transitions, the two-level 

approach will allow to separate the influence of more precise and objective raw sensoric datastreams from less 

reliable, more ambiguous interpreted sensoric and meta-sensoric data streams. 



Inputs 

Requirement No. ADD1 

Name: SOIAA Module 

Description: Specification / Documentation/ Dependent inputs UH 

Detailed description of the SOIAA module is given in WP 5 Specs. The task 6.1 requires not 

necessarily a finished version of SOIAA, but a detailed understanding of what SOIAA is 

doing, and the necessary input information detailed in the SOIAA requirements. 



Requirement No. ADD2 

Name: External Safety filtering 

Description: To safely reintegrate SOIAA into the CORBYS structure it is necessary that some safety 

filtering for the outputs of the SOIAA module is provided. 



Output 

Requirement No. DAA3 

Name: Specification of SOIAA Integration Interface 

Description: A detailed account of how the different outputs of the SOIAA system should be integrated 

into the overall CORBYS framework. 

Reason / Comments: This requires in-depth discussions with our partners, specifically regarding the level of 

cognitive involvement in low level task of SOIAA. 


6.5 Integration of cognitive control modules (Task 6.3, UB) 


� Integration of cognitive control modules (TASK 6.3, UB) 

The cognitive control modules that will be developed in Work-Packages 4 and 5 should be integrated with the 

models of every robotic subsystem. All subsystems will be consolidated into one functional model of the 

cognitive robotic system. 

51




Inputs 

Requirement No. CCM1 

Name: Semantically-Driven Self-Awareness Module (SDSA) sub-system 

Description: Program Code / Documentation 




Name: Situation Assessment Architecture 






Description: Program Code/ Documentation 




Name: Expectation Modelling Engine 

Description: Program Code/ Documentation 




Name: SOIAA sub-system 





Name: Identification and anticipation of human purposeful behaviour and information flow 





Name: Self-motivated gait and goal generation sub-system 





Name: Definition of the cognitive sub-system structure 

Description: The definition of the structure of the cognitive sub-system is output of the Task 6.1 and 

represent the joint work of UR, UH, UB and VUB. 



° - The form of the input will be determined in discussion with the partner responsible 

52


Outputs: 


Name: Cognitive sub-system 

Description: The cognitive sub-system will be responsible for overall behaviour of the robotic system. It 

will be responsible for long-range activities based on high-level goals and anticipation of the 

environment. The cognitive control subsystem will be also responsible for re-planning 

activities in a case when that situation demands. 





Name: Connection to sensing network sub-system. 

Description: Sensing network sub-system should provide pre-processed sensor data for cognitive 

modules in appropriate time rate. 

Reason / Comments: The form of sensor data – TBD. 



Name: Connection to actuation sub-system 

Description: The sub-system will provide following information to actuation sub-system: requests for 

control action, control parameters and /or referent trajectories for real-time control units. 




Name: Connection to HRI 

Description: Connection interface – TBD. 





Name: ROS environment 

Description: The cognitive sub-system will be one of the entities of the robot control architecture that will 

be implemented using the ROS (Robot Operating System) software. 

Reason / Comments: ROS is framework for robot software development. It provides libraries and tools to help 

software developers create robot applications. It provides hardware abstraction, device 

drivers, libraries, visualisers, message-passing, package management, and more. 


53


6.6 Experimenting and evaluating simulations (Task 6.4, VUB) 



Inputs 

Requirement No.: EES1 

Name: Model of mechanical design for CORBYS system 

Description: CAD representation of mechanical construction 




Name: Design and modelling for actuation system 

Description: Information and documentation 




Name: Models for motor control units and cognitive sub-system 

Description: MSC Adams, MATLAB files 




Name: Models of cognitive control sub-system 

Description: MSC Adams, MATLAB files 



Processing 


Name: Simulation scenarios for healthy persons 

Description: Development of various experiments for testing all functionalities and possibilities of the 

robot, evaluation of accuracy and performance in execution time. Type of experiments will 

be decided during development of project. 




Name: Simulation scenarios for impaired persons 

Description: A number of different simulation tests will be performed in function of typology of endusers 

– per their locomotion abnormalities 




Name: Experiments and evaluation for optimal parameters settings 

Description: To obtain more clear results we will develop some simulation scenarios for adapting the 

optimal settings in diverse situations. 



54



6.6.2.1 Other 


Name: Methodology and design of result data 

Description: The data interpretation and sharing will be adopted. 



6.7 Architecture revision and improvement (Task 6.5, UH) 

According to simulation results obtained in previous task revision and improvement of control architecture, 

cognitive modules and simulation models will be performed. 



Inputs 

Requirement No.: ARI1 

Name: Simulation Results from Task 6.4 

Description: 




Name: Measurable Cognitive Success Criteria 

Description: To determine if the status of the project is satisfactory, or where improvement should be made 

it is necessary to define some measurable criteria that the results of Task 6.4 can be compared 

with. The exact criteria are yet to be decided, and should be informed by the overall CORBYS 

design documentation and requirements. 



Outputs 


Name: Improvement of Cognitive Architecture 

Description: This outputs specific nature depends heavily on the not yet available test results. In case the 

test results are satisfactory, in regard to the system specifications, then this output becomes 

optional. 



6.8 Final architecture integration and functional testing (Task 6.6, UB) 


� Final architecture integration and functional testing (Task 6.6, UB) 

In this task, all modules and system components that have been researched and developed through WP 3-5 

55


will be combined into the final cognitive control architecture. Thereby the research and development on 

cognitive control architecture and all modules will be finished and verified in order to accomplish real system 

evaluation. 



Inputs: 

Requirement No. FAI1 

Name: Sensor network architecture and sensor processing sub-systems 


Reason / Comments: All sensor data shall be displayable in the different kinds of user interface 



Name: SDSA sub-system 


Reason / Comments: The form of the input will be determined in discussion with the partner responsible. 



Name: Situation Assessment Architecture 

Description: Program Code / Documentation° 









Name: Expectation Modelling Engine 





Name: SOIAA sub-system 





Name: Identification and anticipation of human purposeful behaviour and information flow 





Name: Self-motivated gait and goal Generation sub-system 




56



Name: BCI sub-system 


Reason / Comments: The BCI results shall be displayable for the user in order to be able to verify system 

decisions. The form of the input will be determined in discussion with the partner 

responsible. 



Name: Actuation sub-system 


Reason / Comments: Hardware components must be accessible for the architecture in order to execute required 

actions. The form of the input will be determined in discussion with the partner responsible. 



Name: Real-time control sub-system 

Description: Program Code / Documentation° 

Reason / Comments: The control sub-system shall be accessible for the architecture as well for giving input to the 

controller module if necessary. The form of the input will be determined in discussion with 

the partner responsible. 



Name: User interface 

Description: Development of user interface as necessary for functional testing and evaluation purposes. 

Reason / Comments: The user interface is required since it is inevitable for testing the whole architecture with all 

its subsystems 



Name: User interface for patients 

Description: An adjusted minimised user interface giving just required and necessary information during 

runtime in order not to overburden the patient with the technical details. This user interface 

serves mainly to initiate/abort actions of the system 




Name: User interface for CORBYS Professional / Expert User 

Description: This user interface is adjusted to suit the needs of the therapists in order to manipulate the 

desired system reactions and change parameters during runtime for achieving the best 

possible therapeutic results 




Name: User interface for developers 

Description: This user interface serves for all kinds of developers being equipped with several modes that 

can display/manipulate every detail of every subsystem in order to debug/improve the 

components of the system 




Name: Cognitive user interface 

Description: This user interface is designed for the specific needs of the cognitive modules, allowing to 

adjust the cognitive modules 

57




Outputs: 


Name: Robot cognitive control architecture 


Reason 

Comments: 

/ Final architecture combining all components into one system 



If observed as a sub-system the software control architecture is the interface between robot hardware and 

robot intelligence. All software sub-systems listed in the table above (list is not finalised) will communicate 

with each other and they will form the “brain” of the robotic system. The software framework that “glues” 

together all other modules will provide basic functionalities of communication between modules, scheduling, 

hardware abstraction and similar. For the communication between the modules, the Robot Operating System 

ROS will be used as a framework for robot software development. The communication will be instantiated in 

C++ in an open manner, using platform and programming language independent interfaces. All information 

will be handled in data-containers that can be configured for every specific need and identified by its recipient 

for further use. Desired data has to be explicitly requested and can contain either single values or a request for 

data streams. 


Name: Interface to sensor network and actuation system (CAN) 

Description: In order to control the robotic system and to sense the environment and robots internal state 

it is necessary to communicate sensor data and exchange it between software modules of 

robotic system. CAN interface will be used for communication. The structure of CAN 

contents should be determined (TBD). 

Reason / Comments: CAN network is widely used protocol in robotics, automation and automotive industries. It 

is supported by a large number of manufacturers and de facto standard for complex robotic 

systems. 



Name: Interface to user interface 

Description: TBD 

Reason / Comments: TBD 




Name: ROS environment 

Description: The cognitive sub-system will be one of the entities of the robot control architecture that will 

be implemented using the ROS (Robot Operating System) software. 

Reason / Comments: ROS is framework for robot software development. It provides libraries and tools to help 

software developers create robot applications. It provides hardware abstraction, device 

drivers, libraries, visualisers, message-passing, package management, and more. 

58






Name: Code standards 

Description: Software design based on MISRA C, MISRA C++ or similar software development 

standards for safety critical systems (applicable only for real—time control sub-modules). 

Usage of some analysis software such as: PC-Lint, LDRA or similar would be beneficial. 

Reason / Comments: In order to make system safe for potential users. 


7 Robohumatic Systems (Graceful RobotHuman InteractiveCooperative 

Systems) 


� Human-robot sharing of cognitive information (TASK 3.5, BBT) 

� Device Ontology Modelling (TASK 4.1, UR) 

� Self-Awareness Realisation (TASK 4.2, UR) 

� Robot response to a situation (TASK 4.3, UR) 

� Expectation as a specification of an anticipated outcome (TASK 4.4, UH) 

� Development of self-motivated gait and goal generator for the cognitive architecture (TASK 5.1, UH) 

� Models and algorithms for the identification and anticipation of human purposeful behaviour (TASK 

5.2, UH) 

� Algorithms for measurement of anticipatory information flow between robot and human and vice 

versa (TASK 5.3, UH) 

� Development of framework for transitional dynamics between robot-initiated and human-initiated 

behaviours (TASK 5.4, UH) 

� User Responsive Learning and Adaptation Framework (TASK 5.5, UR) 

� Cognitive adaptation of low level controllers (TASK 5.6, UB) 

7.2 BCI Cognitive Information (Task 3.5, BBT) 


� Human-robot sharing of cognitive information (TASK 3.5, BBT) 



Input 

Requirement No. BCI1 

Name: Execution Mode 

Description: Indicates which operation between training and decoding is going to be used. 

59


BCI requires a machine free training stage before users can work the technology. The 

training process modifies some internal parameters that successively the decoding process 

uses. This procedure must be observed for each cognitive-related task is planned to be used. 

A training phase is also needed for the artefacts removal processing (Training artefacts). 

The possible values of the execution mode input are 

� Training motion * 

� Training feedback 

� Training attention 

� Training artefacts 

� Decoding motion 

� Decoding feedback 

� Decoding attention 

� Decoding motion & feedback 

� Decoding motion & attention 

� Decoding feedback & attention 

� Decoding motion & feedback & attention 

� Stop 

The independence of the cognitive-related tasks allows their simultaneous execution in the 

decoding process (i.e. Decoding motion & feedback, Decoding attention & feedback, etc.). 

A stop input value has been also added to allow the interruption of the running processes. 

*Motion, feedback and attention are the abbreviations for intention of legs motion, feedback 

error-related potential and attention states respectively. 




Name: Configuration File 

Description: Values of the optional parameters, a default setting is provided 

A list of possible optional parameters is available below: 

� Number of electrodes to be used. 

� Sampling rate of the EEG signal. 

� Decoders parameters. 

A complete list will be provided depending on the results of the ongoing CORBYS research. 



Output 


Name: Raw EEG 

Description: Electroencephalographic signal acquired by the BCI hardware 




Name: Filtered EEG 

Description: Electroencephalographic signal filtered from occurring artefacts 

60




The tables above show the inputs and outputs of the Brain Computer Interface (BCI) subsystem, beyond them 

each cognitive-related subtask may need and/or provide more information (see following tables). 

Subtask 3.5.1 -- Intention of legs motion 

Several studies have demonstrated the appearance of EEG activity preceding human voluntary movement. 

These signals are associated to motor task preparation, and dissimilar from those during the actual execution. 

In this subtask we will study the appearance of such kind of signals related to right and left legs movement. 

Output 


Name: Intention of legs motion decoding flag 

Description: Information about which leg the subject is going to move. It also provides a “no movement” 

output value 

The intention of legs motion decoding flag output can take the following values: right leg, 

left leg and no movement. 




Name: Decoding accuracy 

Description: It provides a numerical value (e.g. a percentage) related to the ability of the BCI subsystem 

in detecting the intention of legs motion. 

Reason / Comments: Information about the performance in decoding intention of legs motion. 


Subtask 3.5.2 -- Feedback error-related potential 

A type of error-related potential is produced when a subject is informed that he has committed an error. The 

brain signal following incorrect feedback differs from the signal following the correct one. Based on this 

theoretical background the subtask 3.5.2 will analyse the presence of feedback error related potential in a 

motor task. 

Input 


Name: Error marker 

Description: Feedback stimulus that informs the subject about the correctness of their response to a 

specific task 

The feedback stimulus presented after the accomplishment of a task, informs the subject 

about the correctness of his response and therefore provide the critical information that 

would enable the error detection. Feedback stimulus, provided by other subsystems, can be 

auditory, visual, somatosensory, etc. 

The error marker input is a time marker that informs the BCI subsystem when the feedback 

is presented to the subject. 

61




Outputs 


Name: Feedback error-related potential decoding flag 

Description: Information about the presence of the feedback error-related potential in the brain signal 

The feedback error-related potential decoding flag output can take the following values: 

present and absent depending on the presence of the feedback error-related potential in the 

brain signal. 






in detecting the feedback error-related potential. 

Reason / Comments: Information about the performance in decoding feedback error-related potential. 


Attention states (Deliverable 3.3, BBT) 

Several studies have focused on the ability of the subject to maintain a consistent behavioural response during 

continuous and repetitive activity. Since in robotic rehabilitation the user has to face repeated and 

unchallenging stimuli, deliverable 3.3 will analyse the user’s attention level in such kind of scenario. 

Outputs 


Name: Attention states decoding flag 

Description: Information about the subject’s attention level during a specific task 

The attention states decoding flag output provides a numerical value indicating the user’s 

level of attention. 






in detecting the attention states. 

Reason / Comments: Information about the performance in decoding attention states. 



It is expected that for some diseases the decoding accuracy related to cognitive processes will not be as high 

as in healthy subjects. 

62


7.3 Device Ontology Modelling (Task 4.1, UR) 

Ontology modelling for all the devices involved in the CORBYS system enables representation of the various 

sensors and actuators that the system uses and interacts with. This ontology contains information about 

properties and constraints of the devices which allow the robot to use the actuators as well as accurately 

interpret the information received from the sensors. The ontological representation of this information allows 

for reasoning to take place in case a specific device model is not readily available but relationships can be 

found. 



Inputs 

Requirement No. DMO1 

Name: Device identification and selection 

Description: Identification and selection of models of appropriate devices which the CORBYS systems 

should be able to use 

Reason / Comments: Selection takes place during specification stage WP2, in discussion with all partners 


Processing 

Requirement No.: DMO2 

Name: Semantics Extraction 

Description: The identified and selected list of devices to be studied and respective semantics to be 

extracted for integrating into the device ontology. Including information on scope, domain, 

spatio-temporal, Finite State Machine representations, functions, roles, responsibilities, 

attributions 




Name: Formulation of Device Ontology 

Description: The identified and selected list of devices, extracted respective semantics used to formulate 

the device ontology 



Output 


Name: Device ontology 

Description: Ontological representation of all selected devices used by the CORBYS systems 

Reason / Comments: To allow the system to effectively use actuators, accurately interpret sensor information and 

allow for reasoning to take place in case a specific device model is not readily available but 

relationships can be found. 



63



Name: Device state integration with Device State Integrator (DSI) 

Description: Device ontology to assist device state integration 



7.3.2 Assumptions and Dependencies 

� Modelling the device ontology depends on input from partners regarding selected sensors and 

actuators, other devices to be used in CORBYS 

7.4 SelfAwareness Realisation (Task 4.2, UR) 

Self-Awareness supported by Situation Assessment is an essential part of a cognitive system. It allows the 

system to interact with the environment in the most efficient and sensible way without the interference of an 

external guide. The CORBYS Situation Assessment Architecture facilitates (i) awareness of available sensors 

and actuators (via the device ontology) and (ii) awareness about the relationship of these devices to the 

environment including humans. CORBYS Situation Assessment Architecture includes a number of modules 

such as semantic integrators for person, device/object/entity, ontological layer for devices, domain specific 

(Plug and Play) ontologies and a blackboard structure acting as a globally accessible facility for situation 

assessment. 

A Device State Integrator (DSI) acts as the interface or interpreter between the sensors/actuators and the core 

system, using the information in the device ontology. DSI fuses all the information and makes it available to 

the core-architecture for further processing via the blackboard. 

Awareness about the relationship to the environment entails positioning, interaction possibilities with the 

environment, potential workflows, process flows, process maps, etc. This information is provided by an 

Object State Integrator (OSI) which analyses the received input information from sensor data and creates a 

cognitive image of the environment. 

A Person State Integrator (PSI) allows for the representation of the interacting person, including information 

such as position of the person, profile integration etc. 

The integrated Situation-Awareness Blackboard (SAWBB) in CORBYS allows access to designated deviceagents 

for relevant updates of profiling knowledge as well as the person’s current states, events, and 

behaviours detected by all the sensor and reasoning sub-systems. It serves to facilitate symbolic knowledge 

integration and inferencing at the higher semantic fusion, and enables appropriate dynamic data/knowledge 

sharing regarding the semantic parametric values of the situated operational context. 

SAWBB is a layered architecture supporting the sharing and integration of data at various levels of abstraction 

including raw data from sensors etc; the upper layer being in the form of an event handler/look up table and 

the second layer taking the form of a database to store historic and profiling information. SAWBB is a part of 

the high level Cognitive CORBYS control architecture; the connection of the Situation Assessment 

Architecture with the module responsible for perceiving, focusing, cognition, learning and responding to 

environment. An important part of this module is the internal representation of the robot internal self-model 

and self-state awareness which allow the system to reason and act based on its status, the context of assigned 

task and event anticipation. Two types of memory will be realised, i.e. one representing the current state of 

64


task execution (M1) and the second being the representation of the world model (M2). 

The Situation Assessment Architecture enables the sharing of raw sensor data from the Blackboard to support 

low level sensor data fusion amongst reasoning sub-systems; particularly to serve those modules within the 

Data Analysis Abstraction and Fusion Layer with variously designated Anticipation, Empowerment, 

Classification, Recognition responsibilities. Accordingly, the blackboard can receive data from any sensor 

configurations on the system e.g. the body area networks in the first demonstrator as well as from the mobile 

robotic system for examining hazardous areas. The output is the data and knowledge that can be made 

available at various levels of semantic abstraction for access as demanded by the agents. 



Inputs 

Requirement No. SAWR1 

Name: Sensor input to Situation Awareness Blackboard (SAWBB) 

Description: From various sensing and data captures device-agents within the system e.g. Person State 

Integrator etc. 




Name: Semantic input to Situation Awareness Blackboard (SAWBB) 

Description: For example, periodic profiling data refresh of the Situation-Awareness Blackboard (SABB) 

with semantically resolved structured profiling knowledge to re-update the operational 

profiling reference values of the Situation-Awareness Blackboard (SABB) for use by all 

relevant sub-systems. 




Name: Event handling request to Situation Awareness Blackboard (SAWBB) 

Description: Requests that flag to indicate that a module should take action (event-handling) 




Name: Sensor input to Person State Integrator (PSI) 

Description: Sensors and devices detecting and tracking person states, to provide input to the PSI so that 

an integrated structured state descriptor of the person at any given time, is sent to the 

SAWBB 




Name: Sensor input to Device State Integrator (DSI) 

Description: Device inputs i.e. from sensors/actuators. All input is asserted onto the SAWBB. 




Name: Sensor input to Object State Integrator (OSI) 

Description: Sensor Input are used to create a cognitive image of the environment, including relationship 

of an object to the environment, positioning, interaction possibilities with the environment, 

65


potential workflows, process flows, process maps etc. 



Processing 


Name: Person ontology modelling 

Description: To formulate relevant ontology deduced from knowledge of the person states. Including 

information on scope, domain, spatio-temporal, Finite State Machine representations, 

functions, roles, responsibilities, attributions. This is used to arrive at a structured person 

state description update which is provided to the Situation-Awareness Blackboard. 




Name: Person state integration 

Description: Integration of person states using person ontology. Combine different hypotheses from 

various sensor modalities to compute structured person state description. 




Name: Device state integration 

Description: Integration of device states using device ontology. 




Name: Object ontology modelling 

Description: To formulate relevant ontology deduced from knowledge of objects and their states likely to 

be encountered by robotic system. 




Name: Object state integration 

Description: Integration of object states to create a cognitive image of the environment. 




Name: Establish working memory structure for SAWBB 

Description: Establish the local and global (shared) dynamically update-able working memory structure. 




Name: Semantic fusion 

Description: Multi-level data fusion to ensure semantic integration to serve situation assessment (selfawareness) 

updates. Selection of appropriate fusion techniques used for the various state 

integrators. 



66


Output 

Requirement No.: SAWR14 

Name: Sensor output from Situation Awareness Blackboard (SAWBB) 

Description: Allow role-based secure read/write access to various sensing and data capture device-agents 

e.g. Person State Integrator etc to enable global and local data and knowledge exchange 

amongst relevant device-agents regarding the relevant states of person. 




Name: Semantic output from Situation Awareness Blackboard (SAWBB) 

Description: For example, periodic uploads to the profiling integration of new profiling related data 

published on the Situation-Awareness Blackboard (SABB), by sensors and detectors, during 

the operational cycle just ended. 




Name: Current context of SAWBB 

Description: The current context that is represented on the SAWBB 




Name: Person state descriptors by PSI 

Description: Semantic parametric descriptors (hypotheses with variable levels of confidence) at various 

levels of abstraction re person states. Integrated structured person state descriptors onto the 

Situation-Awareness Blackboard (SAWBB) 




Name: Device state descriptors by DSI 


levels of abstraction re device states. 




Name: Object state descriptors by OSI 


levels of abstraction re states of objects/entities in the environment 




� Same as Requirement “DMO5” 

� All modules interface with SAWBB 


� SAWR: including SAWBB, PSI, DSI, OSI 

o The operation of this module is automated. 

67


o Operating systems required/supported: MS Windows XP/Vista/7 


Internal 

� PSI depend on SAWBB 

� DSI depend on SAWBB 

� OSI depend on SAWBB 

� All other modules depend on SAWBB 

7.5 Robot response to a situation (Task 4.3, UR) 

For an effective control system, the behaviour is divided into reflexive and learned (reflective) behaviour. A 

hardware and software co-design will be used for this purpose. Reflexes, as in humans, are pre-compiled, i.e. 

they are automatic without the need for elaboration or thinking. In CORBYS, this will be realised via 

hardware implemented algorithms for obstacle avoidance, navigation, safety issues etc. Learned reflective 

behaviour on the other hand is constantly updated and adapted to the situation, this will be implemented in 

software. 

Hardware Reflex Capability will entail the implementation of invariant reflexes of the system such as 

avoidance of certain obstacles, using low level recognition algorithms realised in an FPGA to allow for fast 

prototyping and re-configurability. On the other hand, Software Learning Modules will realise the learning 

behaviour in software using machine learning for pattern discovery, pattern-directed inference and learning 

and refinement of the cognitive map. 



Inputs 

Requirement No. RRS1 

Name: Inputs and connections to the FPGA sub-system from the mechatronics sub-system 

Description: Any inputs directly feeding into the FPGA sub-system, signal conditioning, Analog to 

Digital conversion (ADC), data rate, data bus width, data format etc. 

Number and type of the sensors attached to the subject 




Name: Inputs to FPGA sub-system from various software modules 

Description: Such as the Situational Awareness modules etc. 




Name: Scalability to support inputs and connections from hardware transducers/devices 

Description: Within the scope and duration of the project, the FPGA sub-system should have enough I/O 

capability available for any future hardware transducers, their data rate, data bus width, data 

68


format, signal conditioning, Analog to Digital conversion etc. 




Name: Continuous monitoring 

Description: Any physical quantities or situations, that may have an impact on the power requirements 

and the architecture of the FPGA hardware sub-system, need to be monitored or may need to 

be polled continuously for any purpose and their criticality. Also special requirements such 

as signal conditioning, ADC need to be identified. 




Name: Event handling 

Description: Events that need to be taken care of during the normal execution and their criticality. Events 

that may have an impact on the power requirements and architecture of the FPGA hardware 

sub-system such as priority interrupts etc 



Processing 

Requirement No.: RRS6 

Name: Identification of reflexive capability to be enabled by the FPGA sub-system. 

Description: To identify the feasibility in terms of implementation in the FPGA Hardware. 




Name: Identification of ‘Navigation’ related reflexive behaviour enabled by the FPGA sub-system. 





Name: Identification of ‘Obstacle avoidance’ related reflexive behaviour enabled by the FPGA subsystem. 





Name: Identification of ‘Safety’ related reflexive behaviour enabled by the FPGA sub-system. 





Name: Identification of open and closed loop processes for the FPGA sub-system 

Description: 

Reason / Comments: Reason: 

For example, in a closed loop process, it may be required to implement a closed loop 

controller algorithm in an on board processor or in FPGA, so it is important to know 

beforehand. 

69




Name: Selection of off-the-shelf FPGA device 

Description: This depends on the above factors and the overall functional requirement from the FPGA 

sub-system 




Name: Selection/design of custom or off-the-shelf FPGA board for the selected FPGA device 


sub-system 




Name: Additional processing and memory devices for the FPGA board 


sub-system 




Name: List of states, conditions and transitions 

Description: This is derived from the functionality of the FPGA sub-system 




Name: Reconfiguration and reprogrammable FPGA sub-system 

Description: The implemented hardware sub-system and the FPGA board should be reprogrammable, 

reconfigurable and have enough processing capability to support any future requirements 

within the scope and duration of the project 




Name: Sensor input level 0 processing on the FPGA sub-system 

Description: If FPGA sub-system is situated between mechatronics and software/cognitive sub-systems, 

raw sensor data can be pre-processed in real-time before it is presented to the software 

modules of CORBYS. To be discussed with partners during specification stage 

Reason / Comments: Reason: 

Fast real-time processing sensor input 


Output 


Name: Outputs and connections to the FPGA sub-system from the mechatronics sub-system 

Description: Any outputs directly feeding into the mechatronics sub-system, signal conditioning, Analog 

to Digital conversion (ADC), data rate, data bus width, data format etc. 

Number and type of the sensors attached to the subject 



70



Name: Outputs to FPGA sub-system from various software modules 

Description: Such as the Situational Awareness modules etc. 




Name: Scalability to support outputs and connections from hardware transducers/devices 

Description: Within the scope and duration of the project, the FPGA sub-system should have enough I/O 

capability available for any future hardware transducers, their data rate, data bus width, data 

format, signal conditioning, Analog to Digital conversion etc. 




Name: Continuous stimulation 

Description: Any physical quantities or situations, that may have an impact on the power requirements 

and the architecture of the FPGA hardware sub-system, need to be stimulated continuously 

for any purpose and their criticality. Also special requirements such as signal conditioning, 

ADC need to be identified. 




Name: Event handling 

Description: Events that need to be taken care of during the normal execution and their criticality. Events 

that may have an impact on the power requirements and architecture of the FPGA hardware 

sub-system such as priority interrupts etc 



Dataflow 


Name: Data paths to/from FPGA sub-system 

Description: Identify any other data paths that may be required to be implemented in the FPGA subsystem. 




Name: Any data paths required VIA FPGA sub-system 

Description: This is when FPGA sub-system is also working as a middle-man between the software and 

mechatronics sub-system. 




User interfaces, hardware/software interfaces i.e. what the system must connect to. 


Name: FPGA sub-system interfacing 

Description: Any special requirements regarding interfacing with rest of the system as covered in inputs 

and outputs sections. 

71







Name: Latency and timing requirements for the input and output of the FPGA sub-system 

Description: This also depends on the functional requirements of the overall FPGA sub-system 




Name: Continuous run time 

Description: This depends on the functional requirements as well and also impacts on heat dissipation 

and power requirements of the FPGA sub-system. 




Name: Power consumption of the FPGA sub-system 

Description: This depends on the selection of the board and the hardware components. This depends on 

outcome of the requirements dealing with I/O, processing and device and component 

selection such as RRS1, RRS2, RRS3 etc. 





Name: Heat dissipation of FPGA and other device on board 

Description: This depends on many of the previous factors such as functionality required from the FPGA 

sub-system. An on chip fan or heatsink is typically required to dissipate the heat from the 

system. Vents in the demonstrator enclosures etc. Inadequate ventilation can cause thermal 

shutdown of the system. Furthermore, the user needs to be protected from exposure to 

excessive heat. 




Name: User safety. 

Description: To prevent shock or injury to the user. To prevent any imposing mechanical movement by 

the Mechatronic system (e.g. using a command master stop or reset). Isolation, physical 

connections. 




Name: Reliability of the implemented hardware 

Description: The implemented digital hardware and firmware must be verified and validated properly by 

using functional simulation etc methods. 



72



Name: Testing of the FPGA sub-system 

Description: The implemented hardware and firmware together with board testing must be carried out to 

make sure the system is safe to run. 




Name: Battery safety and reliability for the FPGA sub-system 

Description: Capacity and size of the battery; whether separate battery for FPGA sub-system or only one 

battery. Type of the battery in light of the scenario context in which the system is used 




Name: Hardware redundancy within the FPGA sub-system 

Description: To make the system fail-safe, functionality to be implemented in more than one hardware 

component on the same board or on a different board. 




Name: System State, steps and recovery in case of emergency or malfunction 

Description: Identify the state the system should be in, in the case of emergency and/or malfunction. The 

steps that should be performed by the FPGA sub-system at its level to aid the recovery of the 

malfunction. 




Name: Alerts for emergencies or malfunctions 

Description: The alerts to be generated, the format in which to raise alerts, signal etc in case of 

emergency or malfunctions. Type of alert, i.e. visual or audio or text etc. 




Name: Mutually exclusive states, scenarios, unsafe or hazardous conditions/states of FPGA subsystem 

Description: These conditions or states should be avoided and should ideally never occur during the 

normal functioning of the system. 




Name: Synchronous vs. Non-synchronous execution of various functionalities 

Description: If there is any special requirement for the above 




Name: Diagnostic and self-test functionality 

Description: It is important to know if the system is ok at the start up. Also it is important to identify 

what is wrong with the system when there is a malfunction or emergency situation. 



73


7.5.2.3 Other 


Name: Requirements for size/shape/dimensions/weight of the FPGA board 

Description: It is important to know if there are any restrictions to these parameters. 




Name: Requirements for the placement of the FPGA board 

Description: Where, what location in the demonstrator the FPGA sub-system board should be placed 




Name: Cost requirements of the FPGA sub-system 

Description: Cost requirements of the FPGA sub-system while purchasing components and boards 




Name: Certification 

Description: To get the sub-system certified for safety and usability 




Internal 

� Depends on SAWBB 

� Depends on all modules 

7.6 SOIAA: SelfOrganizing Informational Anticipatory Architecture (Tasks 4.4, 

5.1, 5.2, 5.3, 5.4; UH) 


� Expectation as a specification of an anticipated outcome (TASK 4.4, UH) 

� Development of self-motivated movement and goal generator for the cognitive architecture (TASK 

5.1, UH) 

� Models and algorithms for the identification and anticipation of human purposeful behaviour (TASK 

5.2, UH) 

� Algorithms for measurement of anticipatory information flow between robot and human and vice 

versa (TASK 5.3, UH) 

� Development of framework for transitional dynamics between robot-initiated and human-initiated 

behaviours (TASK 5.4, UH) 

74


7.6.1 Preamble: Interface with the project 

7.6.1.1 Research Scenario Considerations 

From the perspective of the SOIAA component whose development is in the responsibility of UH, and in the 

understanding of UH of the documents provided by IRSZR there appear to be at least two main “axes” 

spanning the space of modes of operation to which SOIAA needs to be exposed. 

The first is a “follow me” mode, where the controller strives to ensure safe interaction between the human and 

the robot. The robot would then practically “walk along” with the human, and should neither hinder nor assist 

its behaviour. This could then be used as a zero-hypothesis “status quo” from which the human’s task could 

start to be influenced. 

In general, the second mode of operation in either demonstrator involves outlining how the robot will interact 

and anticipate the needs of the human. For example, with regards to the first demonstrator, the second mode of 

operation would strive to fully impose a desirable (e.g. from the perspective of therapy) walking pattern on the 

patient. The needed movement would in this case be estimated from the current torque supplied by the sensors 

in the exoskeleton’s motors. This would basically force the human subject to move in the specified way, 

where the applied force would be steadily reduced (under the supervision of a therapist) as the human starts to 

walk normally on his own. 

From the perspective of the development of the SOIAA component, one of the main scientific challenges is 

the formalisation of the different modes of operation (such as the “follow me” mode, or the “human-robot 

interaction” - mode) in terms of the universal approach of the SOIAA component described in the next 

section. The various modes of operation should ideally also allow inclusion of established human movement 

models, such as for instance the minimum jerk principle or the 2/3 power law (Viviani and Flash 1995) in a 

form accessible to the SOIAA framework to be smoothly integrated into the cognitive process. 

With respect to the cognitive requirements in regard to the second demonstrator, the exploration scenario, it is 

important to establish the extent to which the CORBYS architecture should be generalisable to that scenario. 

This includes clarification of the range to which the CORBYS algorithms and techniques will work without 

change for both robots, and possibly for other cognitive robots, and to which extent the development will be 

specifically aimed to work for a particular demonstrator. In this context, a discussion of specific cognitive 

requirements for each demonstrator inside the CORBYS framework is particularly vital. 

7.6.1.2 Issues for SOIAA Interface 

To clarify the interface of the cognitive UH SOIAA component inside the CORBYS architecture, it is 

important to determine the level at which the cognitive component provided by UH will interface with the 

overall project. With regards to the first demonstrator, this can be summarised in the following key question: 

“How much cognition is involved in walking?” 

From the perspective relevant to the SOIAA architecture, the cognitive process of walking is not high-level as 

would be if every action taken and every muscle moved involved processing complex semantic 

representations. In particular, human movement typically does not involve a conscious process of 

contemplation such as e.g. characterised by the following: “To get 10 centimetres ahead I am going to lift my 

leg and move it slightly forward so as a result my body will then tilt slightly forward and gain some forward 

momentum” or some other kind of precise rule-based system. Rather, the approach espoused by SOIAA is 

based on optimisation principles which express the balances of cognitive load in terms of informational 

quantities. In this view, the control of a quasi-automatic behaviour (such as straight walking) would be 

reflected in limited cognitive processing load which, could be furthermore treated as largely detached from 

higher cognitive levels and would only be modulated by them to a limited degree. The key insight is that, 

according to these principles, we expect that the cognitive information processing can be modelled on these 

75


different levels using the same way. Thus, conceptually, the SOIAA architecture does not have to distinguish 

a priori on which level it interacts with the CORBYS sensorimotor command loop and is not limited to one 

particular level of control. 

In the spirit of ensuring the timeliness and novelty of the cognitive research component of the CORBYS 

project, the present proposal for the interface requirements will emphasise that the interface supports the 

universality of the proposed cognitive solution in that it will not limit a priori (with exception of aspects of 

safety) the level at which SOIAA interacts with the CORBYS sensorimotor command loop. However, this 

does not preclude that clear preferences be formulated by the CORBYS Consortium which cognitive levels 

(e.g. high-level) should be addressed with priority. 



Inputs 

Requirement No. SOIAA1 

Name: Sensorimotor data from the robot platforms 

Description: To develop and train a first prototype of the SOIAA algorithm UH requires a large data set 

of Sensorimotor data from the robot platforms. This data set should comprise timestamped 

low level sensor values, regarding various robot components or other available information, 

such as from the BCI (in the case of the first demonstrator), and would include, but would 

not be limited to: 

� positions 

� orientations 

� velocities 

� accelerations 

� forces 

� absolute positions/orientations if available 

� proprioceptive information about actuator activity 

� BCI channel information 

� (semantic) tags of events; especially to be used for debugging 

Reason / Comments: Any available sensor data should be accessible by SOIAA (and it should be possible for 

SOIAA to select between raw and, where applicable, suitably pre-processed data). The 

format it is provided in should foreshadow the interface which will later be used for the 

CORBYS sensorimotor command loop to interact with the SOIAA algorithm in the robot. 



Name: Documentation Sensorimotor data from the robot platforms 

Description: The sensor motor data should be accompanied by a document relating parameter values to 

the respective sensors and actuators, detailing the type of the parameter, its minimal and 

maximal values, and other relevant information. Furthermore, it should identify which of the 

provided parameters would be available on the onboard robot system prototype, and which 

of them are supplementary “lab-only” data which will only be available as test and training 

data (e.g. hand-created semantic tags, absolute positions determined with sensors outside of 

the robot, etc.). 




76


Name: Parameter Space Specifications 

Description: For SOIAA to take full advantage of the information-theoretic methodology it is based on, it 

would be highly desirable for SOIAA to have additional information available about all the 

parameters and data points provided by the CORBYS sensorimotor command loop. This 

should address questions such as, but should not be limited to: 

� Is the parameter continuous or discrete? 

� If discrete, what are the possible state transitions? What is the topology of the 

parameter space? 

� If continuous, what are the extreme values the variable can assume; what is the 

manifold structure of the parameter space? 

� What is the resolution of the physical sensor measuring the continuous variable? 

� What is the value measured in (meter, degrees, etc)? 

Reason / Comments: To apply the information-theoretic methodology to full effect, it is necessary that 

quantitative, low-level data is made available to the SOIAA component. 



Name: Structural Information regarding Parameter Space 

Description: 

- An operational structure formalising the effects of and constraints on actions (outputs) on 

the values of the overall system, formalised in mathematical terms, such as e.g. semigroup 

or Lie group action, invariances, and similar; 

- An interventional structure indicating which variables are causally subject to change if a 

certain action (output) variable is changed. One form in which this could be expressed is as a 

Causal Bayesian Network on the variable space. 

If such explicit structures are not available a priori, weaker substitutes which can be derived 

from empirical data could be used instead, among others e.g.: 

� Laplacian models of data (Kondor and Lafferty 2002) 

� graphlet library (e.g. Kondor, Shervashidze, Borgwardt 2009) 

Reason / Comments: It would be an advantage if the SOIAA component had access to further information about 

the structure of parameter space. 


Output 


Name: Self-motivated movement and goal generator 

Description: Variations of Klyubin’s (2004, 2007) Empowerment methodology are used to identify both, 

salient, high empowered states and trajectories in the state space of the human-robot system. 

Reason / Comments: Discussion with the relevant partners is needed to determine the exact specifications of this 

state space. 



Name: Models and algorithms for the identification and anticipation of human purposeful behaviour 

Description: The “Relevant Goal Information” approach, a specific kind of conditional Bayesian 

modelling, will be used to identify the most likely candidates for sub goals from the salient, 

high empowered output states of SOIAA5. 

Reason / Comments: This module is dependent on SOIAA5 

77




Name: Measurement of anticipatory information flow between robot and human and vice versa 

Description: 


Different formalisms for “information flow” are to be used to get a quantitative 

measurement indicating the causal dominant partner in the symbiotic human-robot relation. 



Name: Development of framework for transitional dynamics between robot-initiated and humaninitiated 

behaviour 

Description: The different quantitative outputs of SOIAA8 are to be combined with further data to 

construct a regulatory feedback control system that transitions the symbiotic human-robot 

relationship into one where the human has maximal causal control. 

Reason / Comments: Dependent on SOIAA7 




Name: Interactive Model of Robotic System 

Description: For such a simulation model UH sees the following options: 

� a simulation model is provided by the robotics/human dynamics partners, e.g. if 

used in their own work. 

This option is highly desirable, even understanding that the model falls short of a fully 

accurate simulation of human motion, to capture the essential kinematic and dynamic 

properties of the system, as seen by the partners with the robotic/therapeutic expertise. 

� if option 1 is not feasible, UH will, as a fall-back option, have to resort to build 

such a model on its own. This will necessarily be only a coarse approximation of 

the human/robot system due to the lack of relevant system expertise on the side of 

UH, but is necessary to ensure practicable progress on the SOIAA architecture 

development while the prototype of the CORBYS sensorimotor command loop is 

being developed for the hardware robots. 

The UH requirements for the simulation model, whether built by the relevant partners 

(option 1) or UH (option 2), needs to have the abilities described in the following section. If 

it turns out to be necessary to resort to option 2, it would be mandatory for UH to have the 

information required to build the simulation model. (see Requirement SOIAA11 and 

SOIAA12) 

Reason / Comments: To allow a large number of runs under controlled conditions and allowing a flexible 

application and test of interventions by the SOIAA component, a simulation model of the 

human-robot system is strictly necessary. This model needs to provide an input/output 

interface of the sensorimotor command cycle of the model to the SOIAA component in 

accordance with the above requirements. 



Name: Functional Dynamics 

Description: The simulation system needs to have the following properties: 

78


� it provides a functional model of the robot-human system 

� it includes the dynamics of robot, sensor, motors 

� with and without safety filter (provided by the relevant partners) 

� it includes a dynamic model of human walkers (in the case of the first 

demonstrator) 

Regarding the human model, a pragmatic approximation is sufficient. It is not expected or 

necessary to implement a high-accuracy model of human walking. A pragmatic model that 

is available early and can be used effectively is preferable to a high-quality model which is 

difficult to tune and use and is available late. 

Reason / Comments: This information is only needed, if UH has to build a functional model themselves. 

Indicative priority Mandatory (conditional on SOIAA10) 


Name: Interventional Dynamics 

Description: The model should provide an interface that would allow the SOIAA component to also 

directly influence the low level dynamics of the robotic system, such as joint controls, 

motors, etc, in addition to providing general behaviour cues (all, however, filtered through 

the protection of the safety unit). For testing purposes, it would also be highly desirable for 

the simulation model if world states could be directly influenced. 

Reason / Comments: This information is only needed, if UH has to build a functional model themselves. 

Indicative priority Mandatory (conditional on SOIAA10) 


Name: Target Specifications 

Description: To work towards the overall project goals the SOIAA component will require a specification 

of the target behaviour of the robot-human system which suits the information-theoretic 

SOIAA framework. In particular, with regards to the first demonstrator this requires a 

clarification about how the therapeutic behaviour is to be integrated, and how teaching hints 

that the robot needs to give should be expressed in a form suitable to the SOIAA cognitive 

framework. Example behaviours could include: 

1. toe walking 

a. CORBYS should restore: target angles; 

2. crouch gait 

a. CORBYS should restore: target angles 

3. stiff knee 

a. insecure weight transfer 

b. should restore knee angles 

4. stiff knee with circumduction 

a. insecure weight transfer 

b. excessive pelvic vertical movement 

c. should restore angles 

In general, it is necessary to communicate to the SOIAA component in a suitable format as 

to what exactly should happen to assist the human in their task. 

To approach and phrase this in a more specific way (in the case of the first demonstrator), 

the following questions might be posed: 

� Is the insecure weight transfer a measurable effect? 

� If so, how could it be measured practically? 

79





� What are the desired target angles and in how far are they universally definable? 

More generally, the following questions might be posed: 

� How is safe movement specified? 

� How is the robot supposed to give teaching hints? 

� How is a “normal” fixed move specified? 

� Can they be modulated, meaning that we could interpolate between the current and 

the target behaviour? 

� Intention extraction: What kind of high level behaviour should SOIAA identify? 


Name: Testing Specifications 

Description: In later stages of the project it will also be necessary to integrate and test the SOIAA module 

with the actual robotic system, including tests with human subjects. For this, a safety 

filtering interface from our partners will be mandatory, which will ensure the safety of the 

human participant and the robot under commands initiated by the SOIAA component, as the 

SOIAA component, while striving to reproduce “natural behaviour” will not contain explicit 

safety provisions. At that point, UH will also require (desirable) sufficient computational 

power to run the algorithm in online mode, to allow a system able to actively intervene in 

the synergic robot-human movement. As a fall-back option for this case if the computational 

power is not available, UH will consider constructing simplified and computationally 

cheaper proxy quantities that would be replacing the full-fledged information-theoretic 

concepts in the online settings. 




Name: Hardware Specifications 

Description: In regard to hardware requirements, the actual CPU and memory requirements of SOIAA 

will be determined during the project, but in general UH strives to create an algorithm that 

has (in decreasing order of priority): 

� online capability 

� onboard capability 

� (desirable) online learning 

Since the information-theoretic algorithms proposed for SOIAA are rather CPU intensive, 

and UH will be looking into methods to create more efficient and possible more light-weight 

versions of the algorithms for this purpose. In general, however, UH expects a need for 

substantial computational power, and once UH will have acquired more experience with the 

specific target domain our partners will be updated with more specific requirements. 



80


7.7 User Responsive Learning and Adaptation Framework (Task 5.5, UR) 

The User Responsive Learning and Adaptation Framework allows for a stable maintained and goal-oriented 

performance continuum in the context of human-robot interaction. This framework enables the creation, 

capturing and storage and thus the data intelligence acquisition of user preferences related to control and 

intervention. Using a canonical set of control parametrics in relation to the variable contexts of the integrative 

human-robot-system (e.g. context elements such as gait descriptors, stability, directionality, admissible and 

inadmissible postures etc.), this framework establishes the semantic parameters necessary for the 

representation of various states of the CORBYS systems in dynamically evolving contexts of states and 

responsive control and/or interventions from either actor (human or robot). 



Inputs 

Requirement No. URLAF1 

Name: Identification of control parametrics 

Description: A canonical set of control parametrics in relation to the variable contexts of the integrative 

human-robot-system 



Requirement No. URLAF2 

Name: User control and intervention preferences 

Description: Creation, capture and storage and thus the data intelligence acquisition of user preferences 

related to control and intervention 



Processing 

Requirement No.: URLAF3 

Name: Offline learning (Reflective capability) 

Description: Offline learning whereby the systems learns to adapt the level and type of intervention and 

control in a particular context responsive to the cumulative data intelligence relation to userpreferences 

as acquired during runtime interaction with the user. This constitutes the 

reflective learning capabilities of the system 




Name: Runtime responsive adaptation 

Description: The system adjusting the level of intervention and control to user’s preferences dynamically 

in response to user-indications. This response will be available rapidly in a pre-compiled, 

reflexive fashion? 




Name: Runtime responsive adaptation 

Description: The response is made available rapidly in a pre-compiled, reflexive fashion i.e. the reflective 

81


learning outcomes at one point become part of the reflexive capability 



Output 


Name: Output of Runtime responsive adaptation 

Description: Recommendations on adjusting the level of intervention and control by the system to user’s 

preferences dynamically in response to user-indications 





Name: Data acquisition from SAWBB 

Description: URLAF needs to interface with SAWBB for information about user preferences, profiling 

etc. 




Name: Adaptive recommendation to FPGA reflexive capability 

Description: URLAF needs to interface with FPGA to provide outcomes of reflective capability i.e. 

recommendations on adaptation as informed by the learning, so that this becomes part of the 

reflexive capability of the system 






Name: Latency 

Description: The reflexive capability enabled adaptive response recommendation needs to be 

communicated by the system in a timely fashion 



7.8 Cognitive adaptation of low level controllers (Task 5.6, UB) 


� Cognitive adaptation of low level controllers (Task 5.6, UB) 

The goal of research is short-term and long-term adaptation of control parameters of real-time controllers 

based on online measurement of interaction of the robot with its environment. Adapting control parameters 

has for an outcome the change in the robot behaviour during the interaction with the environment. 

82




Inputs: 

Requirement No. CAL1 

Name: Self-awareness modules 





Name: SOIAA architecture 





Name: Low level control algorithms 




Outputs: 


Name: Adaptation sub-system layer 

Description: The layer that will be interface between low-level control modules and cognitive modules. 

The module should take care of real time requirements of low level control modules. 



Processing: 


Name: Adaptation of control parameters 

Description: In the CORBYS rehabilitation system adaptation of control parameters has the advantage 

that assistance of the system to the user can be automatically tuned to the patient’s changing 

needs in the short term or long term period. Moreover, since several different control laws 

will be used for controlling the different robotic sub-systems of the CORBYS demonstrator, 

the transition in between the different control laws based on anticipation of the environment 

will be researched. 



Data flow: 


Name: Adaptation layer data flow 

Description: The sub-system is intermediate layer between real-time control modules and cognitive 

83


control higher levels, the information that the system retrieves comes from both directions. 





Name: Adaptive parameters 

Description: From real-time control modules the sub-system will retrieve information about the status of 

the control units and their performance while from higher level cognitive control modules it 

will retrieve information about the interpreted sensory data, control requests, desired control 

behaviour and etc. 





Name: Real-time environment 

Description: Since the module has to be able to provide real-time update of low-level controllers with 

control parameters, referent values, actions and others it is necessary that the module 

operates in real time. A real time framework such as OROCOS or similar will be used. 





Name: Real time hardware 

Description: In order to achieve real-time functionality. 






Name: Hard real-time operation 

Description: Hard real time operation will be tested first in laboratory environment and later in evaluation 

phase. 





Name: Safety checking of control parameters 

Description: The layer will feed in control parameters to low-level control modules and therefore a safety 

84


checking of control values has to be implemented. This will be agreed with partners that are 

involved in this task. 



8 System Integration and Functional Testing (WP8, SINTEF) 

� Sub-System Conformance Testing (TASK 8.1, SCHUNK) 

� Integration of Sub-Systems (TASK 8.2, SINTEF) 

� Conformance Testing of integrated System (TASK 8.3, SINTEF) 

Work Package 8 deals with the integration of CORBYS hardware and software subsystem components into a 

complete, operational system that can eventually be evaluated in clinical trails in WP 9. System integration is 

the process of bringing together subsystem functionalities and ensuring that they work together as a system. 

A prerequisite in order to obtain a working system is obviously that the intended functionalities of all 

subsystem components have been successfully realised and verified (tested) before the integration with other 

subsystems takes place. Without such subsystem testing, many errors in the complete system must be 

anticipated: The origin can be hard to identify in a complex system, the debugging can be intricate, and also 

the subsystem responsibility easily becomes fragmented among the contributing partners. 

The Work Package is therefore defined to contain three main activities (tasks). First, it is verified through 

testing in Task 8.1 that the sub-system components have been developed according to the target specifications. 

Only sub-system components that are deemed to conform to the targets are then cleared for the next stage, the 

integration of sub-systems into an integrated system (Task 8.2). Finally, when the system has become as 

complete as time and the supply of sub-systems allows, the integrated system will be subject to system level 

conformance testing in Task 8.3 to verify the overall system functionality with respect to target specifications. 

8.1 Conformance testing on subsystem and system level 


� Sub-System Conformance Testing (TASK 8.1, SCHUNK) 

� Conformance Testing of integrated System (TASK 8.3, SINTEF) 



Inputs 

Requirement No. CTREF1 

Name: Functional target specification of all sub-system components 

Description: Functional target specifications will be established, and will define metrics and procedures 

to be used in the system integration stage 




Name: Documentation of all sub-system components 

Description: The documentation required in order to integrate sub-system components into a complete 

system, such as: 

85


� Functional specification 

� Mechanical design drawings 

� User manual 

� Source code 

� Interface definitions 

� Installation guidelines 




Name: Hand-over of components from sub-system developers to system integrators 

Description: The hardware and software components must be made available for the system integration 

partners 

Reason / Comments: Must plan for a component hand-over to WP 8 partner. Typically, this can be a visit of the 

sub-component partner to perform the installation and to verify that the system works as 

expected in the new environment 


Requirement No. CONF1 

Name: Definition of test parameters and test rig setups. 

Description: The finalised robotic system including mobile platform and the powered orthosis 

demonstrator need to be tested according to the specifications. A scenario and use case 

profile with a specialised test rig design is to be detailed. 



Processing 

Requirement No.: CTREF4 

Name: Conformance testing requirements must be established 

Description: Predefined test methodologies based on predefined test metrics must be established. 




Name: Identification of test metrics and procedures for conformance testing 

Description: Shared metrics for approving that a system or sub-system conforms to target specifications 

must be established. Typically, these standards will be defined between the partners 

handing over and receiving sub-systems, and will vary and be adapted to the nature of the 

system or sub-system in question. 




Name: Conformance testing parameters – minimum requirements for all testing 

Description: Test plans shall as a minimum contain the following test items: 

� Definition of test environment (site) and test personnel (partners) 

� Conformance to interface definitions 

� Conformance to functional requirements 

� Conformance to performance related requirements (these might preferably be 

carried out by the sub-system developer before the integration stages) 

� Conformance to safety/ hasard requirements 

� Conformance to documentation requirements 


86




Name: Conformance testing test protocol design 

Description: Test plans shall have unique definition of test objects (physical components shall be 

uniquely marked and software shall have correct version numbering). It shall further 

contain information about test site, test date and test personnel 

The test protocol when feasible will be designed with the following information for each test 

item: 

� Unique test item number 

� Description of test activity 

� Description of expected test result (which should be in accordance with target 

specifications) 

� Check box field for entering test result with the following alternatives: 

Passed/Failed 

� Field for entering test observation (in particular observations when the “Failed” box 

was checked. 



Requirement No.: CONF2 

Name: 

Mechanical and electrical design of necessary test rigs. 

Description: Rigs and setups to test actuators of orthosis through and mobile manipulator need 

mechanical design, bill of materials, electrical design and software structure. 




Name: Production of test rigs 

Description: Manufacturing and assembly, cabling, functional test, software design for test procedures. 




Name: Sensor verification 

Description: Sensor function will be verified with third party measurement techniques and third party 

sensors. Options for calibration and necessary maintenance intervals will be presented. 



Output 


Name: Sub-system test reports from conformance testing 

Description: For CORBYS sub-systems, the results of conformance testing will be stored as project 

internal development documentation. 




Name: Data sheet and measurement/test protocols 

Description: Conformance with requirements and specifications has been proven with a final data sheet 

87


and test protocols 





Name: Evaluation of sub-system conformance testing outcomes 

Description: For sub-system conformance testing, the outcomes will be determined as a result of 

conformance testing involving the partners delivering (technology development work 

packages) and receiving technology (system integrators) 




Name: Sub-system components not meeting all conformance testing requirements 

Description: It must be anticipated that not all sub-systems will meet all conformance test requirements. 

In these cases, there will have to be a discussion about the severity of the failed items, the 

possibility to correct these items at a later stage (during integration stages), and the overall 

system consequences of not integrating the component in question. 




Name: Evaluation of integrated system conformance testing outcomes 

Description: For integrated system conformance testing, the outcomes will be determined as a result of 

conformance testing involving WP 8 (system integrators) on one side and WP 9 (clinical 

evaluation) partners 





Name: Sub-system responsibility 

Description: The sub-system functionality and performance is the responsibility of the partner(s) 

developing the sub-system during all testing and integration stages. 

Reason / Comments: After the integration stage, the responsibility will have to be shared with all partners, but all 

partners have responsibility to offer and provide guidance, service and other assistance 

during the integration and evaluation stages 




Name: CORBYS gait system target operating environment 

Description: The CORBYS gait rehabilitation system will be designed to work in typical health 

institution environments. 

Reason / Comments: The CORBYS gait rehabilitation system will not be suitable for outdoor all-weather usage. 



Name: CORBYS gait system target operating mode (ambulatory/not ambulatory) 

88


Description: The CORBYS gait rehabilitation system will be designed to be ambulatory, e.g., with 

wheels, moving around during gait rehabilitation therapy. 





Name: Sub-system conformance testing equipment 

Description: Special, dedicated test equipment, jigs et cetera required to perform conformance testing are 

supplied by the developing partner of the sub-system 






Name: Sub-system conformance testing of performance 

Description: Performance testing is not the main concern of conformance testing, but should have been 

carried out by developers of the sub-system prior to integration. 

Some limited performance testing should however be included in the conformance testing to 

verify functionality (such as to verify that an increase in control system load gives the 

expected feedback) 





Name: Sub-system evaluation of user hasards and safety 

Description: User safety is a prime concern for all medical devices. During conformance testing and in 

integration stages, hasards and safety concerns of all sub-systems shall be evaluated as part 

of the sub-system hand-over to integration partners. Hasards and safety aspects shall be 

considered for all CORBYS users - both patients and professionals. 




Name: Sub-system evaluation of reliability 

Description: The developer of the sub-system is responsible for ensuring that their sub-system 

components operate reliably. The developer’s effort to ensure this should be discussed 

during conformance testing. 



89



Internal 

� Depends on CORBYS Functional Specification 

� Depends on CORBYS overall output 

8.2 Integration of SubSystems 


� Integration of Sub-Systems (TASK 8.2, SINTEF) 



Inputs 

Requirement No. SIREF1 

Name: Sub-systems to be integrated must successfully have passed conformance testing 

Description: Only sub-systems that meet conformance test requirements will be integrated into the 

system 



Requirement No. SIREF2 

Name: Sub-systems to be integrated must be accompanied by sufficient documentation 

Description: The documentation required in order to integrate sub-system components into a complete 

system, such as: 

� Functional specification 

� Mechanical design drawings 

� User manual 

� Source code 

� Interface definitions 

� Installation guidelines 

Reason / Comments: Same as CTREF6 


Requirement No. SIREF3 (same as CTREF7) 

Name: Hand-over of components from sub-system developers to system integrators 

Description: The hardware and software components must be made available for the system integration 

partners 

Reason / Comments: Must plan for a component hand-over to WP 8 partner. Typically, this can be a visit of the 

sub-component partner to perform the installation and to verify that the system works as 

expected in the new environment 


Processing 

Requirement No.: SIREF4 

Name: System integration site 

90


Description: The physical location where the system integration activity takes place must be decided. 

Reason / Comments: Alternatively, at least a plan for moving the system from one site to other must be fixed. 



Name: System integration process for installing new sub-components 

Description: New sub-system components cleared for system integration will be installed in a joint effort 

of the sub-system developer and the system integrator. 

Reason / Comments: From case to case, it will have to be decided what is the most practical. Examples of joint 

efforts can range from remote installation of software or physical face to face meetings to 

install physical components. 



Name: System integration functionality testing should be carried out when new components have 

been installed 

Description: Frequent testing of the system in the integration stages is important in order to verify that 

new components do not give rise to problems in or conflicts with other sub-systems 




Name: Fine-tuning and optimisation of functionality of the integrated system and Sub-system to 

sub-system optimisation 

Description: Many system features can only be implemented and tested when integrated as a complete 

system. This will be a combined effort of the system integrators and the developers of the 

sub-systems. System integrators must ensure that time and access are given for sub-system 

developers to meet and improve the combined, integrated system 




Name: Physical design process of integrated system 

Description: Physical design and construction shall be based on dimensionally correct models developed 

using 3D CAD tools such as SolidWorks or ProEngineeer. 

For moving components, the system shall be evaluated/inspected at all end points to detect 

potential physical conflicts 

Reason / Comments: Discuss with other partners which CAD system and exchange format to use 


Output 


Name: System integration final output 

Description: The System Integration activity shall have as an output an integrated system that passes the 

final conformance tests of Task 8.3 





Name: System integration status and issues update 

Description: Status and issues in integration shall be made available to all project partners by 

91


documentation of plans, timelines and issues stored on the CORBYS file sharing system 

(WebDAV) 




Requirement No.: SIREF11 (confer CTREF12) 

Name: Sub-system responsibility 

Description: The sub-system functionality and performance is the responsibility of the partner(s) 

developing the sub-system during all testing and integration stages. 

Reason / Comments: After the integration stage, the responsibility will have to be shared with all partners, but all 

partners have responsibility to offer and provide guidance, service and other assistance 

during the integration and evaluation stages 



Operation environment requirements correspond 1:1 to the requirements for conformance testing: CTREF13 

and CTREF14 


Name: Integrated system power resources 

Description: The Consortium need to decide whether a battery or a mains powered system will be 

targeted 





Name: Resource limitations encountered in the system integration stage 

Description: Resource limitations emerging in the integration stage (for example power or processing 

will have to be dealt with in a process involving the system integrators and the partners 

sharing the limited resource) 






Name: Integrated system performance testing 

Description: The integrated system will be subject to performance testing. These tests will partly be to 

done to verify that sub-components performs as planned, and partly to verify that system 

features of the integrated system performs. 

Performance test parameters will be derived/extracted from target system specifications 



92




Name: System evaluation of user hazards and safety 

Description: User safety is a prime concern for all medical devices. During integration stages, hasards and 

safety concerns of all sub-systems shall be evaluated on a continuous basis, as well as in 

dedicated risks hasard meetings. The purpose of these meetings is to identify potential risks 

and find strategies to mitigate them. Risks and hazards evaluation shall consider all CORBYS 

users – both patients and professionals 

Reason 

Comments: 

/ 



Name: System integration reliability assessment 

Description: The integrated system will not be subject of extensive reliability tests (such as accelerated 

wear tests, climate chamber testing, and repeatability testing of sensor/actuators). The system 

will however not be released until a situation deemed as stable with a low frequency of 

failures have been reached. 

Reason / An acceptable failure rate has to be decided, but as a minimum, failures should not be 

Comments: expected during the time span of a typical rehabilitation session. 


8.2.2.3 Other 


Name: Integrated system must be possible to transport in Europe 

Description: The integrated system will have to be moved between various locations in Europe. The 

assembly quality of all components must therefore be sufficient to allow safe transport using 

commercial freight services 

Reason / The integrated system will have to be moved for display at one or more conferences, as well 

Comments: as when it will be deployed in the evaluation effort in clinical environments 



Name: The integrated system must be properly insured and declared at customs when transported 

across borders 

Description: With substantial accumulated effort and component value follows the need for insurance. 

With the component value follows also the need to avoid taxes at each customs border. 

Reason 

Comments: 

/ 



Name: Number of gait rehabilitation devices to be developed 

Description: There will only be developed at least one complete one integrated system 

Reason 

Comments: 

/ 



Internal 

93




9 Evaluation (WP9, IRSZR) 

� Specification of evaluation methodology, benchmarking, metrics, procedures and ethical assurance 

(TASK 9.1, UR) 

� Training on CORBYS system (TASK 9.2, IRSZR) 

� Continuous assessment of the technology under the development (TASK 9.3, IRSZR) 

� Evaluation of the researched methods on autonomous robotic system (TASK 9.4, UB) 

� Evaluation and feedback to development (TASK 9.5, NRZ) 

9.1 Evaluation methodology, benchmarking, metrics, procedures and ethical 

assurance (Task 9.1, UR) 


� Specification of evaluation methodology, benchmarking, metrics, procedures and ethical assurance 

(Task 9.1) 



Inputs 

Requirement No. UIREF1 

Name: Functional Specification of CORBYS 

Description: This will inform the identification of metrics and procedures to be used in the evaluation 

methodology 

Reason / 

Comments: 


Requirement No. UIREF2 

Name: Overall output of CORBYS project 

Description: The outcomes of the project will be used in the identification of appropriate existing systems 

to be used as benchmarks 

Reason / 

Comments: 


Processing 

Requirement No.: UIREF3 

Name: Formulation of evaluation methodology 

Description: To follow established methodologies (e.g. UI-REF, Badii 2008) for identification of 

evaluation metrics and procedures i.e. requirements UIREF4, UIREF5 and UIREF6 

Reason / 

Comments: 

94




Name: Identification of benchmarks 

Description: For the purpose of the benchmarking exercise, the CORBYS system viewed generically as a 

meta-controller controlling semi-autonomous sub-systems can be specified with respect to the 

identified terms of reference so as to be able to conduct comparative studies with existing 

meta-controllers that can be identified as comparable with respect to the established terms of 

reference. Thus the work entails the establishment of meta-controller comparators so that the 

performance of the CORBYS controller can be evaluated and compared with other 

comparable meta-controllers within a formal benchmarking framework. 

Reason / 

Comments: 



Name: Identification of evaluation metrics and procedures 

Description: To identify the evaluation metrics and specify the evaluation procedures to be followed in 

CORBYS. 

Reason / 

Comments: 



Name: Ethical assurance 

Description: To identify the evaluation metrics and specify the evaluation procedures to be followed in 

CORBYS in the context of ethical assurance and user involvement. 

Reason / 

Comments: 


Output 


Name: Set of metrics, procedures, benchmarks, ethical assurance and evaluation methodology 

Description: Comprehensive list of identified evaluation metrics and procedures to be followed in 

CORBYS. For benchmarking, the terms of reference by referring to generic properties of the 

required control performance: 

1. Dynamicity: typical real-time response (e.g. in ms) 

2. Complexity of Control: Number of states of each sub-system and number of states of 

control transition dynamics between the two systems 

3. Required Control Envelope: Number of spatio-temporal dimensions of required 

control, e.g. linear motion, rotational motion, 2D, 3D, etc 

4. Delivered Performance Envelope: Dynamicity, complexity and dimensionality levels 

of the controller 





Name: Evaluation of CORBYS outcomes 

Description: The formulated evaluation methodology, identified benchmarks, metrics and procedures will 

be used in the Evaluation WP 

95





Internal 



9.2 Training on CORBYS system (Task 9.2, IRSZR) 

CORBYS demonstrator I prototype will be used and tested in real application environments by industrial 

partners. This process will feed refinement and improvement of the CORBYS system. Conclusions regarding 

further improvements will be derived by measuring all benefits achieved by the implemented solution. 

Training for staff, end-users, and evaluators will be carried out. 



Inputs 

Requirement No. TRAINING1 

Name: Overall output of CORBYS project 

Description: The outcomes of the project will be used for training purposes 



Processing 

Requirement No.: TRAINING2 

Name: Formulation of training strategies 

Description: Appropriate training strategies to be devised in collaboration with experts and the whole 

consortium, so that CORBYS solutions are used optimally 




Name: Training process 

Description: Training of personnel, staff, evaluators, end-users of industrial partners to use relevant 

CORBYS solutions in their application domains 



Output 


Name: Conclusions regarding refinement / further improvement of CORBYS solutions 

Description: Results of training and testing will be documented 

96




9.3 Continuous assessment of the technology under the development (Task 9.3, 

IRSZR) 

Developed technologies and algorithms will be tested for their feasibility, interaction with users and expected 

performance. This will be done by assessing various sub-components of the system with selected patients in 

form of clinical testing. Different pathologies (e.g., stroke) as selected in the requirements elicitation stage 

specific to the first demonstrator, will be considered. The results will provide feed back on ergonomic issues, 

feasibility of sensory and actuation subsystems as well as about the overall feasibility of the intelligence 

platform that makes decisions on the current stage and future directions of rehabilitation. 



Inputs 

Requirement No. CATD1 

Name: Overall output of CORBYS 

Description: Including subsystems and the overall system prototype 



Processing 

Requirement No.: CATD2 

Name: Phase I trials 

Description: Each specific aspect of the overall system e.g., powered orthosis, mobile platform, sensory 

system and cognitive control subsystems, will be tested with able-bodied individuals 

simulating specific gait conditions related to various identified pathologies. This is expected 

to form the core load of system evaluation, hence it would not be appropriate to overburden 

patients with this task that can not help in their rehabilitation. 




Name: Phase II trials 

Description: Once the subsystems function adequately, patients falling under the identified pathological 

categories will evaluate and experimentally validate functioning of the technology under 

development. Here the goal will not be to draw any general conclusions on the therapeutic 

efficiency of the proposed system but rather to provide sufficient proof-of-concept that may 

facilitate randomised controlled trials. Inclusion and exclusion criteria will be determined 

and a set of reliable clinical outcome measures related to balance and walking (10-meters 

walk, 6-minutes walking tests etc.) will be selected to determine the possible therapeutic 

effects. 

In addition, patients will also undergo the instrumented gait analysis in the kinesiology 

laboratory at IRSZR, helping to determine not only functional improvement but also any 

qualitative changes in the kinematic and kinetic features of a particular gait. 

Patients will negotiate various objects or obstacles within their path; this will allow testing 

97


of the cognitive capability in the system, which aids the patients with obstacle negotiation or 

avoidance. 



Output 


Name: Feedback on ergonomics and feasibility 

Description: Testing and validation will result in feedback for engineers to optimise the developed 

solutions in terms of feasibility and ergonomics 



9.4 Evaluation of the researched methods on the second demonstrator (Task 

9.4, UB) 

Task: The evaluation of the generalizability of the CORBYS cognitive techniques developed will be done on 

the second demonstrator representing the robotic system for investigation of contaminated / hazardous 

environments. 



Inputs 

Requirement No. EASD1 

Name: Detailed specification of the evaluation scenarios. 

Description: The robot working scenarios appropriate for evaluation of CORBYS cognitive modules have 

to be designed. It is to specify in details which aspects of cognitive modules will be evaluated 

with second demonstrator. 

Reason 

Comments: 

/ 



Name: Integration (wrapping) of modules of existing control architecture (e.g. robot arm control 

module) of the second demonstrator into CORBYS control architecture. 

Description: High level CORBYS cognitive control modules to be evaluated on existing robotic system for 

examining hazardous environments. 

Reason / Detailed strategy will be defined later. It depends on requirements of CORBYS control 

Comments: (software) architecture. To be discussed in WP6-Cognitive Control 



Name: Analysis of usability of existing sensors for CORBYS demonstration 

Description: Existing sensors (e.g. vision sensors) are to be analysed in the CORBYS context 

Reason / It is to be tested whether existing sensors can provide necessary information for cognitive 

Comments: modules. 


98


Processing 

Requirement No.: EASD4 

Name: Situation Awareness 

Description: Situation Awareness input to semi-autonomous (autonomous) robot control 

Reason / 

Comments: 



Name: Identification and anticipation of human co-worker purposeful behaviour 

Description: SOIAA input to semi-autonomous (autonomous) robot control for overtaking initiative when 

needed 

Reason / 

Comments: 



Name: Extension of existing sensor modules (e.g. vision) 

Description: Extension of functionalities of existing sensor modules such as vision module with 

functionality of human tracking and action recognition to be done by UB if needed. To be 

specify in details with partners 

Reason / 

Comments: 


Outputs 


Name: Evaluation of CORBYS cognitive control architecture on the existing robotic system 

Description: Demonstration of generality of CORBYS cognitive control architecture 

Reason 

Comments: 

/ Partial and full testing TBD 


9.5 Evaluation and feedback to development (Task 9.5, NRZ) 

This task will undertake evaluation of developed solutions and present conclusions and findings to feed back 

the process of developmental refinement, improvements and re-adjustments where necessary, with an 

evolutionary approach. 



Inputs 

Requirement No. EVAL1 

Name: Evaluation methodologies and strategies 

Description: Evaluation methodologies and strategies as set out in requirement UIREF3 

Reason / 

Comments: 


99



Name: Evaluation results of demonstrators I and II 

Description: Evaluation results of demonstrators I and II as output of tasks 9.3 and 9.4. 

Reason / 

Comments: 



Name: Evaluation of demonstrators I 

Description: Training and evaluation of CORBYS demonstrator I (trials with patients in NRZ). 

Reason / 

Comments: 


Processing 

Requirement No.: EVAL4 

Name: Evaluation of CORBYS solutions 

Description: Evaluation of technologies, algorithms, subsystems, components developed as part of 

CORBYS project 

Reason 

Comments: 

/ Evaluation of Corbys solutions 



Name: Evaluation of demonstrator I 

Description: Continuous assessment of the technology under development in UB and NRZ 

Reason 

Comments: 

/ 


Output 


Name: Results of evaluation process 

Description: The outcomes of the evaluation process will be documented, and used for refinement and 

improvement of technologies. 

Reason / 

Comments: 


100


StateoftheArt in Enabling Technologies 

From a CORBYS-specific technology needs viewpoint: 

10 StateoftheArt in Sensors and Perception (SINTEF) 

10.1 Introduction to sensors and perception 

10.1.1 Document Scope and Purpose 

The purpose of this section is to describe the current state-of-art with respect to sensing in order to understand 

the human in human-robot interfaces. The section does not take into account sensor principles directed 

towards interpreting signals from the brain. The brain-computer interface state-of-the-art is presented in 

section 15. 

10.2 Sensor principles for perception in humanrobot interaction 

This section reviews the state-of-the-art of sensor principles underlying the understanding of the human in 

human-robot interactions. The section has been grouped as follows according to the type of information 

extracted: 

� Physiological sensors, which provides information about human physiology 

� Inertial measurement units, which provides inertia information for specific parts of the human body 

� Mechanical sensors, which provides information about contact forces, force distribution (pressure) 

and information about absolute and angular positions of specific body parts. 

� Environmental sensors, which measures the (physical) environment of the human-robot system - 

such as collision avoidance sensors 

The human-robot interface in a gait rehabilitation system has a number of characteristics that need to be 

considered when analysing the human sensing concepts state-of-art and its relevance in the CORBYS setting: 

� Sensors should be safe to use. The sensor readouts will enter into a robotics control system. It is 

therefore crucial that the sensor principles are capable of reliably detecting correct values, or 

alternatively, detect failure situations so that erroneous sensor readings do not cause harmful robotic 

actions. Further, the sensors should not cause other health or safety concerns during the time the 

patient uses the CORBYS system. In this respect, it is anticipated that the CORBYS system will be 

used in rehabilitation training sequences lasting from several minutes to a few hours, and that the 

sensor principles should be compatible with safe and harmless operation during this time frame. 

� CORBYS patients are monitored while doing physical activities of variable intensities. Sensor 

concepts that have the potential of being used while the patient is carrying out physical activities are 

therefore necessary. Many clinical physiological measurement systems are based on evaluating the 

patient while they are resting, for example while sitting down or lying in a bed. Physical activity 

easily introduces movement artefacts and noise from for example muscle activation which lower the 

accuracy of measurements. 

� The benefits of using the CORBYS system should not be outweighed by the hassle of using it. The 

two main users of the CORBYS system are the rehabilitation patient and the rehabilitation therapist. 

From these two user perspectives it should be easy and quick to install the system (for the lack of a 

better word) and configure the patient in the gait rehabilitation system. It is therefore of interest to 

lower the number of interfaces (such as patches and cables) that must be attached to the patient, in 

particular components that must interface the patient’s body directly. From the therapist’s perspective, 

it should also be pointed out that the need for a considerable number of sensors attached at multiple 

locations on the body adds to the risk of placing sensors in wrong locations, such as mixing the left 

101


and right side of the body. In the case of CORBYS, it is therefore of interest to consider ways of i) 

effectively combining multiple sensors in a limited number of unobtrusive devices, ii) reducing the 

number of cables required, or iii) introduce solutions allowing wireless communication of sensor 

readings. Due to user comfort, safety and unobtrusiveness requirements, this review does not 

consider invasive or implanted sensors (sensors puncturing the human skin), breath gas analysis 

sensor principles (which requires sensors or a mask in front of the mouth and nose) or body fluid 

analysis concepts. 

10.2.1 Physiological sensor concepts 

Physiological sensors aim to measure aspects of the human physiology. 

10.2.1.1 Heart rate 

The heart rate is the rate at which the heart beats. As a result of the heart contractions, blood flow through the 

arteries, causing a measurable pulse throughout the body. In normal situations, the heart rate and the pulse are 

the same, but the distinction between pulse and heart rate are made because there are situations when the heart 

is beating, but is not able to pump blood. The heart rate increases with the increasing physical effort, and it 

can also increase due to psychological stress. 

The electrical signals caused by the nerve signals in muscles, and in particular the signals associated with 

periodic contractions of the heart have been known for more than hundred years. Willhelm Einthoven was 

awarded a Nobel Prize in 1924 for his contributions to the field of electrocardiography, see for example 

Rivera-Ruiz et al (2008), see Figure 15. 

Figure 15: Willhelm Einthoven’s setup for measuring the ECG signals (left) 

and the resulting observed schematic PQRST heart signature. 

It is interesting to note that apart from some changes in electrode positions, the measurement is essentially the 

same today. While Einthoven used buckets filled with salt solutions as electrodes, the main technology 

development has taken place within the field of electrode technology. Skin electrode measurements are 

however associated with substantial problems due to physical movements, which causes large DC offset 

values to measurement signals due to polarisation effects as illustrated in Figure 16. Further, noises are 

introduced due to any muscular activity, thereby making electrode measurements more challenging for 

patients that are physically active. 

102


Figure 16: Illustration of the ECG signal and the effect of an abrupt mechanical disturbance. 

A full diagnostic (12 lead) ECG investigation, aiming at identifying heart disease conditions, employs 10 

electrodes placed on the body as shown in Figure 17. Based on the mapping of the resulting curves, it is 

possible to identify malfunctions of the heart, for example due to infarction damaged tissue. In addition to 12 

lead ECG aiming at geometrical mapping of the heart electrical signals; other diagnostic ECG measurements 

are done to characterise various heart rate variability conditions. 

Figure 17: 12 lead ECG 

Within CORBYS, it is not an objective to identify diagnostic heart condition information. Rather, CORBYS 

will employ the heart rate as an ingredient in the assessment of physical effort and possibly as a co-indicator 

of intended actions or psychophysiological states. As CORBYS deals with users engaged in activity, it is 

essential to select electrodes and measurement configurations that allow accurate monitoring while the user is 

active. The most common, and also potentially most accurate, method being used on active people is based on 

simplified ECG measurements using only 3 or 2 electrodes to extract the heart rate. 

In clinical applications, AgCl terminated electrodes are frequently used. These are glued, attached or patched 

to the patient’s skin. Silver chloride terminated electrodes can cause skin irritation with prolonged use. 

Further, they are associated with adhesive patches which can be uncomfortable to remove after use. 

Commercial heart rate monitors used in sports and fitness usually employs electrodes manufactured from 

elastic, conductive textile, or conductive rubber or polyurethane. The textile electrodes are backed by soft 

rubber or similar to increase pressure against the skin, improving electrical contact. A disadvantage with all 

these electrode versions is that a direct mechanical and electrical contact with the body is required at all times. 

The use of capacitive, “non-contact” electrodes has also been demonstrated, but these can not compete with 

the performance of contact electrodes at the present time. 

103


Figure 18: ECG electrodes. To the left is an example of disposable electrodes 

(Ambu Centre Snap ECG Electrode 1 ), in the middle a Garmin sports belt for heart rate monitoring, and to the right capacitive 

electrodes from Quasar 2 . 

10.2.1.2 Electromyography (EMG) 

Electromyography is a technique for evaluating and recording the electrical activity produced by skeletal 

muscles. Skin surface electrodes can be used to record signals from the muscles. The distance from the skin 

to the muscles of interest in CORBYS is short and the muscle groups are large. The muscle activity is related 

to the root mean square (rms) value of the signal. There are commercially available systems, e.g. from Plux 3 

and Biometrics Ltd 4 (Figure 19). In these systems several sensors are connected to a connection system that 

can send the recorded data to a computer. Systems with wired and wireless sending exist. Sensors and system 

from Biometrics Ltd are shown in Figure 19. Also for EMG capacitive, “non-contact” electrodes has also been 

demonstrated, e.g. Quasar 2 . 

Figure 19: The left images show the Biometrics Ltd EMG monitoring system. 

The right graph shows extensor EMG and 3 repetitions of maximum grip strength of a normal subject. The EMG trace is the 

raw data. 

10.2.1.3 Electrodermal response (EDR) 

Electrodermal response (EDR) is a method of measuring the electrical conductance of the skin, which varies 

with its moisture level. EDR measurements have for many years been based on DC voltage or current and 

therefore the method has also been termed galvanic skin response (GSR). EDR is of interest because the 

sweat glands are controlled by the sympathetic nervous system, so skin conductance is used as an indication 

of psychological or physiological arousal. Skin conductance levels are not in direct correlation with the sweat 

production or the evaporation from skin, but rather the sweat activity with the filling of the sweat ducts and 

the reabsorption process. 

1 http://63.134.192.29/cart/Results.cfm?category=16 

2 http://www.quasarusa.com/ 

3 www.plux.info/EMG 

4 www.biometricsltd.com/ 

104


Prototypes used for measuring electro dermal activity (sweat) have been made and reported [Tronstad et al, 

2008] [Poh et al, 2010]. There are also some commercial systems available, e.g. from Plux 5 . 

10.2.1.4 Humidity/sweat sensors 

In addition to the information obtained from EDR type measurement, it is also possible to apply humidity 

sensors that do not have to be in direct contact with the skin of the patient. Humidity sensors might be used to 

measure relative humidity inside the clothing as an indication of sweating, and methods for using two sensors 

for measuring sweat rate have also been reported [Salvo et al, 2010]. Humidity sensors are often combined 

with temperature to measure relative humidity (e.g. SHT21 6 ). Relative humidity is often used instead of 

absolute humidity in situations where the rate of water evaporation is important. SINTEF has carried out 

climate chamber testing on a jacket with integrated relative humidity sensors. Changes in relative humidity 

are correlated with the change in heart rate and tie in well with subjective reported sweat activity [Seeberg et 

al, 2011 (submitted)]. A humidity sensor does not need to be glued to the skin. 

10.2.1.5 Respiration 

The respiration rate is the number of breaths within a certain amount of time, often breaths per minute. The 

ventilation rate is a related parameter, specifically addressing the flow of gas entering or leaving the lungs 

with measurement units of volume per time. While respiration can be measured outside the patient’s chest, 

measurements of the ventilation require the use of a breathing tube or spirometer. Ventilation measurements 

are therefore considered too obtrusive for the CORBYS patient. 

A change in respiratory rate or volume might be an indication of physical or psychological stress, and hence 

might be a useful input into the CORBYS system. An individual can be aware and adjust his/her respiration 

rate, and the rate is also affected by physical or mental activities, talking and so forth, it is therefore difficult to 

apply the respiration rate as a single indicator as physiological state. In CORBYS the motion of the thoracic 

cage can be measured by impedance plethysmography (with electrodes on both sides), by inductive 

plethysmography (inductive sensor that changes values of inductance when stretched across the chest or 

abdomen), piezo resistive sensors (resistance changes when stretched across the chest or abdomen), or 

capacitive displacement sensors (measuring the change in area between two capacitor plates that slide parallel 

to one another). There are commercially available systems that also report the respiratory rate (like the 

Hidalgo Equivitale system 7 ), and more stand alone sensors like the respPlux 8 . 

10.2.1.6 Pulse oxygen saturation 

The oxygen content of arterial blood can be measured by optical techniques. The saturation partial O2 

pressure (SpO2) is a measure for the percentage of haemoglobin binding sites in the bloodstream occupied by 

oxygen. Since it measures the arterial blood the pulse can also be extracted from the measurements. SpO2 

measurements are easily affected by movements and measurements can only be carried out on certain areas 

like the earlobe and finger tip (using transmission photo-plethysmography). In normal circumstances the 

oxygen level is between 95% and 99%, and is not correlated with psycho-physiological states. SpO2 is 

continuously monitored whenever a patient’s oxygenation may be unstable as in intensive care, critical care, 

and emergency department areas of a hospital. SpO2 is also checked periodically in the case of congestive 

heart failure (CHF) and chronic obstructive pulmonary disease (COPD) patients. For CORBYS patients, the 

oxygenation is usually not an issue. It might also be a problem to do accurate measurements during gait 

5 www.plux.info/EDA 

6 www.sensirion.no 

7 www.equivital.co.uk 

8 www.plux.info/resp 

105


rehabilitation. 

Most SpO2 sensors are disposables that are glue attached to the patient. Validated products sending SpO2 and 

heart rate data via Continua standard Bluetooth protocol are also available (e.g. Nonin Onyx II 9560 9 ). 

Examples of both disposable and reusable sensors are shown in Figure 20. 

10.2.1.7 Blood pressure 

Figure 20: Nonin Onyx II 9560 and disposable 7000A Adult sensor 

Blood pressure (BP) is the pressure exerted by circulating blood upon the walls of blood vessels. A person's 

BP is usually expressed in terms of the systolic pressure over diastolic pressure (mmHg). The term blood 

pressure usually refers to the pressure measured at a person's upper arm (at the level of the heart). It is 

measured on the inside of an elbow at the brachial artery, which is the upper arm's major blood vessel that 

carries blood away from the heart. The clinical, gold standard blood pressure measurement is however done 

invasively. 

Blood pressure is influenced by many factors, such as diet, activity level, exercise, disease, drugs or alcohol, 

stress, obesity and so-forth. Further, it is a parameter that varies throughout the body depending upon gravity, 

blood viscosity, artery flexibility and several other parameters. 

Blood pressure is usually done as a point measurement during a consultation with a medical professional. It is 

based on either palpation, ascultatory measurements or oscillometric measurements. In either case, the 

measurement of the blood pressure involves a measurement of the pressure at which the blood pressure of an 

artery or vein is becoming significantly constrained. These measurements are therefore not suitable for 

continuous, long term monitoring. 

Simple, continuous and comfortable monitoring of blood pressure is currently not possible using 

commercially available products [Kirstein et al, 2005] [Sola et al, 2011]. One promising research direction is 

the analysis of the arterial pulse wave velocity, which can be measured without physical constraints on an 

artery or vein. The research is however novel, and challenged by movement artefacts, so it is not seen as a 

realistic alternative for BP measurements in CORBYS . 

10.2.1.8 Skin/Core temperature 

The clinical interest for core temperature measurements is related to illness or hypo- or hyperthermia. The 

skin temperature varies widely and is also strongly affected by the thermal environment. Both skin and core 

temperature will increase during physical effort but the human thermal regulating system will effectively limit 

the temperature changes. Both skin and core temperature are slowly varying parameter. Skin temperature can 

be measured with different types of temperature sensors, such as thermocouple, resistance, or infrared 

emission sensors placed in contact with the patient’s skin. The core temperature is typically using a rectal 

9 www.nonin.com 

106


probe or by infrared emission measurements in the ear. 

10.2.2 Inertial measurement units (IMU) 

An inertial measurement unit (IMU) uses inertial sensors such as movement sensors (accelerometers) and 

rotation sensors (gyroscopes), possibly in combination with other movement sensors. One frequent example 

in this respect is magnetometers, which help orient the object you want to track with respect to the earth 

magnetic field. In order to be able to track objects, an IMU must have accelerometer and gyroscope sensing 

in at least as many directions of freedom as the object does; typically 3 axis of acceleration and 3 axes for 

rotations. 

Accelerometers and gyroscopes measures changes of inertia during movements and rotations. For 

accelerometers, accelerometer values can be a direct measure of force via Newton’s second law: 

The equation also displays that if the acceleration measurement is integrated once with time, one gets the 

velocity, and by integrating a second time, on gets the position. The same can be done for gyroscope 

measurements, where changes in degrees/second squared can be integrated to yield angular velocity and 

integrated again to obtain the absolute angle. 

It is however important to be aware that there are several challenges in order to obtain accurate measurements. 

First, as this is integration measurements, any measurement error, such as offset error, a drift or a scaling error 

will be integrated and can become substantial. In order to perform correct integration of the accelerometer 

data is important to be aware of the direction of gravity at all times. The magnetometer is valuable for 

providing this orientation information. Further it is crucial to know the initial position, from which the 

integration process is started. In the CORBYS setting, it will therefore be important to establish “home” 

positions where the locations of the various IMUs are known. 

IMU are essential components in robotics and are also widely used in navigation systems. For human 

measurements, it is probably most convenient to select a compact, integrated IMU suitable for attachment on 

the human. As an example, the xSens MTx 10 device that offers 3D motion, 3D rotations and 3D 

magnetometer in an integrated unit weighing 30 g, and intended for biomechanics applications. 

10 http://www.xsens.com/en/general/mtx 

Figure 21: The xSens MTx inertial monitoring unit offering simultaneous 

3 axis measurements of movements, rotations and magnetic field. 

107


10.2.3 Mechanical sensors 

10.2.3.1 Pressure/Force sensors 

Force and force distribution (pressure) sensors are widely used in biomechanics and gait analysis in 

particular. There are several flexible thin force and pressure sensors on the market, e.g. from 

Tekscan 11 . Pressure sensors are comprised of numerous individual sensing elements and a 

corresponding sensor map and complex software are necessary to decode the signal. Tekscan have 

e.g. systems for in shoe, as shown in Figure 22. The complex pressure sensors might be difficult in 

real time evaluation and feedback in CORBYS gait rehabilitation system. Force sensors can detect 

and measure a relative change in force or applied load to a surface, detect and measure the rate of 

change in force, identify force thresholds and trigger appropriate action and detect contact and/or 

touch. Tekscan standard force sensors are shown in the far right image in Figure 22. 

Figure 22. Tekscan F-Scan® System , 

an in-shoe plantar pressure analysis (left and middle-left), and FlexForce A201 sensors, standard lengths from 19.7 cm to 5.1 

cm, with sensing area diameter of 0.95 cm (middle-right ) and FlexForce A401 sensor with sensing area diameter of 1 cm. 

10.2.3.2 Joint angular sensing 

Goniometers are devices capable of transforming an angular position into a proportional electrical signal. 

Commercial sensors for biomechanics applications are available, like the Plux 12 goniometer angelPlus, and the 

twin or single goniometers from Biometrics Ltd 13 (Figure 23). A torsiometer will detect the torque on a shaft 

by measuring the twist of a given length of the shaft. Single axis torsiometers are designed for measurement 

of rotations in one plane, e.g. forearm pronation/supination or neck axial rotation. Torsiometers are available 

from e.g. Biometrics Ltd (Figure 23). All these sensors must be attached to the skin using glue. 

11 http://www.tekscan.com/ 

12 www.plux.info/angle 

13 www.biometricsltd.com/ 

Figure 23: Goniometers from Plux (left) and Biometrics Ltd 

(twin axis in middle left, single axis in middle right, single axis torsiometers to the right). 

108


10.2.4 Environmental sensing 

Beyond the human-robot interface, the CORBYS gait rehabilitation system will not need extensive system 

environment sensing. The main needs are related to the need for collision avoidance and for ensuring a safety 

zone for the system users. This would most likely be accomplished by using a simple optical (laser beam) or 

mechanical switching systems to prevent the system from having collisions or from falling down a stair. 

10.2.5 Physiological multi sensor devices 

Several combined sensor systems measuring multiple parameters exist. Also validated chest belts sending 

heart rate data via a documented Bluetooth protocol are commercially available e.g. Hidalgo Equivital 

system 14 , as shown in Figure 24. This systems measures heart rate, respiratory rate, chest skin temperature, 

activity, posture and also has a fall detection algorithm as well as a Physiological Welfare Indication based on 

heart rate and respiratory rate. 

Figure 24: Hidalgo Equivital measuring unit (left) and belt (right) 

SINTEF has also developed a sensor belt sending data over Bluetooth to a cell phone (which in turn sends 

data to a server so that a nurse may follow the patient). The system is to be tested in home monitoring of 

patients with congestive heart failure in the US autumn 2011. The chest unit is shown in Figure 25, and 

includes simple ECG (heart rate), 3D accelerometer, 3D gyroscope and IR skin temperature sensor. 

Figure 25: SINTEF ESUMS chest unit belt 

10.3 Interpretation of multiple sensor signals 

The development of sensor fusion algorithms has for several years been viewed as an important perceptual 

activity in mobile robotics [Murphy, 1996]. Sensor fusion is a term used to describe how multiple 

independent sensors are combined to extract and refine information not available through single sensors alone. 

14 www.equivital.co.uk 

109


In the CORBYS context, perception sensing will be used for the following purposes: 

� User safety 

� Physical effort 

� Characterisation of gait parameters 

� Psycho-physiological states 

� User intention 

With few exceptions, extraction of meaningful parameters in these cases requires a multi sensor approach and 

can not be drawn based on single sensors alone. Compared to other gait rehabilitation research, CORBYS is 

in a quite unique position in being able to combine physiological sensors with EEG information. Below, we 

outline what type of sensor information might be of interest within CORBYS to assess the gait rehabilitation 

session and to provide sensory input to the CORBYS cognitive robotic control algorithms. 

Ensuring user safety: 

For user safety it is important to ensure that the position of extremities and joint angles are reasonable. For 

CORBYS gait rehabilitation system it will then be monitoring: 

� Mechanical switches 

� Consistency between measured body positions and robot actuator position 

Assessing physical effort: 

Physical effort will tell the therapist about the work load of the patient, and may also be input to the cognitive 

control system. For assessing physical effort the following parameters are judged most valuable: 


� Respiratory rate 

� Activity information 

� Force sensors 

� Temperature 

� Sweat/humidity 

� EMG 

� EEG 

� Input from robot actuators 

Characterisation of gait 

Characterisation of movements is also of interest. Characterisation on biomechanics is already a big field and 

applications range from gait rehabilitation [Avor and. Sarkodie-Gyan, 2009 ] to technical analysis within the 

field of sport [Myklebust et al, 2011]. 

Within the CORBYS settings we consider combination of these parameters most promising in the 

characterisation of gait: 

� Inertial measurement units information 

� Force sensors 

� Angular/torque measurements 

110


� EMG 

� Input from robot actuators 

Identifying psycho-physiological states 

Extensive research effort has been put into the field of physiological computing, which is the term used to 

describe any computing system that uses real-time physiological data as an input stream to control the user 

interface 15 . The review article Fundamentals of Physiological Computing [Fairclough, 2009] summarises 

some of the challenges and the complexity in developing a physiological computing system that employs a 

real-time measure of psycho-physiology to communicate the physiological state of the user to an adaptive 

system. 

As summarised in this article the physiological state of the user has been represented e.g. as one-dimensional 

continuum of frustration, anxiety, task engagement, mental workload, or two-dimensional space of activation 

and valence. The detection of negative emotions may be particularly relevant for computing applications 

designed to aid learning [Picard et al, 2004]. Heart rate is one of the most common parameters used to detect 

stress of the affective state [Rani et al, 2002], together with EMG, EDR and facial expression [Rani et al, 

2004] [Kulic and Croft, 2007]. 

Within the CORBYS settings we consider combination of these parameters most promising in order to 

identify psycho-physiological states. 


� EMG 

� EDR 

� EEG 

Identifying intention 

CORBYS has a vision of being able to identify and help assist the user carry out his/her intentions. This is 

however very challenging since there are no or few clear manifestations of intention in physiological 

measurements, and the sensor information is blurred by all other factors impacting the measurements. It is 

probably therefore wise to limit the ambitions to being able to identify whether the patient wants to carry on 

making a cyclic movement such as walking or whether the patient wants to stop. For this purpose the 

following CORBYS components can be considered: 

1. EEG 

2. EDR 

3. EMG 

4. Heart rate 

10.4 Summary on technology gaps and priorities for development in CORBYS 

The SOA analysis shows that many sensor concepts are developed and relatively mature. It is therefore to a 

limited extent necessary to develop entirely new sensor concepts. The best path for innovation in the field is 

through a clever combination of existing sensor concepts. The combination can either be through a physical 

integration of different sensor concepts, or it can be the combination of sensor data in order to come up with 

15 http://www.physiologicalcomputing.net/ 

111


innovative and informative new knowledge about the state and condition of the monitored subject. The 

technology gaps can therefore be separated into a hardware part and a software part. 

With respect to hardware the challenge is to build the most relevant sensors in a system that is 

� Simple and safe to use 

� Effectively combines and integrates multiple sensors in integrated devices 

� A shared sensor interface that makes possible accurate time stamping and simultaneous analysis of 

data 

� Choose and handle a mix of wired and wireless sensors 

In software the challenge is to develop relevant sensor fusion algorithms for the CORBYS gait rehabilitation 

platform, and to verify and validate these. 

Not all sensors and algorithms described in the sections above can be included; hence a prioritisation will be 

required. 

11 StateoftheArt in Situation assessment (UR) 

This section describes the current state-of-the-art on the approaches, methods and tools for automatic situation 

assessment to enhance a decision made by the system making the process in a mixed-initiative intimate manmachine 

interactivity context. 

11.1 Introduction 

Widespread commercial availability of technology has shifted human-computer interaction paradigm from a 

conventional keyboard-mouse combination to more flexible modes of interactivity. Such systems comprise of 

a number of modalities by which they may acquire user input to perform a function explicitly or implicitly 

requested by the user. The collection of input from various modalities enables the development of intelligent 

systems that are context-aware and aid in the shift from traditional systems to a more natural approach for 

interaction and usability (Hurtig and Jokinen, 2006). 

Context-aware interactive systems aim to adapt to the needs and behavioural patterns of users and offer a way 

forward for enhancing the efficacy and quality of experience (QoE) in human-computer interaction. The 

various modalities that contribute to such systems each provide a specific uni-modal response that is 

integratively presented as a multi-modal interface capable of interpretation of multi-modal user input and 

appropriately responding through dynamically adapted multi-modal interactive flow management. 

Multimodal systems provide increased accuracy and precision (and in turn improved reliability) in terms of 

context-awareness and situation assessment by incorporating information from a number of input modalities. 

This approach offers a kind of fault-tolerant way of managing modalities. In the case that one of the 

modalities fails or contains noisy data, information from other modalities can be used to minimise ambiguity 

regarding situation assessment arising from a failed or noisy modality. The improvement in more reliable 

context- sensing and thus more appropriately responsive behaviour by the interactive system is said to be a 

likely outcome of multimodal fusion (Corradini et al. 2005). 

Various application areas exist for such multimodal systems that can make use of feasible hardware devices to 

acquire input not only at a neuro-motor but also at a physiological level for instance: in a patient monitoring 

system, monitoring heart rate, perspiration, blood pressure etc. Further examples of multimodal systems 

112


include anomaly detection, assisted living, PDA with pen-gestures, speech and graphics input capability etc. 

11.2 Situation Assessment 

Endsley (2000) defines situation assessment as “the perception of elements in the environment within a 

volume of time and space, the comprehension of their meaning and the projection of their status in the near 

future”. According to Lambert (1999, 2001, 2003, 2006), situation assessment involves assessment of 

situations where situations are “fragments of the world represented by a set of assertions”. This differs from 

object recognition in the sense that it requires a shift in the procedure from numeric to symbolic 

representations, a problem coined as the semantic challenge for information fusion (Lambert, 2003). Lambert 

(2006) proposes an interesting approach to semantic fusion using a formal theory by following a development 

path that involves sequential construction of the problem in terms of philosophy, mathematics and 

computation. This approach is illustrated using a formal theory for existence in Lambert’s work (Lambert, 

2006). 

Level 1: 

Perception of 

elements in 

current 

situation 

Situation Awareness 

Level 2: 

Comprehension 

of current 

situation 

Goals and 

objectives 

preconceptions 

Figure 26: Lambert's approach to semantic Fusion 

There exist two main stages for integration and fusion of multimodal input according to Corradini et al (2005) 

namely: 

• Integration of signal at feature level, 

Feedback 

• Integration of information at semantic level. 

Level 3: 

Projection of 

future status 

Environment 

Decision 

The feature level signal fusion, also referred to lower level fusion is related to “closely coupled and 

synchronised” modalities such as speech and lip movements. This does not scale with ease, requires extensive 

training data sets and incurs high computational costs (Corradini et al. 2005). Higher level symbolic or 

semantic fusion on the other hand, is related to modalities that “differ in time scale characteristics of their 

features”. This entails time stamping of all modalities to aid the fusion process. Semantic fusion involves a 

number of benefits such as off-the-shelf usage, reusability, simplicity etc. (Corradini et al. 2005). Semantic 

fusion is a process that unifies input at a “meaning level” from various modalities that are part of a multimodal 

system (Gibbon et al. 2000). It is said to occur in two steps: a) input events for a user’s command are taken 

from various modalities and fused at a low level to form a single multimodal input event that signifies the 

113 

Action


user’s command; b) this multimodal input event is then semantically interpreted at a higher level to deduce the 

intended meaning of the user’s action by “extracting and combining information chunks” (Gibbon et al. 2000). 

The main aim of multimodal fusion is to make sense of the user’s intended meaning by fusing the partial 

streams of input that constitute a user’s command from various input modalities of the system. Hurtig and 

Jokinen (2006) cite a “classical example of coordinated multimodal input” as the Put-that-there system 

proposed by Bolt (1980). This system presented a scenario where the user of a system specified input 

commands in a natural manner by way of speech and hand gestures. According to Hurtig and Jokinen (2006), 

multimodal input fusion can be categorised as a process that occurs over three levels: i) signal-level, 

ii) feature-level and iii) semantic, where semantic fusion integrates the understanding acquired from inputs of 

various individual modalities into a single comprehension set that signifies the intended meaning of the user’s 

input. 

Hurtig and Jokinen (2006) propose a three level semantic fusion component which involves temporal fusion 

that creates combinations of input data, statistically motivated weighting of these combinations, and, a 

discourse level phase that selects the best candidate. The weighting process uses three parameters namely 

overlap, proximity and concept type of the multimodal input events (Hurtig and Jokinen, 2006). Overlapping 

and proximity are related to temporal placement of the input events where proximity is said to play a very 

important role especially in modalities such as speech and tactile data (Hurtig and Jokinen, 2006). Concept 

type refers to user commands which would be incomplete if the system were to consider only one input 

modality for example a user pointing to a location and speaking a phrase that only incompletely describes the 

location (Hurtig and Jokinen, 2006). Once the weighted ranking has been performed, a list of these candidates 

is passed on to the discourse level phase that selects the best ranked final candidate for system response 

construction. If no candidate fits, the system requests the user to repeat his/her command (Hurtig and Jokinen, 

2006). 

Multimodal fusion encapsulates the union of a number of data from a number of input channels in a 

multimodal interactive system (Landragin, 2007) where each channel may represent a distinct modality 

whereby a user may interact actively or passively with the system. Active input can be categorised as a direct 

usage of the system on the part of the user by way of speech or gestures etc. whereas passive input may be 

understood as input acquired from the user as a result of a monitoring activity on the part of the system. 

According to Landragin (2007), multimodal fusion is distinguished in terms of several sub-processes namely: 

a) multimodal coordination, b) content fusion and c) event fusion. Multimodal coordination associates two 

activities acquired by different modalities for the formation of a “complete utterance” (Landragin, 2007). The 

output of this sub-process contains a set of paired “hypotheses”, with an associated confidence level, which 

are ingested by the multimodal content fusion sub-process to develop an improved comprehension of 

otherwise partial information (Landragin, 2007). The last sub-process, multimodal event fusion, then unifies 

the “pragmatic forces of mono-modal acts” to create a complete understanding of the user’s input. Landragin 

(2007) uses the general communication categories as considered in the study of classical natural language, and 

lists them as being the following: 1) inform, 2) demand, 3) question and 4) act. Inform, demand and question 

are fairly easy to understand as scenarios where the user may be providing information to the system, 

requiring the system to do something and querying the system to provide him/her with some information. Act 

is the general category which comes into play when the system is not able to label the user’s input act into any 

of the aforementioned three categories (Landragin, 2007). 

The fusion process is divided into five levels by the Joint Director of Labs / Data Fusion Group revised model 

(JDL/DFG) namely: pre-processing (level 0), object refinement (level 1), situation refinement (level 2), 

114


impact assessment or threat refinement (level 3), and lastly process refinement (level 4). This model, 

however, does not incorporate human-in-the-loop, and another level called user refinement (level 5) has been 

proposed to “delineate the human from the machine in the process refinement” allowing for the human to play 

an important role in the fusion process. (Blasch and Plano, 2002) 

• Level 0: Sub-object assessment 

• Level 1: Object assessment 

• Level 2: Situation assessment 

• Level 3: Impact assessment 

• Level 4: Process refinement 

• Level 5: User refinement 

Level 0 and 1 deal with sub-object and object assessment respectively, making use of information from 

multiple sources to arrive at a representation of objects of interest in the environment. In level 2, a 

relationship between the identified objects is established. At the end of this level, once the situation 

assessment process is complete, the system has achieved situation awareness as the objects detected in the 

environment and the various ways in which they are related or connected with each other is known by the 

system. Level 3 allows the system to predict effects of actions or situations on the environment. Level 4 

attempts to refine the outcomes of levels 1, 2 and 3. 

Corradini et al. (2005) list a number of approaches and architectures for multimodal fusion in multimodal 

systems such as carrying out multimodal fusion in a maximum likelihood estimation framework, using 

distributed agent architectures (e.g. Open Agent Architecture OAA (Cheyer and Martin, 2001)) with intraagent 

communication taking place through a blackboard, identification of individuals via “physiological 

and/or behavioural characteristics” e.g. biometric security systems using fingerprints, iris, face, voice, hand 

shape etc. (Corradini et al. 2005). It is stated by Corradini et al (2005) that modality fusion in such systems 

involve less complicated processing as they fall largely under a “pattern recognition framework” and that this 

process may use techniques for integrating “biometric traits” (Corradini et al. 2005) such as the weighted sum 

rule as in Wang et al. (2003), Fisher discriminant analysis (Wang et al. 2003), decision trees (Ross and Jain, 

2003), decision fusion scheme (Jain et al, 1999) etc. 

Corradini et al (2005) also list a number of systems fusing speech and lip movements such as using 

histograms and multivariate Gaussians (Nock et al. 2002), artificial neural networks (Wolff et al. 1994; Meier 

et al. 2000) and hidden Markov models (Nock et al. 2002). 

Some systems use independent individual modality processing modules such as speech recognition modules, 

gesture recognition module, gaze localisation etc. Each module carries out mono-modal processing and 

presents the output to the multimodal processing module which handles the semantic fusion. These systems 

are ideal for introducing a shelf framework where various showcases may be developed for different 

application domains applying re-usable off-the-shelf components each handling a single modality in full. 

Other systems include “quick set” (Landragin, 2007) which offers the user the freedom to interact with a mapbased 

application using a pen-and-speech cross-modal input capability. The system presented in Elting (2002) 

enables the user to specify a command by way of speech, pointing gesture and the input from a graphical user 

interface into a “pipelined architecture”. The system put forward by Wahlster et al (2001) is a multimodal 

115


dialogue system which fuses speech, facial expression and gesture for “both input and output via an 

anthropomorphic and affective user interface”, as an Embodied Conversational Agent (ECA) or Avatar 

(Corradini et al. 2005). In all such systems, input events are assigned a weight or “confidence score” that is 

considered during high level fusion for creating a set of situation interpretations that are ranked by their 

combined score (Corradini et al. 2005). 

11.3 Relevant approaches 

We review different approaches and techniques related to situation assessment for solving the problem of 

identification of objects, scenarios, situations of interest in an operational environment. 

11.3.1 Rulebased Expert Systems 

These are knowledge based systems consisting of a rule base and a fact base. Rules encode domain 

knowledge whereas facts represent the state of the environment. The outcome of a rule is deduced by a 

preceding condition part of the rule using logic, where facts act as the input to the condition. When rules are 

executed, new facts corresponding to the conclusion are added to the fact base. The reasoning of the rulebased 

systems is controlled by an inference engine. To achieve situation awareness, this type of system can 

be used with ontological mappings of the environment in terms of the objects, relations, situations etc. within 

this environment. While such systems are easily deployed and various expert tools are available that require 

only the rule definitions to be specified by the user, the major drawback of these systems is that they are 

largely deterministic in nature. Therefore, a lack of provision for uncertainty values for rules introduces 

limitations and imposes demands on the user regarding careful consideration in developing the rules (Russell 

and Norvig, 2003). Rule-based systems also lack the possibility to perform temporal reasoning. Learning 

new rules in an automatic fashion is not so straight forward either. This necessitates manual human input 

which can be a tedious task. 

11.3.2 Fuzzy Logic 

Fuzzy logic deals with reasoning that is approximate rather than exact. Fuzzy logic variables can have truth 

value ranging between 0 and 1, as opposed to two-valued logic (true or false), hence allowing handling of 

partial truth where truth value can range between completely true and completely false (Novak et al. 1999). 

In classical set theory, elements are introduced to a set in binary terms, overseen by a hard condition, whereby 

an element either belongs to or does not belong to the set. In fuzzy set theory, elements can be assessed 

gradually in terms of their candidature using a membership function that defines the degree to which an 

element can be considered to be inside or outside of the fuzzy set. The use of fuzzy logic allows for complex 

real world modelling. Data fuzzification allows a transition from numerical to fuzzy format, followed by an 

evaluation or fuzzy inferencing of the fuzzy conditions. The output is then de-fuzzified i.e. transited back to 

numerical format. There exist several fuzzy inference engines such as Java based Jess (Orchard, 2001) and 

Prolog based Flint (Shalfield, 2005) among others. 

11.3.3 Casebased Reasoning 

Case Based Reasoning (CBR) is a method used in computing to allow systems to solve problems by recalling 

how similar problems were dealt with previously. The system is given a set of basic cases and a set of rules 

by which it can alter these cases. When given a problem, the system finds the most relevant case (or multiple 

cases) and if required, modifies it to solve the problem. The basic premise for CBR is that similar cases can 

be satisfied by similar solutions. Hence, if a similar case to the received input case is found present in the 

CBR repository, the solution can be adapted to the current case and the problem can be solved. If the 

modified solution is satisfactory, the new solution is retained in the repository together with the problem. 

116


Problem 

4 

Retention 

Solution 

1 

Retrieval 

Figure 1: CBR Process 

Case Based Repository 

1. The problem is entered into the system and analysed. 

2. The case which closest matches the details of the problem is selected. 

3. This case is modified to better fit the problem using predefined rules. This becomes the solution 

which is presented. 

4. The solution is analysed for its effect and is stored for future use along with an indication of its 

successfulness. This allows the system to learn. 

The system should be able to learn not only from its successful solutions, but also from its failed solutions 

(success driven and failure driven learning). Successful solutions can be stored so that the solution can be 

reused without regenerating it. Solutions which failed to solve the problem can be used to generate better 

solutions for use at a later stage. CBR can be used to identify situations presented as cases from a repository 

of known situations; however, the mapping of a situation as a case and subsequent measurement of similarity 

between two cases poses a problem. Furthermore, the existence of an associated solution becomes irrelevant 

as soon as the situation is matched to a template or a case from the repository. 

11.3.4 Bayesian Networks 

2 

Selection 

Bayesian networks are probabilistic graphical models that represent a set of random variables and their 

conditional dependencies via directed acyclic graphs. Bayesian networks are commonly used for probabilistic 

reasoning in the context of situation assessment (Das et al. 2002; Bladon et al. 2002; Higgins 2005). Bayesian 

networks are based on Bayes theorem which computes posterior or inverse probability for a proposition i.e., 

given the prior or unconditional probabilities of A and B, and knowing the conditional probability of B given 

A, what is the conditional probability of A given B? Bayesian nodes in the networks represent propositions, 

random variables, unknown parameters or hypotheses. Nodes are connected by edges, that represent 

conditional dependency, and unconnected nodes therefore represent variables that are conditionally 

independent. Each node has an associated probability function that gives the variable probability represented 

by the node, upon receiving a set of values as input. Priori probabilities and conditional probabilities have to 

be specified for each node in the network. Inverse probabilities for each node can then be computed using 

Bayes rule as new input is received into the network. 

In the context of situation assessment, Das et al. (2002) lists two important points that must be observed when 

117 

Application of 

rules to modify 

selected cases 

Figure 27: Case-Based Reasoning process 

3 

Presentation


using Bayesian networks in real-time environments i.e., rapid modelling of complex situations via Bayesian 

networks, efficient Bayesian network inference based on incoming evidence. For rapid modelling, each 

network can be constructed in real-time from smaller Bayesian networks to assess a specific situation, 

whereas for efficient inference, a network can be broken up into sub-networks and distributed over a physical 

network of computers, employing parallel processing technologies (Das et al. 2002). 

While Bayesian networks satisfactorily handle uncertainty and casual relationships and are fairly 

straightforward in terms of implementation, the set of variables they support is finite and each variable has a 

fixed domain of possible values (Russel and Norvig, 2003). Regular Bayesian networks lack the concept of 

objects and relations and hence are unable to fully benefit from structure of the domain (Howard and 

Stumptner, 2005). In order to enable application of Bayesian networks in complex domains, relational 

probabilistic models attempt to rectify these limitations (Howard and Stumptner, 2005; Russell and Norvig, 

2003) by including generic objects and relations. Bayesian networks also lack support for temporal reasoning 

and dynamic Bayesian networks have been proposed in this regard (Russell and Norvig, 2003). Also, a large 

amount of training data is required, as Bayesian methods are based on probability theory, to approximate the 

probability distributions. 

Sutton et al. (2004) present a Bayesian blackboard (called AIID) for information fusion that is a conventional, 

knowledge-based blackboard system in which knowledge sources modify Bayesian networks on the 

blackboard. It is proposed to carry several advantages as temporal reasoning can range from “data-driven 

statistical algorithms up to domain-specific, knowledge-based inference”; and “the control of intelligencegathering 

in the world and inference on the blackboard can be rational … grounded in probability and utility 

theory”. (Sutton et al. 2004) 

11.3.5 Blackboard systems 

A blackboard system is an Artificial Intelligence application based on the blackboard architectural model. In 

such systems, the blackboard acts as a Common Knowledge Base (KB) which is continuously updated by 

several specialised Knowledge Sources (KS) to reach a viable solution to a problem. Each Knowledge Source 

updates the blackboard with a partial solution and iteratively, a complete solution is worked towards 

collaboratively by all subscribers (KSs) of the blackboard. The blackboard i.e. the common KB retains the 

state of the problem solution, while the KSs make changes to the blackboard. 

The blackboard model is suitable to tackle problems that are complex and not well-defined, and especially 

where the solution is a sum of its parts. The main features of blackboard architecture are multiple sources, 

multiple competing hypotheses, multiple levels of abstraction, feedback to the sources and the blackboard 

acting as a common knowledge base or an associative memory. Blackboard systems were created to deal with 

complex problems that are best addressed by an approach of incremental solution development. They have 

been used extensively in AI problems, and there exist several examples in the literature and market e.g. 

Hearsay I (Reddy 1973), Hearsay II (Erman, 1980), blackboards addressing signal and image understanding 

(Carver, 1991), planning and scheduling (Sadeh, 1998; Smith, 1985), machine translation (Nirenburg, 1989) 

workflows (Stegemann et al, 2007), and spatial-temporal geographic information analysis (Riadh, 2006). 

In essence, a blackboard system is a task independent architecture, implying that it addresses a range of 

problems such as design, classification, diagnosis etc. Owing to its integrated design whereby multiple 

knowledge sources work towards a solution incrementally, more than one technique may be employed, each 

as a separate knowledge source or reasoning system e.g. KS1 can be case-based while KS 2 can be heuristic i.e. 

rule-based (Hunt, 2002). 

118


Blackboard architectures offer several benefits such as configuration flexibility as the KSs are not rigidly 

connected, rather cooperate via the blackboard, hence KSs can be removed or added to the system without 

explicit declaration to all subscribers at any point in time. Furthermore, the user is presented with a selection 

of choices as the set of KSs may contain more than one source carrying out the same function to arrive at a 

partial solution, in which case the most efficient may be chosen. The blackboard system manages multiple 

levels of abstraction, using hierarchy supported by KSs, where the KSs may operate at different levels of 

abstraction and/or may operate between two abstraction levels for information transfer. The main 

disadvantages of these systems include a lack of support for task specific strategies, i.e. lack of support for 

domain specific problem solving strategies and they have to be implicitly added if required. Also, as the 

blackboard is a common knowledge base, it is directly changed by the knowledge sources and therefore 

consideration must be taken to oversee the responsible behaviour of knowledge sources. The scope of the 

system as a whole is not maintained, making it difficult to identify problems (Hunt, 2002). 

Salient architectures, tools and technologies 

Hearsay (or Hearsay I) was developed to solve complex speech recognition problems with a hierarchical 

abstraction space of complete and partial interpretations. It was domain restricted and not a general purpose 

problem solving system, however it led to an extension in the form of a new design or a sequel. Hearsay II 

proposed the architecture of blackboard as known today for the first time. It responded to questions and 

retrieved documents from a collection. It was not restricted to any specific domain. In Hearsay II, the 

knowledge sources work in an opportunistic fashion, i.e., each KS monitors the blackboard for an opportunity 

to contribute to the development of the solution. If such an opportunity is detected, the KS posts an activation 

entry on the agenda of the scheduler indicating its intention to contribute. The scheduler determines the 

suitability of any KS activation, using information in a common knowledge base and the potential impact of 

the knowledge source. 

HASP/SIAP, a signal interpretation system based on the blackboard model of Hearsay II, used AI techniques 

to address the surveillance problem of the activities of submarines and ships in the surveillance region. 

Instead of the scheduler agenda approach, HASP employed a control mechanism based on predefined events, 

which triggered a sequence of KSs relevant for an occurring event. Another system called TRICERO system 

also addressed a similar application area, employing blackboard architecture in the domain of distributed 

computing, to monitor a region of airspace for various aircraft activities and understand the signals to assess 

the situation. The problem is partitioned into independent sub-problems and solved using blackboard systems. 

BB1 is a task independent blackboard control architecture that offers two blackboards, namely a domain 

blackboard, which is essentially the blackboard as we know it, and a control blackboard, which deals with 

generation of a solution to the control problem. The control blackboard in BB1 works the same as standard, 

the only difference being that KSs are control problem solvers and the solution on the blackboard determines 

which domain knowledge source is to be activated next. In this way, BB1 offers increased flexibility in its 

control mechanism, albeit with additional computational complexity, as opposed to Hearsay II. A well-known 

example implementation using BB1 is a mission-planning system for an autonomous vehicle (Hayes-Roth 

1985). 

Hearsay III provides a completely generic architecture which can be used to develop a specialised problem 

solver for a specific domain. The primary goal of Hearsay III is to provide representation and control 

facilities in order to enable the user to create a customised expert system. 

Van Brussel et al. (1998) used a blackboard model in a behaviour-based mobile robot system enabling task 

execution in unstructured real-world environments. The behaviour model consists of a number of motion 

119


behaviours including reflexes, voluntary motion behaviours, and knowledge acquisition modules providing 

task-oriented information about the robot environment. Zhang & Zhu (2004) present a multi-agent based 

architecture using a distributed communication infrastructure and an event-driven situation evaluation agent 

for outdoor mobile robot navigation where event-driven control is used to handle the dynamic change in the 

environment. Smartfire (Petridis and Knight, 2001) uses the blackboard architecture in a hybrid CBR system 

for the automatic set-up of fire field models which enables fire safety practitioners who are not expert in 

modelling techniques to use a fire modelling tool. This hybrid system demonstrates the power of blackboard 

with a common knowledge representation and mechanisms for adaptation and conflict resolution. 

Psyclone is a generic framework for AI, which includes support for multimodal input (vision, audio, speech 

and other sensor input) and modular cognitive processing and multimodal output generation (speech and 

animated output). It is a message-based middleware for developing large distributed systems (Thórisson 

2005). It introduces the concept of a whiteboard, an extension of the blackboard model which allows 

heterogeneous systems to be connected together. The Psyclone framework consists of a number of 

whiteboards, each functioning as a publisher/subscriber of information. Data is retained by whiteboards for a 

time period, in a searchable database for later retrieval by modules needing historical information. Psyclone 

acts as a central connection point for modules using whiteboards. It also allows supervision of the system at 

runtime using a built-in monitoring system. Time-stamped information on whiteboards can be viewed and the 

content of individual messages can be read. 

11.4 Summary on technology gaps and priorities for development in CORBYS 

The state-of-the-art review highlights various existing concepts that have already been developed and are 

fairly well-established in literature. It is foreseen that a maximally optimum combination of existing concepts 

in light of requirements for CORBYS will provide solid foundations for the CORBYS situation assessment 

and awareness cognitive architecture. The following aspects have, in particular, been identified as points of 

interest where the CORBYS project can contribute to research and development of context-and-situationaware 

cognitive architecture: i) observation (and learning) of cyclic patterns of gait, ii) identification of 

deviation from established normal gait pattern, iii) rectifications suggested to human-in-the-loop, 

iv) facilitation for user refinement (level 5 in the revised model JDL/DFG model). In order to present generic 

architecture for situation awareness, the blackboard solution is seen as the most suitable for several reasons. 

Components developed in CORBYS can be used as reusable off-the-shelf publishers and subscribers of the 

blackboard system and this enables ease of deployment in other application domains, where CORBYS 

solutions may be used. Moreover, through multiple publisher and subscriber components, the use of 

blackboard will enable isolation of the CORBYS cognitive architecture from the application domains e.g. the 

two CORBYS demonstrators. Domain specific requirements can be addressed by specialist publisher 

components developed to work with the blackboard. A major technology gap/challenge is to develop suitable 

fusion algorithms for the CORBYS gait rehabilitation platform, and for the CORBYS mobile robot for 

examination of hazardous areas. 

12 StateoftheArt in Behaviour Generation, Anticipation and Initiation 

(UH) 

12.1 Introductory Comments 

The state-of-the-art of various forms of cognitive mechanisms concerning behaviour anticipation, -generation 

and initiation will be considered. 

120


First, the general outline will be established. In particular “bio-inspired” or “biologically motivated” 

mechanisms, and behaviour anticipation, generation and initiation are conceptually interconnected. 

Concerning the notion of bio-inspired mechanisms, one needs to distinguish the notions that are implied with 

this tag. Bio-inspired algorithms are generally understood to be algorithms that are imitating or mimicking in 

an abstracted form mechanisms that are prevalent in nature. Well-known examples are Neural Networks 

which mimic the networks of computation in the brain or Genetic Algorithms which mimic Darwinian 

evolution, or Ant Colony Algorithms which mimic the foraging process of biological ants. All these 

algorithms have in common that the analogies to nature are largely on the level of mechanisms, and it is 

common to all that in the context of most engineering-oriented applications they usually do claim to be a 

sufficiently accurate model of biological processes. However, it is increasingly established that biologically 

accurate processes are often not captured by these simplified mechanisms, although they prove useful and 

effective in a wide selection of problem-solving scenarios. 

The second view is that of trying to accurately model actual processes taking place in biology, as, for 

example, in computational neuroscience, population biology and other. Here, the goal is typically the 

modelling of phenomena actually occurring in biology and the purpose is indeed the (at least qualitatively) 

accurate modelling of salient biological observations. The models are often computationally relatively 

expensive, but they aim for significantly increased faithfulness to biological phenomena. Nevertheless, in a 

considerable number of cases, even an accurate modelling of phenomena may contribute only to a limited 

degree to understand what the achieved biological purpose is. In addition, these models are not easy to adapt 

for technical purposes, although there has been some successful adaptation of spiking neural networks (Maass, 

2003). 

The third view which is the view which will dominate the subsequent discussion, adopts a stance between 

these two extremes. Instead of copying mechanisms “naively” or creating detailed models, a recent family of 

approaches considers not mechanisms, but rather principles governing biologically relevant cognitive 

processing. The idea behind this is to not be concerned with the mechanisms that are implemented in the 

biological substrate, but rather with the principles that these mechanisms have evolved or adapted to 

implement. Examples for such principles proposed in the past are the minimum jerk or the two thirds power 

law (Viviani and Flash, 1995) for limb movement. Here, no specific neural or cognitive mechanisms are 

postulated, but rather that the outcome of the behaviour generation in biological organisms fulfils certain 

mathematical conditions (such as the Fermat Principle-like minimum jerk variation principle). 

The advantages of using principles to determine behaviours are manifold: 

� Principles detach the form of the behaviour generated or modelled from its “implementation”. If a 

principle is found to be valid for a behaviour, one needs not implement a detailed, physiologically 

accurate neuronal control model for the behaviour generation, but rather a high-level computation of 

the principle which may or may not be an accurate model of the biological substrate. 

� Such principles are usually simpler to formulate and contain significantly fewer assumptions and 

parameters than the usual detailed biological models. They are less arbitrary than bio-mimicking 

computational models such as Neural Networks or Genetic Algorithms which, while having fewer 

parameters than the full-fledged models, have deficits in actually capturing the essence of the 

behaviours of biological organisms. 

� At the same time, such principles can be quantitatively formulated and allow one to create precise 

quantitative and often algorithmic implementations of the principles; as said before, the detailed 

121


algorithms may not have anything to do with the microscopic dynamics of the biological substrate, 

but they aim to capture the mesoscopic behaviour of the organism. 

� Principles allow a transparent view into current forms and possible variants of behaviours. Since not 

the microdynamics is modelled, they encompass are only comparatively few degrees of freedom and 

it is easier to identify which changes affect which part of the dynamics. 

� Principle-based behaviour analysis and generation makes it possible to hypothesise about evolutionary 

incentives for the emergence of these behaviours. In this way, a principle under investigation turns 

often out to be accompanied by a number of further principles related to it, derived from it, or which 

are in force in parallel to it. This provides additional natural pathways for refinements of the 

methodology. 

For these reasons, principle-based methodologies will be those most concentrated on, however, where 

relevant, in the light of realising the practicality of the methods, other, traditional methodologies will also be 

referred to, keeping, however, always in mind that they will serve to implement principles. 

Amongst important principles that have been investigated in the past (e.g. the two thirds power law or the 

minimum jerk variation principle), one class of principles has found significant attention in the last few years. 

This class of principles is based on Shannon information theory. It is characterised by versatility, universality, 

transparency, mathematical guarantees (e.g. for robustness) in some circumstances. Most importantly, there 

are strong indications that the behaviour of biological organisms may be governed to a not insignificant 

degree by informational principles (Laughlin et al., 1998, Laughlin, 2001, Polani, 2009). The CORBYS 

SOIAA component will be centrally based on these principles whose current state-of-the-art will be described 

in the following section. 

12.2 InformationTheoretic Principles 

12.2.1 Biological Context 

Information-theoretic behaviour principles form a special case of behaviour “bio-inspired” or “biologically 

motivated” anticipation, generation and initiation mechanisms, as the mathematical principles on which they 

rely can be formulated on a quite abstract level, but still have high biological relevance. The core hypothesis 

of the SOIAA component in the CORBYS project operates on this assumption. 

Key is the hypothesis that it is not necessary to accurately mimic human (or biological) behaviour by the robot 

for a smooth interaction and rather that it is sufficient to have a simplified interaction model. This, however, 

needs to capture characteristic properties, the essence of biological behaviour without attempting to be over 

accurate. 

The present approach is based on mounting evidence that biological information processing might be 

following informational optimality principles (typically minimisation or maximisation, depending on context) 

with respect to information processed by the organism or agent. Here information is taken into consideration 

strictly in the sense of Shannon (this will always be assumed in the following). 

Shannon information acts as a natural measure of unperturbed sensorimotor data flow throughput as well as of 

redundancy and its relevance for organismic behaviour has been hypothesised already in the 1950s and then 

re-emphasized beginning in the 90s (Barlow, 1959, Barlow, 2001, Attneave, 1954, Atick, 1992). From a 

different angle, namely the perspective of cybernetics, especially Ashby’s Law of Requisite Variety should be 

mentioned (Ashby, 1956). Its recent extensions in (Touchette and Lloyd, 2000, Touchette and Lloyd, 2004) 

122


demonstrate quantitatively the role of information in the sensorimotor loop: the difference in entropy 

reduction in the environment that an agent achieves in open-loop and closed-loop mode is limited by the 

information that the agent takes in. This is a fundamental limit that holds for any system that can be 

probabilistically modelled. It does not depend on any other particulars of the model. As such, it exemplifies 

the universality character of information-theoretic descriptions of agents which holds independently of their 

computational model. This relevance is universal when it holds to the informational interaction (and all 

physical interactions are at the same time informational) of an agent with its environment. 

However, the relevance even seems to translate into how biological organisms process information internally 

or how information is generally processed in evolutionary context. For instance, evolutionary fitness models 

that are suitable to be cast into Kelly Gambling scenarios (Kelly, 1956) can be combined with Howard’s value 

of information (Howard, 1966) in a coherent framework (Donaldson-Matasci et al., 2010). 

There are indications that the internal processing of sensoric stimuli seems also to be governed by 

informational optimality principles. This seems to hold particularly true for neural signals (Rieke et al., 1999, 

Laughlin et al., 1998, Parush et al., 2011). The basis for that assumption is partly due to metabolic reasons, 

since (Shannon) information processing is metabolically expensive. It is expected that evolutionary pressure 

will act as to optimise the information processing channels of an organism (Brenner et al., 2000, Laughlin, 

2001, Polani, 2009). Under this hypothesis, an under-exploited informatory channel will either be evolved 

away until it is optimally used, or else be increasingly used to improve the utility that can be gained through 

its existence. It follows that, in the adaptive equilibrium case, one expects that an existing information 

channel for an adapted organism (i.e. an organism that is in a suitable sense of its “informational ecology” in 

balance with its environment) will be exploited to its fullest by this organism. It will in general not hold true 

for an organism out of balance, i.e. an organism that only recently entered a new ecological niche and did not 

yet (on individual as well as on population level) have sufficient time to adapt to that niche. 

These assumptions are also consistent with the assumption that sensorimotor abilities of animals and humans 

operate on the basis of a Bayesian model (Schrater and Kersten, 2002, Körding and Wolpert, 2004). Evidence 

for the Bayesian character of organismic decision making is not limited to higher organisms. In fact, it turns 

out that even insect behaviour demonstrates some consistency with Bayesian modelling. An example for that 

is the infotaxis model that was introduced by (Vergassola et al., 2007) to model the search behaviour of a male 

moth for its female mate through a very sparse, event-based olfactory signal. In this model, isolated 

pheromone detection events, through an inverse Bayesian model, and through a seeking behaviour that is 

consistent with maximizing information about the location of the mate (named infotaxis by the authors) are 

sufficient to reconstruct a search-and-home dynamics that is surprisingly consistent with what is observed as 

behaviour of actual moths. It is, of course, not assumed that the male moth is indeed “implementing” a full 

Bayesian model and an infotaxis dynamics, and in fact, it is more plausible to assume that the brain of the 

organism will in most likeliness implement a proxy or surrogate dynamics that, in the scenarios of relevance 

will exhibit infotaxis-analogue behaviour. Nevertheless, the close similarity demonstrates that the assumption 

of near Bayes- and information-optimal behaviour provides a powerful model while not necessarily of 

mechanisms, but of general character of organismic behaviour generation and the incentives that drive it. 

12.2.2 Cognitive Modelling Context 

The identification of information-theoretic concepts as governing the behaviour of organisms provides 

quantitative pathways towards a systematic modelling of cognitive architectures based on these principles. 

However, although information theory is an old and long-established field, its successful use for cognitive 

modelling has expanded in a very significant only in the last decade, boosted by a series of advances. 

123


12.2.2.1 The Information Bottleneck Principle 

The information bottleneck principle was introduced in (Tishby et al., 1999). It implements an informationtheoretic 

version of sufficient statistics. As it uses information-theoretic measures to quantify informational 

sufficiency, the information bottleneck principle provides a constructive way to obtain optimal 

approximations for sufficient statistics. In the case that the distributions are modelled by exponential families, 

sufficient statistics can be perfect achieved, where not, they are optimally approximated. 

Another interpretation of the information bottleneck is its ability to tag relevant information, selected from a 

total body of information. Unlike in the classical interpretation of Shannon information theory, where 

information is completely devoid of semantics, in the information bottleneck, one is able to “colour” 

information according to its relevance, distinguishing between relevant and irrelevant information. This is 

achieved by introducing a relevance variable Y which denotes the feature or quantity of interest. This variable 

is now available to the system (or the random) variable X only via another variable Z which may contain a 

significant amount of total information which is not relevant in the sense that it does not convey any 

information about the relevance variable Y. The system (or random) variable X has only direct access to the 

latter variable Z which screens the relevance variable Y from it. 

Formally, one considers the Markovian Sequence Y�Z�X where Y is the relevance variable. One then 

maximises the mutual information I(X;Y) while keeping I(X;Z) constant. Thereby, one aims to limit that 

information that X takes in about Z which is not relevant to Y, the optimisation running over all probabilistic 

projections p(x|z). This constrained optimisation can be transformed into an unconstrained optimisation by 

maximising I(X;Y)��I(X;Z) for various values of � which achieve various levels of information intake of X 

from Z. 

This process defines an efficient frontier of solutions which denote the optimal trade-offs between the 

information intake from Z and relevant information intake from Y. The parameter � allows one to control 

whether one rather puts emphasis on capturing the relevant information as completely as possible or one is 

rather interested in keeping the information bottleneck around Z as tight as possible. 

12.2.2.2 UtilityRelevant Information 

The relevant information concept is not limited to identifying prescribed information in the relevance 

variable Y. For a setting where actions taken by an agent can be formulated, it is possible to reformulate the 

relevant information setting to take utility into account. The (unconstrained) optimisation (here a 

minimization over the policy �(a|s)) now is carried out over the functional I(S;A)��E[Q(S;A)] where E 

denotes the expectation and Q denotes the utility of action A taken in state S (Polani et al., 2006). This 

optimisation turns out to be formally equivalent to the well-known rate-distortion problem from information 

theory (Blahut, 1972), with Q replacing the distortion function. An important distinction in the interpretation 

is that the distortion function typically measures distortions between signals which — nominally — live in the 

same state space, whereas Q has not the character of a distortion as actions A and states S are concepts coming 

from entirely separate spaces. Nevertheless, the rate-distortion formalism transfers seamlessly to the new 

situation. 

However, in the presence of delayed reward structures, the utilities Q cannot be directly computed, but depend 

on the complete process that the system undergoes when proceeding according to policy �(a|s). Thus, the 

utility of a concrete state s under a concrete action a (the lowercase characters denote a concrete value rather 

than a random variable) and following policy � is given by where s' is the state following s, P(s,s',a) is the 

transition matrix and R(s,s',a) is the reward structure. 

124


In this case, the optimisation of the functional I(S;A)��E[Q(S;A|�)] requires a more intricate procedure for its 

computation since now not only I(S;A), but also Q(.;.|�) depends on the policy �. This computation can be 

carried out by a double fixed point iteration (Polani et al., 2006), and can be extended also to consider 

cumulated processed information, such as the information-to-go of the whole agent-environment system 

(Tishby and Polani, 2011) or the lookahead information (van Dijk and Polani, 2011b). See also (Saerens 

et al., 2009) for a related concept. 

These computations provide an informatory interpretation of goal-directed, or, more precisely, utility-driven 

behaviour. Conceptually, this corresponds to a regularisation of the usual Markovian Decision Process 

optimisation via Shannon information quantities. This approach offers two important advantages. First, it 

connects the informational interpretation of biological cognitive processes with the formal framework of 

utility-based decision making. Second, on the mathematical level, the informational regularisation provides a 

tie break in the case of undetermined solutions of same performance and, in addition, it offers 

stability/robustness guarantees under a PAC-Bayesian perspective (McAllester, 1999). These guarantees can 

be interpreted as implementing a “least committed” future of an agent’s behaviour (Tishby and Polani, 2011). 

This is advantageous from a number of perspectives. Least commitment implies that the agent implements the 

principle of information parsimony which assumes that minimal informational load is a natural way to model 

minimal cognitive load which, in turn, formed the basis of the biologically plausible information processing 

hypotheses expounded in the earlier sections. 

The second important aspect, however, is that a least committed future also corresponds to the least biased 

assumption about what an agent is going to do in the future. Not only is this attractive from a purely 

information-theoretical typicality perspective (Cover and Thomas, 1991), more importantly, it corresponds to 

transferring the Jaynesian maximum entropy principle from a simple Bayesian setting to a full-fledged 

Markovian Decision Process model. In the context of CORBYS, where intentions and goals need to be 

inferred, it provides a natural approach to make minimally committed assumptions about the future behaviour 

during generation or during analysis of behaviour. 

12.3 SelfOrganised Behaviour and Goal Generation 

As said above, explicit goals and tasks can be incorporated by modelling the reward structure R(s,s',a) 

explicitly. However, in many cases, goals, tasks and intentions of a biological agent are unclear or ill-defined. 

It is therefore instrumental that one has a model for goal- or task-generation in the cases of unspecified or 

unknown reward structure. 

12.3.1 Generic BehaviourGeneration 

The problem of constructing self-motivated behaviours has been identified already in pioneering work by 

Schmidhuber (1991). Under the framework of artificial curiosity, he identified possible incentives for 

intrinsically driven behaviours without an externally structured reward concept. 

Steels, (2004) proposed the autotelic principle for learning systems. This principle is directly motivated by 

the psychological concept of flow (Csìkszentmihályi, 1978) which aims at achieving an optimal balance 

between effort and ease during the process of engaging in an activity (physical or mental). The idea behind 

the psychological concept is that too little effort creates boredom and frustration, but too much effort will 

create stress and thereby again reduce the effectiveness of the activity. This again leads to frustration, 

however, this time via overload. In the flow state, a subject will experience an optimal match between 

engagement, progress, and feedback. The autotelic principle was an attempt to formalise this principle to 

provide quantitative characteristics that could be implemented in an AI system. 

125


Another approach is based on learning progress (Kaplan and Oudeyer, 2004) which considers the speed by 

which an agent succeeds in learning a given error function. The idea is that it is not the performance in 

reaching a goal that the agent aims at optimising, but the speed by which the agent improves. When a 

saturation process begins as an agent becomes better at achieving a goal, this will cause the agent at the same 

time to improve less and less due to the law of diminishing returns, and the agent will then switch to learn a 

different goal. Thus, the agent will keep looking for goals which have an increased level of novelty in them. 

This philosophy is also pursued in Schmidhuber (2002) where learning is modelled ab initio as a compression 

scheme. Learning progress then directly and universally expresses itself as code growth rate during 

compression of the learning process. The actual improvement is measured by the increase of the effectiveness 

of the compression, not just the length of the compressed output. As an illustratory example, one can consider 

the discovery of new laws of motion which allow to compress laws learnt earlier much more effectively. Such 

compression gradients form the incentive structure of this ab initio model. The problem with this model is 

that it considers only universal Kolmogorov-type compression schemes. On the one hand, they offer various 

optimality guarantees, but, on the other hand, they are only of theoretical relevance due to their strongly 

asymptotical character, i.e. the universal guarantees can only be established for sufficiently long learning runs 

which are typically orders of magnitude outside the range that is available to an artificial or biological agent. 

12.3.2 PrincipleBased SelfMotivated Models 

The approaches discussed in the previous section define generic concepts which can be implemented in 

manifold ways and depend on the particular instantiation of the learning models or compression schemes. 

Principle-based models are now more important where the intrinsic motivation principle is directly embedded 

into and “implemented” by the formalism. 

Strictly speaking, Schmidhuber’s universal compression gradient also belongs in the class of principle-based 

models. However, since the particular compression scheme is not canonically defined, and the formalism 

becomes insensitive to the scheme only in the asymptotic case (which is typically not realisable), we have 

grouped it above together with the approaches characterised by generic concepts rather than concrete 

principles. 

One learning concept is ISO-learning which is based on modelling low-level anticipatory feedback loops (Porr 

et al., 2003). Another important concept for implementing intrinsic self-motivation was the homeokinesis 

concept introduced in (Der et al., 1999, Der, 2000, Der, 2001). Given the embodiment of a concrete agent, 

homeokinetic control is defined by constructing behaviour of an agent in such a way that it maximises 

predictability of its sensoric stimuli in the future. This is achieved by an internal model of the agent that is 

using a learning rule to minimise the predictive error for future stimuli encountered by the agents. 

Importantly, this approach encapsulates the embodiment as a core component of the model. It is only defined 

in the context of the complete sensorimotor loop and elevates the body into a central part of the cognitive 

process, in opposition to many approaches from traditional AI; this perspective thus provides a quantitative 

grounding of the embodied intelligence perspective (Brooks, 1991, Paul, 2006, Pfeifer and Bongard, 2007). 

The early “naive” homeokinesis approach had the problem that it tended to favour situations where the 

prediction for the agent is simple. However, since this has a propensity to send the agent into steady states, 

the model was extended by a mechanism to ensure a rich sensorimotor stimulus spectrum (Der et al., 2006). 

For this, the estimated sensorimotor dynamics of the system is considered as a dynamical system whose 

Lyapunov exponents are estimated. The agent then moves towards states which have the most negative time- 

126


reversed Lyapunov exponents in sensoric stimulus space, while maintaining a high level of predictability; 

these are the maximally unstable (thus structure-rich) points of the sensorimotor dynamics, but the 

predictability still maintains structure in the agent dynamics. With the success and transparent interpretation 

of information theory-based methods (discussed below), the homeokinesis approach was generalised to use 

the information-theoretic language: the “unstable-yet-predictable” concept was translated into the concept of 

predictive information in (Ay et al., 2008). Predictive information (Bialek et al., 2001, Shalizi, 2001) is the 

mutual information that the past of a time series contains about its future. This approach makes it possible to 

convert the original dynamic systems approach into an information-theoretic setting. A high value indicates 

both good predictability, but, at the same time, it is more expressive than the criterium of minimum future 

error, because only sensorily rich pasts can achieve high mutual information values. An impoverished sensory 

past/future relation, even if predictable, does not achieve high predictive information since there is not much 

to predict in this case. The approach can be generalised to incorporate the learning process itself (Still, 2009). 

An alternative approach to utilising predictive information has been exploited in an evolutionary context 

where predictive information has been used to evolve locomotor patterns in simulated snakes with limited 

sensorics (Prokopenko et al., 2006a, Prokopenko et al., 2006b). Here, the dynamics is not adapted online, 

but sensorimotor dynamics are optimised via an evolutionary algorithm to maximize the predictive 

information statistics (using a computationally cheaper Renyi variant of the predictive information). 

The information-theoretic approach is highly versatile. It has found extensions to encompass major classes of 

information processing in sensorimotor loops, ranging from simple minimal models of control and 

encompasses various methods for the formulation of behaviour generation and adaption on the basis of 

information theory (Lungarella and Sporns, 2005, Lungarella and Sporns, 2006, Polani et al., 2007, Klyubin 

et al., 2004, Klyubin et al., 2007). The methodology makes it possible to model many aspects of agentenvironment 

interaction, such as autonomy itself (Bertschinger et al., 2008), or the formation of joint concepts 

in a group of agents (Möller and Polani, 2008). 

Central for CORBYS is the construction of intrinsically self-motivated behaviour and thus a degree of 

autonomy for the agents which should enable them to propose, initiate or estimate behaviours. As a basis for 

this, the universal utility empowerment will be used. Empowerment is the external channel capacity of an 

organism (Klyubin et al., 2005a, Klyubin et al., 2008). It is typically defined as a function over the states (or, 

in more general situations, as context function, see Capdepuy et al. 2007a). It is not, as the case with 

predictive information, a function of the trajectory, rather, a function of state and thus it acts as a universal 

utility. This, in particular, means that empowerment is always defined universally, i.e. in the same generic 

way, depending only on the particular embodiment and the dynamics of the environment; and, by acting as 

utility, the agent aims to maximise this utility in a greedy fashion by hillclimbing through the states guided by 

the local empowerment gradient. 

Empowerment is an expression of the least commitment idea in the absence of a concrete goal or reward. It 

rewards being in states which are least committed with respect to future perturbations or goals (Klyubin et al., 

2008). Empowerment provides intrinsic, embodiment-based saliency criteria for desirable states of the world. 

This includes the identification of states affording novel manipulative degrees of freedom (Klyubin et al., 

2005a) or corresponding to states of maximum centrality in state-action graphs (Anthony et al., 2008), as well 

as gradients for sensory feature adaptation (Klyubin et al., 2005b) and natural points of stability (Klyubin et 

al., 2008). 

Computation of empowerment in continuous scenarios provides an alternative to optimal control models for 

the control of dynamical systems and can avoid the backup of value data throughout the dynamical 

127


programming framework and can allow an online travel of the agent through sensorimotor space towards 

“meaningful” salient states without the necessity of a full exploration (Jung et al., 2011). Furthermore, the 

Lagrange conjugate of empowerment can be taken into consideration, namely the computation of the action 

sequences that achieve the maximum channel capacity. An information bottleneck constraint can then be 

imposed on these action sequences. This procedure highlights dominant proto-behaviours in a self-organised 

way, creating preferred “eigenbehaviour” structures (Anthony et al., 2009). 

12.4 Behaviour Anticipation, Generation and Initiation 

12.4.1 Anticipation 

Anticipation as a concept appears in widely disparate scenarios. It encapsulates implicitly more than mere 

time series prediction inasmuch as it implies preparation for an action to be taken that needs to be set up 

before an event to be most effective. 

As traditional anticipation approach, in Bosman and Poutré (2007), a Genetic Algorithm-based approach to 

carry out stochastic dynamic optimisation in an online fashion is used, which incorporates anticipation of 

future events. Here, the authors essentially formulate the anticipation problem as an optimisation problem 

solved by a Genetic Algorithm, but in an online fashion. 

More explicitly, the preparation aspect becomes clear in an antagonistic anticipation scenario. In Kott and 

Ownby (2005) predictive methodologies are offered to anticipate probable enemy actions. Such a scenario is 

of particular complexity, as one needs to incorporate reasoning about deception; this is identified by the 

authors to render anticipation particularly difficult, in particular in conjunction with the inevitable emotional 

and psychological backdrop under which an antagonistic dynamics takes place. In particular, such a scenario 

requires reasoning about likely “disinformation” attempting to mislead an observer about possible goals: thus, 

belief and intent recognition, deception discovery as well as planning become part of the scenario. For that 

purpose, the authors combine emotion modelling (Gratch and Marsella, 2004) and pheromone-based 

multiagent (Parunak et al., 2004) approaches to explore trajectory spaces to model the complex factors 

involving battlefield operations and the relevant anticipatory tasks. A simpler, MUD-oriented rule-based 

anticipation model for antagonistic scenarios is shown in Darken, (2005). Note, however, that in the context 

of CORBYS, the setting is inherently cooperative, so that there is no necessity to take the specific 

complications of antagonistic scenarios into consideration. 

The distinction between weak and strong anticipation (Dubois, 2003, Stepp and Turvey, 2010) which refers to 

the level at which the future of the system is anticipated by an agent; in weak anticipation, the agent contains a 

predictive model of what the system will be doing in the future, purely based on past observation. In a 

strongly anticipatory system, the agent is for all purposes part of the system, in that it — implicitly — can use 

the future states of a system for the anticipatory action. This can be seen as closely related to the relation 

between excess entropy and statistical complexity (Shalizi, 2001) where the first encompasses all (also 

implicit) relations between past and future, while the latter must necessarily force them through the 

Markovian bottleneck of the present time slice (Shalizi and Crutchfield, 2002) and might therefore be much 

more inefficient. In Nery and Ventura (2010) this distinction is used in the framework of their Event 

Segmentation Theory to segment streams into event-separated continuous segments, thereby bridging the gap 

between continuous state space representation and discrete temporal events when a continuous stream makes a 

transition to another qualitatively different continuous stream. An informational perspective how to extract 

anticipatory events from an event sequence consisting of contingent events with given delays is given in 

Capdepuy et al., (2007b) and coupled with an action selection in Capdepuy et al. (2007c). 

128


A notorious example for anticipatory failure is represented by the anticipation of extreme events (Nadin, 

2005). Such examples are catastrophic events such as earthquakes or financial collapses for which often 

power-law type post-hoc models can be given, but which violate the principle suggested in Nadin (2005) that 

“a model unfolding in faster than real time [would appear] to the observer as an informational future”, in that 

they screen the future from the present observer. A particular challenge is, as the author argues, that for these 

models it is difficult to decide when predictions are appropriate because rare, but large events are 

distinguished by the fact that their occurrence is not just not easily predicted according to models valid in 

more “stable” phases of temporal development, but that the models do not even provide sufficient 

introspection to detect this situation. 

Seizures can be considered “extreme” events in the brain. In Mormann et al. (2006), a review is given over 

existing seizure anticipation models based on EEG data. Methods that have been reported to be more 

successful are based on dynamical systems models such as Lyapunov exponents, accumulated signal energy, 

simulated neuronal cell models or phase synchronisation, however, later analysis seems to put these results in 

doubt and the current state of knowledge is considered inconclusive. 

On the level of intentional actions, however, it has been demonstrated that EMG registers the influence of 

anticipation on future events; in particular, it appears in preparatory postures anticipating voluntary movement 

(Brown and Frank, 1987). The anticipatory movement takes place in response to an expected task e.g. to 

preserve balance in anticipation of a push or pull action. The combination of anticipation with attentional 

mechanisms is believed to be controlled by the prefrontal cortex, by modulating sensory pathways in a “topdown” 

fashion (Liang and Wang, 2003). 

This biological evidence indicates the fundamental relevance of anticipation not only to prepare both active 

behaviour which requires suitable alignment activities, but also from the standpoint of the preparation of 

cognitive processing resources. This is corroborated by studies using principled information-theoretic 

methods (van Dijk et al., 2010, van Dijk and Polani, 2011a). In these, it is assumed that limited informational 

resources are allocated for the working memory keeping track of current goals (i.e. one places constraints on 

goal-relevant information). This minimal assumption gives rise to salient decision transition points for 

behaviour strategies, such as intermediate goals. This model acts as a kind of proto-attentional mechanism. 

In addition, it highlights the link between attention, anticipation, and their emergence from constraints in the 

available informational resources. 

An agent’s actions carried out in the context of goal-directed behaviour must necessarily reveal information 

about its goals and/or purposes. This information, called digested information, can be identified by other 

agents that have the same goal as the first agent (Salge and Polani, 2011). 

For the CORBYS project, the properties discussed in the last two paragraphs are of relevance. The digested 

information principle under which actions reveal information about the agent’s intentions indicate that human 

actions should provide the SOIAA architecture with clues to the intentions of the human. To handle this 

possibly sparse information, it is necessary to impose additional regularisation constraints on its estimation. 

For this purpose, the studies of task structuring by constrained goal-relevant information (van Dijk et al., 

2010, van Dijk and Polani, 2011a) offer natural approaches to regularisation. 

Generally, anticipatory behaviour requires the combination of abilities to predict causally driven as well as 

goal-directed dynamics. The first component aims at predicting a dynamics according to e.g. a “passive” 

physical law, the second addresses the fact that agents, such as humans, are not passively following laws but 

initiate behaviours with certain purposes. The “least commitment” philosophy described earlier provides a 

natural framework to extract information about possible future purposeful trajectories from information about 

129


past observations. 

The emphasis on self-motivated behaviour generation in the section above is grounded in the necessity of 

having a picture of possible candidates for the behaviour of the robot, but also of the human and how to 

organise the goal/initiative transition between them. 

12.4.2 Causality and Information Flow 

Detecting the initiation of behaviour requires the detection of causality and of information flow. For causality 

detection, it is necessary to employ suitable causality models, namely Causal Bayesian Networks (CBN, 

(Pearl, 2000)). These are Bayesian Networks endowed with additional interventional semantics. With this 

interventional model, it is possible to structurally model causal influence. Given observational variables only, 

established algorithms such as IC* can be used for the systematic reconstruction of the causal structures for 

parts of the networks under certain, relatively strong conditions (Pearl, 2000). 

Very recently, a class of methods has been introduced in which the requirements that are necessary for a 

reconstruction of the causal structure of the network have been considerably relaxed (Peters et al., 2009, 

Hoyer et al., 2009, Peters et al., 2010). These methods introduce an additional, weak, assumption, namely 

the conceptual independence of mechanisms and initial conditions, i.e. they consider the CBNs as having node 

distributions which are separately specified from the conditional distributions assigned to the CBN edges. 

This seemingly minor assumption allows an approximate causal reconstruction in many cases where 

traditional methods such as IC* would fail. 

Another class of methodologies for causal reconstruction is based on recent information-theoretic results 

(Steudel and Ay, 2011). The use of information theory to characterise causality has been recognised already 

in Lloyd (1991), however, until recently it has been mostly implemented as an essentially predictive model, 

such as Schreiber’s transfer entropy (Schreiber, 2000), or as minimal non-redundant information transport 

requirements, such as determined by directed information (Massey, 1990). However, a causal picture of 

information flow had already been sketched in Lloyd, (1991), Tononi and Sporns (2003) and fully developed 

in Ay and Polani (2008) and Lizier et al., (2007), where the latter assumes the existence of a time-consistent 

compositionality structure (see also Ay and Wennekers, 2003). 

12.5 Technological Gaps 

Technological gaps exist for the following set of tasks: 

� goal/behaviour generation 

� goal/intention extraction 

� causal information flow estimation 

� transitional dynamics 

On the general level, the gaps that need to be addressed are the transfer to continuous and high-dimensional 

state spaces, the combination of the various levels of the process, as well as the adaptation of the formalisms 

for the particular task of human-robot intentional transfer interactions. 

12.5.1 Goal/Behaviour Generation 

For the goal and behaviour generation, various problems in conjunction with the high-dimensional and 

complex structure of the deployment scenario need to be solved. Methods for high-dimensional 

130


empowerment computation need to be developed, and extended towards online performance. 

It is furthermore necessary to extend the empowerment methodology used in SOIAA by a mechanism to 

introduce externally known saliency points and goals into the empowerment framework. 

12.5.2 Goal/Intention Extraction 

In this context the main gaps consist of suitable regularisation formalisms that make the combination of 

observed behaviour possible with prospective/hypothetical tasks/goals generated in the above step. In 

addition, the formalism needs to work in the continuous environment. 

12.5.3 Causal Information Flow Estimation 

For the identification of the initiative-taker, it is necessary to identify the causal information flow in agenthuman 

interaction. Here, a number of gaps exist. It is necessary to develop methods to determine the causal 

flow from observational continuous high-dimensional data. Online capability is also an existing gap, but it 

may be mitigated by the ability of an on-line algorithm in SOIAA to probe the causal structure of the system 

and thus to turn a purely observational system into a partly interventional system and simplify the flow 

detection. 

12.5.4 Transitional Dynamics 

A gap exists at defining the transitional dynamics shifting between robot-initiative and human initiative can be 

seen as a regularisation task on the level of active behaviour. The gap to be addressed here is the fact that on 

an action level, intermediate actions may not exist, or may lead to undesired states; thus transitions in 

behaviours may need to exhibit phase transitions from one category of admissible behaviours to another. This 

transition needs therefore be aligned with the goals/purposes of the human-robot system as well as with 

feasibility/admissibility criteria. 

13 StateoftheArt in Architectures for Cognitive Robot Control (UB) 

For a long time engineers have been using control systems which use models and feedback loops in order to 

control real-world systems. Limitations of the model based controllers led to research and development of 

intelligent control techniques such as adaptive, fuzzy, neural, genetic and etc. (Aström et al., 1995)(Brown et 

al., 1994)(Passino 1998). Intelligent control tries to emulate biological intelligence to cope with uncertainty. 

It either seeks ideas from human who performs control tasks or borrows ideas from how biological systems 

solve control problems and applies that in order to solve control problems. However, artificial systems are far 

behind the biological systems concerning response generation (motor control). As the need to control 

complex systems increases, it is important to look beyond engineering and computer science as the challenges 

cannot be met by merely improving the software engineering and programming techniques. Rather the 

systems need built-in capabilities to deal with these challenges. Looking at natural intelligent systems, the 

most promising approach for handling them is to equip the systems with more powerful cognitive 

mechanisms. For example, humans are able to process variety of stimuli in parallel, to “filter” those that are 

the most important for a given task to be executed, to create an adequate response in time and to learn new 

motor actions with minimum assistance. This process is called cognitive control in psychology, and it is 

unique to humans and some higher-class animals (Botvinick et al., 2001). In recent years it has been a 

challenge for control engineers to find ways to realise such cognitive control functionality reflecting human’s 

robust sensori-motor mechanisms in robots (Kawamura, 2004). Although a lot of effort has been invested in 

131


research of learning algorithms and processing and perception of sensory data, how to efficiently respond 

(generate motor actions) to changes in dynamic environment is still one of the unsolved problems. This is the 

problem that has been emphasised by the European Commission (European robots, 2009): “Continued 

research is necessary to improve control systems and versatile hardware, particularly for robots designed to 

move around different environments.” 

An unanswered question is “to which extent the cognitive controlled robotic systems should copy control 

structures of human’s motor-sensory system?” It is obvious that in some aspects the human body is much 

more advanced in comparison to robotic systems. However, at the same time technical systems have some 

advantages over the human’s body motor-sensory system such as high resolution of sensors, versatility of 

sensor types and small feedback delays (Kawato, 1999). As robots are not limited to human sensors or 

effectors, e.g. robots can have lasers and wheels and non-human grippers, the robotic systems researchers 

generalise some of the structures found in cognitive architectures, as well as relax the adherence to human 

timing data for the performance of nonhuman sensors or effectors (Benjamin, 2004). 

13.1 Architectures for cognitive control of robotic systems 

Cognitive architectures represent attempts to create unified theories of cognition (Newell, 1990), i.e. theories 

that cover a broad range of cognitive issues, such as attention, memory, problem solving, decision making, 

learning from several aspects including psychology, neuroscience, and computer science. These architectures 

strive to explain, implement and measure a range of human cognitive activities. The specification of a 

cognitive architecture consists of its representational assumptions, the characteristics of its memories, and the 

processes that operate on those memories (Vernon, 2006). There are a number of cognitive architectures. The 

three pre-eminent architectures are EPIC (Kieras and Meyer, 1997), Soar (Lehmann et al., 2006) and ACT-R 

(Anderson et al. 2004). Each of these architectures has achieved a degree of success and is used in one or 

more applications. However, not all existing cognitive architectures can be employed in controlling complex 

robotic systems. For example Benjamin et al., (2004) used Soar to develop ADAPT cognitive architecture for 

robotics claiming that existing cognitive architectures such as Soar, ACT-R and EPIC, do not easily support 

certain mainstream robotics paradigms such as adaptive dynamics. Thus, the ADAPT cognitive architecture 

is a representative of the architectures for whose development the motivation comes from robotics where 

researchers want their robots to exhibit sophisticated behaviours including use of natural language, speech 

recognition, visual understanding, problem solving and learning, in complex analogue environments and in 

real-time. A growing number of robotics researchers have realised that programming robots one task at a time 

is not likely to lead to a robot with such general capabilities, so interest has turned to cognitive robotic 

architectures as a natural way to try to achieve this goal. 

For the purposes of this review the term cognitive architecture will be taken in the general and non-specific 

sense when considering an adequate cognitive architecture which includes elementary building blocks for 

technical cognition and intelligence of the robot. By this we mean the minimal configuration of a robotic 

system that is necessary for the system to exhibit cognitive capabilities and behaviours: the specification of 

the components in a system, their function, and their organisation as a whole. The focus is on cognitive 

architectures designed for controlling complex robotic systems that are supposed to function in dynamic realworld 

environments including humans. This is because two robotic systems that are going to be used as 

demonstrators in the CORBYS project are a mobile gait rehabilitation system to be developed during the 

project lifetime and an existing mobile robot used for contaminated /hazardous environment investigation. The 

first one is tightly coupled with a human, physically and psychologically as a robot gives physical support to 

the human while the human should accept a robot as a part of own body or of own physical abilities. Though 

the second demonstrator is not in direct physical contact with human it, as well as the first demonstrator, has 

132


to “work” synergistically with human in real-time. Therefore, the main focus in this review is on architectures 

for control of cognitive real-time robotic systems. 

While it is clearly impossible to cover all architectures that have been developed, the aim is to present an 

exploratory overview of the pre-eminent architectures that have been successfully utilised for control of 

robotics systems in recent years. As the main objective of the CORBYS project is to design, develop and 

validate integrated generic cognitive robot control architecture, the review will cover several architectures that 

have been implemented in different robotic cognitive systems. The architectures will be analysed related to 

the technologies which have been identified as needed to meet the challenges of nowadays and future robotics. 

The focus will be on the following technologies that will be built on in CORBYS: 

Sensing and Perception: Sensing is transformation of physical entities such as contact, force, sound and etc. 

into internal digital representation. Perception is the extraction of key properties from these digital 

representations and integration of sensory data over time. 

Human-Robot Interaction: Human robot interaction is ability of robotic system to mutually communicate with 

humans. That communication can be multi-modal: voice, physical contact, gestures and different types of 

user interfaces. 

Real-time control: Control system that is able to operate in real time, where real-time operation is defined as: 

“The operating mode of a computer system in which the programs for processing of data arriving from the 

outside are permanently ready, so that their results will be available with the predetermined periods of time; 

the arrival times of the data can be randomly distributed or be already a priori determined depending on the 

different applications” [DIN 44300]. 

Planning: Planning is calculation and selection of actions, motions, paths and missions. 

Learning: Learning is change of robot behaviour based on practice, experience or teaching. 

Communication: This property is concerned with hardware and software communication. Hardware 

communication usually is based on industrial communication interfaces (CAN, Profibus, LIN, RS232,USB 

and others), while software communication between software modules of architecture (often called the 

middleware) is based on different protocols, like RPC (remote procedure call) or CORBA (common object 

request broker architecture). There are two basic approaches in communication between software modules: 

publish-subscribe and client-server. 

System (software) architecture: Architecture defines the structure of system components, their 

interrelationships, and the principles governing their design and evaluation over time. 

13.1.1 System architectures 

Although an “all win” architecture has not yet been developed, recent research has been mainly dedicated to 

the use of hybrid open architectures. These architectures enable sophisticated control in complex 

environments. Hybrid architectures replaced purely reactive or purely deliberative approach to the design of 

robot control architecture as many researchers have argued that neither a completely deliberative nor 

completely reactive approach is suitable for building robotic systems. The purely deliberative architectures 

are known as Sense-Plan-Act (SPA) architectures as they are characterised by sensing or gathering data, 

planning to take new actions based on this data, and acting out these plans. One of the most famous of the 

SPA robots was Shakey, developed in the Stanford Research Institute (Nilsson, 1984). However, in realworld 

applications Shakey’s planning system, like the planning systems of other SPA based robots, was 

133


unable to perform the job in a timely fashion. The system would produce a plan, but before this plan could be 

executed in full, it often became invalidated by changes in the real-world. Because of the inability of SPA 

architectures to perform real-world operations, researchers searched for a robotic control method that did not 

rely on high level reasoning so that the school of reactive and behavioural robotics emerged. This approach 

basically tries to produce “intelligent behaviour” driven by sensory data, the robot reacts intelligently to what 

it senses (Brooks 1990). Architectures from the reactive and behavioural robotics school were typified in their 

rejection of a symbolic representation of the outside world, along with a bottom-up approach to achieving 

complex behaviours (Ross, 2004). Thus, on the one hand SPA robotics failed to produce good experimental 

results because the level of planning and other cognitive tasks attempted was too complex to cope with in a 

real-world environment. On the other hand reactive robotics suffered in another way, in that only immediate 

reactive actions could be performed but not complex tasks which needed to be ’thought about’ ahead of their 

execution as the cognitive functions were not supported. The natural progression in the design of a robot 

control architecture was therefore the development of architectures comprising of both reactive and 

deliberative components. Therefore this overview only concerns architectures which are hybrid in nature. 

The hybrid cognitive architectures are mainly characterised by a layering of capabilities where low-level 

layers provide reactive capabilities, supporting fast perception, control and task execution on a low-level and 

high-level layers provide the more computationally intensive deliberative capabilities including symbolic 

reasoning as a necessity for recognition and interpretation of complex contexts, planning of intrinsic tasks, and 

learning of behaviours. In such a layered architecture, a robot’s control subsystems are arranged into a 

hierarchy with higher layers dealing with information at increasing levels of abstraction. A key problem in 

such architectures is what kind of control framework to embed to manage the interactions between the various 

layers. There exists: 

Horizontal layering-where layers are each directly connected to the sensory input and action output. 

In effect, each layer itself acts like an agent, producing suggestions as to what action to perform. 

Vertical layering-Sensory input and action output are each dealt with by at most one layer each 

Typical three layers architecture based on vertical decomposition of components that supports cognitive 

functions is illustrated in Figure 28. 

Figure 28: The illustration of typical three layers architecture 

Classic example of three layered architecture is ATLANTIS (Gat, 1997) where the above illustrated three 

layers have the following functionalities: 

The Controller: collection of reactive “behaviours” where each behaviour is fast and has minimal 

internal state 

The Sequencer: decides which primitive behaviour to run next; does not do anything that takes a long 

134 

.


time to compute, because the next behaviour must be specified soon 

The Deliberator: slow but smart; can either produce plans for the sequencer, or respond to queries 

from it 

The majority of architectures to follow Gat’s architecture adopt the basic idea of layered architecture but adapt 

it to meet the challenges of complex robotic systems. Also, it should be noted that not all architectures are 

equally hybrid, with many hybrid architectures choosing to omit an explicit planning layer in favour of using 

plan libraries or a behaviour sequencer. These will be illustrated in the next section where several preeminent 

examples are analysed aiming to identify core elements which are essential to providing intelligent 

robot control architecture. 

13.2 Cognitive architectures used for controlling different robotic systems 

13.2.1 Armar – a cognitive architecture for a humanoid robot 

In order to develop architecture that supports fast perception, control and task execution on a low-level as well 

as recognition and interpretation of complex contexts, planning of tasks execution, and learning of behaviours 

at a high-level (Burghart et al.2005) chose a three-layered architecture Armar adapted to the requirements of a 

humanoid robot. It is a mixture of a hierarchical three-layered form on the one hand and a composition of 

behaviour-specific modules on the other hand as described in (Vernon et al. 2006). 

135


System Architecture: 

The layout of Armar architecture is given in Figure 29. It is based on parallel behaviour-based components 

interacting with each other, comprising a three-level hierarchical perception sub-system, a three-level 

hierarchical task handling system, a long-term memory sub-system based on a global knowledge database 

(utilising a variety of representational schemas, including object ontologies and geometric models, Hidden 

Markov Models, and kinematic models), a dialogue manager which mediates between perception and task 

planning, an execution supervisor, and an ‘active models’ short-term memory sub-system to which all levels 

of perception and task management have access. These active models play a central role in the cognitive 

architecture: they are initialised by the global knowledge database and updated by the perceptual sub-system 

and can be autonomously actualised and reorganised. As such Armar is a representative of an architecture 

following the classical three-layers architectures which has a common database shared by all layers 

Figure 29: Armar architecture (Burghart et al.2005) 

Robotic platform: 

Robot prototype described in Burghart et al. (2005) is a humanoid robot with 23 degrees of freedom 

consisting of five subsystems: head, left arm, right arm, torso and a mobile platform. The upper body of the 

robot is modular and light-weight while retaining a similar size and proportion of an average person. The 

control system of the robot is divided into separate modules. Each arm as well as torso, head and mobile 

platform has its own software- and hardware controller module. The head has two DOFs arranged as pan and 

tilt and is equipped with a stereo camera system and a stereo microphone system. Each of the arms has 7 

DOFs and is equipped with 6 DOFs force torque sensors on the wrist. The arms are equipped with 

anthropomorphic five-fingered hands driven by fluidic actuators. The mobile platform of the robot consists of 

a differential wheel pair and two passive supporting wheels. It is equipped with front and rear laser scanners 

and it hosts the power supply and the main part of the computer network. 

136


Figure 30: A humanoid robot Armar (Burghart et al.2005) 

Sensing and Perception: 

The perception sub-system consists of low-, mid-, and high-level perception modules. The low-level 

perception module provides fast interpretation of sensor data without accessing the system knowledge 

database. It provides typically reflex-like low-level robot control. Within this module data coming from 

sensors such as joint position sensors, the force torque sensors located in the robot’s wrists, tactile sensor 

arrays used as artificial sensitive skin, and acoustic data for sound and speech activity detection are processed. 

The low-level perception module communicates with both the mid-level perception module and the task 

execution module via the active models. The mid-level perception module provides a variety of recognition 

components and communicates with both the system knowledge database (long-term memory) as well as the 

active models (short term memory). The high-level perception module provides more sophisticated 

interpretation facilities such as situation recognition, gesture interpretation, movement interpretation, and 

intention prediction. 

Planning: 

The task handling sub-system comprises a three-level hierarchy with task planning, task coordination, and 

task execution levels. Robot tasks are planned on the top symbolic level using task knowledge. A symbolic 

plan consists of a set of actions, represented either by XML-files or Petri nets, and acquired either by learning 

(e.g. through demonstration) or by programming. The task planner interacts with the high-level perception 

module, the (long-term memory) system knowledge database, the task coordination level, and an execution 

supervisor. This execution supervisor is responsible for the final scheduling of the tasks and resource 

management in the robot using Petri nets. 

Control (Execution): 

A sequence of actions generated are passed down to the task coordination level which then coordinates 

(deadlockfree) tasks to be run a the lowest task execution (control) level. In general, during the execution of 

any given task, the task coordination level works independently of the task planning level. 

Communication: 

A dialogue manager, which coordinates communication with users and interpretation of communication 

events, provides a bridge between the perception sub-system and the task sub-system. Its operation is 

effectively cognitive in the sense that it provides the functionality to recognise the intentions and behaviours 

of users. 

Learning: 

A learning sub-system is also incorporated with the early generations of Armar robot learning tasks and action 

sequences off-line by programming by demonstration or tele-operation. On-line learning has been work in 

progress in new generations of Armar (Dietsch, 2011). For instance Armar-III actively investigates its 

137


environment. In fact, entities therein only become semantically useful objects through the actions the robot 

performs on. 

Real-time control: 

Fast reactive components (low-level) and building blocks for recognition or task coordination (midlevel) act 

in real-time. Active models used for real-time perceptions and tasks are retrieved from the global knowledge 

database and stored in a cache memory. In addition, they have a certain a degree of autonomy for adaptation. 

The execution supervisor is responsible for the priority management of tasks and perceptual components. 

13.2.2 ISAC: IMA (Intelligent Machine Architecture) 

The IMA (Intelligent Machine Architecture) was specifically developed to ease the integration of 

heterogeneous software components. The general premise is that all software components can be modelled as 

basic agents which are loosely coupled over a DCOM (Distributed Component object model) connection. 

Here the application of multi-agent based hybrid cognitive architecture IMA in the humanoid robot ISAC 

where all aspects of the robots control system are modelled around agents is presented. It is implemented with 

symbolic components (software agents) which embed different connectionist algorithms. The layout of IMA 

cognitive architecture is depicted on Figure 31. 

Figure 31: IMA multiagent-based cognitive robot architecture (Kawamura et.al. 2004) 

Robotic platform (description, actuation and sensing): 

ISAC is an upper body part humanoid robot that has six DoF arms, pneumatically actuated by 

agonist/antagonist McKibben muscles. Each arm has four finger and thumb. ISAC is equipped with an active 

stereovision system, laser motion detectors, stereo microphones to localise the position of the sound source, 

touch sensors on fingers, proximity sensors on palms and two force sensors between hands and arms. 

System (software) architecture: 

The IMA architecture is collection of software modules (agents). Each IMA agent encapsulates all aspects of 

138


single element (logical or physical) of a hardware component, computational task or data. Software agents 

communicate through asynchronous message passing. The agents are implemented as atomic or compound 

agents. Atomic agents encapsulate a single resource and it does not depend on other agents. Compound 

agents contain or depend on other agents for their primary function. 

Sensing and Perception 

In the IMA architecture each stimulus is processed by a different perception agent (different processing 

techniques have been used for different sensors). To each sensor an IMA agent that processes sensor 

information and stores the result in short-term memory is assigned. For example, there are separate visual 

agents that perform recognition of objects, object localisation, face recognition and etc. The modular 

approach makes it possible to add additional sensors. 

Human-robot Interaction: 

The two software agents are responsible for HRI (Kawamura et al., 2000), one of them is human-agent that 

encapsulates information that robot determined about human and other is self-agent that addresses the 

humanoid’s cognitive aspects. 

Real-time control: 

The initial usage of IMA architecture in the ISAC caused problems with real-time control (Kawamura et. al. 

2004). Asynchronous message passing has caused latency in passing control inputs to real-time controllers. 

Therefore, separate encapsulation of multi-agent tasks (such as visual serving) has been carried out. Head, 

arm and hand agents are responsible for controlling the head, arm and hand actuators. They accept commands 

from one or more clients and carry out command arbitration. The agents also provide the clients with 

information about its current state (for example joint position). The actuation agents pass referent joint angles 

and velocities to servo controllers. The servo control loops have been realised using a QNX real-time 

operating system. 

Planning: 

The self-agent is responsible for planning actions of the robot. The most important modules of the Self-agent 

are Central-Executive-Agent (CEA) and First-Order-Response (FOR). CEA module makes decisions and 

invokes skills necessary for performing the given task that has been selected by Intention-Agent based on 

perceived sensory information. The FOR module is responsible for creating the reactive responses of the 

robot. 

Learning: 

The system is equipped with different types of memories in order to support learning. The memory structure is 

modular and divided into three components: short-term memory, long-term memory and working memory. 

Short-term memory holds most recent sensory information on the current environment in which ISAC robot 

operates. Long-term memory holds learned behaviours, semantic knowledge and past experience and it has 

three structures: semantic, procedural and episodic. Working-memory holds task-specific information that 

authors call “chunks”. The working-memory uses temporal difference learning algorithms and neural network 

to provide learning in IMA architecture. 


The communication between agents is based on asynchronous message passing. Any agent can be accessed 

by any other agent. By increasing the number of agents the system faced with communication “lock-ups” 

13.2.3 iCub 

The cognitive architecture of iCub robot is depicted in Figure 33. The architecture “..comprises a network of 

139


competing and cooperating distributed multi-functional perceptuo-motor circuits, a modulation circuit which 

effects homeostatic action selection by disinhibition of the perceptuo-motor circuits, and a system to effect 

anticipation through perception-action simulation.” (Sandini et al., 2007). 

Robotic platform (description, actuation and sensing): 

The iCub child size humanoid robot with 53 degrees of freedom and it is designed to crawl and sit. His hands 

and arms are designed for dexterous manipulation and it also has visual, vestibular, auditory and haptic sensor 

capabilities. The sensory system consists of: vision, touch, audio and inertial sensors system. 

Figure 32: The layers of iCub architecture (Sandini et al. 2007) 

Figure 33: iCub cognitive architecture (Sandini et al. 2007) 

140


System (software) architecture: 

The iCub architecture is based on YARP (Yet Another Robot Platform) (Metta, 2006)], open-source 

framework that supports distributed computing. Figure 32 shows the layers of the iCub architecture. The 

lowest level API-0 of system architecture is used for accessing hardware components, by formatting and 

unformatting IP packages into appropriate classes and data structures. IP packets are sent to the robot by GBit 

Ethernet connection. High-level cognitive processes are implemented as YARP processes. 

Real time control: 

In order to achieve real-time control of the robotic system, control of joint actuators is carried out by digital 

signal processing (DSP) units that are mounted on robotic platform. The iCub software runs in parallel on a 

distributed system of computers that are connected via Gbit Ethernet with the Hub unit (PC/104) that is also 

located on the robotic platform. 

Planning: 

In the brain, the basal ganglia are responsible for action selection and therefore in modulation circuit is present 

action selection circuit that corresponds to ganglia. 


Multiple CAN bus structure is used to communicate with control/driver and AD cards on the robotic system, 

while Gbit Ethernet is used to communicate data between the robotic system and distributed system of 

computers. YARP architecture supports building robot control system as a collection of programs 

communicating in peer-to-peer way, with broad range of connection types (tcp, udp, multicast, XML/RPC, 

tcpros and etc.) 

13.2.4 CareObot 

System architecture: 

The Care-O-bot System is controlled by a heterogeneous hybrid layered software architecture, which 

combines reactive and deliberative control. It consists of several basic modules, as displayed in the following 

figure. 

Figure 34: Architecture of Care-O-bot (Hans et. al. 2001) 

141


The communication between the user and the robot is realised via the man-machine interface and drives the 

task execution. Corresponding orders are sent to the symbolic planner, which serves as a task planning 

component. It converts orders from the user interface into a list of possible actions, which leads to a global 

goal. Once the list is generated, the execution module chooses a suitable action from the list and sends it to 

the robot control module in order to execute it. Information is shared via a world-model in the database 

module. 

Robotic platform: 

The Care-O-bot 3 system is a highly integrated and compact service robot. Its main components are a mobile 

base, a torso, a manipulator, a tray and a sensor carrier. The system can be seen in the following figure: 

Figure 35: Care-O-bot 

It has altogether 28 DOF, which includes a 7 DOF light-weight arm, equipped with a 7 DOF gripper, and a 5 

DOF sensor head with a laser scanner, stereo-vision cameras and a 3D-TOF-camera. The mobile platform is 

capable of omnidirectional movement and the torso is flexible such that simple gestures can be performed. 

For user-interaction, there are acoustic devices and a touch screen is integrated into the tray. 

Planning: 

The task planning is handled by the symbolic planner at the high-level, which generates a list of actions to 

reach the goals specified by the user, by receiving input from the man-machine-interface and the database. 

The planner for the Care-O-bot system is based on the Action Description Language (ADL). It uses the Fast- 

Forward-planner (FF-planner), working with the Planning Domain Description Language (PDDL). Thus it 

uses a world model and a list of the abilities of the robot, both being saved in the systems database, in order to 

create a list of possible actions that can be accomplished next. 

Real-time Control (Execution) 

In order to realise the possible actions from the list of the symbolic planner, the execution module selects a 

suitable action and executes it. For that purpose, the low-level control of the robot is designed using a realtime 

framework. Depending on the context, open- and closed-loop controllers are realised. 

Reasoning: 

142


The actions, chosen from the list of the symbolic planner, are executed by the execution module. This is 

realised as a BDI (Believe-Desire-Intention)-theoretic agent architecture. The agent is able to determine 

intentions dynamically at runtime, based on known facts, current goals and available plans. Here both topdown 

goal-based reasoning and bottom-up data-driven reasoning are possible. 

Sensing and perception 

Considering the environment perception, sensory data is gathered continuously. This data is further processed 

and interpreted in order to acquire information about the environment. The results are then given to the 

system and possibly displayed to the user through the man-machine-interface. 

13.3 CORBYS enabling potential and constraints (current gaps/shortcomings) 

The following aspects of cognitive robot controlled systems have been identified as points of interest where 

the CORBYS project will contribute to research and development of cognitive controlled robotic systems: 

a) Cognitive control adaptation 

b) take-over/hand-over of goal-setting initiative between robot and external agent 

Cognitive control adaptation 

In reviewed cognitive architectures, the control of actuators is carried out by general control techniques that 

are usually used in robotics (position servo controllers and different form of force controllers) where setpoints 

for real-time (RT) controllers are provided by cognitive modules which initiate action of RT 

controllers. However, one important missing aspect is adaptation of real-time controllers by cognitive 

modules. This will be highly influenced by time latency of cognitive loops and communication paths between 

RT controllers and cognitive modules. In CORBYS both long term and short term adaptation of RT 

controllers will be investigated. 

14 StateoftheArt in Smart integrated actuators (SCHUNK) 

14.1 Introduction to Smart Integrated Actuators 

Smart Actuators are considered to be highly integrated mechatronical units incorporating a motor and the 

complete motion control electronics in one single unit. For reasons of minimised interfacing a smart actuator 

provides a bus communication interface and runs all necessary motion control tasks internally. 

Smart actuators were introduced at the beginning 1990s as microcontrollers reached the necessary computing 

performance for motor control. The progressing microcontroller technology integrating a microprocessor, 

capture compare and timer units enabling PWM control as well as analogue, digital and communication ports 

has started the ongoing process of miniaturisation of sensors and motor control electronics. At the same time 

power electronics developed rapidly by increasing drain current capacity in field effect transistors (especially 

the MOSFET technology) at lower internal resistance as well as by introducing new semiconductor 

technologies like IGBT (insulated gate bipolar transistor). 

Depending on the application and component sizing a smart actuator can also incorporate a speed reducing 

gear head thus enabling the device to provide higher torque output. 

Smart actuators can be distinguished by these criteria: motor technique and control system. 

14.2 Basic Actuator Technologies 

State-of-the-art smart actuators have been presented with fluidic drives as well as with electromagnetic 

143


motors. 

An example of a smart fluidic actuator is the Rotary Elastic Chambers – Actuator developed at IAT in 

Bremen. The modules comprise very few fully integrated components with matched mechanical and 

electrical interfaces: fluidic vane motor, sensors, control elements, as well as an electronic unit and control 

algorithms. 

Figure 36: Rotary Elastic Chambers – Actuator (IAT 

Bremen) 

With integrated control unit 

144 

Figure 37: Flexible Fluidic actuator (AIA KIT, Karlsruhe) 

Shows the working principle of the vane motor 

Another fluidic actuator principle has been presented by Festo AG. The “fluidic muscle” provides a linear 

type of actuation, but does not include a control unit within the actuator. 

Other smart actuator solutions are mostly based on electromagnetic motors. The motor techniques used here 

are stepper motors, brushed DC motors and permanent magnet synchronous motors. 

Dimensioning an appropriate electromagnetic motor for a given application influences the choice of the motor 

technique. The most important criteria are housing size, weight, scalability, price, delivery standards but also 

costs resulting from motor control electronics and software. Depending on the motor choice the expenditure 

on drive control can be very different. Therefore some advantages and disadvantages of the named motors 

techniques are explained. It must be considered that smart actuators in most cases are also being used for 

measurement of actual data like position, speed, temperature and motor current. 

14.2.1 DC motor 

Until some years ago mechanically commutated DC motors had been used for highly dynamical servo drives. 

The negative features of this motor technique are based on the principle design where the rotating coil is 

centred inside the motor housing and heat is unable to dissipate appropriately. In addition the mechanical 

commutation limits the stall current but also the current at higher rotation speed because of brush fire. Low 

cost DC motors have limited life time because of brush wear. Brush fire can cause high frequency noise 

causing trouble in the power supply of other components connected. 

14.2.2 Stepper motor 

Stepper motors are an inexpensive alternative for small power drives (


motor for the application planned. 

14.2.3 Permanent magnet synchronous motor 

In applications with small and medium power drives permanent magnet synchronous motors (PMSM) have 

replaced standard DC motors today. This is due to the advantages PMSM provide, e.g. service free, high 

torque and overload capacity because of the missing mechanical commutation. In addition PMSM are better 

able to dissipate heat since the stator windings are oriented to the housing. This results in a high ratio of 

torque to housing size. On the other hand motor control for PMSM is highly complex and expensive. 

14.3 Control techniques, Interfacing, standardised drive modules 

The control techniques for electromagnetic motors have been widely discussed in science. Vector modulation 

is the state of the art for PMSM making precise motor current control possible. The available feedback 

systems are based on incremental encoders or resolvers. Absolute encoders have been widely introduced, but 

are not available as standard components for small sized drives yet. 

Most smart actuator solutions are based on simple two-wire bus communication such as the CAN bus, 

Profibus or other field bus systems. Recently Ethernet based smart actuators have been presented. In order to 

provide real time interfacing new transport layer protocols have been introduced according to the OSI/ISO 

layer model, e.g. Ethercat, Ethernet/IP, Profinet or SercosIII. 

Figure 38: Dynamixel smart actuators including reduction gear, controller, motor and driver 

The power supply in most cases is based on DC power (typically 24VDC), because most smart actuators have 

been optimised for small dimensions not allowing for voltage conversion. Some of the smart actuator 

products follow a modular approach by standardising flange sizes, torque capacity, electrical interfacing and 

controls. A modular system is scalable and provides an easier way to extend the application for further use or 

reuse of the actuators and sensors for new applications. 

145


Figure 39: SCHUNK modular smart actuator system with rotary and linear drives 

as well as gripper modules 

14.4 CORBYS enabling potential and constraints (current gaps/shortcomings) 

With regards to the CORBYS project there are a few criteria to discuss on the dimensions of the motors and 

control subsystem. 

Miniaturisation is a major issue because many drives must be incorporated into a relatively small 

demonstrator (gait rehabilitation). In order to move human joints in an exoskeleton high torque motors with 

minimised dimensions are needed. An actuator providing a high torque – at low weight is the resulting 

requirement for the highest power density. Another aspect for dimensioning the drive is heat. The drives will 

need to be placed near the human skin and must therefore be limited in heat emission. Both demonstrators are 

mobile systems powered by batteries. This leads to several requirements with regards to power stability and 

tolerance for a wide voltage range (typically 19... 30VDC). The safety and reliability is a clear issue, because 

the user must expect highest robustness during their training sessions. The drives must provide a high short 

time overload capacity, because smaller motors with small power consumption and small size would assist in 

the dimensioning of the actuators. Minimising friction in motor and gear heads is another important issue 

helping to significantly improve the sensibility of the drive. 

The cycle time aspect is based on the controls system coordinating the actuator motion independent of the 

training program and the sensor feedback system. Ideally, an open control architecture is foreseen allowing 

the control system to access the drive data and parameters at all times and on different levels. Finally minimal 

noise is desired in order to enable better acceptance by the human user. Price issues need to be discussed from 

a dissemination point of view. 

14.5 Technology innovation requirements Gaps filter elements 

Analysing the state of the art in actuator technology revealed several gaps. The first to name is integration of 

actuators with requirements as described above in a dimension limited mobile device with battery power. In 

order to keep battery lifetime high the complete system must be optimised for power consumption and high 

efficiency drive. The high power density of the system leads to possible sensitivity in data transfer, bus 

communication or malfunction due to power loss. The actuators need to provide variable stiffness, depending 

on the current control and training scheme. At the same time safety is important to reduce the risk of injuries. 

For easy service a modular approach and a simplified interfacing with electrical connectors is preferred. 

146


15 StateoftheArt in NonInvasive Brain Computer Interface (BBT) 

The objective of CORBYS project is to develop the underlying design principles of a cognitive robotic system 

with the focus on augmenting human locomotion capabilities. One of the key points of these principles is the 

way that the human and the robot interact and share their cognitive capabilities. In this area, the large 

majority of research was concentrated in a uni-directional way, where the human delivers orders to the 

machine responsible for the execution of the commands. The focus has been on exploring and improving the 

way that robots understand natural human expressions (e.g. understanding natural language, gestures, etc). 

However, very little research has concentrated on a mutual exchange of cognitive information. This is 

because this information mainly resides in the human brain and progress was still required in brain computer 

interfacing. A few years ago there was much research in understanding how to decode this information from 

brain imaging techniques, leading to one of the objectives of CORBYS: to use brain computer interface 

technology to online decode human cognitive information. This information will be used to build a natural 

way of communication between the human and the robot, and to guide the design and operation principles of 

the cognitive robot. 

In this context the Brain-computer interfaces (BCI) emerge as systems that make it possible to translate the 

electrical activity of the brain in real-time using commands to control devices. They do not rely on muscular 

activity and can therefore provide communication and control for people with devastating neuromuscular 

disorders such as the amyotrophic lateral sclerosis, brainstem stroke, cerebral palsy and spinal cord injury. It 

has been shown that these patients are able to achieve EEG controlled cursor, limb movement, and prosthesis 

control and have even successfully communicated by means of a BCI (Birbaumer et al, 1999; Hochberg et al, 

2006; Buch et al, 2008). 

The remainder of this Deliverable reviews current non-invasive BCI technology applied to the control of 

robots focussing on the CORBYS technology gaps and innovations. Subsequently, the main components of a 

BCI system, Hardware, Software, Basic Signal Processing and Decoding will be described in the following 

sections, showing the state of the art, the constraints and the respective technology innovations. 

15.1 Invasive vs. NonInvasive BCI Technology and Robotics 

Recently there has been a great surge in research and development of brain-controlled devices for 

rehabilitation. The most significant characteristic of these systems is the use of invasive or non-invasive 

methods to record brain activity. Invasive techniques require a clinic intervention to implant the electrodes on 

the cortex, while non-invasive methods place the sensors outside the skull (e.g., Electroencephalogram EEG). 

The research arena is dominated by the US for invasive techniques and research in animals, while the EU 

leads in the development of non-invasive techniques with direct application on humans. On the one hand, US 

teams have achieved significant results using invasive recording methods to control artificial devices. A rat, 

for example, was able to move a robotic arm in one dimension to get water (Chapin et al, 1994), monkeys 

were trained to move a cursor on a screen and adjust its size (Carmena et al, 2003) and to self-feed using a 

robotic arm with a gripper, to grab food in three dimensions (Velliste et al, 2008), and a patient had sensors 

implanted directly in the cortex and learned to open and close a prosthetic hand and control a simple robotic 

arm in two dimensions (Hochberg et al, 2006). These achievements show that direct and online control of a 

robotic device is possible with invasive brain recordings. However, these settings involve technical and 

clinical difficulties for humans, such as the maintenance of the electrodes in the cortex, infection risks, and 

damage to the brain (Mc Farland & J.R.Wolpaw, 2008). 

On the other hand, in line with CORBYS, much research in BCI has focused on non-invasive recording 

147


methods to be accessible for a wide range of users. Currently, Electroencephalography (EEG) is the most 

widely used technology. Following the human non-invasive brain-actuated robot control demonstrated in 

2004 (Millán et al, 2004), research has addressed control of other rehabilitation devices such as wheelchairs, 

robotic arms, small-size humanoids, and even teleoperation robots for telepresence applications (some of them 

developed by members of the Consortium). The majority of research in brain computer interfaces for robot 

control use non-invasive methods to record brain activity, and have explored the way that humans can deliver 

control orders to the machine using brain waves. For example, wheelchairs (Iturrate et al, 2009; Luth et al, 

2007;, Rebsamen et al, 2007) were driven by using a BCI that detects the using steady-state potentials or P300 

evoked potentials, or with detection of mental tasks (Vanacker et al, 2007). Other results focused on the 

motion of a robotic arm in two dimensions using motor imagery (Mc Farland & Wolpaw, 2008), controlling 

the opening and closing of a hand orthosis (Pfurtscheller et al, 2000) with sensory motor rhythms, using motor 

imagery to move a neuroprosthesis (Muller-Putz et al, 2005), using P300 potentials to control a humanoid 

robot (Bell et al, 2008), and the teleoperation of a mobile robot remotely located to develop navigation and 

exploration also with P300 potentials (Escolano, 2009). However, none of these projects has explored the 

type of human cognitive information that could be extracted in this process and how it could be used at all the 

autonomy levels of the robotic system. 

Technological Gaps in Merging Non-Invasive BCI and Robotics. Nevertheless, we are still far from any 

successful deployment of non-invasive brain-controlled devices, which is due to the fact that the already 

mentioned devices share common shortcomings that require substantial scientific advances: 

1. The mental protocol for robot control is not natural for the user, i.e., the user’s intention is not explicit 

in the control. For example, in one of the wheelchairs, the user had to concentrate on rotating a 3D 

figure to turn right or on complex arithmetic operations to turn left. 

2. There is no mutual self-adaptation between the human and the controlled device. Usually, adaptation 

is considered only at BCI level to take into account variability in brain activity across subjects and 

time, but there is not a mutual adaptation of the human to the robot and vice versa. 

3. There is a lack of general and modular software architecture that successfully integrates the different 

BCI technologies, and that also complies with the requirements of hardware and software robotic 

architectures. This is important for large scale or integration projects. 

4. The working scenarios are very simple controlled situations given the current state-of-the-art in 

autonomous robotics. For instance, the most complex tasks achieved are to open or close a robotic 

hand or robot navigation in a two dimensional world. 

Innovation in CORBYS 

In robotic related rehabilitation programs and in many other robot contexts it has been suggested that human 

cognitive processes, such as motor intention, attention, and higher level motivational states are important 

factors with potential to build a natural and increased cognitive interaction with the robot (Tee et al, 2008). 

The possible innovations of CORBYS are to build a BCI to decode and detect motor intentions in real-time 

and a rich cognitive information to be used in the subsequent levels of the robot hierarchy. This general 

objective needs innovation in several areas of brain computer interfacing, such as the development of signal 

processing and machine learning techniques in order to detect in real-time neural processes preceding 

movement and of cognitive processes related to the human execution of the task. In addition to this, a brain 

computer software architecture will have to be developed as an integration tool for the BCI system with interconnections 

with all the modules of the project. The previous aspect, plus other more related to the 

148


deployment of the application (human locomotion rehabilitation) will be addressed in the next sections. 

15.2 Hardware for Noninvasive BrainComputer Interfaces 

Neural activity produces not only electrical activity but also other types such as magnetic and metabolic 

changes that can also be measured non-invasively. However, this type of measurement requires 

magnetoencefalography MEG (for the magnetic fields); and positron emission tomography PET, functional 

magnetic resonance fMRI and optical imaging (for the metabolic activity). These measurement techniques 

require sophisticated devices and facilities out of scope of a technology focused on a wide range of people. In 

addition to this, these methods are still technically demanding and costly, and most of them also have low 

temporal resolution which is a strong limitation for rapid communication (i.e. information transfer rate) being 

one of the most important features of communication devices. 

Focusing on the electrical brain activity, the electroencephalogram (EEG) is a sensory system that places 

several sensors in the surface of the scalp. These sensors allow the scalp recordings of neural activity in the 

brain by measuring the potential changes over time between a signal electrode and a reference one. An extra 

third electrode (ground) is used for getting differential voltage. Minimal EEG configuration consist of one 

active electrode, one reference electrode and one ground electrode (Teplan, 2002). These measurements are 

amplified and digitalised, and subsequently transferred to the computer. The EEG is the most accepted brain 

recording technique for brain-computer interfacing, since it is relatively cheap, portable, easy to use and has a 

low set-up cost. For these reasons, in line with the BCI community, the EEG is most recommended recording 

technology for CORBYS. However, there are two main dimensions that affect the selection of the type of 

EEG technology: the sensor technology and the sensor acquisition system. 

On the one hand, the sensors are the bottleneck of today non invasive BCI since they need long time 

preparation due to the use of conductive gel. Standard gel based EEG electrodes are passive electrodes, which 

require the skin to be cleaned, brushed, scraped in order to reduce their impedance and achieve sufficient 

signal quality. This abrasive skin treatment makes the subject uncomfortable and, when measured very 

repeatedly, exposed them to the risk of irritation, pain and infection (Teplan, 2002). Moreover, the use of 

electrolytic gel may cause electrical short between two electrodes in close proximity when the montage is not 

correctly carried out. To reduce time preparation, the active electrodes were developed which reduce artefacts 

and signal noise resulting from high impedance between the electrode(s) and the skin. The system setup 

becomes faster but the conductive gel is still used. Due to the previous reasons these wet sensors are not 

suitable for daily use in normal living environments, and in the last few years there has been an increasing 

interest in developing dry electrodes (they eliminate the need of conductive gel). These dry electrodes are 

based on a completely different sensory technology. The main features of both, wet and dry electrodes are 

listed below: 

1. Contact impedance: wet electrodes showed a value that is the half of the dry one (Searle & Kirkup, 2010). 

2. Static interference: the influence of an electrically charged object moving near recording electrodes (nonstationary 

electric fields) was analysed. Dry electrodes were tested in unshielded and shielded conditions, 

showing in both case smaller interference levels than wet types. In particular, dry electrodes, when 

shielding was employed, showed an interference level 40 dB lower than that experienced by wet 

electrodes (Searle & Kirkup, 2010). 

3. Motion artefact: change in potential at the electrode/skin interface, due to skin movement, caused 

unwanted signals. Movement tests were conducted showing for dry electrodes an artefact value 20 dB 

higher than that for wet types at the commencement of the trials. Whereas at the end of the trial period, 

149


due to a reduction in skin/electrode interface effects for dry electrodes over the length of the trial, RMS 

artefact values for dry electrodes were 8.2 dB lower than wet electrodes. This outcome may be attributed 

to the geometry of the dry electrodes housing (Searle & Kirkup, 2010). 

4. Usability: wet electrodes with its lengthy application time, subject discomfort and irritation, the need for 

extensive caregiver support and training, have made the technology impractical. Instead the dry electrode 

with its easy and reliable application in a home environment and its use without any skin preparation or 

gel application, may help the widespread use of BCI technology (Mason, 2005). 

5. Stability: the wet electrode signal tends to become unstable over time due to the use of conductive gel, it 

dries during prolonged measurements reducing signal quality and signal performance. On the other hand 

dry electrodes are like wet types at the beginning but become more stable with ongoing time (Matteucci et 

al, 2007). 

6. Portability: typical EEG equipments using wet electrodes are not mobile, they have high power 

consumption and a mass of wires connecting the cap to a PC that process, store, display and analyse the 

signals. However, the requirements for power, control, and read-out are reduced in dry electrodes, making 

a more feasible a wireless solution (Sullivan et al, 2008). 

Some of these features have been corroborated by research teams that have developed working prototypes of 

dry EEG sensors, and demonstrated that the signal obtained can be largely comparable to wet electrodes 

(Popescu, 2007). They have also been tested in BCI paradigms such as alpha/mu rhythms and flash visual 

evoked potential (FVEP). They showed a high correlation with signals recorded in parallel with wet standard 

electrodes in the alpha/mu rhythms experiment and a negligible difference in the FVEP experiment (Gargiulo 

et al, 2010). In addition to the previous results, the N100 auditory evoked potential, the auditory evoked P300 

event related potentials and the sensory-motor rhythm (SMR) for 2-class motor imagery were also tested. Dry 

electrodes showed a signal close to the one recorded with gel-based electrodes between at least 7 Hz and 44 

Hz Bristle-sensors (Grozea et al, 2011). 

150


Figure 40: EEG Dry Devices 

On the other hand, clinical acquisition systems for dry devices are not available on the market, instead, several 

commercial offerings have been made that could fulfil BCI requirements. An important key feature is that 

these systems are focussed on every day use of EEG equipment over a long-term period. For that reason, 

particular attention is paid to the headset design which has to be lightweight, comfortable and at the same time 

ensures coverage in the area of interest. One of the main key points of these systems is that they are portable 

since they incorporate wireless connection link with a computer. For instance, the Enobio system (Figure 

40(b)) developed by Starlab, is a device with 4 dry active digital electrodes placed on the forehead, it shows 

similar responses compared to Active Two system from Biosemi (gel based device) (Cester et al, 2008; Riera 

et al, 2008;, Starlab, n.d). A Neural Impulse Actuator (NIA) (Figure 40(e)) is also available, a OCZ 

Technology product, it uses three carbon-fibre sensors placed in the forehead (OCZ Technology, n.d); instead, 

NeuroSky provides with MindSet (Figure 40(d)) a sensor system with just one single dry sensor at EEG 

position FP1 (International 10-20 system) (NeuroSky, n.d). These are wearable wireless devices, however all 

of them have constrained in EEG research possibilities since the sensor location do not cover the areas of 

interest in the BCI community (like the sensory-motor cortex) and they sense the active in the prefrontal areas 

focalising in mental activity such as attention and meditation. So far, Epoch (Figure 40(a)), commercialised 

by Emotiv, is the most complete in fulfilling BCI requirements: it collects data from 14 saline sensors (they 

eliminate the electrolytic gel of wet electrodes but a saline solution has to be used) placed in a wearable 

wireless neuroheadset, located at AF3, AF4, F3, F4, F7, F8 FC5, FC6, T7, T8, P7, P8, O1 and O2. P300 

experiments were already conducted (Campbell et al, 2010). Unfortunately the Epoch neuroheadset does not 

support experiments involving motor cortex (brain area widely used in BCI) (Emotiv, n.d). 

151


The g.SAHARA (Figure 41(a)) electrode system, a g.tec product, consists of an 8 pin electrode made of a 

special golden alloy. The golden alloy and the 8 pins reduce the electrode-skin impedance. EEG recordings 

are performed at frontal, central, parietal and occipital regions of the head (a mechanical system is required 

that holds the electrode to the skin with a constant pressure at every possible recording location) (g.tec 

Technologies, n.d). The g.SAHARA can be used in combination with the g.MOBIlab+ (Figure 41(b)), a 

portable biosignal acquisition and analysis system. 

Figure 41: Dry and portable recording EEG system provided from g.tech: dry electrodes (left) 

and a portable bioginal acquisition and analysis system (right) 

15.3 Software for Noninvasive BrainComputer Interfaces 

Although the field of brain-computer interfaces is relatively young, significant research has been developed in 

the design of models and components to control interfaces and devices (see [Berger, 2008] for review). 

Despite this being a key point for the development and deployment of BCI-controlled devices, very little work 

has been devoted to design general, modular, and flexible software architectures for BCI (Mason et al, 2007). 

On one hand, the major existing BCI frameworks are very constrained in terms of usability and scalability. 

Some of these frameworks have been developed by private companies as small demonstrators of their 

technologies, being basically assessors of their provided hardware (g.tec Technologies, n.d; BioSig Project, 

n.d). Very specific open-source systems exist for biomedical signal processing and biosignal analysis, such as 

Matlab toolboxes (Delorme & Makeig, 2004) or software libraries (BioSig Project, n.d). 

The OpenEEG (OpenEEG, n.d) is an open-source project that promotes the creation of amateur software for 

neurofeedback using OpenEEG hardware designs. OpenEEG software is very specific and simple, but with a 

lack of a common architecture, incomplete documentation and that has not been extensively tested 

(OpenEEG, n.d). In addition, some other companies have developed proprietary software architectures for 

neurofeedback (BioEra, n.d.; BioExplorer, n.d.; Soft-dynamics, n.d.; Zengar Institute Inc., n.d.). Recently, 

other three software platforms have been proposed: BCI++ (Maggi et al, 2008), BF++ (Brainterface, n.d.) and 

OpenViBE (Renard, 2010). The first one is a C/C++ framework for designing a BCI system, it also includes 

some 2D/3D features for BCI feedback. The second one is another C++ framework originally oriented 

towards neurofeedback applications, although later versions can support any kind of BCI system. The latter 

has rapidly emerged in the last few years, it has a set of software modules that can be easily and efficiently 

integrated to design BCI for both real and virtual environments. It is highly modular and operates 

independently from the different software targets and hardware devices. Anyway its graphical interface is still 

not intuitive and usable for non-programmers users. On the other hand, BCI2000 (Schalk et al, 2004) is the 

result of a joint effort of several laboratories and is currently the most utilised worldwide software platform 

for BCI. 

152


High-level Tools: Usability is a crucial part for a BCI software platform and its acceptance and spread relies 

on how easy-to-use the platform is for both non-programmers and developers. As a result, such a platform 

should provide its users with high-level tools featured by flexibility to customise the software and build 

different applications, as well as usability for non-programmers. As illustrated in Table 4, some examples 

exist. However, many of them require programming skills (e.g. code skeleton generators, Matlab analysis 

tools), while those aimed at non-programmers are too complex and limited (e.g. scene editors). The 

innovation at this point would be to develop graphical tools in a user-centred approach and minimise third 

party dependencies. Facing the use cases of both programmers and non-programmers, the major requirements 

would be captured. Those requirements must lead the development of this kind of tools to obtain powerful as 

well as usable software applications that support flexible BCI designs. 

Technological Gaps in Merging Non-Invasive BCI and Robotics: A software analysis of BCI systems of 

standard requirements (Mason et al, 2007) applied to this platform reveals, however, some limitations when 

used as a general software architecture and in particular in the context of the robotic project CORBYS: 

1. Recording software: only one recording technique can be used at a time for one application (e.g., EEG or 

MEG). This makes complex testing or validating a complementary approach to control devices. 

2. Only one type of neural paradigm can be processed at a time: As a result, it is not possible to use multiple 

control channels simultaneously (e.g., simultaneous robot control and online robot error recognition). 

3. It is a one-processing technique: a single signal-processing module can be processed simultaneously, 

which is a limitation in applications with redundant processing modules (for a given neural paradigm) or 

with parallel processing (of several paradigms as in the previous example). 

4. It is single application software: only one device can be controlled at a time, which prevents the 

development of applications where humans control several devices simultaneously (e.g., two arms for 

manipulation). 

5. It does not have interaction functionalities with other software architectures. This type of interface will 

open the possibility of interaction with other systems and re-using many existing algorithms already 

present in robotics architectures such as Stage/Player (Gerkey et al, 2003), OROCOS (Bruyninckx, 2011), 

CARMEN (Montemerlo et al, 2003), ROS (ROS.org, n.d.) or CoolBOT (CoolBOT Project, n.d.) . 

6. Non portable and single-platform: it relies on third party component for compilation and execution (it can 

only be compiled in Borland under Windows operating system). 

7. Lack of high-level tools: it does not provide graphical tools or software components aimed at nonprogramming 

users in order to facilitate construction and customisation of BCI applications. 

Table 4: Main platform comparison 

153


Table 4 shows a brief comparison between the main BCI frameworks to-date. It illustrates some drawbacks 

such as portability, in some cases due to third party dependencies (e.g. Matlab or development environments), 

or a lack of usable high-level tools aimed at non-programmers (those offered generate weakly customisable 

BCIs or are too complex). These drawbacks are a limitation for future developments in BCI research and its 

applications. Furthermore, to the best of our knowledge, there are no programming abstraction and support 

tools to facilitate modularity and flexible integration for this development. This is another limitation for large 

scale projects and for non-programming experts. 

Innovation in CORBYS: In this context, the innovation required in CORBYS would be to design a software 

architecture for BCI research and development that overcomes these constraints. This platform will be a 

software that supports multi-neural paradigm, multi-signal processing, and multi-application (which covers 

some of the limitations of existing BCI platforms). In addition to this, the new software architecture would be 

portable and multi-platform (supporting embedded systems for distributed intelligence in general-purpose 

micro-controllers), scalable and distributed (to balance workload and hardware capabilities), with real-time 

functionalities and with inter-connectivity to support collaborative applications for robotic platforms. These 

features are important for CORBYS, since the key point is to provide a framework to decode several mental 

processes simultaneously while providing a real-time integration framework portable and with connectivity 

with the robotic system. 

154


Figure 42: Example EEG artefacts 

15.4 The Role of EEG Artefacts in NonInvasive BCIs 

One of the most important aspects in biomedical signal processing is to acquire knowledge about noise and 

artefacts in order to minimise them. Artefacts are undesirable signals that can occur in the signal acquisition 

process and interfere with neurological phenomena. They may change the characteristics of EEG signal, 

limiting the accurate evaluation of the running brain processes, and be even incorrectly used as the source 

control in BCI systems (Fatourechi et al, 2007). This is the reason why it is of upmost importance to develop 

automatic computer based methods that handle with artefacts, and this technology is a fundamental 

requirement for a BCI system (Srnmo & Laguna, 2005). In EEG recordings (and as a consequence in EEG- 

BCI systems) a wide range of artefacts can occur. While some of these artefacts can be easily identified, there 

are others that may have similar characteristics to the neural activity and result extremely difficult to 

recognise. One possible categorisation of artefacts is based on their origin: technical (originated from outside 

the human body, such as the 50/60 Hz power-line noise, changes in electrode impedances, etc); and 

physiological (arising from a variety of bodily activities, such as potentials introduced by eye or body 

movements, muscular activity, cardiac activity, etc. Below (figure 4) some examples of artefacts are shown. 

155


As mentioned before, the artefacts may change the characteristics of EEG signal, limiting the accurate 

evaluation of the running brain processes, and be even incorrectly used as the source control in BCI systems. 

This is the reason why several automatic methods for artefact processing have been developed in the BCI 

literature. 

On the one hand there is the artefact Rejection to be taken into account, which is based on the rejection of 

contaminated trial, implicating loss of valuable data (Ramoser et al, 2000; Millán et al, 2002). Due to the very 

large number of undesirable signal in BCI systems, not all the contaminated trials can be rejected. Usually the 

most artefact affected epochs are excluded from the analysis. Therefore, the “cleaned” data are not 

completely free of artefacts. This methodology is only usable for offline analysis. In online real-time 

applications of a BCI system it is not possible to have time periods when the artefact contaminated signals are 

rejected, and as a consequence the BCI system cannot be used to control the device. 

On the other hand, there is the artefact removal, where the objective of these techniques is to remove the 

artefacts as much as possible while keeping the related neurological phenomenon intact. There are several 

types of artefact removal techniques: 

1. Linear filtering: used to remove artefacts located in frequency bands that are not useful for the 

application of interest (Barlow, 1984; Ives & Schomer, 1988). Low-pass and high-pass filtering can be 

used to remove EMG and EOG artefacts respectively. The main advantage of this method is its 

simplicity, however this fails when neurological phenomena and the artefacts overlap or lie in the same 

frequency (Geetha & Geethalakshmi, 2011). 

2. Linear combination and regression: a common technique for removing ocular artefacts from EEG signals 

(Croft & Barry, 2000), it uses a linear combination of the EOG contaminated EEG signal and the EOG 

signal. One problem of this approach is that subtracting EOG signal from the EEG one may also remove 

part of it. Regression techniques can be used to remove head-movement, jaw clenching and saliva 

swallowing artefacts (Geetha & Geethalakshmi, 2011). 

3. Principal component analysis: strictly related to the mathematical technique of singular value 

decomposition (SVD), it requires uncorrelation between the artefacts and the EEG signal. This method 

has been reported to be not completely efficient with EOG, EMG and ECG artefacts, especially when 

they have comparable amplitude to the EEG signal (Lagerlund et al, 1997; Geetha & Geethalakshmi, 

2011). 

4. Blind source separation: a technique generally based on a wide class of unsupervised learning 

algorithms, it identifies the components that are attributed to artefacts and reconstruct the EEG signal 

without their contribution. Independent component analysis (ICA) is the most utilised (Choi et al, 2005). 

This method has been widely used to remove ocular artefacts, and also EMG and ECG artefacts in 

clinical studies. Its main advantage is that it does not rely on the availability of reference artefacts, 

however it usually needs prior visual inspection to identify artefact components (Geetha & 

Geethalakshmi, 2011). 

5. Others: wavelet transform (Browne & Cutmore, 2002), nonlinear adaptive filtering (He et al, 2004) and 

source dipole analysis (SDA) (Berg & Scherg, 1994), even if they have so far a limited application in 

BCI systems. 

156


Technological Gaps in Artefacting for Non-Invasive BCI and Robotics: Historically, EEG signals have 

been considered noise prone, and even if human cognition often occurs during dynamic motor action most 

EEG studies examine subjects in static conditions (e.g. seated). However, the CORBYS project requires the 

patient to be walking during the use of the BCI system. During locomotion a large number of mechanical 

artefacts arise, which are associated for instance with head movements and can have amplitudes that are one 

order of magnitude greater than the corresponding brain related EEG signal. These kinds of artefacts are nonstationary, 

since the kinematics and kinetics of human walking exhibit both short-term (step to step) and longterm 

(over many steps) variability (Gwin et al, 2010). 

Very few studies have been realised on EEG during human locomotion. These include recording brain 

activity during pedalling a stationary bicycle (Jain, 2009), and during walking and running on a treadmill 

(Gwin et al, 2010). 

In the latter, the EEG signals of eight healthy volunteers were recorded from 248 active electrodes during a 

visual oddball discrimination task while simultaneously they walked and ran on a treadmill. A two-stage 

approach was used to remove locomotion artefacts: first a channel-based template regression procedure was 

applied, and then an Infomax independent component analysis (ICA) followed by a component-based 

template regression. Results showed that during walking condition locomotion the artefacts slightly 

contaminate the EEG signals in an event-related paradigm. However, during running conditions, the EEG 

signals are strongly affected by movement artefacts. The artefact removal technique implemented in the study 

was successfully applied separating brain EEG signals from gait related noise. This is believed to be the first 

study of EEG and event-related potentials from a cognitive task recorded during human locomotion. Its 

results show the feasibility of removing gait related movement artefact from EEG signals. However, notice 

that the type of EEG signals that this method separates are event-related responses, which are of different 

nature that the process that the CORBYS project will address (spontaneous activity). As is well-known in the 

BCI community, the EEG event-related responses are much more robust features (since they are more 

stationary) that the ones of the spontaneous EEG to be detected and classified. Then, it is expected that the 

impact of the EEG artefacts during locomotion will be much stronger in the CORBYS context than in the 

previous study. 

Required innovation in CORBYS The innovation that CORBYS will require is to develop signal processing 

techniques to address and correctly filter EEG artefacts acquired during human locomotion. Notice that these 

techniques will have to accommodate the non-stationary nature of the spontaneous process that will be used in 

CORBYS. In addition to this, these techniques for movement artefacts removal will need to support online 

analysis with a low calibration time, they will need to work with a low number of electrodes (maximum 

number of 16), and will need to require low computational resources. This innovation step will be challenging 

and crucial for the deployment of the BCI in the project, since the complete performance of the BCI will 

strongly depend on the precision and robustness achieved. 

15.5 Decoding the Cognitive Process Required in CORBYS 

As mentioned in the introduction of this document, the general objective of CORBYS needs innovation in 

several areas of brain computer interfacing, such as the development of signal processing and machine 

learning techniques in order to detect in real-time neural processes preceding movement and of cognitive 

processes (such as the feedback potentials and the attention) related to the human execution of the task. 

15.5.1 EEG Decoding of Motor Intentions 

Several studies have demonstrated the appearance of EEG activity preceding human voluntary movement. 

157


These signals are associated to motor task preparation, and dissimilar to those during the actual execution, in 

Figure 43 EEG activities related to preparation and execution are shown. The intention to move is related 

with at least three cortical activities, specifically, the readiness potential (or Bereitschaftspotential -BP), the 

contingent negative variation (CNV) and the event-related (de)synchronisation (ERD/ERS). 

Figure 43: Voluntary hand movement phases: motion intention, 

motion execution and motion end 

The readiness potential starts approximately 2.0 s before the movement onset and, as pointed out in the first 

reports (Kornhuber & Deecke, 1964, 1965) and in subsequent studies (Deecke et al, 1969, 1976; Kutas & 

Donchin, 1980), rapidly increases its gradient about 400ms before the muscular activity. It is characterised by 

a maximal activity in the midline centro-parietal area and by a symmetrical and wide scalp distribution 

independently of the site of movement. The BP occurrence, in respect to the movement onset, differs among 

subjects and movement conditions. The early and the late segment of the BP are called “early BP” and “late 

BP“ respectively due to their different scalp distribution. The late BP, for its asymmetric distribution 

associated with unilateral hand movement, has been studied as the lateralised readiness potential (LRP) (Coles 

et al, 1988). 

The contingent negative variation, a similar slow negativity preceding movements, occurs during the interval 

between a warning stimulus (S1) and a subsequent imperative stimulus (S2) before the movement onset 

(Walter et al, 1974; Tecce, 1972). The earlier part of the CNV can be distinguished from the later one (or 

terminal CNV -tCNV) based on the different scalp distribution. The former is generated in response to S1 and 

it is maximal over the frontal cortex; the latter can start up to 1.5 s before S2 and it is characterised by a 

maximal activity over the frontal and prefrontal cortices indicating cognitive preparation, and over the primary 

motor cortex (M1) and supplementary motor area (SMA) reflecting motor preparation (Rohrbaugh et al, 1976; 

Rosahl & Knight, 1995; Ikeda et al, 1996). 

The event-related desynchronisation/synchronisation (ERD/ERS) reflect the power changes of EEG 

oscillatory activity of various frequency bands, they are related to various tasks including voluntary 

movement. In the case of motor task preparation alpha (8-13 Hz) and beta (14-30 Hz) are the frequency bands 

to analysed (Shibasaki & Hallett, 2006). The ERD, corresponding to a power decrease, is related to increased 

neural activity of the associated cortical area, while the ERS, corresponding to a power increase, is associated 

with decreased activation (Pfurtscheller et al, 1997). The ERD starts about 1.5 s before the movement onset 

and it is localised over the contralateral precentral and postcentral area of the scalp. At approximately 750- 

158


500 ms before the muscular activity desyncronization with relevant magnitude can be measured over the 

ipsilateral scalp as well. Instead, ERS starts about 1 s prior to the movement over the occipital cortex 

(Bastiaansen et al, 1999). 

Several research labs have analysed these EEG signals. Most of them have been focusing on upper 

extremities pre-movement signals, mainly in the discrimination between right and left hand. Just few of them 

have been investigating the lower extremities (e.g. legs). So far there are no studies in discriminating between 

right and left leg. The following list provides an insight into a representative variety of works related to premotor 

potentials that are currently being pursued in research groups: 

� Morash and his group utilised CNV and ERD/ERS to predict which of four movements (right-hand 

squeeze, left hand squeeze, tongue press on the roof of the mouth, and right foot toe curl) was about to 

occur (Morash et al, 2008). EEG signals were recorded from eight, right-handed, healthy, BCI naive 

subjects using 29 channels over sensorimotor areas. The movement was instructed with a specfic 

stimulus (S1) and performed at a second stimulus (S2). A spatial filter (ICA) and a temporal filter 

(DWT) were used in the pre-processing while an off-line evaluation was done using a naive Bayesian 

(BSC) classifier. An average classification accuracy of 40% was reached. The results of this study 

suggest that the ERD/ERSD is the most specific neural signal preceding movements, related to the 

particular movement. 

� Pfurtscheller and his group analysed similar intention of movements (left index finger, right index 

finger and right foot) to show the feasibility of automatic methods for the selection of optimal 

electrode position and optimal number of channels for an EEG-based brain state classifier (Peters et 

al, 2001). Data collected from three healthy subjects recorded with 56 electrodes, developed and 

presented in a previous study (Pfurtscheller et al, 1994), were used. Spatial and frequency filtering 

were applied, especially for the former several techniques were compared showing common average 

reference (CAR) and Laplace filter as the best. Classification was performed using an artificial neural 

network (ANN) with three perceptrons for each channel, where its output is an ”opinion“ in the 

majority voting process for the automatic selection. High classification accuracy was obtained for 

left/right index finger discrimination (93%) and for left/right index finger and right foot 

discrimination (89%). Anyway the data were collected from subjects who were prompted to specific 

tasks by a computer. 

� The Berlin Brain Computer Interface Group has given an important contribution with various studies. 

Blankertz utilised the LRP to predict single-trial EEG potentials preceding voluntary finger movement 

(Blankertz et al, 2003). Eight healthy subjects were instructed to press keyboard keys in a self-chosen 

order and timing. The signal was first filtered using a Fourier transform filtering technique and after 

classified with a linear classifier (LDA). They managed to predict the laterality of imminent hand 

finger movements (right vs. left), and demonstrate the possibility to achieve good accuracy even at 

fast motor command rates (2 taps per second) with a single-trial EEG paradigm. The latter result is 

very relevant, since it takes into account fast motor sequences condition, and start to analyse how 

after-effects of one movement superimpose on the preparation of a consecutive movement (offline). 

In a similar experimental setting Blankertz compared different kind of classifiers, Support Vector 

Machines (SVMs) and variants of Fisher Discriminant, reaching high accuracy level (>96%). Beyond 

the previous classifier a second one was trained, to distinguish movements events from the rest 

(Blankertz et al, 2002). Krauledat showed the possibility of using the LRP in motor task classification 

even in time critical context (Krauledat et al, 2004). The EEG was recorded with 27 up to 120 

electrodes while the subject had to respond as quickly as possible with finger movements to different 

159


stimuli. After a band-pass filtering relying on the fast Fourier transform a LDA classification, to 

discriminate pre-movement potentials of left vs. right hand finger movements, was performed. The 

LRP curves showed a similar shape in spontaneous motor activity and in critical situations. 

Technological Gaps and related CORBYS innovation: As mentioned before, most of existing works have 

focused on upper extremities pre-movement signals, mainly in the discrimination between the right and left 

hand. Just few of them have been investigating the lower extremities (e.g. legs). So far there are no studies in 

discriminating between right and left leg. CORBYS will progress the state-of-the-art in building a real-time 

anticipatory BCI based on the intention of motion of the human feet, distinguishing between the movement of 

the left and right legs. A fully integrated and online anticipatory BCI that will improve current recognition 

rates and will widen the scope of known neural protocols that can be used in these BCIs will be developed. It 

is noticed that the use of intention/execution of motor imagery as a protocol for BCIs is attracting interest in 

robot mediated therapies, since it has been demonstrated that during motor execution, attention to task related 

features of the movement has an effect on motor performance (Tee et al, 2008). This cognitive information 

has an impact in the robotic scenario of CORBYS since the information about the intention of feet movements 

will be used by high-level modules of cognitive control architecture for perception. Though the development 

of the CORBYS anticipatory BCI will be driven by the CORBYS demonstrator this BCI technique is not 

limited only to robot-assisted rehabilitation but it can be applied to any other robot-control application. 

15.5.2 EEG Decoding of Feedback Potentials 

In neuroscience and neuropsychophysiology it is well known that the usage of event-related brain potentials 

(ERPs), evoked responses elicited by the presence of an internal or external event (Wolpaw et al, 2002), to 

study the underlying mechanisms of human error processing and monitoring (Ferrez & Millán, 2004). 

Different types of errors have been described such as when a subject performs a choice reaction task under 

time pressure and realises that he has committed an error (Christ et al, 2000) (response ErrPs); when the 

subject is given feedback indicating that he has committed an error (Nieuwenhuis et al, 2004) (feedback or 

reinforcement ErrPs); when the subject perceives an error committed by another person (observation ErrPs) 

(Coles et al, 2004); or when the subject delivers an order and the machine executes another one (Ferrez & 

Millán, 2004) (interaction ErrP). Recently it has been demonstrated that the errors are also elicited when the 

subject perceives an error in the robot operation in both simulated and real scenarios (Iturrate et al, 2010). In 

addition, several works have shown that it is possible to use signal processing and machine learning 

techniques to perform automatic single trial classification of these ErrPs (Ferrez & Millán, 2004; Iturrate et al, 

2010; Dal Seno, 2009). 

Feedback is usually an event perceived by a person or an animal as a return of an executed task, given as a 

result of a conduct that was or not appropriate. Humans’ learning mainly depends on the ability to distinguish 

between positive and negative feedback, this discrimination is detectable in the brain activity (Nieuwenhuis et 

al, 2004). Furthermore, it is known that some skills do not develop properly if feedback inputs are absent. In 

the last few years, therapists have used positive/negative feedback to improve their practice and the 

motivation of patients whose advance is slow (Tee et al, 2008). Recently, in the field of Brain-Computer 

Interfaces (BCI), there has been an increasing interest in their online detection. This is because they carry 

information to measure indirect parameters of the human learning process that could be used to maximise the 

performance of the therapeutic strategy (Wolpaw et al, 2002). 

Technological Gaps and related CORBYS innovation There are very few works in the design of a BCI 

system for feedback ERPs online classification (Lopez et al, 2010). The experimental protocol used was the 

same as proposed in (Miltner et al, 1997), where the subjects were required to estimate a certain amount of 

time (1 s) pushing a button when they believed that the time had elapsed. After each response a feedback 

160


stimulus was provided indicating the accuracy of the previous estimation. Each trial starts with a warning cue. 

Brain activity of five healthy subjects was measured using 32 electrodes placed at FP1, FP2, F7, F8, F3, F4, 

T7, T8, C3, C4, P7, P8, P3, P4, O1, O2, AF3, AF4, FC5, FC6, FC1, FC2, CP5, CP6, CP1, CP2, Fz, FCz, Cz, 

CPz, Pz and Oz. The signals were classified employing a Support Vector Machines (SVM) with radial basis 

function. This study analysed the requirements of the classifier in terms of the amount of training data, its 

performance among sessions and the possibility of fast re-training in order to achieve good performances 

using data from previous sessions. Also an online analysis of the data was performed showing an average 

recognition rate of (78%). CORBYS will progress the state-of-the-art by designing a real-time detection 

system of feedback errors in robotic applications, analysing the different feedback modalities (e.g. visual, 

auditory, vibrotactile, etc) that best suit for the robotic rehabilitation scenario defined. 

15.5.3 EEG Decoding of Attentional States 

Cognitive processes are produced and controlled within the central nervous system (CNS), accordingly brain 

and physiological activity of the body reflect these states. Cognitive states change the patterns of 

physiological signals (e.g. heart rate, skin temperature, respiration, etc.), several biosensors have been used to 

identify them; stress, relaxation and exhaustion conditions were analysed the most often (Shi et al, 2007; Zhai 

& Barreto, 2006; Kulic & Croft, 2007). Over the last few decades several studies have put in evidence the 

relation between attention, or other relevant mental conditions, and EEG spectral features. For instance, in 

Jung et al, (1997) a power spectrum estimation was combined with principal component analysis (PCA) and 

artificial neural networks to estimate a local error rate in a sustained attention task. Others, instead, have 

focused on specific EEG rhythms. In Kelly et al, (2003) and Huang et al, (2007) alpha band power was 

examined to investigate the attentional demands and the brain dynamics following vehicle deviation in 

sustained attention tasks, respectively. In addition alpha, the gamma band, with frequencies greater than 30 

Hz, was analysed to determine its enhancement during a visual spatial attention task (i.e. moving-bar-like 

paradigm) (Gruber et all, 1999). In Haufler et al, (2000), log-transformed EEG power spectral estimates for 

various frequency bands were compared during a selfpaced visuospatial task from skilled marksmen and 

novice gunmen. Beyond attention, increased in alpha (Foster, 1990; Lindsley, 1952; Brown, 1970) and theta 

powers have been interpreted as a signs of relaxation (Teplan et al, 2009), in Teplan et al, (2006) this was 

shown during long term audio-visual stimulation. Furthermore, related to attention, clinical studies were 

conduced on Attention Deficit Hyperactivity Disorder (ADHD), suggesting that theta/beta self-regulation 

reduces its symptoms (Barry et al, 2003; Monastra et al, 2005), quantifying the deficit (Clarke et al, 2001; 

Koehler et al, 2009) and represents the basis of the neurofeedback treatment (Lubar, 1991; Linden et al, 1996; 

Friel, 2007). Recently, there is evidence that the states of attention and non attention can be discriminated 

achieving up to 89% classification accuracy rate in an online environment using a novel approach to extract, 

select and learn EEG spectral-spatial patterns. This new approach combines advanced signal processing and 

machine learning: the filtering pre-processing step consists of two stage, a filter-bank (FB) and common 

spatial patterns (CSP) filters, while a mutual information technique selecting best features with a linear 

classifier were applied to measure the attention level (Hamadicharef et al, 2009). Aside from visual attention, 

attentional modulation of auditory event-related potentials was reported. Listening to two concurrent auditory 

stimuli, the event-related EEG is modulated by the user selective attention to one or the other (Hillyard et al, 

1973; Ntnen, 1982, 1990). These results were exploited to develop a BCI paradigm, in which the subject 

could make a binary choice by focusing its attention (Hill et al, 2004). 

Technological Gaps and related CORBYS innovation: During motor execution, attention to movement 

task related features plays a fundamental role affecting motor performance (Ingram et al, 200; Zachry et al, 

2005). Furthermore, several studies have investigated in the attention function during the learning process of 

novel sensorimotor transformation and the adaptation process to novel force perturbations, reporting its 

161


importance in learning a novel motor task (Lang & Bastion, 2002). These results encouraged a focus on the 

role of attention in pattern recovery for unhealthy subjects (e.g. post-stroke motor rehabilitation (Blanc-Garin, 

1994)). In a robotic rehabilitation scenario repeated and unchallenging stimuli may decrease attention to 

sensory information in the motor control loop (Tee et al, 2008). These outcomes suggest that motor relearning 

in robotic rehabilitation can be made more optimal by measuring the patient’s attention during the tasks and 

consequently modulating the feedback, increasing attention to task relevant features. CORBYS will make 

progress in real-time recognition of the degree of attention and stress/relaxation from EEG data while the user 

is interacting and operating with the robot using existing models from clinical and psycho/physiological areas 

to find new correlations with the biosensors. 

162


StateoftheArt Relevant to the two Demonstrator Application Domains 

16 StateoftheArt in Gait Rehabilitation Systems (VUB) 

Gait training, over ground or on a treadmill, has become an essential part of rehabilitation therapy in patients 

suffering from gait impairment caused by disorders such as stroke, spinal cord injury, multiple sclerosis and 

Parkinson's disease. Its effectiveness is increasingly evidenced by clinical trials and advancements in 

neuroscience. Seemingly trivial, the notion of “(re)learning to walk by walking” hides some of the key 

research questions that puzzle not only the field of rehabilitation science. 

Similar to the neurological principles underlying human walking itself, the principles underlying motor 

learning and neural recovery are not yet fully understood and are the subject of ongoing research. As a 

consequence, research efforts in the field are focused on quantifying the rehabilitation process and identifying 

rehabilitation practice that maximises outcome. In one of the existing practices, body-weight supported 

treadmill training (BWSTT), the patient's body weight is partially supported by an overhead harness while 

his/her lower limb movements are assisted by one up to three physiotherapists. The strenuous physical effort 

encumbering the therapists and the resulting short training session duration was one of the main reasons for 

introducing robotics into gait rehabilitation. Although this introduction was envisaged by therapists as well, it 

was mainly driven by engineering, strengthened by technological advancements in robotics and prior research 

into powered exoskeletons for humans. The advantages that were initially aimed at by automating therapy, 

namely enhancing intensity, repeatability, accuracy and quantification of therapy, are indeed easily associated 

with robotics. However, a robot operating in close physical contact with an impaired human requires an 

approach to robot performance that differs significantly from the viewpoint of industrial robotics. Accurate 

repeated motion imposed by a position controlled robot is considered contraproductive for many reasons: a 

lack of adaptable and function specific assistance, a limitation of the learning environment, reduced 

motivation and effort by the patient. Nowadays, the field of rehabilitation robotics is increasingly convinced 

by a human-centred approach in which robot performance is focused on how the robot physically interacts 

with the patient. 

A focus on the human in the robot puts emphasis on the adaptability and task specificity of robotic assistance 

required to achieve “assistance-as-needed”. At the same time, safety of inter-action, preventing harm and 

discomfort, is mandatory. Variable stiffness or variable impedance is a promising concept in robot design and 

control that addresses both safety and adaptability of physical human-robot interaction (pHRI). It implies that 

the robot gives way to human interaction torques to a desired and adjustable extent. This adds to the high 

requirements that were already imposed by the application, for instance with regard to wearability (compact 

and light weight design, adjustable to the individual) and actuator performance (high torque output, high 

power-to-weight ratio). Hence, in the development of novel prototypes rehabilitation roboticists are faced 

with the challenge of combining suitable design concepts, high performance actuator technologies and 

dedicated control strategies in view of improved physical human-robot interaction. Improvement, that should 

lead to a better insight into the effects and effectiveness of robot-assisted rehabilitation and ultimately, leads to 

better therapies. 

16.1 Gait rehabilitation 

In persons with damage to the central nervous system, for instance due to stroke (brain damage) or incomplete 

SCI (spinal cord damage), task-specific and intensive gait training leads to (partial) recovery of motor 

163


function and improved gait (Barbeau and Rossignol, 1994; Dietz et al.,1994; Hesse et al., 1995). In addition, 

there are several secondary positive effects on the physical and mental condition of these patients (Hidler et 

al., 2008). Gait training prevents many of the secondary complications that often result from neurological 

injury and gait impairment (e.g. joint stiffening, muscle atrophy, cardiovascular deterioration, pneumonia, 

deep venous trombosis). Therefore, treadmill training is a well-established practice nowadays in rehabilitation 

centres for the neurologically impaired. 

The driving force behind neural recovery is neural plasticity, the ability of neural circuits, both in the brain 

and the spinal cord, to reorganise or change function (Elbert et al., 1994). This process was clearly evidenced 

in prior animal research, revealing the existence of so called “central pattern generators” at the level of the 

spinal cord that allow to reinstill animal gait through training following spinal cord lesion. However, neural 

recovery has proven to be much more complex in humans, as human walking involves both spinal control and 

brain control (Yang and Gorassini, 2006). For rehabilitation to be successful neural plasticity should thus be 

maximally promoted. Although the mechanisms underlying neural recovery are not yet fully understood, 

there is a growing consensus about the major enabling principles. Sensory input from the muscles and joints 

to the central nervous system (afferent input) is crucial (Ridding and Rothwell, 1999; Harkema, 2001). Also, 

these sensory cues should match as closely as possible with those normally involved in the task to be 

relearned. Some critical cues of human locomotion have been established, but are subject of ongoing research 

(Behrman and Harkema, 2000). Another important requirement for recovery is that training should be 

intensive and that it should be started as early as possible after the injury to maximise outcome (Sinkjaer and 

Popovic, 2005). The need for intensive training and the aim of relieving therapists from the physical strain 

induced by manually assisted gait training, triggered the application of robotic assistance to gait rehabilitation. 

The development and use of gait rehabilitation robots both in rehabilitation practice and in research labs has 

strengthened the validity of some concepts that are believed to underlie gait retraining in general and also to 

increase the effectiveness of robot-assisted training itself. A key finding is that assisting movements (too 

much) may result in reduced effort and decreased motor learning (Marchal-Crespo and Reinkensmeyer, 2008). 

This is evidenced by motor learning studies in unimpaired subjects involving robotic assistance to learn a 

movement task, and appears to apply to neural recovery as well. Some studies explored the benefits of 

amplifying movement errors instead of correcting them, which was found to improve short term motor 

learning, as reported for instance in Reisman et al., 2007. It was also shown that the training should be 

adapted to the skills of the subject: similar to providing too much assistance, providing too little is 

counterproductive (Emken et al., 2007). Another important aspect to training is active participation, which is 

promoted by motivation. The robotic training environment should trigger the subject to self-initiate and 

actively contribute to the movements and also to sustain efforts (Lotze et al., 2003). The suggestion that effort 

may be more important than (robotic) assistance, questions the rationale behind the use of robots in movement 

therapy (Reinkensmeyer et al., 2007). Nonetheless, many rationales for using robotic assistance in gait 

rehabilitation can be found in literature (Guglielmelli et al. 2009; Marchal-Crespo and Reinkensmeyer, 2009). 

Previously unexplored movements provide novel sensory information to the patient, assistance makes gait 

training more safe and intense, and helping the patient accomplish desired movements is an important 

motivating factor (an extensive overview can be found in Marchal-Crespo and Reinkensmeyer (2009)). 

The aforementioned concepts are encompassed by a best practice in assistance-based robotic therapy 

commonly referred to by “assistance-as-needed”, implying that the robot should assist only as much as needed 

and only where needed. Hence, the level of assistance should be adaptable and task (or function) specific. 

Newly developed robot technology for gait rehabilitation is increasingly focused on this paradigm. The 

following section puts emphasis on general concepts and hardware. 

164


Figure 44: Gait rehabilitation robots: end-effector based (e.g. HapticWalker) 

and exoskeleton based (e.g. Lokomat®). Related applications supporting the development of exoskeleton based gait 

rehabilitation robots: human performance augmenting exoskeletons (e.g. HAL), assistive exoskeletons (e.g. ReWalk®), 

powered prosthetics (e.g. C-leg®) 

16.2 Gait rehabilitation robots 

In the course of merely ten years, the number of rehabilitation devices, and from a broader perspective, the 

advancements in assistive technology, have grown remarkably. Although common challenges are faced in the 

development of robots for the upper limbs, this overview is limited to devices for the lower limbs. Similarly 

to rehabilitation robots for the upper limb, gait rehabilitation robots can be categorised according to their 

underlying kinematic concept into end-effector based and exoskeleton based robots (Guglielmelli et al., 2009). 

End-effector based robots interact with the human body in a single point (through their end-effector), whereas 

exoskeleton based robots interact with the human body in different points across human joints. The latter 

typically have an anthropomorphic, serial linkage type structure that acts in parallel with the lower limbs. 

Seldom, there are gait training devices not belonging to any of these two categories. String-man, a device 

consisting of tensioned wires attached to the body is an example (Surdilovic et al., 2007). 

16.2.1 Endeffector type devices 

Commercially available end-effector type devices are the GT1 Gait Trainer and its successor G-EO 

(Rehastim, Germany). These are based on a doubled crank and rocker gear system, driving two 

programmable footplates, generating gait-like movements of the lower limbs (Hesse and Uhlenbrock, 2000). 

165


The HapticWalker is based on the same concept of permanent foot-machine contact 

Figure 45: End-effector type devices: (from left to right) HapticWalker®, G-EO®, ARTHuR 

(Schmidt et al., 2005a). This concept is also typically found in parallel type rehabilitation robots with a 

platform for rehabilitation of the ankle/foot and for balance training as for instance in Saglia et al. (2010); 

Yoon and Ryu (2005). ARTHuR is a unilateral 2 DOF device using a back driveable two-coil linear motor 

and a pair of lightweight linkages to drive a footplate (Emken et al., 2006). It has been used primarily to study 

motor learning principles and to evaluate a teach-and-replay procedure with impedance adaptation (Emken et 

al., 2008). 

16.2.2 Exoskeleton type devices 

Most gait rehabilitation robots are exoskeleton based and prototype development in this type of device is often 

supported and stimulated by advancements in related applications: assistive exoskeletons, human performance 

augmenting exoskeletons and powered prosthetics (Figure 46). Rehabilitation exoskeletons, assistive 

exoskeletons and human performance augmenting exoskeletons are the three main types of powered 

exoskeletons for humans and the past decade has seen a multitude of research prototypes and devices of this 

sort. As their common rationale is the assistance of human gait, these exoskeletons are not easily categorised 

and sometimes assistive exoskeletons and human performance augmenting exoskeletons find their way to a 

rehabilitation setting. Also, from an engineering viewpoint, these applications have common requirements 

with respect to the performance of the actuators, the weight and compactness of the structure and the 

wearability of the design. There remains however a clear distinction of basic functionality. Rehabilitation 

exoskeletons are aimed at recovery of impaired function, whereas assistive exoskeletons assist impaired 

function and human performance augmenting exoskeletons augment sound function. Powered prosthetics 

replace lost function. Although acting in series with the human body instead of in parallel, powered lower 

limb prostheses also inspire the development of gait rehabilitation exoskeletons, as they require high 

performance actuators, gait phase detection and user-oriented control. An overview of the state-of-the-art in 

powered lower limb prosthetics can be found in Martin et al., 2010; Versluys et al., 2009. 

16.2.2.1 Commercially available devices 

To date there are two commercially available exoskeleton-based gait rehabilitation robots: the AutoAmbulator 

(Healthsouth, US) and the Lokomat (Hocoma, Switzerland). Both devices consist of a treadmill, an overhead 

suspension system with a harness and a robotic orthosis attached to the patient's lower limbs, assisting the hip 

and the knee bilaterally. The Lokomat, in particular, has undergone substantial testing with patients and, as 

opposed to AutoAmbulator, is extensively reported in literature. Lokomat, originally purely position 

166


controlled, uses ball screw actuators and joint-space impedance control to achieve naturalistic joint trajectories 

at the hip and knee. Various patient-cooperative control strategies have been investigated (Jezernik et al., 

2004; Duschau-Wicke et al., 2008) as well as a hardware extension of the system with additional actuated 

DOF (ab/adduction of the hip and lateral and vertical pelvic displacement, see Bernhardt et al., 2005a), but 

most functionalities were not transferred to the device that is currently on the market and in use in 

rehabilitation centres. The KineAssist (Kinea Design, US), having no lower body structure, is primarily 

intended for adaptable body weight support and walking balance training of stroke patients. 

16.2.2.2 Research prototypes 

Figure 46: Commercially available exoskeleton type devices: 

(from left to right) Lokomat®, AutoAmbulator®, KineAssist® 

Several research groups recognised the need for assistance-as-needed control strategies and for more 

physiological gait movements, both considered essential to increase the effectiveness of robot-assisted gait 

training. Most research efforts are focused on introducing adaptable compliance (or variable impedance) into 

the hardware and/or the control of the system and on extending the number of DOF of the exoskeleton, i. e. 

active DOF (actuated) and/or passive DOF (passive elements or none). 

Bilateral prototypes In LOPES, besides flexion/extension of the knee and hip, lateral and forward/ backward 

displacement of the pelvis and abduction/adduction of the hip are assisted (Veneman et al., 2007). Bowdencable 

based series elastic actuators are used to power the exoskeleton's joints for reasons of inherent safety and 

force tracking performance (Veneman et al., 2005). The device is intended for use in stroke patients and focus 

is on task-specificity of assistance by means of virtual model control (Ekkelenkamp et al., 2007). PAM and 

POGO use pneumatic cylinders to compliantly assist five out of six DOF of the pelvis and flexion/extension 

of the knee and hip. Zero-force control and impedance control are used consecutively in a teach-and-replay 

procedure (Aoyagi et al., 2007). Both LOPES and PAM/POGO are treadmill based devices. The 

WalkTrainer (Stauffer et al., 2009) is a mobile overground walking device, that consists of a mobile base with 

an active body weight supporting harness, a pelvic orthosis (6 actuated DOF) and two leg orthoses (3 actuated 

DOF each) (Allemand et al., 2009). It combines task-space impedance control of the orthoses with closedloop 

functional electrical stimulation (FES) of the paraplegic patient's leg muscles. Unilateral and single-joint 

prototypes in addition to bilateral prototypes, several unilateral rehabilitation exoskeletons, comprising one or 

more powered joints, have been developed. ALEX is a leg exoskeleton of which the hip and knee joint are 

actuated by linear drives (Banala et al., 2009). A force field controller is implemented in task-space that 

displays a position dependent force field acting on the foot. In Sawicki et al. (2005) ankle-foot and kneeankle-foot 

orthoses powered by McKibben type pneumatic muscles are investigated for task-specific 

167


rehabilitation. This actuator type has also been implemented in, amongst others, a bilateral prototype reported 

in Costa and Caldwell (2006) and in an ankle rehabilitation device for stroke patients in combination with 

springs (spring over muscle actuator) reported in Bharadwaj and Sugar (2006). A different type of pneumatic 

muscle, the pleated pneumatic artificial muscle, is used in KNEXO, a powered knee exoskeleton controlled by 

an interaction-oriented trajectory controller (Beyl et al., 2009). 

Figure 47: Bilateral prototypes: (from left to right) LOPES, PAM/POGO, WalkTrainer 

Figure 48: Unilateral and single joint prototypes: 

(from left to right) ALEX, Ankle foot orthoses of University of Michigan, KNEXO, SUE 

SERKA is an active knee orthosis for gait training focusing on stiff knee gait in stroke patients (Sulzer et al., 

2009). It is driven by a rotational series elastic actuator, capable of providing nearly zero to high assistive 

torque, while keeping the added mass low by means of remote actuation through Bowden cables. AKROD is 

a knee orthosis with an electro-rheological fluid (ERF) variable damper component to correct hyperextension 

of the knee and stiff knee gait in stroke patients (Weinberg et al., 2007). Entirely passive systems have been 

developed as well. The gravity balancing orthosis (GBO) compensates the gravitational torques acting at the 

hip and the knee of the combined system (orthosis and leg) during swing by means of a dedicated spring 

mechanism (Banala et al., 2006). SUE is a passive bilateral exoskeleton with torsion springs in the hip and 

168


knee joints optimised for propulsion of the legs during swing in treadmill walking (Mankala et al., 2007). 

16.2.2.3 Assistive exoskeletons 

Also in the field of assistive exoskeletons, a multitude of devices and prototypes have been developed, some 

of them also envisaged for use in rehabilitation or performance augmentation. The ReWalk (Argo Medical 

Technologies, Israel) is a bilateral robotic suit for the mobility impaired that is near to being released to the 

market. Some exoskeletons are specifically aimed at assisting the elderly, such as the walker based 

exoskeleton EXPOS reported in Kong and Jeon (2006) and its successor SUBAR (Kong et al., 2009), others 

focus entirely on body weight support, such as the Moonwalker (Krut et al. (2010)) and the Bodyweight 

Support Assist by Honda. A combination of a quasi-passive exoskeleton with functional electrical stimulation 

(FES) is proposed in Farris et al. (2009). Many single joint exoskeletons have been developed. The DCO 

(Hitt et al., 2007) and the AAFO (Blaya and Herr, 2004) are examples of active ankle foot orthoses making 

use of series-elastic actuators to assist in push-off or to correct dropped foot gait. 

Figure 49: Assistive exoskeletons: (from left to right) ReWalk., Body Weight Support Assist, SUBAR 

Figure 50: Power augmenting exoskeletons: (from left to right) BLEEX, Sarcos Exoskeleton, 

MIT's Quasi-passive Leg Exoskeleton, HAL 

16.2.2.4 Human performance augmenting exoskeletons 

The majority of human performance augmenting exoskeletons for the lower limbs has been designed for load 

169


carrying augmentation (e.g. carrying a backpack) with military applications in mind, such as BLEEX 

(Kazerooni and Steger, 2006), the Sarcos exoskeleton (Sarcos, US) and NTU exoskeleton (Low et al., 2005). 

Their control strategies are dedicated to unimpaired users. A quasi-passive leg exoskeleton, using a fraction 

of the power consumed by the aforementioned devices, is reported in Walsh et al. (2007). The robot suit HAL 

(Cyberdyne, Japan), of which the control relies on muscle EMG measurements, is currently being evaluated as 

an assistive exoskeleton for the mobility impaired. For an extensive overview of powered lower limb 

exoskeletons the reader is referred to Dollar and Herr (2008). 

16.3 Robot control strategies for gait assistance 

In the previous section a general overview of the state-of-the-art in robot-assisted gait rehabilitation was given 

with emphasis on hardware. As mentioned in the introduction, various control strategies are being explored 

for robot-assisted rehabilitation of gait. This section deals with the state-of- the-art in robot control for gait 

assistance. The principles of motor learning and recovery in robot-assisted neuro-rehabilitation are not yet 

fully understood and difficult to translate into controller design guidelines (Reinkensmeyer et al., 2007). 

Control strategy developments therefore tend to be based on general concepts in rehabilitation, neuroscience 

and motor learning and many advances are, for the time being, engineering-driven, seeking improvements in 

robot control to realise those general concepts. 

Figure 51: High-level control strategies in robot-assisted gait rehabilitation: 

(from left to right) assistance based (LOPES), challenge based (ARTHuR), virtual-reality based (Lokomat), non-contact 

coaching based (USC's socially assistive robot for stroke patient 

16.3.1 Assistance based control 

According to Marchal-Crespo and Reinkensmeyer, 2009 the existing high-level controllers can be divided into 

four categories, depending on the underlying approach to achieve recovery: 

� Assistance based: the robotic device provides functional assistance, i.e. assistance of a functional task 

such as supporting the body-weight, advancing the limbs, ... (e.g. Duschau-Wicke et al., 2008; 

Ekkelenkamp et al., 2007; Aoyagi et al., 2007; Agrawal et al., 2007) . 

� Challenge based: in which a functional task is made more difficult or challenging, as opposed to 

assistance based strategies envisaged to facilitate a task (e.g. Yoon and Ryu, 2005; Lam et al., 2008; 

Emken and Reinkensmeyer, 2005). 

� Virtual-reality based: a haptic interface is used to simulate specific environments and/or activities 

170


(e.g. Schmidt et al., 2005b; Zimmerli et al., 2009). 

� Non-contact coaching based: a mobile robotic coach, making no physical contact, encourages the 

patient (e.g. Mataric et al., 2007). 

For a robotic device the terms assistance and challenge both relate to the way in which the device physically 

interacts with the patient in order to accomplish a certain task. As such they define a spectrum of possible 

control strategies that could apply to different patients with different impairment levels or to a single patient 

throughout his/her recovery. Challenge based strategies in robotic therapy are often based on the device 

providing resistance against movements or amplifying movement errors with respect to a reference trajectory 

(Marchal-Crespo and Reinkensmeyer, 2009). These challenge based strategies are likely to be more effective 

in mildly affected patients and, especially in gait recovery, these would require a combination with an 

assistance based strategy to safely support patients with insufficient strength and/or motor control. In the 

following overview focus is on assistance based strategies. 

A suitable way of differentiating between assistive control strategies is in terms of the activity level of the 

robot and the human-in-the-robot (Veneman et al. 2007). At the one side of the spectrum of possible assistive 

control strategies one can envisage a system that is controlled to minimally interact with the human (patientin-charge), 

at the other the system is controlled to, at least partly, take over human function (robot-in-charge). 

The effectiveness of either strategy will likely depend on the residual strength and motor control of the 

patient: severely impaired patients would require a robot in a robot-in-charge mode to ensure safety and 

continuity of movements, whereas patients with sufficient walking ability would benefit more from a device 

in a patient-in-charge mode that only intervenes when required during movements initiated and largely 

controlled by the patient. The ideal limits of the assistance level scale, namely full assistance (100% robot-incharge) 

and zero assistance (100% patient-in-charge), respectively correspond with the robot providing the 

required power to generate its own and the patient's motion, and with the robot perfectly compensating its own 

dynamics, such that the presence of the device is not felt. Non-ideal solutions such as the uncompensated 

dynamics of the robot, yield negative assistance and cause a shift towards a challenge based situation. 

16.3.2 Assistanceasneeded 

As discussed previously some properties of assistance based therapy can negatively affect motor learning and 

neural recovery. The assistance influences and eases the task to be relearned and as such it may induce a 

decrease of the patient's effort (Israel et al., 2006) and reduced or altered motor learning (Marchal-Crespo and 

Reinkensmeyer, 2008). This observation has led to a paradigm shift in robot-assisted rehabilitation of gait 

towards “assistance-as-needed”. In order to maximise assistive therapy outcome the assistive controller 

should assist as much or little as needed for the specific task, while triggering the patient to maximise his/her 

own efforts. Initially, assistive controllers were conceived as (high feedback gain) position-based controllers 

with a fixed target trajectory (Colombo et al., 2001). Such an assistive environment does not promote nor 

does it allow patient induced variability of the gait pattern and therefore it can be considered almost equivalent 

to a pure robot-in-charge mode. With the assistance-as-needed paradigm focus has shifted towards adaptivity 

of assistance, both in the targeted function/task (task-specificity) and in the assistance level. The adaptivity of 

assistance is envisaged between different individuals, different phases in therapy and also online, within the 

individual training session. From the state-of-the-art overview in the introductory chapter it is clear how the 

paradigm shift has influenced robot design on the hardware level: compliance as a means to provide 

variability has found its way to actuator design, either in the form of an intrinsically compliant actuator, a 

passive compliant element in series with a stiff actuator, or a passive compliant element alone. The following 

171


non exhaustive overview is aimed at listing some of the different ways of implementing the assistance-asneeded 

concept on the control level. 

Variability in assistance is generally achieved by means of one or a combination of the following concepts: 

� task/function specific assistance 

� adaptivity of the assistance level 

� adaptivity of timing 

� adaptivity in space 

16.3.2.1 Task/function specific assistance 

Task/function specific assistance implies that the assistance is tailored to the gait function(s), joint(s) or 

limb(s) that need(s) it and in the meantime assistance is reduced where it is useless or adverse. Hybrid forceposition 

control has been used to promote free motion during swing (force control) while using position 

control during stance (Bernhardt et al., 2005b, Lokomat). A similar approach has been conceived for training 

of the hemiparetic: position control (Bernhardt et al., 2005b, Lokomat) or impedance control (Vallery et al., 

2009, LOPES) of the impaired side and force control of the unaffected side. Another example of task-specific 

assistance is the use of Virtual Model Control (Ekkelenkamp et al., 2007, LOPES). A virtual model simulates 

a specific action that needs to be performed on the patient by means of the exoskeleton (e.g., foot lift during 

swing, body weight support, ...). 

The virtual model control is implemented by means of impedance control (in joint space or task space) 

defining the interaction between the actual joint/limb motion and a moving or fixed target position. For 

several reasons different research groups have used force control to apply the zero assistance mode (patientin-charge 

mode) also to the entire device: to record unassisted gait for use as target trajectories in a 

position/impedance control scheme (Aoyagi et al., 2007, PAM, van Asseldonk et al., 2007, LOPES) and as a 

reference or baseline for the assisted mode (van Asseldonk et al., 2008, LOPES). In order to approximate the 

ideal zero assistance level the robot's dynamics are modelled and partly compensated. In that case, instead of 

controlling the actuator output towards zero force/torque, the interaction forces/torques between the robot and 

the human are minimised. 

16.3.2.2 Adaptivity of the assistance level 

Adaptivity of the assistance level is often accomplished by using a measure of the patient's effort or a measure 

of how well the patient performs a task either directly as a feedback control signal or indirectly as a means to 

scale one or more control parameter(s). Patient-driven motion reinforcement (Bernhardt et al., 2005b, 

Lokomat) belongs to the first category, since a support torque is calculated as the product of a scale factor and 

the (modelled) active torque exerted by the patient. In Duschau-Wicke et al., 2008 (Lokomat) the support 

torque, proportional to the error between the actual and target trajectory, is recalculated at every gait cycle by 

an iterative learning controller. The second category groups several variations on the parameter scaling 

approach depending on the underlying control scheme. A forgetting factor is used for instance on the PD 

gains of a position control scheme (Emken et al., 2008, ARTHuR) and on an error-based learning controller 

(Emken et al., 2005, ARTHuR) reducing the assistance over time and ensuring the patient is sufficiently 

challenged. In Riener et al., 2005 (Lokomat) the impedance control parameters are scaled with the patient's 

effort, such that a larger contribution of the patient allows for larger trajectory deviations. 

172


16.3.2.3 Adaptivity of timing 

Adaptivity of timing is always related to adaptivity in space and vice versa, since a gait pattern is defined both 

in space and time. Moreover, for position based control (e.g. position control, impedance control) the timing 

and the amplitude of the target trajectory affect the assistance level as well. Imposing a target trajectory 

without imposing a related timing has been done by using a set point control or path control. In Banala et al., 

2007 (ALEX) PD set point control is used in which the target position is only switched to the next set point if 

the actual position is close enough to the current set point. In Aoyagi et al., 2007 (PAM) the target trajectory 

of the PD controller is determined on the basis of the actual state of the robot and the target trajectory of the 

robot defined in state-space. A moving window limits candidate target trajectory points to points close to the 

actual state. The difference in timing between the two points is fed to a synchronisation algorithm that selects 

the appropriate target trajectory point. The approach in Duschau-Wicke et al., 2008 (Lokomat) combines 

impedance control with the aforementioned set point generation method and the use of a moving window. In 

addition, an automatic treadmill speed adaptation algorithm changes the treadmill speed according to an 

admittance control scheme, using the measured interaction force between the human and an external fixed 

reference as an input. Instead of altering time dependent trajectories, some methods generate a position 

dependent force field or velocity field. The force field controller proposed in Banala et al., 2009 (ALEX) 

displays a force tangential to a required foot trajectory inside a “virtual tunnel”, a normal force towards the 

target trajectory outside that tunnel and a damping force to limit foot velocity. In Cai et al., 2006 two velocity 

field controllers are proposed. One has a virtual tunnel with tangential velocity inside and inward spiralling 

velocity outside, whereas the other has a small moving window with tangential velocity inside and a radial 

velocity field outside pointed towards the windows centre. 

16.3.2.4 Adaptivity in space 

Adaptivity in space is either achieved by altering the target trajectory of a position-based controller or it is 

intrinsic to a non-position-based controller (egg. force controller). The aforementioned force field and 

velocity field controllers are examples of the latter: the actual position can vary almost freely within the 

boundaries of the virtual tunnel. The force controllers discussed in the paragraph about task/function specific 

assistance also allow for patient-induced gait pattern adaptation. The same goes for the impedance controllers 

used in Lokomat (Riener et al., 2005), LOPES (Veneman et al., 2007) and WalkTrainer (Stauffer et al., 2009) 

and for the position controllers implemented in devices with intrinsically compliant or back driveable 

actuators (ARTHuR, PAM, POGO (Reinkensmeyer et al., 2006), KNEXO (Beyl et al., 2009). The trajectory 

tracking controller implemented in KNEXO provides a tunable limitation of the assistive torque allowing for 

large deviations from the target trajectory. Different target trajectory adaptation algorithms for position-based 

control have been proposed in Jezernik et al., 2004. These algorithms calculate a target gait pattern adaptation 

that minimises the active patient torque. In Vallery et al., 2009 the target trajectory of the impedance 

controller for the impaired leg is based on the recorded motion of the unimpaired leg fed to a so called 

“complementary limb motion estimation” algorithm. This allows for a patient- induced adaptation of gait 

both in space and in timing. The use of surface EMG sensors in a so called proportional myoelectric control 

should be mentioned as well (Gordon and Ferris, 2007; Lee and Sankai, 2002). Providing an output torque 

proportional to processed EMG signals indirectly puts the human in control of the timing and the level of 

assistance. EMG-based torque control thus fits in the previous categories as well. 

173


17 StateoftheArt in Robotic Systems for examining Hazardous 

Environments 

One of two demonstrators that will be used in CORBYS project to demonstrate and evaluate technologies 

developed is an existing autonomous robotic system consisting of a robot arm mounted on a mobile platform 

that can be used for inspection of contaminated/hazardous environments in teleoperation application or as a 

co-worker of a human. 

Disaster management aims to reduce, or avoid, potential damage from the hazards, to assure prompt and 

appropriate assistance to victims of disaster, and to achieve rapid and effective recovery. The main 

prerequisite of effective management of devastation is a fast and clear verification of possible contamination. 

Dependent on occurred scenario the first response teams are equipped with different techniques. For example 

the auxiliary fire brigades in Germany deploy NBC (nuclear-biological-chemical) Reconnaissance Vehicles 

(NBC RVs). The NBC RVs are mainly used for measuring, detecting and reporting radioactive and/or 

chemical as well as for recognising and reporting biological contamination (Meissner et al., 2005). However, 

the NBC RV is not permitted to enter contaminated area. In case of required measurements inside 

contaminated zone, the crew members, dressed in special overalls, has to leave the vehicle and continue the 

measurements. One of the common techniques for the verification of contamination is collection of samples 

in affected zones. Bruemmer (Bruemmer et al. 2002) describes baseline sample collection for laboratory 

analysis as a three phase process where, after an initial radiation survey performed by radiation control 

technician and after collecting of video coverage by a video technician, in the last phase a team of sampling 

technicians is sent into the facility to collect samples used to determine contamination levels (Figure 52). 

Typically, this data is then used to aid decontamination and decommissioning planning activities. 

Figure 52: Baseline sample collection for laboratory analysis (Bruemmer et al. 2002) 

In order to reduce humans’ exposure to danger during the exploration of contaminated/hazardous 

environments, in recent years significant effort has been made in developing and employment of mobile 

robotic systems. Existing Reconnaissance Robots, also known as Security Robots, enable mainly a visual 

investigation of the affected areas and/or chemical measurements which, however, do not require collection of 

samples. Security Robots consist of mobile platforms equipped with different sensors (cameras, laser 

scanners, etc.) which allow autonomous or teleoperated navigation. They can be used for indoor or outdoor 

applications. For example OFRO robot of a leading robotics company Robowatch Industries Ltd. from Berlin 

(http://www.robowatch.de) is the first mobile security robot worldwide for outdoor surveillance being able to 

determine the actual cause of an alarm, evaluate the situation and take counter measures. Mobile security 

robots are often equipped also with microphones and are used as rescue robots such as CRASAR robots used, 

among other incidents, in the World Trade centre (http://www.crasar.org) or KOHGA3 ground robot (Figure 

174


53) used by Japanese roboticists to assist with rescue and recovery operations after the 

http://spectrum.ieee.org/static/japans-earthquake-and-nuclear-emergencyearthquake that struck Japan in 

March 2011 (Guizzo, 2011). Robotic systems for Clearing of Explosive Devices represent a important group 

of Mobile Security Robots (http://www.rheinmetall-detec.com). The bomb disposal robotic systems consist of 

robot arm with gripper mounted on a mobile platform (Figure 53) and different sensors enabling remote robot 

control. Mobile platforms enable applications on staircases, flat or mountain areas. However, even equipped 

with robot arms, robots which are mainly used for clearing of explosive devices can not provide satisfactory 

sample quality required for reliable analysis. 

KOHGA3 robot 

EOD-Robot for clearing 

of explosive devices 

Rheinmetall robotic 

system 

Figure 53: Mobile security robots 

17.1 Robotic system for automated sampling 

(a) (b) 

175 

Mobile security robot OFRO 

(RoboWatch) 

The University of Bielefeld designed a robotic system, which handles samples in a laboratory environment 

[Poggendorf, 2004]. Further significant systems belonging to the group of mobile security robots for 

sampling are the Packbot developed by iRobot Corporation [Yamauchi, 2004], Telerob’s Safety Guard and 

tEODor [Saffiotti, 2004] and the NAT-II and T.S.R. EOD-robots from Elektroland 

[http://www.elektroland.com.tr]. However, most of those systems focus on teleoperation where the robot arm 

serves as an elongation of the human arm leading to a low level of automation. In contrast the mobile robotic 

system for safe automated sampling, which will be used as the second CORBYS demonstrator, includes a 

redundant robot arm for dexterous manipulation in unstructured environment. RecoRob is navigated by the 

user but it performs the collection of samples autonomously. It has been developed within the German 

national project RecoRob (Kuzmitcheva et al., 2009). The RecoRob objective is that the mobile investigation 

and robotic sampling platform replace the human investigation team and transfer the sampling/investigation 

circle from conventional manual (Figure 54(a)) to the structure shown in the Figure 54(b).


Figure 54: Autonomous vs. manual investigation of contamination area including sampling 

To meet the project objective, Reco Rob has to satisfy the following basic requirements: 

� the system must allow the user to guide the mobile unit for acquiring information concerning the 

air/water/soil relevant parameters (like contamination) and to gather samples for further investigations, 

� the sample gathering must be performed cross-contamination free, 

� the commands for guiding the unit as well as data acquired must be sent instantly to base using the 

wireless communication, 

� the transferred measurement data as well as information regarding stage of the task execution must be 

visualised on an ergonomic way, 

� the movement of the unit must be realised manually using a control joystick, or automatically, 

� a vision system is needed on the mobile to provide impression of tele-presence, 

� the mobile unit must report also the GPS data, for localisation and for data map generation, 

� in the case of an emergency, the mobile unit must be able to return to base executing a track-back of the 

route, 

� the construction of a mobile unit and/or components must allow decontamination in order to provide 

multiple usage of the system. 

To satisfy the functional requirements the system is endowed with various hardware devices as can be seen in 

Figure 55. For actuation the SCHUNK/degrees of freedom (DoF) lightweight robot arm is mounted on the 

mobile platform. The sensor system consists of a force-torque sensor in the manipulators wrist, a stereo 

camera system for 3D reconstruction, a camera for workspace observation and a camera for thermal 

inspection of the work area. 

(a) 

(b) 

Figure 55: (a) Hardware setup of the RecoRob reconnaissance robotic system, 

(b) Mapped Virtual Reality (MVR) used as simulation environment in the RecoRob system 

17.2 Intelligent automated investigation of a hazardous environment 

To overcome the difficulties and limitations of teleoperation it is absolutely necessary to move beyond 

teleoperation and develop robot intelligence that can be interleaved with human intelligence. The Idaho 

National Engineering and Environmental Laboratory (INEEL) has developed a mixed-initiative robotic 

system which can shift modes of autonomy on the fly, relying on its own intrinsic intelligence to protect itself 

176


and the environment as it works with human(s) to accomplish critical tasks (Bruemmer et al. 2002). With this 

system communication dropouts no longer resulted in the robot stopping dead in its tracks or, worse, 

continuing rampant until it has recognised that communications have failed. Instead, the robot may simply 

shift into a fully autonomous mode. Also in a remote situation, the robot is usually in a much better position 

than the human to react to the local environment, and consequently the robot may take the leadership role 

regarding navigation. As leader, the robot can then “veto” dangerous human commands to avoid running into 

obstacles or tipping itself over. For this the robot has to provide the operator with the situational awareness 

required for controlling a robot (Valero et al., 2011). 

The CORBYS will build on state-of-the-art robotic systems for examination of hazardous environment 

endowing them with cognitive capabilities to support autonomous sampling in two experimental scenarios. 

One of the most fascinating areas of future work is the need for the robot to be imbued (to be filled) with an 

ability to understand and predict human behaviour. The robot’s theory of human behaviour may be a rule set 

at a very simple level, or it may be a learned expectation developed through practiced evolutions with its 

human counterpart. Interaction between the robot and human may be through direct communications (verbal, 

gesture, touch, radio communications link) or indirect observation (physically struggling, erratic behaviour, 

unexpected procedural deviation). Interaction may also be triggered by the observation of environmental 

factors (rising radiation levels, the approach of additional humans, etc.) 

18 Conclusion 

In this document, the domain of robotic cognitive systems and their application in two demonstrators is 

introduced. Knowledge and requirements elicitation process, interaction with clinical and robotic experts (for 

gait rehabilitation systems and autonomous robotic systems respectively), and prioritised collection of 

requirements for CORBYS systems are presented in detail. The requirements engineering analysis base 

reports the requirements engineering methodology (UI-REF), followed by the procedure and findings of 

knowledge elicitation from clinical partners regarding the first demonstrator, entailing important discussions 

on end-user demographics, gait biomechanics in normal and pathological walking. 

Finalised requirements for the first and second demonstrator are then presented, followed state-of-the-market 

in light of CORBYS solutions. In total, 345 requirements have been gathered, out of which 309 are classed as 

mandatory, 22 as desirable, and 14 as optional requirements. All the requirements are detailed under the 

mechatronic control systems of CORBYS, human control system of CORBYS, and the Robohumatic systems. 

Requirements for system integration and functional testing for the CORBYS solutions as well as evaluation 

are also reported in this document. The requirements prioritisation process has used a number of prioritisation 

filters as reported in chapter 4, leading to a final sub set of 44 main requirements for the CORBYS project 

falling under three main categories: Cognitive systems, Demonstrator I specific, Demonstrator II specific. 

Lastly, detailed state-of-the-art reviews are presented in the various relevant areas as applicable to and in 

scope of the project. These include sensors and perception, situation assessment, anticipation and initiation, 

cognitive robot control architectures, smart integrated actuators, non-invasive BCI, gait rehabilitation systems 

and hazardous area examining robots. 

177


19 References 

A surveillance based model to calculate the direct medical costs in Europe – Final Report. DG Sanco 

Public Health / Consumer Safety Institute, Amsterdam, 2004: 

http://www.eurosafe.eu.com/csi/eurosafe.nsf/projects. 

Agrawal, S., Banala, S., Mankala, K., Sangwan, V., Scholz, J., Krishnamoorthy, V., Hsu, W. (2007) 

Exoskeletons for gait assistance and training of the motor impaired, in Proceedings of the IEEE 

International Conference on Rehabilitation Robotics, pp. 1108-1113. 

Allemand, Y., Stauffer, Y., Clavel, R., and Brodard, R. (2009). Design of a new lower extremity orthosis 

for overground gait training with the WalkTrainer, in Proceedings of the 2009 IEEE International 

Conference on Rehabilitation Robotics, pp. 550-555. 

Almeida e Costa, F., Rocha, L. M., Costa, E., Harvey, I., and Coutinho, A., editors (2007). Advances in 

Artificial Life (Proc. ECAL 2007, Lisbon), volume 4648 of LNCS, Berlin. Springer. 

Anderson JR, Bothell D, Byrne MD, Douglass S, Lebiere C, Qin Y., (2004). An intergrated theory of the 

mind. Psychol Rev 111:1036–1060 

Anthony, T., Polani, D., and Nehaniv, C. (2009). Impoverished empowerment: ‘meaningful’ action 

sequence generation through bandwidth limitation. In Kampis, G. and Szathmáry, E., editors, Proc. 

European Conference on Artificial Life 2009, Budapest. Springer. In Press. 

Anthony, T., Polani, D., and Nehaniv, C. L. (2008). On preferred states of agents: how global structure is 

reflected in local structure. In (Bullock et al., 2008), pages 25–32. 

Aoyagi, D., Ichinose, W., Harkema, S., Reinkensmeyer, D., and Bobrow, J. (2007). A robot and control 

algorithm that can synchronously assist in naturalistic motion during body-weight-supported gait training 

following neurologic injury, IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 

15, no. 3, pp. 387-400. 

Ashby, W. R. (1956). An Introduction to Cybernetics. Chapman & Hall Ltd. 

Assessing disability. Münchener Rückversicherungs-Gesellschaft, 2004. München: 

www.munichre.com/publications/302-04093_en.pdf. 

Aström, K.J., Wittenmark, B., (1995). Adaptive Control. Reading: Addison-Wesley. 

Atick, J. J. (1992). Could information theory provide an ecological theory of sensory processing. Network: 

Computation in Neural Systems, 3(2):213–251. 

Attneave, F. (1954). Informational aspects of visual perception. Psychol. Rev., 61:183–193. 

Avor, J.K. and Sarkodie-Gyan, T. (2009). An approach to sensor fusion in medical robots, IEEE 11th 

International Conference on Rehabilitation Robotics, Kyoto International Conference Center, Japan. 

Ay, N. and Polani, D. (2008). Information flows in causal networks. Advances in Complex Systems, 

11(1):17–41. 

Ay, N. and Wennekers, T. (2003). Dynamical properties of strongly interacting markov chains. Neural 

Networks, 16(10):1483–1497. 

Ay, N., Bertschinger, N., Der, R., Güttler, F., and Olbrich, E. (2008). Predictive information and 

explorative behavior of autonomous robots. European Journal of Physics B, 63:329–339. 

Badii A, User-Intimate Requirements Hierarchy Resolution Framework (UI-REF): Methodology for 

Capturing Ambient Assisted Living Needs, Proceedings of the Research Workshop, Int. Ambient 

Intelligence Systems Conference (AmI’08), Nuremberg, Germany November 2008. 

Banala, S. K., Agrawal, S. K., Fattah, A., Krishnamoorthy, V., Hsu, W., Scholz, J., and Rudolph, K. 

(2006). Gravity-Balancing Leg Orthosis and Its Performance Evaluation, IEEE Transactions on Robotics, 

vol. 22, no. 6, pp. 1228-1239. 

178


Banala, S. K., Kim, S. H., Agrawal, S. K., and Scholz, J. P. (2009). Robot Assisted Gait Training With 

Active Leg Exoskeleton (ALEX), IEEE transactions on neural systems and rehabilitation engineering, vol. 

17, no. 1, pp. 2-8. 

Banala, S. K., Kulpe, A., and Agrawal, S. K. (2007). A Powered Leg Orthosis for Gait Rehabilitation of 

Motor-Impaired Patients, in Proceedings of IEEE International Conference on Robotics and Automation, 

pp. 4140-4145. 

Barbeau, H. and Rossignol, S. (1994). Enhancement of locomotor recovery following spinal cord injury, 

Current Opinion in Neurology, vol. 7, pp. 517-524. 

Barlow, H. B. (1959). Possible principles underlying the transformations of sensory messages. In 

Rosenblith, W. A., editor, Sensory Communication: Contributions to the Symposium on Principles of 

Sensory Communication, pages 217–234. The M.I.T. Press. 

Barlow, H. B. (2001). Redundancy reduction revisited. Network: Computation in Neural Systems, 

12(3):241–253. 

Barlow, J. S. (1984). Emg artifact minimization during clinical eeg recordings by special analog filtering. 

Electroencephalogr. Clin. Neurophysiol., 58(2):161–174. 

Barry, R.J., Clarke, A.R. and Johnstone, S.J. (2003). A review of electrophysiology in 

attentiondeficit/hyperactivity disorder: I. qualitative and quantitative electroencephalography. Clinical 

Neurophysiology, 114(2):171–183. 

Bastiaansen, M.C.M., Bocker, K.B.E., Cluitmans, P.J.M. and Brunia, C.H.M. (1999). Event-related 

desynchronization related to the anticipation of a stimulus providing knowledge of results. Clinical 

Neurophysiology, 110(2):250–260. 

BCC Research, http://www.bccresearch.com/report/HLC045A.html HEALTHCARE Prosthetics, 

Orthotics and Cosmetic Enhancement Products Report Code:HLC045APublished:October 2005 

Behrman, A. and Harkema S. (2000). Locomotor training after human spinal cord injury: a series of case 

studies, Physical Therapy, vol. 80, no. 7, pp. 688-700. 

Bell, C.J. et al. (2008) Control of a humanoid robot by a noninvasive brain-computer interface in humans. 

J. Neural Eng., 5(2):214–220. 

Benjamin, D., Lyons, D., Lonsdale, D., (2004). Adapt: A cognitive architecture for robotics, in 2004 

International Conference on Cognitive Modeling, A. R. Hanson and E. M. Riseman, Eds., Pittsburgh, PA. 

Berg, P. and Scherg, M. (1994). A multiple source approach to the correction of eye artifacts. 

Electroencephalogr. Clin. Neurophysiol., 90(3):229–241. 

Berger, W. (2008) International assesment of research and development in brain-computer interfaces. 

World Technology Evaluation Center. 

Bernhardt, M., Frey, M., Colombo, G., and Riener. R. (2005). Hybrid force-position control yields 

cooperative behaviour of the rehabilitation robot Lokomat, in 9th International Conference on 

Rehabilitation Robotics (ICORR 2005), pp. 536-539. 

Bernhardt, M., Lutz, P., and Frey, M. (2005). Physiological Treadmill Training with the 8-DOF 

Rehabilitation Robot LOKOMAT, in BMT 2005, Jahrestagung der deutschen Gesellschaft für 

biomedizinische Technik. 

Bertschinger, N., Olbrich, E., Ay, N., and Jost, J. (2008). Autonomy: an information-theoretic perspective. 

Biosystems, 91:331–345. 

Beyl, P., Van Damme, M., Van Ham, R., Vanderborght, B., and Lefeber, D. (2009). Design and control of 

a lower limb exoskeleton for robot-assisted gait training, Applied Bionics and Biomechanics, vol. 6, no. 2, 

pp. 229-243. 

Bharadwaj, K. and Sugar, T. G. (2006). Kinematics of a Robotic Gait Trainer for Stroke Rehabilitation, in 

Proceedings of the 2006 IEEE International Conference on Robotics and Automation, pp. 3492-3497. 

Bialek, W., Nemenman, I., and Tishby, N. (2001). Predictability, complexity and learning. Neural 

Computation, 13:2409–2463. 

179


BioEra, n.d. BioEra - visual designer for biofeedback. [online] Available at 

[Accessed 13 May 2011]. 

BioExplorer, n.d. BioExplorer . [online] Available at [Accessed 13 

May 2011]. 

BioSig Project, n.d. The BioSig Project. [online] Available athttp://biosig.sourceforge.net/ [Accessed 13 

May 2011]. 

Birbaumer, N. et al. (1999). A spelling device for the paralysed. Nature, 398:297–298. 

Bladon, P., Hall, R. J., Wright, W. A. (2002). Situation Assessment using Graphical Models, Proceedings 

of the Fifth International Conference on Information Fusion, Vol. 2, Pages 886-893. 

Blahut, R. (1972). Computation of channel capacity and rate distortion functions. IEEE Transactions on 

Information Theory, 18(4):460–473. 

Blanc-Garin, J. (1994). Patterns of recovery from hemiplegia following stroke. Neuropsychological 

rehabiliation, 4(4):359385. 

Blankertz, B., Curio, G. and Mller, K.-R. (2002). Classifying single trial eeg: Towards brain computer 

interfacing. In Advances in Neural Inf. Proc. Systems (NIPS 01), 14:157–164. 

Blankertz, B., Dornhege, G., Schäfer, C., Krepki, R., Kohlmorgen, J., Müller, K.R., Kunzmann, V., Losch, 

F., and Curio, G. (2003). Boosting bit rates and error detection for the classification of fastpaced motor 

commands based on single-trial eeg analysis. IEEE Trans. Neural Syst. Rehabil. Eng., 11:127–131. 

Blasch, E and S. Plano, JDL Level 5 fusion model: user refinement issues and applications in group 

tracking, SPIE Vol 4729, Aerosense, 2002, pp. 270 – 279. 

Blaya, J. A. and Herr, H. (2004). Adaptive control of a variable-impedance ankle-foot orthosis to assist 

drop-foot gait, IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 12, no. 1, pp. 

24-31. 

Bolt, R.A. (1980). Put-that-there: Voice and gesture at the graphic interface. Computer Graphics, 

14(3):262-270. 

Bosman, P. A. N. and Poutré, H. L. (2007). Learning and anticipation in online dynamic optimization with 

evolutionary algorithms: The stochastic case. In Proc. GECCO 2007, pages 1165–1172, New York. ACM 

Press. 

Botvinick, M. M., Braver, T. S., Barch, D. M., Carter, C. S., Cohen, J. D., (2001). Conflict monitoring and 

cognitive control. Psychological Review, 108:624–652.M.S. 

Brainterface, n.d. BF++ 2.0: The Body Language Framework. [online] Available 

at [Accessed 13 May 2011]. 

Brenner, N., Bialek, W., and de Ruyter van Steveninck, R. (2000). Adaptive rescaling optimizes 

information transmission. Neuron, 26:695–702. 

Brooks, R. A., (1986). A Robust Layered Control System for a Mobile Robot, IEEE Journal of Robotics 

and Automation, Vol. 2, No. 1, pp. 14–23. 

Brooks, R. A. (1991). Intelligence without representation. Artificial Intelligence, 47(1–3):139–159. 

Brown, B. (1970). Recognition of aspects of consciousness through association with eeg alpha activity 

represented by a light signal. Psychophysiology, 6:442–452. 

Brown, J. and Frank, J. (1987). Influence of event anticipation on postural actions accompanying 

voluntary movement. Exp. Brain Res., 67:645–650. 

Brown, M., Harris, C., (1994). Neurofuzzy Adaptive Modeling and Control, Prentice-Hall: Englewood 

Cliffs. 

Browne, M. and Cutmore, T. R. (2002). Low-probability event-detection and separation via statistical 

wavelet thresholding: an application to psychophysiological denoising. Clin. Neurophysiol., 113(9):1403– 

180


1411. 

Bruemmer, D. J., Marble J. L., Dudenhoeffer, D. D., Anderson, M. O., Mark D. McKay, M. D. (2002). 

Intelligent Robots for Use in Hazardous DOE Environments, in Robotics and Intelligent Machines in the 

U.S. Department of Energy: A Critical Technology Roadmap 

Bruyninckx, H. (2001). Open robot control software: the orocos project. In IEEE International Conference 

on Robotics and Automation, volume 3. 

Buch, E. et al. (2008). Think to move: a neuromagnetic brain-computer interface (bci) system for chronic 

stroke. Stroke, 39(910). 

Bullock, S., Noble, J., Watson, R., and Bedau, M. A., editors (2008). Artificial Life XI: Proceedings of the 

Eleventh International Conference on the Simulation and Synthesis of Living Systems, Winchester 5–8. 

Aug. MIT Press, Cambridge, MA. 

Burghart, C., Mikut, R., Stiefelhagen, R., Asfour, T., Holzapfel, H., Steinhaus, P.,, Dillman, R., (2005). A 

cognitive architecture for a humanoid robot: A first approach, IEEE-RAS International Conference on 

Humanoid Robots (Humanoids 2005), pp. 357–362. 

Business Wire, http://www.allbusiness.com/health-care/medical-devices-equipment-prosthetic/11781497- 

1.html Research and Markets: Market Tracking: Wheelchairs in Europe Top 5 2009 Publication: Business 

Wire Date: Thursday, February 12 2009 

Cai, L. L., Fong, A. J., Otoshi, C. K., Liang, Y., Burdick, J. W., Roy, R. R., and Edgerton, V. R. (2006). 

Implications of assist-as-needed robotic step training after a complete spinal cord injury on intrinsic 

strategies of motor learning, Journal of Neuroscience, vol. 26, no. 41, pp. 10 564-8. 

Campbell, A.T., Choudhury, T., Hu, S., Lu, H., Mukerjee, Rabbi, M.K., M. and Raizada, R.D.S. (2010). 

Neurophone: Brain-mobile phone interface using a wireless eeg headset. 

Capdepuy, P., Polani, D., and Nehaniv, C. L. (2007a). Constructing the basic umwelt of artificial agents: 

An information-theoretic approach. In (Almeida e Costa et al., 2007), pages 375–383. 

Capdepuy, P., Polani, D., and Nehaniv, C. L. (2007b). Construction of an internal predictive model by 

event anticipation. In Butz, M., Sigaud, O., Pezzulo, G., and Baldassarre, G., editors, Proceedings of the 

Third Workshop on Anticipatory Behavior in Adaptive Learning Systems, LNCS/LNAI, pages 218–232. 

Springer. 

Capdepuy, P., Polani, D., and Nehaniv, C. L. (2007c). Grounding action-selection in event-based 

anticipation. In (Almeida e Costa et al., 2007), pages 253–262. 

Carmena, J.M. et al. (2003). Learning to control a brain-machine interface for reaching and grasping by 

primates. PLoS Biology, 1(2):193–208. 

Carver, N., Lesser, V. (1991). A New Framework for Sensor Interpretation: Planning to Resolve Sources 

of Uncertainty, Proceedings of AAAI-91, 724-731. 

Cester, I., Dunne, S., Riera, A., Ru ni, G. (2008). Enobio: Wearable, wireless, 4-channel 

electrophysiology recording system optimized for dry electrodes. 

Chapin, J.K. et al. (1999). Real-time control of a robot arm using simultaneously recorded neurons in the 

motor cortex. Nature Neuroscience, 2(7):664–670. 

Cheyer, A., and Martin, D., The Open Agent Architecture, In: Journal of Autonomous Agents and Multi- 

Agent Systems, vol. 4, no. 1, pp. 143-148, March 2001 

Choi, S., Cichocki, A., Park, H. M. and Lee, S. Y. (2005). Blind source separation and independent 

component analysis: A review. Neural Information Processing-Letters and Review, 6(1):1–57. 

Christ, S., Hohnsbein, J., Falkenstein, M., Hoormann, J. (2000). Erp components on reaction errors and 

their functional significance: A tutorial. Biological Psychology, 51:87–107. 

Clarke, A. R., Barrya, R. J., McCarthyb, R., and S., Mark. (2001). Eeg-defined subtypes of children with 

attention-deficit/hyperactivity disorder. Clinical Neurophysiology, 112(11):2098–2105. 

181


Coles, M. Bekkering, H. Van Schie, H., Mars, R. (2004). Modulation of activity in medial frontal and 

motor cortices during error observation. Nature Neuroscience, 7:549–554. 

Coles, M.G.H., Gratton, G. and Donchin, E. (1988). Detecting early communication: using measures of 

movement-related potentials to illuminate human information processing. Biol Psychol, 26:69–89, 1988. 

Colombo, G., Wirz, M., and Dietz, V. (2001). Driven gait orthosis for improvement of locomotor training 

in paraplegic patients, Spinal Cord, vol. 39, pp. 252-253. 

CoolBOT Project, n.d. CoolBOT Project’s site. [online] Available at:http://www.coolbotproject.org 


Corradini, A., Mehta, M., Bernsen, N.O., Martin , J.-C., Abrilian, S. (2005). Multimodal Input Fusion in 

Human-Computer Interaction: On the Example of the NICE Project. Data Fusion for Situation 

Monitoring, Incident Detection, Alert and Response Management, E. Shahbazian et al. Ed. IOS Press. 

Costa, N. and Caldwell, D. (2006). Control of a biomimetic "soft-actuated" 10 DoF lower body 

exoskeleton, in Proceedings of the IEEE/RAS-EMBS International Conference on Biomedical Robotics 

and Biomechatronics, pp. 495-501. 

Cover, T. M. and Thomas, J. A. (1991). Elements of Information Theory. Wiley, New York. 

CRASAR-Center for Robot-Assisted Search and Rescue at Texas A&M University [online] Available at: 

 

Croft, R. J. and Barry, R. J. (2000). Removal of ocular artifact from the eeg: a review. Neurophysiol. Clin., 

30(1):5–19. 

Csíkszentmihályi, M. (1978). Beyond Boredom and Anxiety: Experiencing Flow in Work and Play. 

Cambridge University Press, Cambridge. 

Dal Seno, B. (2009). Toward An Integrated P300-And ErrP-Based Brain-Computer Interface. PhD thesis, 

Politecnico di Milano. 

Darken, C. J. (2005). Towards learned anticipation in complex stochastic environments. In Proc. Artificial 

Intelligence for Interactive Digital Entertainment Conference (AIIDE). 

Das, S., Grey, R., Gonsalves, P. (2002). Situation Assessment via Bayesian Belief Networks, Proceedings 

of the fifth International Conference on Information Fusion, Annapolis, Maryland 

Deecke, L., Grozinger, B. and Kornhuber, H.H. (1976). Voluntary finger movement in man: cerebral 

potentials and theory. Biol Cybernet, 23:99–119. 

Deecke, L., Scheid, P. and Kornhuber, H.H. (1969). Distribution of readiness potential, pre-motion 

positivity and motor potential of the human cerebral cortex preceding voluntary finger movement. Exp 

Brain Res, 7:158–168. 

Delorme, A. and Makeig, S. (2004). Eeglab: an open source toolbox for analysis of single-trial eeg 

dynamics. Journal of Neuroscience Methods, 134:9–21. 

Der, R. (2000). Selforganized robot behavior from the principle of homeokinesis. In Groß, H.-M., Debes, 

K., and Böhme, H.-J., editors, Proc. Workhop SOAVE ’2000 (Selbstorganisation von adaptivem 

Verhalten), volume 643 of Fortschritt-Berichte VDI, Reihe 10, pages 39–46, Ilmenau. VDI Verlag. 

Der, R. (2001). Self-organized acqusition of situated behavior. Theory Biosci., 120:1–9. 

Der, R., Hesse, F., and Martius, G. (2006). Rocking stumper and jumping snake from a dynamical system 

approach to artificial life. J. Adaptive Behavior, 14(2):105–115. 

Der, R., Steinmetz, U., and Pasemann, F. (1999). Homeokinesis – a new principle to back up evolution 

with learning. In Mohammadian, M., editor, Computational Intelligence for Modelling, Control, and 

Automation, volume 55 of Concurrent Systems Engineering Series, pages 43–47. IOS Press. 

Deutsche Industrie Norm für Begriffe der Informationsverarbeitung: Teil 5: Begriffe, Aufbau digitaler 

Rechensysteme 

182


Dietsch, J., (2011), Imitating Ourselves in Silicon, IEEE ROBOTICS & AUTOMATION MAGAZIN. 

Dietz, V., Colombo, G., and Jensen, L. (1994). Locomotor activity in spinal man, Lancet, vol. 44, pp. 

1260-1263. 

Dollar, A. and Herr , H. (2008). Lower extremity exoskeletons and active orthoses: challenges and stateof-the-art, 

IEEE Transactions on Robotics, vol. 24, no. 1, pp. 144-158. 

Donaldson-Matasci, M. C., Bergstrom, C. T., and Lachmann, M. (2010). The fitness value of information. 

Oikos, 119:219–230. 

Dragon Runner Reconnaissance Robot. [online] Available at: 

[Accessed 15.07.2011] 

Dreinhöfer KE, Merx H., Puhl W. (2003). In Schwerpunkbericht Decade of Bone and Joint Diseases. 

Hrsg. Statistisches Bundesamt. 

Dubois, D. M. (2003). Mathematical foundations of discrete and functional systems with strong and weak 

anticipations. In Anticipatory Behavior in Adaptive Learning Systems, Lecture Notes in Computer 

Science, pages 107–125. Springer. 

Duschau-Wicke, A., Zitzewitz, J. v., Lünenburger, L., and Riener, R. (2008). Patient-driven cooperative 

gait training with the rehabilitation robot Lokomat, in ECIFMBE-IFMBE Proceedings 22, pp. 1616-1619. 

EC Directorate-General Health & Consumer Protection, Public Health. Musculoskeletal Problems And 

Functional Limitation, The Great Public Health Challenge for the 21st Century (Grant agreement 

S12.297217), University of Oslo, Department of General Practice and Community Medicine, The Bone & 

Joint Decade 2000 – 2010, Oslo, October 2003 

Edelman, G. M., (1987). Neural Darwinism: The Theory of Neuronal Group Selection, New York: Basic 

Books. 

Ekkelenkamp, R., Veltink, P., Stramigioli, S., and Van der Kooij, H. (2007). Evaluation of a Virtual Model 

Control for the selective support of gait functions using an exoskeleton, in Proceedings of the IEEE 10th 

International Conference on Rehabilitation Robotics, pp. 693-699. 

Elbert, T., Flor, H., Birbaumer, N., Knecht, S., Hampson, S., Larbig, W., and Taub, E. (1994). Extensive 

reorganization of the somatosensory cortex in adult humans after nervous system injury, Neuroreport, vol. 

5, no. 18, pp. 2593-2597. 

Elektroland Company, Ankara, Turkey, http://www.elektroland.com.tr/ 

Elting, C., Strube, M., Moehler, G., Rapp, S., and Williams, J., The Use of Multimodality within the 

EMBASSI System, M&C2002 - Usability Engineering Multimodaler Interaktionsformen Workshop, 

Hamburg, Germany, 2002 

Emken, J. L. and Reinkensmeyer, D. J. (2005). Robot-Enhanced motor learning: accelerating internal 

model formation during locomotion by transient dynamic amplifcation, IEEE Transactions on Neural 

Systems and Rehabilitation Engineering, vol. 13, pp. 33-39. 

Emken, J. L., Benitez, R., and Reinkensmeyer, D. J. (2007). Human-robot cooperative movement training: 

Learning a novel sensory motor transformation during walking with robotic assistance-as-needed, Journal 

of NeuroEngineering and Rehabilitation, vol. 4, no. 8. 

Emken, J. L., Harkema, S. J., Beres-Jones, J. A., Ferreira, C. K., and Reinkensmeyer, D. J. (2008). 

Feasibility of manual teach-and-replay and continuous impedance shaping for robotic locomotor training 

following spinal cord injury, IEEE Transactions on Biomedical Engineering, vol. 55, no. 1, pp. 322-334. 

Emken, J. L., Wynne, J. H., Harkema, S. J., and R. D. J. (2006). A Robotic Device for Manipulating 

Human Stepping, IEEE Transactions on Robotics, vol. 22, no. 1, pp. 185-189. 

Emken, J., Bobrow, J., and Reinkensmeyer, D. (2005). Robotic movement training as an optimization 

problem: designing a controller that assists only as needed, in Proceedings of 9th International Conference 

on Rehabilitation Robotics, pp. 307-312. 

Emotiv, n.d. EPOC neuroheadset. [online] Available at http://www.emotiv.com/store/hardware/epocbci/epoc-neuroheadset/ 


183


Endsley, M. (2000). Theoretical underpinnings of Situation Awareness: a critical review, Situation 

Awareness Analysis and Measurement, Mahwah. 

Erman, L. D., Hayes-Roth, F., Lesser, V. R., Reddy, D. R. (1980). The Hearsay-II speech-understanding 

system: Integrating knowledge to resolve uncertainty, In ACM Computing Surveys, volume 12 (2), pages 

213–253. 

Escolano, C., Antelis, J., and Minguez, J. (2009). Human Brain-Teleoperated Robot between Remote 

Places. IEEE International Conference on Robotics and Automation (ICRA). 

European Bone and Joint Health Strategies Project, A Public Health Strategy to Reduce the Burden of 

Musculoskeletal Conditions, (Grant Agreement : SI2.304 598), The Bone & Joint Decade, Department of 

Orthopedics, University Hospital, SE-221 85 LUND, Sweden, ISBN 91-975284-0-4 

European Land Robot trials, 2006. [online] Available at: < http://www.rheinmetall-detec.com/> [Accessed 

15 July 2011]. 

European Opinion Research Group EEIG. Health, Food and Alcohol and Safety. Special Eurobarometer 

186. 2003. European Commission. 

Fairclough, S.H. (2009). Fundamentals of physiological computing, Interacting with Computers 21, pp. 

133-145. 

Farris, R. J., Quintero, H. A., Withrow, T. J., and Goldfarb, M. (2009). Design of a Joint-Coupled Orthosis 

for FES-Aided Gait, in 2009 IEEE International Conference on Rehabilitation Robotics, pp. 246-252. 

Fatourechi, M., Bashashati, A., Ward, R.K., Birchi, G.E. (2007). Emg and eog artifacts in brain computer 

interface systems: A survey. Clinical Neurophysiology, 118:480–494. 

Ferrez, P.W. and Del R Millan, J. (2004). Error-related eeg potentials generated during simulated 

braincomputer interaction. IEEE Transactions on Biomedical Engineering, 55(3):923–929. 

Foster, D. (1990). Eeg and subjective correlates of alpha-frequency binaural-beat stimulation combined 

with alpha biofeedback. 

Fowler, F.J., Gill, H.S. (1990). The industrial orthopedic rehabilitation market: a niche opportunity, Hosp 

Technol Ser. May;9(13):1-13. 

Friel, P.N. (2007). Eeg biofeedback in the treatment of attention deficit/hyperactivity disorder. Alternative 

Medicine Review, 12(2):146–151. 

g.tec Technologies, n.d. g.SAHARA, active dry electrode system. [online] Available at 

[Accessed 13 May 

2011]. 

g.tec Technologies, n.d. g.tec medical engineering. [online] Available at < http://www.gtec.at/> [Accessed 

13 May 2011]. 

Gargiulo, G., Calvo, R.A., Bifulco, P., Cesarelli, M., Jin, C., Mohamed, A. and Van Schaik, A. (2010). A 

new eeg recording system for passive dry electrodes. Clinical Neurophysiology, 121(5):686–693. 

Gat, E., (1997). On three-layer architectures. In Artificial Intelligence and Mobile Robots. MIT/AAAI 

Press. 

Geetha, G. and Geethalakshmi, S.N. (2011). Scrutinizing di erent techniques for artifact removal from 

eeg signals. International Journal of Engineering Science and Technology, 3(2). 

Gerkey, B. P., Vaugh, R. T. and Howard, A. (2003). The player/stage project: Tools for multi-robot and 

distributed sensor systems. In Proceedings of the International Conference on Advanced Robotics, pages 

317–323, Coimbra, Portugal. 

Gibbon, D., Mertins, I., Moore, R. (2000). Handbook of Multimodal and Spoken Dialogue Systems: 

Resources, Terminology and Product Evaluation, Springer 

Gordon, K. E. and Ferris, D. P. (2007). Learning to walk with a robotic ankle exoskeleton, Journal of 

Biomechanics, vol. 40, pp. 2636-2644. 

184


Gratch, J. and Marsella, S. (2004). A domain-independent framework for modeling emotion. Journal of 

Cognitive Systems Research, 5(4):269–306. 

GROUND ROBOTS – 510 PACKBOT®. [online] Available at: 

[Accessed 10 July 2011]. 

Grozea, C., Voinescu, C.D. and Fazli, S. (2011). Bristle-sensors-low-cost flexible passive dry eeg 

electrodes for neurofeedback and bci applications. Journal of Neural Engineering, 8(2). 

Gruber, T., Muller, M., and Elbert, T. (1999). Selective visual-spatial attention alters induced gamma-band 

responses in the human eeg. Clinical Neurophysiology, 110:2074. 

Guglielmelli, E., Johnson, M. J., and Shibata, T. (2009). Guest Editorial Special Issue on Rehabilitation 

Robotics, IEEE Transactions on Robotics, vol. 25, no. 3, pp. 477-480. 

Guizzo, E. (2011), Japan Earthquake: More Robots to the Rescue, IEEE Spectrum Magazine, May 2011 

issue 

Gwin, J.T., Gramann, K., Makeig, S. and Fer, D. P. (2010). Removal of movement artifact from 

highdensity eeg recorded during walking and running. J Neurophysiol., 103:3526–3534. 

Hamadicharef, B., Zhang, H.H., Guan, C., Wang, C.C., Phua, K.S., Tee, K.P. Tee, and Ang, K.K. (2009). 

Learning eeg-based spectral-spatial patterns for attention level measurement. In IEEE International 

Symposium on Circuits and Systems (ISCAS). 

Hans, M. and Baum, W. (2001). Concept of a hybrid architecture for Care-O-bot. In Proc. IEEE 

International Workshop on Robot and Human Interactive Communication (RoMan), pp. 407- 411. 

Harkema, S. J. (2001). Neural plasticity after human spinal cord injury: application of locomotor training 

to the rehabilitation of walking, The Neuroscientist, vol. 7, no. 5, pp. 455-468. 

Haufler, A.J., Spalding, T.W., Santa Maria, D.L., and Hatfield, B.D. (2000). Neuro-cognitive activity 

during a self-paced visuospatial task: comparative eeg profiles in marksmen and novice shooters. 

Biological Psychology, 53:131–160. 

Hayes-Roth, B., Buchanan, B., Lichtarge, O., Hewett, M., Altman, R., Brinkley, J., Cornelius, C., Duncan, 

B., and Jardetzky, O. (1986). PROTEAN: Deriving Protein Structure from Constraints, Proceedings of 

AAAI-86, 904-909. 

He, P., Wilson, G. and Russell, C. (2004). Removal of ocular artifacts from electro-encephalogram by 

adaptive filtering. Med. Biol. Eng. Comput., 42(3):407–412. 

Hericks, M., Krebs, U., Kuzmicheva, O., (2011). A Mobile Reconnaissance Robot for Investigation of 

Dangerous Sites, the 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 

2011), San Francisco, California, 2011 (accepted) 

Hesse S. and Uhlenbrock, D. (2000). A mechanized gait trainer for restoration of gait, Journal of 

Rehabilitation Research and Development, vol. 37, no. 6, pp. 701-708. 

Hesse, S., Bertelt, C., Jahnke, M. T., Scha_rin, A., Baake, P., and Malezic, M. (1995). Treadmill Training 

with partial body weight support compared with physiotherapy in nonambulatory hemiparetic patients, 

Stroke, vol. 26, pp. 976-981. 

Hidler, J., Hamm, L. F., Lichy, A., and Groah, S. L. (2008). Automating activity-based interventions: The 

role of robotics, Journal of Rehabilitation Research & Development, vol. 45, no. 2, pp. 337-344. 

Higgins, R.P. (2005). Automatic Event Recognition for Enhanced Situational Awareness in UAV Video, 

SIMA 2005, Atlantic City, NJ. 

Hill, N.J., Lal, T.N., Bierig, K., Birbaumer, N. and Schlkopf, B. (2004). Attentional modulation of 

auditory event-related potentials in a brain-computer interface. Biomedical Circuits and Systems. 

Hillyard, S.A., Hink, R.F., Schwent, V.L. and Picton, T.W. (1973). Electrical signs of selective attention in 

the human brain. Science, 182:177–180. 

Hitt, J., Oymagil, A. M., and Sugar, T. (2007). Dynamically controlled ankle-foot orthosis (DCO) with 

regenerative kinetics: incrementally attaining user portability, 2007 Proceedings of the IEEE International 

Conference on Robotics and Automation Roma, Italy, pp. 1541-1546. 

185


Howard, C., Stumptner, M. (2005). Situation Assessments Using Object Oriented Probabilistic Relational 

Models, Proceedings of the 7th International Conference on Information Fusion, Vol. 2 

Howard, R. A. (1966). Information value theory. IEEE Transactions on Systems Science and Cybernetics, 

SSC-2:22–26. 

Hoyer, P. O., Janzing, D., Mooij, J. M., Peters, J., and SchÃlkopf, B. (2009). Nonlinear causal discovery 

with additive noise models. In Koller, D., Schuurmans, D., Bengio, Y., and Bottou, L., editors, Advances 

in Neural Information Processing Systems 21: Proceedings of the 2008 Conference, pages 689–696, Red 

Hook, NY. Curran. 

Huang, R.-S., Jung, T.-P., and Makeig, S. (2007). Multi-scale eeg brain dynamics during sustained 

attention tasks. Proceedings of the 2007 IEEE International Conference on Acoustics, 4:1173– 1176. 

Hunt, J. (2002). Blackboard Architectures. JayDee Technology Ltd. 2002. 

Hurtig, T., Jokinen, K. (2006). Modality Fusion in a Route Navigation System, Workshop on Effective 

Multimodal Dialogue Interface in International Conference on Intelligent User Interfaces, Sydney, 

Australia. 

Ikeda, A., Lunders, H.O., Collura, T.F., Burgess, R.C., Morris, H.H., Hamano T., et al. (1996). Subdural 

potentials at orbitofrontal and mesial prefrontal areas accompanying anticipation and decision making in 

humans: a comparison with bereitschaftspotential. Electroenceph Clin Neurophysiol, 98:206–12. 

Ingram, H.A., van Donkelaar, P., Cole, J., Vercher, J.L., Gauthier, G.M. and Miall, R.C. (2000). The role 

of proprioception and attention in a visuomotor adaptation task. Exp Brain Res, 132:114126. 

Injuries in the European Union Statistics Summary 2005-2007, Robert Bauer, Monica Steiner (KfV), 

European Commission, Health and Consumers (DG Sanco) Vienna November 2009 

Israel, J., Campbell, D., Kahn, J., and Hornby, T. (2006). Metabolic costs and muscle activity patterns 

during robotic- and therapist-assisted treadmill walking in individuals with incomplete spinal cord injury, 

Physical Therapy, vol. 86, no. 11, pp. 1466-78. 

Iturrate, I., Antelis, J., Kubler, A., and Minguez, J. . Non-Invasive Brain-Actuated Wheelchair based on a 

P300 Neurophysiological Protocol and Automated Navigation. IEEE Transactions on Robotics (in press). 

Iturrate, I., Montesano, L., and Minguez, J. (2010). Elicitation and Online Recognition of Error-Related 

Potentials During Observation of Robot Operation. In Annual Conference of IEEE Engineering in 

Medicine and Biology Society (EMBS). 

Iturrate, I., Montesano, L., and Minguez, J. (2010). Robot Reinforcement Learning using EEG-based 

reward signals. In IEEE International Conference on Robotics and Automation (ICRA), 2010. 

Ivansson, J. (2002). Situation Assessment in a Stochastic Environment using Bayesian Networks, Masters 

Thesis at Department of Electrical Engineering, Linköping University, 

Ives, J. R. and Schomer, D. L. (1988). A 6-pole filter for improving the readability of muscle contaminated 

eegs. Electroencephalogr. Clin. Neurophysiol., 69(5):486–490. 

Jacko, J.A., Sears, A. (2002). The Human-computer Interaction Handbook: Fundamentals, Evolving 

Technologies and Emerging Applications, 2nd ed. CRC. 

Jain, A., Hong, L., and Kulkarni, Y. (1999). A Multimodal Biometric System Using Fingerprint, Face and 

Speech, Proceedings of 2nd Int'l Conference on Audio- and Video-based Biometric Person Authentication, 

Washington D.C., pp. 182-187 

Jezernik, S., Colombo, G., and Morari, M. (2004). Automatic gait-pattern adaptation algorithms for 

rehabilitation with a 4-DOF robotic orthosis, IEEE Transactions on Robotics and Automation, vol. 20, no. 

3, pp. 574-582. 

Jung, T., Polani, D., and Stone, P. (2011). Empowerment for continuous agent-environment systems. 

Adaptive Behaviour. Published online 13. January 2011. 

Jung, T.P., Makeig, S., Stensmo, M. and Sejnowski, T.J. (1997). Estimating alertness from the eeg power 

spectrum. IEEE Transactions on Biomedical Engineering, 44(1):60–69. 

186


Kaplan, F. and Oudeyer, P.-Y. (2004). Maximizing learning progress: an internal reward system for 

development. In Iida, F., Pfeifer, R., Steels, L., and Kuniyoshi, Y., editors, Embodied Artificial 

Intelligence, volume 3139 of LNAI, pages 259–270. Springer. 

Kawamura, K., Dodd, W., Ratanaswasd, P. (2004). Robotic body-mind integration: next grand challenge 

in robotics, in Proceedings of the 2004 IEEE International Workshop on Robot and Human Interactive 

Communication (Kurashiki, Okayama, Japan), pp. 23–28. 

Kawamura, K., Peters, A.R., Wilkes, M.D., Alford, A.W, Rogers, T.E. (2000). ISAC: Foundations in 

human-humanoid interaction. IEEE Intelligent Systems 15 (4): 38–45. 

Kawamura, K., Peters, R.A., Bodenheimer, R.E., Sarkar, N., Park, J., Clifton, C.A., Spratley, A.W., 

Hambuchen, K.A. (2004). A parallel distributed cognitive control system for a humanoid robot. 

International Journal of Humanoid Robotics 1 (1): 65–93. 

Kawato, M., (1999). Internal models for motor control and trajectory planning. Current Opinion in 

Neurobiology, 9:718-727. 

Kazerooni, H. and Steger, R. (2006). The Berkeley Lower Extremity Exoskeleton, Journal of Dynamic 

Systems, Measure- ment and Control, vol. 128, pp. 14-25. 

Kelly, J. L. (1956). A new interpretation of information rate. Bell System Technical Journal, 35:917–926. 

Kelly, S.P., Docktree, P., Reilly, R.B., and Robertson, I.H. (2003). Eeg alpha power and coherence time 

courses in a sustained attention task. Proceedings of the International Conference on Neural Engineering, 

pages 83–86. 

Kieras, D. E., Meyer, D. E., (1997). An overview of the EPIC architecture for cognition and performance 

with application to human–computer interaction. Human-Computer Interaction, 12(4): 391–438. 

Kirstein, K.-U., Sedivy, J., Salo ,T., Hagleitner, C., Vancura, T., and Hierlemann, A. (2005). A CMOSbased 

Tactile Sensor for Continuous Blood Pressure Monitoring, Proceedings IEEE Conference, Design, 

Automation and Test in Europe. 

Klyubin, A. S., Polani, D., and Nehaniv, C. L. (2004). Organization of the information flow in the 

perception-action loop of evolved agents. In Proceedings of 2004 NASA/DoD Conference on Evolvable 

Hardware, pages 177–180. IEEE Computer Society. 

Klyubin, A. S., Polani, D., and Nehaniv, C. L. (2005a). All else being equal be empowered. In Advances 

in Artificial Life, European Conference on Artificial Life (ECAL 2005), volume 3630 of LNAI, pages 

744–753. Springer. 

Klyubin, A. S., Polani, D., and Nehaniv, C. L. (2005b). Empowerment: A universal agent-centric measure 

of control. In Proc. IEEE Congress on Evolutionary Computation, 2-5 September 2005, Edinburgh, 

Scotland (CEC 2005), pages 128–135. IEEE. 

Klyubin, A. S., Polani, D., and Nehaniv, C. L. (2008). Keep your options open: An information-based 

driving principle for sensorimotor systems. PLoS ONE, 3(12):e4018. 

Klyubin, A., Polani, D., and Nehaniv, C. (2007). Representations of space and time in the maximization of 

information flow in the perception-action loop. Neural Computation, 19(9):2387–2432. 

Koehler, S., Lauer, P., Schreppel, T., Jacob, C., Heine, M., Boreatti-Hummer, A., Fallgatter, A. J., and 

Herrmann, M. J. (2009). Increased eeg power density in alpha and theta bands in adult adhd patients. 

Journal of Neural Transmission, 116(1):97–104. 

Kong, K. and Jeon, D. (2006). Design and Control of an Exoskeleton for the Elderly and Patients, 

IEEE/ASME Trans-actions on Mechatronics, vol. 11, no. 4, pp. 428-423. 

Kong, K., Moon, H., Hwang, B., Jeon, D., and Tomizuka, M. (2009). Impedance Compensation of 

SUBAR for Back-Drivable Force-Mode Actuation, IEEE Transactions on Robotics, vol. 25, pp. 512-521. 

Körding, K. P. and Wolpert, D. M. (2004). Bayesian integration in sensorimotor learning. Nature, 

427:244–247. 

Kornhuber, H.H., Deecke, L. (1964). Hirnpotentialanderungen beim menschen vor und nach 

willkurbewegungen, dargestellt mit magnetband-speicherung und ruckwartsanalyse. Pflugers Arch, 

281(52). 

187


Kornhuber, H.H., Deecke, L. (1965). Hirnpotentiala nderungen bei willkurbewegungen und passiven 

bewegungen des menschen: Bereitschaftspotential und rea erente potentiale. Pflugers Arch, 284:1–17. 

Kott, A. and Ownby, M. (2005). Tools for real-time anticipation of enemy actions in tactical ground 

operations. In 10th International Command and Control Research and Technology Symposium, McLean, 

VA. 

Krauledat, M., Dornhege, G., Blankertz, B., Losch, F., Curio, G. and Mller, K.-R. (2004). Improving speed 

and accuracy of brain-computer interfaces using readiness potential features. Engineering in Medicine and 

Biology Society, pages 4511–4515. 

Krichmar, J. L., Edelman, G. M., (2003). Brain-based devices: Intelligent systems based on principles of 

the nervous system. In IEEE/RSJ International Conference on Intelligent Robotics and Systems, Las 

Vegas, Nevada. 

Krut, S., Benoit, M., Dombre, E., and Pierrot, F. (2010). MoonWalker, a Lower Limb Exoskeleton able to 

Sustain Bodyweight using a Passive Force Balancer, in 2010 IEEE International Conference on Robotics 

and Automation, pp.2215-2220. 

Krut, S., Benoit, M., Dombre, E., and Pierrot, F. (2010). MoonWalker, a Lower Limb Exoskeleton able to 

Sustain Bodyweight using a Passive Force Balancer, in 2010 IEEE International Conference on Robotics 

and Automation, pp.2215-2220. 

Kulic, D. and Croft, E.A. (2007). Affective State Estimation for Human-Robot Interaction, IEEE 

Transactions on Robotics, Vol 23, No 5, pp. 991-999. 

Kulic, D. and Croft, E.A. (2007). A ective state estimation for human-robot interaction. IEEE transaction 

on Robotics, 23(5):991–1000. 

Kutas, M. and Donchin, E. (1980). Preparation to respond as manifested by movement-related brain 

potentials. Brain Res, 202:95–115. 

Kuzmitcheva, O., Gräser, A., (2009), A Concept of a Robotic System for Safe Sampling Procedure during 

Reconnaissance of CBRN-Disaster, 3rd Int. Advanced Robotics Programme IARP-RISE’2009 - Brussels, 

Belgium. 

Lagerlund, T.D., Sharbrough, F.W., and Busacker, N.E. (1997). Spatial filtering of multichannel 

electroencephalographic recordings through principal component analysis by singular value 

decomposition. Clin. Neurophysiol., 14(1):73–82. 

Lam, T., Wirz, M., Lünenburger, L., and Dietz, V. (2008). Swing phase resistance enhances exor muscle 

activity during treadmill locomotion in incomplete spinal cord injury, Neurorehabil Neural Repair, vol. 22, 

no. 5, pp. 438-446. 

Lambert, D. A. (1999). Assessing Situations, Proceedings of 1999 Information, Decision and Control, pp. 

503 - 508. IEEE. 

Lambert, D. A. (2001). Situations for Situation Awareness, Proceedings of the Fourth International 

Conference on International Fusion, Montreal 

Lambert, D. A. (2003). Grand Challenges of Information Fusion, Proceedings of the Sixth International 

Conference on Information Fusion. Cairns, Australia. pp. 213 -219. 

Lambert, D.A. (2006). Formal Theories for Semantic Fusion Lambert, 9th International Conference on 

Information Fusion, pp 1 – 8, 

Landragin, F. (2007). Physical, semantic and pragmatic levels for multimodal fusion and fission, Seventh 

International Workshop on Computational Semantics (IWCS-7), Tilburg, The Netherlands 

Lang, C.E. and Bastion, A.J. (2002). Cerebellar damage impairs automaticity of a recently practiced 

movement. J Neurohysiol, 87:1336–1347. 

Laughlin, S. B. (2001). Energy as a constraint on the coding and processing of sensory information. 

Current Opinion in Neurobiology, 11:475–480. 

Laughlin, S. B., de Ruyter van Steveninck, R. R., and Anderson, J. C. (1998). The metabolic cost of neural 

information. Nature Neuroscience, 1(1):36–41. 

188


Lawrence RC, Helmick CG, Arnett FC, Deyo RA, Felson DT, Giannini EH et al. (1999). Estimates of the 

prevalence of arthritis and selected musculoskeletal disorders in the United States. Arthritis Rheum, 

41:778-799. 

Lee, S. and Sankai, Y. (2002). Power assist control for leg with HAL-3 based on virtual torque and 

impedance adjustment, in IEEE International Conference on Systems, Man and Cybernetics, vol. 4, pp. 6- 

9. 

Lehmann, J., Laird, J., Rosenbloom, P (2006), A Gentle Introduction to SOAR, an Architecture for Human 

Cognition: 2006 Update, Web: http://ai.eecs.umich.edu/soar/sitemaker/docs/misc/GentleIntroduction- 

2006.pdf 

Leigh R. Hochberg et al. (2006). Neuronal ensemble control of prosthetic devices by a human with 

tetraplegia. Nature, 442:164–171. 

Liang, H. and Wang, H. (2003). Top-down anticipatory control in prefrontal cortex. Theory in 

Biosciences, 122(1):70–86. 

Linden, M., Habib, T. and Radojevic, V. (1996). A controlled study of the effects of eeg biofeedback on 

cognition and behavior of children with attention deficit disorder and learning disabilities. Applied 

Psychophysiology and Biofeedback, 21(1):35–49. 

Lindsley, D. (1952). Psychological phenomena and the electroencephalogram. Electroenceph. Clin. 

Neurophysiol. 

Lizier, J., Prokopenko, M., and Zomaya, A. (2007). Detecting non-trivial computation in complex 

dynamics. In (Almeida e Costa et al., 2007), pages 895–904. 

Lloyd, S. (1991). Causality and information flow. In Atmanspacher, H. and Scheingraber, H., editors, 

Information Dynamics, pages 131–142. Plenum Press. 

Lopez, E., Iturrate, I., Montesano, L., and Minguez, J. (2010). Real-time recognition of feedback 

errorrelated potentials during a time-estimation task. In Annual Conference of IEEE Engineering in 

Medicine and Biology Society (EMBS). 

Lotze, M., Braun, C., Birbaumer, N., Anders, S., and Cohen Cohen L. G. (2003). Motor learning elicited 

by voluntary drive, Brain, vol. 126, no. 4, pp. 866-872. 

Low, K. H., Liu, X., and Yu, H. (2005). Development of NTU Wearable Exoskeleton System for Assistive 

Technologies, in Proceedings of the 2005 IEEE International Conference on Mechatronics & Automation, 

pp. 1099-1106. 

Lubar, J.F. (1991). Discourse on the development of eeg diagnostics and biofeedback for attentiondeficit/hyperactivity 

disorders. Applied Psychophysiology and Biofeedback, 16(3):201–225. 

Lungarella, M. and Sporns, O. (2005). Information self-structuring: Key principle for learning and 

development. In Proceedings of 4th IEEE International Conference on Development and Learning, pages 

25–30. IEEE. 

Lungarella, M. and Sporns, O. (2006). Mapping information flow in sensorimotor networks. PLoS 

Computational Biology, 2(10). 

Luth, T., Ojdanic, D., Friman, O., Prenzel, O., and Graser, A. (2007). Low level control in a 

semiautonomous rehabilitation robotic system via a Brain-Computer Interface. 

Maass, W. (2003). Computation with spiking neurons. In Arbib, M. A., editor, Handbook of Brain Theory 

and Neural Networks, pages 1080–1083. MIT Press, Cambridge, 2 edition. 

Macfarlane GJ, Croft PR, Schollum J, Silman AJ. Widespread pain: is an improved classification possible? 

Journal of Rheumatology 1996; 23(9):1628-1632. 

Maggi, L., Parini, S., Perego, P., and Andreoni, G. (2008). Bci++: an objectoriented bci prototyping 

framework. In 4th international brain-computer interface workshop. 

Major and Chronic Diseases Report 2007, Directorate-General for Health & consumers, European 

Commission, April 2008 

189


Mankala, K. K., Banala, S. K., and Agrawal, S. K. (2007). Passive Swing Assistive Exoskeletons for 

Motor-Incomplete Spinal Cord Injury Patients, in Proceedings of the 2007 IEEE International Conference 

on Robotics and Automation, pp. 3761-3766. 

Marchal-Crespo, L. and Reinkensmeyer, D (2008). Haptic guidance can enhance motor learning of a 

steering tasks, Journal of motor behaviour, vol. 40, no. 6, pp. 545-557. 

Marchal-Crespo, L. and Reinkensmeyer, D. J. (2009). Review of control strategies for robotic movement 

training after neurologic injury, Journal of NeuroEngineering and Rehabilitation, vol. 6, no. 20. 

Martin, J., Pollock, A., and Hettinger, J. (2010). Microprocessor Lower Limb Prosthetics: Review of 

Current State of the Art, Journal of Prosthetics & Orthotics, vol. 22, no. 3, pp. 183-193. 

Mason, S. (2005). Dry electrode technology: What exists and what is under development? 

Mason, S., et al. (2007). A comprehensive survey of brain interface technology designs. Annnals of 

Biomedical Engineering, 35(2). 

Massey, J. (1990). Causality, feedback and directed information. In Proc. Int. Symp. Inf. Theory Applic. 

(ISITA-90), pages 303–305. 

Mataric, M. J., Eriksson, J., Feil-Seifer, D. J., and Winstein, C. J. (2007). Socially assistive robotics for 

post-stroke rehabilitation, Journal of NeuroEngineering and Rehabilitation, vol. 4, no. 5. 

Matteucci, M., Carabalona, R., Casella, M., Di Fabrizio, E., Gramatica, F., Di Rienzo, M., Snidero, E., 

Gavioli, L. and Sancrotti, M. (2007). Micropatterned dry electrodes for braincomputer interface. Journal 

Microelectronic Engineering, 84(2). 

Mc Farland, D.J. and Wolpaw, J.R. (2008).. Brain-computer interface operation of robotic and prosthetic 

devices. IEEE Computer Society, pages 52–56. 

McAllester, D. A. (1999). Pac-bayesian model averaging. In Proceedings of the Twelfth Annual 

Conference on Computational Learning Theory, Santa Cruz, CA, pages 164–170, New York. ACM. 

McCarney, R., Croft, P.R. (1999). Knee pain. In: Crombie IK, Croft PR, Linton SJ, LeResche L,Von Korff 

M, editors. Epidemiology of pain. Seattle: IASP Press: 299-313. 

Meier, U., Stiefelhagen, R., Yang, J., and Weibel, A. (2000). Towards Unrestricted Lip Reading, 

International Journal of Pattern Recognition and Artificial Intelligence, vol. 14, no. 5, pp. 571-585. 

Meissner, A., Schönfeld, W. (2005). Data Communication Between the German NBC Reconnaissance 

Vehicle and Its Control Center Uniut, From Integrated Publication and Information Systems to 

Information and Knowledge Environments, Lecture Notes in Computer Science, 2005, Volume 3379/2005 

Merx, H., Dreinhöfer, K.E.., Schrader, P., Sturmer, T., Puhl, W., Gunther, K.P., Brenner, H. (2003). 

International variation in hip replacement rates. Ann Rheum Dis, 62-3:222-6. 

Metta, G., Fitzpatrick, P., Natale, L. (2006). Yarp: yet another robot platform. International Journal on 

Advanced Robotics Systems 3(1), 43–48 

Millan, J., Franze, M., Mourino, J., Cincotti, F. and Babiloni, F. (2002). Relevant eeg features for the 

classification of spontaneous motor-related tasks. Biol. Cybern., 86(2):89–95. 

Millan, J.d.R., Renkens, Mourino, F., J., and Gerstner, W. (2004). Noninvasive Brain-Actuated Control of 

a Mobile Robot by Human EEG. IEEE Transactions on Biomedical Engineering, 51(6). 

Miltner, W.H.R., Braun, C.H. and Coles, M.G.H. (1997). Event-related brain potentials following 

incorrect feedback in a time-estimation task: Evidence for a generic neural system for error detection. 

Journal of Cognitive Neuroscience, 9(6):788–798. 

Möller, M. and Polani, D. (2008). Common concepts in agent groups, symmetries and conformity in a 

simple environment. In (Bullock et al., 2008), pages 420–427. 

Modular Reconnaissance and EOD Robot ASENDRO, 2007. [online] Available at: 

[Accessed 

15 July 2011]. 

190


Monastra, V.J., Lynn, S., Linden, M., Lubar, J.F., Gruzelier, J., and LaVaque, T.J. (2005). 

Electroencephalographic biofeedback in the treatment of attention-deficit/hyperactivity disorder. Applied 

Psychophysiology and Biofeedback, 30(2):95–114. 

Montemerlo, M., Roy, N., and Thrun, S. (2003). Perspectives on standardization in mobile robot 

programming: the carnegie mellon navigation (carmen) toolkit. In IEEE/RSJ International Conference on 

Intelligent Robots and Systems, volume 3, pages 2436–2441. 

Morash, V., Bai, O., Furlani, S., Lin, P., and Hallett, M. (2008). Classifying eeg signals preceding right 

hand, left hand, tongue, and right foot movements and motor imageries. Clinical Neurophysiology, 

119(11):2570 – 2578. 

Mormann, F., Elger, C. E., and Lehnertz, K. (2006). Seizure anticipation: from algorithms to clinical 

practice. Curr. Opin. Neurol., 19:187–193. 

Muller-Putz, G.R. et al. (1977). Eeg-based neuroprosthesis control: A step towards clinical practise. 

Neuroscience Letters, 382(1-2):169–174. 

Murphy, R.R (1996). Biological and Cognitive Foundation of Intelligent Sensor Fusion, IEEE 

Transactions on Systems, Man and cyberntices-Part A: Systems and Humans,Vol 26, No1, pp. 42-51 

Myklebust, H., Nunes, N., Hallén, J., and Gamboa, H. (2011). Morphological analysis of acceleration 

signals in cross-country skiing, Proceedings of Biosignals - International Conference on Bio-inspired 

Systems and Signal Processing (BIOSTEC 2011), Rome, Italy. 

Nadin, M. (2005). Anticipating extreme events — the need for faster-than-real-time models. In Extreme 

Events in Nature and Society, Frontiers Collection, pages 21–45. Springer, New York/Berlin. 

Natvig, B, Nessiøy, I, Bruusgaard, D, Rutle, O. (1995). Musculoskeletal symptoms in a local community. 

Eur J Gen Pr; 1:25-28. 

Natvig, B., Bruusgaard, D., Eriksen, W. (2001). Localised low back pain and low back pain as part of 

widespread musculoskeletal pain, two different disorders? Scand J Rehabil Med, 33:21-25 

Nery, B. and Ventura, R. (2010). Online event segmentation in active perception using adaptive strong 

anticipation. Technical Report RT-701-1, Instituto de Sistemas e Robótica, Lisboa, Portugal. 

Newell, A., (1990), Unified Theories of Cognition. Cambridge MA: Harvard University Press. 

Nieuwenhuis, S., Holroyd, C.B., Mola, N., and Coles, M.G.H. (2004). Reinforcement-related brain 

potentials from medial frontal cortex: origins and functional significance. Neuroscience and Biobehavioral 

Reviews, 28:441–448. 

Nilsson, N. J., (1984). Shakey the Robot. Technical note 323 AI Center, SRI International Menlo Park, 

CA. 

Nirenburg, S., Lesser, V., Nyburg, E. (1989). Controlling a Language Generation Planner, Proceedings of 

IJCAI-89, 1524-1530. 

Nock, H. J., Iyengar, G., and Neti, C. (2002). Assessing Face and Speech Consistency for Monologue 

Detection in Video, Proceedings of ACM Multimedia, Juan-les-Pins, France. 

Novák, V., Perfilieva, I. and Močkoř, J. (1999) Mathematical principles of fuzzy logic Dodrecht: Kluwer 

Academic. ISBN 0-7923-8595-0 

Ntnen, R. (1982). Processing negativity: An evoked-potential reflection of selective attention. 

Psychological Bulletin, 92(3):605–640. 

Ntnen, R. (1990). The role of attention in auditory information processing as revealed by event-related 

potentials and other brain measures of cognitive function. Behavioral and Brain Sciences, 13:201–288. 

OCZ Technology, n.d. NIA GAME CONTROLLER. [online] Available at 

[Accessed 13 May 2011].NeuroSky, n.d. 

What We Do. [online] Available at < http://www.neurosky.com/People/WhatWeDo.aspx > [Accessed 13 

May 2011]. 

191


OFRO – Mobile outdoor surveillance at the highest level [online] Available at: 

[Accessed 20 July 2011] 

OpenEEG, n.d. BrainBay – an OpenSource Biosignal project. [online] Available 


OpenEEG, n.d. BWView -- recorded brain-wave viewing application. [online] Available 


OpenEEG, n.d. Welcome to the OpenEEG project. [online] Available at 

http://openeeg.sourceforge.net/doc/ [Accessed 13 May 2011]. 

Orchard, R. (2001). Fuzzy Reasoning in Jess: The FuzzyJ Toolkit and FuzzyJess, Proceedings of the Third 

International Conference on Enterprise Information Systems (ICEIS), pp. 553-542 

Palmer, K.T., Walsh, K., Bendall, H., Cooper, C., Coggon, D. (2000). Back pain in Britain: comparison of 

two prevalence surveys at an interval of 10 years. BMJ; 320:1577-1578. 

Parunak, H. V. D., Brueckner, S., and Savit, R. (2004). Universality in multi-agent systems. In 

Proceedings of Third International Joint Conference on Autonomous Agents and Multi-Agent Systems 

(AAMAS 2004), pages 930–937. IEEE. 

Parush, N., Tishby, N., and Bergman, H. (2011). Dopaminergic balance between reward maximization and 

policy complexity. Frontiers in Systems Neuroscience. 

Passino, K. M., Yurkovich, S., (1998). Fuzzy control. Menlo Park: Addison Wesley Longman. 

Paul, C. (2006). Morphological computation: A basis for the analysis of morphology and control 

requirements. Robotics and Autonomous Systems, 54(8):619–630. 

Pearl, J. (2000). Causality: Models, Reasoning and Inference. Cambridge University Press, Cambridge, 

UK. 

Peters, B.O., Pfurtscheller, G. and Edlinger, G. (2001). Automatic di erentiation of multichannel eeg 

signals. IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, 48(1). 

Peters, J., Janzing, D., and SchÃlkopf, B. (2010). Identifying cause and effect on discrete data using 

additive noise models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence 

and Statistics (AISTATS 2010), pages 1–8. 

Peters, J., Janzing, D., Gretton, A., and SchÃlkopf, B. (2009). Detecting the direction of causal time 

series. In Danyluk, A., Bottou, L., and Littman, M. L., editors, Proceedings of the 26th International 

Conference on Machine Learning (ICML 2009), pages 801–808, New York, NY. ACM Press. 

Petridis, M., Knight, B. (2001). A blackboard architecture for a hybrid CBR system for scientific software. 

Pfeifer, R. and Bongard, J. (2007). How the Body Shapes the Way We think: A New View of Intelligence. 

Bradford Books. 

Pfurtscheller, G. et all. (2000). Brain oscillations control hand orthosis in a tetraplegic. Neuroscience 

Letters, 292(3):211–214. 

Pfurtscheller, G., Flotzinger, D. and Neuper, C. (1994). Di erentiation between finger, toe and tongue 

movement in man based on 40 hz eeg. Electroenceph. Clin. Neurophysiol., 90:456–460. 

Pfurtscheller, G., Neuper, C., Andrew, C. and Edlinger, G. (1997). Foot and hand area mu rhythms. Int J 

Psychophysiol, 26:121–35. 

Picard, R.W., Papert, S., Bender, W., Blumberg, B., Breazeal, C., Cavallo, D., Machover, T., Resnick, M., 

Roy, D. and Strohecker, C. (2004). Affective learning – a manifesto, BT Technology Journal 22 (4), pp. 

253-269 

Poggendorf, I., (2004) Einsatz eines Serviceroboters zur Automatisierung der Probenentnahem und des 

Probenmanagements während Kultivierung tierischer Zellen in einer Technikumsumgebung, Universität 

Bielefeld 

192


Poh, M-Z, Swenson, N.C. and Picard, R.W. (2010). A Wearable Sensor for Unobtrusive, Long-Term 

Assessment of Electrodermal Activity, IEEE Transactions on Biomedical Engineering, vol. 57, no. 5, 

pp.1243-52. 

Polani, D. (2009). Information: Currency of life? HFSP Journal, 3(5):307–316. 

Polani, D., Nehaniv, C., Martinetz, T., and Kim, J. T. (2006). Relevant information in optimized 

persistence vs. progeny strategies. In (Rocha et al., 2006), pages 337–343. 

Polani, D., Sporns, O., and Lungarella, M. (2007). How information and embodiment shape intelligent 

information processing. In Lungarella, M., Iida, F., Bongard, J., and Pfeifer, R., editors, Proc. 50th 

Anniversary Summit of Artificial Intelligence, pages 99–111. Springer-Verlag: Berlin, Heidelberg, New 

York. 

Popescu, F., Fazli, S., Badower, Y., Blankertz and Mller, KR. (2007). Single trial classification of motor 

imagination using 6 dry eeg electrodes. PLoS ONE, 2(2). 

Porr, B., von Ferber, C., and Wörrgötter, F. (2003). Iso learning approximates a solution to the inversecontroller 

problem in an unsupervised behavioral paradigm. Neural Computation, 15(4):865–884. 

Prevalence of disability and long-standing health problems (unintentional injuries only, population aged 15 

to 64). Labour Force Survey. Eurostat, 2002. 

Prokopenko, M., Gerasimov, V., and Tanev, I. (2006a). Evolving spatiotemporal coordination in a 

modular robotic system. In Nolfi, S., Baldassarre, G., Calabretta, R., Hallam, J. C. T., Marocco, D., Meyer, 

J.-A., Miglino, O., and Parisi, D., editors, From Animals to Animats 9: 9th International Conference on the 

Simulation of Adaptive Behavior (SAB 2006), Rome, Italy, volume 4095 of Lecture Notes in Computer 

Science, pages 558–569, Berlin, Heidelberg. Springer. 

Prokopenko, M., Gerasimov, V., and Tanev, I. (2006b). Measuring spatiotemporal coordination in a 

modular robotic system. In (Rocha et al., 2006), pages 185–191. 

PRWeb, http://www.prweb.com/releases/orthopedic_prosthetics/knee_prosthesis/prweb8072141.htm 

Friday, May 20, 2011 Global Orthopedic Prosthetics Market to Reach US$19.4 Billion by 2015, 

According to a New Report by Global Industry Analysts, Inc. San Jose, CA (Vocus/PRWEB) January 19, 

2011 

PRWeb, http://www.wheelchairbilling.com/a262996-world-wheelchairs-market-to-exceed-4-2.cfm San 

Jose, CA (PRWEB) March 6, 2008 

Ramoser, H., Muller-Gerking, J. and Pfurtscheller, G. (2000). Optimal spatial filtering of single trial eeg 

during imagined hand movement. IEEE Trans. Rehab. Eng., 8(4):441–446. 

Rani, P., Sarkar, N., Smith, C.A., and Kirby, L.D. (2004). Anexity Detecting Robotic System – Towards 

Implicit Human-Robot Collaboration, Robotica, Volume 22, pp. 85-95. 

Rani, P., Sims, J., Brackin, R., and Sarkar, N. (2002). Online Stress Detection using Psychological Signal 

for Implicit Human-Robot Cooperation, Robotica, Volume 20 (6) pp. 673-686. 

Rebsamen, B., Teo, C.L., Zeng, Q., Ang, M.H., Burdet, E., Guan, C., Zhang, H., and Laugier, C. (2007). 

Controlling a Wheelchair Indoors Using Thought. IEEE Intelligent Systems, 07:1541–1672. 

Reddy, D. R., Erman, L., Neely, R. (1973). A model and a system for machine recognition of speech, 

IEEE Transactions on Audio and Electroacoustics, vol. AU-21, 229-238. 

Reinkensmeyer , D., Aoyagi, D., Emken, J., Galvez, J., Ichinose, W., Kerdanyan, G., Maneekobkunwong, 

S., Minakata, K., Nessler, J., Weber, R., Roy, R., de Leon, R., Bobrow, J., Harkema, S., and Edgerton, V. 

(2006). Tools for understanding and optimizing robotic gait training, Journal of Rehabilitation Research & 

Development, vol. 43, no. 5, pp. 657-670. 

Reinkensmeyer, D., Galvez, J., Marchal, L., Wolbrecht, E., and Bobrow, J. (2007). Some key problems for 

robot-assisted movement therapy research: a perspective from the University of California at Irvine, in 

Proceedings of the IEEE 10th International Conference on Rehabilitation Robotics, pp. 1009-1015. 

Reisman, D. S., Wityk, R., Silver, K., and Bastian, A. J. (2007). Locomotor adaptation on a split-belt 

treadmill can improve walking symmetry post-stroke, Brain, vol. 130, no. 7, pp. 1861-1872. 

193


Renard, Y., Lotte, F., Gibert, G., Congedo, M., Maby, E., Delannoy, V., Bertrand, O. and Lecuyer, A. 

(2010). Openvibe: An open-source software platform to design, test and use brain-computer interfaces in 

real and virtual environments. Presence: teleoperators and virtual environments, 19:35–53. 

Reprinted from the Lancet, 353, Tunstall-Pedoe H, Kuulasmaa K, Mahonen M, et al. For the WHO 

MONICA project, Contribution of trends in survival and coronary-event rates to changes in coronary heart 

disease mortality: 10-year results from 37 WHO MONICA Project populations, 1547-57, 1999, with 

permission from Elsevier. 

Rheinmetall robotic system (http://www.rheinmetall-detec.com) 

Riadh, F. (2006). Analysing spatial-temporal geographic information based on blackboard architecture and 

multi-agent systems, IJCSNS International Journal of Computer Science and Network Security, VOL.6 

No.8A. 

Ridding, M. C. and Rothwell, J. C. (1999). A_erent input and cortical organisation: a study with magnetic 

stimulation, Experimental Brain Research, vol. 126, no. 4, pp. 536-544. 

Rieke, F., Warland, D., de Ruyter van Steveninck, R., and Bialek, W. (1999). Spikes. A Bradford Book. 

MIT Press. 

Riener, R., Lunenburger, L., Jezernik, S., Anderschitz, M., Colombo, G., and Dietz, V. (2005). Patientcooperative 

strategies for robot-aided treadmill training: _rst experimental results, IEEE Transactions on 

Neural Systems and Rehabilitation Engineering, vol. 13, no. 3, pp. 380-394. 

Riera, A., Soria-Frischi, A., Caparrini, M., Grau, C. and Ruffini, G. (2008). Unobtrusive biometric system 

based on ectroencephalogram analysis. ERASIP Journal on Advances in Signal Processing. 

Rivera-Ruiz, M., Cajavilca, C., and Varon, J. (2008). Einthoven’s String Galvanometer - The First 

Electrocardiograph, Tex Heart Inst J, 35(2):174-8 

Robowatch Industries Ltd. (http://www.robowatch.de) 

Rocha, L. M., Bedau, M., Floreano, D., Goldstone, R., Vespignani, A., and Yaeger, L., editors (2006). 

Proc. Artificial Life X. 

Rohrbaugh, J.W., Syndulko, K. and Lindsley, D.B. (1976). Brain wave components of the contingent 

negative variation in humans. Science, 191:1055–7. 

ROS.org, n.d. ROS. [online] Available at: [Accessed 13 May 2011]. 

Rosahl, S.K. and Knight, R.T. (1995). Role of prefrontal cortex in generation of the contingent negative 

variation. Cereb Cortex, 5:123–34. 

Ross, R. (2004). The SharC Cognitive Control Architecture, Technical Report 

Ross, A., Jain, A., Information Fusion in biometrics, Pattern Recognition Letters, vol. 24, no. 13, pp. 2115- 

2121, 2003 

Rumelhart D. E., McClelland J. L., and The PDP Research Group, (1986) Eds.,Parallel Distributed 

Processing: Explorations in the Microstructure of Cognition. Cambridge: The MIT Press. 

Russell, S., Norvig, P. (2003). Artificial Intelligence - A Modern Approach, Second Edition, Prentice Hall, 

New Jersey. 

Sadeh, N. M. (1998). A Blackboard Architecture for Integrating Process Planning and Production 

Scheduling, Concurrent Eng.: Res. and Apps, vol. 6, no. 2. 

Saerens, M., Achbany, Y., Fuss, F., and Yen, L. (2009). Randomized shortest-path problems: Two related 

models. Neural Computation, 21:2363–2404. 

Saffiotti, A., (2004) Platforms for Rescue Operations, AASS Mobile Robotics Laboratory, Örebro 

University, Örebro, Sweden. 

194


Saglia, J. A., Tsagarakis, N. G., Dai, J. S., and Caldwell, D. G. (2010). Control Strategies for Ankle 

Rehabilitation using a High Performance Ankle Exerciser, in Proceedings of the 2010 IEEE International 

Conference on Robotics and Automation, pp. 2221-2227. 

Salge, C. and Polani, D. (2011). Digested information as an information theoretic motivation for social 

interaction. Journal of Artificial Societies and Social Simulation (JASSS), 14(1):5. 

Salvo, P., Di Francesco, F., Costanzo, D., Ferrari, C., Trivella, M.G. and De Rossi, D. (2010). A Wearable 

Sensor for Measuring Sweat Rate, IEEE Sensors Journal, vol. 10, no. 10, pp. 1557-58. 

Sandini, G., Metta, G., and Vernon, D. (2007). The iCub Cognitive Humanoid Robot: An Open-System 

Research Platform for Enactive Cognition, in 50 Years of AI, M. Lungarella et al. (Eds.), Festschrift, 

LNAI 4850, Springer-Verlag, Heidelberg, pp. 359-370. 

Sanket G. J. (2009). Eeg during pedaling: Brain activity during a locomotion-like task in humans. Master 

Thesis. 

Sawicki, G., Gordon, K., and Ferris, D. (2005). Powered lower limb orthoses: applications in motor 

adaptation and rehabilitation, in 9th International Conference on Rehabilitation Robotics (ICORR 2005), 

June-July, pp. 206-211. 

Schalk, G., McFarland, D.J., Hinterberger, T., Birbaumer, N., and Wolpaw, J.R. (2004). Bci2000: a 

general-purpose brain-computer interface (bci) system. Biomedical Engineering, IEEE Transactions on, 

51(6):1034–1043. 

Schmidhuber, J. (1991). A possibility for implementing curiosity and boredom in model-building neural 

controllers. In Meyer, J. A. and Wilson, S. W., editors, Proc. of the International Conference on Simulation 

of Adaptive Behavior: From Animals to Animats, pages 222–227. MIT Press/Bradford Books. 

Schmidhuber, J. (2002). Exploring the predictable. In Ghosh, A. and Tsutsui, S., editors, Advances in 

Evolutionary Computing, pages 579–612. Springer. 

Schmidt, D.C. (1998). The design and performance of real-time object request brokers. Computer 

Communications, 21(4), 294-324. 

Schmidt, H., Hesse, S., and Bernhardt, R. (2005). Haptic Walker - A novel haptic foot device, ACM 

Transactions on Applied Perception, vol. 2, pp. 166-180. 

Schmidt, H., Piorko, F., Bernhardt, R., Kruger, J., and Hesse, S. (2005). Synthesis of perturbations for gait 

rehabilitation robots, in Proceedings of the 2005 IEEE International Conference on Rehabilitation 

Robotics, pp. 74-77. 

Schneider, W. (1999). Working memory in a multilevel hybrid connectionist control architecture (CAP2). 

In Miyake, A., Shah, P., editors, Models of Working Memory: Mechanisms of Active Maintenance and 

Executive Control, New York:Cambridge University Press. 

Schrater, P. and Kersten, D. (2002). Vision, psychophysics and bayes. In Rao, R. P. N., Olshausen, B. A., 

and Lewicki, M., editors, Probabilistic Models of the Brain: Perception and Neural Function, Neural 

Information Processing Series, pages 37–60. A Bradford Book. The MIT Press. 

Schreiber, T. (2000). Measuring information transfer. Phys. Rev. Lett., 85:461–464. 

Searle, A. and Kirkup, L. (2010). A direct comparison of wet, dry and insulating bioelectric recording 

electrodes. Physiological Measurement, 21(2):271–283. 

Seeberg, T., Hjelstuen, M., Austad, H.O., Færevik, A.L.H., Tjønnås, M.S., Storholmen, T.C.B. (2011). 

Smart Textiles - Safety for Workers in Cold Climate, Submitted to Ambience 2011 

Shalfield, R. (2005). Flint Reference, Logic Programming Associates, http://www.lpa.co.uk, London 

Shalizi, C. R. (2001). Causal Architecture, Complexity and Self-Organization in Time Series and Cellular 

Automata. PhD thesis, University of Wisconsin-Madison. 

Shalizi, C. R. and Crutchfield, J. P. (2002). Information bottlenecks, causal states, and statistical relevance 

bases: How to represent relevant information in memoryless transduction. Advances in Complex Systems, 

5:1–5. 

195


Shi, Y., Choi, E.H.C., Ruiz, N., Chen, F., and Taib, R. (2007). Galvanic skin response (gsr) as an index of 

cognitive load. In Conference on Computer and Human Interaction, pages 2651–2656. 

Shibasaki, H. and Hallett, M. (2006). What is the bereitschaftspotential. Clinical Neurophysiology, 

117(11):2341–56. 

Sinkjaer, T., and Popovic, D.B. (2005). Trends in the Rehabilitation of Hemiplegic Subjects, Journal of 

Automatic Control, vol. 15, pp. 1-10. 

Smith, S., Ow, P. S. (1985). The Use of Multiple Problem Decompositions in Time Constrained Planning 

Tasks, Proceedings of IJCAI-85, 1013-1015. 

Sobocki, P., Pugliatti, M., Lauer, L., Kobelt, G. (2007). Estimation of the cost of MS in Europe: 

Extrapolations from a multinational cost study. Mult Scler Jul 10 [epub]. 

Soft-dynamics, n.d. Soft Skill and dynamical Systems. [online] Available at 


Sola, J., Chetelat, O., Sartori, C., Allemann, Y., and Rimoldi, S.F. (2011). Parametric Chest Pulse Wave 

Velocity: A Novel Approach to Assess Arterial Stiffness, IEEE Trans. Biomed. Eng. 58(1), pp. 215-223. 

Srnmo, L., Laguna, P. (2005). Bioelectrical signal processing in cardiac and neurological applications. 

Starlab, n.d. Enobio, Wireless brain monitoring. [online] Available at: 


Stauer, Y., Allemand, Y., Bouri, M., Fournier, J., Clavel, R., Metrailler, P., Brodard , R., and R. F. The 

WalkTrainer - A new generation of walking reeducation device combining orthoses and muscle 

stimulation, IEEE Transactions 

Steels, L. (2004). The autotelic principle. In Iida, F., Pfeifer, R., Steels, L., and Kuniyoshi, Y., editors, 

Embodied Artificial Intelligence: Dagstuhl Castle, Germany, July 7-11, 2003, volume 3139 of Lecture 

Notes in AI, pages 231–242. Springer Verlag, Berlin. 

Stegemann, S.K., Funk, B., and Slotos, T. (2007). A blackboard architecture for workflows. In Johann 

Eder, Stein L. Tomassen, Andreas L. Opdahl, and Guttorm Sindre, editors, CAiSE Forum, volume 247 of 

CEUR Workshop Proceedings. CEUR-WS.org. 

Stepp, N. and Turvey, M. T. (2010). On strong anticipation. Cognitive Systems Research, 11(2):148–164. 

Steudel, B. and Ay, N. (2011). Private communication. Submitted. 

Still, S. (2009). Information-theoretic approach to interactive learning. EPL (Europhysics Letters), 

85(2):28005–28010. 

Sullivan, T.J., Deiss, S.R., Jung, T.P., and Cauwenberghs, G. (2008). A brain-machine interface using drycontact, 

low-noise eeg sensors. Circuits and Systems 2008. ISCAS 2008. IEEE International Symposium 

on, pages 1986–1989. 

Sulzer, J., Roiz, R., Peshkin, M., and Patton, J. (2009). A Highly Backdrivable, Lightweight Knee 

Actuator for Investigating Gait in Stroke, IEEE Transactions on Robotics, vol. 25, no. 3, pp. 539-548. 

Surdilovic, D., Zhang, J., and Bernhardt, R. (2007). STRING-MAN: Wire-robot technology for safe, 

_exible and humanfriendly gait rehabilitation, in Proceedings of the 2007 IEEE 10th International 


Sutton, C., Morrison, C., Cohen, P.R., Moody, J. and Adibi, J. (2004). A Bayesian blackboard for 

information fusion. In: P. Svensson and J. Schubert, Editors, Proceedings of the Seventh International 

Conference on Information Fusion, June 2004 vol. II, International Society of Information Fusion, 

Mountain View, CA, USA 

Symmons, D., Turner, G., Webb, R., Asten, P., Barrett, E., Lunt, M., et al. (2002). The prevalence of 

rheumatoid arthritis in the United Kingdom: new estimates for a new century. Rheumatology; 41:793-800. 

196


Tecce, J.J. (1972). Contingent negative variation (cnv) and psychological processes in man. Psychol Bull, 

77:73–108, 1972. 

TechnoRobot – Rapidly Deployable Remotely Operated Less-Lethal Support Robots. [online] Available 

at:< http://www.army-technology.com/contractors/unmanned_vehicles/technorobot/> [Accessed 15 July 

2011]. 

Tee, K. P. et al. (2008). Augmenting cognitive processes in robot-assisted motor rehabilitation. In 

Proceedings of the 2nd Biennial IEEE/RAS-EMBS International Conference on Biomedical Robotics and 

Biomechatronics, Scottsdale, USA. 

Telerob. [online] Available at: < http://www.army-technology.com/contractors/mines/telerob> [Accessed 

16 July 2011]. 

Teplan, M. (2002). Fundamentals of eeg measurement. Measurement Science Review, 2(2). 

Teplan, M. et al. (2006). Eeg responses to long-term audio-visual stimulation. Int. J. Psychophysiol, 

59:81–90. 

Teplan, M., Krakovsk, A., and tolc ,S. (2009). Eeg characterization of psycho-physiological rest and 

relaxation. In Measurement Conferece. Proceedings of the 7th International Conference. 

The O&P EDGE, http://www.oandp.com/articles/news_2007-09-28_01.asp September 28, 2007 

Thórisson, K. R., List, Di Pirro, T., J., Pennock, C. (2005). A Framework for A.I. Integration, Reykjavik 

University Department of Computer Science Technical Report, RUTR-CS05001. 

Thórisson, K.R., List, T., Pennock, C., DiPirro, J. (2005). Whiteboards: Scheduling Blackboards for 

Semantic Routing of Messages & Streams, AAAI-05 Workshop on Modular Construction of Human-Like 

Intelligences, Twentieth Annual Conference on Artificial Intelligence, Pittsburgh, PA, July 10, 16-23. 

Tishby, N. and Polani, D. (2011). Information theory of decisions and actions. In Cutsuridis, V., Hussain, 

A., and Taylor, J., editors, Perception-Action Cycle: Models, Architecture and Hardware, pages 601–636. 

Springer. 

Tishby, N., Pereira, F. C., and Bialek, W. (1999). The information bottleneck method. In Proc. 37th 

Annual Allerton Conference on Communication, Control and Computing, Illinois, Urbana-Champaign. 

Tononi, G. and Sporns, O. (2003). Measuring information integration. BMC Neuroscience, 4:31. 

Touchette, H. and Lloyd, S. (2000). Information-theoretic limits of control. Phys. Rev. Lett., 84:1156. 

Touchette, H. and Lloyd, S. (2004). Information-theoretic approach to the study of control systems. 

Physica A, 331:140–172. 

Tronstad, C., Gjein, G.E., Grimnes, S., Martinsen, Ø.G., Krogstad, A-L, and Fosse, E. (2008) Electrical 

measurement of sweat activity, Physiological Measurements, 29 pp. 407–415 

Valero, A., Randelli, G., Botta, F., Hernando, M., Losada, D. R., (2011), Operator Performance in 

Exploration Robotics - A Comparison Between Stationary and Mobile Operators, J Intell Robot Syst, 

Springer 

Vallery, H., van Asseldonk, E., Buss, M., and van der Kooij, H. (2009). Reference trajectory generation 

for rehabilitation robots: complementary limb motion estimation, IEEE Transactions on Neural Systems 

and Rehabilitation Engineering, vol. 17, no. 1, pp. 23-30. 

Van Asseldonk, E. H. F., Veneman, J. F., Ekkelenkamp, R., Buurke, J. H., Van der Helm, F. C. T., and 

Van der Kooij, H. (2008). The Effects on Kinematics and Muscle Activity of Walking in a Robotic Gait 

Trainer During Zero-Force Control, IEEE Transactions on Neural Systems and Rehabilitation 

Engineering, vol. 16, no. 4, pp. 360-370, August. 

Van Asseldonk, E., Ekkelenkamp, R., Veneman, J., Van der Helm, F., and Van der Kooij, H. (2007). 

Selective control of a subtask of walking in a robotic gait trainer(LOPES), in IEEE 10th International 


197


Van Brussel, H., Moreas, R., Zaatri, A., Nuttin, M. (1998). A behaviour-based blackboard architecture for 

mobile robots, Industrial Electronics Society, 1998. IECON '98. Proceedings of the 24th Annual 

Conference of the IEEE , vol.4, no., pp.2162-2167 vol.4. 

Van Dijk, S. and Polani, D. (2011a). Grounding subgoals in information transitions. In Proc. IEEE 

Symposium Series in Computational Intelligence 2011 — Symposium on Adaptive Dynamic 

Programming and Reinforcement Learning. Accepted. 

Van Dijk, S. and Polani, D. (2011b). Look-ahead relevant information: Reducing cognitive burden over 

prolonged tasks. In Proc. IEEE Symposium Series in Computational Intelligence 2011 — Symposium on 

Artificial Life. Accepted. 

Van Dijk, S. G., Polani, D., and Nehaniv, C. L. (2010). What do you want to do today? relevantinformation 

bookkeeping in goal-oriented behaviour. In Proc. Artificial Life, Odense, Denmark, pages 

176–183. 

Vanacker, G., Millan, J.d.R., Lew, E., Ferrez, P. W., Moles, F.G., Philips, J., Brussel, H. V., and Nuttin, 

M. (2007). Context-Based Filtering for Assisted Brain-Actuated Wheelchair Driving. Computational 

Intelligence and Neuroscience. 

Varshney, P. K. (1997). Multi-sensor data fusion. Electronics and Communication Engineering Journal, 

9(6): 245-25. 

Velliste, M. et al. (2008). Cortical control of prosthetic arm for self-feeding. Nature, pages 1098–1101, 19. 

Veneman, J., Ekkelenkamp, R., Kruidhof, R., Van der Helm, F., and Van der Kooij, H. (2005). Design of a 

series elastic- and Bowden cable-based actuation system for use as torque-actuator in exoskeleton-type 

training, in 9th International Conference on Rehabilitation Robotics (ICORR 2005), pp. 496-499. 

Veneman, J., Kruidhof, R., Hekman, E., Ekkelenkamp, R., Van Asseldonk, E., and Van der Kooij, H. 

(2007). Design and evaluation of the LOPES exoskeleton robot for interactive gait rehabilitation, IEEE 

Transactions on Neural Systems and Rehabilitation Engineering, vol. 15, no. 3, pp. 379-386. 

Vergassola, M., Villermaux, E., and Shraiman, B. I. (2007). ’infotaxis’ as a strategy for searching without 

gradients. Nature, 445:406–409. 

Vernon, D., Metta, G., and Sandini, G., (2006). A Survey of Artificial Cognitive Systems: Implications for 

the Autonomous Development of Mental Capabilities in Computational Agents, IEEE Transactions on 

Evolutionary Computation, Special Issue on Autonomous Mental Development. 

Versluys, R., Beyl, P., Van Damme, M., Desomer, A., Van Ham, R., and Lefeber, D. (2009). Prosthetic 

Feet: State-of-theart Review and the Importance of Mimicking Human Ankle-Foot Biomechanics, 

Disability and Rehabilitation: Assistive Technology, vol. 4, no. 2, pp. 65-75. 

Viviani, P. and Flash, T. (1995). Minimum-jerk, two-thirds power law, and isochrony: converging 

approaches to movement planning. Journal of experimental psychology. Human perception and 

performance, 21(1):32–53. 

Waddell, G. (1998). The back pain revolution. Edinburgh: Churchill Livingstone. 

Wahlster, W., Reithinger, N., and Blocher, A. (2001). SmartKom: Multimodal Communication with a 

Life-Like Character, Proceedings of Eurospeech, Aalborg, Denmark. 

Walsh, C. J., Endo, K., and Herr, H. (2007). A Quasi-passive Legacy Exoskeleton for load-carrying 

augmentation, Inter- national Journal of Humanoid Robotics, vol. 4, no. 3, pp. 487-506. 

Walter, W.G., Cooper, R., Aldridge, V.J., McCallum, W.C. and Winter, A.L. (1964). Contingent negative 

variation: an electric sign of sensorimotor association and expectancy in the human brain. Nature, 

203:380–4. 

Wang, Y., Tan, T., Jain, A.K. (2003). Combining Face and Iris Biometrics for Identity Verification, 

Proceedings of 4th International Conference on Audio- and Video-Based Biometric Person Authentication, 

Guildford, UK. 

Weinberg, B., Nikitczuk, J., Patel, S., Patritti, B., Mavroidis, C., Bonato, P., and C. P. (2007). Design, 

Control and Human Testing of an Active Knee Rehabilitation Orthotic Device, in Proceedings of 2007 

IEEE International Conference on Robotics and Automation, pp. 4126-4133. 

198


WHO MONICA Project [http://www.ktl.fi/monica/] (accessed on12.07.07). 

Wolff, G.J., Prasad, K.V., Stork, D.G., and Hennecke, M. (1994). Lipreading by neural networks: visual 

processing, learning and sensory integration, Proc. of Neural Information Proc. Sys. NIPS-6, Cowan, J., 

Tesauro, G., and Alspector, J., eds., pp. 1027-1034. 

Wolpaw, J.R., Birbaumer, N., McFarland, D.J., Pfurtscheller, G., and Vaughan, T.M. (2002). 

Braincomputer interfaces for communication and control. Clinical Neurophysiology, 113(6):767–91. 

Yamauchi, B. M. (2004). PackBot: A Versatile Platform for Military Robotics, Proceedings of 2004 SPIE, 

Orlando, USA, pp. 228-237 

Yang, J. F. and Gorassini, M. (2006). Spinal and Brain Control of Human Walking: Implications for 

Retraining of Walking, Neuroscientist, vol. 12, no. 5, pp. 379-389. 

Yoon, J. and Ryu, J. (2006). A Novel Reconfigurable Ankle/Foot Rehabilitation Robot, in Proceedings of 

the 2005 IEEE international Conference on Robotics and Automation, pp. 2301-2306. 

Zachry, T., Wulf, G., Mercer, J. and Bezodis, N. (2005). Increased movement accuracy and reduced emg 

activity as the result of adopting an external focus of attention. Brain Research Bulletin, 67:304–309. 

Zengar Institute Inc., n.d. NeurOptimal® Neurofeedback System. [online] Available 


Zhai, J. and Barreto, A. (2006). Stress detection in computer users based on digital signal processing of 

noninvasive physiological variables. In Engineering in Medicine and Biology Society, 28th Annual 

International Conference of the IEEE, pages 1355–1358. 

Zhang, H-C., Zhu, M-L. (2004). Self-organized architecture for outdoor mobile robot navigation. Journal 

of Zhejiang University Science 

Zimmerli, L., Duschau-Wicke, A., Mayr, A., Riener, R., and Lunenburger, L. (2009). Virtual reality and 

gait rehabilitation augmented feedback for the lokomat, in Proceedings of the 2009 IEEE Virtual 

Rehabilitation International Conference, pp. 150-153. 

199

D2.1 Requirements and Specification - CORBYS

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?