707.031: Evaluation Methodology W15/16

This specialized course concentrates on evaluation methodologies for research and innovation on interactive systems. The course applies a participative methodology based on project prototypes. A number of novel projects have been selected to address individual aspects of the empirical methodology. Projects chosen for WS2015/16 include: Glove for gesture and haptic feedback, augmented reality visualizations, recommendation of visualizations, interactive topic analysis.

The student taking this course will learn through experience to:

As a student, you will acquire skills to formulate, design and validate technology by conducting and participating in a critical evaluation process. The skillset target of this lecture will let you re-think and plan research and also product development processes in terms of evaluations and their results.


Inffeldgasse 13, 6th Seminar Room KTI


to the course outline @ TU Graz

Class Times

This year (WS 2015 -2016), Evaluation Methodology is a block lecture. Every day from 09.11.2015 to 20.11.2015 at 14:00 - 16:30

Timetable [TENTATIVE]

DAY 1: Introduction 09.11.2015
    Topics :
  • General Information,
  • Why we evaluate: evaluation in product design process, evaluation in research process
  • How to measure: the model human processor,
  • Project introduction, project choices, sketching
Due: project alternatives, project choices
DAY 2: Visual Perception 10.11.2015
    Topics :
  • Visual perception, visual encoding, colors
  • Visual depth perception
  • Eye tracking
  • Project AR Vis
DAY 3: Visual Cognition 12.11.2015
    Topics :
  • Cognitive system model,
  • Cognitive load ,
  • Situation awareness,
DAY 4: Evaluation Formulation 16.11.2014
    Topics :
  • Research questions and hypotheses
  • Sampling methods: between/within,
  • Sampling methods: qualitative, quantitative, performance measures,
DAY 5: Crowd Studies 13.11.2015
    Topics :
  • Crowdsourced experiments
  • Structure of massively
  • Designing controllability in online studies
  • Project: VizRec,
DAY 6: Tactile Acquisition and Processing 16.11.2015
    Topics :
  • Haptics,
  • Sensory feedback,
  • Project: Glove transhaptics,
DAY 7: Descriptive Statistics 17.11.2014
    Topics :
  • Descriptive methods, experimental model,
  • Assumptions, Normality, Homogeneity of Variance,
  • Correlations,
DAY 8: Two-Sample Experimental Designs: Analyses and Reporting 18.11.2014
    Topics :
  • Between groups and repeated measures designs,
  • T-Test, Mann-Whitney U,
DAY 9: Multi-Sample Experimental Designs: Analyses and Reporting 19.11.2014
    Topics :
  • Between groups and repeated measures designs,
  • One way ANOVA, repeated measures ANOVA,
  • Complex designs: two (or more) independent variables
  • Complex designs: mixed model between and within groups factors
DAY 10: Evaluation / Putting it all together 20.11.2014
    Topics :
  • Wrap-up,
  • Choosing evaluation methods,
  • Reporting methods,
  • Assignment reports,
  • Lecture evaluation,


Eduardo Veas, PhD


Eduardo Veas is Interim Profesor in Computer Science at the Faculty of Engineering, National University of Cuyo, Argentina. He was formerly deputy division manager of the Knowledge Visualization group at the Know-Center. His interests lie in the field of human computer interaction, in particular in virtual and augmented reality, visualization and human (visual) perception. Eduardo has accumulated experience in user centric design methodology and in designing (non-conventional) interfaces, from prototyping and design of specialized devices, to the implementation and evaluation of novel interfaces for visualization of sensor data. He obtained his PhD in Computer Science in 2012 from Graz University of Technology, his Msc. in Information Science and Technology from Osaka University.


AR Signs: AR displays and perception of ISO signs


In this project, you will lead studies using a head mounted display to view overlays of ISO signs. Augmented Reality (AR) aims to extend a person’s view of the world with computer generated information in real-time, has targetted industrial processes since its introduction as a field. Advances in miniaturization and sensor technologies sparked a range of novel applications where AR promises an ideal solution in industrial processes. ISO sings are ubiquitous in industrial settings. They are used to alert dangerous areas, locate equipment, indicate escape routes and others. This project analyses perceptual effecfs of overlaying ISO sings on the real world. This project evaluates how ISO signs are perceived through AR. In particular, we are interested to find out if different industrial backgrounds pose interference that confuses signs. We also want to establish which sign categories are better perceived in AR.
Lead: Carla Barreiros and Eduardo Veas


Carla is researcher at Know-Center and PhD Candidate at KTI. Carla's research interests span a broad spectrum of wearable and mobile technology to present information in ways that affect people. A key challenge in her work is to reach people emotionally with digital interfaces. Along these lines she is investigating methods to represent information with natural interfaces.
Sensory substitution/augmentation


In this project you will study whether/how people learn to decode information from a tactile display prototype. Much of the digital information we deal with is presented visually and we have acquired a broad understanding and basic methodology to design for visual interfaces. However, at times, people are already engaged in an activity that requires visual concentration or visual stimulation cannot be used for other reasons. This project systematically studies how to represent information with a tactile display. You'll specifically help us find out characteristics of a vocabulary and messages that can be conveyed with the tactile display.
Lead: Granit Luzhnica, MSc.


Granit is researcher at Know-Center and PhD Candidate at KTI. His research interests dwell on the topic of sensors and actuators for mobile multimodal interaction. Granit is involved in designing and testing a number of mobile prototypes involving sensors at Know-Center. A key challenge in his work is the elicitation of behaviour patterns. With each prototype, it becomes necessary to determine patterns that define a behaviour from raw sensory data. His expertise in machine learning and classification algorithms is key to investigate models (activity, behaviour) from sensory data.
Interactive Topic Modelling


This projet investigates how we organize knowledge from large text collections. The ability to analyze and organize large collections, to draw relations between pieces of evidence and build knowledge are all part of an information discovery process. Continuous shifts in goals and information needs guide the acquisition of knowledge – cognitive constructs of topics related by exemplary resources–. This project investigates the search process and knowledge building process with a tool of our design. We are interested in finding out if people put trust on a tool recommending content and how knowledge is constructed interactively using this tool.
Lead: Cecilia di Sciascio, Lic.


Cecilia is researcher at Know Center and PhD candidate at KTI. Her research interests concentrate on search and recommendation interfaces for large collections of data. She has developed a self contained system for search and exploration of scientific content, which is currently integrated into the EEXCESS platform. A key challenge in her work has been the nature of unstructured data and the definition of methods to represent features that can be used to establish relationships therewith. Cecilia's work complements visual analytics and recommender systems techniques to deliver the right information in a clear, understandable way.
Recommending Visualizations


This project investigates the effects of recommending visualizations. Coming up with a good visual representation is a challenge for most people. It takes a number of iterations until the right configuration of a visualization is found. This project uses a multi-stage recommender system, which prepares recommendations appropriate for a dataset and matching the interests of the user as recorded in her profile. Participating in this project, you will investigate methods for users to express their preferences for visualizations and datasets. Additionally, you will work on studies that help us understand the aspects of interaction that people apply the most as well as those which are problematic.
Lead: Belgin Mutlu, MSc.


Belgin is researcher at Know Center and PhD candidate at KTI. Her research is dedicated to investigate methods to recommend visualizations. She has studied a number of algorithms to recommend visualizations: rule based, collaborative filtering, content based and hybrind. The key challenge in her research is to find a way to simply derive and establish the features that make a visualization useful for a particular person. Her research is defining new ways to interact with data, whereby visualizations are recommended for the data at hand using algorithms that account for user preferences and past experience.


30% being present and active in the lecture 40% evaluator credits 30% participant credits

The course is project based. For each class students take turns to moderate and take part in evaluations. For the course, the student will:

  • choose a project topic to design and evaluate from concept to prototype
  • choose a project topic to moderate (play the role of user for peer projects)


Continuous evaluation serves as a research methodology in many fields of Computer Science. This lecture is intended to familiarize you with a range of methods applied when continuous evaluation is used in research, desing and innovation. The following is a list of the main textbooks that will be referenced throughout the lecture. You can borrow them from the university library, or come to Eduardo's office to browse them (make an appointment).

User Centric Design and Human Factors

Foundations for Designing User Centred Systems. Ritter, Baxter, Churchill, is a thorough reference for the UCD methodology. It dedicates a good portion of the book to understanding people, human factors, behavior and cognition. The chapter on evaluation provides a good, general entry point to the different aspects of evaluations. In the lecture, this book serves as reference for UCD but also for considerations about the user capabilities. Link

Formal Evaluation: Statistics for Analysis

Discovering Statistics Using R (Andy Field, Jeremy Miles, Zoe Field) is our main reference to the implementation of statistics methods used for analyses of evaluation results. It is a thorough reference with example implementations in R, covering the major aspects of research and analysis of evaluation results introduced in this lecture. It is a recommended read, which will be heavily used in weeks 3-6, and will surely accompany the students beyond the lecture in their carreer.

Quantifying the User Experience: Practical Statistics for User Research (Jeff Sauro and James R Lewis)

Human-Computer Interaction:An Empirical Research Perspective (I. Scott MacKenzie)

Research Methodology

Doing Psychology Experiments. David Martin

How to Design and Report Experiments. Andy Field, Graham Hole

Research Methods in Human-Computer Interaction (Lazar, Jinjuan, Hochheiser)

Human Computer Interaction (Dix, A., Finlay, J., Abowd, G., and Beale, R. )

Information Visualization 3rd Ed.: Perception for Design (Colin Ware)

Desing and Critique

The Design of Everyday Things (Donald Norman)

Emotional Design: Why We Love (or Hate) Everyday Things (Donald Norman)


Sketching User Experiences: Getting the Design Right and the Right Design (Bill Buxton)

Sketching User Experiences: The Workbook (Saul Greenberg, Sheelagh Carpendale, Nicolai Marquardt, Bill Buxton)

What this course is NOT

This course is not an introduction to HCI nor to visualization. For general courses on the topics covered visit:

This course will not deal with web usability, as that topic is extensively covered by Keith Andrews course on Information Architecture and Web Usability. Instead the course concentrates on providing a solid background on the different evaluation methods as applied in a UCD continuous evaluation process. This course is no replacement for an advanced statistics lecture. Descriptive and inferential statistics methods will be used to guide the preparation of data, the performance of appropriate statistical tests, their interpretation and reporting of results.

Who should attend this course

This course is particularly suited for post-graduate students (Msc.) and researchers (PhD.) investigating or developing novel metaphors to interact with computers and machines. It is also suited for researchers trying to understand the user behaviour and how it is influenced by technology. This includes psychology majors as well as educators. Students in the final year of their Bak. may benefit from this course too.