We have conducted usability evaluations for the 4diac IDE using various common usability evaluation methods. During this evaluation we found numerous issues that may very well affect other tools as well.
Extensive usability evaluations take a lot of time and effort though, so we have looked into ways to reduce the overhead for setting up and conducting user tests as the most common and usually most effective method.
By extending the existing RCP Test Tool plug-in we were able to create a plug-in that allows defining and executing experiments with real users in real environments while minimizing the effort for the evaluator.
We will demonstrate how to perform such instrumented evaluations and outline our results, both in terms of specific usability findings and how the approach performs in contrast to traditional methods.