At the University of Illinois, we have a fairly large group that is working on improving Integrated Development Environments (IDEs), especially for refactoring. Recently, our group has developed Photran, an IDE for Fortran, and several refactoring tools for parallelism like Immutator, ReLooper, and DPJizer.
Naturally, we want to know whether we are making the IDE better. In general, there are two ways to study an IDE, doing lab or field studies. In a lab study, the researchers invite a few participants to perform a set of tasks using the IDEs. If there are not too may participants and the tasks are not too long, it might be possible to screencast the interactions of the participants with the IDE and later watch the screencasts to analyze the interactions. Typically, in a field study, researchers study a number of programmers working on their daily tasks over a long period of time. It is both tedious and error-prone to analyze the large volume of data collected in such long-term field studies. Therefore, field studies on IDEs require tools that collect the interactions of programmers with the IDE in machine-readable formats that are amenable to automated analyses.
We have developed CodingSpectator as a platform for conducting user studies based on Eclipse. It automatically uploads the collected data to a central repository and keeps the history of data submitted by each participant. CodingSpectator uploads the collected data efficiently by just submitting the new data since the last upload. In addition, it reminds the participants about the availability of its new releases. Moreover, CodingSpectator is extensible because it does not impose a particular format on the collected data and can upload any data that third-party data collectors put in a designated folder called the watched folder.
We designed CodingSpectator to automatically prompt the participants at the startup of Eclipse at most once a day to upload the collected data. A participant can also manually initiate a data upload at any other time. CodingSpectator creates the watched folder in every workspace and uploads the contents of this folder to a central repository regularly. It stores the contents of the watched folders of all participants in a single Subversion repository. Subversion keeps the histories of the submitted data automatically. This history lets researchers both examine the evolution of the collected data and revert to an older version of the data in case of an accidental data loss.
Incremental changes to the data collector during a long-term study are inevitable. We have been constantly evolving CodingSepectator during our study. Whenever we release a new version of CodingSpectator, we'd like our participants to upgrade to benefit from the latest improvements. Therefore, CodingSpectator automatically notifies our participants about the availability of a new version.
CodingSpectator adds a small icon to the status line of Eclipse to indicate its presence. Some of our participants do not want CodingSpectator to collect data from some of their workspaces. We recommend these participants that they do not open their private workspaces in an instance of Eclipse with CodingSpectator. CodingSpectator adds a visual indicator to Eclipse so that our participants can easily tell whether CodingSpectator is installed on an instance of Eclipse.
In summary, CodingSpectator is more than just a data collector. It provides a reusable infrastructure for conducting user studies on Eclipse. CodingTracker is an example of a research project that has reused the infrastructure of CodingSpectator for conducting user studies. If you need help in using CodingSpectator for your user studies, please contact us.