1.4 Tracking Metrics

The progress of this project will be tracked with a set of metrics. These metrics will be quantifiable variables that reflect the level of activity in our project. Several of the metrics we use will be accumulated automatically as a component of the software. Others will require input from evaluations provided by site staff and project personnel. The metrics will use will track:

1. The number of individuals who use "Windows to the Universe" at each site, relative to the total number of individuals entering the site. This will provide us information on the initial attractiveness of the display and front page of the system. We do not have a good estimate of the percentage that indicates success for this metric - indeed the goal is not that we have the largest number of fleeting users but instead to provide a valuable learning experience to users. Instead of using this metric as a quantifier of success, we intend to use this information to suggest if we need to improve the display and front page of Windows to the Universe.

2. The frequency of reuse of the system by individuals who repeatedly visit deployment sites. This metric, as well as the following two, will provide information about the interest individuals have in the system. Again, it is difficult to anticipate a value for this metric which implies success until we have begun initial deployment of the system. However, it seems a reasonable goal that 25% of the individuals who have initially used the system and revisit the site will be interested enough to reuse the system again. We intend to re-evaluate the reasonableness of this success metric during evaluation of the first deployed version during the second year of the project.

3. The duration of individual sessions. If an individual is interested in the system, we anticipate that their sessions will be of longer duration than those who are turned-off by it. Again, it is difficult to anticipate a value for this metric which implies success until we have begun initial deployment of the system. However, it seems a reasonable goal that at least 25% of the individuals who use the system will engage in sessions of 5 minutes duration or longer. We intend to re-evaluate the reasonableness of this success metric during evaluation of the first deployed version during the second year of the project.

4. The depth of penetration into the system by individuals. If an individual is interestedc in the system, we anticipate that they will delve deeper into the stack of information provided over the course of their use of the system. This includes tracking use of the system over multiple interactions, assuming reuse by a percentage of users. It is difficult to anticipate a value for this metric which implies success, particularly in view of the different behaviours to be expected from children and adults. However, it seems reasonable to hope that 10% of initial users will be interested enough to browse the full stack of information and that 10% of individuals that reuse the system at a given site will delve to a deeper level on successive interactions. Again, we intend to re-evaluate the reasonableness of this success metric during evaluation of the first deployed version during the second year of the project. In addition to these metrics, we will also evaluate the use of the system by different groups, including children, adults, females, males, and minorities. This information will be used to determine if the system is less attractive to some groups than to others, allowing us to make revisions that will improve the system to be as inclusive and interesting as possible to a wide-range of the general public.