Usability Testing: International Children’s Digital Library

I was interested in testing a site designed specifically for children. The International Children’s Digital Library is an online library that lets children or parents/teachers browse for children’s books in many languages and read them online.

I tested with two child testers and one adult/parent tester. My script follows. I found that as I went through it I had to simplify it even more than I already had because the children were confused by some of the language (for example, “what strikes you” or “your impressions”). Parentheses in the script indicate changes or explanations I made when the children appeared confused.

Usability test script

(adapted from Krug, 2006)

Hello! Thank you for participating in this library site test!

Today I will be asking you to try using a website so we can see whether it works as intended (whether it works well, how easy it is to use). We will be looking at the website for the International Children’s Digital Library. I will ask for your impressions (ask what you think) of the home page of the website, then I will ask you to do a couple of things: I will ask you to browse for a book you might want to read, and then I will ask you to find out how to register for an account on the website (you won’t actually register, I will just be asking you to figure how you could if you wanted to).

The first thing I want to make clear is that we’re testing the site, not you. You can’t do anything wrong here. In fact, this is probably the one place today where you don’t have to worry about making mistakes.

As you use the site, I’m going to ask you as much as possible to try to think out loud: to say what you’re looking at, what you’re trying to do (what you want to click), and what you’re thinking about the site. This will be a big help to me. Also, please don’t worry that you’re going to hurt my feelings. We’re doing this to improve the site, so we need to hear your honest reactions.

If you have any questions as we go along, just ask them. I may not be able to answer them right away, since we’re interested in how people do when they don’t have someone sitting next to them to help. But if you still have any questions when we’re done I’ll try to answer them then. And if you need to take a break at any point, just let me know.

Do you have any questions so far?

1. Before we look at the site, I’d like to ask you just a few quick questions.

• How would you describe your computer and Internet knowledge—how familiar are you with using the computer and web browsing? (NOTE: I had to explain to the children that “web browsing” was “clicking around and looking around at stuff on web sites.”)
• What kinds of sites are you looking at when you browse the Web?
• Do you have any favorite Web sites?OK, great. We’re done with the questions, and we can start looking at things.

Test Script

Scenario #1: ICDL Home Page

1. (starting from Google) Let’s go to the ICDL home page, www.childrenslibrary.org
2. First, I’m going to ask you to look at this page and tell me what you think about it: what strikes you about it, whose site you think it is, what you can do here, and what it’s for. Just look around and tell me what you think.

You can scroll if you want to, but please don’t click on anything yet.

• What is this site for? What can you do here?
• Does the site look fun? Do you think you would like to use it?
• What do you think about how the site looks? (Additional prompts if needed: Do you like the colors? What do you think about the number of links?)
  

  Scenario #2: Finding a Book to Read

Thanks. Now I’m going to ask you to try doing a couple of specific tasks. And again, as much as possible, it will help us if you can try to think out loud as you go along.

1. What I’d like you to do now is look for a book that you might want to read (for adult tester: a book you think your child would want to read). How do you think you might do that? OK, let’s try that. [Wait a few minutes, help where needed]
2. [After book record is chosen] Now I’d like you to try and open this book so you can read it. [Wait]
3. [After books is successfully opened, or if 10 minutes pass without success] Great! Thanks. [If applicable, say: ] We can come back and read this book later, I promise.
4. What did you think of this site? Was it easy to find a book to read?

Thanks, that was very helpful.

Scenario #3: Figuring Out How to Register

Thanks. OK, now I am going to ask you to figure how you can register as a library member at this site. You won’t actually be registering, we just want to figure out how to do that.

1. We’ll start this task from the library’s home page. Do you know how to get back there? [Help if needed]
2. How do you think you would register if you wanted to be a member of this site? OK, let’s try that.
3. [After appropriate registration form (child or adult) is successfully opened, or if 10 minutes pass without success] Thanks! [If applicable say:] Let’s look at this registration form. Do you think it looks easy or hard to fill out? Do you have any thoughts about it?
4. How would you go back to the library’s home page?

OK, that was the last task!

Do you have any questions for me, now that we’re done?

Thank you so much for participating in the site test! I really appreciate it!

Tester Profiles

Tester 1

Nine year old tester who uses the computer 1 or 2 hours a day. Primary occupation: fourth grade. Tester 1 said “I don’t do that [i.e. computer use and web browsing] a lot. Tester 1 said http://www.coolmath-games.com was his favorite website.

Tester 2

Eleven year old tester who uses the computer 1 or 2 hours a day. Primary occupation: sixth grade. Tester 2 said he was “pretty unfamiliar” with computer use and web browsing. Tester 2 said he did not have any favorite websites.

Tester 3

Adult tester who uses the computer 6-8 hours per day (uses computer for work). Primary occupation: telecom engineer/administrator. Also a parent. Tester 3 said he was highly familiar with using the computer and web browsing. Tester 3 said he had several favorite networking/technical sites, and also really liked ESPN, and IMDB.

Test Results

One of the things I concluded from this test is that it is probably really hard to design a website that will be used by both kids and adults. My child testers found the site pretty easy to use for the most part and completed tasks quickly. They were comfortable just clicking around and trying things.

My one adult tester, on the other hand, got very frustrated with the site functionality. This was contrary to my expectations. I chose a children’s library site to test because I was curious about design for children, since it seemed like children, who have less experience and fewer expectations of site conventions, might be harder to design an intuitive site for.

This site’s primary audience is children. It makes sense that they would design a site that kids will enjoy and find easy and fun to use. Based on two kid tests, it seems like they did that, but at the price of making it a somewhat annoying site for adults to use. The child testers were more willing to browse, whereas the adult tester was very focused on finding the most efficient path quickly. In this case, having experience and expectations about how sites behave worked against him. He said several times “Well this isn’t what I expected to happen.” The children never said anything like this.

For this site, it seems like many conventions were not followed, and this was not a problem for the child testers, since they had few expectations. The adult tester, however, struggled with the unusual choices.

Obviously I didn’t test enough users to make generalizations about how children vs. adults use the web, but the results did make me wonder if children in general are more willing to explore, less impatient, and more likely to be “link dominant” versus “search dominant.” My adult tester clearly wanted an efficient search strategy and found the site defective because there wasn’t one. Both child testers seemed more willing to just look around at what was there and browse rather than trying to immediately identify the most efficient search strategy.

Task Completion

All of the users completed all of the tasks, which I tried to make pretty simple since I was testing children. The overall completion rate was 100 percent. However, my adult user went down many wrong paths and became frustrated. A less persistent user might well have quit and gone to another site, especially if he really was a parent looking for a book for his child. Lots of children’s ebooks are pretty cheap on Amazon and are super easy to search and find!

Suggestions

Home Page

I’m wary of making suggestions since this seems like a situation where it’s going to be “fix one thing, break another.” The site worked pretty well for the children, and was pretty irritating to the adult. There are some changes I would make, however.

ICDL Home Page

I would add a Search box on the home page. The adult tester commented unfavorably on the absence of a search box at the top of the home page, where he expected to see it. The large and centrally located “Read Books” icon is intended to take its place, making the page cater to link dominant users and ignore search dominant users. This didn’t seem to be a problem for the kids but the adult tester really wanted a search box. Adding in a search box would not overburden the home page, especially if my second suggestion were followed, which is to drastically reduce the number of choices on the home page.

There is just way too much on the home page. Tester 2 and Tester 3 both commented on how busy the home page was. Tester 2 commented that it was hard to choose because there “so many icons.”

Many of the options are redundant. For example, you can reach a search page by clicking the large Read Books icon in the center of the page, from the top menu under “Read Books” and also under “Help,” and on the left nav menu from no less than three of the links there! Certainly some redundancy can be desirable, to accommodate different ways people prefer to look for things, but this seems beyond what is necessary. A great deal of cleaning up could be done by eliminating redundancies.

I’m wondering about indicating some sort of prominent choice between “Search or Click Here to Browse Books” but that’s where I feel I might fix it for the adults and break it for the kids. This site is for children and “browsing” is not a term that registered with the child testers. The term “Read Books” obviously worked well for them, and I would hesitate to change that.

All testers completely ignored the large amount of “library news and notes,” mission statement, information about the “Foundation” and so on. Clearly, the home page is being used partly to market the library and solicit support as well as to be a functional library page–the result is way too much going on. A lot of “happy talk” promoting the library could be moved to a secondary page that is reached from a basic link on the home page such as “Support Us!” or “Learn About Our Mission!” or something like that. Designers might be concerned that people won’t donate or support if they don’t see it on the home page, but the testing showed that site visitors didn’t see it anyway–there is just too much stuff there.

My guess is almost all site visitors will arrive at the home page wanting to browse/search for books and read them, or to sign up or sign in to save books to their shelf. Given those likely priorities a much simpler home page interface is called for.

Simple Search

Simple Search Page

I found the Simple search page very interesting since it introduces access points that don’t make a whole lot of sense to adults, but really are ways that children think about books. One of the more interesting design choices was the ability to search by color. I’ve worked in K-6 and 6-8 school libraries, and kids ask all the time for “the blue book with the owl on the cover” or something like that. This site actually lets you search by the color of the cover and then shows an image of the book cover, so you really could search for a “blue book with an owl on it.” Tester 1 really approved of this feature because (the adage notwithstanding) “a cover tells you a lot about a book.” On the other hand, Tester 3, an adult and non-librarian, thought searching by the the color of a book was ridiculous.

Tester 1 clicked the main “Read Books” icon on the main page and found the SImple Search page immediately and liked it. Tester 2 chose various searches from the top menu on the home page, first choosing Recently Added Books, going back and choosing Award Winning Books, clicking Back again and choosing Full Book list (all from the top menu). Then he started scrolling through the long list of choices and selected a book. Although these strategies were different, they were both accomplished very quickly, and with no sign of frustration.

Tester 3 on the other hand, spent a long time looking for “Search” on the home page. First he tried the top menu Read Books > Keyword Search option and came up with too many book choices. Then he tried Read Books Advanced Search and was baffled by the Advanced Search page.

Advanced Search

Advanced Search Page

The advanced search page lets you enter keywords and then…do what? Tester 3 didn’t want to click on the links because he expected them to take him off of this page. Actually when clicked the links open up and let you check boxes to filter for various criteria, but this is not apparent before you click them. Tester 3 was expecting a typical text entry advanced search where you enter something like subject: art, language: Italian and so on. This page was baffling and he retreated back to try the Simple Search page. He eventually found a book using the Simple Search page but it took him much longer and he experienced far more frustration than the child testers. I would recommend altering the page labeled Advanced Search to be a more conventional text entry type search, or at least call the page something else so that experienced computer users don’t get tripped up by their expectations of what an “Advanced Search” looks like!

Conclusion

Overall, this website works reasonably well for is target demographic (children) but could be improved by making it less busy. After reducing the visual clutter, a few accommodations for adults/experienced web users would be desirable. Search dominant users are disadvantaged on the home page and this should be fixed.

Testing Process

This was a really interesting process to go through. I really expected the tasks to be a breeze for my adult tester and more challenging for the child testers, but the opposite was true. On the other hand, the adult tester “thought out loud” with ease throughout the test and the children required a more concrete explanation of what that meant, and even after that I needed to prompt them as we went through the test.

Testing children requires even more simple and clear instructions. If I were testing children again I would make the language even simpler. It’s challenging because you don’t want to ask too many leading questions, but they seem to need more prompting. Abstract skills like “thinking out loud” and “giving your impressions” confused them a little. Now that I have tested children, I would spend more time simplifying the testing script and making statements very concrete (such as “tell me what you think you might click on”) and avoid the abstract statements “what strikes you?”) as much as possible.

After the fact I read a little bit about usability testing for children, and one reference mentioned that it is desirable to avoid the word “test” completely (Carraro, 2011). Adults don’t get tested a lot but school-aged children do. My child testers initially had a hard time accepting the fact that weren’t being tested even though I stated it at the beginning. I restated this a few times throughout the test.

The children seemed shy and quiet at the beginning of the test, probably nervous at this very unfamiliar activity. It is even more important to establish rapport with children since they are more likely to be shy with strangers or with unfamiliar conditions/activities (Hanna, Risden & Alexander, 1997). Even though these children know me well (since I am their mother!) it took them a long time to warm up. I would focus more time in the future on talking to the children on neutral kid-friendly topics to establish rapport (especially with children I did not know) and in chatting casually about what the test was for. By the end of the test both child testers warmed up and did a lot more “thinking out loud.” Hanna, Risen and Alexander mention the desirability of switching the order of tasks around so that time placement in the test does not affect the way children respond (Hanna, Risden & Alexander, 1997, p. 11). Their concern was children slowing down, balking, or beginning to act very silly at the end when they are tired (which might be an issue with younger children than I tested, or with longer testing sessions). By contrast, in my testing I found that at the end the children were most comfortable and volunteered the most extensive information, so it would be desirable to have different tasks placed at the end.

With regards to my adult tester, I found that not only are conventions your friend (Krug, 2006, p. 34), but altering them can cause significant annoyance to users who are familiar with them. This presents a problem for this kind of site, where you are trying to make something that appeals primarily to children but that adults can also use. It’s a significant challenge to make a site that works for two very different user groups.

REFERENCES

Carraro, J.M. (2011, July 21). Five things you should consider when testing with children [Web blog]. Retrieved from http://blog.usabilla.com/five-things-you-should-consider-when-testing-with-children/

Hanna, L., Risen, K. & Alexander, K. (1997). Guidelines for usability testing with children. Interactions, September/October 1997. Retrieved from http://www.microsoft.com/usability/UEPostings/p9-hanna.pdf

Krug, S. (2006). Don’t Make Me Think: A Common Sense Approach to Web Usability. Berkeley, CA: New Riders Publishing.