Lightning usability testing with OneSearch

From the Spring 2015 Newsletter

Is a single search box a usable interface? Can students use the library website to find what they need? These were the questions that prompted our springtime usability study. 

Usability testing is a best practice among website administrators. A moderator will sit with a participant and ask him/her to complete a number of tasks. Based on the success rate, the website design may be changed to be more usable. We used a “lightning” usability test approach to compare students’ intuitions of different library search methods.

What’s OneSearch? It’s a single-search-box portal to books, ebooks, videos, and a large number of articles and digital materials available through the library. In library-speak, OneSearch is a web-scale discovery service.

Article contents

  1. Background
  2. Methodology
  3. Findings
  4. Appendix 1: students' search strings
  5. Appendix 2: lightning usability script

Background

The CUNY Office of Library Services, the central group that supports all CUNY libraries, rolled out OneSearch in Fall 2014. Since then, we’ve included the search box in the fourth tab on the Library’s homepage, with the explanation that it is a “beta” service. While we at John Jay have tended not to incorporate OneSearch as a primary part of our library instruction, we have been averaging between 400–600 searches in OneSearch each day. In our roles as librarians, we have observed students using it—and liking it—more and more. But how were they using it? Were they really finding what they wanted? What are the pitfalls we’d have to address when teaching OneSearch?

We asked John Jay students to show us how they use the library website, with results both expected and surprising. This usability study was conducted by Robin Davis, Prof. Janice Dunham, Prof. Karen Okamoto, and Allie Verbovetskaya (OLS) on February 4, 5, and 9. Students who participated were rewarded with MBJ $10 cafeteria vouchers, generously provided by the Faculty-Student Engagement Fund (coordinated by Christie Graziano at the Office of Student Transition Programs). 

Methodology

We run usability studies on the Library website at least once a year, and lately we’ve begun to favor “lightning” or “quick and dirty” usability tests. Conventional usability studies have 5–7 participants with sessions lasting 30–90 minutes. When we did such tests before, we felt that we did not receive significant data from such a small number of participants. Moreover, we advertised through library-related channels and on printed-out signs in the library, so our users were self-selecting as heavy library users. We needed a selection of users who would better represent the college’s student body. Lastly, as Prof. Dunham pointed out, the conventional usability study is not a typical use case. Students are often searching for library materials quickly in between other research tasks; they usually do not feel under pressure, enclosed in a small room with a librarian scribbling notes while they are audio-recorded. We wanted to create conditions that more closely mimicked a real use case. 

We set up a table with laptops in the busy Kroll Atrium during Community Hour. Our signs offered the free $10 cafeteria coupons in exchange for five minutes of student feedback on “a website.” Students lined up, often chatting with their friends. Once a student sat down with us, we told them that we were testing the library website and they’d be given 3 tasks to do on the laptop:

  • Find out if a book, The Polar World, is available in the Library
  • Define peer review, either from memory or from referring to any resource
  • Find a peer-reviewed article about different ways college students deal with stress

As they completed the tasks, we took notes in pen on our script worksheets. The most important thing we did was to write down their path (e.g., homepage > catalog > title search “the polar world” > scrolled down page > gave up) as well as their search strings (e.g., college students AND stress). We rated their performance of each task for our own notes.

Because we were only selecting for the kind of student who wants a free lunch (but who wouldn’t?), we netted a range of users: all year levels and the full range of library experience. Some knew exactly which databases they wanted to use, and some had never used the library website before. We had 39 participants in total. 

Findings

Students tend to use natural language search strings in OneSearch and find less relevant articles. Students tend to use keyword search strings in other library databases and get more relevant results.

The research question students were given read, “You need to find a peer-reviewed article about different ways college students deal with stress. What’s the first relevant article you see that you’d want to read? Look anywhere you’d usually look.” Librarians noted if the chosen article was “relevant” if it was peer-reviewed and was about at least one way that college students manage stress. (Note that we were looking at intuition and ease of use here, rather than testing our students’ research savviness. They were only searching for a couple of minutes on a fake assignment.) Fewer students who used OneSearch found “relevant” articles compared to those who used Academic Search Complete, JSTOR, and other library databases.

These findings do not mean that OneSearch does not give relevant results! Finding articles in OneSearch is easy since the index is enormous, but finding articles relevant to a given research question requires a certain level of searching skills. With a good search string like “college students AND stress management,” OneSearch delivers very good search results. But a bad search string will deliver confusing results, like when students typed in “different ways college students deal with stress” (a natural-language phrase) or “peer review student stress” (misunderstanding how to use keywords). Most students who chose to use OneSearch used these poorly-constructed search strings, whereas students who used other library databases broke the question into keywords and used two search boxes connected with AND. See the bottom of this article for the actual search strings that students used.

It is probable that because we used OneSearch’s one-big-box look on our default page, students took the cue that that they could use the search box like Google, which can handle those kinds of natural-language searches. But at heart, even though its index is web-scale, OneSearch is still a library database that responds best to keywords and Boolean operators. Because of this usability study, we are considering defaulting to OneSearch’s advanced search interface.

OneSearch is much easier to use than CUNY+ when performing title searches.

All of the students who searched for The Polar World (1964) in OneSearch found it. None of the students who searched for The Polar World in CUNY+ found it. CUNY+ is limited in two major ways: first, search results cannot be sorted by relevance, and are sorted by date by default. Second, “title begins with” searches require a/an/the to be stripped from beginning of the title, a fact no students in this study realized, even with the hint highlighted below the CUNY+ search box. Based on this finding, we are considering using OneSearch as the primary access point for records in the CUNY+ catalog. OLS has provided OneSearch widgets that can be limited to material type, including print books & ebooks.

Students like the tabs design.

The default tab shown on our test homepage was OneSearch. Over half of the students clicked the Books & Media tab when asked to find a book. Two-thirds of students clicked the Articles & Databases tab when asked to find an article. Some students even noted that one thing they liked about the library website was the tabs box. Narrowing down a library search before clicking the search button may be intuitive for these students, even the ones who said that they hardly ever used the library. Based on this finding, we will be keeping our tabs box, and using the aforementioned type-specific widgets in each. 

Library classes are a big benefit.

Having had a library class session was extremely beneficial to students. Our “one-shot” library sessions usually last under two hours, and most students only get one or two of these class sessions throughout their entire career at John Jay. One-shots are often bemoaned among librarians, as it is truly difficult to pack so many research skills into such a short time. But from our data, students who had a library session have a significant advantage in research skills.


With these findings, we will be redesigning our OneSearch tab box as well as customizing the John Jay-specific OneSearch interface. We will also use the intuitions and pitfalls we came across to guide how we present OneSearch in library class sessions and at the reference desk. We appreciate the participation of so many students in a project so valuable to the Library!


Appendix 1: search strings

Search strings used in OneSearch
  • different ways college students deal with stress
  • different ways college students deal with stress
  • different ways college students deal with stress 
  • different ways students deal with stress
  • peer review
  • peer review student stress
  • stress AND college
  • stress peer review
  • student stress
  • ways that college students deal with stress
 
Search strings used in other library databases
(Academic Search Complete, CQ Researcher, Gale Virtual Reference Library, JSTOR, Opposing Viewpoints, PsycInfo, and SocIndex)
  • college AND stress
  • college AND stress
  • college AND students AND stress AND new york city 
  • college stress
  • college student AND stress
  • college students AND stress
  • college students AND stress
  • college students AND stress
  • college students AND stress
  • college students deal with stress
  • college students stress
  • different ways college students deal with stress
  • different ways college students deal with stress
  • stress AND college students
  • stress management AND college students
  • stress management AND college students
  • stress management AND college students
  • student stress
  • ways college students deal with stress
 
Search strings used in Google or Google Scholar
  • how do students deal with stress in college
  • peer review article stress
  • peer reviewed articles on stress

Appendix 2: lightning usability testing script

Lightning usability script

Note: we were testing for several things in this script, including whether our new discovery service was a usable way to search for books by title. The questions were broad enough that they also served as a litmus test for students’ library skills, which informs our instruction. Feel free to be inspired by our script and try out a version at your own library!

1. Intro. Hi, I’m [name], a librarian here at John Jay. What’s your name/major/year? … We’re testing the library website to see what works and what doesn’t work. This will probably take around 10 minutes. I’ll be giving you three tasks to do. I’ll be timing you, but work at the pace you usually work at when you do your homework. There are no right or wrong solutions to these tasks. We’re testing the website, not you. So if you can’t find something, that’s a problem with the website, not you! Do you have any questions?... All right, let’s get going.

  • Student information: first-year / soph. / junior / senior / senior+ / graduate
  • Major: _______________
  • Have you had a library intro session before? yes / no

2. Find a book: You’re looking for a book with the title The Polar World for a class paper. Does the library have it? [If the student has trouble finding it, say: Okay, at this point, just describe to me what you would normally do next.]

  • Path to find book: _______________
  • Time on stopwatch:    _______________
  • Private: How easy was it for the student to find this book? 1 / 2 / 3 / 4 / 5

3. Find an article, part 1: This is a two-part question. You will need to find a peer-reviewed article for a class paper. First, show me how you would find the definition of peer review if you don’t already know. You can look anywhere you’d usually look. Then tell me what you think it means.

  • Path to find information: _______________
  • Student’s definition of peer review: _______________
  • Time on stopwatch: _______________
  •  Private: How do you rate the student’s understanding of peer review? 1 / 2 / 3 / 4 / 5

Article, part 2: Now, let’s search for a specific article. You need to find a peer-reviewed article about different ways college students deal with stress. What’s the first relevant article you see that you’d want to read? You can look anywhere you’d usually look.

  • Path to find article: _______________
  • Time on stopwatch: _______________
  • Private: How easy was it for the student to find the article? 1 / 2 / 3 / 4 / 5
  • Private: Is it a relevant article? yes / no

4. Debrief: We’re almost done. Tell me what you think of the tasks you did today and your thoughts overall on the library website and the library as a whole. What do you like? What bugs you?

  • Likes:   _______________
  • Dislikes: _______________

Robin Davis

More from the Spring 2015 Newsletter »