Competency N

Introduction

The fourteenth competency in the SJSU iSchool MLIS program is to "evaluate programs and services using measurable criteria".

Evaluation can be defined as a process for making judgements about the qualities of something. The purpose behind evaluation is to provide feedback to influence making decisions or policies. Without evaluation an individual, group, or organization is blind to their progress towards an aim or goal. Evaluation is used by information organizations to "[determine] the success, impact, results, costs, outcomes, or other factors related to a library activity, program, service, or resource use" (McClure, 2008, p. 179). An information organization uses evaluation as a means to determine how well they meet the needs of their patrons or clientele and how wisely they are using their allocated resources. Evaluation helps focus the organization on the priorities and efforts that give the greatest impact and benefit to their communities. This process of evaluation helps information organizations ensure that the day-to-day work and efforts are aligned with the organization's mission, vision, and strategic goals.

As the purpose and focus of information organizations shifts they have an important need to show their continued value in a changing society. To show their value they will need to ask the right questions to find answers that demonstrate their importance and value within society. Traditionally information organizations have focused on building their collections and so the value and quality of the organization was tied to the size of their collections (Hernon and Altman, 2010, p. 2). Assessing and evaluating the value of an information organization requires asking questions better focused on their modern-day purpose and function.

Asking the right questions and then gathering the right data to accurately measure how things really are within an organization is a challenging aspect of evaluation and assessment. Information organizations (or more specifically libraries) do well at collecting usage statistics. The idea has been that if "libraries are used, they are of value to the users" (Tenopir, 2012, p. 6). These kinds of measures are focused on measuring the inputs and outputs to libraries but miss the processes and outcomes that come from a library. This includes understanding the needs of library patrons and their satisfaction with library services.

The evaluation process utilizes measures and quality standards. Measures identify the observed state or value of a particular program or service at a particular moment in time. The evaluation process can utilize many different types of measures including performance measures (i.e. how well something is performed), output measures (i.e. the result of a program or service), and outcome measures (i.e. the impact of a program or service on something else). Quality standards are the desired quality or performance level of an organization's program or service. Evaluation criteria focuses on the following questions (McClure, 2008, p. 182-183):

As the right questions are asked and measured information organizations will position themselves to be able to demonstrate their value to their patrons and communities. Coursework in the SJSU MLIS program provided training and experience to evaluate programs and services using measurable criteria.

Evidence

The following projects and assignments demonstrates experience evaluating programs and services using measurable criteria.

  1. Reference interview evaluation
  2. Evaluation research project proposal
  3. Discussion post: Evaluating the library's physical space
  4. Discussion post: Evaluating library IT service
  5. Discussion post: Evaluating library patron feedback

LIBR 210 Reference Interview Evaluation

Evaluating the interaction between patrons and the staff of an information organization is one way to determine whether the organization is meeting the needs of its community. Reference interviews are one type of interaction where staff of the organization converse with a patron to identify their information need and direct them to resources that will satisfy that need. Evaluating reference interviews helps an information organization know the best way to provide reference services to its patrons or clientele.

An assignment for a reference and information services course (LIBR 210) provided an opportunity to evaluate a hypothetical reference interview. The assignment shows a possible interaction between patron and reference librarian involving a legal question. The interactions were measured against the RUSA (2013) Guidelines for Behavioral Performance of Reference and Information Service Providers as well as the RUSA (2001) Guidelines for Medical, Legal, and Business Responses. The success of this reference interview involves striking an appropriate balance between serving the patron in finding resources to answer their questions while also following the ethic and written library policy that the librarian is not able to provide authoritative legal counsel. A RUSA guideline was correlated with each reference interview transaction. The interactions were successful based on their matching the ideal described in the guidelines. The RUSA guidelines are valuable tools to help evaluate the quality and usefulness of reference services. These guidelines help reference librarians stay objective through the reference interview process while still allowing the patron freedom to express their views and beliefs. The guidelines are also beneficial by helping the librarian focus on helping patrons learn to learn by showing them resources that answer their information needs.

INFO 285 Research Project Proposal

The evaluation of services and programs offered by an information organization requires collecting data that will help answer questions that help measure the success of the service or program. A final project for a research methods course (INFO 285) involved writing a research study proposal to evaluate a chosen library service or program. This project focused on submitting a proposal to evaluate an online archival finding aid interface for the L. Tom Perry Special Collections at the Harold B. Lee Library (HBLL) on the campus of Brigham Young University (BYU). Online archival finding aids are used by archives and special collections to show staff and patrons what items are contained in an archive or collection and where those items are located.

The finding aid interface for the Perry Special Collections was redesigned seven years ago. Because of the rapid changes in mobile computing technology since the interface was redesigned this research study would evaluate how well the finding aids interface meet the needs of staff and patrons using mobile computing devices. To help measure the effectiveness of the finding aid interface data will be collected by conducting a usability study of the interface.

Archival library staff, campus faculty, undergraduate students, and graduate students will be asked to participate in the study. Participants will perform tasks using mobile devices (i.e. a tablet and smartphone) that require them to navigate and use the finding aid interface. A researcher will observe the completion of these tasks and record the results on an evaluation form that determines whether a task was completed or not and the length of time it took to complete the task. Surveys will be given to participants before and after the usability test. The pre-test survey will provide background information about the participant. The post-test survey will gauge customer satisfaction with the interface and allow comments to be shared. In addition to data collected from a usability study server logs for the finding aid interface web application will be collected and anonymized. These general usage statistics will help to see the popularity of the EAD interface, areas within the web application that received the most use, and how many visitors to the web application used a mobile computing device.

The surveys include ordinal, nominal, and interval variables that can be interpreted using quantitative measures like frequency, averages, and variability. These measures can help identify patterns related to study participant characteristics, their use of the finding aid interface, and the level of satisfaction with the interface. Usability test question results will be analyzed to identify the success level of a participant with the finding aid interface. Usability test questions that are a struggle for a majority of participants to complete successfully will indicate ineffective areas in the finding aid interface that may need to be redesigned for use on a mobile device. The survey results will be correlated with the usability test results to identify patterns that would indicate the effectiveness of the finding aid interface in meeting the needs of the participant. Qualitative comments in the post-test survey will be plotted on a quadrant analysis chart (Hernon and Altman, 2010, p. 156-158) to visually identify how well the service meets of the study participants. The chart compares customer satisfaction with service quality and would reveal how the expectations of the participant coincides with their level of satisfaction. The qualitative analysis results can then be compared with the usability test results. Usability test tasks where a participant is unsuccessful should correspond to low customer satisfaction and low service quality. Usability tests where the customer is successful should map to high customer satisfaction and high service quality. By performing statistical analysis on data collected from surveys, a usability study, and the server logs from a web application this research study proposal demonstrates a way to evaluate a library service using measurable criteria.

INFO 285 Discussion Post: Evaluating the Library's Physical Space

A crucial step in evaluating services and programs is asking the right questions that lead to relevant answers for evaluating the service or program. These questions and answers need to be aligned with measurable criteria that can effectively evaluate the service or program. A discussion post for a research methods course (INFO 285) explored an initial idea for evaluating a library's use of their physical space. The evaluation would try to answer what the impact would be on an academic library if they moved a majority of their physical collection to compact storage and freed up more space on each floor for other uses.

This discussion post explores quantitative and qualitative measures that could be used to answer this question. Answering this question requires understanding how the current library materials are utilized. Quantitative data about the library's current inventory and its utilization over time could be used. If there is high utilization of the current physical items available to patrons removing those items may impact patrons' use of the library and its materials. Qualitative data could be gathered using surveys and interviews to evaluate how students, faculty, and staff at the university use the physical items in the library. Evaluating the impact of moving physical items in an academic library would be based on measures related to current and historical usage of the library materials.

INFO 285 Discussion Post: Evaluating Library IT Service

Library IT staff provide various services to the library and its community. With the increase of a library's reliance on computing hardware and software to provide the services and programs offered by a library it is crucial to evaluate the effectiveness and usefulness of Library IT services. One question to help evaluate this effectiveness is:

Does equipment work when it has been repaired?

To answer this question involves knowing the context and scope of this particular issue. This would initially involve identifying the existing equipment available in the library. Gathering this information could be done with surveys or interviews with each division in the library. A tracking system could be developed to facilitate knowing what equipment is utilized in the library and by whom. A next step in answering this question involves asking tracking when and how equipment fails. When failure occurs a staff member would be assigned to repair the equipment. The time it took to repair the equipment would also need to be tracked. Once an item is repaired and the original employee or department reporting the failure is notified they would need to review the repair and verify that the problem was resolved. The organization would need to agree upon the methods and measures for tracking equipment failures, repairs, and verification of the repair. This agreement would include knowing what constitutes equipment being repaired. Without this agreement there would be a level of subjectivity relative to the technician's and patron's view of repaired equipment. There needs to be a clear understanding of what is considered a quality service offering. This discussion post demonstrates the need to understand the context of a service or program being evaluated and the need to have a clear set of measurable criteria and quality standards to effectively compare a service's or program's current level of quality.

INFO 285 Discussion Post: Evaluating Library Patron Feedback

To be an adaptive and continually improving information organization that is relevant and valuable in the lives of those the organization serves requires focusing on the needs of patrons or clientele. Evaluating a service or program includes measuring the effectiveness, service quality, impact, and usefulness of the service or program in relation to its patrons or clientele. To perform this evaluation effectively for an organization requires receiving and handling patron feedback.

A discussion post for a research methods course (INFO 285) discussed issues related to an organization's use and handling of customer feedback. Evaluation efforts should utilize customer feedback as a way to evaluate the success of a service or program. This user feedback system would require working with library staff to determine where the information organization stands in relation to receiving and responding to customer feedback. Asking and answering the following types of questions can motivate the organization's staff and leadership to evaluate where they are in regard to meeting the needs of patrons and what improvements can be made to elevate the level of service.

Establishing a customer feedback system would be part of the measurable criteria that an organization uses to evaluate its services and programs. Establishing a system for receiving and handling patron or clientele feedback builds confidence in the organization being able to respond to the needs of its patrons or clientele. Confident patrons lead to satisfied patrons which in turn can lead to loyal patrons (Hernon and Altman, 2010).

Conclusion

At their core information organizations are the people that work within them. People are imperfect beings that make mistakes and can lead an organization down inefficient and ineffective paths and activities. Evaluation corrects for these imperfections and redirect an organization and its staff towards service excellence. Evaluation also helps information organizations and their staff adapt to their changing environment. To ensure success in adapting to a changing environment and maintain relevance and value in the lives of an organization's patrons or clientele evaluation is best performed regularly on an ongoing, iterative basis. Information organizations need to develop an evaluation culture that is focused on measuring (and improving) services and programs offered by the organization. This requires identifying and utilizing measurable criteria and agreed upon quality standards that are used when evaluating an organization's services and programs. Coursework completed in the SJSU MLIS program provided experience evaluating services and programs using measurable criteria.

References

American Library Association. (2008). Guidelines for behavioral performance of reference and information service providers. Retrieved from http://www.ala.org/rusa/resources/ guidelines/guidelinesbehavioral

American Library Association. (2008). Guidelines for medical, legal, and business responses. Retrieved http://www.ala.org/rusa/resources/guidelines/guidelinesmedical

Hernon, P. & Altman, E. (2010). Assessing service quality: Satisfying the expectations of library customers. Chicago: American Library Association.

Hiller, S. (2012, September/October). What are we measuring, and does it matter? Information Outlook, 16(5), 10-12, 41.

McClure, C. R. (2008). Learning and using evaluation: A practical introduction. In K. Haycock & B. E. Sheldon (Eds.), The portable MLIS: Insights form the experts. (p. 179-192). Westport, Connecticut: Libraries Unlimited.

Trochim, W. (2006). Introduction to Evaluation. Retrieved from http://www.socialresearchmethods.net/kb/intreval.php

Tenopir, C. (2012). Beyond usage: measuring library outcomes and value. Library Management, 33(1/2), 5-13.