Heuristic Evaluation for Google News Beta

By Donald Fox,2014-07-19 20:14
9 views 0
Heuristic Evaluation for Google News Beta

    Mary Hodder, Zhanna Shamis, Diana Stepner

    Course: IS214

    Final Project: Google News

    Due Date: May 12, 2003

1 Overview

    We identified the Google News Beta (“Google News”) as the target of our group project for IS214

    during the first week of February, 2003. Since then we have been using the site in support of our class assignments and in order to complete general search requests that arise. Throughout this time we have found Google News to be a very “usable” site.

1.1 Positive aspects of Google News

    Google News incorporates a simple design which supports rapid navigation and comprehension. Such a structure allows users to quickly scan, search, and access news. As a result, based on our observations, individuals who visit Google News tend to be satisfied with the experience.

1.2 Areas of improvement for Google News

    Adding the word “beta” to the tagline of a site or product implies it is still under development. Despite our positive experience with Google News, the “beta” reference seemed to be appropriate.

    The minor issues encountered during the evaluation period fell into the following categories:

    ; Terminology Ambiguous words or phrases were used in critical features and functions

    ; Navigation Common web site navigation principles were not followed

    ; Consistency Variances exist between the layout and structure of Google and Google News

1.3 Recommendations for Google News

    When individuals say they are satisfied with a web site” words such as sufficient or adequate come to

    mind. These terms do not seem appropriate for a company which states its technology is “… better

    than any other online service and is known as the “World's Best Search Engine". The following

    recommendations could help Google News to achieve the same level of recognition.

    ; Perform user testing in order to identify terms which convey a concise meaning and better

    represent the functions available on the Google News site.

    ; Improve usability by incorporating common web navigation practices such as consistency

    between sites and the inclusion of common web practices and standards (e.g. link highlighting)

    ; Leverage Google‟s search expertise to incorporate improved functionality within the Google

    News site for example adding category and periodical specific searching

2 Approach

    Three methods of usability evaluation were carried out on the Google News site from February 2003 to May 2003. The following method were used Interviews, heuristics, and user task analysis.

2.1 Purpose

    The primary was to apply practices taught during IS214 to a “real world” application. The secondary

    objective was to evaluate a site which, despite a high level of visibility due to the Google name, is still in the early stages of release. At such a time, opportunities might exist to identify improvements and possibly impact some aspects of the site‟s development.

2.2 Methods

    Tests were carried out on individuals who use Google and Google News as well as other online news and search sites. These individuals were targeted as they were considered a likely audience for Google News. Both the interview script and heuristic evaluation were developed in conjunction with course assignments. As a result, iterative development was carried out in order to complete the corresponding assignment. Due to the level of insight gained through initial testing, a similar try-revise-launch strategy was used for user task analysis. Modifications were made after feedback was gathered on the preliminary list of scenarios.

    ; Interviews: Questions were asked independently and following the User Task Analysis.

    Individuals were asked a series of questions in order to evaluate whether their General News

    Patterns and Online News Patterns influenced their Site Use of Google News

    ; Heuristic Evaluation: Web site evaluation heuristics identified by Dick Miller and Jakob Nielsen

    were used to review Google News. Findings were rated according to location and severity.

    ; User Task Analysis: Individuals were presented with a scenario and directed to complete a

    supporting task. Comments regarding the available and expected features as well as the

2.2.1 Interviews


    As outlined above, the reason for including interviews as an evaluation method was to gain feedback from individuals on three areas:

    ; General News Patterns: From what sources and how often do people obtain news

    ; Online News Patterns: How do people locate online news items, how important is it to them to

    be able to search by specific criteria, and do people want to customize their online news


    ; Site Use: How often do people use Google News, do they visit it more or less often than similar

    sites, and how do they rate their overall satisfaction with its available features and


These areas were identified, because they allow evaluation of Google News‟ site layout, content

    organization, and supporting features as well as identify the level of satisfaction with Google News content. In addition the responses provided a method to compare findings with those obtained via the other two evaluation methods (e.g. Heuristic Evaluation and User Tasks Analysis). Specifically it is interesting to gather individuals‟ perceptions through interviews and then use this insight to compare (or contrast) with people‟s responses after using the same features to accomplish a specific task.

    Along the same lines, after conducting heuristic evaluation, ideas were formed around what aspects of Google News people would like and dislike. The interview responses helped gain insight into other people‟s perspectives and overcome any preconceived notions.

    Target Group

    The audience for the interviews consisted of users of Google news, other news portal sites such as Yahoo! News (, and particular online publication sites such as the New York Times online ( and the San Francisco Chronicle ( Such

    individuals were selected because their active browsing and searching of news related content online had made them familiar with how news sites lay out their pages, filter their content for the web, design their navigation and features, and brand themselves visually. As a result of this experience and exposure, such users had specific opinions relating to a news site‟s content, layout, features, and

    functions. As a result they were more adept at recognizing and proposing changes and/or additions to the content, features, and functions offered by Google News.

In order to identify “if the content satisfies users need for news”, users who rarely use online news

    sources were also interviewed. Such individuals offered insight into roadblocks which would need to be overcome in order to migrate offline news users to online news sources. In addition their comments could be used to help redesign Google News‟ content, layout, features, and functions;

    expand the target audience; and, improve general browsing and searching for news online. Summary

    Due to the waning presence of war, people appeared to be more interested in receiving up-to-date news than they would be under "normal" situations. As a result, we found people relied on multiple news sources. Individuals indicated that they would probably alter their behavior and focus on one site for news in the coming months. Despite indicating they would consolidate their news viewing, some people noted they would still go to other sites for coverage of sports and entertainment.

    In regards to customizing their homepage, people tended to the extremes and had either no expectation or very specific requirements for customizing news. Depending on their view, Google News lack of customization was satisfactory. Similar feedback also applied to search. As all the headline news items are found on the home page, individuals who did not want an intense interaction with the site were satisfied with Google News layout. On the other hand, for those desiring a customized news presentation, there was also a desire to search in a more specialized way. For example, the ability to search within a specific category, such as Entertainment or timeframe, was mentioned by a number of individuals.

    Interview Questionnaire

    See Appendix A

2.2.2 Heuristic Evaluation


    The heuristics evaluation of Google News was conducted using two sets of criteria - Dick Miller‟s and

    Jakob Nielsen‟s heuristic. Aspects of the site were reviewed, violations identified, and severity ratings assigned. Recommendations for correcting the violation were included as well. In order to maintain consistency, the rating scale follows the premise posted on Nielsen‟s site. It is based on a

    combination of frequency, impact, and persistence of usability problem. The ratings follow a 0 to 4 scale and map to the following information:

    0 = No usability problem

    1 = Cosmetic problem only: fix as time allows

    2 = Minor usability problem: low priority fix

    3 = Major usability problem: high priority fix

    4 = Usability catastrophe: imperative fix


    As outlined above, Google News incorporates a simple design which makes navigation and comprehension quite simple tasks. Such a structure allows users to quickly scan, search, and access news. As a result, the minor issues that were identified stand out and their correction would truly benefit the overall experience of using Google News.

    The minor issues that were common to our evaluations fell into the following categories:

    ; Terminology: Usage of terms such as “ago” in the time-stamping of articles, “relevance” in the

    sorting of articles, and “in the news” in the classification of articles imposed Google‟s views on

    users. Instead of doing so, Google News could provide more tangible metrics. For example

    including the actual date and time that an article was posted would eliminate questions

    regarding the definition of “ago”. Providing a mechanism to sort by popularity (e.g. number of

    times viewed) would eliminate any disagreement around article relevance. Along the same

    lines, given an articles‟ inclusion in Google News it is already apparent that the item is “in the

    news”. As a result, applying the “in the news” classification is redundant and can be

    eliminated or simply titled in a more descriptive manner.

    ; Navigation: Under this category, link color inconsistencies and transfer between certain

    aspects of the site were raised most frequently. In regards to link color, the color used to

    identify a link had been visited varied between sections of Google News. As a result, it was

    difficult to quickly determine if a link was new or had previously been visited. Such behavior

    or lack there of, breaks with a well-known, yet unspoken, web standard of keeping users

    constantly informed of where they have been. Given Google‟s overall focus on ease-of-use

    and awareness of HCI principles, we were surprised to encounter this issue.

    ; Consistency: To prevent confusion and increase the likelihood of positive user experience,

    websites typically focus on consistency. In most cases, Google News follows this principle and

    incorporates practices (e.g. terminology, links, etc.) that are found on Google‟s main site. The

    differences we identified that raised the most concern were: the inclusion of a redundant

    search button (e.g. Google News includes Search the Web and Search News vs. Google

    includes Search the Web and I‟m Feeling Lucky), lack of “Advanced Features” on Google News,

    and variations in the buttons found across the bottom of the Google News‟ pages depending

    on the area of the site being visited.


    Despite the issues identified above, the overall navigation of Google News is intuitive especially for

    browsing and searching. As a result none of the violations identified deemed Google News ineffectual in its presentation of news. Instead the items seemed to be more characteristic of Google News‟ status as a “beta” site. Before removing this classification, Google should review the structure

    of Google News and clear up any items which slow users‟ ability to search, browse, or scan for news.

    Heuristic Evaluation

    See Appendix B and Appendix C

2.2.3 User Task Analysis


    Five individuals were asked to perform seven tasks. The tasks covered search (general, subject), casual browsing, page layout, features, and documentation. These areas were selected because they represent the main aspects of the Google News site. Scenarios were provided to set the context and make the tasks seem more realistic. Each individual was asked to work through the tasks and provide feedback both during and upon completion. The “talk out loud” method was used, because it

    provides insight into actual perceptions and attitudes while addressing a specific task. Such information serves as a contrast to feedback gathered upon completion. At that time, the individual may have thought through (e.g. refined) their input or simply be reflecting upon the elation felt after completing a task.


    To facilitate analysis, observations were gathered in an excel document. Each task took between two to four minutes to complete. As a result, most people finished all seven within 20 to 30 minutes. Such timing worked well, because individuals perceived 30 minutes as the threshold between just enough and too much time to spend participating in this activity. In addition, given the timing, individuals did not feel overwhelmed when a few interview questions were asked upon completion.

    Observations can be broken down across the categories being covered. They are as follows:

    ; Search (general, subject): Even though participants were able to accomplish the tasks quickly,

    they identified a number of improvements, which had they been available, would have helped

    their search. Specifically individuals would have liked to have been able to limit their search

    to certain news sources (i.e. New York Times, Washington Post etc.) and categories (e.g.

    sports). These categories are represented down the left hand navigation bar within Google


    ; Casual browsing: Sort by Date and Sort by Relevance are easy to locate on the search results

    page. Some individuals questioned the meaning of “Relevance”. These comments reflected

    the issues outlined in the Heuristics section of this document

    ; Page layout: Unfortunately the top portion of the Google News homepage seemed to present

    the most confusion. When asked to identify “Top News Stories” individuals debated about

    whether the items mentioned underneath the two main stories should also be considered "Top

    News Stories". In the end most individuals came up with the same figure (e.g. there were

    seven Top News Stories), but improvements to the page layout could eliminate confusion

    around what seems to be a simply classification task.

    ; Features: Google News offers very few features. Those which are provided are easy to locate

    and use. As a result, people completed this task very quickly. If additional features are

    provided, hopefully their inclusion will not detract from Google News‟ overall ease-of-use.

    ; Documentation: Most sites provide links to corporate or comprehensive information (e.g.

    About Us, Employment, Site Map, Help, etc) within one of the navigation frames. Google News

    follows this common practice and provides a link to the “About Google News” page within the

    left hand navigation bar. Individuals who use the Internet quite frequently quickly found this

    link. A few people commented that it would have been more appropriate to include the link at

    the bottom of the page. Even though quite a few sites place links to corporate and sitewide

    information at the bottom of the page (e.g. About Us, Employment, Site Map, Help, etc) doing

    so may require scrolling. By including the “About Google News” link on the left hand

    navigation, it is immediately visible when an individual enters Google News.


    Google is well-known for its ease of use. As Google‟s area of expertise is search, it makes sense that individuals had little or no difficulty completing the tasks which focused on search. In addition, due to the simple page layout again reflective of Google‟s minimalist structure completion of the page

    layout, features, and documentation tasks were completed quickly and very few recommendations were provided. The improvements which were identified tended to focus on more detailed features, such as limiting searches to specific news sources and categories. Such requests could be incorporated into an Advanced Search option. Even though Google‟s main site offers this option, it is

    currently not available within Google News.

    Heuristic Evaluation

    See Appendix D


    1. From what sources do you obtain news information? (circle all that apply)

    a. Television

    b. Radio

    c. Newspaper (physical)

    a. News magazine (physical)

    b. Online (Mainstream Sources: newspapers/publications/content sites)

    c. Online (Non-Mainstream Sources: portals, weblogs, community sites, newslists)

    d. Other _________________________________________________________________________

    2. How much time do you actively devote to reading/viewing/listening to news per week?

    a. < 1 hr/wk.

    b. 1-3 hrs/wk

    c. 4-7 hrs./wk

    d. 8-10 hrs/wk.

    e. > 10 hrs/wk.

    3. How often do you use the Google News beta site?

    a. Never

    b. Every Day

    c. Once a week

    d. Once a month

    e. Other. If so, how often? ______________

    4. If you use the Google News beta site, how would you compare it with other news aggregators such as Yahoo! News or MSN News?

    a. Better than other sites

    b. Comparable to other sites

    c. Worse than other sites

    d. Please explain your choice:




    5. If you read news online, how would you describe your method(s) for retrieving/accessing news (circle all that apply)

    a. Search for specific articles

    b. Browse current/ archived headlines in Site‟s main layout (ex: NY Times front page layout)

    c. Browse sections by topic of interest

    d. Other (please describe) _______________________________________________________

    6. How important to you is the ability to search for news content within a particular region? (i.e. Bay Area, Hong Kong, Middle East)

    a. Very Important

    b. Important

    c. Somewhat Important

    d. Not Important

    e. Don‟t Care

    7. How important to you is the ability to search for news content by publication? (i.e. New York Times or

    a. Very Important

    b. Important

    c. Somewhat Important

    d. Not Important

    e. Don‟t Care

    8. How important is the ability to customize your online news by: (circle all that apply)

    a. Subject (i.e. Sports, Technology)

    b. Region (i.e. Bay Area, Middle East)

    c. News Source (i.e. New York Times, Washington Post)

    d. Page Layout (i.e. select where articles appear on the page)

    e. Related articles

    f. Defining alerts (i.e. receive a message when news on a specific subject is posted)

    g. Other _________________________________

    e. Customizing news is not important to me

    9. How important is it to have access to historical news (e.g. news more than 30 days old)?

    a. Very Important

    b. Important

    c. Somewhat Important

    d. Not Important

    e. Don‟t Care

    10. If you use Google News beta, are you satisfied with the overall ease of your use?

    a. Haven't used it yet

    b. Yes

    c. No - If no, please explain how it is not easy to find what you are looking for:



    11. If you use Google News beta, are you satisfied with the articles and content available on the front page and subject pages?

    a. Haven't used it yet

    b. Yes

    c. No - If no, please explain how the front page or subject page articles and content

    returned are not satisfactory::



    12. If you use Google News beta, are you satisfied with your search results when you conduct specific searches on news articles?

    a. Haven't used it yet

    b. Yes

    c. No - If no, please explain how items returned are not satisfactory:




    Dick Miller

Web Site Evaluation Criteria by Dick Miller



Severity Ratings

    0 = I don't agree that this is a usability problem at all

    1 = Cosmetic problem only: need not be fixed unless extra time is available on project 2 = Minor usability problem: fixing this should be given low priority

    3 = Major usability problem: important to fix, so should be given high priority 4 = Usability catastrophe: imperative to fix this before product can be released

Appropriate use of web technologies:

    ; Miller:

    o Leading-edge technologies are not employed for their own sake, but in order to meet

    an important business, learning, or information need. For example, use of frames is

    kept to a minimum, and do not interfere or confuse users‟ mental models of the site.

    In addition graphics are kept to a minimum and are used only for providing necessary

    information. The same notion applies to animated graphics, sound and video. They

    should not be used unless they meet an important business, learning, or information

    need that cannot be addressed with simpler technology. If these advanced are used,

    accommodation is made for users who turn off graphic information displays in their


    ; Google News:

    o Google News is relatively simple in design and does not employ any large graphics, or

    other technologies that would required users to have special programs (ie QuickTime,

    Acrobat, Windows Media applications) in order to view content. Frames are used, but

    kept to a minimum and are consistently used within the site. Graphics used consist of

    the Google News logo, the Google menu and Google News menu. Thumbnail photos

    are used which link to the stories they are associated with, and are small and

    appropriately labeled with text links to the same stories. [Severity rating: 0]

    o Google News has a Text Version link at the far left of the site. This link takes the user

    to a text only version, without thumbnails. The text version follows the same site logic,

    with TOP STORIES listed first. Five or six stories, with the same coloring and logic are

    listed, and a menu appears in the upper left to jump to section headings like Business,

    Heath, etc. If more stories are desired, however, the user must go back to the more

    graphical version of Google news, to link to the section, scroll to the Business section

    header, and click on Business, to see maybe twenty stories in the section. It would be

Report this document

For any questions or suggestions please email