DOC

Template for a Usability Evaluation Plan

By Jane Austin,2014-05-15 21:19
7 views 0
Template for a Usability Evaluation Plan

    Attachment C

    Usability Evaluation Plan

    Draft SEPA Project Checklist Test

    Issues or objectives

    ; Is the form logical and helpful to the applicants and the agencies?

    ; Do applicants fill out all the information?

    ; Can applicants provide the requested information?

    ; Does the form ask for the right and/or sufficient information?

    ; Do the questions get answered correctly?

    ; Can the agencies find the information?

Participants

    Characteristic 1 Characteristic 2 Characteristic 3 Applicant Agency Reviewing

    Applicant Proposal Lead Agency - Participant Participant Agency

    GMA or Non Phase I Phase II Participant 12GMA Phase III Boeing City of Renton Professional/ Complex GMA

    experienced 4 5 Professional/ Complex GMA single 3experienced agency reviewer 4 5 Professional/ Complex Non GMA

    experienced 4 5 Nonprofessional/ Simple GMA

    inexperienced 4 5 Nonprofessional/ Simple GMA

    inexperienced 4 5 Nonprofessional/ Simple Non GMA

    inexperienced 4 5 Nonprofessional/ Complex GMA

    inexperienced 4 5 Nonprofessional/ Complex GMA

    inexperienced 4 5 Nonprofessional/ Complex Non GMA

    inexperienced single agency 3 reviewer

     1 Describes the lead agency for the proposal as GMA or non GMA; subcategories include local vs. state and agencies with single reviewers of the checklist vs. multiple reviewers of the checklist 2 Will identify at least one reviewing agency for each test. The reviewing agency will likely vary between another local jurisdiction, WA DFW, WA DOE, and WA DNR. 3 It is assumed there are differences in how an agency with a lot of specialized staff reviews and processes a proposal versus how an agency with a few staff covering multiple responsibilities processes a proposal. 4 Will be identified by the local jurisdiction and will meet the characteristics requested. 5 Will be a Western or Eastern Washington county or city that volunteers to participate and has identified an applicant and proposal which meets the characteristics such as Thurston County, Clark County, City of Burien, Walla Walla County

    Characteristic 1 Characteristic 2 Characteristic 3 Applicant Agency Reviewing

    Applicant Proposal Lead Agency - Participant Participant Agency

    GMA or Non Phase I Phase II Participant 67GMA Phase III WA DOT WA DOT Local jurisdiction Professional/ Complex Non GMA - state

    experienced, 8state agency Professional/ Complex GMA or Non

    experienced, GMA 8 local agencyLogging DNR Nonprofessional/Simple Non GMA state applicant inexperienced Note: It is possible there are differences in how a proposal is reviewed and processed between Eastern

    Washington and Western Washington. At this time, differences have not been identified and the test

    assumes no differences.

    Schedule and Locations

    ; The testing will occur during March, April and May. Each test will be

    scheduled for a different day.

    ; Projects and tests will be scheduled in geographic areas which represent GMA

    (including at least 3 local lead agency projects) and represent non-GMA

    (including at least 3 local lead agency projects).

    ; Complex or simple projects can occur anywhere. Less time would be needed

    if the user is located close to the tester’s work site.

    To be completed:

    Participant Location Date Time

     6 Describes the lead agency for the proposal as GMA or non GMA; subcategories include local vs. state and agencies with single reviewers of the checklist vs. multiple reviewers of the checklist 7 Will identify at least one reviewing agency for each test. The reviewing agency will likely vary between another local jurisdiction, WA DFW, WA DOE, and WA DNR. 8 Because state agencies and local jurisdictions will also serve as an applicant, it will be appropriate to conduct a test for a state agency applicant and a local jurisdiction applicant. The geographic location for these two tests is not critical. Less time would be needed if the user is located close to the tester’s work site .

    Recruiting

    ; Recruiting will be done by the participating lead agencies

    ; Recruiting will be initiated as soon as the testing plan is reviewed by

    the project checklist drafting committee. It will be ongoing during the

    testing (March 1 April 30)

    ; Agency staff and/or applicants who have previously indicated a

    willingness to participate will be contacted and invited to participate.

    Those agencies which commit to participate will begin screening

    possible applicants (e.g. during preapplication conferences). When

    agencies identify potential participants, the information will be

    forwarded to the test manager at the Dept. of Ecology and a test date

    identified. Recruiting will continue until all the participant slots have

    been filled.

    ; Assumptions:

    ; Sufficient agencies participate to meet the characteristics and

    numbers of testers needed

    ; Most applicants are identified within a month (in order to meet the

    2 month time frame).

    ; Applicants agree to be shadowed by an observer

    Activities and techniques to be used during the site visit with the applicant ; Tester meets the user at the host agency’s office or, in the case of a

    professional applicant such as a consultant, at the user’s office.

    ; The host agency (or professional user) provides space, a table and 2 chairs for

    the user to work and for the tester to observe the user working ; Tester introduces the test and the process

    ; User reads and signs the legal form

    ; User fills out the PreTest Questionnaire

    ; Tester sets up the audio recording equipment

    ; Tester asks follow-up questions during the PreTest Interview

    ; Tester turns on the audio recording equipment and begins taking notes ; User begins filling out the checklist and talks about what he/she is doing and

    thinking

    ; Tester observes and takes notes; tester may ask questions to clarify what the

    user is doing and thinking

    ; After completion of Sections 1 and 2 or after 1 ? to 1 ? hours, the user fills

    out the first PostTest Questionnaire

    ; Tester asks follow-up questions during the first PostTest Interview

    ; There will be a break at this time or approximately two hours after start ; Change tapes

    ; After the break, the user continues filling out the checklist, possibly starting

    Section 3

    ; Tester observes and takes notes; tester may ask questions to clarify what the

    user is doing and thinking

; After 1 ? to 1 ? hours or 20 minutes before lunch, the user will complete the

    second PostTest Questionnaire

    ; Tester asks follow-up questions are asked during the second PostTest

    Interview

    ; Break for lunch (1/2 to 1 hour, depending on the user’s preference)

    ; Change tapes

    ; After lunch, the user continues filling out the checklist ; Tester observes and takes notes; tester may ask questions to clarify what the

    user is doing and thinking

    ; Another PostTest Questionnaire, PostTest Interview, and break (change tapes)

    may occur or the user may finish and the final PostTest Questionnaire and

    PostTest Interview will occur.

    ; Turn off audio equipment

    ; Thank the user for their assistance

    ; If the user does not finish, there would be one final PostTest Questionnaire

    and PostTest Interview over the phone, scheduled after the user finishes the

    checklist

(Ginny I wasn’t sure how to link the above to the objectives)

    Schedule for each site visit session with the Applicant

Time (minutes) Activities (one-person note-taker/tester)

    5 minutes greet user

    15 minutes get release forms signed, set up audio taping equipment,

    collect user profile information with questionnaire

    1 ? hours observe the user start the task

    20 minutes user fills out post test questionnaire, ask some post-test

    questions that will be asked again at other times during the

    day

    15 minutes break, change the tape

    1 ? hours Observe the user continue the task

    20 minutes user fills out a new post test questionnaire, ask same post-

    test questions as earlier

    ? hour to 1 hour Break for lunch, change the tape

    1 ? - 2 ? hours User continues the task, go until finished or stop at 1 ?

    hours if not close to being finished

    20 minutes User fills out a new post test questionnaire, ask same post-

    (optional) test questions as earlier

    15 minutes Break, change the tape

    1 ? hours User continues the task until finished or time is up

    last 20 minutes User fills out final post test questionnaire, ask final interview

    questions, turn off and pack taping equipment, thank

    everyone, (give out gifts??)

    Activities and techniques to be used during the site visit with the Lead Agency Reviewer

    Note: Tester talks with the user or primary user prior to the session. A schedule is developed which accommodates the user’s typical process. The test may involve observing only one user or may involve observing a series of users.

Scenario 1 one agency user tests whole document

    ; Tester meets the user at the user’s office or business.

    ; User provides a chair for the tester to observe the user working

    ; Tester introduces the test and the process

    ; User reads the information

    ; User fills out the PreTest Questionnaire

    ; Tester sets up the audio recording equipment

    ; Tester asks follow-up questions during the PreTest Interview

    ; Tester turns on the audio recording equipment and begins taking notes

    ; User begins reviewing and editing the checklist and talks about what he/she is

    doing and thinking

    ; Tester observes and takes notes; tester may ask questions to clarify what the

    user is doing and thinking

    ; After completion of Sections 1 and 2 or after 1 ? to 1 ? hours, the user will

    fill out the first PostTest Questionnaire

    ; Tester asks follow-up questions during the first PostTest Interview

    ; There will be a break at this time or approximately two hours after start

    ; Change tapes

    ; After the break, the user continues to review and edit the checklist, possibly

    starting Part 3

    ; Tester observes and takes notes; tester may ask questions to clarify what the

    user is doing

    ; After 1 ? to 1 ? hours or 20 minutes before lunch, the user will complete the

    second post test questionnaire

    ; Tester asks follow-up questions during the second Post test interview

    ; Break for lunch (1/2 to 1 hour, depending on the user’s preference)

    ; Change tapes

    Note: At this point users may change. Other staff within the agency may be responsible for reviewing and editing specific portions of the checklist. If others are involved, the tester will have worked with the primary user to set up appointments in the afternoon with each of the other users. The length of time scheduled with each user will partly depend on the time they currently spend reviewing the existing checklist and what the primary user recommends.

Scenario 1a - Continue after lunch with same user:

    ; After lunch, the user continues reviewing the checklist

; Tester observes and takes notes; tester may ask questions to clarify what the

    user is doing and thinking

    ; Another PostTest Questionnaire, PostTest Interview and break (change tapes)

    may occur or the user may finish and the final post test questionnaire and

    interview will occur.

    ; Turn off audio equipment

    ; Thank the user for their assistance

    ; If the user does not finish, there would be one final PostTest Questionnaire

    and PostTest Interview over the phone, scheduled after the user finishes the

    checklist

    Scenario 1b Continue after lunch with a new user and repeat with each additional user ; Tester meets the user at the user’s office or business.

    ; User provides a chair for the tester to observe the user working ; Tester introduces the test and the process

    ; User reads the information

    ; User fills out the PreTest Questionnaire

    ; Tester sets up the audio recording equipment

    ; Tester asks follow-up questions during the PreTest Interview

    ; Tester turns on the audio recording equipment and begins taking notes ; User begins reviewing and editing the checklist and talks about what he/she is

    doing and thinking

    ; Tester observes user reviewing the checklist and takes notes; tester may ask

    questions to clarify what the user is doing and thinking ; After user completes his/her review, the user will fill out the Post test

    Questionnaire

    ; Tester asks follow-up questions during the PostTest Interview

    ; Turn off audio equipment

    ; Thank the user for their assistance

    Schedule for each site visit session with the Lead Agency Reviewer

    Example for Scenario 1b

    Time (minutes) Activities (assumes one-person note-taker/tester)

    5 minutes greet user

    10 minutes get information sheet read by user, set up audio taping

    equipment, collect user profile information with pretest

    questionnaire

    1 ? hours observe the user start the task

    20 minutes user fills out post test questionnaire, ask some post-test

    questions that will be asked again at other times during the

    day

    15 minutes break, change the tape

    1 ? hours Observe the user continue the task

    20 minutes User fills out (final) post test questionnaire, ask (final)

    interview questions

    ? hour to 1 hour Break for lunch, move to different location, if needed, and set

    up tape, change the tape

    5 minutes greet new user

    10 minutes get information sheet read by user, set up audio taping

    equipment, collect user profile information with pretest

    questionnaire

    ? hour to 2 hours observe the user start the task until completed

    20 minutes User fills out final post test questionnaire, ask final interview

    questions, thank user

    15 minutes Break, move to different location if needed, and set up tape,

    change the tape

     ? - 2 hours Repeat the previous 5 steps, until the end of the day or until

    finished with all scheduled users

    Note: this process also applies for a Reviewing Agency Reviewer except that only one user will be observed. The site visit with the Reviewing Agency Reviewer will be scheduled. The timing will depend on the lead agency completing its review and processing of the checklist.

    Activities and techniques to be used during the site visit with the Reviewing Agency Reviewer

    This activity will essentially be the same as the after lunch activity for the Lead Agency Reviewer

Schedule for each site visit session

    This schedule will essentially be the same as the after lunch activity for the Lead Agency Reviewer except

    that it may occur during any prescheduled 1-3 hour period of the day

Evaluation teams

    ; Patty Betts, Department of Ecology, will conduct the visits and will act as

    observer and note taker

Materials needed

    ; See attached checklist

    Checklist for planning what materials will be needed for a site visit

Check Material

Audio and video tape recording

     power strip

     extension cord

     audio recorder

     tapes for audio recorder

     batteries for audio recorder

     Extra microphone

     extra batteries for microphones

    still camera (may be useful at agency offices to see the environment which

    is provided to the applicant when they come in and for the agency

    reviewers)

     film for still camera

     extra batteries for still camera

Note taking

     Preformatted notetaking sheets (with boxes for time, important points, and

    location on checklist user is working on)

     notebook if taking notes on paper

     pencils, pens

Papers and other materials for working with users

     script for introducing test

     folder for each participant

     copy of correspondence that went to that site (e.g. invitation, checklist

    worksheet)

     release form for each participant at site (and extras)

     information from recruiting or screening questionnaire

     Copies of “Questionnaire About the Project Applicant (Questionnaire About

    the Agency Reviewer)

     Copies of post-evaluation questionnaires

     Gifts Ecology posters?

Report this document

For any questions or suggestions please email
cust-service@docsford.com