System Test Approach - University of Michigan

By Maria Boyd,2014-03-28 09:09
8 views 0
System Test Approach - University of Michiganof,OF,Test,test

University of Michigan Administrative Information Services

    MAIS System Development Life Cycle (SDLC)

    M 5000A - System Test Approach Document

System Test Approach

I. Introduction

    This System Testing Approach document is designed for University of Michigan Administrative Information Services’ (MAIS) System Development Life Cycle (SDLC) projects. The emphasis is on testing critical business processes, minimizing the time necessary for testing while also mitigating risks. It’s important to note that reducing the amount of testing done increases the

    potential for problems after go-live.

A. Purpose of Testing

    Testing ensures that the system still meets the client needs (functional & non-functional requirements). The intent of System Test is to find defects and correct them before go-live. There is no approach or method to guarantee a system completely free of defects. However, following a System Test approach will assist in mitigating risks and ensuring a successful project.

B. Assumptions

    There is a detailed listing of assumptions in Section IV. For System Testing to be successful, there are critical work efforts that need to be accomplished in the preceding phases, Requirements/Analysis, Design and Development.

II. Test Approach

    A. Test Approach Description

    The general methodology/approach for the SDLC System Testing Phase at MAIS will involve the following steps:

    1. Test all the critical impacted business functions at least once.

    2. Confirm the key contacts list is updated.

    3. Review reports needed for tracking System Testing, update if necessary.

    4. Review the Issues Management process, update if necessary.

    5. Confirm the test conditions, cycles, and plans based on functional requirements are updated.

    Plans should include conditions for testing users security access.

    6. Ensure test conditions are grouped by business process. There should be a separate test plan

    that follows a transaction through the entire system versus just at the entry or exit point.

    (End-to-end test plan).

    7. Confirm the interface listing worksheet is updated. This should include all database links

    and the access granted (i.e. Read Only, Read/Write, etc).

    8. Confirm the critical path business processes are updated.

    9. Identify any risks that may jeopardize the schedule completion.

    10. Review the plan and results with Customer and Business Process Owners; obtain

    Acknowledgment of System Test Plan(s).

For each Phase of System Testing:

    11. Execute the test condition.

    784826653.docx Page 1 of 10 3/28/2014

University of Michigan Administrative Information Services

    MAIS System Development Life Cycle (SDLC)

    M 5000A - System Test Approach Document

12. Check the output against the expected results.

    13. Evaluate and document any unexpected results.

    14. Make sure that any required corrections are migrated and re-tested.

    15. Make sure that final testing components (conditions, input, and expected results) are accurate,

    complete and documented in such a way to make them repeatable and reusable. 16. Review and obtain Acknowledgment of System Test results where appropriate (i.e. new


B. Test Phases

    There are separate phases of testing which are designated on the timeline within the overall System Test phase. Each phase may include several types of testing. When possible, some of these phases may be done concurrently. The following is a list of the test phases included in the overall System Test timeframe. However, some of these phases are not covered in detail in this document. There are separate PS Upgrade Methodology Approach documents for those noted, to be used for reference in case additional test types are needed.

Note: It is anticipated that most SDLC projects will require SIT1 and SIT2 tests only.

    The types of testing listed in SIT1 and SIT2 are described in the table in the following section.

    ; System Integration Testing I (SIT1)

    This phase includes integration, system, user testing, security, some end-to-end, and

    regression test types. All RTP conditions with a criticality of A and those with a B and

    a High or Medium level of change will be tested. Refer to the Upgrade Prioritization

    Approach Document for details on the RTP prioritization. The testers may include

    Developers, Business Analysts, MAIS Help Desk, Business Process Owners and End


    ; System Integration Testing II (SIT2)

    This phase includes regression testing (all failed SIT1 conditions), end to end, batch,

    system, integration, and user testing. All RTP conditions that are Low B’s and C’s will

    be tested, per module discretion. Refer to the Upgrade Prioritization Approach

    Document for details on the RTP prioritization. The testers may include Developers,

    Business Analysts, MAIS Help Desk, Business Process Owners, and End-Users. Note:

    There is a separate Upgrade Approach document available for Batch testing, to be

    used for reference.

    ; Parallel Testing

    This phase validates all processes work together to support the business functions, by

    running the new system alongside the existing system. Data and results comparisons

    can easily be made; ensuring successful operation of the new system. There is a

    separate PD Upgrade Methodology Approach document for this testing effort, to be

    used for reference.

784826653.docx Page 2 of 10 3/28/2014

    University of Michigan Administrative Information Services

    MAIS System Development Life Cycle (SDLC)

M 5000A - System Test Approach Document

; Load Testing

    This phase validates that critical functions will meet production performance

    requirements during peak transaction volumes. There is also a separate PS Upgrade

    Methodology Approach document available for this testing effort, to be used for


    Note that it may be possible to perform a simple analytical Load Test, without

    necessarily using a Load Testing tool. This can be done by selecting a targeted

    number of concurrent users. Next, one can specify a mix of users by developing user

    profiles (what those different users typically would do), along with associated typical

    transactions that would be performed by those users. By picking the most frequently

    used and/or the most compute-intensive transactions, you can obtain useful results

    when done correctly.

; Model Office

    This phase enables end-users an opportunity to log into the system, perform their

    typical tasks on the new system, verify their security access, validate their procedures

    and get comfortable with the new system. This phase is scheduled after the hard-

    freeze and if additional defects are found during this phase, migrations will require

    additional levels of sign-offs. Participation in this phase is at the discretion of each

    impacted group. There is a separate PS Upgrade Methodology Approach document

    available for this testing effort, to be used for reference.

; Infrastructure/Gateway Testing

    There is a separate PS Upgrade Methodology Approach document available for this

    testing effort, to be used for reference.

    C. Testing Type Descriptions

    The following is a list of test types that will be performed during SIT1 and SIT2:

Test Type Focus System

    Test Phase

    Testing to find errors in complete functions and SIT1 & SIT2 Integration Testing

    processes within and between units. Ensure

    everything has been linked together correctly.

    Validate that the system functionality performs as SIT1 & SIT2 System Testing

    specified. Functional requirements define how the

    system should perform.

    Validate a transaction through the entire system, not SIT1 & SIT2 End-to-End Testing

    just at entry and exit points. This means a transaction

    is followed throughout the various modules it may

    touch. Must be coordinated.

    784826653.docx Page 3 of 10 3/28/2014

University of Michigan Administrative Information Services

    MAIS System Development Life Cycle (SDLC)

    M 5000A - System Test Approach Document

    Test Type Focus System

    Test Phase

    SIT1 & SIT2 Regression Testing Ensures that the application doesn’t negatively

    impact previously migrated objects/modules. Re-tests

    the application to ensure that a fix did not cause

    another portion to break that was previously working.

    This is done as objects are migrated to fix errors.

    SIT1 & SIT2 Security Testing Eliminates security accessibility errors.

    User Testing Same focus as Integration and System Testing SIT1 &/or

    (executing RTP conditions), performed by Users. SIT2 per

    Also, validates production-ready and data integrity. module

    discretion Note: There is no separate User Testing “phase”

    how users are incorporated into the overall testing

    effort may vary by module. This is the opportunity

    for users to validate functionality prior to the hard


III. Status and Reporting Metrics

    A. Defect Tracking Approach

    During all phases of system testing, defects will be logged, including performance tuning needs. The database will provide the necessary information for developers, as well as provide management with reports. The developers need the name of the reviewer, date discovered, the requirements, test case id, severity code, and any other relevant information. All high and medium priority issues must be resolved or have a work-around in place to go-live.

B. System Test Phase Tracking Approach Metrics to Utilize

    System test will be tracked by phase and within each phase the business processes by module. Leads will be responsible for ensuring that time is entered into Planview and all stats are reported to the system test phase lead in a timely manner. The format of the status reports being utilized for design and development will be used for system testing, as well as the reusable test plan reports. There may be additional reporting requirements, as well.

    Metrics that will be tracked include but are not limited to:

    1. Number of conditions, by priority (ABC), that are in progress, in error, passed, deferred,

    or not started.

    2. Percentage of conditions tested.

    3. Percentage of conditions passed.

    4. Percentage of A, B, and C conditions. (Including % of each priority that has been tested

    and passed.)

    784826653.docx Page 4 of 10 3/28/2014

University of Michigan Administrative Information Services

    MAIS System Development Life Cycle (SDLC)

    M 5000A - System Test Approach Document

    5. What cycle/sub-cycle the conditions are in.

    6. Total number of conditions per cycle/sub-cycle.

    7. Total number of conditions per priority.

    8. Percentage of Effort Complete

    9. Percentage of Effort Earned.

    10. Percentage of Effort Remaining.

    11. Number of Open Testing Incidents by priority.

C. System Test Status/Issues Meeting

    During system testing, it is recommended that a weekly status meeting be held at the same scheduled time on the same day each week. This meeting will support a communication process to inform the project team on the testing status, problems or issues, and changes. The agenda will include a review of open high and medium issues, the overall system test plan, and any action items from the previous week. Meeting minutes should be taken that include a list of attendees, discussion points, and action items. Developers and reviewers should be represented in the meeting, as needed.

D. Acknowledgement

    Where deliverables require “acknowledgement”, this means that the responsible party has reviewed the documentation, feels that the documentation and/or testing was sufficient. Prior to system testing, test plans will require acknowledgement. During system testing, test conditions will require acknowledgement if requested. Reports will be available to provide to the users regarding status of system testing.

IV. Assumptions and Risks for the System Test Approach

    A. Assumptions for System Test Approach

    The following assumptions are critical to the successful accomplishment of this Approach to System Testing:

    Project Management

    ; All critical data should be captured in all phases of the project.

    Requirements/Analysis and Design

    ; The Requirements/Analysis phase will assess the upgrade’s impact on each business

    process area.

    ; The Requirements/Analysis phase will determine the prioritization needed for each

    business process for the following phases including Design, Development, and Test


    784826653.docx Page 5 of 10 3/28/2014

University of Michigan Administrative Information Services

    MAIS System Development Life Cycle (SDLC)

    M 5000A - System Test Approach Document

    ; The Requirements/Analysis phase will identify all cross module impacts and touch

    points within modules.

    ; The Requirements/Analysis phase will determine the timing of object level work efforts

    for the following phases (Design, Development and Testing). In some cases, the work

    may be done outside of the set dates for each phase, based on priority and dependencies.

    For example, a global change like the HE x-lat table change should be completely done

    prior to other changes being started.

    ; Reusable Test Plans (RTPs) will be updated as needed during Requirements/Analysis

    and Design phases. RTPs must be complete prior to the applicable System Test phase.

    ; The Requirements/Analysis phase will determine what Business Processes need to be

    included in Load Test and will identify Interfaces and Remote Database Access (RDA). Data Conversion Validation

    ; Data conversion validation will be done.


    ; If applicable, objects for Load Testing are completed prior to the beginning of the System

    Test phase, so initial system testing can be performed prior to Load Testing.

    ; As development occurs, necessary updates will be made to the RTP’s.

    Unit Testing

    ; During Development and Unit Testing Phase, all necessary Unit Tests were performed. Test Plans

    ; Test Plans will include a level of detail necessary to enable them to be re-used for various

    test phases.

    ; Each module will have their System test plans complete by the start of each applicable


    ; There will be a separate end-to-end test plan, which covers the entire business process.

    ; ODS and Data warehouse will have separate test plans and perform additional testing.

    Note: Groups should include test cycles for ODS and DW in their end-to-end testing.

    ; External interfaces to and from PS are identified with requirements and key contacts in

    the test plans. (Note: There is a separate document for this. The document will then

    indicate which cycle/sub-cycle/condition testes the interface.)

    ; Internal interfaces to/from specific PS modules are identified in the test plans.

    ; Listing of Database links and the access the links provide (Read, Read/Write or Update,

    Read/Write or Update/Delete) is provided with the test plans.

    ; Appropriate resources (Technical, Functional Leads, Help Desk Staff, Business Analysts,

    etc) assist in creation of test conditions, scripts (if applicable) and test execution. Technical

    ; The System Testing instance is full copy/size of production, with production data.

    ; Batch jobs will be run during their normal “Production” times.

    784826653.docx Page 6 of 10 3/28/2014

University of Michigan Administrative Information Services

    MAIS System Development Life Cycle (SDLC)

    M 5000A - System Test Approach Document

    ; A separate environment should be available for Parallel Test, and Load Test. ; The environments (Demo, Dev and Test/QA) will be functioning and available during all

    scheduled test activities. Adequate technical support will be provided to allow the team

    to meet the approved project schedule.

    ; Migrations into System Test environment will be coordinated through a release

    management plan.

    ; Any migrations after SIT2 will require additional signatures.

    ; Backups are occurring on a regular basis to ensure the ability to restore the database. ; Security setup will be complete prior to system testing beginning for the MAIS and user


    ; A plan for disaster recovery is in place. A separate project for updating the disaster

    recovery process and testing that process will be performed independently of System

    Testing. It is outside of scope of an SDLC project.

    ; Document naming and repository standards are established and followed. ; Plan for contention between SIT1, Parallel and Load testing (people and hardware). ; Testing will be performed via the web (not 2-tier or 3-tier, unless applicable). ; Technical resources will be available for performance tuning of critical processes.

B. Risks for System Test Approach

    ; Load testing and Parallel testing instances on same box as System Integration Test. ; Ability to coordinate timing of Load and Parallel testing.

    ; Reusable Test Plans not complete with test conditions defined.

    ; Production support responsibilities.

    ; Availability of Resources (technical and functional).

    ; Development not complete prior to the start of SIT1.

    ; Users not exposed to some form of training.

    ; Security is not setup up before testing begins.

    ; Peak processing times for existing systems during System Testing need to be reviewed.

V. Deliverables

    A. Deliverables from System Test Phase Lead and Methodology Team The following deliverables are required.

    ; Updated System Test Process Flows - if necessary (M 5000B System Test Phase

    Process Flow)

    ; Updated System Test Plan Template (Reusable Test Plan format), updated as

    necessary (M571 System Test Plan TEMPLATE)

    ; Updated System Test Approach Documents (as needed)

    o Updated System Test Approach Document (Overview) (M 5000 System

    Test Approach)

    o Updated Parallel Test Approach (M 5000A2 Parallel Test Approach)

    784826653.docx Page 7 of 10 3/28/2014

University of Michigan Administrative Information Services

    MAIS System Development Life Cycle (SDLC)

    M 5000A - System Test Approach Document

    o Updated Load Test Approach (M 5000A3 Load Test Approach)

    o Updated Batch Test Approach (M 5000A4 Batch Test Approach)

    o Updated Model Office Approach (M 5000A5 Model Office Approach)

    ; Updated Metrics/Tracking Reports

    ; Updated High Level Overall System Test Plan Work Breakdown Structure (WBS)-

    Identify critical path, note where may have conflicts and cushions in timeframe. B. Deliverables from Each System Test Phase

    The following deliverables are required from each phase for it to be considered completed.

    ; Schedule - updated business processes and adjusted hours. This is required at the

    beginning of the System Test Phase.

    ; M571 System Test Plan

    o Master reusable test plan (RTPs) with appropriate rating scale assigned to

    each business process.

    o Test scripts and process flows as necessary

    o A subset of the RTP for SIT1

    o A subset of the RTP for SIT2/User Test the focus here is end-to-end


    o Model Office does not require a test plan, however one may be provided.

    It may be the same as used for SIT1, SIT2/User Test.

    o All internal and external touch points/interfaces and key contacts must be


    ; STAT

    ; TI (Testing Instance) Database (Resolution of issues or work around in place) 784826653.docx Page 8 of 10 3/28/2014

University of Michigan Administrative Information Services

    MAIS System Development Life Cycle (SDLC)

    M 5000A - System Test Approach Document

VI. Appendix

    A. Glossary

    Application Is a set of software components that provide a business function.

    Bug A fault or that portion of a program that fails under specific

    conditions. Also known as a defect.

    Business Function A single business activity such as claim entry or a major business

    function such as claim processing. A specific purpose or capability of

    an application.

    Business Condition An order of steps or functions necessary to perform a major business


    Change Control Procedures The process by which a change is proposed, evaluated, approved or

    rejected, scheduled, and tracked.

    Client/Customer Also known as an application or subject matter expert assigned by the

    owner of the application. A person that possesses specific detailed

    knowledge of an application, system, or business process. In regards

    to testing, the individual(s) assigned to work with the development

    team in defining the test plan, developing acceptance test cases and

    conditions, and reviewing system and acceptance test results.

    Database A collection of data fundamental to a system or enterprise.

    DBA Database administrator has the responsibility for those functions

    needed to manage data in a database

    Defect Also known as a fault or bug, a defect is a portion of a program that,

    when executed under specific conditions cause a failure.

    Design Phase A part of the software development process whose primary purpose is

    to decide how the system will be implemented. During design

    strategic and tactical decisions are made to meet the required

    functional and quality requirements of a system.

    Development Team The group of individuals with primary responsibility for code

    enhancement, upgrade, conversion, or development of a system

    application. This may include a manager, team leader, business

    analyst, technical analyst, technical coders, and DBA.

    Discrepancy A disagreement or inconsistency between the expected and actual


    Error An action that creates a defect also known as fault or bug.

    Failure The actual result from the test case execution does not match the

    expected result.

    Fault Portion of a program that when executed under specific conditions

    cause a failure. Also known as bug or defect.

    Independent Tester The tester is not a part of the development team who test based on the

    knowledge of the functional and performance requirements of the


    Interface Any electronic exchange or sharing of data between applications. The

    applications may exchange information in a variety of methods.

    Directly, by sending or receiving data to/from each other via online

    connections. Data feeds such as magnetic media or electronic data

    interchange (EDI). Indirectly, by mechanisms such as placing files on 784826653.docx Page 9 of 10 3/28/2014

University of Michigan Administrative Information Services

    MAIS System Development Life Cycle (SDLC)

    M 5000A - System Test Approach Document

    a bulletin board, to be pulled by other applications. The interfacing

    applications may also share files accessible to both (e.g. master files),

    where by one application updates the data and the other reads it from

    the same file.

    Path A sequence through a program of statement or instructions that starts

    at an entry, junction, or decision and ends as another junction,

    decision, or exit.

    Project Scope A concise and accurate description of the deliverables to be expected

    from the project and that meet specified requirements as agreed

    between the Project's Stakeholders.

    Review A walkthrough in which one leads one or more members of a

    development team through a deliverable to ask questions, note

    possible errors, standard violations or any other problem.

    Reviewers Those responsible for ensuring a product is completed and fault free at

    a quality review.

    Test Approach Specifies the types and levels of tests as well as test case techniques

    and strategies.

    Test Case A document describing the expected result and execution procedures

    for a specific input process.

    Test Case Repository A collection of test cases stored in a library for future use and thus

    reduces development time.

    Test Criteria A set of measurements that define the success of a test.

    Test Environment The sum of all technologies, tools, people, and procedures needed to

    carry out the testing events for an application.

    Test Plan A document that contains verbiage to describe the test case and

    condition, inputs, expected results, dependencies, rating of risk,

    pass/fail criteria and the necessary details required to execute the test


    Testing The process of exercising or evaluating a unit, component, system by

    manual or automated means to verify that it meets requirements or to

    identify differences between expected and actual results.

    Validation The process of evaluating software at the end of a development

    process to ensure compliance with software requirements

    Verification The process of determining whether or not the products of any given

    phase of the software development cycle meet the requirements

    established during the previous phase.

784826653.docx Page 10 of 10 3/28/2014

Report this document

For any questions or suggestions please email