DOC

VV&A techniques state-of-the-art, draft (TECH1)

By Steve Mitchell,2014-08-17 06:18
12 views 0
VV&A techniques state-of-the-art, draft (TECH1)

    WEU Unclassified

    THALES JP 11.20

    Common Validation, Verification and Accreditation

    Framework for Simulation

    REVVA

    VV&A Techniques State-of-the-Art

    TECH2

Work Element: WE 3100

    Prepared by: TNO, UNI-C

    Distribution: ALEKTO, CERT, DATAMAT, FMV, FOI, TNO,

    UNI-C

    Approved for Release to: JP 11.20 Consortium

     JP 11.20 Management Group

    Document Number

    THALES JP 11.20-WE3100TECH2-D3102

     Version Date: 17-08-2010 Number of Pages: 109

    This document is subject to the terms and conditions of the WEAG MoU Supplement JP 11.20

    THALES JP 11.20-WE3100TECH2-D3102

    EXECUTIVE SUMMARY

    This report gives a survey of the present V&V techniques state-of-the-art with special focus on validation of ―black box‖ simulations. A full chapter is devoted to each of the

    three main sources of information, ―The Recommended Practices Guidelines‖ from DMSO, ―The ITOP initiative‖ and the Dutch experience in Training Simulators. The DMSO chapter introduces the six important V&V steps, verifying user requirements, developing the V&V plan, validating the conceptual model, verifying the design, verifying the implementation, and validating the results. The ITOP chapter describes in detail their proposal for contents and format of the Transfer Package based on a claims-argument-evidence structure and used for communicating V&V results from one organisation to another. In the chapter on VV&A of training simulators a measure of effectiveness for training, ToT (―Transfer of Training‖) is introduced and some methodological pitfalls in the domain of training are described.

    In two subsequent chapters material from other research domains that could benefit the V&V agent have been gathered. The first of these, chapter 5, deals with managing, planning, decision making and contents management topics picked up from business and corporate management research. The other, chapter 6, covers argument-structuring techniques to support a proper and convincing presentation of VV&A cases. The subject matter of chapter 6 is fetched from research in computer-supported cooperative work and from the safety research.

    It is concluded in the final chapter 7 that the next step in moving the art of V&V from its roots in software engineering to meet the challenges of building confidence in large complex simulations of probabilistic scenarios could benefit of using Bayesian Belief Networks and multi-criteria decision aids of which the latter are covered in depth in chapter 5.

    It is beyond of the scope of the present report, TECH2, to give a critical analysis of all the different methods, techniques and tools presented. TECH2 and its companion,

     WEU Unclassified This document is subject to the terms and conditions of the WEAG MoU Supplement JP 11.20 17-08-2010 Version 2.2 Page 2 of 109

    THALES JP 11.20-WE3100TECH2-D3102

    RESRC2, are followed by a third report, GUIDE, which will guide in the selection and use of these practices.

     WEU Unclassified This document is subject to the terms and conditions of the WEAG MoU Supplement JP 11.20 17-08-2010 Version 2.2 Page 3 of 109

    THALES JP 11.20-WE3100TECH2-D3102

    REPORT DOCUMENTATION PAGE

    1. Report number as used internally by Company responsible for document

THALES JP 11.20-WE3.1TECH2-D3102

2. Name and Address of Company responsible for document

     UNI-C, DTU bldg. 304, DK-2800 Lyngby

     3. WEU Contract number

     4. Name and Address of Contracting Authority

     WEAO Research Cell, Rue de la Régence, 4,

     B-1000 Bruxelles

    5. Name and Address of SLIE

     CERT, ONERA

    6. Report Classification and Caveats on use 7. Date of first issue

     WEU Unclassified 26-02-2004

    8. Document title

    VV&A techniques state of the art, TECH2

    9. Author(s)/Company

    Bjarne Stig Andersen, UNI-C

    Martijn van Emmerik, TNO

    Jørgen Moth, UNI-C

     10. Descriptors, Key words

    Thales 11.20, WE3.1, VV&A, techniques, validation, state-of-the-art

11. Abstract (Must be WEU Unclassified)

    This technical report delivers a state-of-the-art report on V&V techniques

     WEU Unclassified This document is subject to the terms and conditions of the WEAG MoU Supplement JP 11.20 17-08-2010 Version 2.2 Page 4 of 109

    THALES JP 11.20-WE3100TECH2-D3102

    CHANGE RECORD

    VERSION DESCRIPTION

     TECH1 was the first draft version of TECH2.

    1.0

     TECH2 is amended with more material on the verification part of 2.0 V&V and with support techniques picked up from other research

    domains.

    2.1 TECH2 version 2.1 has corrected a number of editorial errors found

    in version 2.0.

    2.2 All figures are now numbered in sequence and their titles are added

    to the Table of Contents. References have been changed to ordinal

    numbers indexing the list of references in the last chapter.

    First 2 2.1 2.2 Version Issue

    27-08-2003 26-02-2004 29-10-2004 11-1-2004 Date

     BSA, BSA, JM JM Written by MvE, JM MvE, JM

     JM, LH LH LH Checked by MvE, RJ

     Approved by

     WEU Unclassified This document is subject to the terms and conditions of the WEAG MoU Supplement JP 11.20 17-08-2010 Version 2.2 Page 5 of 109

    THALES JP 11.20-WE3100TECH2-D3102

    Table of Contents

1 INTRODUCTION ........................................................................................................10

    1.1 WORK ELEMENT OVERVIEW ...................................................................................10

    1.2 ABBREVIATIONS .....................................................................................................10

    1.3 GLOSSARY ............................................................................................................11

    1.4 BACKGROUND .......................................................................................................11

    1.5 V&V TECHNIQUES AND SIMULATION TAXONOMIES ...................................................13

    1.5.1 Techniques ......................................................................................................... 13 1.5.2 Simulations ......................................................................................................... 15 2 DMSO RECOMMENDED PRACTICES GUIDE .......................................................16

    2.1 BACKGROUND .......................................................................................................16

    2.2 VERIFICATION METHODOLOGIES AND ACTIVITIES .....................................................17

    2.2.1 Verification and the Simulation Model Development Life Cycle ............. 17

    2.2.2 Methodologies and classes ............................................................................. 20 2.2.3 Commercial Verification Tools........................................................................ 23

    2.3 VALIDATION METHODOLOGIES AND ACTIVITIES ........................................................26

    2.3.1 VV&A for Legacy Simulations......................................................................... 26

    2.4 VALIDATION OF LEGACY AND BLACK BOX SIMULATIONS ..........................................28

    2.4.1 V&V Agent role .................................................................................................. 28 2.4.2 V&V functions ..................................................................................................... 31 2.4.3 V&V Agent‘s Relationship with Other Roles................................................ 32

    2.4.4 Documentation Requirements ........................................................................ 33

    2.4.5 Cost implications and Resourcing ................................................................. 34

    2.5 VALIDATION TECHNIQUES ......................................................................................36

    2.5.1 Data V&V ............................................................................................................. 37 2.5.2 Comparison techniques ................................................................................... 39 Statistical techniques ........................................................................................................ 41 3 ITOP, INTERNATIONAL TEST OPERATIONS PROCEDURES ............................43

    3.1 GENERAL ..............................................................................................................43

    3.2 INTRODUCTION ......................................................................................................43

    3.2.1 Cases ................................................................................................................... 43 3.2.2 Claims .................................................................................................................. 43 3.2.3 Levels ................................................................................................................... 44

     WEU Unclassified This document is subject to the terms and conditions of the WEAG MoU Supplement JP 11.20 17-08-2010 Version 2.2 Page 6 of 109

    THALES JP 11.20-WE3100TECH2-D3102

    3.2.4 Credibility ............................................................................................................. 45 3.2.5 The Transfer Package ...................................................................................... 45

    OLES AND RESPONSIBILITIES ...............................................................................45 3.3 R

    3.4 PROCEDURE ..........................................................................................................47

    3.5 DOCUMENTATION ..................................................................................................48

    3.5.1 Claims .................................................................................................................. 51 3.5.2 Evidence .............................................................................................................. 56 3.6 CONCLUSION .........................................................................................................57

    4 VV&A OF TRAINING SIMULATORS .......................................................................58

    4.1 INTRODUCTION ......................................................................................................58

    4.2 VALIDITY OF TRAINING SIMULATORS ........................................................................58

    4.3 OBJECTIVE VALIDATION METHODS ..........................................................................61

    4.4 SUBJECTIVE VALIDATION ........................................................................................63

    4.5 DISCUSSION ..........................................................................................................65

    5 OTHER V&V AGENT TECHNIQUES AND TOOLS ................................................66

    5.1 V&V PLANNING ......................................................................................................66

    5.2 REQUIREMENTS ANALYSIS......................................................................................67

    5.3 EVIDENCE COLLECTION AND OTHER RESOURCE REPOSITORIES ..............................69

    5.4 DECISION MAKING ..................................................................................................71

    5.4.1 Cost/Benefit Analysis (CBA) ........................................................................... 72 5.4.2 Pareto Analysis .................................................................................................. 72 5.4.3 Paired Comparison Analysis........................................................................... 73 5.4.4 Grid Analysis ...................................................................................................... 74 5.4.5 Decision Trees ................................................................................................... 74 5.4.6 Plus/Minus/Implications (PMI) ........................................................................ 75 5.4.7 Force Field Analysis ......................................................................................... 76 5.4.8 Six Thinking Hats............................................................................................... 77 5.4.9 Multi-criteria analysis (MCA) ........................................................................... 79 6 ARGUMENTATION TECHNIQUES AND TOOLS ...................................................87

    6.1 SAFETY CASE TOOLS .............................................................................................87

    6.2 TOOLS FOR COLLABORATIVE SENSE-MAKING ...........................................................90

    6.2.1 Graphical support .............................................................................................. 90 6.2.2 Exploiting corporate memory .......................................................................... 96 6.2.3 Design rationale, argumentation, and wicked problems .......................... 98

    7 DISCUSSION AND CONCLUSION ....................................................................... 102

    7.1 PRESENT VALIDATION TECHNIQUES ..................................................................... 102

     WEU Unclassified This document is subject to the terms and conditions of the WEAG MoU Supplement JP 11.20 17-08-2010 Version 2.2 Page 7 of 109

    THALES JP 11.20-WE3100TECH2-D3102

    ANDIDATES FOR FUTURE TECHNIQUES ............................................................... 104 7.2 C

    7.3 OTHER ISSUES ................................................................................................... 104 8 REFERENCES ........................................................................................................ 106

     WEU Unclassified This document is subject to the terms and conditions of the WEAG MoU Supplement JP 11.20 17-08-2010 Version 2.2 Page 8 of 109

    THALES JP 11.20-WE3100TECH2-D3102

    Figures

    Figure 1: M&S development life cycle [MacKenzie et al., 2002]. .....................................18 Figure 2: Taxonomy of V&V techniques ...........................................................................20 Figure 3: Modelling Language Sample Screen ................................................................22 Figure 4: VV&A Flow Diagram [DMSO, 2001] ..................................................................27 Figure 5: VV&A for New M&S Development [DMSO, 2001] ............................................28 Figure 6: Results Validation [DMSO, 2001] ......................................................................37 Figure 7: Training Effeciveness .........................................................................................60 Figure 8: Taxonomy Life Cycle [P. Russom] ....................................................................71 Figure 9: Force Field Analysis Example [www.mindtools.com] .......................................77 Figure 10: Argument Structure (SHIP) ..............................................................................88 Figure 11: Argument Construction [KMI-TR-25, 1997] .....................................................94 Figure 12: Graphical Argumentation [QuestMap] .............................................................95 Figure 13: SEAS Knowledge Management ......................................................................97

     WEU Unclassified This document is subject to the terms and conditions of the WEAG MoU Supplement JP 11.20 17-08-2010 Version 2.2 Page 9 of 109

    THALES JP 11.20-WE3100TECH2-D3102

1 INTRODUCTION

    1.1 Work Element Overview

    Working Element 3100 is part of Work Package 3000 of JP 11.20. WP3000 shall deliver a V&V techniques state-of-the-art and guidelines for these techniques and their selection. WP3000 consists of three Working elements, WE3100 to deliver the V&V techniques state-of-the-art report, WE3200 to deliver guidelines for VV&A resources selection, and WE3300 to deliver the guidelines for VV&A techniques selection. The present report, TECH2, is the final version of the draft TECH1 deliverable, the aim of which is to deliver a V&V techniques state of the art report.

    The information in TECH2 is like TECH1 gathered and extracted from the Internet resources that were identified in our previous RESRC1 and in the RESRC2 report. Although we do not agree in every detail with their results and recommendations most input has come from DMSO resources. When studying the several relevant papers and reports available on the Internet it has been evident that the DMSO ―Recommended Practices Guide‖ is the current standard ―state-of-the-art‖ resource.

1.2 Abbreviations

    AFDRG Anglo-French Defence Research Group

    AI Artificial Intelligence

    DMSO DoD Modeling and Simulation Office

    ITOP International Test Operating Procedures

    M&S Modelling and Simulation

    NMSG NATO Modelling and Simulation Group

    SME Subject Matter Expert

    VV&A Verification, Validation and Accreditation

    WE Work Element

    WP Work Package

    WEU Unclassified This document is subject to the terms and conditions of the WEAG MoU Supplement JP 11.20 17-08-2010 Version 2.2 Page 10 of 109

Report this document

For any questions or suggestions please email
cust-service@docsford.com