DoDI 5000.98 and DoDM Information
Explore the latest updates surrounding the rollout of DoD Instruction 5000.98 and  5 related DoD Manuals:
  • Full Spectrum Survivability and Lethality
  • TEMP/TES
  • M&S VV&A
  • AI-Enabled Autonomous Systems
  • Software

DoD 5000.98 establishes policy, assigns responsibilities, and prescribes procedures for operational test and evaluation (or OT&E), as well as live fire test and evaluation (or LFT&E) of DoD systems and services acquired via the Defense Acquisition System or via other non-standard acquisition plans. 
 
5000.98 DoDI impacts all DoD Components, including the Office of the Secretary of Defense, Military Departments, Joint Chiefs of Staff, Combatant Commands, the Inspector General, Defense Agencies, & all other DoD entities. It covers all DoD systems obtained through the Defense Acquisition System, following any pathway outlined in the Adaptive Acquisition Framework. DoDI 5000.98 also covers systems under special access controls per Directive 5205.07, and  non-standard acquisition systems like the Missile Defense System. This broad scope ensures comprehensive coverage across various DoD activities and  systems, promoting standardized & effective acquisition practices. The 5000.98 DoDI supersedes the previous 5000.89 DoDI about operational & live fire testing & evaluation.

 DOT&E welcomes your questions and comments.  Your insights and feedback are crucial in establishing a clear path forward for the T&E enterprise.
DoD Instruction (DoDI) 5000.98
DoD Manuals (DoDM's)
Additional Information
DoDI and DoDM FAQs
  • Process Oriented Questions
    ♦  Is integrated testing a priority for DOT&E?
    DOT&E strongly encourages integrated testing, and it is core to the new policy. The new policy requires that the T&E WIPT consider all available information in the planning of OT and LFT and the use of such data in operational and live fire evaluations. DOT&E is coordinating closely with OSD (R&E) DTE&A to develop guidebooks in areas where integrated testing is critical, including M&S VV&A, cyber, and other critical areas of the TEMP/T&E Strategy. The guidebooks will provide information on how to implement integrated testing so that it is seamless to the T&E enterprise.
    ♦  To what programs do these apply? Are these changes retroactive? Is there a grace period?
    No, these changes are not retroactive. The changes will apply to new programs and programs that have TEMP updates in the future. For programs that are in the TEMP staffing process DOT&E will work closely with the program to ensure that the intent of the policy is implemented and that any changes needed are provided in such a manner that we do not significantly impact program cost, schedule, performance. DOT&E has developed an implementation phasing that focuses on leveraging elements of the policy that are easiest to implement now and that will bring the biggest impact to T&E of programs. Guidebooks are currently under development for aspects of the policy that are more forward-leaning. We will work with the community to time phased policy adherence with the availability of guidebooks and examples. We welcome programs to engage with DOT&E to develop examples that illustrate the new policy and shape the implementation.
    ♦  Why did DOT&E decide to use DoDMs as opposed to the historical approach of using guidance memos issued by the Director?
    Connecting DoDI with supporting DoD Manuals enables the formalization of guidance across the T&E community (not just oversight programs)
    ♦  Facilitates standardization across the DoD with greater authority stemming from the formal coordination process.
    ♦  Broadens coordination and stakeholder involvement; developing a DoDIs and DoDMs involves broad coordination, leading to more comprehensive guidance, and transparency.
    ♦  Enhances longevity as DoDIs and DoDMs provide stable guidance across leadership tenures.
    ♦  Clarifies roles and responsibilities; DoDIs and DoDMs specifically assign roles and responsibilities, making it easier to hold parties accountable for compliance.
    ♦  How will the OTAs and program offices understand DOT&E’s new or changed expectations?
    The DoDI, DoDM, and supporting guidebooks provide a structure for policy and guidance. The DoDI focuses on strategic intent and key stakeholders. The DoDMs move toward how to implement the policy in critical challenge areas. DOT&E will be issuing guidebooks to further support the T&E community in translating policy into practice. These guidebooks will provide clear practice expectations for OTAs and programs. They can also provide more specific guidance that is updated to keep pace with the current state of the technology, reflect the evolving needs of the T&E community, or the development of new examples and tools. OTA’s and program offices can stay up to date on policy developments through the DOT&E website for the 2024 policy update. Frequently asked questions – such as this one – will be routinely updated on this website. Tutorial and briefing materials will be posted here as well as well as links to all the guidebooks as they are developed.
  • DoDI 5000.98 Questions
    ♦  Does the "throughout the acquisition lifecycle" language effectively mean that we will no longer be taking programs off oversight once their OT&E program is complete?
    No, it does not mean programs will never come off oversight. Programs will remain on oversight if there are planned substantive changes to the program’s capabilities through future system updates or the threat changes. If the operational environment, threats, or targets change significantly, the program should re-evaluate survivability, lethality, and potentially effectiveness. DOT&E prioritizes the allocation of resources in alignment with overall DoD strategy. As we start to see AI-enabled programs and other emerging technologies, DOT&E may prioritize keeping them on oversight until the update T&E process is proven.
    ♦  Does the updated policy mean all programs go on live fire oversight?
    No, DOT&E will maintain separate oversight lists for OT&E and LFT&E. DOT&E’s operational evaluation has always included the evaluation of effectiveness and suitability, while including survivability and lethality if applicable. All programs on OT&E oversight need to meet this minimum threshold.
    ♦  How does DOT&E envision OTAs maximizing Contractor Testing (CT) during OT&E, and what guardrails will be established to prevent overstepping (or biasing operational evaluations)?
    The changing nature of systems means that the operational evaluation must consider relevant sources of information outside of dedicated operational tests to inform operational evaluations.

    DOT&E envisions data and information from CT being useful in planning operational tests and informing operational evaluations in several ways (although there may be others):
    ♦  Informing the selection of scenarios for dedicated operational testing. This may include identification of significant factors, the identification of edge cases, or areas of high uncertainty that can be reduced in OT.
    ♦  Providing more comprehensive testing of software and/or AI components to inform operational evaluations.
    ♦  Providing information on system integration testing.

    Contractor Test data like Development Test data can be used for operational assessment if it is validated by DOT&E as representative. It is important to ensure that operational evaluations reflect the operational environments with operationally realistic data. This may (or is often not) be the case in CT. The policy states that reporting activities will use statistical inference methods, to include Bayesian models to incorporate relevant information, while reducing the risk of biasing reported results.
    ♦  Please explain what Figure 1 “T&E across the acquisition life cycle” is showing.
    Figure 1 emphasizes early engagement by OT&E and LFT&E organizations, codifying the importance of “shifting left” and incorporating operational realism early in the acquisition lifecycle. It highlights that data from subcomponent, component, subsystem, systems, and systems-of systems all provide information toward DOT&E’s assessment of operational effectiveness, suitability, survivability, and lethality. A common data repository should support T&E activities across the acquisition lifecycle.
    ♦  How will DOT&E work with the services to prioritize development of Full-Spectrum lethality and survivability test capabilities to include M&S given major gaps in the ability to currently test or evaluate?
    DOT&E will take a phased approach to implementing full spectrum survivability and lethality. The initial phase will focus on what we can do now without new tests or test resources. The intermediate phase will focus on enhancing current testing. The final phase will focus on novel tests. Resourcing, planning, and carrying out tests with more than one attack vector will require communication and collaboration among test personnel. Likewise, combining results across disciplines will require tighter coordination of analysts. DOT&E is advocating for programs to think about testing against each relevant threat vector in the spectrum and prioritize testing through risk-based approaches. DOT&E plans to develop a cross-service working group to develop a baseline for what is currently possible for full-spectrum testing, identify gaps for coordinated kinetic and non-kinetic threats, and prioritize needs. DOT&E will work with TRMC and across the Services to develop plans for implementing this part of the guidance. We expect this will take time to implement correctly but is necessary to reflect the current landscape of warfighting.
  • DoDM 5000.100 (TEMP/TES) Questions
    ♦  What type of events should be listed in Table 2, Section 2.3? What is the cut off or level of detail for this section?
    All DT, IT, OT, and LFT events that will inform the evaluation of operational effectiveness, suitability, survivability, and lethality should be included. These events will also be documented in the IDSK. The risk-based level of test assessment should reflect what an adequate independent OT&E should include.
    ♦  Where does the evaluation framework fit into the TEMP, Section 2.1, 2.3, or somewhere else? Or does the IDSK replace the evaluation framework?
    The IDSK contains the key test and M&S data tied to decisions across the acquisition lifecycle. This includes all relevant information previously captured in the developmental evaluation framework (DEF), the operational evaluation framework (OEF).
    ♦  Can examples be provided of how Risk-Based Level of Test Assessment effectively scopes OT&E, considering the requirement for production-representative testing under realistic combat conditions by typical military users?
    DOT&E is currently working on a guidebook, which will provide examples. DOT&E is also looking for program partners to pilot the new guidance and help shape best practices for implementation. There are numerous ways that tests across the acquisition lifecycle can include operational realism, which could help identify problems early, but also inform a reduced scope of dedicated OT&E.
  • DoDM 5000.102 M&S VV&A Questions
    ♦  Does DOT&E need to approve M&S strategies or V&V plans?
    DOT&E does not need to approve standalone M&S strategies or V&V plans. However, DOT&E needs to review those strategies and plans as part of the TEMP/TES and test plan approvals to ensure that adequate data will be collected to enable DOT&E’s independent evaluation.
    ♦  What about V&V plans that come after the TEMP approval?
    The policy emphasizes early engagement to set clear expectations. Ideally, all V&V plans should be developed in concert with the TEMP/TES to ensure that adequate test resources are available to support the V&V plan and data collection strategy. In cases where this is not possible, the V&V plans should be made available for review prior to test plan approval. DOT&E has the authority to disapprove any TEMP/TES or test plan if they have not first reviewed relevant aspects of a V&V plan.
    ♦  Do we have guidance on VV&A of models based on data?
    The standards for VV&A have not changed and previous DOT&E guidance remains relevant. Currently, DOT&E maintains supplemental M&S guidance and resources, including best practices, case studies, and a handbook, on its guidance page on the DOT&E extranet site. The forthcoming M&S VV&A guidebook will provide additional explanation and examples. Follow-on video modules and case studies will emphasize rigorous statistical methodologies and uncertainty quantification. The guidance update focuses heavily on how statistical methods and experimental design can be used to make data-driven arguments for validation. However, direct data comparisons will not always be possible. In those cases, programs should leverage data and extrapolation to the extent possible while quantifying uncertainty in untested conditions. They should also leverage subject matter expertise to understand if models are producing reasonable outcomes.
  • DoDM 5000.99 Full Spectrum Survivability
    ♦  What problem is Full Spectrum Survivability and Lethality trying to solve?
    The ability of systems and operators to defend themselves against a realistic adversary has always been a part of OT (combat representative environment) and has included the full-spectrum threat space. However, warfighting is evolving into a coordinated kinetic and non-kinetic threat space, therefore we must consider the impact of coordinated kinetic and non-kinetic threats on system survivability and all the attack vectors we can employ from a lethality perspective. Our ability to fully evaluate has been hindered by test capability and tools. The full spectrum survivability guidance focuses the need for new test capabilities and can ideally drive necessary resources for inclusion throughout testing.
    ♦  How does DOT&E plan to implement full spectrum survivability and lethality (FSSL), how will adequacy be determined?
    DOT&E will take a phased approach to implementing full spectrum survivability and lethality. The initial phase will focus on what we can do now without new tests or test resources. The intermediate phase will focus on enhancing current testing. The final phase will focus on novel tests. DOT&E will work with the test community to determine what is an adequate survivability or lethality evaluation as full spectrum test capabilities develop and evolve.
    ♦  Will the Live Fire Strategy or more importantly, the Alternative Live Fire Strategy include ALL Survivability testing?
    Full Spectrum Survivability and Lethality (FSSL) is the assessment of a system’s survivability and lethality. Operational tests, developmental tests, live fire, integrated tests, real world events, and other data sources are part of FSSL. FSSL and LFT&E are not the same, rather LFT&E is a subset of FSSL.

    The process of requesting a Waiver (Alternative Live Fire Strategy) only applies to those parts under LFT&E. As defined, a Waiver is granted if a program can show the cost of Full Up System Level testing is “unreasonably expensive and impractical.” The NDAA language expands survivability and lethality testing to include non-kinetic threats. Therefore, it is possible that a waiver could be requested for non-kinetic threat tests if the tests are destructive in nature or unreasonably expensive.
    ♦  Who is responsible for cyber testing based on the new policy?
    OTAs will continue to be responsible for conducting cooperative vulnerability and penetration assessments (CVPAs) and adversarial assessments (AAs) to support the evaluation of operational effectiveness, operational suitability, and survivability against cyber threats. A clarification will be provided in forthcoming guidebooks issued alongside the DoDI and DoDMs. The CVPA and AA will be complemented from cyber testing conducted across the acquisition lifecycle.
    ♦  Who are the LFT&E organizations?
    Live fire test and analysis organizations are numerous. Therefore, examples of organizations responsible for the conduct and M&S are: The Army has the Aberdeen Test Center, DEVCOM Analysis Center, and many others. The Air Force uses the 704th Test Wing, Air Force Life Cycle Management Center Combat Effectiveness &
    Vulnerability Analysis Branch for analyses, and other organizations to support. The Navy has Naval Air Warfare Centers at China Lake, Patuxent River, Indian Head, Lakehurst.