Forgotten Elements of Systems Engineering – Test and Evaluation

Complexity is in systems engineering’s very nature. As systems engineers, we create solutions to complex problems. So, from time to time, we forget important elements of the system’s lifecycle. We might even believe we are remembering them, but we do not give these elements the time and attention they desperately need. In part one of the three-part series of “The Forgotten Elements of Systems Engineering,” we will discuss how many systems engineers fail to remember Test and Evaluation (T&E) planning.

T&E is a part of the Verification and Validation in a system lifecycle. It is the plan to execute test cases, test plans, test procedures, modeling and simulation, and report the issues found. Unfortunately, many develop the T&E late in the lifecycle, often starting halfway through. Stakeholders, such as the test developers, do not interact in time to understand the system’s needs. They make the perfunctory Test and Evaluation Master Plan (TEMP) early in the project. The TEMP contains countless TBDs, but the plan does not evolve with the system’s changes. When the time comes to test, they often have to start with little or nothing in the way of a real T&E plan.

NDIA suggests “incorporating T&E expertise early in the acquisition cycle (ITEA Journal).” Not creating a plan for the T&E early in the lifecycle causes a lot of problems, mainly schedule delays and increased costs. If test requirements (detailed metrics used to determine performance and operational suitability) and plans (which include facilities, equipment, integrated schedules, and costs) are developed early, users can validate the systems. If it passes this criterion, it will meet its operational needs. Showing the users how the system is validated provides confidence that testing has been sufficient to take the risk of introducing a new system to the operational environment. Bringing in a new system that doesn’t work costs time and money, but in many cases, it can even cause loss of life.

It’s important for stakeholders, especially the test developers, to interact early in the system’s lifecycle to better understand all the test needs. At a company I used to work with, the test developers would complain to me that systems come in for testing without any notice. The test developers had no understanding of the system, sometimes the system wouldn’t even come with requirements, and the test developers were expected to get straight to testing and evaluating this foreign system. This is unacceptable, and the test developers should be refining the TEMP throughout the lifecycle.

If you look at the V-model, you can see that each step going down the left side of the V-model corresponds to the right. The entire right side is devoted to testing. The V-model encourages T&E to receive output in each phase of the lifecycle. Due to improved modeling languages and model-based systems engineering (MBSE) tools, T&E is much easier to perform earlier and simultaneously with other steps in the lifecycle. LML was designed to support the complete lifecycle, so aspects of that are considered in the language. Most of the entities are common to other phases (i.e., Assets, Actions, etc.). Perhaps the greatest influence T&E had on LML was in the Measure of entity attributes. If you look closely at them you will see information that is extremely useful for T&E. The “Value” can be the measured value, while the other values (Projected, Threshold, and Objective) are often estimated values. LML also included Improvement Direction and Tolerance, as information you would want to measure. The Measure entities relate to Time and Location. A number of the Labels are also T&E related (COI, MOE, MOP, MOS, TPM, etc.). The depth of LML is one of the many reasons Innoslate uses LML as one of its main modeling languages.

Having the language entities needed to capture test results, as well as the processes, procedures, and equipment, should improve the inclusion of T&E in the early phases of design, as well as capture the data. Simulators such as the Monte-Carlo and Discrete Event simulators enable you to assist in testing physical aspects of the system.

T&E is one of the most forgotten elements in systems engineering. Forgetting T&E can seriously delay a system’s project and dramatically increase the costs and risks. Remember these three rules to maintain proper T&E processes: 1) Use modern modeling languages and software tools to start planning early in the system’s lifecycle. More defined modeling languages allow for modern MBSE tools that can help you create test plans, test reports, and test procedures. Tools like Innoslate have built-in automated modeling. 2) Involve all the stakeholders, especially the test developers throughout the lifecycle. 3) Evolve the T&E plan with the system’s changes.

If you can remember these rules, you will have a successful system project!

Take a deep dive into our extensive resource library! Access whitepapers, user stories, and informative blogs to gain in-depth knowledge about cutting-edge practices in the industry.