Choosing the Best Automation Strategy for Electronic Systems Testing

Updated: Aug 30, 2019

As covered in my previous blog post, the advantages of automation are tremendous in the right situations. This post aims to determine what these right situations are, in order to understand when and how much to deploy the energy necessary for production automation.

Operational objectives of the transfer to production

Want to skip the objectives and head straight to the results ? Scroll-down!

Extent of test coverage

The extent of the test coverage you desire for your product is one of the major operational objectives determining how much automation it requires. The coverage is generally driven by the type of system you aim to produce. For example, a medical equipment necessitates a much higher test coverage than a low-end consumer electronics product. We can generally agree that the more a product needs a high coverage, the more energy should be expended to automate the test. However, this is not always right. However if we are talking strictly of the extent of automation as a percentage of tests automated vs manual, then it is the low coverage products who will have a higher percentage.

It is easy to accept that an FM radio can be fully tested by a computer and test equipment, without human interaction. However if I delivered to you a nuclear power plant and told you no humans participated in any quality or safety verifications on it, you would clearly doubt its test coverage and its quality. Simply put, very high test coverage products still need human involvement to reach their quality objectives, even if they require a lot of energy to automate.

Unit delivery throughput

A straightforward operational objective is the target unit delivery throughput of the production line. This objective is often stated in units per week and/or per month.

The most important factor affecting the throughput is the test time. A main objective of the test automation should therefore be to allow a production line to reach its throughput target by reducing the total test time to a minimum, by removing time intensive manual operations and efficiently chaining test activities and measurements.

Other factors increasing the throughput are:

A high yield Yield is primarily affected by the quality of the systems as they are manufactured prior to the tests. But a shoddy automation can drastically lower the yield and create false fails and unnecessary retests, which will cause the total test time to increase and the throughput to decrease.

Efficient debugging

Automation can help debugging by adding (or tweaking) tests to target debugging steps when a failure is detected. For example, it is possible to add measures taken at intermediary points to automatically investigate a failure of the measure taken at the end point of a sub-system. This targeting, as well as complete and standardized test outputs can increase the efficiency of the debugging phase and allow the turn-around of more units per week.

Full test cost per unit

Depending on the margin targeted for the product, a varying portion of the cost is earmarked to testing. The last main variable in the decision you face is therefore not the maximum per unit test cost itself - as this is a fixed parameter - but the allocation of this cost in the different activities and parts necessary to create the testbench.

Let’s first define the full test cost of a system as the sum of the following parts:

The cost of test labor

The cost of testing 1 unit (time * salary) * volume

  • including retests in case of less than 100% yield

  • including maintenance and support from engineers

  • including debugging

The cost of the bill of material

For all testbenches on the production line:

  • Test PCs

  • Test equipment (automated or not)

  • Mechanical fixtures

  • Golden units of other parts of the system or electronic cards emulating them

The cost of design, development and deployment

Non-Recurring Engineering (NRE) for all testbenches on the production line

  • Design of electronic cards and mechanical fixtures

  • Design and development of software tools

  • Design and development of the test sequences

  • Procedures, training and deployment

Total volume of production

To obtain the cost per unit we must divide the total cost by the target volume, which is our last operational objective affecting our decision. As we will see, the volume is the main driver of the choice we face.

Deciding whether to automate - an example

The decision on whether to automate the production of a system can be defined by the following sentence

Can you, with your automation plan,

  • Reduce the active test time, and thus labor cost,

  • While raising the BOM and development cost less than the amount saved on labor costs,

  • While allowing you to meet your coverage objective,

  • And to reach your unit throughput objective

A good way to go about it is to elaborate different strategies with different levels of automation which allow you to reach your targeted coverage. Then evaluate the costs and target optimizations that save time and money.

The Example

Let’s consider the following example. For its purpose, the operational objectives of the transfer to production are set as:

  • A target volume of 1000 units

  • A target cost per unit of < 40 $

  • A target delivery throughput of 100 units per week

  • A high target test coverage for a consumer electronics product, allowing the test to be fully automated and not require manual tests.

3 strategies have been evaluated, one not at all automated, one half-automated and one fully automated. For the sake of the example, let’s assume all three strategies meet the coverage objective, as establishing a failing strategy is not practical.

Cost of labor

The following table shows the cost of labor calculation as determined for our example.

  • The not-automated strategy uses a fully manual test. It’s tester cost per hour is higher because engineers are required to perform and supervise some of the tests. Note that it fails to meet the throughput target.

  • The half-automated strategy is based on a manual first half and automated second half. Test time gains are made, especially the active test time, leading to important labor cost savings.

  • The fully automated strategy removes all manual tests. Slight test time and cost gains are made.

Cost of the testbench's Bill of Materials

The next table lists the bill of material cost for all strategies as determined for our example. Notice that the half and not-automated strategies share the same BOM, the difference being the operator of the equipment for some of the tests (a tester for the not-automated strategy or a computer for the half-automated strategy).

The fully automated strategy involves different equipment and fixtures to allow the complete automation which explains the cost increase.

Cost of testbench development

The next table lists the cost of development of each strategy as determined for our example. The more complete the automation, the higher the investment in the development of the software tools, the test sequences and the mechanical fixtures to support them. However, procedures and training require more time when manual testing is involved.

Summary of costs

This last table recaps the sub-costs and calculates the total test cost per unit.

As we can see from the above table, the half-automated and not-automated both meet the test cost target, while the fully automated does not. However, as calculated above, since the not-automated strategy does not meet the target throughput objective, only the half-automated strategy is a valid one.

Even if the not-automated strategy had reached required throughput, the cost savings made by the half-automated strategy would have led us to choose to automate.

Although the end cost looks similar, it is important to note that the half-automated strategy also leads to higher quality standards due to the standardization of procedure and results brought by the test software.

By iterating the target volume parameter, we can calculate that for the fully automated strategy to reach the target cost of 40 $, the volume would have to be > 1515 units. For it to become the cheaper strategy, and therefore the selected one, the volume would need to be > 8110 units.

As I hope I have demonstrated, the decision of whether to automate your production line should be made with all the operational objective variables involved known and accounted for. It is a decision that cannot simply be made tentatively with preconceived intentions.

Tack’s experience in production test automation can help you to make the right decision on the amount of energy to invest in the automation of your production line. If you would like to discuss in more details on the variables involved in that decision, or if you would like to inquire about our services or our expertise, please leave us a message at